Agree that the k8s tax, as described, is a huge issue. But I think the biggest issue is immaturity of the ecosystem, with complexity coming in second. You can at least throw an expensive developer at the complexity issue.
But when it comes to reliable installations (even helm charts for major software is a coin flip in terms of whether they’ll work), fast moving versioning that reminds me of the JavaScript Wild West (the recent RBAC on by default implementation comes to mind, even if its a good thing), and unresolved problems around provider-agnostic volumes and load balancing... those are headaches that persist long after you’ve learned the difference between a replicaSet and a deployment.
To further this point about the ecosystem, and this is AWS specific. You need, or have needed, to install a handful of extra services/controllers onto your EKS cluster to get it to integrate the way most would expect with AWS. Autoscalling? Install and configure the autoscaler. IAM roles? Install Kube2IAM. DNS/ALB/etc etc? etc etc etc.
After a slog you get everything going. Suddenly a service is throwing errors because it doesn't have IAM permissions. You look into it and it's not getting the role from the kube2iam proxy. Kube2iam is throwing some strange error about a nil or interface cast. Let's pretend you know Go like I do. The error message still tells you nothing specific about what the issue may be. Google leads you to github and you locate an issue with the same symptoms. It's been open for over a year and nobody seems to have any clue what's going on.
Good times :) Stay safe everyone, and run a staging cluster!
But when it comes to reliable installations (even helm charts for major software is a coin flip in terms of whether they’ll work), fast moving versioning that reminds me of the JavaScript Wild West (the recent RBAC on by default implementation comes to mind, even if its a good thing), and unresolved problems around provider-agnostic volumes and load balancing... those are headaches that persist long after you’ve learned the difference between a replicaSet and a deployment.