I see a lot of comments to the effect that this is a complicated setup, especially for one person. I really think most people need to re-evaluate their stacks without all of the marketing of cloud providers and gigascalers.
This setup is very similar to the startup that I run. We have used k8s from day one, despite plenty of warnings when we did, but it has been rock solid for 3 years. We also decided to run Postgres and Redis on k8s using Patroni + Sentinel and the LVM storage class. This means we get highly performant local storage on each node and we push HA down to the application.
Was there a learning curve? Yes, it took a solid week to figure out what we were doing. And we've had the odd issue here and there, but our uptime has exceeded what we would have had with a cookie cutter AWS setup, and our costs have been a fraction (including human capital costs).
I am solo builder and runner of https://atomictesselator.com and I have used k8s from day one. I also agree with you, it's been rock solid, and the ability to horizontally scale so easily is great. I dont even know which physical machines in my rack the jobs/pods are running on, I just have a high level dashboard (lens) and I can see the CPU+GPU utilization hit 100% as everything self balances, it's great, this has been very helpful scaling my GPU workload which has increased a lot recently
Same. The simple fact that every PR can deploy a full stack, including RDS and managed Redis if desired, automatically in it's own namespace with proper DNS pointing to services is a massive win for us. As in, if you label the PR then it all happens automatically and then it all shuts down automatically when the PR is closed.
- Every service is deployed via a Helm chart and using containers
- GitHub actions build the container and deploy the helm chart
Some of the details that matter:
- ACK is used to create AWS services (RDS, Redis, etc.) via Helm charts (we also have a container option for helm charts as it's faster and less expensive)
- External Secrets is used to create secrets in the new namespace and also do things like generate RDS passwords
- ExternalDNS creates DNS entries in Route53 from the Ingress objects
- Namespace name is generated automatically from the branch name
- Docker images use the git hash for the tag
Some things that are choices:
- Monorepo although each service aims to be as self-contained as possible.
- Docker context is the git root as this allows for a service to include shared libraries from the repo when creating a container. This is for case where we break the previous rule.
K8s is overkill though. Google cloud run gives you the good bits with less work and it's still reasonably priced. Azure container apps in theory should be similarly good, but it's kind of buggy and poorly engineered.
CloudRun with Django works well, but for a complete setup, you need to configure, manage and pay other services, too, like Cloud SQL for PostgreSQL, Memorystore for Redis, API Gateway, Observability, Cloud Build, Artefact Registry, Cloud Key Management, IAM, Pub/Sub, CDN, and Cloud Storage. When you have a live service, you probably also want to pay for Google Cloud Customer Care. The same is true for AWS and Azure.
Depending on training and experience, it may be easier to setup a GCP than a Kubernetes based solution, but it's still not exactly trivial. And, once a fitting Kubernetes platform is up and running, it's almost trivial to add new services, as the article describes.
I think, even in 2024, and even for a one-person business like the authors', starting with Kubernetes is not a bad decision. I'm currently building my own Kubernetes based platform, but on a less expensive service (Hetzner Cloud). I use a separate single-node Kubernetes "cluster" with Rancher for management, and ArgoCD for continuous deployment from git. Currently I only have a single-node "playground" cluster up and running for my "playground" workloads, to save costs, but I will upgrade this to a proper HA cluster before a service goes live.
Later, I can still upgrade to GCP, or in my case probably Azure. Until then, I don't need to worry about large cloud bills, which is also a good thing.
In my experience, people who think k8s is really hard to user are either: a) someone who had k8s forced on them without getting a chance to learn it (or had no interest in the first place) and was stuck with its complexity, or b) has never actually used k8s and is just parroting what they've heard.
The former situation does suck. K8s is amazing and once you understand it, it is easy to work with. But if you haven't learned the concepts and core resources, it will definitely appear as a black box with a ton of complexity.
But I think for many of us who have used Kubernetes a lot, it is a no-brainer in a lot of situations as it doesn't matter which cloud provider you're using (for the most part), you get a common and familiar interface.
I usually find that it's B. The number of professional programmers who don't have the slightest clue how a database actually works, much less the ability to just spin up a container containing one, is shocking.
Sadly I think that extends to most people about most things. I’m certainly guilty of stating something I’ve heard but don’t actually know from time to time (though I do make a conscious effort not to).
It is funny (or depressing, depending on your point of view) to catch people doing this by asking them when was the last time they did X or how they know whatever they’ve stated and then seeing the gears turn.
I'm not saying it's an impossible monster but I can setup and deploy a project from scratch with cloud run in ~5 gcloud cli commands while being somewhat protected from my own ignorance.
Caution: Working with CloudRun with ignorance on any level can lead to very large bills: https://news.ycombinator.com/item?id=25372336 - The article discussed there is a good cautionary tale: https://archive.is/D5qc8 - I prefer to explore and experiment on a less scalable solution which is usually also less scalable in terms of costs.
From the fact that the LoC count of the implementation is beyond "millions".
How about this equivalence: I appreciate how extremely sophisticated GCC is, and the very well optimized output it generates. It is still is 100x more complex internally (and thus, error prone, buggy, more complex to modify when needed) than TinyCC, for instance.
Um, ok. I think you're massively underestimating the LoC of the services mentioned (I would bet significant sums that cumulatively it would exceed a basic k8s deploy).
I'm also not really sure what the relevance of LoC here is though? The Linux Kernel is a large codebase...but surely you don't object to using that?
This setup is very similar to the startup that I run. We have used k8s from day one, despite plenty of warnings when we did, but it has been rock solid for 3 years. We also decided to run Postgres and Redis on k8s using Patroni + Sentinel and the LVM storage class. This means we get highly performant local storage on each node and we push HA down to the application.
Was there a learning curve? Yes, it took a solid week to figure out what we were doing. And we've had the odd issue here and there, but our uptime has exceeded what we would have had with a cookie cutter AWS setup, and our costs have been a fraction (including human capital costs).