As a one-man team operating a 10+ year old SaaS product, I’ve done two things that have helped keep things sustainable. The frontend continues to be server generated. PJAX style partial page updates is (mostly) dynamic enough. The maintenance burden of operating a JavaScript frontend is too high to justify for a 1-2 person team.
The second thing I’ve done is avoid containers. VMs work fine for many types of applications. That’s especially true if provisioning and configuration is mostly automated. A product like Google Cloud Run looks interesting for a few low-volume backend services that I operate. Containers are a good fit there because it simplifies operations rather than increases complexity. The core infrastructure will remain VMs for the foreseeable future.
I began doing Ruby and Ruby on Rails (RoR) development in 2006. Ruby had been around for awhile but the growing popularity of RoR was driving a lot of changes in the ecosystem in those years. It was a fun time but I was working for a large company that allowed me the time to keep up with everything. I see today’s frontend and container ecosystems in the light of that experience. I’m now more content to wait it out and let the dust settle as much as I enjoy learning new technology.
I'm glad you shared this. I've been reading the comments here with much interest but simultaneously also a sense of impostor syndrome, for I've been running my one-man SaaS on generic LAMP since 2005, and aside from adopting jQuery early on, I haven't much touched any other piece of exciting tech that's come out since.
My old and boring stack works just fine on the VMs that it runs on, and and continues to support me and my family financially with little to no downtime and decent work/life balance.
Of course I'm aware of the cons of programming in PHP versus other languages, but after 20 years of coding in it, I like to think that my code isn't all that bad.
Still, I'm reading some comments here and feel like I'm waaay behind the tech curve with my use of such an old stack, and am somewhat scared to admit it.
Doing your own thing while supporting your family is awesome. It's a difficult thing to do.
The SaaS I run isn't mine. Back when you were starting your SaaS, I was probably spending all of my time chasing the latest shinny tech :) Some of that was worthwhile but some not. The experience taught me to slow down a bit. Sometimes I wonder if I've slowed down too much. We'll see.
> feel like I'm waaay behind the tech curve with my use of such an old stack, and am somewhat scared to admit it.
If it pays your bills and gives you the work life balance you want, why do you care?
Running on the perpetual treadmill of cutting edge tech (especially with JS and containers where everything new is old every other month) is nothing to strive for. I know plenty of engineers who are very up to date on the latest trends but have no personal lives or families and are basically married to their job. I don’t envy them, I envy you.
Nice work. Your SaaS has been providing you income for 15 years.
Tell us about your SaaS. What were your challenges early on? What were your hardships? And what were your lessons learned? What would you do differently?
I'm a private guy. But I will say that it's a Greek startup, birthed in my English dorm room (Comp Sci grad; taught myself programming at age 11 by reverse engineering GORILLA.BAS; shout-out to all you DOS 6.22 babies #486dx2 #duneiichani #zerocool #snowcrash #bbs #qbasic #mode13h #nehe #0x5f3759df #lamp #vi).
> What were your challenges early on?
From a life standpoint, it would have to be student loans. But in crisis lies opportunity, because by working to pay these off, I programmed database integration layers between a bunch of local companies that ultimately led to a sweet SaaS system.
From a technical standpoint, there were many fun challenges, such as 2,000+ days of uptime with multi-master replication. But the biggest challenge was probably getting an early version of Quixote and a cranky SOAP protocol to talk to a bunch of SAP EDRs; that's when I learnt people skills.
> What were your hardships?
The usual stuff... paying the rent, feeding myself, no free time. It was all systems go back in the day, and dating wasn't really an option; but I'm thankful for lots of genuine friends, who welcomed my intermittent visits.
From a business standpoint, the biggest hardship was my constantly wrestling with the idea that I could do anything but I couldn't do everything. Once I concluded that exercise I found peace, and then everything else just clicked into place.
> And what were your lessons learned?
1. There are multiple SaaS strategies. One such strategy is to go super niche. Emphasis on super.
2. Going super niche doesn't scale, but it will allow you to prioritise free time - whilst living a steady, balanced lifestyle.
3. To go super niche, build your SaaS in such a way that you can run it from your mobile phone.
4. Always keep an eye on your phone, so that when the need for customer comms arises, it flows with elegant efficiency.
5. The quality of your communication with your customers is your competitive edge. Excel at this to justify your pricing.
6. Optimise all your business processes around minimising your phone's notifications - the goal is to maximise your free time.
7. At your leisure, routinely check in on your customers for a friendly catch-up. This enhances your competitive edge and bargaining power.
8. Always add value to your customers, and if you can't add value, walk away. You'd be surprised how much you can earn and still add value.
9. Enjoy time. Discover who you are. Explore life with your family and friends. Exercise your body and mind outdoors.
0. Help others. Volunteer your skills and resources. I advise several entities and it brings me great joy.
> What would you do differently?
Perhaps I'd have picked a name that was easier to recall, and easier to spell.
But when you go super niche, it doesn't really matter what name you pick.
You didn't seem to get much of a reaction, probably because of your recommendations to go with proven (read: old & boring) tech, but I appreciate it. Front-end churn is real and, coupled with rapidly evolving and ever-abstracting containerization approaches, staying on top of it is a full-time job, best done as an employee.
FWIW, I disagree with GP. I'm more productive in React than in SSR, and I'm much better at code design and architecture with it. React is pretty much old and boring by now.
The apparent difference in our situations is that you know React and everything that comes with it. It seems like that'd be a good choice for you.
I'm not proficient in any of the currently popular frontend architectures. Gaining that proficiency and moving a 10+ year old product to a SPA type architecture would be a massive investment. I enjoy learning new things and pushing things forward but I can't justify that investment when it comes to the frontend. Instead, I invest in other areas to move forward in.
Yeah same here. I did years of generating JS heavy frontends with PHP for Ajax apps. Once I learned to keep all data on the server and functionality on the client I’ve become so much more productive when slapping together an MVP with react (and any backend capable of spinning up a rest, though I prefer python or go at this time.)
If I may chime in here, I would humbly suggest that it is best done by whoever enjoys it.
Sometimes that might be your employee, and sometimes that might be you.
I guess it depends on where your passion lies, and what your objectives are?
I imagine that achieving one's financial ambitions in a relaxed and enjoyable work environment is great - irrespective of which part of the stack one enjoys tinkering with the most.
> The second thing I’ve done is avoid containers. VMs work fine for many types of applications. That’s especially true if provisioning and configuration is mostly automated. A product like Google Cloud Run looks interesting for a few low-volume backend services that I operate. Containers are a good fit there because it simplifies operations rather than increases complexity. The core infrastructure will remain VMs for the foreseeable future.
VMs (even self-updating ones) can be built easily using templates with tools like Packer.
And without containers, a lot of things can be done very simply, often more easily, using modern versions of systemd. You get ad-hoc namespacing, ro storage, resource (CPU/network/memory/I/O) limiting, CPU pinning, private networking, logging and much more by the power of a single line more in the .service file. Throw a tight selinux configuration on top of it and you got a pretty good per-service, secure "container" on your system. Additionally, you get smart .service dependency handling, service auto-restarts, notified restarts if you care to implement it, scheduled tasks, and so on in a multiprocess environment that makes you just sad thinking back to having to resort to stuff like supervisord.
After finally getting a SaaS product off the ground a few months ago, I cannot agree more.
I fell into the shiny object trap so many times before. This time, I stuck to what I know inside and out and jazzed things up where I could. Long live the monolith.
Honestly, having gone down this path a year back, just use Heroku for your backend. Maybe Netlify if you want to separate your frontend and backend.
The amount of complexity and interdependencies these "here's my stack" posts describe are always a huge cognitive overhead for running a one-person SaaS.
Heroku costs more, but there's a reason for it. If you're even remotely making money from your product, and are alone in running the show, it's a no brainer to drop a couple hundred dollars a month to eliminate 99% of your ops headaches: the documentation, the add-on's, the fantastic UI, decent reliability, and ease of operations are totally worth it. Literally the worst thing one can say about Heroku is the cost, and possibly, the inability to build certain complex architectures (which most single-person SaaS services aren't).
I recommend just writing a Docker container instead. That's the ultimately portability and you can run in much more efficient environments (Cloud Run, Fargate, DigitalOcean Apps, Fly.io) and still scale up to large Kubernetes-based clusters if/when you need it.
Docker containers avoid the lock-in of a particular platform or CLI, and they can also be used as cheap CI/CD with multi-stage builds within the container.
Docker means you're maintaining an operating system, watching out for security vulnerabilities, patching software. It's still work, and if you screw it up, you're hosed.
If you're on Heroku or Google App Engine, you deploy your application code. That's it.
Docker containers (on Linux) don't have a guest OS. They all share the same kernel and only the userland is different depending on which distro you choose for your base.
You can choose to build on a scratch/blank image just as easily, which means nothing except your app itself is in the image.
You can deploy Docker containers to Heroku. I've done this (just requires a simple heroku.yml file and heroku stack:set container). If the project grows to the size where Heroku's costs become prohibitive, I can easily switch over to Digital Ocean with some tweaks to the docker config.
And also Heroku. I'm not disagreeing, just saying Heroku becomes 'just another platform', and if it still looks expensive, you can go elsewhere, if it's cheaper/better in some way (I have no idea nor horses) then go with it.
My SaaS has grown from $50/month costs to over $2k/month costs on Heroku over the last few years and I'm always eyeing AWS and others as a cheaper alternative, but honestly the ease of mind Heroku gives me is more than enough to cover the costs. I've had a few "hugs of death" over the years when posts went viral and scaling up both in quality and number of servers was literally just a few mouse clicks and then I was back online. Other times, when e.g. Postgres fell over from heavy use, their support brought it back online for me free of charge (and quickly), and offered a list of helpful suggestions that I could do to keep it from happening again.
I see paying extra for Heroku like paying extra for a devops guy. If you're comfortable enough in AWS to handle things in the heat of downtime, it's probably worth moving because it's significantly cheaper. If you aren't, then Heroku might be a pretty invaluable resource even when growing into the upper echelons of cost.
Heroku is a great product if you want development speed right from the start. I have used them in the past, and it was a very nice experience overall.
For many projects this is more than enough. However, in my case I would be paying 2-3x more if I was using Heroku.
With Kubernetes, in case I wanted to deploy new projects or even spin up a "testing" env, I can use the same stack/cluster, and not have my costs increase.
I do realize that this is not for everyone. But for me it just made sense, and enabled me to work on new features faster than before.
I just wonder how much you're spending per month. Going from $200 to $600 in exchange for not having so many moving parts would be 1000% worth it to me. Usually the margin on SaaS businesses is in the 60% range, most of which isn't infrastructure, so it's a rounding error until you get to the size where you have full time people working on infra.
Taking my first GKE project (done essentially as two man team in the side, and then maintained at quite low level of work most the time)...
The vendor whom we replaced = somewhere above $5000/mo (just infra)
Our final, most costly, GCP spend, including support contract = about $2000/mo (this involves a bunch of inefficiencies caused by developers unable to live with container - or dynos)
Expected Heroku cost, based on their online calculator= starts at $3000/mo, probably more
Expected lower time needed developing on Heroku = negligible
Heroku is a god send for getting off the ground, however the biggest issue for us hasn’t been cost (it’s high, but worth every cent) it’s been server location.
We’re an Aussie company, and a large percentage of our customers are in Australia. Having servers only in the US and Europe mean that Australian customers speeds are slower than they should be, and private spaces are far too expensive to warrant it.
I support this sentiment so much. I've been building something in my spare time. You are one person, the value provided with Heroku and the time saved is worth the cost versus AWS, at least for a project of this size and scope. It would take a lot more customers and capacity on the server to justify the ROI.
I've been spending some time looking at moving to heroku, but having trouble figuring out how to use it for a real production for deploy for "only" a couple hundred dollars a month.
Is that what you're doing? "Standard" dynos and just a few of them are working for you, I assume, for that budget? In my tests, with a Rails app, "standard" dynos are looking surprisingly slow, possibly unacceptably so. Very curious to hear about other people's heroku formations if anyone wants to share. What platform you're on (Rails or something else), and what your response times look like, would also be interesting.
Checkout Hatchbox.io, just started using it myself. You basically pay a flat monthly rate for their Heroku-esque management service and then pay for the servers on your provider of choice separately (e.g. DO, AWS, etc).
Are you finding it mature enough to "just work"? The reason someone pays for something like heroku is they put a high value on "just work", I am really not looking to be troubleshooting weird bugs or figuring out weird edge cases in my ops, that's the whole point here!
I'll take a look, it definitely looks interesting, thanks!
Hard to say because I've only been using it for a week, but so far it has just worked (which is what I am looking for as well, I don't have the time or patience for anything less).
I've been looking for something between Heroku (too expensive, too many features) and Dokkku (too much hassle, I have to maintain my own security and patches) and this looks like exactly the thing I wanted, except it looks like it's only for Ruby? I use a combination of Python, Elixir, and Haskell, so I'd love it if they did support other langs.
Take a look at digitaloceans new app platform or render.com. Render allows you to run distributed Elixir, which is nice. And with digitalocean, if you outgrow app platform you could just move to their hosted k8s.
These both are k8s under the hood but with everything extracted away to a heroku level, but much cheaper. Additionally, they both provide static site hosting like netlify.
With a standard dyno, you get something like 1/15th of a core on the underlying EC2 instance. IMO, they're best for workers where latency isn't an issue.
For user-facing work, I use performance dynos.
Heroku _is_ expensive, but in exchange, you don't have to worry much at all about the ops side of things.
YMMV, it may not be suitable if you need to be profitable each month and can't afford to spend the baseline cost of, say, $1,000/month.
Thanks. I agree that heroku is an amazing service, a really impressive product.
But once you are using a single performance dyno, you are unlikely looking at a bill that's only "couple hundred dollars a month".
Thanks for confirming you use performance dynos for user-facing stuff -- I thought I was going insane discovering that it didn't look like standard dynos were suitable for that for me -- is "everyone else" using them though? The heroku docs imply they are indeed... standard (that's the name), and the performance dynos are for unusual performance needs.
Which is not what it was looking like to me.
Based on my current investigations, I agree being prepared to spend $1000/month is a better back of the napkin to-start-with estimate.
Which, sure, is quite possibly still a value compared to the number of hours you'd be spending setting up and maintaning something else. Quite possibly! But it's not "a couple hundred dollars a month".
For the one man projects I'm thinking of starting there's a good chance I will make at best 100 a month for the first couple months, having probably only one customer for that time.
Agree on Heroku as well - the cognitive overhead they help take care of is well worth the money. There are other commenters saying "but it's expensive!", but your comparison should be Heroku vs. a Devops person (or yourself), not Heroku vs. <cloud_service_that_you_manage_yourself>.
My one complaint is that their support seems to have gotten worse over the years - they used to have great support engineers across the board, but have hired poorly recently, specifically in off-shore markets, so depending on what time you send in a ticket request, you may get someone who doesn't fully read your ticket or comprehend the situation, and just fires back a canned reply.
This usually gets solved if there is enough back-and-forth as they then seem to escalate it properly, but I have noticed that the initial reply to support tickets are not as good as they used to be.
It takes me about 2 hours to setup docker container + Github pipeline. I don't see why this is difficult. I am not saying go all out and manage your kubernetes cluster. I am saying that packaging your application in a docker container and running it in a VM is pretty straight forward.
Heroku is terrible value for money. You can very easily use one of the dozens of free & open source deployment managers or CLIs for EC2 and get the same ease of use at 1/10-1/20 of the cost. It's good for hosting a hobby project for free or $5/mo and forgetting about it, but it doesn't make any sense for a startup.
Is there an open-source Heroku alternative? I've used Heroku in the past and the "just deploy application code" nature is great for most applications but you're locked in to one vendor. Cloud Foundry?
I tried using AWS, Digital Ocean and other hosting platforms for my hobby projects, but every time I keep running into different problems and eventually give up and go back to Heroku - and things just work.
I would never recommend Heroku for my employer or for any company that has real production loads and a budget for administrators that know how to set it all up: Heroku does seem to get prohibitively expensive at larger scale. But for hobby projects that get couple of hundred requests per day this is more than enough, and if one of them gets too successfull, I'll just pay someone for a day or two of work for migrating my code to other hosted platform - and since I never use vendor-specific tech, this shouldn't be too hard to do.
Here's my self-serving advice for every one-man SaaS:
Use Heroku.
It's super easy to get started. And when you reach $2000+ in monthly bills, hire me to move things over onto dedicated or EC2 so that you'll get 10x that performance for the same price.
As for the actual language, I think PostgreSQL + Ruby Backend + JS Frontend is still the easiest way to get started.
All those great architectural ideas don't matter much until you have your first $5k monthly in reliable revenue. And by then, you'll have a completely different perspective than what you think you need now.
Say, you’re a business, and want to set up your own e-commerce site. Whether it makes money immediately or not, is not a concern, as the business will pay for it, and absorb the initial loss. And the development costs to build the site are already in the $50,000+ range.
So with all this upfront expense, why don’t the business just go with EC2?
Unless this is for some silly hobby website of yours, that you have no clue will make any money or not. Then in this case, maybe your suggestion is valid.
They start the comment saying its targeted at one man SaaS. Even e-commerce, there are many who will not drop 50k on development and just start with shopify.
Okay, this thread is making me nervous about Heroku. I'm building a web app that runs on it and I clearly have not given the pricing scaling as much thought as I apparently need to.
My site is a firebase powered SPA, so the only thing Heroku has to handle is the initial pageload....I would think that the standard 1x dyno would be sufficient for that....which is not very expensive...
I meant some sort of JavaScript framework. I'd say that part mostly depends on personal preference. I'm quite happy with Angular, but I've also had lots of smaller one-off pages where I used jQuery.
if you truly want to save some $$$, host it on your residential line and proxy the requests through an external load balancer.
could also rope your friends/family into it if you want your app to be "distributed", might require a lot of work though especially if they are not very technology oriented.
For many newcomers starting out, the Rails CRUD generators are good enough to generate their entire admin area. And Rails comes with sane security defaults.
I agree that GraphQL + React will be pretty and work well. But for one-man start-ups, the focus is usually on "looks good enough and kinda works".
So I believe I'd go with Bootstrap & Glyphicons, just to get results faster.
I use this stack and I'm pretty sure I could pump out any SaaS business in about two weeks with it. Host on Heroku, use whatever analytics system you want.
I'm not the person you asked but I have my frontend hosted separately from the Heroku backend. CORS was a bit of a pain at first, mostly because I had never had to deal with it before, but after initial setup I never think about it.
We use Django REST and an angular frontend. Django is hosted on heroku, and angular is hosted on netlify. Netlify makes it super easy to proxy API requests from the front-end to our API backend.
Just the discussion on this post alone, is a goldmine for anyone trying to figure out what worked, and what not for others. This is why I love HN.
To clarify a point about my article, I think everyone should do what works for them. Just because my unconventional stack for a solo founder works great for me, doesn't mean it will do the same for you.
Your mileage may vary, and it all comes down to tradeoffs.
Also, I'm sorry if I haven't been able to reply to each question, this post got way more attention than what I expected.
- it seems like you just launched your SaaS a couple of weeks ago. Why didn't you launch it earlier by removing some not strictly necessary/MVP features, such as the security login emails, the ability to manage multiple websites, a less scalable but simpler architecture ... ?
- how much time have you spent developing the product? I've been working on my personal SaaS for more than one year, planning to launch it in the next few weeks. I'm still working at my 4/5 full time job, and I really wonder how you managed to deliver this beautiful product while still working fulltime. That would be a wonderful opportunity for blog article BTW.
1) Fathom, Simple Analytics, and Plausible are great. They have done an incredible work too. Panelbear has already most (if not all) of the features you might find in their offerings. However, I do intend to dive into more advanced functionality, while keeping it simple to use. I will be talking more about it when I feel confident about what I can deliver.
2) I launched in September, but only with a few posts here and here. It was also a lot more barebones back then. I try to release a new feature each week, and traffic has been slowly ramping up. I guess I just haven't had much time to promote it, but I plan to keep writing blog posts if there's something interesting.
3) I try to limit work on Panelbear to 2 hours per day. If I want to be in this for the long-term, it's gotta be sustainable, and fun. I try to leverage most of the tools I already know, even if unconventional like Kubernetes, to help me focus on shipping rather than learning a new tool.
Glad you like the design :) I took courses in design when I was studying, but I actually struggle a lot with it. It has helped me appreciate just how much work goes into a good design.
It's cool that this stack works for the author, but if anyone is just starting out on their own one-person journey to build a product, I don't think they should follow the stack in this article.
There are too many dependencies and too much complexity here. Kubernetes is overkill for 95% of applications, especially single founder SaaS businesses. Clickhouse may make sense for an analytics product but caring and feeding it is non-trivial (see: ZooKeeper dependency and all the problems you get once you move to a distributed database).
Most people would be better off with a single beefy machine running Linux, Postgres and whatever web app framework they know. The whole "cattle not pets" thing is fine once you obtain product market fit and your product needs to scale up. At that point you'll have time and money to do it - before then you're just wasting cycles.
I agree with you. I wouldn't blindly recommend my stack to everyone, one could argue it's probably even overkill for my SaaS project right now. At this scale I could get away with a couple of $20/mo instances, and call it a day.
However, it just made sense for me due to the simplified operations, and re-using most of the learnings from my full-time job. It has enabled me to move faster in shipping customer-facing features.
If I wanted to start a new project, I can leverage the same stack and be up and running in minutes. Which means I can take advantage of my cumulative efforts for any future project.
It's definitely not for everyone as you said, but it might be a good solution for many :)
Your stack is great. If anything, I think many one-man SaaS will benefit from using k8s. That's the superpower of k8s, one person can go from 1node to 1000nodes if they need that scaling with minimal effort. Cobbling together shell scripts and tools for one or two servers is just as much work as using k8s, so long as you don't run your own and use a managed service. Best of luck, I think your list is pretty reasonable.
Eeesh I dunno. I agree that the paradigm has broad scaling capablities, but the amount of time I spent chasing down weird behavior in our (managed) cluster at my last job was a total waste, and the company would have been much further along if we'd spent that time on literally anything else. Somebody still needs to manage the managed cluster, IMO.
I think the key thing is that you already know the stack and didn't have to learn it. I wouldn't recommend it to someone for whom the learning curve will be brutally steep, but if you already know it I'm sure it's pretty game changing in terms of scale you could handle just being a solo dev.
> However, it just made sense for me due to the simplified operations, and re-using most of the learnings from my full-time job. It has enabled me to move faster in shipping customer-facing features.
As someone who is mostly in the same boat as you, you have got a strong point in terms of favouring tricks you know best.
The amount of code I have had to throwaway made me not want to invest in things that require lots of resources to run and maintain (in other words, things I'd not be comfortable with managing all by myself).
I almost always default to managed and serverless products: This won't necessarily be a winning strategy for everyone to say the least as it brings its own complexity to the table. It is pragmatic; however (like you call out), to trade one set of complexities (one that I am comfortable with) for another.
Hey that's great! I think everyone should do what works for them.
Using a tool "just because" it worked for others, seems to be what often drives many decisions in this industry. For example, would you go for the 2,200+ microservice architecture from Uber[1] just because they say it solved all their problems? They probably have their reasons, and they are unlikely to be the same as everyone else.
I know my stack as a solo founder is unconventional, but it's what has worked best for me. And I'm happy with my choice.
If serverless does the trick for you, then focus on that, and keep building the features that your customers actually care about.
It is more complex than what I'm looking at right now, but I still appreciate the read as it helps me think about some things I can do now to be better prepared for a more scalable environment later. Thanks. Oh, and I had one (serious, but off topic) question... what are you using to brew your fresh cup of coffee?
I second this. If what you are familiar with is <stack-that-is-not-cool-by-popular-opinion> but know it would work, go with it. You don't have to struggle with a new tool if something similar is already in your toolbox, and you already have a lot of experience using it. Of course, do this unless you are building to learn (and maybe ship), rather than to just ship. And if the new tool proves to be a better one, you can upgrade then.
> The whole "cattle not pets" thing is fine once you obtain product market fit and your product needs to scale up. At that point you'll have time and money to do it - before then you're just wasting cycles.
Is this true? At my last company we wasted a bunch of time every week dealing with the accumulation of ad-hoc changes in different environments creating different behavior. When we pivoted to a continuous deployment model (we also moved from EC2 to ECS/Fargate), all of this time went away as did all of the guess-work and fear of deploying to production (because production very rarely deviated from lower environments with respect to behavior, because all environments were reproducible). It also enabled us to stand up whole environments with the press of a button, which made it a lot easier for developers who were interating on infrastructure (we had a lot of async ECS/Lambda workloads). I'm sure if we were smarter or had the right kind of discipline we could have made pet EC2 instances work, but we benefited a lot from moving to cattle/containers.
NOTE: I also suspect that cattle/EC2 is a lot harder than cattle/containers, but that may also just my inexperience with the former. I would really like to know how to do reproducible EC2 instances properly, but from the research I've done, it seems like one has to reinvent a good chunk of Kubernetes.
There may be a rare exception here or there where network effects or some other unique market need require a startup to treat their infrastructure as cattle, but otherwise it's absolutely true. Running `git init --bare` on a server, setting up some remote origins, slapping some bash scripts into the git hooks folder, some .service files for systemd, and then running `git push origin dbhost && git push origin apihost && git push origin ...` is a hell of a lot simpler than even getting a toy version of kubernetes up and running. For the developer with average linux skills, it's probably even easier than figuring out which cloud orchestration monstrosity to choose. Throw in Github actions and baby, you've got a stew going!
Maybe I'm misunderstanding the whole "pets vs cattle" thing but you don't need to lose repeatability or give on process altogether; it's a false dichotomy. Just don't dive headfirst into the massive complexity that is "scalable" devops - you can still use terraform or docker compose or whatever to deploy your application and provide repeatable development/staging/test infrastructure.
> is a hell of a lot simpler than even getting a toy version of kubernetes up and running.
Getting it off the ground is one thing, keeping things reproducible so you can stand it back up reliably if the instance goes down is another thing. You need to know what are the relevant changes that people have made to the instance over its lifetime in order to reproduce them on the new machine. Maybe this is not what people mean by "pets vs cattle", but they seem pretty closely related if nothing else. I guess the thing I care about is "reproducibility" and for all of its complexity, Kubernetes makes the happy path pretty clear in this regard whereas with EC2 you need Ansible and a whole bunch of playbooks to reproduce things like SSH, logging, monitoring, process management, etc that Kubernetes or ECS or whatever give you out of the box. And if you find that you need to scale beyond one or two instances (not everything is a CRUD webapp, after all) especially autoscaling, then it seems like the EC2 story becomes even more complicated (you now need something to automatically invoke Ansible with the right playbooks for each node). I'm probably biased by my experience (aren't we all?) but it seems like Kubernetes or ECS/Fargate are simpler in these regards.
> Getting it off the ground is one thing, keeping things reproducible so you can stand it back up reliably if the instance goes down is another thing.
Are we talking about pet servers or kubernetes here? Jokes aside, I think you're underselling how complex Kubernetes is and overselling how complex logging, monitoring, process management, et al are. You don't get any of those things with kubernetes "out of the box" unless you've got an ops team or someone's got a year plus experience running one cloud provider's version. It seems really easy cause we've got one click clusters and a bunch of great UIs but all the time people spend chasing down problems and wrangling the extra complexity of k8s in production gets papered over because it's the hot new thing.
The bash history for the setup above fits on a single page and since the advent of Nix and now Terraform, repeatability hasn't been a problem. The harder part is discipline among coworkers but Linux user/group permissions go a long way towards making it readonly for debugging purposes while still giving fine grained control over servers.
> Jokes aside, I think you're underselling how complex Kubernetes is and overselling how complex logging, monitoring, process management, et al are. You don't get any of those things with kubernetes "out of the box" unless you've got an ops team or someone's got a year plus experience running one cloud provider's version.
I'm pretty sure cloud providers give you logging and monitoring out of the box (as well as a bunch of other stuff) and Kubernetes itself is a process manager with its own "SSH" (kubectl exec) and so on. I'm not wed to Kubernetes either--you could use ECS/Fargate or whatever. But to do something comparable in an EC2 environment requires a fair amount of experience just to get to get the basic things that Kubernetes (or ECS) gives you out of the box. Of course, if you're well-versed in VMs already then I don't think you'll get much value out of a container solution for simple workloads, but if you're not familiar then I think container solutions are easier. But again, I might be biased by my experience.
> And if you find that you need to scale beyond one or two instances (not everything is a CRUD webapp, after all) especially autoscaling, then...
Have you ever been working a solo project that needed autoscaling? If so what was it doing that was so resource intensive per $ earned that it was still a solo project?
Picture an analytics application--something that has to crunch a bunch of data per request, and your users are mostly visiting from 9-5. Depending on the size of the user's data source, they could easily use a whole CPU and a lot of memory for each request and requests can take up to 60s or maybe even multiple minutes. You need to run more than a handful of machines, so you can either overprovision or autoscale, but in either case you need some amount of reproducibility to wrangle all of those machines and you'll probably want to plan for them to come and go because you probably don't want to pay for a bunch of high-end machines sitting idle every night. Optimization is nontrivial so you'll get around to it eventually and maybe even hire, but hiring your first employee is a lot harder than hiring your 101st employee, so you're not going to do it right now but you still need a solution to your problem...
I did in a past life. I'd prefer not to say which one because this is an anonymous account and I don't want to make it too easy to associate to my real world identity.
I think it entirely depends on the path you're taking. For example, I'd disagree with the quoted section you made for a different reason. I'd almost always suggest early stage startups to use something like Heroku (if it works for them), which gives them the ability to pay a small premium in order to never have to spend too much time dealing w/ devops stuff. Pre product-market-fit, that's wasted time/effort unless specific devops needs are essential to getting there.
> Is this true? At my last company we wasted a bunch of time every week dealing with the accumulation of ad-hoc changes in different environments creating different behavior.
That is an issue a disciplined one-man operation won't have since (generally) they have a Dev and a Prod environment, the changes would be made in Dev first then duplicated in Prod.
If you have 2+ developers, you have environment drift like you experienced because (generally) Dev is the local environment on the laptop or otherwise a distinct environment for each developer. So now you have 3+ environments you need to sync and then discipline starts to fall apart as you have to communicate every change.
What happens to the disciplined single dev when their EC2 instance falls over in either environment? How do you get things back in sync without code to automatically stand things up in the right state? And if you're going through the work of writing/maintaining that code, are you saving anything over using Kubernetes (consider out of the box, Kubernetes gives you logging, ssh, process management, service discovery, etc)?
Not everyone uses EC2 which is prone to fall over.
And the disciplined single dev documents their changes. That is why it works in a 1 man shop but not at scale, there is always someone who is unreliable once the group gets large enough.
My take would be that once you need another environment than staging+prod, that's the time to move to at the very least move to containers, and probably think very hard about either a hosted Kubernetes or something like Fargate.
Indeed OP uses this sort of stack in his full time job and has learned about all the kinks over the years, while getting paid to do it!
I'm working on a side project and it's literally a bunch of .py files in a folder. Libraries include fastapi and sqlite3. I cannot think of a reason to use container orchestration over a "single beefy machine" in almost any use case.
Not having to recreate your Python env, not having to worry about dependencies (once they're pinned), the ability to run the app on a hosted service like Fargate instead of managing the server...
SaaS business single founder here, moved from a mix of GCP's Cloud Run, Cloud SQL, and Compute Engine to DO's managed Postgres and Kubernetes, and it has had the follow effects:
- Much cheaper, reducing bill from >100$/month to ~40$. Important for early stage startups
- More performance, easier scaling. Found that my application was much better suited to run in K8S, but this is definitely specific to my use-case
- Consolidation of resources. Except Postgres/Redis, everything is in my cluster, simplifying management and maintenance
For my business, this was a great move, but as others have said, I wouldn't recommend it to everyone. K8S is a great and powerful tool, but also is very complex.
I’m running something similar and have to admit that the ~70$/month for CloudSQL grinds my gears a little. The pricing for Cloud Run on the other hand is pretty sweet.
Agree. The databases we're my biggest costs on GCP, and DO offers more for less.
Cloud Run, as you said, is pretty great. In my other start-up, we're using Cloud Run and Cloud SQL, and again, Cloud SQL is the biggest cost (~95-99% of the bill)
Did you experience the downtime issues with DigitalOcean managed Kubernetes that the author also experienced? I'm also considering DOKS so would be great to know!
Note: source for this information is from discussions with DO via support and the kubernetes slack. I am not affiliated with DO in any way.
The downtime issues with DOKS are directly related to the resources allocated to the control plane, which is not set up in a HA capacity. The resources assigned are directly related to the size/number of nodes you use. API-heavy applications can very easy knock out the control plane, and in that situation is takes it a (relatively) long time to recover on it's own (I would typically see ~4hr before it became responsive again).
Their support team is able to modify the master resources for a given cluster (to assist with recovery), but the turn-around time on that shouldn't be considered "production ready".
At this point my advice for DOKS would be:
- Are you using very basic, out-of-the-box Kubernetes to host "apps"? You will probably be fine, but be sure to have a back-up.
- Are you planning to use Operators, or anything that heavily interacts with the kube-api? I would recommend not using it, or over-provisioning your cluster (which would very quickly offset the advantage of the "free" managed masters).
I know that they are working on fixing some of these reliability issues and I have hopes that it will be more stable shortly.
At this time I have a "stable" cluster which (unless their support lied to me) had master resources manually increased by their support team after the 3rd incident of it dying. I haven't had an issue since then.
Availability of your applications, and availability of the kubernetes control plane, are not typically correlated. One of the nice things about how Kubernetes orchestrates is that the master is not in the path of requests. Ie, in many situations the Kubernetes master may be unavailable, but your applications will be unaffected and can still serve traffic.
I guess he means the Kubernetes API which is what most of the controllers, and other components talk to.
That was my case too. It was mostly when I installed the prometheus-operator that things would go south, even if the nodes themselves were perfectly healthy and underutilized.
Gotcha, thanks for the info! Do you know if were just doing run of the mill Pod deploys of like wordpress and little MVC apps and stuff with no K8s api calls we should be fine?
I like the autoscaling and the infrastructure as code, and price for DOKS is good.
I wish companies would have better visibility to allow them to approve DO for cloud stuff. Struggled to get the necessary compliance docs under NDA from DO
When people bring up "cattle, not pets" I tell the story of my uncle, who runs a small dairy farm in Wisconsin. He has about fifty head of cattle, gave each of them names, and takes care of them when they're sick.
Point being that it takes very large scale for the principles people refer to when they say "cattle, not pets" to kick in -- even when you're actually raising cattle. But by the same token, automated reproducible configuration is a huge win whether you have 1 server or 10,000.
I believe 100% that a one-man SaaS needs to leverage their existing knowledge even more than a team. So for me it would be a no-brainer nodejs, dynamo, etc. - the tech I use every day and know in and out. You should have a very good reason to stray from your core if you actually want a business and not just a fun coding challenge.
This. You will want to focus on product design, customer acquisition, financing, investment etc etc. These alone are more than enough for a team, let alone one person. You don't want to be wasting your time and energy fixing server issues or obscure browser bugs at 3 am.
Agree. Blindly recommend this to anyone is not a good idea! It comes down to author's skill sets, and if the author is truly interested in tech, or just some guy want to implement an idea. Personally I am full stack and know how to make server scale without using Kubernetes. Since I've being using commonJS since it's inception, it is pretty easy to write web apps that scales using a light weight server layer like v8.
>Most people would be better off with a single beefy machine running Linux, Postgres and whatever web app framework they know
The problem is back ups. You've got to figure them out yourself. And there's a decent chance that you do it wrong leading to potentially catastrophic data loss.
IMHO, managed DB + S3 equivalent and then doing your own server is the sweet spot. You don't have to worry about data loss and it's still pretty cheap and flexible.
If you're using something like DigitalOcean, you can enable full system backups for 20% of the cost of the 'droplet' (VPS) a month in one-click. Probably quite a safe bet.
Personally, I run a Kubernetes cluster that all my projects go on by default. DigitalOcean hosts mine and my bill is pretty low. I set it up for this exact reason, so I could continue to use my knowledge of containers and container scheduling systems to get easy wins early on.
Setting up Kubernetes is a fixed cost in the beginning with clear payoffs. Things like microk8s, Rancher or GKE/EKS make it a lot easier to set up than it used to be. Deploying your app is then just a simple Helm chart, or another way of getting deployments out. If I was running any kind of business that was making more than zero dollars I would absolutely go with Kubernetes. I don't think it's as complicated as people think at any rate, and it lets you use premade Helm charts which can simplify the deployment of things like Clickhouse.
No it's not, assuming your database sits outside k8s there's absolutely nothing that needs to be fit inside k8s, a simple machine running docker if needed is more than sufficient.
K8s is such a huge mental overhead that if it's not already naturally in your head because you've had extensive experience (or apparently born geniuses) no one with limited tech budget should ever do this.
All of these tools (Terraform, Kubernetes, Github Actions/Other CI) seem like overkill until a node dies and you need to recreate it, or configuration gets messed up and you don't remember how you deployed it in the first place. It sounds like the OP has had experience with these problems. If these aren't problems you have interest in solving, then Heroku or a similar platform would be the way to go.
Running a business on a single node is not advisable. If your scale is that small, or uptime doesn't matter, then sure go with that. If you're running on multiple nodes, you're going to need a way to schedule containers to your cluster one way or another. Scheduling and orchestrating is where complexity get introduced - Kubernetes wrangles this complexity.
You can run a lot on a $5 server. I would probably run 3 $10 servers for a small business at least, which feels like still within the realm of a cheap budget for business infrastructure costs.
As far as Kubernetes complexity goes, I think this is a self-perpetuating idea on internet tech communities. I don't think it compares to the complexity of actually programming the web apps themselves. There's like 6 resource types you have to be familiar with to work with Kubernetes, and most of that will just be Deployments (where containers actually get scheduled and run, health checks, etc). The others are for configuration and routing.
I have had significantly better uptime with my production dashboard app running on an EC2 instance that has been up and running for 3 years straight, than every other team I am aware of that uses kubernetes. The more complexity you add, the worse things get.
Of course, you can't most of the time run production apps on a single machine, but my general approach has been to add the least amount of infra overhead - in our case it has been elastic beanstalk. It has been reliabile and zero touch for us for years now. I have heard of some people saying EB is good till it works and when it doesn't you're screwed - my backup plan for that case is to continue keeping that EC2 instance running, and generally make sure you can run your app without too much hassle in a fundamentally different stack at any given time.
In the end, by depending on RDS and EB we have never had to worry about infra or scaling in our team. This is in stark contrast to every other team I've seen or worked with, where upgrading your eks version, or certificates expiring or etcd getting unresponsive or environment variables leaking between multiple prod and stage or someone accidentally deploying to prod because they didn't specify the correct name space in their deploy script or a hundred other reasons made their k8s deployment a nightmare. You can argue that this is a sign of bad training or practice, but I'm working with the same engineers on this stack that's so much more foolproof so when can we start blaming the tool a bit as well?
> configuration gets messed up and you don't remember how you deployed it in the first place
Configuration getting messed up sounds more like a k8s problem than anything else. In any case, if you aren't using git and don't know how to at least back things up you really should be asking yourself if a tech company is right for you, as a one man/woman operation. There are countless other ways to make money online. Etsy, Craigslist, Shopify, etc.
> until a node dies and you need to recreate it
The way to solve both of these problems on a single node operation is to create a snapshot/clone of your machine. At that point you would really just need to worry about the IP address changing and/or updating DNS when restoring the image. The best part is you can take this "master" copy and deploy it as a staging server. Or even horizontally scale by throwing a load balancer on top. You can even get your VM exactly how you want it locally and then deploy it to DO, Linode, Vultr or whatever.
> Configuration getting messed up sounds more like a k8s problem than anything else. In any case, if you aren't using git and don't know how to at least back things up you really should be asking yourself if a tech company is right for you, as a one man/woman operation. There are countless other ways to make money online. Etsy, Craigslist, Shopify, etc.
I don't know if this was to intended to directed at me, but all of the tools mentioned imply using them with git or other version control, hence Github actions / CI. Kubernetes does not spontaneously mess up your configuration, well intentioned developers do. Willingness to learn new-ish technology puts you in a much better standing for running a tech business.
The snapshot/restore method is fine if you don't plan on updating very often, but this isn't a good alternative to infrastructure as code. I would make a base image that bootstraps your node to a cluster in Packer and then hand off all the deployment of nodes/DNS/Loadbalancer wiring in Terraform. Then just snapshot your database volumes as normal and deploy with your orchestration.
Kubernetes honestly isn't that complicated. It's complexity really only scales with the complexity of the system you're running on it. If you're just bringing up a database and a backend on top of it, it's honestly very simple to get running on a single linux box using something like microk8s. And, as a benefit, you can move everything over to EKS or GKE in like a few hours max, if you ever need to scale.
I think (one of) the big win(s) with k8s is testing in namespaces at a pace that doesn't really slow you down. If you think you'd want to do end2end against an environment entirely built by code, k8s is a really good building block.
If you think a one man band and his ruby app can probably go pretty far with self-discipline, but if I'm choosing to do any sort of IaC I'd probably recommend looking at k8s as a) a faster solution, b) a more versatile solution and c) largely vendor agnostic solution.
Point being, I'd probably recommend folks learn to use k8s before they invest time learning lambda, EC2, or ECS, but if you're a small team and you already have lambda, EC2, or ECS, and it's working for you, then it's hard to argue for change.
That's not to say you shouldn't use Kubernetes as a solo founder eventually. Kubernetes does a lot of the devops heavy lifting for you making your job as a solo founder much easier.
A lot of people here are recommending stacks that have worked for them, or cautioning against over-engineered stacks that didn't. All of that is useful, but I'd boil a lot of this down to one simple axiom:
If you're a professional engineer and you're starting a tech side project with the goal of shipping - use what you know.
I do a lot of work with node, cloudfront, s3, and React for my day job. All of my side projects are built on node, cloudfront, s3, and React. This works well for me because my goal with side projects right now is to be productive, not to learn new tech.
An important clarification is that I use the tech that I use in my day-to-day, not what my company uses. Just because I have coworkers who know kubernetes back-to-front doesn't mean I do!
Also, there's nothing wrong with doing side projects with totally crazy/new tech stacks as a way to explore and learn new tech. I just think it's good to be really intentional with whether the goals of a project are educational/process-oriented or results-oriented.
Serial one-man SaaS builder here with a pair of soft-landings and one big hit:
I would agree with others who have opined that this is overkill. You couldn't pay me to use Terraform or Kubernetes for my SaaS unless it was a much larger team.
My stack usually consists of this: A framework I am comfortable with that removes a lot of boilerplate (happens to be the one I know best in the language I know best), the database I know best, the memory store I know best, Turbolinks for load times, a theme from ThemeForest (with the commercial license), self-hosted analytics, Amplitude, and Heroku.
The author mentions "lessons learned" regarding Clickhouse and Kubernetes. I'm not sure what their original goal was when starting their SaaS. Usually mine is financially motivated more than learning things.
What I'm really saying is that just go with what you know best. Having to scale is a good problem to have. Pre-optimization is the root of all evil — asking "is x the right stack for y?" will always be yes when you ask in the x subreddit.
Please, please, please listen to this person’s advice about Terraform and the like.
The last startup I was at hired a DevOps engineer to come in to completely containerize our simple Rails monolith at a time where they were just simply way too small to warrant it.
Development velocity plummeted. Deploys started to take half an hour. Features weren’t being released. The professionals our app was servicing lost trust in the engineering team.
All in the name of checking off a box on the CTO’s list of “things that Google does”.
Not sure what 'containerize' means in this context, but adding a Dockerfile, building the image and changing your box to use docker should definitely not reduce velocity or increase deploy time to half an hour...
With docker you have to worry about how containers talk to each other. Which at a bare minimum means exposing ports and adding a network layer. Suddenly you find yourself in the weeds with docker-compose and debating whether or not it's actually suitable for a production environment, when they recommend using swarm instead. Then you find out swarm is on shaky ground and now you're hearing about nomad and ten different other solutions just to avoid k8s.
Infrastructure-as-code is a wonderful idea. But the reality is it's a full time job. There will always be orphaned containers and things that need cleaned up and automated. And if you don't, then your environment will run out of space or RAM and all devs will be sitting there doing nothing.
Building docker images also takes time, especially if you don't optimize the image. The simplest deploy workflow I've seen was a git hook (post-update, I think) that ran a few commands when the repo is updated. It'd really be hard to make up for the efficiency of that workflow after messing about with docker for days.
There's a smell if containers are forcing you to radically change your ops strategy. Networking between boxes doesn't change, networking between processes on the same boxes is slightly more complicated but you can always fallback to host if really needed. None of that pushes you to use kube / swarm, they solve completely different issues.
To clarify, this has already helped me to ship features to my customers quickly. I actually spend little time working on the stack. Last time I touched the infrastructure components was probably a few weeks ago.
Panelbear was originally built and launched in a weekend ($5/mo DO instance with SQLite), and I've been working on it for the past 3 months. I have been releasing a new feature every week. That is, in addition to my full-time job, so most of the time we're talking about 2 hours per day tops.
I'd also like to clarify that I am not suggesting this stack should be used by other solo founder SaaS. I actually advice against it in my post :)
I just wanted to share my stack, as it has worked really well in my case, and I thought it'd be interesting material for discussion.
I didn't mean to refute what you shared. In fact I was reinforcing some of your points. Doing things the cheap way is arguably the best way. Learning new tech while building is good, but I wouldn't recommend new tech to build a SaaS.
My concern was copycats. Some starters reading your post may have this feeling that "because Panelbear did it with Clickhouse and Kubernetes and Django, that's the only way to do it." What it really should read is "because Panelbear used what he knew best, he was able to focus on getting customers; that's the only way to do it."
More broadly, can you help me with an opinion on my present struggle? Here goes:
I have a lot of ideas, a couple of which solve problems for me and I think are even _good_ ideas. However, none have shipped because I'm spending an inordinate amount of time building out what is essentially my own SaaS template. Is it worth investing the time in doing that, or are some of the commercial SaaS templates out there (sjabloon, bullettrain, etc)
I prefer Python/Flask, but I'm trying to learn Rails now, since a lot of the small saas ecosystem seems to exist there.
I'm a Rails guy. I've been working with it for the last 10 years and it (now) allows me to be hyper-productive. I've put other languages and frameworks into production but nothing allows me to focus like Rails does.
There is a meaningful amount of boilerplate that remains for any stack if you're using it for a SaaS. It may not hurt to invest in a commercially-available solution that takes care of billing.
For some products (super-hyper-MVP), I've gotten away with having a `companies` table with a `state` column which was either `active`, `inactive`, or `past_due`. I would update this manually from Stripe every day. When it became a chore (aka "I am spending an hour on this every day and my time is better spent elsewhere") I only then wrote code to handle it.
There are solutions like Recurly or Chargebee that take a way a lot of pain on a platform basis. Those may be helpful in managing some of the complexities, but they come at a cost: financial, vendor-based, flexibility.
It's important to keep in mind that your ultimate goal is to get customers validating your idea. While it's important that you conduct a meaningful transaction in exchange for solving their problem, that does not mean you need to handle every single last billing case.
Accept their money, let your solution solve their problem, and repeat.
If you want to talk more my email is in my profile.
That's great insight and I really appreciate you taking the time to reply. I have it in my head that I need to get to a state where I have the boilerplate "done" - able to handle accounts, users (for areas where the two need to be 1:M or vv), admin, and basic subscription billing. I'm trying to get to the point where I have this thing, and each MVP is just an incremental build on top, but instead I have this half-baked thing and a lot of undelivered ideas. I suppose there are no shortcuts and I need to hammer at it, but as you said, not go nuts with handling edge cases.
I'm working on a product to help developers quickly get started and manage their SaaS projects. If you have a moment I'd like to hear more about your experience. To quote the other commenter: "If you want to talk more my email is in my profile." :)
I agree about Kubernetes but not about Terraform. Terraform is easy to understand, and it's basically just declarative configuration. You have some Cloudflare records for your domain? You can put them in a file, wiki or declare them in Terraform. So maybe just use Terraform and get the benefit of free deployment?
Right now the operational effort is pretty minimal to be honest. I spend most of my time developing new features and talking my customers.
I know it seems like a lot of tools, but most of them are set up once and forget. Except for the occasional round of dependency updates. But even for that I have automated dependency/container scanning to detect vulnerabilities, and get notified if updating an image is recommended.
Most of the stack consists of "platform" tools, such as the ingress-nginx, external-dns, or cert-manager. So it's stuff I take with me for every project. It's like a personal Heroku I've been building, but way cheaper and fully customizable.
I've been iterating on this stack over the years with various other projects, and also learned a lot from running Kubernetes in production at my full-time job, so it's not something I'd advise for someone who just wants to deploy their MVP for the first time.
What would you advise for someone who doesn't have the pre-existing k8 expertise etc?
The ops environment still looks overwhelming to me, if there's a way to do it sustainably without a really heavy-weight ops infrastructure, I haven't figured it out. Like, something that to me requires like a full-time job just to do ops, although maybe not if you've spent some years learning the ops tools, but then same same.
I don't think diving into kubernetes makes any sense for a 1 person shop if you don't already know it. People love to evangelize about k8s more than your average tech but it's just one (IMHO overkill) way to do things. A better bet I think is to leverage tools from cloud providers. They specifically offer many different easy to use scaling platforms for a premium for small/not experienced teams. Heroku has been mentioned a bunch, something like Elastic Beanstalk on AWS is super easy for the first year or two of a company. Or just go for a single machine somewhere and worry about the 5% chance you need more than that when that happens.
You can get really far with the managed services from your cloud provider.
If you're already familiar with Docker containers, just that already gives you plenty of options. I think every platform now provides a way to just run containers behind a load balancer with a few simple steps.
Yes, maybe it's a form of vendor lock-in, but if your worry is acquiring customers, and growing a company, Kubernetes is probably not helping you in this case.
Replace Python/Django with Java/Kotlin/SpringBoot and you pretty much have my preferred project stack, which also organically grew due to being exposed to these platform tools in my day job.
Kubernetes/Terraform and friends are really useful as a solo-dev if, and only if you have the required experience with them.
"I know it seems like a lot of tools, but most of them are set up once and forget. Except for the occasional round of dependency updates. But even for that I have automated dependency/container scanning to detect vulnerabilities, and get notified if updating an image is recommended."
This is an insightful article. A learning for me was how the author managed to switch multiple cloud providers even while being a one man SAAS.
Kubernetes is overwhelming because of 2 things: Overload of terminology, and Distributed systems day 2 operations.
Author succeeds so well because he has a starter stack (install, CI pipeline, secure, monitor, scale) maintained as code, and it works every time.
For someone starting new, the learning curve is high. Bootstrapping is easy, but day 2 on kubernetes is hard. Benefits outweigh is learning curve imho.
Specific to DigitalOcean kubernetes API server issues, we have rolled out some resource provisioning improvements for newly created clusters very recently. Also making ongoing improvements to backend monitoring.
Thanks for the response! To clarify, I still a happy customer of DO, and would still use it for some other projects.
I have also had a fair share of issues with AWS services in the past (Athena, they took care of it a few months later), and GCP too. And I understand that no product is ever perfect.
Very, very similar to the tech stack we have in my engineering teams (other than we use Flask and SQLAlchemy instead of Django). Really like it. I've not heard of ClickHouse before, but that looks interesting. (If I had a requirement around time series, I was thinking of checking out Timescale - https://www.timescale.com.)
Seeing a lot of criticism on this thread, but honestly as a founder the best stack for your company is the one you know. Whether you use Kubernetes/Terraform or manually rsync a bunch of files to a server under your desk – do whatever works for you.
Adding onto that - your priority is to be able to ship features and have an ease of maintaining them. This setup works for you, so it's great to see it be shared.
Django has just so much stability, it was an obvious decision for me to go with it. And that's speaking as someone who has mostly worked with NodeJS/React for the past 5 years.
I love the pythonic principle of "There should be one obvious way to do it". This helps me cut down on decision making which is something quite annoying to deal with as sole-tech-founder. Now I get to focus on the business, and technology is just a matter of execution.
Important thing to keep in mind with this article is that this is for OPs data analytics SAAS where he's ingesting a mass amount of data, has different uptime requirements, etc than many other SAAS applications.
Fellow solo founder here. I feel like this is pretty overkill, most notably k8s (and terraform.) The last company I worked at used k8s and it felt overkill even for them. But if it works for you, great, but I definitely think that'd require spending too much of my time on devops. Maybe I should do a "counter" blog post on using Rails, Heroku, Redis and Postgres. :)
A good rule of thumb for success is to have more paying customers than there are layers in your tech stack.
Another rule of thumb is to have a setup so basic that articles like this are no fun to read or write. The limiting factor in a one-bro setup is complexity, and this guy maxed out about one paragraph in to his description.
Seems it's another cookie-less, privacy-friendly Google Analytics alternative. This time from Germany!
It's pretty cool that there's so much going on in this space, but is it really wise going into the market with seemingly the exact same features that Plausible, Simple Analytics, Fathom, etc. offer? Even Cloudflare recently entered the privacy-friendly analytics market with a free product, so it seems that market is getting really crowded. Then again, every (commercial) website probably needs such a solution, so maybe the market is big enough.
I wonder though if these cookie-less solutions can give people the insights they need? You can do only very simple overall statistics and funnel analyses with this approach. Maybe that's enough for most websites though?
Yes, I understand I'm entering a crowded market, but Panelbear is still in the early stages, and I'm trying to understand what problems my customers have that I could solve / how much value can I bring to them.
A crowded market also means more options for you as a customer. Specially when it comes to privacy-friendly tools, not only analytics, I'd argue that there's not enough options yet.
I initially built it for myself, as I was not fully satisfied with the existing solutions. And I do intend to offer more features than what you currently see, but unfortunately my time is limited and it's going to take a while to bring Panelbear to the feature-level I'd like it to be :)
- GKE kubernetes
- Managed Postgres (CloudSQL)
- GCS buckets for file storage
- Cloudflare for DNS
- Countour for ingress
- Keel for automating deployment updates
- Mailgun
- Sentry (errors)
- Node-RED various little automations such as healthchecks
Planning to introduce Elasticsearch too, so far I have been testing the operator and it seems to be pretty good quality.
From maintenance perspective GKE is great, as long as the bills are paid, no need to login there pretty much ever. As a one man company this means a lot, I try to never spend any time on ops.
Wow, there is a like a million tools in use here. Before I clicked the link I was expecting the tool set to be extremely minimal and focused, e.g. HTML+CSS+javascript/node.js. I currently have a bunch of projects using PHP+javascript+HTML+CSS and I am seriously considering dropping PHP and focusing more on javascript, and thus using node.js instead of PHP, so I no longer have to keep up to date with PHP. Anyway, great you can manage it all and that it's working for you.
All JS is the place to be. TypeScript is an amazing dev experience, NodeJS is decently performant, and then it's all portable as hell thanks to the web.
My feeling is that they are comparable in a proverbial "straight line" computational sense, but nodeJS' non-blocking event loop / IO makes it superior for handling a webserver's request / event model under higher concurrent user counts
At my startup we use a pretty similar tech stack, except we are on Heroku for infrastructure which makes our life a lot easier, and gives us things like review apps quite easily.
One piece we've struggled with is just mypy doesn't seem to play too nicely with Django. It definitely feels like a world of difference between the type support we get in our React app with Typescript and mypy with Django. Would be curious what OP's experience has been with that.
I run three different SaaS products by myself, spend a couple of days per week consulting and lead a pretty full life with young kids, volunteer firefighting, hobbies, etc. Each of my products probably equates to around a 1/4-Man SaaS.
They're all server-rendered HTML with a little ajax sprinkled around the place where required for interactivity. I use Materialize CSS to give me a leg-up on the design side but the next thing I release will be on Tailwind UI.
They're all written in Go using a simple MVC-ish pattern. I write a lot of integration tests and a few unit tests.
I use Google's hosted PostgreSQL database. It's not cheap but it's not expensive either and it's been rock solid. The frontends are all on Cloud Run as of a few months ago which has been wonderful. Super reliable and fuss-free. Takes care of all the application logging, etc.
Task queue stuff is handled by a library on top of Postgres. It won't scale to a billion jobs per second or anything but it handles load just fine. If I ever start pushing it's boundaries it's all pluggable so I can switch it back to SQS or PubSub or whatever else.
The task workers run on a single-node k8s cluster hosted on GKE. A lot of people grump about Kubernetes being "overkill" but it doesn't have to be complicated. A single YAML manifest defines your service and your environment variables, you tell GKE to make it happen, you go play in the sunshine. It takes care of restarts, logging, full disks, etc etc etc all for you.
I use Amazon SES for emails. I had some real problems with deliverability until I ponied up for a dedicated IP address and now it's fine.
I use Freshdesk for ticketing. I don't love it at all but I don't yet hate it enough to replace it with something else.
* .NET Core Backend (Business logic + Kestrel Web server + Let's Encrypt + JSON Web API)
* NOSQL Database: The filesystem
* SPA InfernoJS front-end (similar to React)
--
The application is performance intensive and must deliver responses sub 100ms at at all times. Each 'request' from the user does a lot of work. I optimized to get all this to run efficiently on a $20 / mo EC2 instance, with no other costs
Taking a different line of questioning: Does anyone have any insights or resources on the non-tech stack a one-man SaaS should use?
At various points, I've looked into opening a single-owner LLC to hold some small side projects I want to put on the internet for fun, but the number of decisions and amount of overhead has always seemed pretty daunting.
If I were to attempt a one liner for the key takeaway from this, it'd be "use frameworks as much as possible, use managed services as much as possible".
It's a position I couldn't agree with more. It might be one of those mistakes you can only truly learn from by making it yourself, but honestly, while DIY is awesome for learning its only place in a business is at the core of what your business value is.
One way I think about it is anything DIY is surface area for either innovation, or mistakes. To mitigate the risk of making mistakes doing things DIY you'd better do it only in areas where innovation could actually benefit your core business value and give you magnified upside. If you innovate outside these areas, I guess it's nice, but you've taken the risk of mistakes for no significant upside. Not to mention a whole load of work invested.
I use plain PHP for backend and HTML for the front-end (plus javascript and libraries from the awesome open source communities). It's just fast enough to ship. And personally, I find it very easy to scale if needed. Don't worry, debugging is not that hard as long as you don't complicate your code.
I personally just use one ore more VPSs on DigitalOcean, which scale pretty well at first and if you get enough traction to need "infinite" horizontal scaling, it might be better to only focus on that once you have the money to hire some sys architect/admins.
I haven't read comments yet, but I bet some people would criticize the tech stack as "too complex" or "not perfect" or "not even mention my favorite tech"...
1. Typically tech stack starts very simple, then evolve into something a bit more complex. It's normal.
2. If this tech stack is the author's most familiar tech stack and it gets jobs done, then it's a perfect tech stack for the author & the business. There's not a tech stack that works well for every developer, every company, every business...
3. Treat this tech stack as a data point. Find other data points on the internet. And you'll come up with your own "perfect tech stack", just for you, for your specific business.
For early stage startups we just released fsKube: Your full-stack kubernetes enabler
fsKube provides as Desktop Web UI to setup entire AWS EKS Kubernetes cluster, with all the components and Django framework template with a full DevSecOps enabled, along with monitoring and operations.
The whole thing within 45 minutes.
Support for Spot Instances, Graviton Processor and AMD AMI coming soon, will reduce already cheap EKS prices further.
I'd definitely appreciate seeing something like that too. After a couple of "build it first" failures I've come to see the wisdom in finding your customers and validating your idea first.
I've read several of these SaaS walkthroughs and it invariably seems to follow this pattern:
1. Author has a blog with a large following on some topic.
2. Finds niche in that topic.
3. Makes a product and pushes it via their blog.
4. "Incredibly" their product gets tons of customers.
Presuming you are using Github and were using Angular, I think "Google Chrome" is missing from the "Other" stack - since I have had problems even in Firefox try to get latest Google WebComponent shinies to work.
How do you begin to learn all of those tools? The coding itself doesn’t scare me, but the configuration of all those tools is something that makes me feel like I have barely learned anything since college.
The easiest way to learn this (or anything, really), is to start small. You don't need to invent Facebook on day 1 - start with something small (probably the backend, that's the simplest) and add new tools when you find need for them.
In the linked post, they mention both Django for the backend and Redis as a cache. Django has caching built-in though, so until you need more complex caching logic, you probably don't need to reach for Redis.
I kinda wish the dogfood part was mentioned earlier... but sure glad I caught it at the end. An article on your analytics bootstrapping would be neat - how (did) you do analytics early?
Would love to hear more about how you deploy, manage and utilize Celery. I think there is a lack of quality case studies on celery-in-production. Thanks for sharing!
How did you land on another analytics SaaS product? I think like every dev I have always dreamed of starting my own thing, but never had the time/energy to compete with existing spaces, and couldn't think of any new markets create. It just seems crazy to me when someone is like: 'oh I am gonna make a tool for people to manage their finances' when 1000 exist, but then they build it and achieve some level of success.
Another one man-team here, I find it fascinating how our stacks have at least a 90% overlap. Also funny that we both build analytics tools :)
For your front to back integration: I suggest you look into django-graphene and reacts apollo-client. That small addition to my stack has, at least, doubled if not tripled my churn speed. Throw in something like Django-channels and you get a modern full async websocket capable reactive framework :)
Glad to see that stack works for author, but honestly one man show's shouldn't be multi lingual stacks at all IMO.
for me php (laravel for back and front with blade templates) or js (express and vuejs for frontend) and docker-compose for deploying and developing stuff is fast and enough stack.
As a fairly new dev, this is really great information for me. Thanks for posting this.
If you don't mind me asking, roughly how much do you spend (time) on this every week? How much of that is split between development, operations and product or business development?
Since I have a full-time job too, my time on this project is pretty limited, usually no more than 2 hours per day.
That's why I tried to automate and save time as much as I could using Kubernetes operators, well supported open source projects, and most importantly, tools I already feel comfortable with.
I don't have hard rules in terms of how I distribute my time, but recently it may have looked like this:
- 60% of the time I'm trying to understand the problem I'm solving (talking to customers, researching, evaluating features).
- 20% working on features, and issues.
- 20% promoting: writing blog posts, documentation, reaching out on Twitter, HN, Indie Hackers. Helping out customers integrate their website / understand how something works.
Of course, there's tons of other smaller things here and there, but those are the big "buckets" of my time.
Once I know what to work on next, I tend to prioritize as follows:
1) At the start of each week I pick 1-3 topics to work on next (for example writing a new blog post, working on a new feature, or even just writing documentation which I am going to next).
2) Each day when I'm working on my project I pick the most impactful thing I could work on, and start with that. I allow myself to explore a topic in depth, sometimes without a clear goal or deadline. After all it's supposed to be fun if I want to stick to it for the long term.
3) I try to respond to customer emails as soon as I can, and understand why they're giving my product a try. That's my gauge for what to work on the following week.
Thanks for taking the time to share this. I've been mildly interested in a side hustle with the primary goal of picking up new technologies but currently lack the motivation to, after a long job search.
It's interesting to see how you split your time and how much thought goes into all this. A followup question, if you could;
> and most importantly, tools I already feel comfortable with
What do you do in situations where X tool would be 'better' than Y tool, but you're comfortable with Y tool? Do you do a comparison of the time/effort it takes to learn AND use X tool vs. Y?
To get an idea off the ground, I use Google App Engine (GAE) with Python, Google's Datastore, Flask, Bootstrap. It works, doesn't cost much and removes a lot of complexity.
GAE's offering of free (and auto-renewing) certificates also makes things cheaper.
Thanks for sharing, I would like to know more about how you define your deployment infrastructure and environment with code(Docker,terraform,kubernetes etc) including migration checklist. I think it deserves its own blog post and youtube video
Thanks for sharing! How are you liking Loki so far? I’m looking to migrate off Google Cloud‘s logging offering to Loki but was worried about its maturity.
I have used https://calculator.aws/ in the past, but to be honest it's difficult to account for the egress costs/networking fees using their calculator. Simple things like networking between Availability Zones/Regions can really creep up in your monthly bills if you don't watch out.
I do track my expenses on a sheet manually, using a ballpark figure. At the end of the month I compare it with my AWS bill. If something seems off, I investigate it further. But I don't try to predict exact figures, more like expected ranges.
I do miss the fixed-cost nature of DigitalOcean/Linode. I didn't have to worry about sudden charges, you pay $20/mo per node and that's it. Lots of bandwidth included.
But for the most part it's fine, as long no bot starts bombarding your services with useless requests, egress costs tend to scale with revenue in my case, as my product's pricing is based on usage.
I also have Cloudflare in front of my AWS stack, it has blocked a few small attacks already, but nothing serious yet.
I still have a lot to learn, but IMO nothing beats struggling with something for a long time, and pushing along until you're no longer terrible at it.
I feel that most of what I know I learned by starting projects on topics I initially new nothing about, and sticking to it.
Regarding on the job learning, I have been working with Python/Scala for my full-time job over the past 3.5 years, and that helped too. Specially in terms of running multi-regional services, and the challenges that come with that.
We run several applications on Kubernetes using AWS (combined >2k req/s at peak time), and that was a big challenge for me from which I learned a lot. Specially when you're on-call, and one faulty node brings down the entire cluster, that fails over to another region and repeat :)
A single core running Python will easily handle more load than most successful SaaS companies ever reach, especially in web dev where most of the real work happens in a database of some flavor.
There absolutely are performance pitfalls, and Python is less power efficient than other alternatives running some kinds of workloads at scale, but in a ton of environments the perf difference doesn't matter.
But while I too had in my head "for web dev most of the real work happens in a database of some flavor", I recently realized that was not true for my app, probably hadn't been true for some time, and probably isn't true of most Rails apps (possibly not the same for non-Rails web apps?). If you've properly eliminated n+1 queries and other inefficient querying, I find that my apps are spending only 20 or 30ms waiting on DB results, and a couple hundred on CPU tasks to render HTML.
I know people are going to reply with "That's because Rails is slow," but I'm not sure that's true for what we're talking about compared to similar Python platforms, I'd be interested in seeing numbers for other real apps. In the Rails community too, people still repeat the assumption "most of your response time is spent waiting on the DB" -- but I dont' think it's actually true anymore (it may have been once).
The "most of your response time is spent waiting on the DB" assumption might not hold for your app. Even if you actively push work to the DB to reduce network bandwidth or something, your workload might fundamentally be comprised of small, easy-to-optimize units of work that the DB handles without any issues.
That said, hundreds of milliseconds sounds slow by an order of magnitude or more for html rendering, even if all the work is done in a batteries-included framework for a dynamic language, and it doesn't jive with my Python experience at all. Do you mind me asking what kinds of tasks are taking that much time?
In this app, mostly lots and lots of thumbnails. It may be some unoptimized code.
But rather than get into the details of my app, and parts that need optimization (whether in my local code or in Rails or in ruby), I'm more curious about the overall concept.
Are you sure that most of your web app's time is spent waiting on the DB? My suspicion has become that this is conventional wisdom that is not actually true of most apps anymore. But I could be wrong. Or I could be right only for Rails and not python because Rails is slower than typical python, or something. I am curious to find out.
> Are you sure that most of your web app's time is spent waiting on the DB?
Positive. It's my first tech job though, and we use C# at work to the extent that matters.
The kind of Python I write off the clock probably isn't a good example of waiting on the DB though. It's usually a thin wrapper around C or assembly (or calling into such a library via numpy, networkx, etc) to do something horrendously expensive that I absolutely would not want to run in vanilla cpython. That said, when I do ordinaryish web-related stuff in Python I'm looking at well under 20ms total elapsed time per call, which is why your ~200ms html rendering time stuck out to me.
time "spent" in the database depends on the speed of DB drivers too, which depends on the language. Our stuff is Java and average query is ~2ms round trip.
Python, Ruby and friends are bad for page speed metrics. REST response time of ~200ms vs ~10ms for our java backends
Yes, it does, as long as performance-bottlenecked code is not in Python (and it usually isn't). In the real world, you may have to extract some hot regions to C++ or Rust or Java, but 99% of your code will scale infinitely.
My team migrated a service handling >1.6k req/s at peak time, from Scala (Finatra), to Python (Django). Same number of servers (3x c5.large), just higher CPU utilization (from ~15% to 70%).
Scala was great, but unfortunately it made it difficult to onboard new team members into the project, so development suffered. Also the rest of our stack is mostly Python, so we couldn't use a lot of the common tooling, and libraries we had built for other projects.
We migrated it within a month, and it's been running for almost a year without issues.
At some point the service even recorded 3k+ req/s at ~200ms 99p latency. Yeah, maybe not Google scale, but more than enough for 99% of the businesses out there.
This service handles about 2.5 billion requests per month from one AWS region. We're a small team, and that's only one of the services we support. Which is great because we don't need to spend all of our time optimizing it.
Notwithstanding this (that Python will scale), Python will force you to scale horizontally a lot earlier than something like Rust. In mid-2013, NewsBlur was using 48 servers to run its Python/Django stack. The workload was reasonably precisely declared (and I could grudgingly see why it used so many servers for its stack), so I was able to run the numbers and confidently concluded that something reasonably efficient (which for me would now mean Rust) could have handled the same workload on one equivalent server, though not with a great deal of headroom, and further vertical scaling might have started running into I/O limitations. But for a one-man thing, I know I would prefer to use a single server as long as convenient, and a 98% reduction in hosting bill is very significant (though that’s a moderately extreme case, and I imagine smaller parts of the code could have been rewritten in a language like Rust to get a meaningful fraction of that resource usage reduction).
The second thing I’ve done is avoid containers. VMs work fine for many types of applications. That’s especially true if provisioning and configuration is mostly automated. A product like Google Cloud Run looks interesting for a few low-volume backend services that I operate. Containers are a good fit there because it simplifies operations rather than increases complexity. The core infrastructure will remain VMs for the foreseeable future.
I began doing Ruby and Ruby on Rails (RoR) development in 2006. Ruby had been around for awhile but the growing popularity of RoR was driving a lot of changes in the ecosystem in those years. It was a fun time but I was working for a large company that allowed me the time to keep up with everything. I see today’s frontend and container ecosystems in the light of that experience. I’m now more content to wait it out and let the dust settle as much as I enjoy learning new technology.