FWIW, I get on the order of $40/hour as a senior with almost 10 years experience, and it allows me not to worry too much about spending (with a wife earning about a third of my salary and two kids). I think I could easily earn at least 50% more if I wanted to work for some rich but soul-crushing corp, but for obvious reasons I don't do that. I guess US cost of living is just insane. (I live in central Europe.)
> FWIW, I get on the order of $40/hour as a senior with almost 10 years experience, and it allows me not to worry too much about spending (with a wife earning about a third of my salary and two kids)
How much do you pay annually out of pocket for health insurance premiums and other healthcare expenses?
In the US that expense is very high, and is a major source of worry for working families.
> How much do you pay annually out of pocket for health insurance premiums and other healthcare expenses?
Very rough estimation: $9000. I'm not sure how much my wife pays - this is paid by the employer and she usually doesn't bother to check. (This is mainly insurance, we seldom use public healthcare.)
My wife is a full-time-mother and is currently uninsured because we'd be looking at doubling the cost of insurance, and paying close to 25k a year for insurance. It is a completely broken system at this point.
I'm similar salary band, I pay 9% of my annual salary for mandatory medical insurance, but it's usually hard to get an appointment in reasonable time so you are going to pay extra 50-100€ for a visit to the same doctor, but in private clinic. And also vaccination and dental is not covered by that 9% payment.
For most white collar jobs like tech here in the US, your out-of-pocket as percentage is income doesn't play a role in how we decide salary bands.
For a family of four, the average health plan is around $10k out of pocket from the employee along with around $20k in employer costs [0]. Yet the median American SWE salary is $187k [1] versus $66k in Poland [2], $93k in Canada [3], and $111k in the UK [4]. Either way an American ends up earning significantly more after healthcare costs and insurance.
The issue is salary expectations at the lower performance band haven't kept up with what is expected at that salary band.
> In the US that expense is very high, and is a major source of worry for working families
When benchmarked against similar peer cities in Canada [5] or the UK [6], CoL is roughly at par yet salaries are significantly higher in the US, especially when comparing peer tech markets like SF [7] versus London [8].
This is the crux of the issue - demanding 100% WFH well past the end of COVID made it hard for us to justify domestic hiring when
1. Async was successfully proven to not impact business operation
2. A reverse brain drain of all nationalities in the US during COVID meant it was easier for employers to work with them to open a hub office or GCC abroad
3. A new grad is demanding salaries that simply don't make the economics of training and hiring new grads work. At $70k-$110k it does, but not beyond that.
4. Companies have now adopted the Netflix model - by cutting low performers, we can actually give higher pay bands to employees who actually have a business impact, as can be reflected in the rise in 75th percentile tech salaries.
I think you make some solid points, but there are major tradeoffs some of the data is not totally convincing. If US workers are so much more highly paid than foreign workers, then we can reasonably expect the best workers to migrate to the US whenever possible. It's pretty easy for Canadians to cross the border. So one reason to hire American developers is for quality. The other is simply that these companies exist in the US, which means collaboration needs to be done in US time zones, which makes overseas workers far less efficient, not to mention the major negative impacts on worker morale. So there can be reasons to hire out of country, but the tradeoffs are significant even when well executed.
> If US workers are so much more highly paid than foreign workers, then we can reasonably expect the best workers to migrate to the US whenever possible
Not really.
No one wants to leave their families, and the upper tier of salaries in alternative geos are high enough to capture the higher talent tier because their salary expectations are based on their domestic condition.
On top of that, the US immigration system is severely backlogged. It can take decades for Chinese and Indian nationals to become green card holders, and we as employees increasingly expect foreign nationals to pay the filing costs - not us.
> other is simply that these companies exist in the US, which means collaboration needs to be done in US time zones, which makes overseas workers far less efficient and having major negative impacts on worker morale
Not anymore. WFH proved async work models can ensure business continuity.
On top of that, the bulk of layoffs during COVID were workers on work visas who were given the option to return to their home countries and open an office there.
This is what Google did in Hyderabad, Bangalore, and Warsaw, Databricks in Bangalore, Amazon in Canada and India, and Nvidia in Bangalore.
Furthermore, we as employers don't really sponsor VPs, Engineering Managers/Directors, Product Managers, and Staff/Principal Engineers on O-1 visas. Most are stuck on some form of EB1/2 or L1/2, and those who apply to O1s who aren't founders or extremely critical to the business are being sponsored but filing out-of-pocket.
It just isn't attractive to immigrate to America long term anymore as a white collar employee in most cases now aside from unicorn roles which employees then use to boomerang back to executive roles or demand US salaries in their home country.
Ideally we need to build a domestic talent pipeline, but universities failed severely by watering down curricula in an attempt to compete with bootcamps, which burnt a lot of employers disincentivizing them from hiring early career, and state and local jurisdictions in the US just don't give us the support or pipeline needed to build a competitive early career hiring pipeline.
For example, in cybersecurity, I can hire someone in Israel who has done offensive security work for a couple years in a military, police, internal security capacity or someone in India who participated in one of the dozens of Police Force, Army, or Home Affairs cybersecurity internship programs. Similar programs like Cyberpatriots and the Cyber Incentive Program (approx $100M) were mismanaged as was found in a 2023-25 investigation by the DHS OIG [0][1] and an entire generation of students of cybersecurity scholarships quit in 2016 when the Trump 1 admin cut funding for cybersecurity scholarship programs.
> No one wants to leave their families, and the upper tier of salaries in alternative geos are high enough to capture the higher talent tier because their salary expectations are based on their domestic condition.
For 2x the salary young men have proven time and again they are willing to take the risk.
> On top of that, the US immigration system is severely backlogged. It can take decades for Chinese and Indian nationals to become green card holders, and we as employees increasingly expect foreign nationals to pay the filing costs - not us.
A challenge for sure. This and what you go on to describe could certainly shift some junior labor from H1B to remote contract. I would expect that shift to have a mostly net zero impact on other American exployment.
> > other is simply that these companies exist in the US, which means collaboration needs to be done in US time zones, which makes overseas workers far less efficient and having major negative impacts on worker morale
> Not anymore. WFH proved async work models can ensure business continuity.
Every non-US team I've worked with and everyone I know that works with offshore still have meetings. It would be incredibly dysfunctional to not have any collaborative time.
> Ideally we need to build a domestic talent pipeline, but universities failed severely by watering down curricula in an attempt to compete with bootcamps, which burnt a lot of employers disincentivizing them from hiring early career, and state and local jurisdictions in the US just don't give us the support or pipeline needed to build a competitive early career hiring pipeline.
Is that really what is happening? Because based on everything I can see, hiring standards are the highest they have ever been. As we get older, we have a bias towards underestimating the capabilities of younger generations, because we can see them making familiar mistakes in real-time.
> For 2x the salary young men have proven time and again they are willing to take the risk
First, sexist, and secondly in 2025 there's no guarantee that you would be able to live in the US long term on a work visa (but it's been this way since the mid-2010s), and if someone really feel the urge to immigrate to the West, then Australia, Canada, Netherlands, and Germany are all easier and (excluding Germany) Anglophone (yes, NL is de facto Anglophone now).
The US just isn't as attractive a location to immigrate to anymore for a large amount of people in white collar roles.
For the cream of the crop who primarily target American BigTech (GAYMAN) or HFTs like Citadel or Jane Street, in India the salary and ESPP grant can afford you household help, a nice condo, and make enough money that you can invest in building generational wealth or angel invest. It was a similar story for Chinese over the past 10-15 years as well.
For Europeans and Canadians, both are extremely turned off to America due to the Trump admin, and at portfolio companies we've seen a significant amount of requests from employees to shift away from the US as a result. Even Israelis increasingly don't target the US anymore because of perceptions and have begun choosing Germany, Czechia, or remaining in Israel.
> I would expect that shift to have a mostly net zero impact on American exployment
A lot of senior managers and leadership in tech companies are in the same boat with work visas as I mentioned before. All visas categories go through the same backlog for naturalization in the US - be you a manager, VC, factory worker, or SWE. Heck the creator of PyTorch is himself on one of these visas despite being employed at Meta.
> Every non-US team I've worked with and everyone I know that works with offshore still have meetings. It would be incredibly dysfunctional to not have any collaborative time
Absolutely, but everyone makes time for Zoom meetings and finds a way to make it work, or people like me will hire someone else who can get it done.
> Is that really what is happening
Yes in cybersecurity and a large portion of databases. I even explained why in other HN comments [0]
This is why most of our dealflow is now in Israel, Eastern Europe, and India. Look at recent exists like Wiz ans PingSafe.
Even recent cybersecurity companies that IPOed like Netskope and Rubrik have overwhelmingly hired in Israel or India and with leadership being Israeli or Indian either in origin or nationality.
> Absolutely, but everyone makes time for Zoom meetings and finds a way to make it work, or people like me will hire someone else who can get it done.
So we are in agreement that work cannot be fully async.
There are clearly some forces at play that are changing how immigration works in the short-term. I think where we differ is mostly this: our anecdotal experiences are radically different (I would say competency is increasing not decreasing as a general trend) and I would never bet against the momentum of the US long-term, nor do I believe that quality tends to remain steady or increase with offshoring. Maybe things are different in your niche. I appreciate being given a view into a different perspective!
> For most white collar jobs like tech here in the US, your out-of-pocket as percentage is income doesn't play a role in how we decide salary bands.
Most people here in the US don't have white collar jobs nor compensation of a software engineer. They work in retail or other blue and pink collar professions.
A Canadian retail worker has much more affordable access to healthcare than their US counterpart.
> A new grad is demanding salaries that simply don't make the economics of training and hiring new grads work
This is just me, might not be representative, but as an indian CS graduate, I was willing to move to the US temporarily if it meant I could make FIRE money and return to India and basically chill out and work at interesting jobs without worrying about needing MNC/unicorn money to live well. I saw broadly two ways to do this - the startup scene, and the ridiculously high new grad salaries (which would enable FIRE in india after a few years) in the US back in early 2020s - when I was graduating. By the time I finished though, those salaries dried up like you are saying, and startup exits are easier in Blore these days and I can get funding from the usual bunch by just domiciling in Spore like flipkart did. Note that I did not expect the current AI bubble to last this long. Potentially could have cashed out on that. My impression is that if you know nvcc exists, you get money thrown at you in the US today... All in all, the idea fizzled out. Among my peers too, the people who went there for the usual M.S at a UC --> california job market pipeline are all people who are sure they want to settle in the US long-term, through the visa troubles and all. Others didn't go.
Point is, those high salaries were a big draw to a lot of people here including me. And in the absence of that companies now need to move here to get access to the same people. I think it is happening, unicorns/GCCs here are now paying quite well. I mean I had almost the median EU tech salary ($60K - 52LPA) for india cost of living at my _starting job_. So without the US starting salary being 150K-200K (a salary unattainable in india anywhere) like it was 5 years ago, it's a hard sell. Senior salaries are still high of course, but if you have to stay there 10+ years, you have a family there, kids there, loans there, and it is basically committing to settling in the US. Meaning that without the added security of the backup plan "return and you're either FIRE or super comfortable", its much more of a commitment to move. My circles have a selection bias of course, and for people who did not manage to get top 10% salaries here, the risk/reward is completely different.
My reply was primarily about American new grads - who overwhelmingly don't do grad school.
And your anecdote is exactly what I am trying to explain on HN.
We as employers are fine paying high salaries to mid-career talent, but it's hard to justify hiring a middle of the pack new grad from CSU East Bay for a $130k base salary new grad role when I can hire a mid-career US returned FAANG dev in BLR or HYD for $70k-90k TC.
We will still hire new grads in the US for a $130k-$180k base, but they will have to actually be worth it. The brutal reality is, if you didn't attend a target CS program for your Bachelors (Stanford, Cal, MIT, UIUC, CMU, UT Austin, GT, UMich, UW, Cornell, Harvard, Columbia), at this point you probably aren't landing a high paying CS job - just like how in India if you didn't get a good JEE score, you're essentially relegated to being stuck at WITCH because you didn't get into a good BTech program, and it's an employer's market
A lot of people on HN assume Indians (and other foreign nationals) only do b*tch work like legacy springboot crap (and ofc plenty of people do), but an equally large cohort is doing legitimately competitive and innovative work.
> The brutal reality is, if you didn't attend a target CS program for your Bachelors (Stanford, Cal, MIT, UIUC, CMU, UT Austin, GT, UMich, UW, Cornell, Harvard, Columbia), at this point you probably aren't landing a high paying CS job
But what about after masters? What are the expectations there? Now of course there is the research path which is publishing well in your masters, working at deepmind or nvidia etc,. But I am talking about people doing non-thesis MS and then entering the job market. Is the base salary for those people still high?
> But what about after masters? ... and then entering the job market. Is the base salary for those people still high?
Not really especially when you factor in international tuition and loan servicing.
While it does depend on the program they attended, their past work experience in India, and their ability to land internships, in general I'd say salaries tend to be at the 25th percentile and below of what is shown on levels.fyi nationally.
But in all honestly, newly graduated masters students would be in the same boat as domestic undergrads if not worse becuase there isn't an incentive to hire an international student domestically and then go through the jumla of filing their H1B while they are on OPT unless they are a unicorn but if you are a unicorn, why would you even do a course based masters?!?
In fact, most of the complaints on HN about "H1Bs undercutting American salaries" tend to be these kinds of students because of how desperate they are to pay off a 1.5 to 2 crore loan with double digit interest rates despite having almost no scope of getting hired here in the US.
> for people who did not manage to get top 10% salaries here, the risk/reward is completely different
Yep.
Personal anecdote wise, I had a cousin who was working at Tech Mahindra in Chandigarh so on the lower end of salaries and got the US MS fuwara and they ended up attending UF (which is expensive and not a target).
We warned him not to come, but he didn't listen.
After spending almost $150k/1.5 crore getting the degree, the only SWE roles he could get could only pay $70k-80k in the NYC/NJ area which meant their take home after tax was rent was a little under $30k a year and this was during the 2022-24 period so no one wants to sponsor and he had a massive loan/EMI to pay off that was in the $17k-22k range a year.
He basically burnt through an extreme amount of family net worth (which was significant as they owned a house in Chandigarh) just to service a loan and basically remain in the same spot as before.
Another cousin did something similar to go to Canada despite us warning them not to do so either and their career is functionally screwed because it's even harder to find a role in Canada, but at least they attended a "college" so the tuition hit wasn't as severe.
In both cases it did not make financial sense for either to leave India for North America - their families had enough money in India that they could have bought significant property in Chandigarh and NCR and build generational wealth despite not necessarily working in the best jobs, but a lot of capital was basically burnt in loan repayment.
Like, let's be honest - the only people coming to the US to do a masters have a significant amount of capital because no one is going to guarantee a 2 crore loan without significant collateral, but at that point there are just better asset classes to diversify into.
From a financial standpoint, if you are Indian, it honestly isn't worth it doing some random coursework based masters in the US because in most cases you just aren't going to be hired by good companies anyhow, because none of us are willing to spend the capital to sponsor for an H1B after OPT unless you somehow ended up at a GAYMAN.
Interesting. Well, hope my friends get into GAYMAN companies then... whats the Y in there anyway? And is microsoft still excluded from the acronym? lol.
Ok, not asking in the context of this thread but in general: why do you think so? With their stake in openai especially and github being a good place to sell AI code assistants (one of the clear revenue streams for LLMs thus far) they seem quite well placed. Is it all their big "enterprise" and government contracts that bring them a lot of money now losing value?
1. Operational work is difficult to justify extremely high TCs
2. Other aspects of the business such as Cybersecurity just lack domestic talent in the US. That's why much of the MS Security portfolio engineering team is in Israel or India or people who are brought in via an L1/2
3. Vast swathes of MS employees from line level ICs up to Engineering Directors were stuck in immigration limbo because of the GC backlog or because of older parents, so a large number began returning to India with a slight shave on TC.
London and Toronto have a similar CoL as the Bay Area and $45/hr is a mid-career tech salaries in both Greater London and GTA.
Edit: can't reply but every single white collar job provides an employer healthcare plan that is equally as competitive as the public healthcare plans in Canada and the UK.
And especially if you were being paid $50/hr as a new grad in 2014.
Edit 2:
> And I was in Michigan.
All the more reason I would have pushed back severely. It's easier to find talent at scale in London or GTA - metros there have a population larger than the entire state of MI, and with a breadth of options beyond UMich Ann-Arbor.
$45/hr is low for GTA. I was making about that in Toronto in 2017 with two years experience, one year vocational degree, and a bachelor's in a completely unrelated field.
The vast majority of jobs in America give healthcare. The quality is vastly superior to London and Toronto, although we pay far more (and our medical professionals are upper middle class rather than middle / lower class). However this is a huge hidden portion of salary that most are not aware of, about $25k for a family of 4, which increases labor costs greatly.
On sheer metrics of access and quality, America kicks the shit out of Europe and Canada
> I was making $50/hour fresh out of college back in 2014. And I worked remotely.
Same for me, except out of high school, and a decade earlier.
That's the sordid tale of the industry. Outside of a handful of FAANG high flyers, pay, in real dollars, has been very steadily decreasing over many decades. But it took high inflation for us to notice.
Now we're in a difficult spot because we feel we need to make more to make up for the considerably higher cost of living, but there is no market willing to pay more — and never was.
The market can bear to pay high salaries for the right talent.
If you can show me you have tangible development skills and can think about the product or feature you develop as a business (eg. Can you justify to me in financial terms the net benefit doing a refactor does versus keeping the status quo) you will be fine.
We aren't going to pay you $300k-$400k TC just to be a code monkey. We expect you to be able to help inform actual business decisions and not be a PITA when thinking about the core metrics that matter for a business - NARR, FCF, and COGS.
So, being a developer who is specialized in a business domain (eg. Being a fullstack developer but with a decade of experience working on Cloud Security products) makes it easier for hiring managers to decide whether or not to hire you. And as a former PM, those kinds of Engineers are the best to work with becuase they understand the pitfalls that exist in a subdomain and have opinions and the ability to justify them.
Those who can upskill or show the ability to upskill are also worth their weight in TC.
And finally, you will have to be located in Tier 1 tech hubs now (Bay Area, Seattle, Austin, NYC). The 5-7 year blip of satellite offices in RTP or Denver or Portland or being 100% WFH in a cabin in Montana is over. The roles at these kinds of offices are the ones that get offshored first.
> And finally, you will have to be located in Tier 1 tech hubs now
Exactly. Adjusted for cost of living, $300,000 in SF or NYC is about $170,000 where I lived back then, so ~$80/hr. Which is, after adjusted for inflation to that time... You guessed it: $50.94/hr!
And you're pointing at high quality talent with considerable experience, not some kid out of high school. Said kid out of high school like I was back then isn't going to find that much in today's market. As you point out, the market has tanked big time — and has been tanking for decades.
As before, we're only just now starting to notice how far behind we’ve fallen because of things recently becoming exceptionally more expensive.
> Adjusted for cost of living, $300,000 in SF or NYC is about $170,000 where I lived back then, so ~$80/hr
I've lived in SF over the same time period as well, and my CoL hasn't changed aside from rent - I've kept the same consistent savings rate - but even rent was largely manageable for me because of job opportunities and a mix of local and state rent control.
That said, I do think being Asian or Latiné means having different buying preferences (eg. the bulk of my shopping is at ethnic grocery stores and my "white people food" is primarily sourced from Costco or TJ).
Ofc, looking visibly ethnic also gives me the ethnic discount at most places I shop at.
> Said kid out of high school like I was back then isn't going to find that much in today's market. As you point out, the market has tanked big time — and has been tanking for decades
Oof, that is actually a good point. Sadly, you are right about that. I don't think hiring managers in the US would take a risk on hiring a HSer even if they have the right skills and domain experience.
-----------
That said, I am starting to come around to your argument.
Jeez--I was making $50/hr in 2004 in one of my first jobs after I finished my PhD and opted out of academia. That pay didn't go that far back then!
This is what upsets me about my career--that logistic pay curve. You initially grow fast and then it tops out and never gets better, but your costs keep rising, particularly as you have a family. I'm paying for one kid's college tuition right now, she has 1.5 years left, and will then enter a dubious job market. My son is 15, so if he goes to college, I won't be paying tuition for him until Fall 2028.
The problem is, I'm no longer a developer. I'm currently a nothing working on figuring out "something." I have a lot of skills and talents, but seemingly not many that will pay. I'm looking at any 2yr. training program that can get me certified to do something useful. It's so freaking bizarre to be sitting here with a degree in CS/Math, an MS in Computer Engineering from a very reputable university, and a doctorate in Information Management, also from a very reputable university, and looking at basically doing blue collar work! My nation has utterly failed me in every possible manner.
I should add that we are looking to move out of our fairly high-cost of living state for a possibly lower cost of living state, but there are complications to that plan, too. My wife doesn't really want to move our son out of the high school he is in. I'm saying that other imperatives need to be addressed before they become full-blown crises. I'm being taxed to death, and costs like insurance are rising fast.
If I were you I'd be tempted to rewrite my PhD as a multi-year "special project". Reason is that hiring a PhD is controversial unless you're a university or a research division of a corporation. In contrast, the CS/Math & MS will always be solid & saleable.
> My nation has utterly failed me in every possible manner.
We've failed ourselves too, though. If I was some random person with endless money burning a hole in my pocket, what would I even do with a CS/Math/Computer engineer/Information manager? It is in no way clear how life is improved by working with such a person. Other industries put a lot of effort into marketing what function they serve. Said random person knows exactly when and why they'd want to hire a plumber, electrician, structural engineer, lawyer, accountant, physician, etc. But us...? We've rested on our laurels thinking Google, Microsoft, and Meta will forever want us, putting no effort into expanding our market.
You have to be able to get the prescription. HMOs (Kaiser specifically) will generally not provide any sort of preventative care in this area unless your numbers are very high. You can’t get access to a cardiologist unless you’ve already had an adverse event.
If you can get time off work and have a PPO, you can get the preventative care.
$10 is the cash price. Your doctor diagnoses, not insurance, and you don't need a specialist to get diagnosed regardless of what your insurer wants. Even a nurse practitioner can prescribe you a statin.
They are all like this to a degree because controversy creates engagement. If a platform is not making you money, is not making you smarter, and not helping you form IRL connections, then I highly recommend disabling it.
Managed hosting is expensive to operate and self-managing kafka is a job in of itself. At my last employer they were spending six figures to run three low volume clusters before I did some work to get them off some enterprise features, which halved the cost, but it was still at least 5x the cost of running a mainstream queue. Don't use kafka if you just need queuing.
Cheapest MSK cluster is $100 a month and can easily run a dev/uat cluster with thousands of messages a second. They go up from there but we've made a lot of use of these and they are pretty useful
I've basically never had a problem with MSK brokers. The issue has usually been "why are we rebalancing?" and "why aren't we consuming?", i.e. client problems.
It's not the dev box with zero integrations/storage that's expensive. AWS was quoting us similar numbers for MSK. Part of the issue is that modern kafka has become synonymous with Confluent, and once you buy into those features, it is very difficult to go back. If you're already on AWS and just need queuing, start with SQS.
Is it arrogance or is it experience combined with a different perspective? One developer may love React because of the component ecosystem and talent pool, and another developer may dislike it because they're writing custom HTML/CSS anyways and React requires them to write way more JS than their preferred approach. Would I ever choose backbone? No. But many developers may be surprised by how little vanilla JS that it takes to build modern web apps. More than ever the tradeoffs of different frontend stacks need to be evaluated on a project-by-project basis.
It is worth learning to use Docker Swarm. Deployments are as simple as pushing a new container to your registry and running one command. I built a free CLI tool rove.dev that simplifies provisioning and does service diffing.
Either that or use a PaaS that deploys to VMs. Can't make recommendations here but you could start by looking at Semaphore, Dokku, Dokploy.
Definitely check out swarm. I've heard so many great things from engineers that use it on large projects, and it takes very little time to learn if you already know the docker cli.
The non-invasive followup for people with positive test results would knock out a lot of the false-positives. At least, thats what I understand of "the usual result of a positive test result for a serious illness, is that a repeat test does not confirm it"
That said, at what level of risk of follow up diagnostic would you baulk? Any procedure which requires a general is bad news, and if you are over 70 its a lot more bad.
Their advertised sensitivity and specificity put them in the ballpark of what other liquid biopsies advertise. The ones I know of target high-risk cohorts where the benefits of other screenings already outweigh the risks of taking them. It doesn't make sense for the average person to be getting periodic full chest CT scans for instance, but it might for a decades-long smoker.
Sure, that's a concern. But for screenings like this the ultimate metric is all-cause mortality (perhaps adjusted for costs and quality of life). It will take several years before we have a clear signal on that.
At first glance, I don't understand the design choice of appending HTML templates to the python controller files. Seems like a lot of complexity just to remove a template render call. What am I missing?
ECS is good, just expensive and still requires more devops than it should. Docker Swarm is an easy way to run production container services on VMs. I built a free golang tool called Rove that provisions fresh Ubuntu VMs in one command and diffs updates. It's also easy-enough to use Swarm directly.
I’ve used a modified version of this for 8 years - I didn’t write it. Updating your ECS Docker image is just passing in the parameter of your new image and updating the cloudformation stack.
Honestly I didn't have a good experience with ECS (Fargate) - I remember I had to write a ton of CF deployment scripts+bash scripts, setting up a private AWS docker registry, having a terrible time debugging while my CF deployment always failed, deploys taking forever, finding out that AWS is too miserly to pay Docker to use the official repo so they are stuck on the free tier, meaning sometimes deploys would fail due to Dockerhub kicking the AWS docker agent out etc. It had limitations like not being able to attach a block volume to the docker instance, so overall I remember spending a week setting up the IaC for a simple-ass CRUD app on Fargate ECS.
Setting up the required roles and permissions was also a nightmare. The deployment round trip time was also awful.
The 2 good experiences I had with AWS was when we had a super smart devops guy who set up the whole docker pipeline on top of actual instances, so we could deploy our docker compose straight to a server in under 1 minute (this wasn't a scaled app), and had everything working.
Lambda is also pretty cool, you can just zip everything up and do a deploy from aws cli without much scripting and pretty straightforward IaC.
A lot of AWS requires way too much config. It is a mystery to me why AWS doesn't lean into extending the capabilities of App Runner. I actually built a whole continuous deployment PaaS for AWS ECS with a Heroku-like UX, ended up shutting it down eventually because although useful, their pricing is pretty awful. What I need to do is figure out how to bring it back, just minus the hosted service so I can use it on corporate projects that require AWS...
Yeah I haven't had any issues with Swarm. Heard good things from people running substantial clusters. Would be interested in hearing about what rough edges people have run into as well!
But if you take the government out of the equation, and instead mandate contribution via project licenses, that might be worth a try.
reply