So this is a good thing even for coreutils itself, they will slowly find all of these untested bits and specify behaviour more clearly and add tests (hopefully).
Man, if I had a nickel every time some old Linux utility ignored a command-line flag I'd have a lot of nickels. I'd have even more nickels if I got one each time some utility parsed command-line flags wrong.
I have automated a lot of things executing other utilities as a subprocess and it's absolutely crazy how many utilities handle CLI flags just seemingly correct, but not really.
This doesn't look like a bug, that is, something overlooked in the logic. This seems like a deliberately introduced regression. Accepting an option and ignoring it is a deliberate action, and not crashing with an error message when an unsupported option is passed must be a deliberate, and wrong, decision.
It certainly doesn't look intentional to me- it looks like at some point someone added "-r" as a valid option, but until this surfaced as a bug, no one actually implemented anything for it (and the logic happens to fall through to using the current date).
It's wrong (and coreutils get it right) but I don't see why it would have to be deliberate. It could easily just not occur to someone that the code needs to be tested with invalid options, or that it needs to handle invalid options by aborting rather than ignoring. (That in turn would depend on the crate they're using for argument parsing, I imagine.)
Could parsing the `-r` be added without noticing it somehow?
If it was added in bulk, with many other still unsupported option names, why does the program not crash loudly if any such option is used?
A fencepost error is a bug. A double-free is a bug. Accepting an unsupported option and silently ignoring it is not, it takes a deliberate and obviously wrong action.
At least from what I can find, here's the original version of the changed snippet [0]:
let date_source = if let Some(date) = matches.value_of(OPT_DATE) {
DateSource::Custom(date.into())
} else if let Some(file) = matches.value_of(OPT_FILE) {
DateSource::File(file.into())
} else {
DateSource::Now
};
And after `-r` support was added (among other changes) [1]:
let date_source = if let Some(date) = matches.get_one::<String>(OPT_DATE) {
DateSource::Human(date.into())
} else if let Some(file) = matches.get_one::<String>(OPT_FILE) {
match file.as_ref() {
"-" => DateSource::Stdin,
_ => DateSource::File(file.into()),
}
} else if let Some(file) = matches.get_one::<String>(OPT_REFERENCE) {
DateSource::FileMtime(file.into())
} else {
DateSource::Now
};
Still the same fallback. Not sure one can discern from just looking at the code (and without knowing more about the context, in my case) whether the choice of fallback was intentional and handling the flag was forgotten about.
> Accepting an unsupported option and silently ignoring it is not, it takes a deliberate and obviously wrong action.
No, it doesn't. For example, you could have code that recognizes that something "is an option", and silently discards anything that isn't on the recognized list.
I would say that Canonical is more at fault in this case.
I'm frankly appalled that an essential feature such as system updates didn't have an automated test that would catch this issue immediately after uutils was integrated.
Nevermind the fact that this entire replacement of coreutils is done purely out of financial and political rather than technical reasons, and that they're willing to treat their users as guinea pigs. Despicable.
What surprises me is that the job seems rushed. Implementation is incomplete. Testing seems patchy. Things are released seemingly in a hurry, as if meeting a particular deadline was more important for the engineers or managers of a particular department than the qualify of the product as a whole.
This feels like a large corporation, in the bad sense.
Yes, it was basically gone for a decade or more. There’s no shared code. Though I’m sure they may have looked at the old code for inspiration for some of the Win32 stuff.
Besides the ecosystem issues, for the phishing part, I'll repost what I responded somewhere in the other related post, for awareness
---
I figure you aren't about to get fooled by phishing anytime soon, but based on some of your remarks and remarks of others, a PSA:
TRUSTING YOUR OWN SENSES to "check" that a domain is right, or an email is right, or the wording has some urgency or whatever is BOUND TO FAIL often enough.
I don't understand how most of the anti-phishing advice focuses on that, it's useless to borderline counter-productive.
What really helps against phishing :
1. NEVER EVER login from an email link. EVER. There are enough legit and phishing emails asking you to do this that it's basically impossible to tell one from the other. The only way to win is to not try.
2. U2F/Webauthn key as second factor is phishing-proof. TOTP is not.
That is all there is. Any other method, any other "indicator" helps but is error-prone, which means someone somewhere will get phished eventually. Particularly if stressed, tired, or in a hurry. It just happened to be you this time.
> 1. NEVER EVER login from an email link. EVER. There are enough legit and phishing emails asking you to do this that it's basically impossible to tell one from the other. The only way to win is to not try.
Sites choosing to replace password login with initiating the login process and then clicking a "magic link" in your email client is awful for developing good habits here, or for giving good general advice.
:c
In that case it's the same as a reset-password flow.
In both cases it's good advice not to click the link unless you initiated the request. But with the auth token in the link, you don't need to login again, so the advice is still the same: don't login from a link in your email; clicking links is ok.
Clicking links from an email is still a bad idea in general because of at least two reasons:
1. If a target website (say important.com) sends poorly-configured CORS headers and has poorly configured cookies (I think), a 3rd-party website is able to send requests to important.com with the cookies of the user, if they're logged in there. This depends on important.com having done something wrong, but the result is as powerful as getting a password from the user. (This is called cross-site request forgery, CSRF.)
2. They might have a browser zero-day and get code execution access to your machine.
If you initiated the process that sent that email and the timing matches, and there's no other way than opening the link, that's that. But clicking links in emails is overall risky.
1 is true, but this applies to all websites you visit (and their ads, supply chain, etc). Drawing a security boundary here means never executing attacker-controlled Javascript. Good luck!
2 is also true. But also, a zero day like that is a massive deal. That's the kind of exploit you can probably sell to some 3 letter agency for a bag. Worry about this if you're an extremely high-value target, the rest of us can sleep easy.
I watched a presentation from Stripe internal eng that was given I forget where.
An internal engineer there who did a bunch of security work phished like half of her own company (testing, obviously). Her conclusion, in a really well-done talk, was that it was impossible. No human measures will reduce it given her success at a very disciplined, highly security conscious place.
The only thing that works is yubikeys which prevent this type of credential + 2fa theft phishing attack.
> At Stripe, rather than focusing on mitigating more basic attacks with phishing training, we decided to invest our time in preventing credential phishing entirely. We did this using a combination of Single Sign On (SSO), SSL client certificates, and Universal Second Factor
(U2F)
I receive Google Doc links periodically via email; fortunately they're almost never important enough for me to actually log in and see what's behind them.
My point, though, is that there's no real alternative when someone sends you a doc link. Either you follow the link or you have to reach out to them and ask for some alternative distribution channel.
(Or, I suppose, leave yourself logged into the platform all the time, but I try to avoid being logged into Google.)
I don't know what to do about that situation in general.
A Firefox plugin/feature, probably also available on other browsers as well. It is useful for siloing cookies, so you can easily be logged into Google on one set of browser tabs and block their cookies on another.
As for any of these cases, we do receive legitimate emails that require being logged in, Google or otherwise
The answer is simple: use your bookmarks/password manager/... to login yourself with a URL you control in another tab and come back to the email to click it
(and if it still asks for a login then, of course still don't do it)
A browser-integrated password manager is only phishing-proof if it's 100% reliable. If it ever fails to detect a credential field, it trains users that they sometimes need to work around this problem by copy-pasting the credential from the password manager UI, and then phishers can exploit that. AFAIK all existing password manager extensions have this problem, as do all browsers' native password-management features.
It doesnt need to be 100% reliable, just reliable enough.
If certain websites fail to be detected, thats a security issue on those specific websites, as I'll learn which ones tend to fail.
If they rarely fail to detect in general, its infrequent enough to be diligent in those specific cases. In my experience with password managers, they rarely fail to detect fields. If anything, they over detect fields.
I think it's more appropriate to say TOTP /is (nearly)/ phishing-proof if you use a password manager integrated with the browser (not that it /doesn't need to be/ phishing-proof)
> U2F/Webauthn key as second factor is phishing-proof. TOTP is not.
Last I checked, we're still in a world where the large majority of people with important online accounts (like, say, at their bank, where they might not have the option to disable online banking entirely) wouldn't be able to tell you what any of those things are, and don't have the option to use anything but SMS-based TOTP for most online services and maybe "app"-based (maybe even a desktop program in rare cases!) TOTP for most of the rest. If they even have 2FA at all.
This is the point of the "passkey" branding. The idea is to get to the point where these alphabet-soup acronyms are no longer exposed to normal users and instead they're just like "oh, I have to set up a passkey to log into this website", the way they currently understand having to set up a password.
Sure. That still doesn't make Yubikey-style physical devices (or desktop keyring systems that work the same way) viable for everyone, everywhere, though.
Yeah, the pressure needs to be put on vendors to accept passkeys everywhere (and to the extent that there are technical obstacles to this, they need to be aggressively remediated); we're not yet at the point where user education is the bottleneck.
At least the crowd here should _know_ that TOTP doesn't do anything against phishing, and most of the critical infrastructure for code and other things support U2F so people should use it.
Urgency is also either phishing (log in now or we'll lock you out of your account in 24 hours) or marketing (take advantage of this promotion! expires in 24 hours!).
A guy I knew needed a car, found one, I told him to take it to a mechanic first. Later he said he couldn't, the guy had another offer, so he had to buy it right now!!!, or lose the car.
I mean, real deadlines do exist. The better heuristic is that, if a message seems to be deliberately trying to spur you into immediate action through fear of missing a deadline, it's probably some kind of trick. In this respect, the phishing message that was used here was brilliantly executed; it calmly, without using panic-inducing language, explains that action is required and that there's a deadline (that doesn't appear artificially short but in fact is coming up soon), in a way quite similar to what a legitimate action-required email would look like. Even a savvy user is likely to think "oh, I didn't realize the deadline was that soon, I must have just not paid attention to the earlier emails about it".
Yeah, this particular situation's a bit weird because it's asking the user to do something (rotate their 2FA secret) that in real life is not really a thing; I'm not sure what to think of it. But you could imagine something similar like "we want you to set up 2FA for the first time" or "we want you to supply additional personal information that the government has started making us collect", where the site might have to disable some kind of account functionality (though probably not a complete lockout) for users who don't do the thing in time.
I had someone from a bank call me and ask for my SSN to confirm my identity. The caller ended up being legitimate, but I still didn't give it...like, are you kidding me?
This has happened to me more times than I can count, and it's extremely frustrating because it teaches people the wrong lesson. The worst part is they often get defensive when you refuse to cooperate, which just makes the whole thing unnecessarily more stressful.
Is there somewhere you'd recommend that I can read more about the pros/cons of TOTP? These authenticator apps are the most common 2FA second factor that I encounter, so I'd like to have a good source for info to stay safe.
1- As a professional, installing free dependencies to save on working time.
There's no such thing as a free lunch, you can't have your cake and eat it too that is, download dependencies that solve your problems, without paying, without ads, without propaganda (for example to lure you into maintaining such projects for THE CAUSE), without vendor lockin or without malware.
It's really silly to want to pile up mountains of super secure technology like webauthn, when the solution is just to stop downloading random code from the internet.
I agree that #1 is correct, and I try to practice this; and always for anything security related (update your password, update your 2FA, etc).
Still, I don’t understand how npmjs.help doesn’t immediately trigger red flags… it’s the perfect stereotype of an obvious scam domain. Maybe falling just short of npmjshelp.nigerianprince.net.
should practice it for ENTER your password, ENTER your 2FA ;)
> Still, I don’t understand how npmjs.help doesn’t immediately trigger red flags
1. it probably did for quite a few recipients, but that's never going to be 100%
2. not helped by the current practices of the industry in general, many domains in use, hard sometimes to know if it's legit or not (some actors are worse in this regard than others)
Either way, someone somewhere won't pay enough attention because they're tired, or stressed out, or they are just going through 100 emails, etc.
Most mail providers have something like plus addressing. Properly used that already eliminates a lot of phishing attempts: If I get a mail I need to reset something for foobar, but it is not addressed to me-foobar (or me+foobar) I already know it is fraudulent. That covers roughly 99% of phishing attempts for me.
The rest is handled by preferring plain text over HTML, and if some moron only sends HTML mails to carefully dissect it first. Allowing HTML mails was one of the biggest mistakes for HTML we've ever made - zero benefits with huge attack surface.
Still would have done nothing in this case, as they pulled the correct email address he uses for npm from another source (public API I think?).
That's exactly why I said all the other "helpful" recommendations and warning signs people are using are never foolproof, and thus mostly useless given the scale at which phishing campaigns operate.
Great if it helps you in the general case, terrible if it lulls you into a sense of confidence when it's actually a phishing email using the right email address.
It's definitely not the world's messaging market. For instance in Japan and many places in SEA, Line is the standard messenger - one many people probably haven't even heard of. Though it does have a nice play on words - are you on Line?
It’s not uncommon. Orkut back in the day was wildly popular in Latin America and India. WhatsApp is the same. I think users in NA have a lot of high quality options as against those in Asia and LatAm who don’t have much reliable options other than ones developed in NA.
You can get an android phone for about one tenth of what a new iPhone costs. That’s why android dominates lower income markets. Apple decided they just don’t want to be there.
Yeah, huge in Latin America in the sense that a lot (most?) business only have a number that they use with Whatsapp (you can't call or even text them). Is it the same in Europe? Since I am from Latin America I never know if people from other continents use Whatsapp as much as we do, and if when I ask them to use Whatsapp I am imposing a new app or it's what they regularly use.
No. Here in Germany WhatsApp is not even that widespread for businesses. But WA is very big here for personal communication, though Signal comes in second (at least amongst older people, and amongst my circle)
I think Europe is not homogenous enough for this, but in the Netherlands at least, there are plenty of companies that you can't call, email or text, but they'll have some other options: a chatbot, a web form, maybe a Twitter account, and sometimes via WhatsApp indeed.
I’m not sure that’s true. I’m fairly certain UK, France, AU, Canada WhatsApp is not vastly more popular than the blue bubble alternative. At least I believe this was the case a few years ago, based on data I’d seen.
> Blue bubble isn't really a thing ever mentioned in France either, not enough iPhone market share.
Nobody uses iMessage. People with iPhone use WhatsApp too.
The user experience of iMessage used to be subpar and now everyone has WhatsApp installed anyway, the feature set is the same and it works on all phone brands so nobody feels like switching.
Same in the UK. The fact that iMessage only works for iOS devices means it's a complete non-starter. What's the point in using a messaging app if you can't add all your contacts to a group? And if you're using a different app for group chats for this reason, then why not use it for 1-1 messaging, too?
I guess that it’s the iPhone’s messenger app? I heard that in that app, fellow iOS users have blue bubble messages and Android / other users have green bubble messages, and all the teens in the US /maybe Canada think it’s lame if you don’t have blue bubbles.
Oh. I remember hearing about that about 15y ago, didn't realise it was still a thing. I suppose because I haven't heard of anyone using iMessage for almost as long!
Hey, you're doing an exemplary response, transparent and fast, in what must be a very stressful situation!
I figure you aren't about to get fooled by phishing anytime soon, but based on some of your remarks and remarks of others, a PSA:
TRUSTING YOUR OWN SENSES to "check" that a domain is right, or an email is right, or the wording has some urgency or whatever is BOUND TO FAIL often enough.
I don't understand how most of the anti-phishing advice focuses on that, it's useless to borderline counter-productive.
What really helps against phishing :
1. NEVER EVER login from an email link. EVER. There are enough legit and phishing emails asking you to do this that it's basically impossible to tell one from the other. The only way to win is to not try.
2. U2F/Webauthn key as second factor is phishing-proof. TOTP is not.
That is all there is. Any other method, any other "indicator" helps but is error-prone, which means someone somewhere will get phished eventually. Particularly if stressed, tired, or in a hurry. It just happened to be you this time.
1. You just requested it, I'm not saying to never click link on transactional emails you requested. You still need to click on those verify email links
2. It replaces entering your password, so you're not entering your password on a link from an email, which is the very wrong thing.
At least you've requested that email, to be able to login. The timing chance for a phishing mail to come here and there is insignificant. OP is referring to communications that are one way street, the (pseudo) organisation to you.
It's very ergonomic for those who discovered the internet via an iPhone, who think Gmail is email. They can't remember their passwords, and wouldn't know where how to recover most cryptographic factors. They have an email account they tend to have access to and use magic links to login , they are very happy with that.
Not promoting the pattern, I also find it worrying the majority of internet users have no basic understanding of authentication and the risk for their digital identity.
I agree. However you use them less often, so its far harder for someone to time it right.
If you use username instead of email address attackers have to guess that too.
One quite serious problem I see quite often is using email plus password for login, and notifying on failed login that the email is not in the system, letting attackers validate which emails are logins.
It happens less often, but it's also more believable that it would be sent without a user action—e.g. "We had a security incident. Please click here to change your password."
And this is exactly the kind of phishing attack that is most effective, as this particular incident shows. So I'd say it's actually a worse phishing vector than magic links.
Or you know, get a password manager like the rest of us. If your password manager doesn't show the usual autofill, since the domain is different than it should, take a step back and validate everything before moving on.
Have the TOTP in the same/another password manager (after considering the tradeoffs) and that can also not be entered unless the domain is right :)
I feel like it's extremely common for the autofill to not work for various reasons even when you aren't being phished. I have to manually select the site to fill fairly often, especially inside apps where the password manager doesn't seem to match the app to the website password.
Passkeys seem like the best solution here where you physically can not fall for a phishing attack.
> I feel like it's extremely common for the autofill to not work for various reasons even when you aren't being phished.
This is how Troy Hunt got phished. He was already very tired after a long flight, but his internal alarm bells didn't ring loud enough, when the password manager didn't fill in the credentials. He was already used to autofill not always working.
This is why I haven't bothered with them (the browser extensions; I have used password managers for years and years) and thus why they weren't there to protect against the attack.
> I feel like it's extremely common for the autofill to not work for various reasons even when you aren't being phished
I dunno, it mostly seems to not work when companies change their field names/IDs, or just 3rd party authentication, then you need to manually add domains. Otherwise my password manager (1Password) works everywhere where I have an account, except my previous bank which was stuck in the 90s and disallowed pasting the passwords. If you find that your password manager doesn't work with most websites (since it's "extremely common") you might want to look into a different one, even Firefox+Linux combo works extremely well with 1Password. Not affiliated, just a happy years+ user.
> Passkeys seem like the best solution here where you physically can not fall for a phishing attack.
Yeah, I've looked into Passkeys but without any migration strategy or import/export support (WIP last time I looked into it), it's not really an alternative just yet, at least for me personally. I have to be 100% sure I can move things when the time ultimately comes for that.
I'm glad you've had such good experience with autofill consistently working for you. My experience has been closer to that of the sibling comments: 60/40 so I often just give up and copy-paste. I actually did try jettisoning 1Password for Proton Pass but that was even worse, so I went back
> without any migration strategy or import/export support
Since you're already a 1Password user, I wanted to draw your attention to the "Show debugging tools" in the "Settings > Advanced" section. From that point, you can say "Copy Item JSON" and it will give you the details you would want for rescuing the Passkey. Importing it into something else is its own journey that I can't help with
You only need read the whole thread however to see reasons why this would sometimes not be enough: sometimes the password manager does not auto-fill, so the user can think it's one of those cases, or they're on mobile and they don't have the extension there, or...
> So pick one that does? That's like its top 2 feature
Still doesn’t work 100% of the time, because half of the companies on earth demote their developer time to breaking 1995-level forms. That’s why every popular password manager has a way to fill passwords for other domains, why people learn to use that feature, and why phishers have learned to convince people to use that feature.
WebAuthn prevents phishing. Password managers reduce it. This is the difference between being bulletproof like Superman or a guy in a vest.
Given recent vuln of password manager extensions on desktop leaking passwords to malicious sites, I have disabled autofill on desktop... And autofill didn't work for me on ycombinator on mobile... Autofill is too unreliable.
You don't need 100%, just a high enough frequency that you wouldn't get used to dismissing the fail on auto pilot. Perfect shouldn't be the enemy of the good?
Then good password managers will still show you only the logins for that domain. If the login is on another domain then you would have saved it anyways when first logging in/registering and if the site moved then you can get suspicious and check carefully first.
All password managers allow copy-paste (which is what happened here) and the popular ones all offer you the ability to search and fill passwords from other domains. It's important to understand why they do, because it's also why these attacks continue to work: the user _thinks_ they are working around some kind of IT screwup, and 9 times out of 10 (probably closer to 99 out of 100) that's correct. Every marketing-driven hostname migration, every SSO failure, every front-end developer who breaks autofill, every “security expert” who was an accountant last year saying password managers are a vulnerability helps train users to think that it's not suspicious when you have to search for a different variation of the hostname or copy-paste a password.
That's why WebAuthn doesn't allow that as a core protocol feature, preventing both this attack and shifting the cost of unnecessary origin changes back to the company hosting the site. Attacking this guy for making a mistake in a moment of distraction is like prosecuting a soldier who was looking the other way when someone snuck past: wise leaders know that human error happens and structure the system to be robust against a single mistake.
Personally a big fan of 1Password. On the topic of autofill, the only website it sometimes won't fill is Reddit, which you know, whatever, I never go there anymore anyway.
As a developer I also love their ssh and gpg integrations, very handy.
I do get it for free from work, but if I had to choose one myself I'd have to pay for I'd probably still pick 1Passwrod.
> I do get it for free from work, but if I had to choose one myself I'd have to pay for I'd probably still pick 1Passwrod.
I wanted to highlight that "getting it for free from work" isn't a sweetheart deal offered just to OP, but a feature of 1Password for Teams, meaning all employees of a business that uses 1Password automatically have a Family license for use at home https://support.1password.com/link-family/
And, for clarity, it's merely a financial relationship: the business cannot manage your Family account, cannot see its contents, and if you have a separation event you can retain the Family account forever in a read only capacity or you can take over the payment (or, heh, I presume move to another employer that also uses 1Password) and nothing changes for your home passwords
He didn't say it didn't have the autofill feature, he said sometimes it doesn't work. I've experienced this pretty routinely with two different managers.
I wish it's that easy. 1Password autofill on Android Chrome broke for me a month ago. Installed all updates, checked settings, still nothing. Back to phishing prone copy paste.
Or go ahead and use them, but abort if your password manager doesn't auto fill. Such abort scenarios include not only a password field without auto fill, but also a total lack of password field (e.g., sites that offer OTP-only authentication), since either way you don't have your password manager vetting the domain.
Happy for you, don't get me wrong, but your post is not particularly news, I'm guessing everyone on HN knows bare metal/VPS providers are cheaper than AWS/Azure/GCP.
And also lacking a bit in details:
- both technical (e.g. how are you dealing with upgrades or multi-data center fallback for your postgresql), and
- especially business, e.g. what's the total cost analysis including the supplemental labor cost to set this up but mostly to maintain it.
Maybe if you shared your scripts and your full cost analysis, that would be quite interesting.
I'm trying to share as much technical across this thread as for your two examples:
System upgrades:
Keep in mind that as per the ISO specification, system upgrades should be applied but in a controlled manner. This lends itself perfectly to the following case that is manually triggered.
Since we take steps to make applications stateless, and Ansible scripts are immutable:
We spin up a new machine with the latest packages and once ready it join the Cloudflare load balancer. The old machines are drained and deprovisioned.
we spin up a new machine We have a playbook that iterates through our machines and does it per machine before proceeding. Since we have redundancy on components, this creates no downtime. The redundancy in the web application is easy to achieve using the load balancer in Cloudflare. For the Postgres database, it does require that we switch the read-only replica to become the main database.
DB failover:
The database is only written and read from by our web applications. We have a second VM on a different cloud that has a streaming replication of the Postgres database. It is a hot standby that can be promoted. You can use something like PG Bouncer or HAProxy to route traffic from your apps. But our web framework allows for changing the database at runtime.
> Business
Before migration (AWS): We had about 0.1 FTE on infra — most of the time went into deployment pipelines and occasional fine-tuning (the usual AWS dance).
After migration (Hetzner + OVHCloud + DIY stack): After stabilizing it is still 0.1 FTE (but I was 0.5 FTE for 3-4 months), but now it rests with one person. We didn’t hire a dedicated ops person.
On scaling — if we grew 5-10×:
* For stateless services, we’re confident we’d stay DIY — Hetzner + OVHCloud + automation scales beautifully.
* For stateful services, especially the Postgres database, I think we'd investigate servicing clients out of their own DBs in a multi-tenant setup, and if too cumbersome (we would need tenant-specific disaster recovery playbooks), we'd go back to a managed solution quickly.
I can't speak for cloud FTE toll vs a series of VPS servers in the big boys league ($ million in monthly consumption) and in the tiny league but at our league it turns out that it is the same FTE requirement.
Anyone want to see my scripts, hit me up at jk@datapult.dk. I'm not sure it'd be great security posture to hand it out on a public forum.
Cloudflare could be considered a point of failure and is another level of complexity compare to doing your own LB (the extra is the external org — actually extra both in terms of tech and of compliance).
Have you considered doing your own HA Load balance? If yes what tech options did you consider
I took for granted that Hetzner and OVHcloud would be prone to failures due to their bad rep, not my own experience, so I wanted to be able to direct traffic to one if the other was down.
Doing load balancing ourselves in either of the two clouds gave rise to some chicken and egg situations now that we were assuming that one of them could be down (again not my lived experience).
Doing this externally was deliberate and picking something with a better rep than Hetzner and OVHcloud was obvious in that case.
The network allows relevant ports from the respective IPs and so does the UFW so the servers can communicate between in each other in a restricted way. Needless to say communication is encrypted with certificates.
Our logging server will switch primary DB in case the original primary DB server is down. Since we are counting on downtime, the monitoring server is by default not hosted in the same places as the primary DB but in the same place as the secondary DB.
We assume that each clod will go down but not at the same time.
