yes, they have. It just costs a shit ton of money and is extremely difficult to get the suits to sign off on TWO full 'cloud services' bills. It generally doubles your cost and workload and increases your uptime by a couple hours/year, assuming you don't have bugs that affect one or the other cloud in your deployment stack.
It's basically a wash for almost all organizations for twice the cost and effort.
also these things don't go down THAT often... well aws, not some others. More uptime that you probably had before. even the stock market takes a few days off every decade. Just ask W.
It's basically what leads to extended downtime almost every time. There are just some things in the stack that are still single points of failure, and when they fail it's a mess.
Sometimes the circular dependencies get almost cartoonishly silly.
Like, "One of the two guys who has the physical keys to the server cage in us-east-1 is on vacation. The other one can't get into his apartment because his smart lock runs into the AWS cloud. So he hires a locksmith, but the locksmith takes an extra two hours to do the job because his reference documents for this model of lock live on an S3 bucket."
We had a pair of machines. And some bright spark set them up to mount each others NFS shares. after a power outage "Holy mother of chicken and egg NFS hangs batman"
That was a weird job, fun, it was a local machine room for a warehouse that originally held the IBM mainframe, it still held it's successor "the multiprise 3000" which has the claim to fame as being the smallest mainframe IBM ever sold. But now the room was also full of decades of artisanal crafted unix servers with pick databases. the pick dev team had done most the system architecture. The best way to understand it is that for them pick is the operating system, unix is a necessary annoyance they have to put up with only because nobody has made pick hardware for 20 years. and it was NFS mounts everywhere, somebody had figured out a trick where they could NFS mount a remote machine and have the local pick system reach in and scrounge through the remote systems data. But strictly read-only. pick got grumpy when writing to NFS not to say anything about how the other database would feel about having it's data being messed with. Thus the circular mount.
Still was not the worst thing I saw. I liked the one system with a SMB mount. "Why is this one SMB?" "Well pick complains when you try to write to a NFS mount, but it's NFS detection code does not trip on SMB mounts." ... Sighs "Um... I am no pick expert but you know why it does not like remote mounts right. SMB does not change that, Do you happen to get a lot of corrupt indexes on this machine?"
"yes, how did you know"
Oh, yeah, re-exporting NFS mounts via SMB was very much a thing in the early 2000s - something to do with their different approaches to flock() vs fcntl() handling. If you ran into locking issues with nfs, then re-exporting via SMB was standard advice.
At some point, the behaviour changed and locks starting conflicting. IIRC, we hit it when upgrading to Debian Etch and took the time to unwind the system and make pure NFS work properly for us. Plenty of people took the opposite approach, and fiddled with the config to make locking a noop on SMB. I know of at least one web hosting company who ended up having to restore a year's worth of customer uploads from backups as a result...
> Our primary and out-of-band network access was down, so we sent engineers onsite to the data centers to have them debug the issue and restart the systems. But this took time, because these facilities are designed with high levels of physical and system security in mind. They’re hard to get into, and once you’re inside, the hardware and routers are designed to be difficult to modify even when you have physical access to them. So it took extra time to activate the secure access protocols needed to get people onsite and able to work on the servers. Only then could we confirm the issue and bring our backbone back online.
There was one (later denied) report that a 'guy with an angle grinder' was involved in gaining access to the server cage.
Why would such a critical server even be accessible with only one set of keys?
I’ve always thought mission critical stuff needs two independent key holders, with key holes placed far apart enough to make it impossible for 1 person to reach both.
I dont know how it is in the datacentre industry, but certainly in other industries that is how its done for anything truly mission critical and also easily tampered with.
I guess it shows very few care enough to pay enough to make that a reasonable upgrade.
They're not actually accessible with 'only one set of keys' in my experience.
You actually have to present your photo ID at the site entry gatehouse, then again to the building entry guard (who will also check you have a work permit and a site-specific safety induction) then you swipe a badge at a turnstile to get from reception into the stairwell, then swipe your badge at a door to get into the relevant floor, then swipe your badge and key in a code to enter the room with the cages then you use the key.
a circular dependency and a single point of failure are not the same thing. If I have a single point of failure and it is down, I fix that and things work again. If I have circular dependency, there is no obvious way to fix anything that is broken any longer.
You might be counting out the value of government and military contracts that might not want to do business with a wild card.
SpaceX is killing it because the US government gives them a bunch of contracts, but if stability is slightly more important than cost or speed, amazon has a contender.
You can definitely average two relatively accurate chronometers but you if you only have two it’s difficult to tell if one is way fast or way slow.
In a perfect world they drift less than a minute per day and you’re relatively close to the time with an average or just by picking one and knowing that you don’t have massive time skew.
I believe this saying was first made about compasses which also had mechanical failures. Having three lets you know which one failed. The same goes for mechanical watches, which can fail in inconsistent ways, slow one day and fast the next is problematic the same goes for a compass that is wildly off, how do you know which one of the two is off?
> In a perfect world they drift less than a minute per day...
A minute per day would be far too much drift for navigation, wouldn't it?
From Wikipedia [1]:
> For every four seconds that the time source is in error, the east–west position may be off by up to just over one nautical mile as the angular speed of Earth is latitude dependent.
That makes me think a minute might be your budget for an entire voyage? But I don't know much about navigation. And it is beside the point, your argument isn't changed if we put in a different constant, so I only mention out of interest.
> Having three lets you know which one failed.
I guess I hadn't considered when it stops for a minute and then continues ticking steadily, and you would want to discard the measurement from the faulty watch.
But if I just bring one watch as the expression councils, isn't that even worse? I don't even know it malfunctioned and if it failed entirely I don't have any reference for the time at the port.
My interpretation had been that you look back and forth between the watches unable to make a decision, which doesn't matter if you always split the difference, but I see your point.
> A minute per day would be far too much drift for navigation, wouldn't it?
Even that was much better than the dead-reckoning they had to do in bluewater before working chronometers were invented. Your ship's "position" would be a triangle that might have sides ten miles long at lower latitudes.
I’ve never heard the bring one or three, I’ve always just heard three. I think that exact saying implies that if you have two and one isn’t working out you’ll go crazy but if you have one you’ll be oblivious until it’s too late.
A well serviced rolex in 2026 with laser cut gears drifts +/- 15sec per day.
One with hand filed gears is going to be +/- a minute on a good day, and that’s what early navigation was using. I have watches with hand filed gears and they can be a bit rough.
Prior to that, it was dead reckoning, dragging a string every now and again to calculate speed and heading and the current and then guesstimating your location on a twice daily basis.
Those two wildly inaccurate systems mapped most of the world for us.
Reading the comment thread here made me realize the idea seems to be that having 2 just means double the probability of one of them failing in some undetectable way. The resulting error magnitude is reduced by half, but the probability of that error is doubled. So it gains you nothing to expected value to have 2. Unlike with 3, where the probability of undetectable failure and the error rate from partial failure are both reduced by the ability to make comparative measurements (eg pick the middle number not the average)
Though not without significant errors, the most amusing to me being that islands had a tendency to multiply because different maps would be combined and the cartographer would mistake the same island on two maps as being separate islands due to errors. A weird case of aliasing I suppose.
The book “Longitude” is fascinating, and discusses the challenges prior to chronometers (many people died), as well as the rewards offered for a precise chronometer, the attempts, etc.
I hope not because I’ve been doing my US taxes on Linux for 15 years.
It’s probably a specific windows desktop app, probably TurboTax by intuit, the company that lobbies to make filing your taxes hard and to destroy any free simple government app to file taxes.
So, not sure why they’d complain about not being able to help shoot their foot off but we all have preferences. :shrug:
My sister was the same, she brought her machine over, I booted a Ubuntu disk and did the disk config in the install and then she set the rest of the stuff up and I haven’t heard from her about it for 5 years, other than that I check if she’s still using it now and again.
If you do regulate. We currently have full regulatory capture in most industries and regulators that are doing their jobs are either hamstrung or the laws are so far behind the industries that they can’t or won’t work.
The key to proper regulation is to keep money and influence from pooling at the top, making it difficult for any single person to buy enough influence.
As it is, we have a dozen monopolies that should be broken up that are making a small section of the population so rich they are essentially above laws.
But, proper regulation can exist if people want it, and more specifically in the case of the USA, legislators want it. Unfortunately, Dems actively prevent it, and republicans are ripping it down, so the rest of us are kinda fucked.
Regulations can work if bypassing the regulation in question does not open up a market that is large enough to keep paying off the regulators.
For example, if there is only one regulator for a country, the companies can pay millions to get it eased up for them, because they can make billions from it.
But if there one regulator for each state, they equation will change and it might not be profitable to pay millions to a regulator of the state, because they cannot make enough profit from selling in the state to justify it.
That is the only way to make it work. Rules don't work forever. Incentives do.
It's basically a wash for almost all organizations for twice the cost and effort.