Pretty sure you're being facetious, but yes, each team of SDEs is responsible for their own operations. Reliability Engineer positions do exist, but there doesn't seem to be a company-wide standard job title.
There is a group (Operational Excellence) that focuses on things you'd expect SREs to focus on, but I think they focus more on building the tools than actual operational support.
Someone was probably promoted for the adoption of this "frugal" hiring practice. This is a culture problem that permeates all of Amazon and goes all the way to the top.
Find a new job. The reputation of your company is going down the drain.
With all due respect, you don't know what you're talking about :)
I, and most people I know have mostly positive Amazon experiences. We are proud of the work we do, and the innovations of the company. I interview a LOT, and I make sure that 100% of my candidates come out with a positive experience, even if they do not get hired.
There are problems at every company, but right now it's fashionable to shit on Amazon, ever since the NYT hit-piece that got a TON of facts straight up wrong, and ever since then every negative Amazon story gets upvoted to the stratosphere. Every company makes mistakes, it's how you deal with them that counts.
Oh and in this case it wasn't a question of frugality, but scaling. Too many interns to interview, not enough time. It was still a bad decision, but it wasn't about being cheap.
Listen, your hiring process reflects your company culture. The fact that HR is in charge and you have to work to change it, rather then you, the people who will be teaching and supervising these interns, sitting down, telling HR "This is what we need, This is what we want, how can we make this scale" and having final approval says a lot.
Further, this expectation of a private quiet clean place to interview for several hours seems rather biased against poor people. Not to mention the test itself sounds like it doesn't account for the possibility of people with motor disabilities, the blind, the deaf and other disabled people.
Honestly, open book tests are a thing that has been extensively studied. Use IRT based adaptive multiple choice tests (like the computerized SAT or GRE where tests adapt to your ability), set a time limit, have a rolling set of questions to prevent knowledge transfer, and have an intern test day every quarter. You can calibrate test responses based on Amazon engineers at their desks.
This is the hard part, and I think it's why open book tests don't happen more. Coming up with good questions is hard. Don't forget that if they're not sufficiently work-like people on HN will still say that your hiring culture sucks.
I haven't worked at Amazon, and I won't speak to whether things are uncommonly bad there. Certainly Microsoft seems to have produced plenty of similar stories while escaping the broad stigma Amazon has acquired.
I will say, though, that I think your bad-reputation timeline is totally off. By the time that NYT piece broke, I had heard "Amazon is a horrible employer" from a half-dozen different places. I didn't even finish the thing, I just shrugged and went "well, matches what people have told me, no new info here".
The first time I was warned away from Amazon, it was over specific (and probably unenforceable) pieces of their internship contract. A friend rejected his offer because it contained a bizarrely broad non-compete that could theoretically have barred him from ever starting a company with anyone else who had ever worked at Amazon. This was 2012.
The second time was from a friend-of-a-friend who spent one year as an SDE. He talked about bureaucracy, burnout, and made the Microsoft comparison. He said it Amazon was reasonable as a ~2 year cash out, but nothing more. This was 2013.
After that, I was still open enough to respond when I heard from a recruiter. The internship interview required invasive rules like the ones at issue here, which proved fundamentally incompatible with my machine/internet setup at the time. I declined it, and scratched off Amazon unless I got news that these things had improved. This was 2014.
How you deal with mistakes is what counts, agreed, but Amazon's reputation struggle is far older and more pervasive than the perception spread by the NYT story. Fair or not (and I honestly don't know), it's a long-standing issue.
I interned at Amazon in 2014 and in 2015. I have not worked there since.
The non-compete and non-solicit for customers & business partners lasted for 9 months from when my internship ended and the non-solicit (titled "Non-Interference") for Amazon employees, contractors & consultants lasted 6 months from when my internship ended.
The interview process was two back-to-back phone screens. I heard from other interns the following summer that it had changed to an online coding test and one phone screen. I had never heard of anything so invasive as what is mentioned here before this article (and the precursor a week or two ago) hit HN.
I was working at Amazon when the NYT hit piece broke. It read like it was about the business/sales/marketing side of the company and it did not at all reflect what I experienced at Amazon.
I have met a few interns who had bad experiences at Amazon but most had good experiences and were invited back. I can only think of one who was invited back but had such a bad experience that they refuse to ever work for Amazon again.
When it comes to things like this, the variation between teams within large companies is much much greater than the variation between large companies.
I don't work for Amazon or have any affiliation with the company, but I can confirm that everyone I know who does work there (or used to) thinks highly of the company overall, and more or less enjoyed their time there. This throwaway account seems to be fairly balanced in perspective.
I'm personally of the opinion that Amazon is not a perfect company, but reports of its "evilness" have been greatly exaggerated. Moreover, I think this is a case of people with negative experiences having more of a reason to chime in than people with positive experiences. That somewhat biases threads, unfortunately.
I'm throwing this comment in here because I think it's easy for people to forget than n=1 anecdotes don't really confirm or deny anything (positive or negative) about a company's overall culture.
I've been told opposite stores from my n=2 sample size. One has said it's miserable and he couldn't wait to get out, the other that it's great and his group is doing really interesting work and I'd love working with him.
As someone who left Microsoft for Amazon, most of the things I hear about Amazon align much more closely with my experiences at Microsoft. We're talking about companies that employ thousands of people, you're going to hear different stories from different parts of the company.
It's entirely possible for a "publicity bubble" to cause all this. A big article got published ripping Amazon, got a lot of shares and views. For a while after that, anything negative about Amazon that seems authentic gets upvoted hard because it's what people want to read, and it lets them feel what they would call justified rage. People like that and people who just want forum karma and attention aggressively seek out or sometimes even make up stories to meet what these upvoters want to read. Stories that buck the narrative get downvoted and ignored. And just like that, you form a big popular impression of something that may be false or grossly exaggerated.
I don't know that the impression is actually false in this particular case, but these things can happen, and you'd be wise to not take the "internet consensus" too seriously.
The closest sibling to the grandparent of your comment, just -four-or-five- [Edit -- went back and counted:] seven comments higher in the thread, explicitly refutes your "publicity bubble" thesis.
> Oh and in this case it wasn't a question of frugality, but scaling. Too many interns to interview, not enough time.
I've interviewed for Google and Apple internships and the process has been extremely pleasant, with the interviewers happy to give their time. With Apple, I got to meet the entire team and spend time with them. I've heard similar stories about the Microsoft interview process (I mean, they even fly you out to Redmond).
What makes Amazon incapable of interviewing their interns adequately?
The only people who I've met in real life who say that Amazon is a good place to work are people trying to hire me (with one exception). Everyone else tells me that either that they hated it and quit, or that they hate it and will quit soon. I work for a competitor so there's absolutely some bias in the people I'm exposed to, but the complete lack of positive experiences related to me by humans in real life is surprising and compelling.
The one exception is a friend who started working at Amazon a few months ago. She loves it. But she's also been handed a pile of money and told to basically do what she wants. They hired her to create a new product line (not tech related) and the execs apparently don't know enough to get involved so she's basically got signoff to just do what she thinks is appropriate. So she loves it. Her husband also works for Amazon and hates it.
I'm sure there are other people at Amazon who like their jobs. I just haven't met any and I'm not sure they constitute a majority.
I had 2 friends working at Amazon and both of them were reasonably satisfied with their jobs. Overworked sure, but not depressed. I did ask them about the negative stories about Amazon's working conditions but both said that it didn't match with their experience. I know I shouldn't generalize from n=2 but I don't know how to reconcile their evidence with what I read online.
Amazon's social media policy discourages employees from posting about their experiences in reply to negative PR. I'm guessing that means that the employees who hate their job anyway disregard the policy, while the ones who care either don't take action or post on anonymous accounts :P
It's a big company. How you experience varies based on what stage your product is at (greenfield? possibly being cancelled next month? Launching? Stable?), whether your direct manager is a nice person, what career stage your manager is at, and what five people you spend most of your time with.
I'm sure it's not the hell that you read about online. If Amazon were as bad as some stories indicate, they wouldn't be able to retain anyone decent. Still, I have heard almost exclusively negative stories about Amazon, even in person.
> ever since the NYT hit-piece that got a TON of facts straight up wrong
The one about the Seattle culture or the warehouse side? I remember both, and I remember the warehouse one being a lot of garbage too. I was working as a contractor doing Amcare stuff and found a lot of issues with the article.
With all due respect to you too, didn't one of your employee's commit suicide? I think something must be wrong if so many people feel perpetually miserable there.
Attempted, he jumped and landed on a balcony 2 stories down or some such.
I don't hold that against amazonm. Any company of a size like amazon is going to have people that may be a bit unstable that get pushed over the edge.
I do however hold their treatment of their warehouse employees against them. When I look at a company I look at all their employees and see how they are treated. When a whole class of employees are treated as replaceable automation its probably the case that they will treat ALL their employees in that way eventually.
As an in-demand software engineer with oncall experience at a well-regarded company, I will not consider jobs that require me to be on call.
Jobs with oncall don't offer more compensation than jobs that don't.
Scheduling my life around being able to answer a page is inconvenient, and waking up in the middle of the night is something I'd rather avoid.
Operational work is often not considered as important as feature development for promotions, so you feel like you're wasting your time when doing it.
In my experience, system quality is completely independent of whether the developers do oncall or not. But I'd welcome objective data that proves otherwise.
There is no upside for me as an individual to take a job with oncall responsibilities.
Companies need to evaluate the loss of man-days when they demand or even suggest on-call duties for engineers. Also the effect of these duties on number of sick days taken, and overall health. Because when I'm up from 3-3:30am fixing something, I'm going to be less useful and emotionally testy that day at the office. When I'm a Good Marine and am up eyeballing a problem from 1am-6am, the same will happen but over a 1-2 week span. And I'll probably get sick.
I really think engineers end up on-call because companies give short shrift to documentation. Maybe if you afforded time to document systems and processes, you could outsource on-call duties, or trust sysadmins to remediate without developer intervention.
Operational work is often not considered as important as feature development for promotions, so you feel like you're wasting your time when doing it.
Which is why developers need to be on call, otherwise they don't build operational support into apps (monitoring hooks, useful logging, etc)
At my company devops is primary on-call, with a developer(s) as backup, if devops can't fix the problem quickly, they escalate to the developer. If they can't reach the developer or the developer can't fix it, then the entire release is rolled back and then the developers bear the brunt of the release schedule changes.
Without developers in that loop, it's hard for devops to get the tools they need to diagnose and fix problems.
Dev Ops is focused more on operations (tooling around releases, monitoring, etc) as well as doing initial triage for system problems to figure out where the problem is, and if they can't fix it themselves, determine which engineering group can help resolve an issue. Most of the Devops staff does development, but more around operational needs, product features are implemented by Dev.
Dev Ops means different things to different companies.
I think it's fair to say that you're just ops since you have the exact same functional role. And you can suffer from the same organizational issues that prompted the creation of the devops movement.
That said, there's a need for a way to divide the industry between traditional ops people and those with the ability to handle modern infrastructure and develop custom tooling where needed. And for better or worse the industry has decided that "devops" is going to be the way to determine that. Now the problem is all of the fakers (both individuals and teams) adopting the moniker. So now we're back to square one and we actually need to evaluate people and teams based on their knowledge and skill.
We're more integrated with Dev than a traditional "ops" role where Dev throws the code over the fence at Ops and says "Deploy it!" and Ops throws it back if there's a problem and says "Fix it!"
Our devops team sits in on design and code reviews to help ensure that operational needs are met early on in the process, and we'll change code to fix bugs or support operational needs. We're not full developers and won't rearchitect entire systems, but we will do bug fixes when we can.
So now we're back to square one and we actually need to evaluate people and teams based on their knowledge and skill
Sounds like your devops and dev teams do devops which is great. I wasn't trying to imply that they don't or that they had the same problems that prompted the movement, only that your roles are divided the same way as traditional dev and ops even if you do in fact do them smarter than average. What I'm getting at is that "devops" as a title is an (internal) marketing term.
You're right that every org defines devops differently but it's totally and unashamedly coopted from a movement which, despite being very generally defined, defines it radically differently. And that's not to say it's always a bad thing either. Sometimes it takes a title to help signal change and make an organization better which in the end accomplishes the goal of the movement.
And I agree, I don't think we ever left that square and we never will. But if prepending "dev" to "ops" wasn't at least somewhat effective in getting something out of people in decision making positions it wouldn't be tacked on to job titles and team names left and right. So at least some people think the marketing terms in a job title are a meaningful shortcut even if smart people like you know better.
Actually devops is supposed to be a culture where the dev team and ops team work together. What people seem to want it to mean is you're an ops team with automation skills. I've yet to see this be a dev team doing ops.
It goes a bit beyond "an ops team with development skills" in my company -- someone from Devops sits in on design and code reviews to suggest changes for operational needs (which almost always means adding metrics so we know how busy it is, and when it breaks or is near breaking, followed closely by adding enough logging so when it does break, we know why and don't need to page a developer to read a stack dump to try to figure it out).
But ultimately, the developer is the one that best understands how their code may break so he needs to instrument the code accordingly. And when the developer knows that giving the right information to Ops may avoid a 3am phone call, they have good incentive to do so.
Typically a DevOps team would mean a team that works on things that make it easier for the whole organization to do DevOps, e.g. tooling for easier building, deploying, testing, configuration and monitoring. It doesn't defeat the purpose of DevOps unless the people at the organization think that it's the only team that needs to be doing DevOps.
So much this. I don't trust an engineer who's never done maintenance work. And I really don't trust an engineer who's never been on call. I find that they write vastly different (usually much simpler) systems. Personally I will now only build a system that I know I can troubleshoot at 4am while drunk.
I 100 percent agree. I'm sorry you feel the need to post under a throwaway, but I'll put my name on this to back your point up.
On call is a deal breaker for my future job searches, and I am considering leaving the company I just joined because they sprung it on me while never mentioning it in the interview process.
On call is justified in my opinion only if someone will die if the problem isn't handled. Or if compensation is dramatically increased and agreed upon in advance.
On call is justified in my opinion only if someone will die if the problem isn't handled. Or if compensation is dramatically increased and agreed upon in advance.
What if the company will die if the problem isn't handled? Many companies (mine included) provide a service that needs to be highly available 24x7 (customers run automated tasks 24x7 against our service). If our site regularly went down for hours at night (or even for an entire weekend) because of a software problem that no one could fix because the developers weren't picking up the phone, we'd lose customers and eventually, the company would go out of business.
Even a 10 minute outage is a significant event and requires a full RCA for customers. We try hard to architect for high availability, but bugs do happen.
The answer here seems simple. Include on-call time in compensation, and fire developers.
Note that for some developers, you're never going to be able to compensate 'on-call' hours appropriately due to their evaluation of the opportunity cost of their time.
Salary shouldn't be a thing that a company can ever hide behind.
Labor laws really do need to cover the maximums that an employee can be expected to work. They should also make going above those maximums exponentially more expensive in 'bonus time'; and the accumulation rate shouldn't magically reset after time, but only after time back on normal/reduced work load and duration.
Also, while I'm on this subject, 'full time' work should really begin at more like 24 hours / week. Benefits for part time work should be pro-rated. (It should never be more cost effective to split a full time job in to part time jobs. That is defrauding the economy and making others pay for the costs of your labor.)
That's just fine, as long as on-call expectations are fixed as part of the job offer, just like salary is. If the on-call expectations change for the worse later, that is exactly the same as a salary cut.
IANAL but I believe (in the US or in NYS at least) this would be grounds for quitting with full unemployment coverage (you are not usually eligible for unemployment benefits if you quit, exceptions include the basic nature/hours/wage of the work being changed without your consent, since that is philosophically similar to having your old job terminated and being offered a new, worse one)
Get any on call expectations in writing when taking an offer if you can, just like you'd expect salary in writing
The real problem then is engineers being unwilling to vote with their two feet.
In a market as friendly to software developers as we're currently in, the correct response to that is totally "great, consider this my two week's notice, I'll go to this other company across the road that pays the same salary but doesn't demand I have on-time call".
(No, the response to every problem with your work environment should be to get a new job, nor is that even a realistic possibility for many. The economic argument still stands.)
It's a valid answer provided that on-call was part of the conditions known during hiring and salary negotiations. If we've negotiated a salary of $X for doing Y, but then you suddenly want Y+on-call, then you're going to either pay more or find another employee, because $X is simply too little for that.
Because, math? That company may have a team of 5 developers, how do you split those 5 developers across 3 8 hour shifts that cover 24 hours/day 7 days/week + holidays + vacations + sickdays to get 24x7x365 coverage without doing an on-call rotation.
When you're in a 100 or 1000 person company, then it's easier to have dedicated after-hours support staff (or staff working from multiple timezones around the world)
No one is saying that it should be "free", it's built into the salary - every company that I've worked at that's had an on-call is very clear about on-call rotations during the interview.
Except you're saying that small businesses should get to ignore that math, and get stuff for free.
" That company may have a team of 5 developers, how do you split those 5 developers across 3 8 hour shifts that cover 24 hours/day 7 days/week + holidays + vacations + sickdays to get 24x7x365 coverage without doing an on-call rotation."
You hire more people so that you can, or you don't make deals you can't afford. Honestly, why is this so hard to understand? Why do you feel so entitled to things you can't afford?
Again, who said anyone is getting anything for free?
My company has 100+ engineers, all of whom do a on-call rotation, they are all aware of it when the sign up, so that on-call duty is included in their salary. The employee can evaluate for themselves whether or not they think the salary is sufficient to cover that, but as far as I know, we've never had a candidate turn down an offer due to the on-call requirement (though I don't know that they'd always tell us that's why)
But really, does any engineer join a small company (in the USA) without assuming that they'll be on-call after hours? Even in early stage companies that haven't launched a product yet, there's still after-hours support to be done to keep dev systems running, fixing broken builds, etc.
And that overseas team often has little-to-no ownership so they call people while they're sleeping or half ass everything. This problem is magnified if they're contracted and not actually employees.
as someone who runs an on-call team, and is woken up frequently still as a co-founder, if someone sprang that on you, you should quit asap, and tell them why. people need to know that on-call is a decision, not a mandate.
the only reason my team puts up with it is because we're 100% remote. they never leave their families, so that's the trade-off.
That sucks, but hey, at least you get compensated for it. I think in some states they even have to pay you for a certain amount of time at minimum even if you only work for 10 minutes.
It happens very rarely. The times it does they do it under the guise of me being potentially fired if I answer wrong. Ie they want immediate clarification of what I did in some crisis but really just want info. Not paid and I'm actively searching for work.
I think its more: Comparing multiple companies, my total yearly income won't be any higher if I pick a company that requires oncall than if I pick one that doesn't, even if the one that requires it compensate for it. I can do the same job at company X for 200k without being on call, 175k at another that requires it, or vice versa...there's no correlation.
So if its not obviously worth my time, I just wont' do it.
As someone occasionally told my job is becoming irrelevant by people here on HN, I hope you folks have some regard for sysadmins who do on call regularly and (likely) get paid drastically less to do it. ;)
I've been told here that now developers can and should do everything, but it doesn't seem like a lot of developers want to be called in at 2am.
I think the term developer is a bit loaded. If you're properly developing an application, you should know what your dependies are, where you store your data, what are the failure condiditions are, how to prove your appication is actually functioning.
In my world, developers barely understand the programming language or framework they're using and are more interested in using buzzword foo rather than solving the problem at hand or dealing with actually running the systems that solve said problems.
As someone who does both dev and ops jobs, together and seperatly, I think your job will be safe for a very long time.
"In my world, developers barely understand the programming language or framework they're using and are more interested in using buzzword foo rather than solving the problem at hand or dealing with actually running the systems that solve said problems."
Oh, man. A thousand times this. Many devs I work with have been writing code that runs on AWS for years, and don't know what RDS means.
