Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Oh, that interview question. “Tell me about something you broke in your last job"


Paged for "$HOSTNAME is down!"

Usual checks to access $HOSTNAME failed

Rushed to office at 6am before important process was about to run that needed that host.

Plugged in keyboard+monitor, dead screen, nothing.

Physically power-cycled server.

Stood in front of monitor+keyboard. It occurred to me it was taking longer than expected to show POST screen. About that time, I got a page saying $ACTUALHOSTNAME is down.

Walk around to the back of the racks. The monitor cable had come detached from the cable extender that I plugged into the server. I had never plugged the monitor in at all, just the extension.

The server wasn't down in the first place, it just lost a virtual interface, which I was paged for, and stupidly tested that virtual interface instead of the REAL name/IP.

And then I raced to the office just so that I could cause an outage.


"The Internet."


I once changed a piece of code that was referenced by every page on the customer facing site (10's millions of visits a day) to use a new function that someone had previously written (and was called on 1 page in the site). I mistakenly didn't really look to closely at the implementation of the function, and didn't realize how badly it's caching strategy was designed. When this code was deployed it instantly caused a thundering herd on our cache servers bringing the site down for about ~40 seconds.

I felt TERRIBLE about it.


My worst was that one time I accidentally took down some services that were essential for order processing via an unrelated, large stress test. My test accidentally consumed a large amount of bandwidth saturating the links to a shared service.

I did all this while sitting 2 feet from a print out of "The 8 fallacies of distributed systems". Bandwidth is indeed not infinite, can confirm.


Due to youth, totally misplaced confidence and poor access rights regime, ran an untested script in production causing it to fork uncontrollably requiring reboot.


It's my favorite. =D




Consider applying for YC's Summer 2026 batch! Applications are open till May 4

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: