Today a tech lead with admin role on GH opened a PR, approved it for himself and merged it, because he could override GH rules. The PR had failing unit tests. It went straight to prod and caused 20 minutes downtime of one functionality. We do test things, sometimes you're just not prepared for all the permutations of the idiocy out there...
They almost certainly do, but there's always ways that the test jig differs from the units in the field, for example:
- The test jig is probably pristine, so no hundreds of hours of telemetry data clogging up the internal storage.
- The test jig might be on ethernet whereas a lot of users would be using wifi.
- The test jig probably targets specific A -> B upgrades rather than testing progressive upgrade across every version that's ever existed.
- The test jig can't cover every permutation of config options.
- The test jig probably only does a bare minimal smoke test after the install, so if the problem takes a bit to kick in, it might not show up.
Not to say that it's certainly any of these, but all are possible contributors. In the coming days it'll become clearer what particular pattern the affected devices follows, and/or clever people with JTAG dongles will reverse engineer the problem and spill the beans.
The test jig should be in expected conditions. We have simulated tests, and we have tests that run on the devices on my desk, but we also have a real world setup for consumer devices in a separate building that could be mistaken for the real deployment environment. That's not feasible for every company, but it's certainly feasible for Samsung. It doesn't mean you'll catch everything, but it does address some of your points.
There's no question about what it should be, but without technical leadership up the chain that understands and insists on this, it's easy to see how it could atrophy over time with cuts and staff turnover.
Like once upon a time, someone established a lab with twenty different units in different states, and put in place a process for validating the releases on it, but that person is long gone, and parts of the lab haven't worked quite right in years, but the parts that do still give a green checkmark, and who wants to stick their neck out and block a release over some baroque process no one even understands, right? It's not like the lab ever seems to really catch a major issue, does it? Just send a :ship: emoji to the slack channel and wait to be assigned your next ticket in the sprint meeting.
