A common metric for how much actual content has changed is the Jaccard Index. Even for large numbers of datasets that are too large to fit in memory it can be approximated with various forms of MinHash algorithms. Some write up here: https://blog.nelhage.com/post/fuzzy-dedup/
There are a few books which admirably explain the assumptions etc.
- A Programmers Introduction to Mathematics (amazing for building intuition, explaining how to read mathematics papers, understanding beauty of proofs etc.)
- Elements of Mathematics for Economics and Finance (a quick jog through typical the high school and college math a non-mathematician would study)
- Papula's Mathematik für Ingenieure und Naturwissenschaftler (rigorous (for practitioners) and comprehensive. Starts from nothing, with sets, then teaches you fractions, then quadratic equations until you're doing vector analysis and Laplace transformations)
- Foundations and Fundamental Concepts of Mathematics by Howard Eves (wide overview of the man developments in mathematics, up to its publication)
What an unimaginable horror! You can't change a single line of code in the product without breaking 1000s of existing tests. Generations of programmers have worked on that code under difficult deadlines and filled the code with all kinds of crap.
Very complex pieces of logic, memory management, context switching, etc. are all held together with thousands of flags. The whole code is ridden with mysterious macros that one cannot decipher without picking a notebook and expanding relevant pats of the macros by hand. It can take a day to two days to really understand what a macro does.
Sometimes one needs to understand the values and the effects of 20 different flag to predict how the code would behave in different situations. Sometimes 100s too! I am not exaggerating.
The only reason why this product is still surviving and still works is due to literally millions of tests!
Here is how the life of an Oracle Database developer is:
- Start working on a new bug.
- Spend two weeks trying to understand the 20 different flags that interact in mysterious ways to cause this bag.
- Add one more flag to handle the new special scenario. Add a few more lines of code that checks this flag and works around the problematic situation and avoids the bug.
- Submit the changes to a test farm consisting of about 100 to 200 servers that would compile the code, build a new Oracle DB, and run the millions of tests in a distributed fashion.
- Go home. Come the next day and work on something else. The tests can take 20 hours to 30 hours to complete.
- Go home. Come the next day and check your farm test results. On a good day, there would be about 100 failing tests. On a bad day, there would be about 1000 failing tests. Pick some of these tests randomly and try to understand what went wrong with your assumptions. Maybe there are some 10 more flags to consider to truly understand the nature of the bug.
- Add a few more flags in an attempt to fix the issue. Submit the changes again for testing. Wait another 20 to 30 hours.
- Rinse and repeat for another two weeks until you get the mysterious incantation of the combination of flags right.
- Finally one fine day you would succeed with 0 tests failing.
- Add a hundred more tests for your new change to ensure that the next developer who has the misfortune of touching this new piece of code never ends up breaking your fix.
- Submit the work for one final round of testing. Then submit it for review. The review itself may take another 2 weeks to 2 months. So now move on to the next bug to work on.
- After 2 weeks to 2 months, when everything is complete, the code would be finally merged into the main branch.
The above is a non-exaggerated description of the life of a programmer in Oracle fixing a bug. Now imagine what horror it is going to be to develop a new feature. It takes 6 months to a year (sometimes two years!) to develop a single small feature (say something like adding a new mode of authentication like support for AD authentication).
The fact that this product even works is nothing short of a miracle!
I don't work for Oracle anymore. Will never work for Oracle again!
Not a very deep CS-y one, but still one of my favourite data structures: Promise Maps.
It only works in languages where promises/futures/tasks are a first-class citizen. Eg JavaScript.
When caching the result of an expensive computation or a network call, don't actually cache the result, but cache the promise that awaits the result. Ie don't make a
Map<Key, Result>
but a
Map<Key, Promise<Result>>
This way, if a new, uncached key gets requested twice in rapid succession, ie faster than the computation takes, you avoid computing/fetching the same value twice. This trick works because:
- promises hold on to their result value indefinitely (until they're GC'ed)
- you can await (or .then()) an existing promise as many times as you want
- awaiting an already-resolved promise is a very low-overhead operation.
In other words, the promise acts as a mutex around the computation, and the resulting code is understandable even by people unfamiliar with mutexes, locks and so on.
Top comment [1] from another good HN discussion [2] about this from late last year. Worth repeating
Here's some of my learnings about getting promoted for those that really want to play that game:
- Only the perception of your work matters
- Attend the social events and get in good with the bosses
- The countability of your major achievements is important. Make the list long, too long to hold in the mind
- At the same time the gravitas of your best achievement is also important since that will be the soundbite that is shared about you behind your back
- Get allies who can proselytize about you behind your back
- Be the best. The difference between one and two is bigger than that between two and three, as far as promotions go
- Take credit for your work (use pronouns I and Me when talking about your work, not We) and do not allow others to take credit for your work
- If it's a teamwork situation with other people on your level, don't do most of the work, because the credit will end up being split 50/50 in the eyes of the bosses even if you did most of it
- Make a very good first impression
- Shape the narrative around the role you played in the success of the mission/team/company
- Get the bosses to make a soft public commitment regarding your competence
- Even if you have a really good boss, all of the above is still important, because they are fallible humans and aren't omniscient
- Actually do good work, it'll make the above easier
Pyroclastic flows have been directly measured at up to 90-130 m/s (200-290 mph), though even more typical speeds of 10-100 m/s (22 - 220 mph) exceed the pace of a walking human in good shape, let alone the elderly, young, or infirm. I'm not aware of specific indicated speeds of the 79 CE Vesuvius flows.
Escape on foot would have been at ~5 kph (3 mph). The safest destination from Pompeii, we can say in 2020 with hindsight, would have been perpendicular to the wind flow, to the north-east, at least 10 km (3+ hours continuous walking), up-hill.
Keep in mind that modern vulcanology, meterology, risk analysis, and disaster mitigation were also in their infancies, as well as wide-area information broadcasting, making such information largely unavailable at the time. In particular, the likely "obvious" safety of water (south-west) lead not only toward the centre of the kill zone, but a dead end as well. Walking strictly away from Vesuvius would reduce risk, but involved a longer path to safety than the NE route.
Few people at the time had access to SUVs, high-speed watercraft, helicopters, motorcycles, or bicycles, to aid in escape.
The wind was blowing onshore, a factor substantially credited to the death of Pliny the Elder, who, with the words "Fortune favours the brave" had attempted a sea rescue with his naval fleet, but found itself trapped at the beach. Pliny died there.
Temperatures within the cloud were a minimum of 100C (212F), and reached as much as 360C (500F) or more, all deadly. Wood and human remains at Herculaneum show carbonisation -- effectively having been turned into charcoal. Temperatures were extreme.
The surges themselves occurred well into the eruption, after substantial damage to the city by both earthquakes and bombardment, lasting overnight.
Total volcanic deposits on Herculaneum, which was spared windblown ash and hence resulted only from pyroclastic flows, were 23 meters (75 feet). Digging your way out of 7 storeys of searing-hot rock is a challenge for many people.
Any remaining survivors at the time of the surge would likely have been in terror, sleep deprived, utterly confused by the transformation to their surroundings, and probably not most able to have escaped in the first place. The eruption occurred during a festival in which the local population would have been increased by tourists and visitors. In Pompeii, though the initial pyroclastic surge saw lower temperatures close to the ground, later surgers (by surviving evidence) saw consistent temperatures throughout the vertical column, even at or below ground level. Surviving in a basement or cellar was not an option.
Total population of Pompeii and Herculaneum is estimated at ~16-20k, with about 1,500 bodies having been identified to date. Total victims clearly lie somewhere between these two numbers, and there's little agreement as to the precise number, though a fair argument can be made that mortality was a minimum of 10-20%.
At some point, if you were within the kill zone of Vesuvious, regardless of your mental state or fitness, you were walking dead. You might not have been dead yet, but there was simply no way out or to survivable shelter. There was no survivable shelter within the zone.
Again: by midnight August 28, if you were anywhere but heading north-east of Pompeii, or north-west of Herculaneum, you were already dead.
Visibility is very important to getting a promotion at a large company. Selling your work is important.
To move up, you must be playing the "choose a good project or team" game for at least 6 months before you try to get promoted. Preferably for a year or more to hit the right checkboxes for multiple cycles.
If you fail to do so, you can do absolutely amazing work but rigid processes and evaluation criteria will conspire to defeat you in a promotion committee setting.
At least, that's true in my company. From my ex-Google peers it seems to be true there as well.
Being in a smaller office means you get fewer of the best projects available to you. Reorgs sometimes steal them. Cancelling projects makes the last half a waste of time from a promotion standpoint.
As for what constitutes a good project. It will:
* Let you lead it
* Have peers at your level + one or two
* Work with multiple other teams
* Ideally work with multiple teams outside of your group, e.g. you're in, say, a chat app team and get to solve issues for some other app
* Good product manager and designers with a well thought out product; it's not fair, but if the project is successful business-wise people will often incorrectly attribute that to your own skills
* Have large engineering scope. Committees get confused and sometimes think simple designs to solve complex problems are bad. You want to solve problems that are complex even in terms of the solution (or can be made to sound complex) to check more boxes.
* Allows you to solve problems for other teams
I haven't optimized for promotion and thats personally hurt me in the quest to get to L6. I very, very close but didn't quite make the leap.
Don't ignore the flaws in this committee process. Exploit them.
It feels like that dev time cost is instead being crowdfunded by energy, disk, CPU and RAM that every user has to pay with and eventually that cost is paid by Earth itself. An app might be cheap or free but in the long time cheaper for the planet and each user would be an app costing a dollar or two more per install. I was using a 2011 laptop with only an HDD, 2 gigs of RAM and no discrete GPU until it actually went and finally broke last year so it irks me especially when I see this handwaving of "computers are cheap", especially from Westerner millenials or developers from SF. The fact that well off people who change machines every few years can even dare to call less well off people with older and shittier hardware "entitled" for wanting performant snappy software just like they got 5-10 years ago when they bought their machine is baffling. Not everyone needs a crazy and new machine, writers, reviewers, sales people, admins, etc. Case in point - G.R.R. Martin uses a DOS machine to write - https://www.youtube.com/watch?v=X5REM-3nWHg .
I.e. Slack and Atom got absolutely lambasted for performance, sluggishness and resource use (while VS Code was applauded, so it's clearly not an Electron specific thing) despite being made by companies valued in billions and based in the most expensive region of the world, one of them even being a paid product.
Or a game with pixel art (I do like it and I understand that particular indie dev optimizing for time with such a niche product so I don't want to name names here) graphic and gameplay only as deep as some better Flash ones from mid 2000s requires as its minimal system requirements several GBs of RAM (for comparison, Doom 3 recommended, not even minimal, was 512 MB in 2003) and disk space, etc.
Or when a graphically simple 2D game requires a 64 bit OS (despite using no 64 bit features seemingly), a non-integrated GPU (and not because of some lack of OpenGL features but due to poor optimization) and runs at 30 FPS on an integrated Intel that has 0 problems with Mincraft with really far draw distance. And it attempts to load hundreds of files (all of the game assets for an entire 4-10 hour long VN) at boot, taking 30 seconds on an HDD. And they could be loaded incrementally (loading what is needed right now only and everything else in the background, even dumbly and fully into RAM as it does now) or packed into SQLite or a ZIP to avoid so much FS access, but no - hundreds of files are being opened at game boot and there are tons of XML assets with 0 compression or minization. But instead the solution to performance woes (in gaming especially but through things like Electron it's seeping into main stream) is apparently to "git gud", "stop being a poor pleb" and getting a new GPU (apparently GTX 950 M is a potato level GPU now and only an idiot would play games on it in 2017) or an SSD so that the developer doesn't have to bother to do the tiniest of optimization.
That 2D game loading all assets, wanting a 64 bit CPU and non-integrated GPU, all for no good reasons, was Tokyo Dark by the way and due to the way the developer carry themselves I have 0 problem name dropping them, I made an entire video about that game, the disk and GPU part is at 15:15 : https://www.youtube.com/watch?v=sCXwgPJGLIE
It feels like what was done with Crash Bandicoot is some interstellar death star level technology in comparison to what some developers do, not even bothering to pack files to reduce FS chatter or load smartly or compress textual assets, they probably had it developed on an SSD, it loaded fast enough for them, it's done and prime for shipping, duh! Just gotta write a hype text about how extensively we tested it and how much effort we put in making it!
I realize I sound like an ass that's ranting and I am writing too lengthy (I did think about writing articles instead of lengthy HN comments like this one so if someone is interested feel free in hitting me up) but some of the stuff just blows my mind in ways I didn't know existed.
It's not even optimization for dev time like Python could feasibly be but sometimes outright waste or lack of basic care, i.e. Slack was apparently launching a full blown browser per organization until recently (or something like that), completely needlessly, now that part is out. At the same time they had this crazy involved (and cute, because it's 2017 and things must be cute) error page: https://slack.com/asdsad , or that semi-notorious reply article from a guy using unix CLI instead of hip BigData(tm) tools to analyze relatively small amount of data (yes, the guy is rubbing it a bit in too badly when he brings out mawk): https://aadrake.com/command-line-tools-can-be-235x-faster-th...
That lack of care is evident in other areas too, i.e. in security it manifests as these SQL injections, IoT botnets, outdated software pwns and plaintext/unsalted+sha1 password debacles. Afterwards it gets justified by "state attack, China or Russia probably" or handwaved like "we store passwords in plaintext to send them to user via email when he forgets them" (an actual explanation I read once..) or "we innovated so fast to deliver SUPERB customer experience that we didn't focus on security" (while 'security' in that case would amount to closing an admin port on an IoT appliance for example..). In general software we get also stuff like that TP-Link repeater (recently on HN) that needlessly queries NTP every 5 seconds, squandering hundreds of megs of transfer per month and basically DDoSing these NTP servers.
It's like this entire mentality that good stuff is too hard or too complicated or too expensive to do (like that Chess guy and his "clever multi-threaded application") while Pareto is very much in effect and even as little as not opening a hundred files at once at game boot or reading the dense man/info pages and thinking for 20 minutes about the problem at hand or back of the napkin math could make a big difference. 10 or 20 minutes or hours of dev time per year is not a big enough reason to squander resources so badly. There is an expression in Polish that seems really apt for developers who "optimize" their time to that degree: korona ci z głowy nie spadnie (the crown won't fall off your head, basically meaning something along the lines that exerting a little effort towards something isn't too much to be reasonably asked/expected of you).
I recall a similar event when someone wanted to stress test something on a webserver and had a few million long file with URLs in it, he did while read line curl $line in bash, it brought his local machine to its knees, probably due to this rapid process creation and destruction. I gave him an xargs with -P and -n to launch a single curl per each 100 URLs instead and it ran no problem and this time the webserver we were testing was on its knees on my much weaker laptop (weakest in the company actually, since I wasn't a programmer and didn't need a strong one), as intended. I'm actually guilty of overengineering myself, since my first try was a Python 3 + requests + grequests script, and only when weeks after I forgot where I put the script and didn't want to rewrite it I ran that xargs version (very Taco Bell eqsue solution actually - https://news.ycombinator.com/item?id=10829512 ).. And that's an anecdote but it feels like people (actual 'professionals' making a paid product and working in $billion+ corps) ship stuff as bad as the original 1 curl per URL script as if it's not a big deal and then it gets justified with some handwaving, "focus on features and not performance and security", "no one is gonna hack a toaster for anything", "computers are fast and cheap", "optimizing for dev time", etc.
It's a typical high volume low margin situation, like Steve Jobs once said during original Mac building that improving a load time by even a few seconds saves lives of people because so many people will use the Mac so often that it will add to a few lifetimes.
https://en.wikipedia.org/wiki/Jaccard_index