Nice post but they quickly disregard Dask, don't explain why, don't test it, even exclude it from the benchmark they quote. I don't know if is the better answer, but it seems worth testing if you want a balance approachable + scalable. That's kind the thing Dask was meant to do.
We run Dask distributed for contents in what their chart covers (10TB+ with 4+ physical hosts) and it's a problem that can't really be discussed without well defined drive read or network latency limitations (if non-local storage). We've found defining recordsize, raid and other filesystem patterns to have an equal or larger impact in itself. Pandas is great because it allows us to validate on a sampling or smaller time window dataset, and then the same engineers can use a very similar implementation for Dask distributed through their scheduler tool.
While throwing hardware at problems isn't always a reasonable solution, it's trivialized to discuss this without IOPS or other read limitations.. and the nvme enterprise market is rapidly developing making this change quickly.
"AI will probably, most likely, sort of lead to the end of the world. But in the meantime, there will be great companies..."
- actual Sam Altman quote, the man is so unhinged he's beyond satire
That would be a phenomenal ROI. In almost all advertising contexts the hit rate is far, far lower. Resulting in everyone drowning in ads to hit the few targets.
To the commenters saying, "but how will people/companies know what to buy?"...
1) Through the same high-quality information channels you usually use today: customer reviews, word of mouth, point of sale, and lived experience. For B2B, you could still have industry events or demos where it is essentially a physical opt-in for advertising.
2) I could not possibly care. The offsetting benefit of living in a world where I am not surrounded by messages trying to deceive me into spending money is well worth losing whatever tiny signal might be in that noise.
HN ownership is complicit. They've been censoring everything "political" recently, as it becomes harder and harder to rationalize this regime's exactions. Used to be that such links would show up in the front page of /active daily, now there can be weeks without any passing by.
All those devs and data scientists and PhDs make a choice though. They could quit and work somewhere else. Even in a tough market their skills are in demand. They choose to work on this every morning.
There is a time when the operation of the machine becomes so odious, makes you so sick at heart, that you can’t take part; you can’t even passively take part, and you’ve got to put your bodies upon the gears and upon the wheels, upon the levers, upon all the apparatus, and you’ve got to make it stop. And you’ve got to indicate to the people who run it, to the people who own it, that unless you’re free, the machine will be prevented from working at all!
(from Mario Savio)
Partly the issue is, it was never the right model.
If you write/wrote code, you were a manager. Probably a better one than your "people manager" ever was, but that is/was arguably a different skill. What you managed was extremely technical, more akin to a line manager or an operations manager than a people manager.
Which, yes, everyone could not be the manager. Everyone who was not the manager could be quality control (QC), and handle the parts that "only a human" could handle.
The problem is, AI fits in nowhere in this equation, rather it throws it out. AI can manage, like any manager it can short circuit it's QC, in pursuit of an arbitrary metric. Even the idea of "human in the loop" is fundamentally flawed - if the human is not at some level the one directing (managing), there is no reason to stop incuding the human, less and less, in the work.
I believe it is the other way around: the LLM decides the type of action and the input to the action; the code validates the permission to act and the acceptability of the input. But, yes it is very different than SQL injection in that way.
reply