I haven’t read the paper yet, but I’d imagine the issue is converting the natural language generated by the reasoner into a form where a formal verifier can be applied.
> what evolution has given us is a learning architecture and learning algorithms that generalize well from extremely few samples.
This sounds magical though. My bet is that either the samples aren’t as few as they appear because humans actually operate in a constrained world where they see the same patterns repeat very many times if you use the correct similarity measures. Or, the learning that the brain does during human lifetime is really just a fine-tuning on top of accumulated evolutionary learning encoded in the structure of the brain.
Not really, this is just the way that evolution works - survival of the fittest (in the prevailing environment). Given that the world is never same twice, then generalization is a must-have. The second time you see the tiger charging out, you better have learnt your lesson from the first time, even if everything other than "it's a tiger charging out" is different, else it wouldn't be very useful!
You're really saying the same thing, except rather than call it generalization you are calling it being the same "if you use the correct similarity measures".
The thing is that we want to create AI with human-like perception and generalization of the world, etc, etc, but we're building AI in a different way than our brain was shaped. Our brain was shaped by evolution, honed for survival, but we're trying to design artificial brains (or not even - just language models!!) just by designing them to operate in a certain way, and/or to have certain capabilities.
The transformer was never designed to have brain-like properties, since the goal was just to build a better seq-2-seq architecture, intended for language modelling, optimized to be efficient on today's hardware (the #1 consideration).
If we want to build something with capabilities more like the human brain, then we need to start by analyzing exactly what those capabilities are (such as quick and accurate real-time generalization), and considering evolutionary pressures (which Ilya seems to be doing) can certainly help in that analysis.
Edit: Note how different, and massively more complex, the spatio-temporal real world of messy analog never-same-twice dynamics is to the 1-D symbolic/discrete world of text that "AI" is currently working on. Language modelling is effectively a toy problem in comparison. If we build something with brain-like ability to generalize/etc over real world perceptual data, then naturally it'd be able to handle discrete text and language which is a very tiny subset of the real world, but the opposite of course does not apply.
> Note how different, and massively more complex, the spatio-temporal real world of messy analog never-same-twice dynamics is to the 1-D symbolic/discrete world of text that "AI" is currently working on.
I agree that the real world perceived by a human is vastly more complex than a sequence of text tokens. But it’s not obvious to me that it’s actually less full of repeating patterns or that learning to recognize and interpolate those patterns (like an LLM does) is insufficient for impressive generalization. I think it’s too hard to reason about this stuff when the representations in LLMs and the brain are so high-dimensional.
I'm not sure how they can be compared, but of course the real world is highly predictable and repetitious (if you're looking at the right generalizations and abstractions), with brains being the proof of that. Brains are very costly, but their predictive benefit is big enough to more than offset the cost.
The difference between brains and LLMs though is that brains have evolved with generality as a major driver - you could consider it as part of the "loss function" of brain optimization. Brains that don't generalize quickly won't survive.
The loss function of an LLM is just next-token error, with no regard as to HOW that was achieved. The loss is the only thing shaping what the LLM learns, and there is nothing in it that rewards generalization. If the model is underparamized (not that they really are), it seems to lead to superposed representations rather than forcing generalization.
No doubt the way LLMs are trained could be changed to improve generalization, maybe together with architectural changes (put an autoencoder in there to encourage compressed representations ?!), but trying to take a language model and tweak it into a brain seems the wrong approach, and there is a long list of architectural changes/enhancements that would be needed if that is the path.
With animal brains, it seems that generalization must have been selected for right from the simplest beginnings of a nervous system and sensory driven behavior, given that the real world demands that.
As someone that has only dabbled in C++ over the past 10 years or so, it feels like each new release has this messaging of “you have to think of it as a totally new language”. It makes C++ very unapproachable.
It isn’t each release but there are three distinct “generations” of C++ spanning several decades where the style of idiomatic code fundamentally changed to qualitatively improve expressiveness and safety. You have legacy, modern (starting with C++11), and then whatever C++20 is (postmodern?).
This is happening to many older languages because modern software has more intrinsic complexity and requires more rigor than when those languages were first designed. The languages need to evolve to effectively address those needs or they risk being replaced by languages that do.
I’ve been writing roughly the same type of software for decades. What would have been considered state-of-the-art in the 1990s would be a trivial toy implementation today. The languages have to keep pace with the increasing expectations for software to make it easier to deliver reliably.
As someone that has been using C++ extensively for the last 25 years, each release has felt as an incremental improvement. Yes, there are big chunks in each release that are harder to learn, but usually a team can introduce them at their own pace.
The fact that C++ is a very large and complex language and that makes it unapproachable is undeniable though, but I don't think the new releases make it significantly worse. If anything, I think that a some of the new stuff does ease the on-ramp a bit.
C++ can be written as the optimal industrial language it is. Simple core concepts year after year. Minimal adaptation.
The key thing to understand you are still using C with sugar on top. So you need to understand how the language concepts map to the hardware concepts. So it’s much more relevant to understand pointer arithmetic, the difference between stack and heap allocations and so on, rather what the most recent language standard changes.
You can write the same type of C++ for decades. It’s not going to stop compiling. As long as it compiles on your language standard (C++17 is fine I think unless you miss something specific) you are off to the races. And you can write C++17 for the next two decades if you want.
Able to make changes preserving correctness over time
Vibecoding reminds me sharply of the height of the Rails hype, products quickly rushed to market off the backs of a slurry of gems and autoimports inserted on generated code, the original authors dipping and teams of maintainers then screeching into a halt
Here the bots will pigheadedly heap one 9000 lines PR onto another, shredding the code base to bits but making it look like a lot of work in the process
Yes, preserving correctness seems like a good metric. My immediate reaction was to think that the parent comment was saying they’d like to see this comparison because AI will come out ahead. On this metric and based on current AI coding it’s hard to see that being the case or even possible to verify.
I was going to comment that it’s a bit weird that dark mode is viewed as a new or special option. But someone else already wrote a whole article on that point [1]
It's not worth twice as much. There was a buyer willing to pay twice as much. For something like this, the fact that value is completely subjective really sticks out. Trying to sell it again tomorrow might go for half or double.
I tend to agree, most platitudes are less impactful simply because your mind inserts what it already knows/autocompletes the phrase meaning because it's something so commonly heard. I read it as something like... you have enough resources to lead a life that has prosperity relative to the limited faculties of a human organism. That seems less catchy, and if you're too literal in your phrasing in writing, then you get the opposite problem where the reader limits their thinking to just what is written. Do you come to a different conclusion?
This phrase comes up alot in mindfulness and self-esteem related writings. I interpret it to mean that you don't have to do anything to justify your existence on the planet or prove your worth. I agree with this. But the phrasing of "you are enough" implies that it is possible to not be enough, but I don't think that's true since there's no bar that you have to meet, you just are.
reply