> Every other kind of software regularly gets vulnerabilities; are LLMs worse?
This makes it sound like all software sees vulnerabilities at some equivalent rate. But that's not the case. Tools and practices can be more formal and verifiable or less so, and this can effect the frequency of vulnerabilities as well as the scope of failure when vulnerabilities are exposed.
At this point, the central architecture of LLM's may be about the farthest from "formal and verifiable" as we've ever seen a practical software technology.
They have one channel of input for data and commands (because commands are data), a big black box of weights, and then one channel of output. It turns out you can produce amazing things with that, but both the lack of channel segregation on the edges, and the big black box in the middle, make it very hard for us to use any of the established methods for securing and verifying things.
It may be more like pharmaceutical research than traditional engineering, with us finding that effective use needs restricted access, constant monitoring for side effects, allowances for occasional catastrophic failures, etc -- still extremely useful, but not universally so.
That's like a now-defunct startup I worked for early in my career. Their custom scripting language worked by eval()ing code to get a string, searching for special delimiters inside the string, and eval()ing everything inside those delimiters, iterating the process forever until no more delimiters were showing up.
As you can imagine, this was somewhat insane, and decent security depended on escaping user input and anything that might ever be created from user input everywhere for all time.
In my youthful exuberance, I should have expected the CEO would not be very pleased when I demonstrated I could cause their website search box to print out the current time and date.
> At this point, the central architecture of LLM's may be about the farthest from "formal and verifiable" as we've ever seen a practical software technology.
This makes it sound like all software sees vulnerabilities at some equivalent rate. But that's not the case. Tools and practices can be more formal and verifiable or less so, and this can effect the frequency of vulnerabilities as well as the scope of failure when vulnerabilities are exposed.
At this point, the central architecture of LLM's may be about the farthest from "formal and verifiable" as we've ever seen a practical software technology.
They have one channel of input for data and commands (because commands are data), a big black box of weights, and then one channel of output. It turns out you can produce amazing things with that, but both the lack of channel segregation on the edges, and the big black box in the middle, make it very hard for us to use any of the established methods for securing and verifying things.
It may be more like pharmaceutical research than traditional engineering, with us finding that effective use needs restricted access, constant monitoring for side effects, allowances for occasional catastrophic failures, etc -- still extremely useful, but not universally so.