I'm the author of AFL. I think this is pretty cool, but also would like to ask you all to hold on to your hats =) Here's the short response I posted to the oss-security mailing list:
Hanno (the blog author) uses american fuzzer lop (AFL) and other fuzzers and has discovered (and reported with test cases) an extraordinary number of bugs: http://lcamtuf.coredump.cx/afl/#bugs
He is doing great work to make the Internet safer.
Google has a big fuzz-farm and Project Zero looking for this type of thing and even they did not find Heartbleed years ago. They are nabbing tons of bugs but there are many that are simply buried.
This seems to me a bit like when you do a maze starting from the finish and it is, for whatever reason, trivial to go from one end to the other.
It is neat that it is 2015 and fuzzers are cool again, though.
No. A fuzzer abstracts away some of the need for intricate, function by function analysis, but you really want to know what the source code is doing to be successful.
Fuzzing "blind" will work...but you will miss a lot without more instrumentation than that.
The only slightly suspicious thing that he did was that he immediately fuzzed only the first packet and no others. Although, being the first packet, it's a logical place to start. So I think the Heartbleed bug would indeed have been reasonably easy to find with this technology.
Actually, also the fact that he selected this particular packet exchange. This is a bit more suspicious, since there are thousands of other possible packet sequences. But again, the initial TLS handshake would be one of the first ones to check, I think.
He mentions in the article that he also found two other bugs but that they were previously discovered and reported by Brian Carpenter.
By fuzzing various file inputs I recently found two issues in OpenSSL, but both had been found by Brian Carpenter before, who at the same time was also fuzzing OpenSSL.
happy to see that someone else saw that! I mostly wrote that post / code as a tutorial on how to write checkers in a symbolic infrastructure, I think it was a little successful. I've been working on making checkers like that better, but that work is depressing because people like the author of AFL spend a lot of time telling me (indirectly) that it will never work, never scale, and never matter.
the parent comment has a link that includes code and a blog post. clang-analyzer uses "symbolic execution". the work I've been doing in extension also uses symbolic execution.
I fear these kinds of tools will be (are?) used as an argument to keep using C/C++. "Just adhere to all these standards and use all these code analysis tools" is a really bad way of handling bugs when I guarantee you it won't be used by those that need it most.
I think we need to start thinking of programming as a true craft. Like watch making. Not something you learn within 2 years just to become a productive app-maker to mass produce more useless apps. But something you grow into by gaining a full understanding about how the computer works. And this requires you to go trough both assembler and C. And by raising the notion of coding as a true craft that takes time and require precision, we will gain more secure applications. Not by hiding the inherent mechanics of the machine treating programmers like kids who need training wheels.
Except we're not making the same kind of thing over and over. Programming's not a craft. It's an abnormally non-routine form of engineering. By all means, teach the college students about doping, logic circuits and binary assembly. But skip C. Skip C++. It's not cool, it's not hardcore, it's just sad.
Whether they are used as a justification to use C/C++ for new projects (which isn't an argument I've seen), it's still better to have some solid tools available to find security issues in the existing C/C++ code that's already out there.
It's really dangerous to think that switching to another language will fix all your security problems. You can have safe memory, but that's about it. Serious exploits like "goto fail" can exist in any language.
No. The state of the art has advanced well past Java. And besides the kinds of bugs that a good type system can fix, there's also bugs borne from usability problems like switch fall-throughs that are trivial to fix with a better syntax.
Since I had something to do with the exploit mitigation chatter, I feel I should note that there's something of difference between fuzzing and exploit mitigation. I'll also happily concede that Address Sanitizer could be a more effective exploit mitigator than omalloc.
To make random() deterministic you can also just seed it with a predetermined number. Not sure if it is any better in this case, but it helps when writing unittests and similar.
The third part of my post here [1] is that progress in security will more likely come from cheap tools than expensive humans. Pardon the self-reference of course. :)
Progress in maintaining huge codebases in general. OpenSSL is ridiculously large for too few maintainers, but you should be so lucky as to have a codebase of maintainable size for the coders you have. My favourite example remains LibreOffice, where the coders are very good but there are too few of them - so they hit the automated tools, static checkers, clang plugins etc really hard.
I'm a big fan of fuzzers and believe the HeartBleed bug could have been found by one. Finding an already known bug would require a very careful researcher (to avoid providing either clues or bias). In any case, thanks for all the work you're doing Hanno!
So wasn't Heartbleed caused by a single overly large value coming directly from the attacker?
So, yeah, fuzzing could find that but it seems like wild overkill. The normal sort of manual range checking one does when implementing a protocol would of worked too...
I'm the author of AFL. I think this is pretty cool, but also would like to ask you all to hold on to your hats =) Here's the short response I posted to the oss-security mailing list:
http://www.openwall.com/lists/oss-security/2015/04/07/8