Yes. The way it works is that two smart hackers go into the office each day and spend eight hours trying to think of as many creative ways of tapping on the target application as possible. Nothing is off-limits except what is agreed up front, but you're obviously expected not to interfere with production operations. The client is generally expected to set up a testing environment substantially similar to production, but usually consultants just have to muddle through with whatever the clients give, which may not (and usually isn't) populated with production data. As long as consultants are given the ability to enter in data themselves, i.e. admin accounts, this is fine, because data entry is their job. Hacking is basically large-scale data entry, and it's as boring as it sounds: very tedious, interspersed with excitement when you see an XSS popup window or figure out a clever way to get a reverse shell.
If after two weeks of this you found no medium-or-higher security vulnerabilities, you were generally considered to not be doing a very good job.
The secret of the industry is that at the end of this process, you are deemed secure. That's the point of the security audit. But if it's not a repeating process, it doesn't work. It may work for that particular version of the application, and it may substantially improve the security in that old vulnerabilities are found and fixed. Let me abandon this train of thought and put it another way:
This post is a press release saying that phpMyAdmin is secure. But that's not how this works. High-severity vulnerabilities are often found near the end of an audit. This is because the consultants have had time to become intimately familiar with the application. But the late stages of an audit are exactly when the consultant's time is mostly spent writing reports for the existing findings, and not doing pentesting. This means that two weeks is often just long enough to start finding serious vulns, since week one can be devoted to pentesting and week two is mostly reporting from Tuesday onward. But that "mostly reporting" process gets the consultant thinking about the application as they're doing the writeups, which -- you guessed it -- leads to realizing that there's something clever they could try. And when they try that clever thing, sometimes it yields a high-severity vuln. It's the opposite of a mechanical, thoughtless process.
That means your results will vary depending on who, specifically, is doing the auditing. If you run your application through the consulting process twice -- same version, same staging data, same everything -- it's likely that you'll get wildly different results, because the pentesters are different people.
It has to be an on-going process in order to be effective. And it can be highly effective. It just costs so much that only the most massive companies can afford this.
That's not to say this audit wasn't effective. It's possible that whoever did the audit found substantially everything. But it was interesting to discover how often this was not the case, in a "How'd they miss this last time?" sort of way.
They're a very worthwhile process for SMEs and companies that haven't had one before. At one of my previous employers (I won't say which) they were marvelling at the ways in which the contractor was able to do privilege escalation by editing a form to change their user level from the "3" or "4" in the drop-down to "1".
The fact that they got full rights so quickly really drove home the need for security to be a feature and for code reviews.
Now the cynical would point out that standard pen testers would have found that, and maybe they would, but the speed at which a contractor could find these issues and then see the full breadth of the surface compared to pen testers was great. And the fact they could explain back what the problem was in terms of code and how it should be rewritten rather than just "found rights escalation in form x" leaving the client to perhaps improperly deal with that.
Overall I was far more impressed watching an auditor doing a few days work than any of the regular pen testing companies I've seen since who mostly seem to point fuzzers at any endpoints they find.
> Now the cynical would point out that standard pen testers would have found that, and maybe they would, but the speed at which a contractor could find these issues and then see the full breadth of the surface compared to pen testers was great.
What's the difference between a contractor and a pen tester?
I consider one a function of how you are employed and the other a function of role. IE the two overlap and are not directly comparable.
If after two weeks of this you found no medium-or-higher security vulnerabilities, you were generally considered to not be doing a very good job.
The secret of the industry is that at the end of this process, you are deemed secure. That's the point of the security audit. But if it's not a repeating process, it doesn't work. It may work for that particular version of the application, and it may substantially improve the security in that old vulnerabilities are found and fixed. Let me abandon this train of thought and put it another way:
This post is a press release saying that phpMyAdmin is secure. But that's not how this works. High-severity vulnerabilities are often found near the end of an audit. This is because the consultants have had time to become intimately familiar with the application. But the late stages of an audit are exactly when the consultant's time is mostly spent writing reports for the existing findings, and not doing pentesting. This means that two weeks is often just long enough to start finding serious vulns, since week one can be devoted to pentesting and week two is mostly reporting from Tuesday onward. But that "mostly reporting" process gets the consultant thinking about the application as they're doing the writeups, which -- you guessed it -- leads to realizing that there's something clever they could try. And when they try that clever thing, sometimes it yields a high-severity vuln. It's the opposite of a mechanical, thoughtless process.
That means your results will vary depending on who, specifically, is doing the auditing. If you run your application through the consulting process twice -- same version, same staging data, same everything -- it's likely that you'll get wildly different results, because the pentesters are different people.
It has to be an on-going process in order to be effective. And it can be highly effective. It just costs so much that only the most massive companies can afford this.
That's not to say this audit wasn't effective. It's possible that whoever did the audit found substantially everything. But it was interesting to discover how often this was not the case, in a "How'd they miss this last time?" sort of way.