Many humans do the same thing. They write tests that mock so much of the actual code, the "test" is not testing much at all, except perhaps the developer's ability to basically turn the code inside out. It's often just a large volume of crap that has to be maintained (or eventually deleted.)
I agree integration tests are best, with some e2e testing for common scenarios.
I worked at a place that required unit tests for every new method or function. Arguments like "this other (integration) test already covers that. why do I have to add another test?" wouldn't fly. PR reviews would often degrade into arguments about testing and how all database access needed to be mocked...
A couple months back, I had Sonnet build me a browser-based VT-100 terminal emulator. I then had it build me a websocket-to-telnet gateway for connecting some old retro systems to it. Both worked pretty well.
Could I have done it on my own? Yes, eventually. The problem is I would've lost interest and moved on to some other useless project before getting that far.
I know people at medium size companies where they are tracking AI costs very carefully. They are pulling back to levels under $100/week in AI spend per engineer, encouraging use of lower quality, lower cost models, etc.
$100/week will get you both a higher tier of Claude and significantly more tokens with GLM5.1 or Kimi, both of which are competitive in a decent harness (for Kimi that means their own CLI - it works well in that, but has a lot of quirks that requires special treatment). Just slightly more will get you all three.
I don't doubt you, but $100 is approximately the cost to company of one hour of dev time. If companies end up being willing to spend only 2% of their dev budget on AI, this bubble is not going to last long.
I would've ended the interview. "I don't want to waste any more of your time. It's clear to me I won't be a good fit here. Thank you for the consideration." <end call>
You can't guarantee one of your customers isn't going to do that dissemination though, right? No spammer is going to sign up saying "I'm a spammer that just got banned from <other service>! Can you guys help?"
It's probably the right approach to onboard a few independent security companies and task them with reviewing multiple OSS projects than it is to onboard each project individually.
I agree. Inexperienced people (not necessarily "dumb") are likely to accept everything at face value, not apply critical thinking skills, and not even check the AI generated output.
Avoid it for as long as you can. I worked at a startup that sold to enterprises. We had 6 employees. The CEO / sales was able to work around the SOC2 requirement every time.
My company had 6 employees, I was the CTO and I can't imagine getting SOC2 certified without using Vanta - that was back in their early access/beta days.
I had no choice - we had so many security assessments spreadsheets sent by potential customers, that getting SOC2 saved us time in the long run.
I like the people at Vanta just fine but it really squicks me out to see people doing Vanta because it's the simplest way for them to clear this dumb hurdle --- that implies that they don't understand SOC2 and are just taking Vanta's word for it.
The problem is, Vanta will ask (suggest? come perilously close to demand?) you do a lot of engineering work that is absolutely not necessary for a SOC2 attestation. Worse still: whatever controls you attest in your SOC2, you're practically locked into. If Vanta has you set up some cloud detection capability, and it turns out as you mature your security organization that it wasn't necessary or even useful, you have a fight on your hands with your Type II auditor about why you stopped doing it.
It's all negotiable. I did audits and attestations at a bank, .. everything's negotiable.
> that implies that they don't understand SOC2
Good engineering and SOC2 compliance can be on similar but not identical paths. If you want SOC2, you're bending your engineering towards that particular standard. Getting SOC2 compliant because it's time, and you have the customers, is just a step, and not a reflection of whatever good engineering you've done. If you can defend it, you can probably keep some of your variances.
If you're a solopreneur and you've never been in/near an audit, and you're committed to a vendor like Vanta, I'd recommend hiring a consultant for even a few hours to give you independent coverage of industry norms and a little coaching on sticking points.
I've been working with an organization that apparently won't give its developers reasonable access to dev cloud environments "because of SOC2." At least, that is the excuse they tell me.
Example: "I need access to EC2" isn't enough. I wind up with a role where I can launch instances, but not list them. I have to send several emails, have meetings, follow ups, sending links to AWS docs, etc. to get them to modify a custom IAM role. Then they still can't figure it out, so I am literally telling someone what to copy-and-paste into JSON to fix the issue. I completely understand more control in higher environments, but this crap adds up and costs weeks in lost productivity.
Yep! It took a month of back-and-forth to do what should have taken less than a day in an environment with less friction. I'm totally frustrated by the project at this point.
I think we're in quite a bit of agreement.. sometimes the SOC2 review exposes gaps and you need to find a way to close them -- where do you look for critical path on that?
Also, SOC2 audits are sometimes coupled with more strenuous ones, so in the umbrella of audit season, you may have to demonstrate things, or records of things.
In my experience it was as simple as connecting to AWS and tagging resources in Terraform. I got it all done in around 3 weeks. So maybe yes, if somebody doesn’t know about SOC2 then Vanta might be getting in the way but it in my case it literally solved all my problems in a month or so
I agree integration tests are best, with some e2e testing for common scenarios.
I worked at a place that required unit tests for every new method or function. Arguments like "this other (integration) test already covers that. why do I have to add another test?" wouldn't fly. PR reviews would often degrade into arguments about testing and how all database access needed to be mocked...
reply