I am a huge fan of mTLS but it is quite difficult to set up and I'm sort of taking a "wait and see" approach, especially with Google pushing ALTS, which sounds pretty sane. My dream is that you can ignore "VPCs" (i.e. trusted networking, firewalls, etc.) and let applications decide whether or not to allow traffic. "You have presented an incorrect TLS certificate, no response for you."
The problem I have is that I'm not sure what method you want to use to inject certificates. The current systems seem to be "something will figure it out for you at runtime, and your application will be none the wiser". (This is things like Istio and Linkerd.) I am not sure I'm convinced this adds much security. Anyone that can talk to the apiserver that injects certificates can probably convince the system to give you a certificate for your rogue container. And, I'm not sure that applications have enough visibility into the networking layer to really add any trust. I'd like for each incoming request to have some provenance information attached with dimensions like "this container was built from trusted source code and each commit that it contains was code reviewed", "this container was launched from a manifest written and applied by a trusted engineer from a trusted machine", and of course "this request was authorized by a human user whose data is being accessed". Right now, I feel like you have to hand-wave to get the first two. Sure, maybe Istio thinks the container is legit, perhaps because it asked a container notary (very early stages of support), and so now it can open a TLS connection to anything in the cluster. And, adding a request-scoped JWT is easy enough to provide the "here is the human user that made the request". I just don't think you can hide this from the application with a service mesh; every application needs to be aware of what's going on so it can say no to suspicious requests. And for that, I don't think we're really there yet. We had this at Google and it worked well, but I haven't found anything that works well in the outside world yet.
With that in mind, I totally understand why people trust the "well, it's all behind a firewall" model. It's not very good, but it is easy. I hope that in the next year or two we can get to where I want to be, but I'm also not sure that anyone cares. "Good is the enemy of great" and a private network seems to fit people's mental model pretty well. The only thing that automatic mTLS gets you right now is protection against "SSL added and removed here ;-)", and it's only the NSA that's really doing that. If you aren't a target of the NSA, your data is being exfiltrated by employees with access, or by simple application level bugs (XSRF, SQL injection).
This is all very rambly and I'm not sure where I'm going with this... but I wish that someone made application-level transport security work and it was trivial to set up.
> Anyone that can talk to the apiserver that injects certificates can probably convince the system to give you a certificate for your rogue container.
This is pretty infrastructure-heavy but I think my ideal solution would be to have a trusted orchestrator service on every machine which has its own certificate and accepts human-signed build artifacts to be run in a new container on that machine. It could verify the signature of each uploaded artifact and send CSRs for them to a hardened signing server, which returns the certs to be mounted (or sent over a standard initialization API) into the new containers.
Your options for compromising this are:
1. Get a malicious build artifact signed and submit it to a machine for execution. This shouldn't be possible without compromising an actual developer's credentials to sign the artifact.
2. Send your own custom CSR to the signing server to get a signed certificate. This shouldn't be possible without compromising the certificate from one of the orchestrator services to sign the CSR.
I think this should work as long as you can guarantee that services can't break out of their containers and as long as there's some hardware root of trust ensuring that your orchestrator service is genuine and the only thing that can read its CSR-signing certificate.
The problem I have is that I'm not sure what method you want to use to inject certificates. The current systems seem to be "something will figure it out for you at runtime, and your application will be none the wiser". (This is things like Istio and Linkerd.) I am not sure I'm convinced this adds much security. Anyone that can talk to the apiserver that injects certificates can probably convince the system to give you a certificate for your rogue container. And, I'm not sure that applications have enough visibility into the networking layer to really add any trust. I'd like for each incoming request to have some provenance information attached with dimensions like "this container was built from trusted source code and each commit that it contains was code reviewed", "this container was launched from a manifest written and applied by a trusted engineer from a trusted machine", and of course "this request was authorized by a human user whose data is being accessed". Right now, I feel like you have to hand-wave to get the first two. Sure, maybe Istio thinks the container is legit, perhaps because it asked a container notary (very early stages of support), and so now it can open a TLS connection to anything in the cluster. And, adding a request-scoped JWT is easy enough to provide the "here is the human user that made the request". I just don't think you can hide this from the application with a service mesh; every application needs to be aware of what's going on so it can say no to suspicious requests. And for that, I don't think we're really there yet. We had this at Google and it worked well, but I haven't found anything that works well in the outside world yet.
With that in mind, I totally understand why people trust the "well, it's all behind a firewall" model. It's not very good, but it is easy. I hope that in the next year or two we can get to where I want to be, but I'm also not sure that anyone cares. "Good is the enemy of great" and a private network seems to fit people's mental model pretty well. The only thing that automatic mTLS gets you right now is protection against "SSL added and removed here ;-)", and it's only the NSA that's really doing that. If you aren't a target of the NSA, your data is being exfiltrated by employees with access, or by simple application level bugs (XSRF, SQL injection).
This is all very rambly and I'm not sure where I'm going with this... but I wish that someone made application-level transport security work and it was trivial to set up.