Yes, but the same can be said about non-compete agreements. Imagine a company invested a lot in research and infrastructure, only to get their key employees poached and bring most of it to a new company in their heads.
Yet, California forbids the non-competes, because it promotes competition and at least somewhat eases capital lock-ins (i.e. two dudes in a garage can start competing with "big ones"). I don't really see much difference with patents here.
Remember early Facebook infra looked very similar to Google, they had to get creative to make new names for internal projects clones of Google ones.
If you are arguing that non-competes shouldn't exist therefore patents shouldn't exist — I think you're framing the issue too broadly.
If a professional joins an organisation, creates something novel and valuable, and then is poached to another organisation to do the same thing — that's exactly what patents are there to protect.
Say they spend 5 years developing a nice kind of lubricant — the company can patent that lubricant. Then if the professional goes and joins another company, even if they have that knowledge, they'd have to conduct new research to find a new approach which doesn't run afoul of the existing patent. That's the point — patent's don't protect your market dominance, they just protect against imitation/copying.
Now say the work of that professional is systems architecture for some Google project, and they leave and join Facebook to build the same systems architecture for Facebook's project — of course that shouldn't be patented (although sometimes it is, and that's a different issue.)
A "restraint of trade" agreement in your employment contract (a "non-compete") is intended protect companies from others copying their product when other IP rights don't exist — and I agree that it shouldn't exist. It quite literally prevents new invention and done through an imbalance of power. But, organisational already have plenty of mechanisms to mitigate that risk — pay the person more, better working conditions, or just pay them a long gardening period.
Yet, California forbids the non-competes, because it promotes competition and at least somewhat eases capital lock-ins (i.e. two dudes in a garage can start competing with "big ones"). I don't really see much difference with patents here.
Remember early Facebook infra looked very similar to Google, they had to get creative to make new names for internal projects clones of Google ones.