I think it's more cost efficient for companies to give the CPUs some absurd advantage (crazy amount of HP, powerful attacks) and call it day.
Problem with AI is even if you spend a lot of time trying to get it right, it can make choices that are confusing to us humans and illicit ridicule from the player base.
Spot-on. In our game the NPC AI is ... exactly what I wanted it to be, smart and extremely "opportunistic". And the gameplay is generally such that your goal is to build structures/layout the buildings in a way that the AI will make the AI most likely to reach their goals.
To me, that's a huge part of our game -- catering to the NPCs, specifically figuring out "why are they missing X? Oh, because they're walking Z distance to the nearest bathroom!". I think the AI is fantastic but, when functioning perfectly as [we would have] expected, we still get complaints about why the 'dumb AI' will do X or Y. Part of that is defintiely a UI/UX issue on our part -- but another part is that players generally aren't used to it and they expect much more "robotic" or obvious/easy behaviors from agents/NPCs.
Emergent & opportunistic behaviors are awesome (IMO at least), but a lesson we learned is that they may not always provide players with the desired or expected gameplay experience. It'll of course always be game-specific, but it's something we will keep in mind & likely simplify further in our next game.
Well, the choice to make AI hordes braindead zombies who hust mindlessly storm you, is confusing to me as a player, too, if the enemie are supposed to be not zombies, but smart special forces.
Titanfall is a pretty good example of how to design good AI. Same goes for Age of Empires, I would say. XCOM, on the other hand, just increases the HP and count of enemies, which is a bad experience.
Isn't that also due to the reason that consoles, once sold, last for close to 10 years and the game developers have to design their games for sub par hardware. Since graphics sell games more, they spend most of their available computation on graphics and other areas rather than coming up with intelligent AI which takes a lot of computation as well. This is probably the reason why we don't have many big single player games.
That’s not the issue as 20 year old hardware can run decent AI for modest numbers of enemies. The reality is most players simply want stupid AI and more importantly exploitable AI.