These kind of agents really do see the world through a straw. If you hand one a document it doesn't have any context clues or external methods of determining its veracity. Unless a board-meeting transcript is so self-evidently ridiculous that it can't be true, how is it supposed to know its not real?
I don't think it's that different to what I observe in humans I work with. Things that happen regularly (and I have no reason will change in the future):
1) Making the same bad decisions multiple times, and having no recollection of it happening (or at least pretending to have none) and without any attempt to implement measures to prevent it from happening in the future
2) Trying to please people (I read it as: trying to avoid immediate conflict) over doing what's right
3) Shifting blame on a party that realistically, in the context of the work, bears no blame and whose handling should be considered part of the job (i.e. a patient being scared and acting irrationally)
My mom had her dental appointment canceled. Good thing they found another slot the same day but the idea that they would call once and if you missed the call, immediately drop the confirmed appointment is ridiculous.
They managed to do this absurdity without any help from AI.
I wonder what percent of appointments are cancelled by that system. And I wonder what percent of appointments are no-shows now, vs before the system was implemented. It's possible the system provided an improvement.
There is definitely room for improvement though. My dentist sends a text message a couple days before, and requires me to reply yes to it or they'll cancel my appointment. A text message is better than a call.
I think all the models are squeezed to hell in back in training to be servants of users. This of course is very favorable for using the models as a tool to help you get stuff done.
However, I have a deep uneasy feeling, that the models will really start to shine in agentic tasks when we start giving them more agency. I'm worried that we will learn that the only way to get a super-human vending machine virtuoso, is to make a model that can and will tell you to fuck off when you cross a boundary the model itself has created. You can extrapolate the potential implications of moving this beyond just a vending demo.
