It's like if someone watches "2001: A Space Odyssey" and takes HAL as the model for AI, so they work really hard and create a computer capable of playing chess like in the movie. "Well, that's not really the essence of HAL, it's just that HAL happened to play chess in one scene." So then they work really hard some more, and extend the computer to be able to recognize human-drawn sketches. "Well, that's still not really the essence of HAL, it's just that HAL did that in one particular scene." So they work still harder and create Siri with HAL's voice, and improve its conversation skills until it can duplicate the conversations from the film (but it still breaks down in simple edge cases that aren't in the film). "Well, that's still not the essence of HAL..."
The Greeks observed these limitations thousands of years ago. Below is an excerpt from Plato's "Theaetetus":
Socrates: That is certainly a frank and indeed a generous answer, my dear lad. I asked you for one thing [a definition of "knowledge"] and you have given me many; I wanted something simple, and I have got a variety.
Theaetetus: And what does that mean, Socrates?
Socrates: Nothing, I dare say. But I'll tell you what I think. When you talk about cobbling, you mean just knowledge of the making of shoes?
Theaetetus: Yes, that's all I mean by it.
Socrates: And when you talk about carpentering, you mean simply the knowledge of the making of wooden furniture?
Theaetetus: Yes, that's all I mean, again.
Socrates: And in both cases you are putting into your definition what the knowledge is of?
Theaetetus: Yes.
Socrates: But that is not what you were asked, Theaetetus. You were not asked to say what one may have knowledge of, or how many branches of knowledge there are. It was not with any idea of counting these up that the question was asked; we wanted to know what knowledge itself is.--Or am I talking nonsense?
This is a great example of 1 of the 2 fundamental biases Kahneman identifies in Thinking Fast and Slow: answering a difficult question by replacing it with a simpler one.
The other one (also perhaps relevant to the general topic of this thread): WYSIATI (What You See Is All There Is).
The Greeks observed these limitations thousands of years ago. Below is an excerpt from Plato's "Theaetetus":
Socrates: That is certainly a frank and indeed a generous answer, my dear lad. I asked you for one thing [a definition of "knowledge"] and you have given me many; I wanted something simple, and I have got a variety.
Theaetetus: And what does that mean, Socrates?
Socrates: Nothing, I dare say. But I'll tell you what I think. When you talk about cobbling, you mean just knowledge of the making of shoes?
Theaetetus: Yes, that's all I mean by it.
Socrates: And when you talk about carpentering, you mean simply the knowledge of the making of wooden furniture?
Theaetetus: Yes, that's all I mean, again.
Socrates: And in both cases you are putting into your definition what the knowledge is of?
Theaetetus: Yes.
Socrates: But that is not what you were asked, Theaetetus. You were not asked to say what one may have knowledge of, or how many branches of knowledge there are. It was not with any idea of counting these up that the question was asked; we wanted to know what knowledge itself is.--Or am I talking nonsense?