Another way to think about it is comparing to how children learn. First, children spend inordinate amount of time just trying to make sense of words they hear. Once they develop their language models, adults can explain new concepts to them using the language. What'd be really exciting is being able to explain a new concept to GPT-n in words, and have it draw conclusions from it. Few-shots learning is a tiny step in that direction.
Children don't spend inordinate amounts of time learning words. In fact, past the first months, children often learn words from hearing them a single time.
I have a 4, 6, and 8 year old, and each of them are still learning words. Yeah they don’t spend 80% of each day learning words, but building up their vocabulary legit takes a looong time.
Oh, absolutely. I'm 31 and I'm still learning words!
But I don't think I've ever spent time to learn a particular word - it's almost always enough to hear it in context once, and maybe get a chance to actually use it yourself once or twice, and you'll probably remember it for life.
If it's a word for a more complex concept (e.g. some mathematical construct), you may well need more time to actually understand the meaning, and you may also pretty easily forget the meaning in time, but you'll likely not forget the word itself.
"But I don't think I've ever spent time to learn a particular word - it's almost always enough to hear it in context once, and maybe get a chance to actually use it yourself once or twice, and you'll probably remember it for life."
I'd strongly bet against this. If it were true, SAT and similar vocabulary tests would be trivial to anybody who has taken high school English, and I think it is not the case that most people perceive the SAT to be trivial.
That's of course correct. Perhaps GPT-3 can do that too? I don't have access to it, but I wonder if it can be taught new words using few-shot learning.
In fact, even GPT-2 gets close to that. Here's what I just got on Huggingface's Write With Transformer: Prompt: "Word dfjgasdjf means happiness. What is dfjgasdjf?" GPT-2: "dfjgasdjf is a very special word that you can use to express happiness, love or joy."
What takes time is all the learning a child needs to go through before they can be taught new words on the spot.
