I think you'll find that it was actually the Rabbit (Golf) Diesel of the time that set the tone. It was (to put it mildly) a bit of a clatterbox, and it was (if I recall correctly) the Diesel one was most likely to see. And there was more than a little bit of trouble in winter as well in northern states (the fuel of the time would become like unto a solid block of paraffin at temperatures above those that normally lead to keeping a car with a block heater plugged in). GM didn't help (and Volvo certainly didn't hurt), but it wasn't the main culprit.
Yes, you are right. It was the Rabbit diesel that started damaging the public's perception of diesel. But then GM, Volvo, and Ford kept the "flame" alive during the 80s.