Exactly. What I always tell vendors when I get a strange look after refusing their plastic bag is that we use it for 5 minutes but it stays in the environment for 1000 years.
They make plastics that biodegrade in a few years. Supposedly even in a landfill they degrade (I've seen this debated, I'm not qualified to figure out who is right). However they are more expensive and so rarely used except when someone wants to make a point about being green.
Nothing wrong unless it is used to store food or water and contains some BPA-like compound (AFAIK it includes most plastic produces nowadays even so called BPA-free).
^^^^THIS!^^^^
There is early evidence that most BPA and pthlalate replacements may be considerably worse for you than the things they are replacing. (At least, worse in the sense that they are possibly more dangerous as endocrine disruptors and hormone mimics, and also possibly harder to remove from the body...)
Best choice is to insist on glass packaging wherever you can. It's inert and infinitely recyclable.
It's the insane amount waste and environmental pollution that single-use plastic generates that is the problem.