No you can only use physical violence in defence of yourself or others against imminent physical harm. I think in the US you can also use it against someone in your home (presumably because by the time in your home you are otherwise defenceless.) Anything else is you just assaulting someone and puts you in the wrong. The law doesn't allow for 'they were winding me up' as a reason to assault someone. I can't believe anyone would think violence was warranted for skipping a queue or speaking to someone in a way you didn't like.
And “OK” is how you described violence in the absence of a genuine threat, which is not an objective statement of “what people will actually do” but a subjective endorsement.
