Sole proprietor/family business/lemonade stand: yes, refuse whoever you want, even if you blatantly say "we don't serve gays."

Corporations/LLC's: these entities have obtained special government privileges (being detached from any personal liability/ ability to seize an owner's assets), and so it seems reasonable that they can be considered to 'meet some public standards' in exchange for these privileges. Anti-discrimination falls into that category in my opinion.

The argument I've heard against the former is that everyone ends up paying taxes and otherwise indirectly contributing to your business, so it isn't right to deny them access.

At the same time, though the business owner also contributed their own taxes to the situation.

I think I agree with your distinction and stance, with the idea being that the market will correct because it's shitty business to turn away a customer over an irrelevant detail like that