The manager of the restaurant should probably learn to be more tolerant, but it's pretty clear on any plausible reading of the U.S. Constitution that he has not violated the First Amendment. It does, after all, start with the words "Congress shall make no law."

But there is an interesting twist. What if the manager tries to kick you out, but you refuse to leave? Then he calls the police and the police drag you out. At least in a certain sense, the government is now enforcing the manager's intolerance. When it comes to the First Amendment, it's pretty clearly established that even on these facts, the First Amendment still has not been violated. But there are other rights where the situation is more complicated. Courts will not enforce, for example, a racially restrictive covenant preventing African Americans from buying property in a certain neighborhood because to do so would violate the Fourteenth Amendment. (See Shelly v. Kreamer, http://en.wikipedia.org/wiki/Shelley_v._Kraemer) But in many respects, the situations are analogous: the government is asked to enforce individuals' otherwise private biases. It's not at all clear (at least to me, or anyone I know who studies this area of law) what accounts for this difference. Food for thought.

Freedom of assembly is also a form of freedom of expression, one that is also constitutionally protected. People have a right (within reason) to refuse to do business with other people as a form of expression. See the Mozilla/Brendan Eich situation as an example of this.

If a restaurant owner kicks you out of a restaurant, he is also expressing himself, just through association instead of speech. Additionally, he has property rights to consider. He is allowed to control the operation of his building and business. On the balance, the restaurant owner should be favored in this sort of dispute.

Why involve the cops? It's not strictly necessary. In many cases, there are private citizens that do the same job (event security, bouncers). But the cops, at the end of the day, are enforcing the the restaurant owner's rights to property and free association. The enforcement of biases only happens indirectly.

How is that unexpected or unwanted compared to the original argument? You have a right to say whatever you feel like, at whatever volume you feel necessary. The land owner has a right to kick you out of her business for whatever reason she deems appropriate; this includes private conversations.

I've been to places that won't seat individuals, only groups. I've been to places that won't seat you if you wear jeans.

I think most people would be pretty upset if they were ejected from a restaurant half-way through a meal for the content of an otherwise-intimate conversation, in contrast to the case of having a quiet meal disturbed by an unwanted intruder. There's an implicit contract in a restaurant that if you're seated and conduct yourself with decorum that your interaction with the restaurant operators will center on the exchange of food for money.

I've been to places that won't seat individuals, only groups. I've been to places that won't seat you if you wear jeans.

The key difference being that you haven't settled in to enjoy your meal in either of those examples, although you may be disappointed to be turned away.

It has nothing to do with being upset. It has to do with the business owner's right to kick you out for pretty-well any reason s/he sees fit. The person who is loudly interrupting your meal will probably be just as upset for being booted for expressing his opinion.

Not being seated is the same as not being allowed to enjoy your meal, regardless of where you fall on the 'I've already been seated' continuum.