I take crap at work for wearing sport coats and pants made of wool. I've given up on ties and suits unless there is a customer visit. A workplace has a culture and though you can exist at the edge of it, you'd better not go outside the lines.

Also, I compliment other men on their outfits when they look especially well put together. No HR complaints yet...