I would stand up to defend your right to read and enjoy a good book by a person that you aren't an expert in.
In general, I try to start by extending the benefit of doubt. It seems like nobody here stopped to wonder if pmarca had any idea that the dude wrote the Fascist Manifesto. And if he didn't, does he still deserve to be shamed for enjoying his other work without Googling him first?
What if you found out the author of the last thing you read was a fascist after the fact? Should we be mad at you?
Do you Google the author of everything you read? If so, that sounds exhausting.
If you call one of the fathers of fascist ideology your “patron saint,” in the context of also listing several other right wing thinkers, I think you know what you’re doing and should expect corresponding disdain.