I don't think that's true anymore. In NYC, at least, the people who are into Japanese culture tend to be black/Hispanic teenage girls, not the classic "basement-dwelling white guy" stereotype. Visit a big Japanese store like Kinokuniya or Bookoff sometime if you have one in your area - I think you'll be surprised!