The same goes for contacts by the way: you can be as careful as you wish keeping your phone number and address and e-mail address private, as long as you are friends with me and I keep my Google Contacts up to date, Google has that data. And I don't think this is avoidable, unless you consider complete isolation from human contact desireable.