>if a site robots.txt file prohibits bots from gathering data,

Pages in a robots.txt file still get put into Google's index they just don't get parsed. Surprised me to find that out; http://www.seomoz.org/learn-seo/robotstxt (see "Why Meta Robots is better than robots.txt").