# Robot Exclusion File: robots.txt, created with RoboGen SE Trial # See www.rietta.com/robogen/ # See www.robotstxt.org/wc/exclusion.html#robotstxt # www.baidu.com/search/spider.htm # Last Updated: 1/20/09 # Format is: # User-agent: # Disallow: | # e.g. Disallow: /lounge.asp User-agent: * Disallow: /sp #Disallow: /news #Disallow: /img #User-agent: baiduspider #Disallow: / #User-agent: Googlebot-Image #Disallow: /