# Throttle MSNBot, who really seems to need it User-agent: MSNBot Crawl-delay: 50 Disallow: /source/ Disallow: /search/ Disallow: /issues/ Disallow: /servlets/ Disallow: /wiki/HelpOn* Disallow: /nonav/source/ Disallow: /nonav/search/ Disallow: /nonav/issues/ Disallow: /nonav/servlets/ Disallow: /nonav/wiki/HelpOn* # Sosospider actually ignores this file, # at least as of 2009-10-01 # But in case it ever gets a clue.... User-Agent: Sosospider Disallow: / # General restrictions # Remember to echo or supercede these in # individual crawler sections # (some ignore general instructions if they get specifics) User-agent: * Disallow: /source/ Disallow: /search/ Disallow: /issues/ Disallow: /servlets/ Disallow: /nonav/source/ Disallow: /nonav/search/ Disallow: /nonav/issues/ Disallow: /nonav/servlets/