User-agent: * Disallow: /cgi-bin/ Disallow: /icons/ Disallow: /badbot_catcher/ Disallow: /sys/ Disallow: /mailman/ Disallow: /img/ Disallow: /beta.shtml Disallow: /~irb/resume/ Disallow: /~irb/resume.shtml Disallow: /aud/ Disallow: /inc/ Disallow: /search/ # Rover is a bad dog User-agent: Roverbot Disallow: / # EmailSiphon is a hunter/gatherer which extracts email addresses for # spam-mailers to use User-agent: EmailSiphon Disallow: / User-agent: NPBot Disallow: / User-agent: psbot Disallow: / # just kinda bugs me User-agent: ia_archiver Disallow: / User-agent: LinkWalker Disallow: / User-agent: CyberPatrol Disallow: /