User-agent: Baiduspider Disallow: / User-agent: Twiceler Disallow: / User-agent: DBLBot Disallow: / User-agent: * Crawl-delay: 4 Disallow: /cgi-bin/ Disallow: /image/ Disallow: /temporary/ # This robot exclusion protocol file is semi-generic. # The site map domain name must be customised, as well as the filename if that is different. # The leading whitespaces may cause problems, and may be removed.