User-agent: * #any other bot Crawl-delay: 10 #wait 10 seconds between accesses Disallow: /images/ Disallow: /includes/ Disallow: /common/