# # traderonline.com: robots.txt # # 2006.04.03 - JVS # -- initial delay set to 20; to be fine tuned as we need # 2006.04.06 -- CHH # Set User-agent: slurp (yahoo) & User-agent: msnbot (msn) to Crawl-delay: 1 (Seconds) # 2007.09.26 arthur.matthews@dominionenterprises.com - added block to maintenance page User-agent: slurp Crawl-delay: 1 User-agent: msnbot Crawl-delay: 1 User-agent: * Disallow: /database-maintenance.html