User-agent: Slurp Crawl-delay: 1 User-agent: TurnitinBot Disallow: / User-agent: LinkWalker Disallow: / User-agent: Flash+Processor Disallow: / User-agent: WebStripper Disallow: / User-agent: Webinator Disallow: / User-agent: Scooter Disallow: / User-agent: ImageBot Disallow: / User-agent: asterias Disallow: / User-agent: Intelliseek Disallow: / User-agent: NPBot Disallow: / User-agent: BDFetch Disallow: / User-agent: HTTrack Disallow: / User-agent: NPBot Disallow: / User-agent: Ocelli Disallow: / User-agent: MJ12bot Disallow: / User-agent: Nutch Disallow: / User-agent: egothor Disallow: / User-agent: CazoodleBot Disallow: / User-agent: twiceler Disallow: / User-agent: bot Disallow: / User-agent: bot/1.0 Disallow: / User-agent: IDBot Disallow: / User-agent: ShopWiki Disallow: / User-agent: Yeti Disallow: / User-agent: Yetibot Disallow: / User-agent: zibber-v0.1 Disallow: / User-agent: zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: Speedy Disallow: / User-agent: psbot Disallow: / User-agent: ImageWalker Disallow: / User-agent: ImageWalker/2.0 Disallow: / User-agent: Fatbot Disallow: / User-agent: Teemer Disallow: / User-agent: Charlotte Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot) Disallow: / User-agent: Baiduspider Disallow: / User-agent: dotbot Disallow: / User-agent: ZoomSpider Disallow: / user-agent: AhrefsBot disallow: / user-agent: sistrix disallow: / User-agent: SemrushBot Disallow: / User-agent: spbot Disallow: / User-agent: BLEXBot Disallow: / User-agent: Bytespider Disallow: / User-agent: GPTBot Disallow: / User-agent: CCbot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: Google-Extended Disallow: / User-agent: FacebookBot Disallow: / User-agent: OmgiliBot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: cohere-ai Disallow: /