# --- Controlled Crawlers with Unofficial Delay --- User-agent: AhrefsBot Crawl-Delay: 5 User-agent: msnbot Crawl-Delay: 5 User-agent: rogerbot Crawl-Delay: 2 User-agent: Yandex Crawl-Delay: 5 User-agent: SemrushBot Crawl-Delay: 3 User-agent: SemrushBot-SA Crawl-Delay: 3 User-agent: CCBot Crawl-Delay: 10 # --- Blocked Crawlers --- User-agent: BLEXBot Disallow: / User-agent: MJ12bot Disallow: / # --- Global Rules for all other Crawlers --- User-agent: * Sitemap: https://www.skodefy.com/sitemap.xml Disallow: /leads Disallow: /lead-otp Disallow: /cdn-cgi/