User-Agent: Screaming Frog SEO Spider Allow: / User-Agent: Googlebot Allow: *.js Allow: *.css User-agent: * Disallow: /admin/ Disallow: /documents/ Disallow: /users_doc/ User-Agent: Googlebot Disallow: /admin/ Disallow: /documents/ Disallow: /search?q= Disallow: */category/* Sitemap: https://www.is-t.net/sitemap.xml