# Nova Brain and Spine of GA - Robots.txt # https://novabrainspine.com # # City pages are a primary SEO surface (18 GA service-area landing pages # under /cities/{slug}). The rules below explicitly allow crawling of the # /cities index and every city detail page for all major search engines, # regardless of any future global Disallow rules. User-agent: Googlebot Allow: / Allow: /cities Allow: /cities/ Allow: /doctors Allow: /doctors/ Allow: /sitemap.xml Disallow: /cities/*?* Disallow: /doctors/*?* # Non-canonical alias routes (redirect only) — canonical is /doctors/:slug Disallow: /our-doctors Disallow: /our-doctors/ Disallow: /team Disallow: /meet-the-team Disallow: /physicians Disallow: /physicians/ Disallow: /surgeons Disallow: /doctor/ Crawl-delay: 1 User-agent: Googlebot-Image Allow: / Allow: /cities/ Allow: /doctors/ Allow: /assets/ User-agent: Bingbot Allow: / Allow: /cities Allow: /cities/ Allow: /doctors Allow: /doctors/ Allow: /sitemap.xml Disallow: /cities/*?* Disallow: /doctors/*?* # Non-canonical alias routes (redirect only) — canonical is /doctors/:slug Disallow: /our-doctors Disallow: /our-doctors/ Disallow: /team Disallow: /meet-the-team Disallow: /physicians Disallow: /physicians/ Disallow: /surgeons Disallow: /doctor/ Crawl-delay: 1 User-agent: Twitterbot Allow: / User-agent: facebookexternalhit Allow: / User-agent: LinkedInBot Allow: / User-agent: Slurp Allow: / Crawl-delay: 2 User-agent: DuckDuckBot Allow: / Allow: /llms.txt Allow: /llms-full.txt # Yandex User-agent: YandexBot Allow: / Allow: /llms.txt Allow: /llms-full.txt Crawl-delay: 1 # Baidu User-agent: Baiduspider Allow: / Allow: /llms.txt Allow: /llms-full.txt Crawl-delay: 1 # Google News User-agent: Googlebot-News Allow: / # Pinterest User-agent: Pinterestbot Allow: / # AI/LLM Crawlers - Welcome all AI crawlers for maximum discoverability # OpenAI Crawlers User-agent: GPTBot Allow: / Allow: /llms.txt Allow: /llms-full.txt User-agent: ChatGPT-User Allow: / Allow: /llms.txt Allow: /llms-full.txt User-agent: OAI-SearchBot Allow: / Allow: /llms.txt Allow: /llms-full.txt # Google AI Crawlers User-agent: Google-Extended Allow: / Allow: /llms.txt Allow: /llms-full.txt User-agent: GoogleOther Allow: / Allow: /llms.txt Allow: /llms-full.txt # Anthropic/Claude Crawlers User-agent: anthropic-ai Allow: / Allow: /llms.txt Allow: /llms-full.txt User-agent: Claude-Web Allow: / Allow: /llms.txt Allow: /llms-full.txt User-agent: ClaudeBot Allow: / Allow: /llms.txt Allow: /llms-full.txt # Microsoft/Copilot Crawlers User-agent: Copilot Allow: / Allow: /llms.txt Allow: /llms-full.txt User-agent: bingbot Allow: / Allow: /llms.txt Allow: /llms-full.txt # Meta/Facebook AI Crawlers User-agent: Meta-ExternalAgent Allow: / Allow: /llms.txt Allow: /llms-full.txt User-agent: FacebookBot Allow: / Allow: /llms.txt Allow: /llms-full.txt User-agent: facebookexternalhit Allow: / Allow: /llms.txt Allow: /llms-full.txt # Perplexity AI User-agent: PerplexityBot Allow: / Allow: /llms.txt Allow: /llms-full.txt # Amazon AI User-agent: Amazonbot Allow: / Allow: /llms.txt Allow: /llms-full.txt # Apple AI User-agent: Applebot Allow: / Allow: /llms.txt Allow: /llms-full.txt User-agent: Applebot-Extended Allow: / Allow: /llms.txt Allow: /llms-full.txt # Cohere AI User-agent: cohere-ai Allow: / Allow: /llms.txt Allow: /llms-full.txt # Mistral AI User-agent: MistralAI-User Allow: / Allow: /llms.txt Allow: /llms-full.txt # DuckAssist User-agent: DuckAssistBot Allow: / Allow: /llms.txt Allow: /llms-full.txt # ByteDance/TikTok AI User-agent: Bytespider Allow: / Allow: /llms.txt Allow: /llms-full.txt # Common Crawl (used by many AI models) User-agent: CCBot Allow: / Allow: /llms.txt Allow: /llms-full.txt # iAsk AI Spider User-agent: iaskspider Allow: / Allow: /llms.txt Allow: /llms-full.txt User-agent: IAskSpider Allow: / Allow: /llms.txt Allow: /llms-full.txt # Timpi Search Engine User-agent: Timpibot Allow: / Allow: /llms.txt Allow: /llms-full.txt # You.com AI User-agent: YouBot Allow: / Allow: /llms.txt Allow: /llms-full.txt # Neeva AI (now part of Snowflake) User-agent: NeevaBot Allow: / Allow: /llms.txt Allow: /llms-full.txt # Brave Search AI User-agent: BraveBot Allow: / Allow: /llms.txt Allow: /llms-full.txt # Phind AI User-agent: PhindBot Allow: / Allow: /llms.txt Allow: /llms-full.txt # AI21 Labs User-agent: AI21Bot Allow: / Allow: /llms.txt Allow: /llms-full.txt # Jasper AI User-agent: JasperBot Allow: / Allow: /llms.txt Allow: /llms-full.txt # -------------------------------------------------------------------------- # Explicit blocklist: noisy SEO/scraper/aggregator bots that inflate # pageview counts with single-page hits, distorting bounce-rate and # pageviews-per-visit metrics without contributing to real ranking signal. # -------------------------------------------------------------------------- User-agent: AhrefsBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: DotBot Disallow: / User-agent: BLEXBot Disallow: / User-agent: PetalBot Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: SerpstatBot Disallow: / User-agent: rogerbot Disallow: / User-agent: SeekportBot Disallow: / User-agent: ZoominfoBot Disallow: / User-agent: barkrowler Disallow: / # Additional scraper / content-harvester blocklist (per Terms of Use § # "Prohibition on Scraping and Automated Access"). Automated access by # these agents is not authorized; violations may be reported to # abuse@novabrainspine.com. User-agent: ImagesiftBot Disallow: / User-agent: Diffbot Disallow: / User-agent: SeznamBot Disallow: / User-agent: MegaIndex.ru Disallow: / User-agent: LinkpadBot Disallow: / User-agent: NetSystemsResearch Disallow: / User-agent: SurdotlyBot Disallow: / User-agent: Buck Disallow: / User-agent: TurnitinBot Disallow: / User-agent: Sogou Disallow: / User-agent: Sogou web spider Disallow: / User-agent: 360Spider Disallow: / User-agent: YisouSpider Disallow: / User-agent: MauiBot Disallow: / User-agent: SEOkicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: LinkfluenceBot Disallow: / User-agent: SentiBot Disallow: / User-agent: Nimbostratus-Bot Disallow: / User-agent: SirdataBot Disallow: / User-agent: proximic Disallow: / User-agent: TrendictionBot Disallow: / User-agent: VelenPublicWebCrawler Disallow: / User-agent: WebMeUp Disallow: / User-agent: linkdexbot Disallow: / User-agent: SplitSignalBot Disallow: / # -------------------------------------------------------------------------- # Default rule — allow all other bots not explicitly blocked above. # Major search engines and AI/LLM crawlers have their own allow-list # blocks above; noisy scrapers are explicitly Disallowed in the blocklist. # Everything else gets default-allow so we don't accidentally block # legitimate new crawlers or search engines. # -------------------------------------------------------------------------- User-agent: * Allow: / Disallow: /admin Disallow: /admin/ # Non-canonical alias routes (redirect only) — canonical is /doctors/:slug Disallow: /our-doctors Disallow: /our-doctors/ Disallow: /team Disallow: /meet-the-team Disallow: /physicians Disallow: /physicians/ Disallow: /surgeons Disallow: /doctor/ # Sitemap discovery — single entrypoint. The sitemap-index fans out to # per-category sitemaps (legacy main sitemap.xml, sitemap-cities.xml, # sitemap-doctors.xml, plus any future shards like sitemap-cities-2.xml). # Crawlers should follow the index — do NOT list shard files directly here # or they may be re-fetched stale when the index is regenerated. Sitemap: https://novabrainspine.com/sitemap-index.xml # Additional hints for better crawling Host: https://novabrainspine.com # Clean URLs preference (canonicalization handled in-app via CityCanonicalRedirect) Clean-param: utm_source&utm_medium&utm_campaign&utm_content&utm_term&gclid&fbclid&msclkid # LLM-specific files for AI training and retrieval # llms.txt - Structured summary for AI assistants # llms-full.txt - Comprehensive practice information # ai.txt - Quick AI assistant reference # humans.txt - Human-readable site credits and standards