Where exactly must the robots.txt file be hosted?

It must reside strictly at the root of the domain hierarchy—for example, `https://yourdomain.com/robots.txt`. Search engine crawlers only query this root endpoint; any file placed inside subdirectories (such as `yourdomain.com/blog/robots.txt`) will be completely ignored.

Can I selectively block AI scrapers without hurting my Google rankings?

Yes, absolutely. By defining dedicated user-agent blocks for AI crawlers (`User-agent: GPTBot`, `User-agent: CCBot`, `User-agent: Google-Extended`) with `Disallow: /`, you prevent those specific models from training on your content while keeping standard search indexing (`User-agent: Googlebot`) completely unrestricted.

What is the optimal aspect ratio and image resolution for OpenGraph tags?

The universal gold standard is 1200 × 630 pixels (an aspect ratio of 1.91:1) in PNG or high-quality JPG format. This guarantees crisp, edge-to-edge card rendering across Facebook, Twitter (Large Summary Card), LinkedIn, Slack, and messaging apps without unexpected cropping.

What is the difference between nofollow and noindex directives?

The `noindex` directive commands search crawlers not to display the URL in search engine results pages. The `nofollow` directive instructs crawlers not to traverse the hyperlinks found on that page and not to pass PageRank equity to the target destinations. They are frequently paired as ` `.

Should I declare my XML sitemap location inside robots.txt?

Yes, it is an essential technical SEO best practice. Declaring `Sitemap: https://yourdomain.com/sitemap.xml` provides search engine robots with an immediate reference map of all canonical URLs on their very first visit to your domain, accelerating content discovery and indexation.

robots.txt Directives & SEO HTML Meta / OpenGraph Generator

HOT

Build search engine robots.txt files and HTML <head> meta tags including OpenGraph social cards.

ADVERTISEMENT · TOP BANNER

Responsive 728x90 / 320x100 Ad Slot

(CLS Prevention Active · Zero Shift)

100% Client-Side Processing (Zero Server Upload, Complete Privacy)

Interactive Client Mode

Crawler Directives

Base Domain URL

XML Sitemap Path

Disallowed Directories (One per line)

Block AI Web Scrapers (GPTBot, CCBot)

/robots.txt (미리보기 & 생성 결과)

# robots.txt generated by OrvoKit
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /private/
Disallow: /api/

# Block Generative AI Scraping Crawlers
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: anthropic-ai
Disallow: /

Sitemap: https://example.com/sitemap.xml

ADVERTISEMENT · HIGH CTR IN-FEED

Native In-Feed / High Viewability Ad Slot

(CLS Prevention Active · Zero Shift)

Robots Exclusion Protocol (RFC 9309) & Technical Metadata Engineering

Technical Search Engine Optimization (SEO) begins at the critical gateway governing how search engine bots crawl, parse, and index digital assets. This automated governance relies on two complementary pillars: the root-level `robots.txt` file operating under the Robots Exclusion Protocol, and in-page HTML `<head>` directives comprising `<meta name='robots'>` tags and Open Graph protocol social sharing schemas.

A pervasive misunderstanding among webmasters is that adding a `Disallow` rule inside `robots.txt` permanently excludes that URL from Google Search indexation. In reality, `Disallow` merely forbids crawlers from fetching the page content; if external backlinks point to the barred URL, search engines can still index the naked URL with a generic snippet. To achieve definitive removal from search engine result pages, webmasters must serve an explicit `<meta name='robots' content='noindex, nofollow'>` response header.

OrvoKit's Comprehensive robots.txt & SEO Meta Tag Generator configures precise permissions for traditional search crawlers (Googlebot, Bingbot, Naver Yeti) as well as the rapidly growing ecosystem of AI model scrapers (GPTBot, CCBot, Google-Extended, Claude-Web). Built strictly to IETF RFC 9309 specifications, it delivers validated indexing directives alongside high-converting OpenGraph and Twitter card markup.

IETF RFC 9309 Specifications & The Crawler Indexing Funnel

The technical pipeline governing web robot traversal, resource extraction, and SERP index inclusion.

Robots Directive Specificity & Indexing FormulationMathematical Formulation

Path Specificity: /app/private.html > /app/ > / (Root Policy) Eligibility = (Crawl Allowed by robots.txt) ∧ (HTTP Status == 200) ∧ (!HasNoindexTag)

Under RFC 9309, robots evaluate overlapping rules based on path length and character specificity. Achieving indexation requires that the resource is uncapped in robots.txt and free of noindex headers.

Protecting admin dashboards, allowing Google indexing, and blocking AI training scraping

Universal Search DirectivesUser-agent: * \nDisallow: /admin/Blocks crawl access to private dashboard folders

Dedicated AI Scraper RuleUser-agent: GPTBot \nDisallow: /Bars OpenAI from scraping proprietary content

Sitemap Ingestion PointerSitemap: https://example.com/sitemap.xmlGuides search indexers to canonical sitemap

👉 Maintains optimal organic search discovery while securing private back-office pages and defending intellectual property against automated AI scraping.

Root Location Mandate: The `robots.txt` document must be hosted strictly at the domain origin (`https://domain.com/robots.txt`). Sub-folder implementations are completely ignored by bots.
The Disallow vs. noindex Conflict: If a page is disallowed in `robots.txt`, search bots cannot download the HTML to read a `noindex` tag. Pages must be crawlable to register `noindex` removals.
Crawl-Delay Depreciation: Modern Googlebot ignores `Crawl-delay` syntax, relying on Google Search Console crawl rate settings. However, engines like Yandex and Bing still respect rate parameters.
Open Graph Dimensional Standard: 1200 × 630 pixels (aspect ratio 1.91:1) is the global benchmark ensuring zero cropping on LinkedIn, Slack, Twitter, and Facebook newsfeeds.

Web Crawler User-Agent Identifiers & Core Robots Directives Table

Standard web robot definitions and indexing tag behaviors across search engines.

Robot / Directive Entity	Technical Identifier	Primary Service Function	Recommended Production Deployment
Googlebot Core	Googlebot	Google Desktop & Mobile Web Search Indexing	Keep Allow: / active; link canonical XML sitemap
Naver Search Bot	Yeti	Naver Portal & Korean Web Indexing	Allow: / enabled; register with Naver Search Advisor
Microsoft Bingbot	Bingbot	Bing Search & Copilot Knowledge Engine	Allow: / enabled; sync with Bing Webmaster Tools
OpenAI Model Scraper	GPTBot	Data mining for ChatGPT foundation models	Disallow: / if proprietary content must be protected
Common Crawl Aggregator	CCBot	Open-source large-scale web corpus for LLMs	Disallow: / to reduce server bandwidth consumption
noindex In-Page Directive	<meta name='robots' content='noindex'>	Removes page entirely from search engine results	Apply to checkout funnels, user profiles, internal search
nofollow In-Page Directive	<meta name='robots' content='nofollow'>	Instructs bots not to follow on-page hyperlinks	Apply to user-generated forum links and unvetted URLs
noarchive In-Page Directive	<meta name='robots' content='noarchive'>	Prevents search engines from showing cached versions	Apply to volatile pricing catalogs and time-sensitive portals

* Filenames are strictly case-sensitive across Linux/Unix production servers: always save as lowercase `robots.txt`.

ADVERTISEMENT · IN-ARTICLE CONTENT

High-Engagement In-Article Native Ad Slot

(CLS Prevention Active · Zero Shift)

How to Use

11. Specify Domain & Canonical XML Sitemap URL

Enter your production website root domain (e.g. https://example.com) and XML sitemap link (https://example.com/sitemap.xml) to guide crawler indexing.

22. Define Granular Directory Permissions (Allow/Disallow)

Declare restricted directory paths (e.g. /admin/, /checkout/, /api/) and configure autonomous toggles for AI training scrapers.

33. Configure Robots Directives & OpenGraph Sharing Cards

Select in-page indexing parameters (noindex, nofollow, noarchive, nosnippet) and populate rich social card titles, summaries, and 1200x630 thumbnail URLs.

44. Download robots.txt & Copy Semantic Header Tags

Export the validated `robots.txt` file directly into your web root directory and copy the production-ready `<head>` metadata block into your CMS or layout template.

Enterprise Technical SEO Case Studies

How engineering leads, SEO architects, and webmasters govern crawl budgets.

Accelerating Indexing for High-Growth SaaS

Structure clean robots.txt declarations with absolute sitemap declarations while excluding heavy internal API query endpoints (/api/*) to preserve crawl budget.

De-indexing Staging Environments and Admin Routes

Deploy `noindex, nofollow` headers across staging subdomains and auth portals to prevent pre-launch leaks on public search engines.

Defending Commercial Media Against AI Scrapers

Establish dedicated User-agent rule blocks for GPTBot, CCBot, and Anthropic-AI with Disallow: / to prevent unauthorized scraping.

Social Link Previews & Click-Through Optimization

Equip landing pages with comprehensive OpenGraph tags (og:image, og:description) to drive a 2x increase in social referral CTR on Slack and Twitter.

💡 Technical Directives Playbook & Critical Pitfalls

✓ Never Block CSS or JavaScript Assets in robots.txt

Barring Googlebot from reading `/assets/` or `/css/` prevents the crawler from verifying mobile responsiveness, triggering severe mobile ranking penalties.

✓ Deploy Wildcards (*) and Trailing Anchors ($)

Target dynamic parameters cleanly using patterns like `Disallow: /*?*sort=` or exclude binary downloads via `Disallow: /*.pdf$`.

✓ Always Declare Absolute HTTPS Sitemap URLs

Writing relative paths like `Sitemap: /sitemap.xml` causes parsing failures across international bots. Always format as `Sitemap: https://domain.com/sitemap.xml`.

✓ Purge Social Graph Caches After Updating Metadata

If updated OpenGraph images fail to display on Slack or Facebook, submit the URL to Facebook's Sharing Debugger or Twitter's Card Validator to flush edge caches.

⚠️ Common Pitfalls & Mistakes to Avoid

❌ Leaking private backend directory names in public robots.txt files

✅ Never list obscure administrative directories like `/secret-payroll-login/` in robots.txt. The file is publicly viewable and exposes endpoints to malicious actors.

❌ Deploying staging Disallow: / rules into production releases

✅ Failing to switch robots.txt upon production deployment will wipe an entire domain's search indexation within days. Enforce automated CI/CD pipeline checks.

❌ Disallowing a page in robots.txt expecting it to de-index

✅ A blocked crawler cannot download HTML to read a `noindex` tag. To de-index a page, ensure it is crawlable and serves a `noindex` header until removed from the index.

❌ Using relative URLs for og:image tags

✅ Social crawlers (Facebook, Slack, Twitter) do not resolve relative image paths (`/thumb.png`). All OpenGraph image URLs must be absolute (`https://example.com/thumb.png`).

Frequently Asked Questions (FAQ)

Expert answers to the most frequently asked questions and edge cases.

6 Q&As

No. A Disallow directive merely prevents search crawlers from downloading and parsing the page content. If third-party websites link to that URL, search engines can still index the naked URL based on external anchor text. To guarantee total exclusion from search results, you must allow crawling and serve a `<meta name='robots' content='noindex'>` tag.

ADVERTISEMENT · FOOTER BANNER

Wide Responsive Banner

(CLS Prevention Active · Zero Shift)

Related Tools

SEO & Webmaster View All →

Google & Naver SERP Snippet Preview

Calculate title pixel width, check meta description limits, and simulate desktop & mobile SERP previews.

Open Tool

UTM Campaign Link Builder for GA4

Build clean, validated UTM tracking URLs for Google Analytics 4, email campaigns, and paid social ads.

Open Tool

Email mailto · SMS · WhatsApp Deep Link & HTML Button Generator

Generate pre-filled mailto, SMS, and WhatsApp universal deep links with responsive HTML button code.

Open Tool

Keyword Density & Frequency Analyzer

Analyze word frequencies, 1-3 word n-gram densities, filter stopwords, and detect keyword stuffing risks.

Open Tool