Meta Tags & Robots Suite

The ultimate crawling and indexing toolkit. Generate valid robots meta tags, X-Robots headers, and professional robots.txt files.

Quick Start Presets

Meta Robots Generator

Additional Directives

Advanced Parameters

Robots.txt Generator

User Agent Matrix

Directories & Paths

Generated Directives

HTML META TAG
X-ROBOTS-TAG (HTTP HEADER)
ROBOTS.TXT

Pro Tip: Use X-Robots-Tag for non-HTML files like PDFs and images.

The Ultimate Guide to Robots Directives

Meta Robots and robots.txt are the dual pillars of search engine crawling control. While meta tags provide page-level instructions, robots.txt manages your site's overall "crawl budget."

How Meta Robots Work

When a crawler (like Googlebot) visits a URL, it first checks for a robots.txt file at the root. If the URL is allowed, it then parses the HTML. The Meta Robots tag in the <head> provides final instructions on whether to index that specific content.

Meta Robots vs. X-Robots-Tag

The X-Robots-Tag is an HTTP response header. It is more powerful than a meta tag because it can be applied to non-HTML assets, such as images, PDFs, and spreadsheets. For a truly professional setup, use meta tags for pages and headers for assets.

Crawl Budget Optimization

On large websites, Google may not crawl every page daily. By using robots.txt to block low-value directories (like internal search results or session-ID parameters), you ensure Google spends its time crawling your most important, high-revenue pages.

Enterprise SEO Strategies

  • Google News: Use the googlebot-news bot target to allow indexing in general search while restricting it in News carousels (using nosnippet).
  • Legal & Privacy: Protect internal staging sites or sensitive legal documents using noindex, nofollow, noarchive.
  • Asset Protection: Use noimageindex to prevent competitors from hotlinking your original photography via Google Image search.

Common Indexing Mistakes

The most frequent error is blocking a page in robots.txt that also has a noindex tag. Since Google is blocked from crawling the page, it can never see the noindex directive, and the page may remain in the index if found via external links.

SEO Cheat Sheet

DirectivePurposeGoogle
indexAllow indexing✅
noindexBlock indexing✅
followFollow links✅
nofollowBlock equity✅
noarchiveNo cached link✅
nosnippetNo text snippet✅
max-snippetLimit length✅
max-imageLimit img size✅
unavailableAuto de-index✅

Robots.txt Myth

Robots.txt is NOT a security tool. Blocked pages can still be indexed if linked from other sites. Use password protection for truly private data.

Implementation Best Practices

✅ Use indexifembedded

Combine with noindex to allow your content to be indexed only when it appears in an iframe on another page.

✅ Future Expirations

Use unavailable_after for seasonal sales or events. The page will auto-remove from SERPs on the date specified.

❌ Robots.txt Conflicts

Never block a URL in robots.txt if you want a noindex tag on that page to be respected by Google.

❌ Case Sensitivity

Directives are case-insensitive, but bot names often are not. Stick to standard lowercase for directives.

Frequently Asked Questions

Robots.txt tells crawlers where they can't go on your site (crawling). Meta Robots tells them what they can't show in search results (indexing).

No. Googlebot ignores the crawl-delay directive in robots.txt. You must manage their crawl rate via Google Search Central.

Yes, but it is redundant for HTML pages. If they conflict, search engines will generally follow the most restrictive directive.

It depends on how often the bot crawls your site. Once the bot sees the noindex tag, the page is typically removed within a few days.

Authoritative Resources

🔗 Google Search Central

Official specifications for robot meta tags and X-Robots-Tag HTTP headers.
🔗 Bing Webmaster Tools

How Bing handles crawling and indexing instructions for professional webmasters.
🔗 Yandex Help

Documentation for controlling YandexBot behavior via metadata and robots.txt.

Popular Categories

📊 SEO Tools 🔢 Calculators 🖼️ Image Tools 📝 Writing Tools

Related Toolkits

🛠️ Technical SEO ✍️ Blogging 📱 Social Media

Quick Start Presets

Meta Tags Generator

0/60
0/160

Technical Configuration

Search Preview & Code

GOOGLE SEARCH PREVIEW
Page Title Appears Here
https://example.com › page-slug
Your meta description will appear here in search engine results. This helps users understand what your page is about.
HTML META TAGS

Mastering SEO Meta Tags

Meta tags are snippets of text that describe a page's content; they don't appear on the page itself, but only in the page's source code. Meta tags are essentially little content descriptors that help tell search engines what a web page is about.

The Importance of Title Tags

The title tag is one of the most important on-page SEO factors. It appears in search results and at the top of web browsers. For maximum impact, keep titles under 60 characters and place primary keywords at the beginning.

Crafting the Perfect Meta Description

While meta descriptions don't directly influence rankings, they are crucial for Click-Through Rate (CTR). A compelling description serves as "organic ad copy," encouraging users to click your link instead of a competitor's.

Enterprise Meta Strategies

  • Brand Consistency: Always append your brand name to titles (e.g., "Keyword | Brand Name").
  • Dynamic Descriptions: Use variables to generate unique descriptions for large e-commerce sites.
  • Character Buffer: Aim for 55 characters for titles and 155 for descriptions to avoid truncation across all devices.

Length Benchmarks

ElementMax
Title60 Chars
Description160 Chars
Keywords10 Tags

Implementation Best Practices

✅ Use @graph

When using multiple schema types on one page, wrap them in a @graph array to show Google how they are connected.

✅ MainEntityOfPage

Explicitly state which entity is the primary focus of the URL to help search engines disambiguate complex pages.

❌ Hidden Content

Never mark up content that is hidden from users. This is a violation of Google's guidelines and can lead to a manual action.

❌ Generic Types

Avoid using Thing or WebPage if a more specific type like Article or Product exists.

Frequently Asked Questions

Major search engines like Google and Bing have ignored meta keywords for years to prevent spam. However, some smaller search engines and internal site search systems may still use them.

Google may rewrite your title tag if they believe a different title is more relevant to the user's search query or if your title tag is too long, too short, or lacks relevance.

Quick Start Presets

Open Graph Generator

Optional Properties

Social Sharing Preview

FACEBOOK / LINKEDIN PREVIEW
No Image Provided
EXAMPLE.COM
Social Share Title
Your social media description will appear here, providing a summary for users on platforms like Facebook and LinkedIn.
HTML OG TAGS

The Power of Open Graph Protocol

The Open Graph protocol enables any web page to become a rich object in a social graph. For instance, this is used on Facebook to allow any web page to have the same functionality as any other object on Facebook.

Why You Need OG Tags

Without Open Graph tags, social networks will guess which image and title to show when someone shares your link. Often, they pick the wrong image or a random string of text. OG tags give you 100% control over your social presence.

Optimizing Social Images

The standard size for Open Graph images is 1200x630 pixels (1.91:1 aspect ratio). Following this ensures your images look sharp and aren't awkwardly cropped in the feed.

Social Specs

Aspect Ratio: 1.91:1

Min Size: 600 x 315px

Max File: 8MB

Frequently Asked Questions

Use the Facebook Sharing Debugger to force Facebook to scrape your site again and update its cache with your new meta tags.

Yes, LinkedIn supports Open Graph tags and follows the same standards as Facebook for titles, descriptions, and images.

Quick Start Presets

X (Twitter) Card Generator

X Feed Preview

X (TWITTER) POST PREVIEW
Preview Image
X Post Title
The description of your content as it will appear when shared on X (formerly Twitter).
example.com
HTML X CARD TAGS

Optimizing for X (Twitter) Cards

Twitter Cards allow you to attach rich photos, videos, and media experiences to Tweets, helping to drive traffic to your website. By adding a few lines of HTML to your web pages, users who Tweet links to your content will have a "Card" added to the Tweet that's visible to their followers.

Choosing Your Card Type

The Summary Card with Large Image is generally the most effective for blog posts and articles, as it provides a large, eye-catching visual. The standard Summary Card is better for smaller blocks of content or when a large image isn't available.

X vs. Open Graph

If X does not find specific twitter: tags on your page, it will fall back to Open Graph tags (like og:title and og:image). However, using explicit X tags allows for better control and platform-specific optimization.

Image Sizes

Summary: 1:1 ratio (min 144x144)

Large Image: 2:1 ratio (min 300x157)

Frequently Asked Questions

The standalone Twitter Card Validator tool has been integrated into the standard post composer. To see a preview, simply paste your link into a new post on X.com.

Yes, X supports WebP in addition to standard JPG, PNG, and GIF formats for Card images.

Schema Type Presets

Schema Generator

JSON-LD Output

STRUCTURED DATA (JSON-LD)

Pro Tip: Paste this code into the <head> of your HTML or use a plugin to inject it. Always validate with Google's Rich Results Test.

Mastering Schema Markup (JSON-LD)

Schema markup is a standard vocabulary that you can add to your website's HTML to help search engines understand your content more effectively. By providing explicit clues about the meaning of a page, you increase the chances of earning "Rich Results" in Google Search.

Why JSON-LD?

Google officially recommends JSON-LD (JavaScript Object Notation for Linked Data) over other formats like Microdata or RDFa. It is easier to implement because it can be placed anywhere in the document and is decoupled from the user-facing HTML.

The SEO Impact of Structured Data

While Schema is not a direct ranking factor, it significantly improves Click-Through Rate (CTR) by enabling rich features like star ratings, price displays, event dates, and FAQ dropdowns directly on the search results page.

Schema Implementation Strategy

  • Entity Accuracy: Only mark up content that is actually visible on the page.
  • Nested Objects: Use nesting to show relationships, such as an Author belonging to an Organization.
  • Validation: Always test your code using the Schema Markup Validator and Google's Rich Results Test.

Validation Tools

Frequently Asked Questions

You can use multiple types, but they should ideally be combined into a single @graph object to show how the entities relate to one another. Avoid marking up unrelated entities on the same page.

Yes. LLMs and Search Generative Experiences rely heavily on structured data to parse facts and relationships reliably. Proper Schema makes your content much easier for AI to "consume" and cite.

While it is possible, it is not recommended as it creates redundancy and increases the risk of conflicting information. It is better to migrate everything to the modern JSON-LD format.

Google recommends placing it in the <head> section, but JSON-LD can technically be placed anywhere in the <body> as well.

Authoritative Resources

🔗 Google Search Central

Introduction to structured data and official Google guidelines for JSON-LD.
🔗 Schema.org Official

The full vocabulary reference for every structured data type and property available.
🔗 Rich Results Test

Google's official tool to test your Schema markup for rich result eligibility.

Popular Categories

📊 SEO Tools 🔢 Calculators 🖼️ Image Tools 📝 Writing Tools

Related Toolkits

🛠️ Technical SEO ✍️ Blogging 📱 Social Media

Canonical URL Generator

Enter the absolute URL of the page you want search engines to prioritize.

Language Alternates (hreflang)

Master Copy Rule

A canonical tag tells search engines which version of a URL is the "master" copy. Use this to prevent duplicate content issues when you have similar pages or multiple URLs pointing to the same content.

Canonical Implementation

HTML LINK TAG
X-ROBOTS-TAG (HTTP HEADER)

Understanding Canonicalization

Canonicalization is the process of picking the best URL when there are several options. This is important because search engines don't like duplicate content and may penalize your site or pick the wrong page to rank if you don't provide a preference.

When to Use Canonical Tags

  • Parameters: When URLs have session IDs or tracking parameters (e.g., ?utm_source=...).
  • Protocols: When the site is accessible via both http and https.
  • Structure: When content lives under multiple categories or folders.
  • Mobile: When there is a separate m.example.com site.

Cross-Domain Canonicals

You can use canonical tags to point to a master article on a completely different domain. This is often used by publishers who syndicate content across multiple websites.

Implementation Checklist

  • Use Absolute URLs
  • One Tag Per Page
  • Consistent Trailing Slashes
  • Lowercase URLs

Frequently Asked Questions

Yes. This is called a "self-referential" canonical tag. It is considered a best practice because it prevents tracking parameters from causing duplicate content issues.

For Google, it is a hint. While Google usually respects it, they may choose to ignore it if they believe another URL is a better canonical choice based on other signals.