Robots Meta Tag: Page-Level Instructions for Crawlers
The robots meta tag is a line of HTML in the head of a page that gives search engines page-level instructions like noindex and nofollow.
Robots Meta Tag is an HTML tag that gives crawlers page-level instructions.
What It Is
The robots meta tag is an HTML meta tag placed in the head of a page that gives search engines instructions for that page.
The most common directives are noindex, which keeps a page out of results, and nofollow, which tells crawlers not to follow its links.
Common Directives
- noindex: do not show this page in search results.
- nofollow: do not follow links on this page.
- noindex, nofollow: apply both.
- nosnippet: do not show a text snippet.
Why It Matters
- It controls indexing page by page without touching robots.txt.
- It is the reliable way to keep sensitive or thin pages out of search.
- Used wrongly, it can hide pages you actually want indexed.
How to Use
- Place it in the head of each page you want to control.
- Use noindex rather than blocking in robots.txt for pages you want de-indexed.
- Combine with the X-Robots-Tag header when you cannot edit HTML.
- Verify the tag in URL Inspection after publishing.
Before: a thank-you page keeps appearing in search results.
After: a noindex robots meta tag is added to the page.
The result: the page drops out of the index while the rest of the site is unaffected.
The lesson: control every page from the head of the page itself.
Quick Tip
The robots meta tag controls only the page it sits on, which makes it the safest tool for page-level indexing control.
Frequently Asked Questions
Robots Meta Tag, in Short
One tag in the head controls one page.
Use noindex to hide, not robots.txt.