Noindex &
Nofollow
What is a Noindex Tag?
A noindex tag is a directive that signals to search engines that they can still crawl a webpage, but they should refrain from including it in their index. Essentially, it prevents a page from appearing in the search engine results pages (SERPs). This is particularly useful for excluding irrelevant or unnecessary pages from search engine indexing.
Google, for instance, will continue to crawl pages marked with a noindex tag but will do so less frequently compared to pages that are indexable. This practice helps optimise the website's crawl budget, which can lead to more efficient indexing.
What is a Nofollow Tag?
Nofollow tags instruct web crawlers not to follow any links found on a page or on specific links (depending on the implementation). Consequently, no link equity is passed on to the target URL when nofollow tags are applied. Using nofollow tags can be valuable in limiting crawling and enhancing crawler efficiency, especially when you want to prevent specific URLs from being crawled and indexed.
Nofollow tags are often employed alongside noindex tags, as the two work in tandem to control the crawling and indexing of a webpage.
How are Noindex and Nofollow Tags Applied?
You can apply these tags to a webpage in two primary ways:
1. HTML (Meta Robots Tag): You can include them within the HTML of the page using the meta robots tag.
The format typically looks like this:
<meta name="robots" content="noindex">
<meta name="robots" content="noindex, nofollow">
2. HTTP Response Header (X-Robots-Tags): Alternatively, you can use the X-Robots-Tags placed in the HTTP response header.
It appears as follows: HTTP/1.1 200 OK
Date: Wed, 27 September 2023 21:42:43 GMT
(...)
X-Robots-Tag: noindex
(...)
These X-Robots tags are particularly handy for controlling the indexing of non-HTML content, such as PDF files, videos, or Word documents, where it's not feasible to add robots directives within the HTML head.
For example, if you wish to block a specific image or video. Regular expressions can also be employed to group files together. For instance:
Files ~ "\.(png|jpeg|gif|webp)$"
Header set X-Robots-Tag "noindex"
This snippet would noindex any image files, regardless of their file extension.