Search engine optimization is a vast field. And when it comes to technical SEO, it has always been a significant part of the search engine optimization process. From XML Sitemaps to duplicate content, crawl issues to HTTPS status codes, and mobile-friendliness to meta tags and keyword cannibalization, being proactive about on-page optimization is a must. These are the elements that can attract Google penalty if not taken care of duly. Here, we shall focus on controlling search engines to view your website content with the help of X robots and meta tags.
Exploring Meta Robot Tags
If you are completely new to technical SEO and do not understand what it is, not to worry. You can find the meta robots tag in the HTML source of a page and is placed within thetags of the page code. It appears somewhat like this:
<meta name=”robots” content=”noindex,follow” />
First of all, “meta name = robots” are considered the user agent, and “noindex,follow” are referred to search engine directives. This entire code/script informs all search engines (Google, Bing, Yahoo, and the rest of all) not to index the web page but to follow the web page links it comes across on that particular web page.
<meta name=”robots” content=”noindex”/>
This code informs all search engines not to index the page and not follow the links it comes across on that particular web page.
If you want to apply this only to a specific search engine e.g., Google, you should make this tag more specific.
The meta robots tag enables you to make minor adjustments and control search engines regarding what content should be indexed and displayed to SERPs’ users.
Importance Of Meta Robot Tags For A Website Owner
As a website owner aka business owner, you must be wondering why you should scratch your head over this when you have outsourced your SEO or plan to do so. Meta tags are an essential part of a solid SEO strategy. Meta robot tags are crucial for every website owner since they impact how your site appears in the SERPs when being searched. Its appearance and impressions also influence how many people will be inclined to click through to the website. And hence, the meta robot tags will impact the website traffic and engagement rates, thereby influencing the SEO strength and rankings.
Meta robots tag are little things in your site’s SEO that you can use to control your Google juice. The meta robots tag can be cracked down to four main functions for the search engine crawlers:
|A signal for the search engine crawler to follow the links in that webpage
|A signal for the search engine crawler NOT to follow the links in that webpage
|A signal for the search engine crawler to index that webpage
|A signal for the search engine crawler NOT to index that webpage
For example, assume that you run a dental clinic and your website has multiple pages that you would not like search engines to crawl and index. Those pages could be Thank You Page, Calendar to Book Appointments, etc. As a website owner, you need to make sure that it has a nofollow and noindex tag on all such web pages.
The meta robots tags are the most commonly used to control whether or not a page is indexed and whether or not search engines crawl the links on that particular web page. Not only this, meta robot tags can also control snippets on the SERPs.
Do not display snippets for a web page on the SERPs:
<meta name=”robots” content=”nosnippet” />
Set the maximum length of a text snippet to 125 characters:
<meta name=”robots” content=”max-snippet:125″ />
Set the maximum duration of a video preview to 12 seconds:
<meta name=”robots” content=”max-video-preview:12″ />
Set the maximum size of an image preview to small:
<meta name=”robots” content=”max-image-preview:small” />
If you do not want to place any restrictions on your snippets, you can add the below sitewide in your page header:
<meta name=”robots” content=”max-snippet:-1, max-image-preview:small, max-video-preview:-1″ />
Exploring X-robots Tag
Here’s an example of what an x-robots-tag header response looks like:
x-robots-tag: noindex, nofollow
Here Are A Few Use Cases When A Website Owner Might Need To Employ The X-Robot Tags
- To control the indexation of content like Flash or Video that is not written in HTML
- To control indexation if you don’t have access to a page’s HTML (specifically, to thesection) or if your site uses a global header that cannot be changed
- To block indexation of a particular element of a page (like an image or video), but not of the entire page itself
- To add criteria whether a page should be indexed or not (for example, if a user has downloaded a lead magnet, index their profile page)
Well, X robots and meta tags is a vast ocean. If you are a beginner in the field of technical SEO, you might come across a few hurdles when instructing search engines how to crawl and index a web page. There are common mistakes associated with it, with the most common being removing pages with a noindex directive from sitemaps, accidentally blocking search engines from crawling an entire site, keeping meta robots directives on a page blocked by robots.txt or adding robots directives to the robots.txt file.
Hiring a right SEO agency to fix your technical SEO issues helps you to keep unwanted pages out of SERP results with proper set up of X robots and meta tags. You can perfectly control how your pages are crawled as well as indexed and prevent search engines from following unwanted links.
No, they aren’t. Crawler directives i.e. Robots.txt – uses the user agent, allow, disallow and sitemap directives to specify where on site which search engine bots are allowed to crawl and not allowed to crawl and indexer directives i.e. X-Robots-tag – allows you to control how specific file types are indexed.
Add the following line to the configuration / a .htaccess file:
Header set X-Robots-Tag “noindex, noarchive, nosnippet”