Everyone who has been involved in website development, or simply worked with websites, has heard of the term robots.txt.
It is a special UTF-8 encoded file that must sit in the root of the domain (https://mypersonaldomen.com/robots.txt) and be no larger than 500 KB. The commands in this file recommend (but do not guarantee) that search robots from Google, Yandex and others follow certain directives. For example, we can ask bots not to index registration pages, search forms, the client area, or any other page.
It is important to understand that these are merely recommendations we wish to convey, and they do not give a 100% guarantee. If an internal or external link to a page blocked in robots.txt is found, there is a certain chance the page will still end up in the search index.
robots.txt file syntax
User-Agent: specifies the name of the search engine bot the rules are addressed to.
Disallow: the path to the page we want to block access to.
Crawl-delay: a directive telling the search engine robot how often it should visit the site.
It can sometimes be useful on high-traffic sites, to reduce the load on the server.
Worth noting:
- The # symbol: used for comments inside the robots.txt file
- File and folder names are case-sensitive
- Host: a directive for Yandex that specifies the site's main mirror
- Sitemap: here you specify the full https path to the sitemap
- * - this symbol stands for any sequence of characters
robots.txt usage examples
Block indexing of all pages (useful while developing a new site):
User-agent: *
Disallow: /
Block the Google robot from indexing the /tmp folder
User-agent: Googlebot
Disallow: /tmp/
Block the Google robot from indexing the file /provider.html
User-agent: Googlebot
Disallow: /tmp/provider.html
Block all bots from indexing .pdf files
User-agent: *
Disallow: /*.pdf$
Allow Yandex robots to index the page provider.html
User-agent: Yandex
Allow: /tmp/provider.html
Path to the sitemap:
User-agent: *
Disallow:
Sitemap: https://mypersonaldomen.com/sitemap.xml
What the noindex meta tag is used for
Unlike our recommendations in the robots.txt file, we can give search engine bots a guaranteed ban on indexing a page.
To do this, add the following line to the page's HEAD:
<meta name="robots" content="noindex, follow"> - forbid indexing but allow following the links on the current page
<meta name="robots" content="noindex, nofollow"> - forbid both indexing and following the links on the current page