A robots.txt is a simple text file placed in the root directory of a website that tells search engine crawlers which pages, posts, or tags of the site they are allowed or not allowed to crawl.
A robots.txt file signals to search engines what your website’s engagement rules are. A big part of doing SEO is conveying the right signals to search engines, and robots.txt is one of the ways to communicate your crawl preferences to search engines.
Search engines continuously check a website’s robot.txt file for instructions on website crawling. We call these directives instructions.
If no robots.txt file is present or if there are no relevant directives, search engines will crawl the entire website.
Although all major search engines proceed with the robot.txt file, search engines may choose to ignore (parts of) your robot.txt file. While the directives in the robots.txt file are a strong signal to search engines, it is essential to remember that the robots.txt file is a set of voluntary guidelines directed at search engines, not commands.
A robots.txt file holds search engine directives, which you can use to restrict search engines from crawling certain parts of your website, giving search engines helpful advice on how they can better crawl your site and avoid duplicate content. The robots.txt file plays an essential role in SEO.
How to Generate a Robots.txt file
From a technical perspective, the robots.txt file is a text file in ASCII format, so it can be created using any simple text editor (Notepad or WordPad). It is usually highly advised to use any type of template that exists on the web to help in its creation.
- Robot.txt File Generator
- Robots Text Generator Tool, by Internet Marketing Ninjas
- Robot.txt Generator by SmallSEOTools
- Robot.txt Generator by Ryte
- Or the applications with Google Webmaster Tools or Bing Tools for web administrators.
It is also reliable that if a CMS is used, in most cases, this file may already be included in the initial installation. If not, it is always an option to install a plugin, an extension, or a module to update its content or, directly, create it. This happens with WordPress, Drupal, Joomla, etc.
Where Should the Robots.txt File Be Placed?
It should eternally be at the root of the server. On a generic website, if your site is www.techgogoal.com, it should appear when you type https://www.techgogoal.com/robots.txt.
The robots.txt file must be placed in the root directory (top-level folder) of your website so that search engines can find it automatically.
For most shared hosting providers (such as cPanel), upload the robots.txt file to the public_html folder (or the document root of your website).
Types of Robots.txt
There aren’t official types of robots.txt files, but there are several common configurations based on what you want search engines to do. Here are the most common ones:
1. By using this robots.txt search engine, bots crawl the entire website.
- User-agent: *
- Disallow:
Or
- User-agent: *
- Allow: /
2. This is for blocking all crawlers
- User-agent: *
- Disallow: /
3. To block specific folders, use this:
- User-agent: *
- Disallow: /admin/
- Disallow: /private/
- Disallow: /tmp/
4. Using a sitemap in a robots.txt for better search engine discovery:
- User-agent: *
- Disallow:
Sitemap: https://example.com/sitemap.xml
5. Use these to set unique instructions for multiple crawlers:
- User-agent: Googlebot
- Allow: /
- User-agent: Bingbot
- Disallow: /archive/
- User-agent: *
- Disallow: /private/
6. A common robots.txt setup for WordPress websites.
- User-agent: *
- Disallow: /wp-admin/
- Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap.xml
The ideal robots.txt relies on your website’s intended use, but for most public websites, a simple file that allows crawling, blocks only non-public areas, and references the sitemap is the recommended approach.
When Executing Robots.txt, Keep in Mind the Following Best Practices
Be careful when making changes to your robots.txt file – this file can make large parts of your website inaccessible to search engines.
The robots.txt file should appear at the root of your website (for example, https://www.abcd.com/robots.txt).
The robots.txt file is only valid for the domain in which it appears, including the protocol (HTTP or HTTPS)
Different search engines interpret directives differently. Usually, the first matching directive always wins. But, with Google and Bing, specificity wins.
Avoid using crawl-delay directives whenever possible. Web crawlers.
Importance of Robots.txt files
The robots.txt file represents an essential role from the SEO point of view. It informs search engines of the best time to crawl your website.
By using the robots.txt file, you can prevent search engines from accessing certain parts of your website, avoid duplicate content, and provide search engines with useful tips on how they can crawl your site more efficiently.
Be cautious when making changes to your robots.txt: This file has the potential to make large parts of your website inaccessible to search engines.