{"id":23316,"date":"2020-02-20T18:13:52","date_gmt":"2020-02-20T17:13:52","guid":{"rendered":"https:\/\/avengering.com\/?p=23316"},"modified":"2020-02-16T11:08:27","modified_gmt":"2020-02-16T10:08:27","slug":"what-is-robots-txtrobot-txt-and-how-effect-in-seo","status":"publish","type":"post","link":"https:\/\/avengering.com\/en\/what-is-robots-txtrobot-txt-and-how-effect-in-seo\/","title":{"rendered":"what is robots txt(robot . txt)? and how effect in SEO?"},"content":{"rendered":"

Preface: What is a Robots.txt file?<\/span>\u00a0What effect does SEO have?<\/span><\/h2>\n
\n
\n

The Robots.txt file is a text file that webmasters can use to guide search engine robots when monitoring a website.<\/span>\u00a0The robot.txt file is part of a protocol called REP that standardizes how robots monitor the web<\/a> and how to access and index content and provide it to users.<\/span>\u00a0The REP protocol also includes how to handle the\u00a0Robots meta tag<\/a>\u00a0and other commands such as phallus links.<\/span><\/p>\n

In effect, robots text files tell user agents (such as search engine robots) which pages to monitor and which paths to work with.<\/span>\u00a0These are given in the form of instructions like Allow and Disallow.<\/span><\/p>\n

\"robot<\/a><\/p>\n

The basic format of this file is as follows:<\/span><\/p>\n

User-agent: [user-agent name]\r\nDisallow: [URL that should not be monitored]<\/pre>\n
This file allows you to write different commands for different user agents.<\/span>\u00a0It is enough to distinguish each set of commands for one user agent with a dash.<\/span>\u00a0Notice the following image:<\/span><\/p>\n
\"R\u00e9sultat<\/p>\n
If multiple sets of commands are created for multiple user agents in the robots.txt file, each user agent reads its instructions.<\/span>\u00a0Consider the following example:<\/span><\/p>\n
\"robots.txt\"<\/p>\n
Given the above commands, the user agent named msnbot only looks at the set of commands in Part I.<\/span>\u00a0If the user agent logged in to the site is none of these options, follow the instructions in Part Two that started with user-agent: *.<\/span><\/p>\n
Examples of robots.txt file<\/span><\/h3>\n
Here are some examples of the robots text file.<\/span>\u00a0Note that this file is located on the root of the site, www.example.com\/robots.txt.<\/span><\/p>\n
Block All Webmasters from Accessing the Whole Website:<\/span><\/p>\n
User-agent: *\r\nDisallow: \/<\/pre>\n
These commands in the robots.txt file tell-all web browsers not to monitor any page in the www.example.com domain.<\/span><\/p>\n
Access all site pages to all monitors:<\/span><\/p>\n
User-agent: *\r\nDisallow:<\/pre>\n
These commands in the robots text file tell-all web browsers that they can monitor all pages of the www.example.com domain.<\/span><\/p>\n
Block access to a specific monitor from a specific folder:<\/span><\/p>\n
User-agent: Googlebot\r\nDisallow: \/ example-subfolder \/<\/pre>\n
These commands tell the Googlebot Monitor that the URLs included at www.example.com\/example-subfolder\/ are not monitored.<\/span><\/p>\n
Block access to a specific webpage from a particular webpage:<\/span><\/p>\n
User-agent: Bingbot\r\nDisallow: \/example-subfolder\/blocked-page.html<\/pre>\n
These commands tell Bingbot Monitor (Bing Monitor Robot) not to monitor the URL only www.example.com\/example-subfolder\/blocked-page.html.<\/span><\/p>\n
How does the robots.txt file work?<\/span><\/h3>\n
Search engines have two main tasks:<\/span><\/p>\n