User-agent: [Essential, one or more for each group] The rule specifies the title of the automated customer known as search engine crawler which the rule applies to. Take Management over exactly where search engine crawlers go With all the robot.txt file, a very small file with large Search engine optimisation https://seotoolstube.com/