Robots.txt is a simple and short text file stored in the root directory of a domain that instructs search engine crawlers like Googlebot, what they are allowed to crawl on your site. In SEO, robots.txt helps crawl the pages of high importance first, preventing them…