The Robots.txt file is what tells the search engines which pages to access and index on your website and which pages not to. You may need to use the full-screen option to see the video more clearly.
1: In The Toolkit , go to Search Engines and then Robots.
2: Paste in your 'Custom Robots.txt'.
3: For any extra information regarding the syntax of Robots.txt files feel free to view the following page below. Find Out More
Directories Settings Analytics Sitemap Hyphens Open Graph
