robots.txt Optimization

robots.txt is used by search engine webcrawlers to determine what to index. robots.txt should be located in your document root. If you don't have one, you can create a blank one. This brief guide will help you with managing webcrawlers through your robots.txt file.

  • Stopping bots entirely. If you wish to stop webcrawlers from indexing your site (this will also stop webcrawlers from ranking your site), you will need to add this code to your robots.txt:
    #Code to not allow any search engines!
    User-agent: *
    Disallow: /
  • Stop bots from crawling parts of your site. You can allow some pages to be indexed, while preventing others from being indexed. You will need to add this code to your robots.txt. Replace "/cgi-bin/", "/tmp/", and "/junk/" with whatever pages you wish to be blocked.
    # Blocks robots from specific folders / directories
    User-agent: *
    Disallow: /cgi-bin/
    Disallow: /tmp/
    Disallow: /junk/
  • Google & Bing. Neither of these webcrawlers obey robots.txt. If you wish to control their webcrawler activity, you will need to make Google and Bing Webmaster accounts with their services.
  • 38 Users Found This Useful
Was this answer helpful?

Related Articles

Is it a problem if I'm not in the US?

Not at all! In fact, we already have customers in countries all over the world.

I uploaded my page but it's not working!

Make sure that you are uploading to the public_html or www folder.  Only the files placed...

What is an error 404 page?

Error 404 indicates that you are requesting a file or a directory that does not exist on the...

Can I run scripts like Wordpress, Joomla, or phpBB on my site?

Absolutely! We make it extremely easy to install over 250 different scripts via the Softaculous...

Can I upgrade my plan or features later?

Upgrades/Addons are available to purchase at any time through your Client Area dashboard. Once...