How to block robots

Written by adrian grahams Google
  • Share
  • Tweet
  • Share
  • Pin
  • Email
How to block robots
Block the robot from your website with the root-level access file. (Stockbyte/Stockbyte/Getty Images)

All major search engines, online advertising services and affiliate networks use automated robot software to crawl and index the content of websites. This ensures that the search engine or network has an up-to-date copy of your site's content in its cache. The automated robots use up some of your website's bandwidth while they crawl the content of your site, and this can slow down the site for other visitors. If you want to block the robot from automatically crawling your website, you can block it with your Htaccess file. This file sits on the root server of your website and controls access to your website.

Skill level:


  1. 1

    Log in to your website host's server through the host's online content-management system or with a File Transfer Protocol tool.

  2. 2

    Double-click on the website root folder to open it. This is the folder that contains the main pages, images, scripts and multimedia content for your website, including the index file. Web hosts usually label the folder Root or Htdocs.

  3. 3

    Click the "Name" or "Filename" tab to display the folder and file list alphabetically.

  4. 4

    Find the ".htaccess" file in the displayed list. The file name starts with a period, so it should be easy to find toward the top of the list.

  5. 5

    Double-click the file to open it. This will automatically open the file in Microsoft Notepad or your computer's default plain-text reader and editor.

  6. 6

    Paste the following code at the bottom of the document, below all other entries:

    order allow, deny

    deny from

    allow from all

  7. 7

    Save the file. This updates the Htaccess file and instructs your Web server to block the bot.

Tips and warnings

  • Repeat this procedure to block other robots from accessing your website.
  • Blocking the bot might prevent Amazon ads and content links from displaying on your website.

Don't Miss

  • All types
  • Articles
  • Slideshows
  • Videos
  • Most relevant
  • Most popular
  • Most recent

No articles available

No slideshows available

No videos available

By using the site, you consent to the use of cookies. For more information, please see our Cookie policy.