SEARCH ENGINE MARKETING
(512) 989-2945

Let us show you how your current site is doing and if your competitors are being favored by the search engines.

 

Get Your Search Engine Evaluation»

Email Address
First Name
Last Name

Week 9: 10 Things to do Before the Spiders Come

Submitted by Ben Finklea on July 9, 2007 - 3:16am.

Tip #9: The robots.txt

This file allows you to filter out the bots that crawl your website, plus you can restrict certain URLs that do not need to be indexed. This is a tremendous resource if used properly.

Post new comment

The content of this field is kept private and will not be shown publicly.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.

More information about formatting options

CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.
Image CAPTCHA
Copy the characters (respecting upper/lower case) from the image.