I got a question from one of the readers. John H asked me: If I don’t have any pages to hide from Search Engines, Do I need to use Robots.txt file?
This is quiet a common question and many SEO consultants are asking webmasters to add robots.txt file mandatory. As I have mentioned in the previous post on What is a robots.txt file ? search engines will first look for your robots.txt file. If they check for www.yourdomain.com/robots.txt it returns a 404 Error.
It’s always better to have a robots.txt file on your site, even if it is just a blank file. This prevents Google and the other ‘bots from filling your log files with 404 errors, and functions just like a “real” non-blank robots.txt file containing.
User-agent: *
Disallow:
If you have a custom 404 error page with your sitemap, when spider check for robots.txt file and see this page it is well and good.
If you don’t have a 404 error page, please go ahead and add a blank robots.txt file as mentioned above and have the spiders see the page not redirected.
