Welcome to the Small Business Ideas Forum! We are a community of over 100,000 small business folks with over 163,000 posts for you to browse. We pride ourselves on being the friendliest forum you will find and we'd love to have you as a member of our community. Please take a moment and register for a free account. If you need any help, please contact Chris Logan.

Small Business Ideas Forum

Small Business Ideas Forum

A friendly place to share small business ideas and knowledge, ask questions, find help and encourage others that are involved in the small business industry. Topics include small business marketing, generating revenue and small business computing.

Go Back   Small Business Ideas Forum > Small Business Marketing - Online > General Search Engine Marketing Information
Register Search Today's Posts Mark Forums Read

Thread Tools
Old 18th August 2009, 08:07 AM   #1
BWelford's Avatar

Join Date: Jun 2004
Location: Langley, BC, Canada
Posts: 88


Even an empty robots.txt is worth adding since the search engine robots tend to look for that as a sign of a complete website.

More importantly you can block robots from spidering some parts of your website that you do not wish to have indexed. That could include parts that are under development. Not all robots observe the instructions in a robots.txt file but the more important ones do.

Barry Welford
Follow Me On Twitter Marketing Right Now
BWelford is offline   Reply With Quote
Register or log in to remove this ad.
Old 18th August 2009, 06:20 PM   #2

Join Date: Jun 2004
Location: Colorado
Posts: 8,046

Its a protocol that search engine robots use to identify what should be excluded from being indexed. Reading through this site will quickly make you an expert. I agree that an empty one is ok, but I've never experienced a search engine not indexing because a robots.txt wasn't in place. I rarely run into a need for one.

Search Engine Guide - Small Business Guide to Search Marketing
Small Business Brief - Small Business Ideas Forum, Articles & News
Logan is offline   Reply With Quote
Old 19th August 2009, 09:23 PM   #3

Join Date: Jun 2006
Location: Greenville, NC
Posts: 64
Default Minor suggestion

As you are developing a site, even if you use a development folder, make sure you include a robots.txt file to exclude your files in development or the pages will get indexed. They'll show up on SERP's and you'll be sorry.

Learned this the hard way.

Casey Turton
caseyturton is offline   Reply With Quote
Old 26th August 2009, 11:38 AM   #4

Join Date: Aug 2009
Posts: 7

robot text prevents spiders to crawl on pages that you don't want to be indexed, its a good idea to add it up on your site.

abuyouts is offline   Reply With Quote
Old 21st September 2009, 04:32 AM   #5

Join Date: Sep 2009
Posts: 5

how can you get one?robot.txt...

vickie21 is offline   Reply With Quote
Old 22nd September 2009, 03:57 AM   #6

Join Date: Mar 2009
Posts: 13

"Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all.

Here's a basic "robots.txt":

User-agent: *
Disallow: /

With the above declared, all robots (indicated by "*") are instructed to not index any of your pages (indicated by "/"). Most likely not what you want, but you get the idea

andrasan is offline   Reply With Quote
Old 23rd September 2009, 11:17 AM   #7
torka's Avatar

Join Date: Mar 2006
Location: Triangle area, NC, USA, North America, Earth (usually)
Posts: 4,101

Search Engine Guide Blogger


Originally Posted by vickie21 View Post
how can you get one?robot.txt...
You don't get one, you make one.

(Then you upload it to your web server.)

Follow the link Logan pointed out (up above) and you'll find a site that explains all about how to create a robots.txt file.



Diane Aull - NineYards.com: Helping Businesses Do Business Online
Whether you think you can, or that you can't, you are usually right.
torka is offline   Reply With Quote
Old 19th January 2010, 03:31 AM   #8

Join Date: Jan 2010
Posts: 10

Be REAL CAREFUL... one wrong statement in your robot.txt file will stop all crawlers... Sign up your site for Google webmaster tools - it will creat one for you to upload to your site

OLCDSarah is offline   Reply With Quote
Old 24th January 2010, 12:43 PM   #9

Join Date: Jan 2010
Posts: 25

So even if I create an empty robot.txt file and upload my sitewill have better place on search engines? That's really interesting! I thought that a robot.txt was very difficult to be done and work well.
Thanks for sharing

Danielle77 is offline   Reply With Quote
Old 30th January 2010, 09:39 PM   #10
Senior Member

Join Date: Jun 2009
Location: CN
Posts: 292
Send a message via MSN to AirForce1

Thanks for the nice sharing on robots.txt and I think sometimes you might really need this file to be under your site root because search engines like Bing would ask for this for a successful sitmap inclusion.


AirForce1 is offline   Reply With Quote


Similar Threads
Thread Thread Starter Forum Replies Last Post
Robots.txt Nutty Ecommerce 3 19th June 2009 10:50 AM
Robots.txt File Tutorial WebBizIdeas.com Search Engine Optimization 2 16th January 2009 07:40 AM
Help with Robots.txt johnnylew Search Engine Optimization 1 22nd July 2005 12:52 PM
Robots.txt Robert Search Engine Optimization 3 17th June 2005 11:13 PM
New Article - Search Engine Marketing FAQ - Robots.txt and Domain Names thejenn General Search Engine Marketing Information 0 28th April 2005 01:02 PM

Thread Tools

Get Updates
RSS Feeds:
RSS Feed for General Search Engine Marketing Information RSS for this Category Only: General Search Engine Marketing Information

RSS Feed for Small Business Ideas Forum RSS for Entire Forum
Forum Rules

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump

Small Business Ideas Forum

At Your Business - Small Business Directory
Free Business Forms - Prewritten Documents
Search Engine Guide
Small business guide to search marketing

Small Business Brief
Fetching the Best Small Business Info

Free Links - Free Advertising
Free Guide - Online Directory

Advertise your business here
Contact us for more details!

Semantic Juice
Register now to access free Quick SEO service!

Rocket Lawyer
Sign up for free 7 day trial. Boost your biz!

Buy UPC Codes
Get your products listed online!

All times are GMT -5. The time now is 10:26 AM.

Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.
Copyright 2004 - 2018 - Privacy