Welcome to the Small Business Ideas Forum! We are a community of over 100,000 small business folks with over 163,000 posts for you to browse. We pride ourselves on being the friendliest forum you will find and we'd love to have you as a member of our community. Please take a moment and register for a free account. If you need any help, please contact Chris Logan.

Small Business Ideas Forum

Small Business Ideas Forum

A friendly place to share small business ideas and knowledge, ask questions, find help and encourage others that are involved in the small business industry. Topics include small business marketing, generating revenue and small business computing.

Go Back   Small Business Ideas Forum > Small Business Marketing - Online > Search Engine Optimization
Register Search Today's Posts Mark Forums Read

Reply
 
Thread Tools
Old 13th January 2015, 12:58 AM   #1
RosieGlover
VIP Contributor
 

Join Date: Mar 2014
Posts: 158
Default Question Relating To Robots.txt File

Hello Friends,

I never created a robots.txt file, through the available online resource I came to know the basic elements used in this file like User Agent and Allow, Disallow attribute.

I have to make the robots.txt file with the aim to allow only Googlebots to crawl my website and no other bots should access and crawl them. What should be the content of the robots.txt file to meet this requirement.

Experts, Please help me.

RosieGlover is offline   Reply With Quote
Register or log in to remove this ad.
Old 13th January 2015, 02:24 AM   #2
johnkca
Senior Member
 

Join Date: Dec 2014
Posts: 106
Default

site maps is one of the imporatant content in robots.txt

johnkca is offline   Reply With Quote
Old 13th January 2015, 05:47 AM   #3
alexjames212
Senior Member
 

Join Date: Jun 2014
Posts: 145
Default

I think this is the code

User-agent: google
Disallow:

User-agent: *
Disallow: /

Correct me if im wrong.

alexjames212 is offline   Reply With Quote
Old 14th January 2015, 02:46 AM   #4
RosieGlover
VIP Contributor
 

Join Date: Mar 2014
Posts: 158
Default

Quote:
Originally Posted by alexjames212 View Post
I think this is the code

User-agent: google
Disallow:

User-agent: *
Disallow: /

Correct me if im wrong.
Thanks for sharing the code snippet, moderators can give us confirmation whether the content for robots.txt file is right or wrong.

RosieGlover is offline   Reply With Quote
Old 14th January 2015, 05:43 AM   #5
alexjames212
Senior Member
 

Join Date: Jun 2014
Posts: 145
Default

yea hoping to get an answer soon.

alexjames212 is offline   Reply With Quote
Old 14th January 2015, 11:47 PM   #6
D_Kotian
Senior Member
 

Join Date: Oct 2012
Posts: 131
Default

I hope that code snippet is right, Still I have doubts on the same, Let some experienced people to review and confirm it.

__________________
Bolas Stellar ERP
D_Kotian is offline   Reply With Quote
Old 16th January 2015, 10:28 AM   #7
torka
Moderator
 
torka's Avatar
 

Join Date: Mar 2006
Location: Triangle area, NC, USA, North America, Earth (usually)
Posts: 4,101

Search Engine Guide Blogger

Default

@alexjames212 - you are correct. According to robotstxt.org, that is the correct code to allow Googlebot but exclude all other bots that obey the robots.txt file directives. (Note: not all bots pay attention to robots.txt, so it's not a "sure thing" when it comes to excluding your site from being visited by bots.)

Which begs the question, why would someone do that? "Bad" bots are extremely unlikely to obey robotx.txt to start with, so they'll go ahead and crawl your site anyway. Generally, it's only the legitimate "good" bots that will pay attention to your robots.txt file. One can get valuable traffic from "good" bots like Bing and other search engines, unless you've forbidden them to crawl your site (which is what this code will do).

Just seems to me that it's like shooting yourself in the foot...

--Torka

__________________
Diane Aull - NineYards.com: Helping Businesses Do Business Online
Whether you think you can, or that you can't, you are usually right.

Last edited by torka; 16th January 2015 at 10:31 AM.
torka is offline   Reply With Quote
Reply   

Bookmarks


Similar Threads
Thread Thread Starter Forum Replies Last Post
What Does It Mean If My Site Has A Robot.txt File? Cindy16 Search Engine Optimization 5 16th May 2014 03:26 AM
What Is The Use Of Robots.txt File. RosieGlover Search Engine Optimization 5 3rd April 2014 08:04 AM
Robots.txt File Tutorial WebBizIdeas.com Search Engine Optimization 2 16th January 2009 07:40 AM
The Ghost Robots.txt File! SmallMerchant Search Engine Optimization 6 25th February 2008 11:13 AM
Easy Backups with WinZip StupidScript Computer Security 0 9th December 2005 06:51 PM



Thread Tools

Get Updates
RSS Feeds:
RSS Feed for Search Engine Optimization RSS for this Category Only: Search Engine Optimization

RSS Feed for Small Business Ideas Forum RSS for Entire Forum
Forum Rules


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


Small Business Ideas Forum


 
At Your Business - Small Business Directory
Free Business Forms - Prewritten Documents
 
Search Engine Guide
Small business guide to search marketing

 
Small Business Brief
Fetching the Best Small Business Info


Used Values - Free Links
Blue Book





All times are GMT -5. The time now is 08:44 AM.


Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.
Copyright 2004 - 2018 - Privacy