Welcome to the Small Business Ideas Forum! We are a community of over 100,000 small business folks with over 163,000 posts for you to browse. We pride ourselves on being the friendliest forum you will find and we'd love to have you as a member of our community. Please take a moment and register for a free account. If you need any help, please contact Chris Logan.

Small Business Ideas Forum

Small Business Ideas Forum

A friendly place to share small business ideas and knowledge, ask questions, find help and encourage others that are involved in the small business industry. Topics include small business marketing, generating revenue and small business computing.

Go Back   Small Business Ideas Forum > Small Business Marketing - Online > Search Engine Optimization
Register Search Today's Posts Mark Forums Read

Reply
 
Thread Tools
Old 18th May 2016, 06:23 AM   #1
AliceFowell
Banned
 

Join Date: Nov 2015
Posts: 23
Default Is It Necessary To Use Robots.txt?

Hi,

My question is, Is it necessary to use Robots.txt?

AliceFowell is offline   Reply With Quote
Register or log in to remove this ad.
Old 18th May 2016, 06:56 AM   #2
williamjohn01
Member
 

Join Date: Dec 2015
Posts: 69
Default

It is important to use in the site. The pages have robots.txt won't crawled by Google. This can be used for highly secured page.

williamjohn01 is offline   Reply With Quote
Old 18th May 2016, 07:45 AM   #3
designstoredxb
Member
 

Join Date: Feb 2016
Posts: 24
Default

Yes of course you can prevent bots to crawl unnecessary pages so that search engine can find only relevant content

designstoredxb is offline   Reply With Quote
Old 19th May 2016, 01:13 AM   #4
jeffscott
VIP Contributor
 
jeffscott's Avatar
 

Join Date: Oct 2014
Location: Singapore
Posts: 927
Default

Robots.txt also improve better indexing for the website.

__________________
Worldwide trusted website for Free Bet365 Football Tips!
jeffscott is offline   Reply With Quote
Old 20th May 2016, 01:14 AM   #5
LisaHandson
Member
 

Join Date: May 2015
Posts: 40
Default

It is not important use robots. txt file but it is advisable to use it, if there is any page on your websites which you don't want search engine to crawl. Robots.txt also improves better indexing for the website.

LisaHandson is offline   Reply With Quote
Old 20th May 2016, 02:05 AM   #6
chrisjohn93
VIP Contributor
 

Join Date: Apr 2015
Location: CA, USA
Posts: 273

Default

Yes it is important for your website to get crawled and indexing by Google.

chrisjohn93 is offline   Reply With Quote
Old 20th May 2016, 05:29 AM   #7
jameswarner
Member
 

Join Date: May 2016
Location: Ahmedabad, Gujarat, India
Posts: 50
Default

Generally, robots.txt contains :

User-agent: *
Disallow: /

Without the “forward slash” in the instructions, search engines are granted access to the entire site. So, the inclusion of this one character in the robots.txt can prevent a website from showing in the search engines.

Errors aside, another benefit of having a robots.txt is that you can specify the location of the Google .xml or Yahoo sitemap with this simple instruction:

jameswarner is offline   Reply With Quote
Old 23rd May 2016, 01:41 AM   #8
randiv1984
Member
 

Join Date: Jun 2014
Posts: 73
Default

I think it os helpful for indexing or crawling.

randiv1984 is offline   Reply With Quote
Old 23rd May 2016, 05:34 AM   #9
anond
Member
 

Join Date: Jun 2015
Posts: 21
Default

It is not necessary unles you have something to get done with this, like blocking some bots. If all bots are welcome, then you don't need it.

__________________
HUBBION - Reach customers over SMS/Email/Voice/Fax
Digital Authority - HIGH authority link building
anond is offline   Reply With Quote
Old 24th May 2016, 01:17 AM   #10
btadman21
Member
 
btadman21's Avatar
 

Join Date: Jul 2015
Location: USA
Posts: 1
Default

It is not necessary to use robots.txt file in website.

But you need to use robots file if there is any page or pages on your websites which you don't want to crawl in search engine, for prevent crawling from search engine..

btadman21 is offline   Reply With Quote
Reply   

Bookmarks


Similar Threads
Thread Thread Starter Forum Replies Last Post
Question Relating To Robots.txt File RosieGlover Search Engine Optimization 6 16th January 2015 10:28 AM
Robots.txt Nutty Ecommerce 3 19th June 2009 10:50 AM
Robots.txt File Tutorial WebBizIdeas.com Search Engine Optimization 2 16th January 2009 07:40 AM
Help with Robots.txt johnnylew Search Engine Optimization 1 22nd July 2005 12:52 PM



Thread Tools

Get Updates
RSS Feeds:
RSS Feed for Search Engine Optimization RSS for this Category Only: Search Engine Optimization

RSS Feed for Small Business Ideas Forum RSS for Entire Forum
Forum Rules


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


Small Business Ideas Forum


 
At Your Business - Small Business Directory
Free Business Forms - Prewritten Documents
 
Search Engine Guide
Small business guide to search marketing

 
Small Business Brief
Fetching the Best Small Business Info


Used Values - Free Links
Blue Book





All times are GMT -5. The time now is 07:36 PM.


Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.
Copyright 2004 - 2018 - Privacy