Welcome to the Small Business Ideas Forum! We are a community of over 100,000 small business folks with over 163,000 posts for you to browse. We pride ourselves on being the friendliest forum you will find and we'd love to have you as a member of our community. Please take a moment and register for a free account. If you need any help, please contact Chris Logan.

Small Business Ideas Forum

Small Business Ideas Forum

A friendly place to share small business ideas and knowledge, ask questions, find help and encourage others that are involved in the small business industry. Topics include small business marketing, generating revenue and small business computing.

Go Back   Small Business Ideas Forum > Small Business Marketing - Online > Search Engine Optimization
Register Search Today's Posts Mark Forums Read

Thread Tools
Old 18th May 2016, 06:23 AM   #1

Join Date: Nov 2015
Posts: 23
Default Is It Necessary To Use Robots.txt?


My question is, Is it necessary to use Robots.txt?

AliceFowell is offline   Reply With Quote
Register or log in to remove this ad.
Old 18th May 2016, 06:56 AM   #2

Join Date: Dec 2015
Posts: 69

It is important to use in the site. The pages have robots.txt won't crawled by Google. This can be used for highly secured page.

williamjohn01 is offline   Reply With Quote
Old 18th May 2016, 07:45 AM   #3

Join Date: Feb 2016
Posts: 24

Yes of course you can prevent bots to crawl unnecessary pages so that search engine can find only relevant content

designstoredxb is offline   Reply With Quote
Old 19th May 2016, 01:13 AM   #4
VIP Contributor
jeffscott's Avatar

Join Date: Oct 2014
Location: Singapore
Posts: 927

Robots.txt also improve better indexing for the website.

Worldwide trusted website for Free Bet365 Football Tips!
jeffscott is offline   Reply With Quote
Old 20th May 2016, 01:14 AM   #5

Join Date: May 2015
Posts: 40

It is not important use robots. txt file but it is advisable to use it, if there is any page on your websites which you don't want search engine to crawl. Robots.txt also improves better indexing for the website.

LisaHandson is offline   Reply With Quote
Old 20th May 2016, 02:05 AM   #6
VIP Contributor

Join Date: Apr 2015
Location: CA, USA
Posts: 273


Yes it is important for your website to get crawled and indexing by Google.

chrisjohn93 is offline   Reply With Quote
Old 20th May 2016, 05:29 AM   #7

Join Date: May 2016
Location: Ahmedabad, Gujarat, India
Posts: 50

Generally, robots.txt contains :

User-agent: *
Disallow: /

Without the “forward slash” in the instructions, search engines are granted access to the entire site. So, the inclusion of this one character in the robots.txt can prevent a website from showing in the search engines.

Errors aside, another benefit of having a robots.txt is that you can specify the location of the Google .xml or Yahoo sitemap with this simple instruction:

jameswarner is offline   Reply With Quote
Old 23rd May 2016, 01:41 AM   #8

Join Date: Jun 2014
Posts: 73

I think it os helpful for indexing or crawling.

randiv1984 is offline   Reply With Quote
Old 23rd May 2016, 05:34 AM   #9

Join Date: Jun 2015
Posts: 21

It is not necessary unles you have something to get done with this, like blocking some bots. If all bots are welcome, then you don't need it.

HUBBION - Reach customers over SMS/Email/Voice/Fax
Digital Authority - HIGH authority link building
anond is offline   Reply With Quote
Old 24th May 2016, 01:17 AM   #10
btadman21's Avatar

Join Date: Jul 2015
Location: USA
Posts: 1

It is not necessary to use robots.txt file in website.

But you need to use robots file if there is any page or pages on your websites which you don't want to crawl in search engine, for prevent crawling from search engine..

btadman21 is offline   Reply With Quote


Similar Threads
Thread Thread Starter Forum Replies Last Post
Question Relating To Robots.txt File RosieGlover Search Engine Optimization 6 16th January 2015 10:28 AM
Robots.txt Nutty Ecommerce 3 19th June 2009 10:50 AM
Robots.txt File Tutorial WebBizIdeas.com Search Engine Optimization 2 16th January 2009 07:40 AM
Help with Robots.txt johnnylew Search Engine Optimization 1 22nd July 2005 12:52 PM

Thread Tools

Get Updates
RSS Feeds:
RSS Feed for Search Engine Optimization RSS for this Category Only: Search Engine Optimization

RSS Feed for Small Business Ideas Forum RSS for Entire Forum
Forum Rules

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump

Small Business Ideas Forum

At Your Business - Small Business Directory
Free Business Forms - Prewritten Documents
Search Engine Guide
Small business guide to search marketing

Small Business Brief
Fetching the Best Small Business Info

Used Values - Free Links
Blue Book

All times are GMT -5. The time now is 07:36 PM.

Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.
Copyright 2004 - 2018 - Privacy