Welcome to the Small Business Ideas Forum! We are a community of over 100,000 small business folks with over 163,000 posts for you to browse. We pride ourselves on being the friendliest forum you will find and we'd love to have you as a member of our community. Please take a moment and register for a free account. If you need any help, please contact Chris Logan.

Small Business Ideas Forum

Small Business Ideas Forum

A friendly place to share small business ideas and knowledge, ask questions, find help and encourage others that are involved in the small business industry. Topics include small business marketing, generating revenue and small business computing.

Go Back   Small Business Ideas Forum > Small Business Marketing - Online > Search Engine Optimization
Register Search Today's Posts Mark Forums Read

Reply
 
Thread Tools
Old 25th March 2014, 06:44 AM   #1
RosieGlover
VIP Contributor
 

Join Date: Mar 2014
Posts: 158
Default What Is The Use Of Robots.txt File.

I don't have any knowledge about on-page, can any one please let me know what is the use of the file as well as the content of the file.

RosieGlover is offline   Reply With Quote
Register or log in to remove this ad.
Old 25th March 2014, 09:57 AM   #2
Measurable
Member
 

Join Date: Mar 2014
Posts: 6
Default

A Robots.txt file is used to give instructions to the search engine spiders on which directories of a website they are allowed to crawl and how they are allowed to crawl them.

As best practice in on page seo, it is generally considered a requirement to have a robots.txt file, even if it is empty. The file is generally placed at the root of the website and uses "allow" or "disallow" commands to direct the search engine spiders.

I don't know about how links are shared on this forum so I won't post one, but if you do a search on "robots.txt examples" I am certain you will find several examples of how a properly formed robots.txt should be constructed.

Measurable is offline   Reply With Quote
Old 29th March 2014, 05:51 AM   #3
Rajiv
Member
 

Join Date: Dec 2013
Location: Jaipur
Posts: 84
Default

It is just a text file in which we tell search which can crawl and not to crawl like you can put your sitemap.html page in the robots.txt file as it in none of the concern with the Google to Google.

Rajiv is offline   Reply With Quote
Old 29th March 2014, 10:09 AM   #4
tvanslooten
Member
 

Join Date: Mar 2014
Location: Minneapolis, MN
Posts: 2
Default

Yes, it's just a simple text file so you can create one using notepad, wordpad, or any plain text editor. After you create it, you'll upload it to the root directory of your site so the file path will be something like this:

yourwebsite.com/robots.txt

I have no affiliation to this site but this article does a great job of explaining what this file is and how to create the various instructions that you can provide in this file:

feedthebot.com/robottxt.html

Travis

tvanslooten is offline   Reply With Quote
Old 1st April 2014, 06:22 AM   #5
gaylerichardson
Senior Member
 

Join Date: Feb 2014
Posts: 217
Default

Quote:
Originally Posted by Measurable View Post
A Robots.txt file is used to give instructions to the search engine spiders on which directories of a website they are allowed to crawl and how they are allowed to crawl them.

As best practice in on page seo, it is generally considered a requirement to have a robots.txt file, even if it is empty. The file is generally placed at the root of the website and uses "allow" or "disallow" commands to direct the search engine spiders.

I don't know about how links are shared on this forum so I won't post one, but if you do a search on "robots.txt examples" I am certain you will find several examples of how a properly formed robots.txt should be constructed.
Hii,

You are right and thank you so much for this information about robots.txt

gaylerichardson is offline   Reply With Quote
Old 3rd April 2014, 08:04 AM   #6
AvniShergill
Senior Member
 

Join Date: Apr 2014
Location: Maharashtra, Mumbai, India
Posts: 215
Default

Quote:
Originally Posted by RosieGlover View Post
I don't have any knowledge about on-page, can any one please let me know what is the use of the file as well as the content of the file.
Robots.txt files are used to tell the Goolge which page to index and which page to avoid. It usually contains allow and disallow attribute.

AvniShergill is offline   Reply With Quote
Reply   

Bookmarks


Similar Threads
Thread Thread Starter Forum Replies Last Post
Remote File Collaboration Yi X Computer Software 3 1st July 2010 06:17 PM
Robots.txt File Tutorial WebBizIdeas.com Search Engine Optimization 2 16th January 2009 07:40 AM
The Ghost Robots.txt File! SmallMerchant Search Engine Optimization 6 25th February 2008 11:13 AM
Easy Backups with WinZip StupidScript Computer Security 0 9th December 2005 06:51 PM
Making An RSS Feed Like Blogs Do StupidScript Blogs and Blogging 1 8th September 2005 05:11 PM



Thread Tools

Get Updates
RSS Feeds:
RSS Feed for Search Engine Optimization RSS for this Category Only: Search Engine Optimization

RSS Feed for Small Business Ideas Forum RSS for Entire Forum
Forum Rules


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


Small Business Ideas Forum


 
At Your Business - Small Business Directory
Free Business Forms - Prewritten Documents
 
Search Engine Guide
Small business guide to search marketing

 
Small Business Brief
Fetching the Best Small Business Info


Used Values - Free Links
Blue Book





All times are GMT -5. The time now is 08:32 AM.


Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.
Copyright 2004 - 2018 - Privacy