Welcome to the Small Business Ideas Forum! We are a community of over 100,000 small business folks with over 163,000 posts for you to browse. We pride ourselves on being the friendliest forum you will find and we'd love to have you as a member of our community. Please take a moment and register for a free account. If you need any help, please contact Chris Logan.

Small Business Ideas Forum

Small Business Ideas Forum

A friendly place to share small business ideas and knowledge, ask questions, find help and encourage others that are involved in the small business industry. Topics include small business marketing, generating revenue and small business computing.

Go Back   Small Business Ideas Forum > Small Business Marketing - Online > Search Engine Optimization
Register Search Today's Posts Mark Forums Read

Reply
 
Thread Tools
Old 14th February 2008, 11:36 AM   #1
SmallMerchant
Member
 
SmallMerchant's Avatar
 

Join Date: Feb 2008
Location: Connecticut
Posts: 66
Send a message via AIM to SmallMerchant
Default The Ghost Robots.txt File!

Hello,

I am wondering if anyone can help. I have searched my site via FTP and through the files directory (where you can see hidden files) and cannot find a robots.txt file. However, google says there is one and it is blocking them!

Furthermore, I go to my URL plus the file name....
http://thesmallmerchant.com/robots.txt/

and it shows that there is a file. Why can I not find it on my server? I also tried to create a file with this name, and upload it thinking it would overwrite what was on there, that didn't work.

Does anyone have any clue how I can find this file?

Oh, and does the following mean it is blocking all search engines?

User-agent: *
Disallow: /

Thanks!! Nick

__________________
New URL Shortner: http://pip.ly
SmallMerchant is offline   Reply With Quote
Register or log in to remove this ad.
Old 14th February 2008, 12:21 PM   #2
Sporkman
VIP Contributor
 
Sporkman's Avatar
 

Join Date: Oct 2007
Posts: 304
Default

Yes, it is blocking all robots (that pay attention to robots.txt) from crawling anywhere on your website.

Sporkman is offline   Reply With Quote
Old 14th February 2008, 12:29 PM   #3
Sporkman
VIP Contributor
 
Sporkman's Avatar
 

Join Date: Oct 2007
Posts: 304
Default

I looked at the html code of your front page, looks like you have one of these at the top:

<meta name='robots' content='noindex,nofollow' />

See:

http://www.robotstxt.org/meta.html
http://googlewebmastercentral.blogsp...-meta-tag.html
http://www.askapache.com/seo/updated...wordpress.html

Still not sure why it's showing a robots.txt file to exist...

Sporkman is offline   Reply With Quote
Old 14th February 2008, 12:45 PM   #4
Sporkman
VIP Contributor
 
Sporkman's Avatar
 

Join Date: Oct 2007
Posts: 304
Default

This might be what's going on:

http://wordpress.org/extend/plugins/kb-robotstxt/

Sporkman is offline   Reply With Quote
Old 14th February 2008, 01:58 PM   #5
Logan
Administrator
 

Join Date: Jun 2004
Location: Colorado
Posts: 8,047
Default

Good work sporkman!

I agree, because there is a / after the robots.txt and it is not located at the directory level or root. It's either a setting with wp or your host depending on your setup.

__________________
Search Engine Guide - Small Business Guide to Search Marketing
Small Business Brief - Small Business Ideas Forum, Articles & News
Logan is offline   Reply With Quote
Old 14th February 2008, 05:47 PM   #6
SmallMerchant
Member
 
SmallMerchant's Avatar
 

Join Date: Feb 2008
Location: Connecticut
Posts: 66
Send a message via AIM to SmallMerchant
Default

Thanks so much for all your help guys! I downloaded that plugin for WP and it worked fine.

Thanks! Nick

__________________
New URL Shortner: http://pip.ly
SmallMerchant is offline   Reply With Quote
Old 25th February 2008, 11:13 AM   #7
rick22
Member
 

Join Date: Feb 2008
Posts: 20
Default

I think the file is situated on the top.... look again.

rick22 is offline   Reply With Quote
Reply   

Bookmarks




Thread Tools

Get Updates
RSS Feeds:
RSS Feed for Search Engine Optimization RSS for this Category Only: Search Engine Optimization

RSS Feed for Small Business Ideas Forum RSS for Entire Forum
Forum Rules


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


Small Business Ideas Forum


 
At Your Business - Small Business Directory
Free Business Forms - Prewritten Documents
 
Search Engine Guide
Small business guide to search marketing

 
Small Business Brief
Fetching the Best Small Business Info


Used Values - Free Links
Blue Book





All times are GMT -5. The time now is 10:54 PM.


Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.
Copyright 2004 - 2018 - Privacy