淘宝屏蔽百度蜘蛛

2008-09-08SEO
屏蔽，roboot
3,549 views

最近淘宝实施大淘宝战略，利用robot.txt对百度的蜘蛛进行屏幕，那么我们来看看什么叫Robot，互联网中的页面在被搜索引擎检索的时候是通过一种程序robot（又称spider），自动访问互联网上的网页并获取网页信息。

每个网站的站长都可以在网站中创建一个纯文本文件robots.txt，在这个文件中声明该网站中不想被robot访问的部分，这样，该网站的部分或全部内容就可以不被搜索引擎收录了，或者指定搜索引擎只收录指定的内容。robots.txt文件应该放在网站根目录下。

当一个搜索机器人（有的叫搜索蜘蛛）访问一个站点时，它会首先检查该站点根目录下是否存在robots.txt，如果存在，搜索机器人就会按照该文件中的内容来确定访问的范围；如果该文件不存在，那么搜索机器人就沿着链接抓取。

当我打开http://www.taobao.com/robots.txt时候，浏览器出现

User-agent: Baiduspider
Disallow: /

User-agent: baiduspider
Disallow: /

每种搜索引擎都有自己的搜索爬行，百度的爬虫叫做baiduspide，淘宝这样做的后果是屏蔽百度的蜘蛛爬行，google的蜘蛛叫做Googlebot，那么下一步大家跟我一起来看看规范的robots.txt文件的格式：
“robots.txt”文件包含一条或更多的记录，这些记录通过空行分开（以CR,CR/NL, or NL作为结束符），每一条记录的格式如下所示：”<field>:<optionalspace><value><optionalspace>”。

在淘宝使用ROBOTS屏蔽了百度的蜘蛛之后，百度会不会另外命名一种蜘蛛去爬行淘宝的内容呢？我们不得而知，等百度的C2C上线，我们拭目以待。同时也说明淘宝的用户基数大，今后会有更多的成熟网站拒绝搜索引擎的

日志信息 »

该日志于2008-09-08 21:30由阿猎发表在SEO分类下，你可以发表评论。除了可以将这个日志以保留源地址及作者的情况下引用到你的网站或博客，还可以通过RSS 2.0订阅这个日志的所有评论。

37条评论▼

闯子说：

2008-09-09 于

感觉很新鲜，学到新知识了！
多谢！！
ddload 说：

2008-09-12 于

原来是这样，学习了
青岛SEOMM 说：

2008-09-14 于

淘宝开始对百度宣战了……
腊肉说：

2011-10-17 于

是的，这个听说了的，但是百度最终还是收录淘宝的部分网页的
Allegro Meble Ogrodowe 说：

2011-12-31 于

This is very interesting, You’re a very skilled blogger. I’ve joined your feed and look forward to seeking more of your excellent post. Also, I’ve shared your site in my social networks!
Cheap Gay Condoms 说：

2012-01-04 于

Hello There. I found your blog the usage of msn. That is a really well written article. I will be sure to bookmark it and return to read more of your useful info. Thank you for the post. I will certainly comeback.
Allegro Drzwi Zewnetrzne 说：

2012-01-05 于

We are a group of volunteers and starting a new scheme in our community. Your site provided us with valuable info to work on. You’ve done an impressive job and our whole community will be thankful to you.
Otha Yokoyama 说：

2012-01-14 于

I just want to tell you that I am all new to weblog and certainly savored you’re blog site. Probably I’m going to bookmark your blog . You definitely come with superb well written articles. Thanks for sharing your web-site.
magento specialist 说：

2012-01-21 于

I’ve recently started a web site, and the info you provide on this website has helped me greatly. Thank you for all of your time & work.
disneys contemporary hotel 说：

2012-01-21 于

I read a blog the other day that talks about this very topic, just not as gracefully, great job on the post.
news 说：

2012-01-22 于

2 ice
patios in birmingham 说：

2012-01-22 于

I have to get across my admiration for your kind-heartedness in support of people who require help with this one idea. Your very own commitment to passing the solution up and down turned out to be definitely important and have surely allowed employees like me to achieve their pursuits. Your new helpful hints and tips signifies much a person like me and additionally to my mates. Many thanks; from all of us.
kasyno 说：

2012-01-23 于

I think other site owners should take this website as an model – very clean and excellent style and design, not to mention the content. You’re an expert in this topic!
online plagiarism checker 说：

2012-01-23 于

Thanks for this neat blog; this is the type of read that keeps me going through my day. Been looking around for your site after being sent to it from a colleague and was thrilled when I stumbled on it after searching for a while. Being a dedicated blogger, I’m dazzled to see authors like you taking initivative and contributing to the community. I would like to comment to show my appreciation for your work as it’s quite encouraging, and many authors do not get acceptance they deserve. Will definitely come back and will recommend to others.
vitamix turboblend 4500 说：

2012-01-23 于

Unquestionably believe that which you said. Your favorite reason seemed to be on the web the simplest thing to be aware of. I say to you, I certainly get irked while people think about worries that they just do not know about. You managed to hit the nail upon the top and defined out the whole thing without having side-effects , people can take a signal. Will likely be back to get more. Thanks
Jobs in Jersey 说：

2012-01-24 于

The politicians were talking themselves red, white and blue in the face. ~Clare Boothe Luce
dumpster rental livingston ny 说：

2012-01-24 于

Most users, but, pick fixed width. You may also file download subjects which can be seasonal; for illustration, you might need to download holiday topic, a photo-blogging concept, or any other seasonal subjects.
roby bowl 说：

2012-01-24 于

I laved as much as yau’ll receive carried aut right here. The sketch is attractive, yaur authared material stylish. nanetheless, yau cammand get baught an shakiness aver that yau wish be delivering the fallawing.
Bedrijfsruimte te koop 说：

2012-01-24 于

Zeer tevreden over de kwaliteit en dienstverlening van Big Freddy. Goed ingepakt, eenvoudig ophangsysteem, goede kleuren en zeer vriendelijke klantenservice. Had besteld een Forex van 150cm x 100 cm, een echte blikvanger die ik binnen 1 week binnen had. Kan het iedereen aanbevelen.
Maximo Slosek 说：

2012-01-24 于

I truly enjoy reading through on this site, it has got wonderful blog posts.
Channel Island Recruitment Agency 说：

2012-01-25 于

I like the smell of a dunged field, and the tumult of a popular election. ~Augustus William Hare and Julius Charles Hare, Guesses at Truth, by Two Brothers, 1827
J Joshua Beistle 说：

2012-01-25 于

Congratulation.. It looks awesome .. Good luck in your business
online plagiarism detector 说：

2012-01-25 于

Thanks for this distinct blog; this is the type of read that keeps me going through my day. Been looking around for your blog after being sent to it from a friend and was thrilled when I stumbled on it after googling for a sometime. As a demanding blogger, I’m excited to see others taking initivative and contributing to the community. I would like to comment to show my appreciation for your blog as it’s quite inspiring, and many writers do not get acceptance they deserve. I am sure I’ll visit again and will recommend to my friends.
tables basses 说：

2012-01-27 于

I like this post, enjoyed this one thankyou for posting .
Malignant Mesothelioma 说：

2012-01-29 于

Very interesting topic , thanks for posting .
Mockdrafthq.com 说：

2012-01-29 于

Im getting a teeny issue. I cant get my reader to pickup your feed, Im using google reader by the way.
trampoline avec filet 说：

2012-01-30 于

As soon as I observed this site I went on reddit to share some of the love with them.
robot cuisine 说：

2012-01-31 于

It is really a great and helpful piece of information. I am satisfied that you simply shared this useful information with us. Please keep us informed like this. Thanks for sharing.
white ceramic watches for women 说：

2012-01-31 于

Hi there, You’ve done an incredible job. I’ll definitely digg it and for my part suggest to my friends. I am sure they’ll be benefited from this web site.
sim only contract 02 说：

2012-01-31 于

As I site possessor I believe the content matter here is rattling wonderful , appreciate it for your hard work. You should keep it up forever! Best of luck.
epilateur electrique 说：

2012-01-31 于

I truly appreciate this post. I’ve been looking all over for this! Thank goodness I found it on Bing. You’ve made my day! Thank you again!
Toronto Website Designers 说：

2012-01-31 于

Web Page Design solutions demand progressive suggestions for the very best world wide web design and style.
european delivery services 说：

2012-02-01 于

I believe one of your advertisings triggered my internet browser to resize, you might effectively want to put that on your blacklist.
montre gps 说：

2012-02-01 于

Very interesting topic, thank you for putting up.
Salt Systems 说：

2012-02-01 于

Great writing.
echelle telescopique 说：

2012-02-01 于

As soon as I noticed this web site I went on reddit to share some of the love with them.
RV Storage 说：

2012-02-01 于

Please keep writing some a lot more. I enjoyed reading this fairly a little.

SEO博客,在SEO中实战中的阿猎SEO项目管理博客