« »
2008-08-12搜索引擎

3,689 views

如何利用robots限定yahoo蜘蛛爬行

yahoo蜘蛛频繁爬行给公司系统部带来了巨大的麻烦,因为蜘蛛的频繁爬行直接导致系统反应过慢,那么如何才能降低yahoo的蜘蛛爬行抓取速度呢?
在yahoo的帮助定义:

在robots.txt 里,YST有一个特定的扩展名,可以通过它对我们的抓取程序设定一个较低的抓取请求频率。您可以加入Crawl-delay:xx指示,其中,“XX”是指在crawler程序两次进入站点时,以秒为单位的最低延时。如果crawler频率对您的服务器是一个负担,您可以将这个延时设定为任何您认为恰当的数字,例如1或10。

    例如,您想设定一个2秒的延时,语句如下:

    User-agent: Slurp
    Crawl-delay: 2

他们给出的帮助建议是设定为2,这个设定为2用来告诉Slurp蜘蛛2次访问之间的间隔,单位是秒.但对于大型网站设定2秒肯定是不合适的,因为雅虎美国和雅虎中国有2套蜘蛛在运行,每个蜘蛛都是遵循Crawl-delay间隔的,因此为了进一步降低Slurp压力,希望达到10秒一次请求的频道 实际的Crawl-delay配置就要*2。
Slurp来源于40多个IP网段,同网段内的蜘蛛协调抓取频度。因此:实际的Slurp设置,需要设置为10*40 = 400秒,才能达到期望的Slurp每10秒访问一次频度。

那么要想限定YAHOO蜘蛛的爬行,那么robots.txt可以写成

 

User-agent: *
Disallow:

User-agent: slurp
Disallow:
Crawl-delay: 100

 

定义说明:

1. 添加了User-agent: *属性。允许所有蜘蛛访问。
2. 更正了Slurp数值,1000为25秒,如果需要,可以设定成10000
3. 添加了sitemaps地址,将b2b网站的新旧sitemasps加入robots,便于蜘蛛抓取.

 

 

本说明来源于雅虎官方日志:
如何控制Yahoo! Slurp蜘蛛的抓取频度
http://ysearchblog.cn/2006/07/yahoo_slurp.html

日志信息 »

该日志于2008-08-12 15:46由 阿猎 发表在搜索引擎分类下, 你可以发表评论。除了可以将这个日志以保留源地址及作者的情况下引用到你的网站或博客,还可以通过RSS 2.0订阅这个日志的所有评论。

相关日志 »

相关日志

26条评论

  1. 好文章,值得一看

  2. 为什么要限制yahoo的爬行呢

  3. Kip Postma 说:

    Howdy very nice website!! Guy .. Excellent .. Wonderful .. I’ll bookmark your website and take the feeds additionally…I’m glad to find a lot of helpful information right here within the post, we need work out more techniques in this regard, thank you for sharing. . . . . .

  4. Mable Manning 说:

    I was wondering if you ever thought of changing the structure of your website? Its very well written; I love what youve got to say. But maybe you could a little more in the way of content so people could connect with it better. Youve got an awful lot of text for only having one or 2 pictures. Maybe you could space it out better?

  5. Mora Cosier 说:

    Hello there! Would you mind if I share your blog with my twitter group? There’s a lot of folks that I think would really enjoy your content. Please let me know. Many thanks

  6. I was just seeking this info for some time. After 6 hours of continuous Googleing, at last I got it in your website. I wonder what is the lack of Google strategy that don’t rank this type of informative websites in top of the list. Generally the top sites are full of garbage.

  7. Randy Balwin 说:

    I just want to say I am just new to blogs and actually enjoyed this web site. Most likely I’m going to bookmark your blog post . You absolutely have awesome articles and reviews. Kudos for revealing your webpage.

  8. Alma Jarecke 说:

    Needed to post you that very small remark to say thanks yet again on the stunning suggestions you have provided on this site. It is certainly tremendously generous of people like you in giving easily what many individuals would’ve advertised as an e-book to make some bucks on their own, certainly seeing that you could have tried it if you decided. These guidelines in addition served to provide a good way to recognize that most people have a similar zeal like my personal own to find out a whole lot more pertaining to this issue. I think there are many more pleasant instances ahead for people who scan through your site.

  9. Whats up very cool website!! Guy .. Beautiful .. Wonderful .. I’ll bookmark your site and take the feeds also…I’m satisfied to find a lot of helpful info here in the submit, we’d like work out extra strategies on this regard, thank you for sharing. . . . . .

  10. By visiting your webpage, the first impression for me is strong. I can’t imagine when and why you share this great topic but don’t spread it with social bookmarking. This information can be published as reference in online journal, or even in press release site. An early improvement in your site is great, can give us more time in your website. Would you mind if I capture several screenshot as my collection, because I’ve joined several researches? General purpose for me is to tell you about this discussion. My critical question for us is the resource that you have used to manage this site. In order to make great discussion, you are great because you post new topic in several areas. But, I suggest in giving personal opinion, please refer to big or authority sites, I am sure you will be fine in giving past or future experiences. In my environment, I am sure your capability to enrich people can be strong advantage for your future.

  11. History is gossip but scandal is gossip made tedious by morality. ~Oscar Wilde

  12. Mike 说:

    I conceive this web site holds some very great info for everyone : D.

  13. Very interesting details you have noted , thanks for posting . “Lefty Wise guy dont carry wallets, they carry their money in a roll….beaner on the outs” by Donnie Brasco.

  14. Some really select content on this internet site , saved to my bookmarks .

  15. here 说:

    I was reading some of your content on this internet site and I think this website is real instructive! Retain posting.

  16. I enjoy your writing style really loving this site.

  17. decodeur tnt 说:

    I like assembling utile info, this post has got me even more info! .

  18. There is perceptibly a bunch to know about this. I consider you made certain nice points in features also.

  19. I’d always want to be update on new content on this website , saved to fav! .

  20. Merely a smiling visitor here to share the love (:, btw outstanding design and style. “Make the most of your regrets… . To regret deeply is to live afresh.” by Henry David Thoreau.

  21. Web Page Design services call for progressive ideas for the very best website page layout and type.

  22. loose tea 说:

    Magnificent goods from you, man. I have understand your stuff previous to and you’re just extremely excellent. I actually like what you have acquired here, really like what you’re stating and the way in which you say it. You make it entertaining and you still take care of to keep it sensible. I cant wait to read far more from you. This is really a great site.

  23. Regards for helping out, wonderful info .

  24. removals 说:

    On this very subject, I try update my blogs almost like I’m speaking to somebody or talking to that reader directly as I consider it helps engage folks a great deal much more than writing content formally particularly when my own weblog is based around the goings ons, every day thoughts and research about this somewhat controversial concern. I discovered that I nonetheless need to have to use a lot more of the strategies you have mentioned in this posting.

  25. Very interesting topic, thanks for putting up.

发表评论 »

返回顶部