众里寻她千百度,蓦然回首,却是一头笨猪

2008-06-26搜索引擎
baiduspider webtrend
1,952 views

由于网站的历史遗留问题，网站存有大量动态url地址（如info_list.php?）,而且参数非常混乱。动态url地址由于存在时间较长，已被搜索引擎大量收录，存于搜索引擎的索引数据库中，且有了很高的权重。
使用webtrends对日志进行了spider爬行分析得出，这些动态url地址有很高的排行优先权，多次都是spider通过这些动态url从数据库中直接爬行至网站，进行抓取。由于spider是通过如info_list.php?此类动态页面爬行至此，在抓取的过程中spider一直在动态网页中循环抓取（因为参数比较混乱，spider会根据参数排列逻辑自己排列组合进行抓取，且不是返回404，所以导致了动态页面无穷多，严重浪费了spider的抓取效率），无法抓取到网站的伪静态页面。
对此情况采取了以下措施：
1、对全站url进行静态化处理，限制不同参数的排列组合，固定参数非正确的排列组合返回404.
2、对静态化的页面进行META规则制定，每个页面拥有自己独立的META信息，避免造成这些页面被收录后，在索引数据库中本站关键词竞争，可以突出重点词。
3、制作静态化页面sitemap.xml ，且通过robots.txt屏蔽各个动态目录。
4、通过google管理员工具删除已存在于google索引数据库中的动态url地址，可以删除整个动态目录。
通过以上方法，经过约1个月时间，目前google.com索引数据库中动态url地址已全部删除。site:该目录以为0. 通过webtrends对日志spider爬行的分析，目前已经批量抓取静态页面。原有排名的动态url页面，已被静态页面取代。spider抓取、索引数据库中的静态页面排名已趋于正常。
通过此次操作，又再次体会出google.com比baidu.com智能的多，其中最重要一点包括google.com可以通过管理员工具很方便的与站长沟通，即能帮助站长解决问题又能节省自己的spider抓取资源。而百度呢？baiduspider根本就不能控制，乱抓。也不明白站长的意思。技术含量极低… … 排序规则、更新规则中又夹杂着n多人为因素。

众里寻她千百度,蓦然回首,却是一头笨猪.

日志信息 »

该日志于2008-06-26 15:27由阿猎发表在搜索引擎分类下，你可以发表评论。除了可以将这个日志以保留源地址及作者的情况下引用到你的网站或博客，还可以通过RSS 2.0订阅这个日志的所有评论。

22条评论▼

瀚霆说：

2011-08-17 于

前辈啊
想学习系webtrends的使用，是在无从下手
能否指点一二
或者给点资料呢
单人操作说：

2011-11-04 于

动态地址有时真的很不给力的
Florentino Holzheimer 说：

2012-01-15 于

I just want to say I am just all new to weblog and actually loved you’re web page. Most likely I’m planning to bookmark your website . You surely have superb well written articles. Bless you for sharing with us your blog site.
How can i make my hair grow faster 说：

2012-01-21 于

hello. I see you are focused on hair growth
disney s photopass promotion code 说：

2012-01-21 于

Your article has been quite helpful in aiding me to resolve a private issue. Thanks
hello 说：

2012-01-22 于

1 cheese
How to make your hair grow faster 说：

2012-01-24 于

Good post. And to be on-topic with the post, I don’t know for you girls but for me hair is really the most important factor for my look. My hair stilist from the saloon where I go gave me around 2 months ago some type of hair oil which makes my hair grow faster and thicker. I don’t want to spam this blog so, who wants to find more details can click on my user name.Also my apologies for my bad english
Channel Island Jobs 说：

2012-01-25 于

Liberalism is trust of the people tempered by prudence. Conservatism is distrust of the people tempered by fear. ~William E. Gladstone, 1866
tables basses 说：

2012-01-27 于

I like this web blog very much so much wonderful info.
site 说：

2012-01-27 于

you are in point of fact a just right webmaster. The site loading velocity is amazing. It kind of feels that you are doing any distinctive trick. Moreover, The contents are masterpiece. you’ve performed a wonderful job in this topic!
sacoche homme lacoste 说：

2012-01-27 于

As soon as I observed this internet site I went on reddit to share some of the love with them.
robotmenager 说：

2012-01-28 于

Unquestionably imagine that that you said. Your favourite justification seemed to be at the net the simplest thing to remember of. I say to you, I certainly get irked while other folks think about concerns that they plainly don’t recognize about. You managed to hit the nail upon the top as neatly as outlined out the whole thing without having side effect , other people could take a signal. Will probably be back to get more. Thanks!
appareil photo bridge 说：

2012-01-28 于

I don’t usually comment but I gotta admit regards for the post on this amazing one : D.
friteuse sans huile 说：

2012-01-29 于

Very interesting information!Perfect just what I was searching for!
online movies 说：

2012-01-29 于

I am very happy to read this. This is the kind of manual that needs to be given and not the random misinformation that’s at the other blogs. Appreciate your sharing this greatest doc.
cliquez ici 说：

2012-01-30 于

I dugg some of you post as I cerebrated they were handy very useful
single 说：

2012-01-31 于

I enjoy what you guys are usually up too. This kind of clever work and reporting! Maintain up the very very good works guys I’ve added you guys to blogroll.
loose tea 说：

2012-01-31 于

I cling on to listening to the news bulletin lecture about receiving boundless online grant applications so I have been looking around for the best site to get one. Could you advise me please, where could i acquire some?
Toronto Website Designers 说：

2012-01-31 于

Web Page Design solutions will need impressive ideas for the best world-wide-web design and style and elegance.
table à langer 说：

2012-01-31 于

I like the valuable information you supply in your articles. I will bookmark your weblog and test again right here frequently. I’m moderately sure I will be informed many new stuff right right here! Best of luck for the next!
sorbetière 说：

2012-02-01 于

But wanna comment on few general things, The website design is perfect, the content is real fantastic : D.
trancheuse 说：

2012-02-01 于

Thanks a lot for being our lecturer on this subject matter. I enjoyed your own article a lot and most of all enjoyed the way in which you handled the aspect I considered to be controversial. You are always extremely kind to readers much like me and assist me to in my living. Thank you.

SEO博客,在SEO中实战中的阿猎SEO项目管理博客