Asp.net Request.Browser.Crawler - 動態爬蟲列表？

Question

我學會了爲什麼Request.Browser.Crawler在C＃中始終是False（http://www.digcode.com/default.aspx?page=ed51cde3-d979-4daf-afae-fa6192562ea9&article=bc3a7a4f-f53e-4f88-8e9c-c9337f6c05a0）。

有沒有人使用某種方法來動態更新Crawler的列表，所以Request.Browser.Crawler會非常有用？

Answer 1

我已經幸福的Ocean's Browsercaps提供的結果。它支持Microsoft的配置文件沒有檢測到的抓取工具。它甚至可以解析出您網​​站上的抓取工具的版本，而不是我真的需要這種詳細程度。

Answer 2

你可以檢查（正則表達式）針對Request.UserAgent。

Peter Bromberg寫了一篇關於在ASP.NET中編寫ASP.NET Request Logger and Crawler Killer的好文章。

這是他在Logger類使用方法：

public static bool IsCrawler(HttpRequest request) 
{ 
    // set next line to "bool isCrawler = false; to use this to deny certain bots 
    bool isCrawler = request.Browser.Crawler; 
    // Microsoft doesn't properly detect several crawlers 
    if (!isCrawler) 
    { 
     // put any additional known crawlers in the Regex below 
     // you can also use this list to deny certain bots instead, if desired: 
     // just set bool isCrawler = false; for first line in method 
     // and only have the ones you want to deny in the following Regex list 
     Regex regEx = new Regex("Slurp|slurp|ask|Ask|Teoma|teoma"); 
     isCrawler = regEx.Match(request.UserAgent).Success; 
    } 
    return isCrawler; 
}