排除某些網址被抓取

Question

我正在寫一個搜尋器，並且在那個抓取器中，我不想抓取某個網頁（排除一些鏈接以使其不抓取）。所以我寫了那個頁面的排除。什麼毛病此代碼。由於這個http://www.host.com/technology/網址獲取調用盡管寫排除。我不希望這個網址http://www.host.com/technology/來抓取的..開頭的URL

public class MyCrawler extends WebCrawler { 

    Pattern filters = Pattern.compile(".*(\\.(css|js|bmp|gif|jpe?g" 
      + "|png|tiff?|mid|mp2|mp3|mp4" + "|wav|avi|mov|mpeg|ram|m4v|pdf" 
      + "|rm|smil|wmv|swf|wma|zip|rar|gz))$"); 

List<String> exclusions; 


    public MyCrawler() { 

     exclusions = new ArrayList<String>(); 
     //Add here all your exclusions 
//I do not want this url to get crawled.. 
     exclusions.add("http://www.host.com/technology/"); 

    } 

    public boolean shouldVisit(WebURL url) { 
     String href = url.getURL().toLowerCase(); 
     System.out.println(href); 
     if (filters.matcher(href).matches()) { 
      System.out.println("noooo"); 
      return false; 
     } 

     if (exclusions.contains(href)) {//why this loop is not working?? 
     System.out.println("Yes2"); 
      return false; 
    } 

     if (href.startsWith("http://www.host.com/")) { 
      System.out.println("Yes1"); 
      return true; 
     } 



     System.out.println("No"); 
     return false; 
    } 

    public void visit(Page page) { 
     int docid = page.getWebURL().getDocid(); 
     String url = page.getWebURL().getURL();   
     String text = page.getText(); 
     List<WebURL> links = page.getURLs(); 
     int parentDocid = page.getWebURL().getParentDocid(); 

     System.out.println("============="); 
     System.out.println("Docid: " + docid); 
     System.out.println("URL: " + url); 
     System.out.println("Text length: " + text.length()); 
     System.out.println("Number of links: " + links.size()); 
     System.out.println("Docid of parent page: " + parentDocid); 
     System.out.println("============="); 
    } 
} 

Answer 1

如果你不想要抓取與排除開始任何URL，你必須做這樣的事情：

for(String exclusion : exclusions){ 
    if(href.startsWith(exclusion)){ 
     return false; 
    } 
} 

此外，if語句不是循環。