呼叫控制器（crawler4j-3.5）內部循環

嗨我打電話controller裏面for-loop，因爲我有超過100個網址，所以我有所有在列表中，我會迭代和crawl頁面，我設置該URL爲setCustomData也是，因爲它不應該離開域。呼叫控制器（crawler4j-3.5）內部循環

for (Iterator<String> iterator = ifList.listIterator(); iterator.hasNext();) { 
    String str = iterator.next(); 
    System.out.println("cheking"+str); 
    CrawlController controller = new CrawlController(config, pageFetcher, 
     robotstxtServer); 
    controller.setCustomData(str); 
    controller.addSeed(str); 
    controller.startNonBlocking(BasicCrawler.class, numberOfCrawlers); 
    controller.waitUntilFinish(); 
}

但是，如果我運行上面的代碼，第一次URL完成後第二次URL入門和打印錯誤如下。

50982 [main] INFO edu.uci.ics.crawler4j.crawler.CrawlController - Crawler 1 started. 
51982 [Crawler 1] DEBUG org.apache.http.impl.conn.PoolingClientConnectionManager - Connection request: [route: {}->http://www.connectzone.in][total kept alive: 0; route allocated: 0 of 100; total allocated: 0 of 100] 
60985 [Thread-2] INFO edu.uci.ics.crawler4j.crawler.CrawlController - It looks like no thread is working, waiting for 10 seconds to make sure... 
70986 [Thread-2] INFO edu.uci.ics.crawler4j.crawler.CrawlController - No thread is working and no more URLs are in queue waiting for another 10 seconds to make sure... 
80986 [Thread-2] INFO edu.uci.ics.crawler4j.crawler.CrawlController - All of the crawlers are stopped. Finishing the process... 
80987 [Thread-2] INFO edu.uci.ics.crawler4j.crawler.CrawlController - Waiting for 10 seconds before final clean up... 
91050 [Thread-2] DEBUG org.apache.http.impl.conn.PoolingClientConnectionManager - Connection manager is shutting down 
91051 [Thread-2] DEBUG org.apache.http.impl.conn.PoolingClientConnectionManager - Connection manager shut down

請幫我解決了上述方案，我要作用中啓動和運行的內部迴路控制器，因爲我有在名單很多的url。

注意：**我正在使用** crawler4j-3.5.jar及其依賴項。

來源

2015-05-19 Selva

嘗試：

for(String url : urls) { 
    controller.addSeed(url); 
}

，並覆蓋shouldVisit(WebUrl)，使其不能離開域。

來源

2015-07-02 21:14:53 mohe2015

呼叫控制器（crawler4j-3.5）內部循環

回答

相關問題