如何避免抓取共享磁盤而不關閉它？

Question

我正在使用Nutch。我計劃抓取共享磁盤而不是互聯網網站。

我擔心的一件事是抓取它會使磁盤變得非常慢。 如何避免抓取共享磁盤而不關閉它？

Answer 1

您可以在conf/nutch-site.xml中設置請求之間的線程數量和等待時間。

嘗試overrinding這些屬性，並將其設置爲你感到舒服的值：

<property> 
    <name>fetcher.threads.fetch</name> 
    <value>10</value> 
    <description>The number of FetcherThreads the fetcher should use. 
    This is also determines the maximum number of requests that are 
    made at once (each FetcherThread handles one connection). The total 
    number of threads running in distributed mode will be the number of 
    fetcher threads * number of nodes as fetcher has one map task per node. 
    </description> 
</property> 

<property> 
    <name>fetcher.threads.per.queue</name> 
    <value>1</value> 
    <description>This number is the maximum number of threads that 
    should be allowed to access a queue at one time. 
    </description> 
</property>