導入模塊（nltk）導致多處理掛起

我跟蹤了一個python多處理頭痛，導入模塊（nltk）。可重複的（希望）代碼粘貼在下面。這對我沒有任何意義，有沒有人有任何想法？導入模塊（nltk）導致多處理掛起

from multiprocessing import Pool 
import time, requests 
#from nltk.corpus import stopwords # uncomment this and it hangs 

def gethtml(key, url): 
    r = requests.get(url) 
    return r.text 

def getnothing(key, url): 
    return "nothing" 

if __name__ == '__main__': 
    pool = Pool(processes=4) 
    result = list() 
    nruns = 4 
    url = 'http://davidchao.typepad.com/webconferencingexpert/2013/08/gartners-magic-quadrant-for-cloud-infrastructure-as-a-service.html' 
    for i in range(0,nruns): 
#  print gethtml(i,url) 
     result.append(pool.apply_async(gethtml, [i,url])) 
#  result.append(pool.apply_async(getnothing, [i,url])) 
    pool.close() 

    # monitor jobs until they complete 
    running = nruns 
    while running > 0: 
     time.sleep(1) 
     running = 0 
     for run in result: 
      if not run.ready(): running += 1 
     print "processes still running:",running 

    # print results 
    for i,run in enumerate(result): 
     print i,run.get()[0:40]

請注意'getnothing'函數的工作原理。它是nltk模塊導入和請求調用的組合。嘆

> python --version 
Python 2.7.6 

> python -c 'import sys;print("%x" % sys.maxsize, sys.maxsize > 2**32)' 
('7fffffffffffffff', True) 

> pip freeze | grep requests 
requests==2.2.1 

> pip freeze | grep nltk 
nltk==2.0.4

來源

2014-03-02 Ziggy Eunicien

我會用類似的問題重定向到其他不使用的多模塊解決方案：

1）Apache的火花可擴展性/靈活性。但是，這似乎並不是python多處理的解決方案。看起來pyspark也受全球口譯員鎖限制？

2） 'GEVENT' 或一般蟒異步處理 '扭曲' http://sdiehl.github.io/gevent-tutorial/

3）grequests用於異步請求 Asynchronous Requests with Python requests

來源

2014-03-07 04:35:21

導入模塊（nltk）導致多處理掛起

回答

相關問題