2017-01-21 47 views
5

我不知道我怎麼會限制這樣的事情也只能使用10個線程在同一時間python如何設置線程限制?

with open("data.txt") as f: 
    for line in f: 
     lines = line.rstrip("\n\r") 
     t1 = Thread(target=Checker, args=("company")) 
     t1.start() 
+0

只是監視線程活着並阻塞,直到其中一個線程退出,如果有10個或更多。 – ForceBru

+0

可能的重複[如何限制在Python中的活動線程數量?](http://stackoverflow.com/questions/1787397/how-do-i-limit-the-number-of-active-threads-in -python) – zondo

+0

[Python線程池類似於多處理池?]可能重複(http://stackoverflow.com/questions/3033952/python-thread-pool-similar-to-the-multiprocessing-pool) –

回答

5

使用Python的ThreadPoolExecutor有max_workers參數設置爲10

事情是這樣的:`

pool = ThreadPoolExecutor(max_workers=10) 
with open("data.txt") as f: 
    for line in f: 
     lines = line.rstrip("\n\r") 
     pool.submit(Checker,"company") 

pool.shutdown(wait=True) 

pool將根據需要自動分配線程,限制分配的最大數量爲10。第一個參數在pool.submit()是函數名稱,參數只是以逗號分隔的值傳遞。

pool.shutdown(wait=True)等待所有線程完成執行。

3

使用ThreadPoolExecutor,並告訴它你要10個線程。

def your_function_processing_one_line(line): 
    pass # your computations 

with concurrent.futures.ThreadPoolExecutor(10) as executor: 
    result = executor.map(your_function_processing_one_line, [line for line in f]) 

...,你將不得不在result所有結果。

+0

如果多個參數? –

+1

這也是可能的。看看[這個很好的答案](https://stackoverflow.com/questions/6785226/pass-multiple-parameters-to-concurrent-futures-executor-map/6976772#6976772)。 – yogabonito

1

(對於Python的2.6+和Python 3)

使用threadPoolmultiprocessing模塊:

from multiprocessing.pool import ThreadPool 

的唯一的事情是,它沒有很好的記載......

1

我編寫了這個嵌套循環來將線程限制爲一個變量。 此代碼依賴於預設的命令數組進行處理。 我從其他答案中借用了一些元素來啓動線程。

import os, sys, datetime, logging, thread, threading, time 
from random import randint 

# set number of threads 
threadcount = 20 

# alltests is an array of test data 

numbertests = len(alltests) 
testcounter = numbertests 

# run tests 
for test in alltests: 
    # launch worker thread 
    def worker(): 
     """thread worker function""" 
     os.system(command) 
     return 
    threads = [] 
    t = threading.Thread(target=worker) 
    threads.append(t) 
    t.start() 
    testcounter -= 1 
    # cap the threads if over limit 
    while threading.active_count() >= threadcount: 
     threads = threading.active_count() 
     string = "Excessive threads, pausing 5 secs - " + str(threads) 
     print (string) 
     logging.info(string) 
     time.sleep(5) 

# monitor for threads winding down 
while threading.active_count() != 1: 
    threads = threading.active_count() 
    string = "Active threads running - " + str(threads) 
    print (string) 
    logging.info(string) 
    time.sleep(5)