如何進行未來調用並等到Python完成？

Question

我有下面的代碼，我有一個用戶名列表，我嘗試和檢查，如果用戶使用net user \domain | find somegroup是在一個特定的Windows用戶組。

的問題是，我可以針對每個用戶名約8羣組該命令，它是緩慢的。我想用期貨發送這些電話，甚至分開線程（如果它使速度更快）。

我只需要在年底前等待我做別的。我如何着手在Python中做這件事？

for one_username in user_list: 
    response = requests.get(somecontent) 

    bs_parsed = BeautifulSoup(response.content, 'html.parser') 

    find_all2 = bs_parsed.find("div", {"class": "QuickLinks"}) 
    name = re.sub("\s\s+", ' ', find_all2.find("td", text="Name").find_next_sibling("td").text) 

    find_all = bs_parsed.find_all("div", {"class": "visible"}) 
    all_perms = "" 
    d.setdefault(one_username + " (" + name + ")", []) 
    for value in find_all: 
     test = value.find("a", {"onmouseover": True}) 
     if test is not None: 
      if "MyAppID" in test.text: 
       d[one_username + " (" + name + ")"].append(test.text) 

    for group in groups: 
     try: 
      d[one_username + " (" + name + ")"].append(check_output("net user /domain " + one_username + "| find \"" + group + "\"", shell=True, stderr=subprocess.STDOUT).strip().decode("utf-8")) 
     except Exception: 
      pass 

Answer 1

（這個答案會忽略HTML解析你的代碼做...你可以隊列成池相同，以這種方式如何排隊net user調用）

首先，讓我們定義一個函數，它的一個(user, group)和tuple返回所需的信息。現在

# a function that calls net user to find info on a (user, group) 
def get_group_info(usr_grp): 
    # unpack the arguments 
    usr, grp = usr_grp 

    try: 
     return (usr, grp, 
       check_output(
        "net user /domain " + usr + "| find \"" + grp + "\"", 
        shell=True, 
        stderr=subprocess.STDOUT 
        ).strip().decode("utf-8"))) 
    except Exception: 
     return (usr, grp, None) 

，我們可以使用multiprocessing.dummy.Pool

from multiprocessing.dummy import Pool 
import itertools 

# create a pool with four worker threads 
pool = Pool(4) 

# run get_group_info for every user, group 
async_result = pool.map_async(get_group_info, itertools.product(user_list, groups)) 

# now do some other work we care about 
... 

# and then wait on our results 
results = async_result.get() 

的results是(user, group, data)元組列表中的一個線程池中運行這一點，你的願望都可以加工。

注：此代碼是目前未測試由於在平臺

Answer 2

看起來好像是producer consumer problem。

主線程應生成任務

class Task: 
    def Task(self,user,group) 
     self.user = user 
     self.group = group 
    def run(self): 
     pass # call command with self.user and self.group and process results 

twp = TaskWorkerPool(4) 
for group in groups: 
    twp.add(Task(user,group)) 
twp.wait() 

Answer 3

在Python 3的差，更簡單的和方便的解決方案是使用concurrent.futures。

concurrent.futures模塊爲異步執行可調用對象提供了一個高級接口。 Reference...

import concurrent.futures 


# Get a list containing all groups of a user 
def get_groups(username): 
    # Do the request and check here 
    # And return the groups of current user with a list 
    return list() 

with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor: 
    # Mark each future with its groups 
    future_to_groups = {executor.submit(get_groups, user): user 
         for user in user_list} 

    # Now it comes to the result of each user 
    for future in concurrent.futures.as_completed(future_to_groups): 
     user = future_to_groups[future] 
     try: 
      # Receive the returned result of current user 
      groups = future.result() 
     except Exception as exc: 
      print('%r generated an exception: %s' % (user, exc)) 
     else: 
      # Here you do anything you need on `groups` 
      # Output or collect them 
      print('%r is in %d groups' % (user, len(groups))) 

注意max_workers這裏指線程的最大數量。

見here哪裏這個例子從何而來。

編輯：

如果你需要做的每個檢查在單獨的線程：

import concurrent.futures 


# Check if a `user` is in a `group` 
def check(user, group): 
    # Do the check here 
    # And return True if user is in this group, False if not 
    return True 

with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor: 
    # Mark each future with its user and group 
    future_to_checks = {executor.submit(check, user, group): (user, group) 
         for user in user_list for group in group_list} 

    # Now it comes to the result of each check 
    # The try-except-else clause is omitted here 
    for future in concurrent.futures.as_completed(future_to_checks): 
     user, group = future_to_checks[future] 
     in_group = future.result() 
     if in_group is True: 
      print('%r is in %r' % (user, group)) 

通過@donkopotamus啓發，itertools.product這裏可以用來生成所有目標。

如果你不需要處理異常，它會更簡單：

import concurrent.futures 
from itertools import product 
from collections import defaultdict 


def check(target): 
    user, group = target 
    return True 

with concurrent.futures.ThreadPoolExecutor() as executor: 
    results = defaultdict(list) 
    targets = list(product(user_list, group_list)) 
    for (user, group), in_group in zip(targets, executor.map(check, targets)): 
     if in_group is True: 
      results[user].append(group) 

    print(results)