任何可能的方式來加快處理這個？

Question

我現在有80個用戶名的列表，我有我的腳本檢查每個用戶名是否存在。然而，它需要比我喜歡的時間長一點，所以我想知道是否有任何事情可以做，以加快需要多長時間來檢查每個用戶名是否存在。

# ------------------------------ 
# Mass Kik Username Checker 
# Script Made by: Ski 
# ------------------------------ 

import requests, threading 

def check(username): 
    try: 
     req = requests.get("http://kik.me/"+username, allow_redirects=False).status_code 

     if req == 302: 
      return False 
     if req == 200: 
      return True 
    except Exception as e: 
     print e 
     exit() 


def _loadList(filename): 
    item_list = [] 
    for item in str(open(filename, "r").read()).split("\n"): 
     item_list.append(item) 
    return item_list 

def _thread(items): 
    global _usernames 
    for username in _usernames[items[0]:items[1]]: 
     exists = check(username) 

     if exists: 
      print username+" exists\n" 
     if not exists: 
      print username+" doesn't exist\n" 

if __name__ == '__main__': 
    _usernames = _loadList("usernames.txt") 

    thread1 = threading.Thread(target=_thread, args=([0, 20],)).start() 
    thread2 = threading.Thread(target=_thread, args=([20, 40],)).start() 
    thread3 = threading.Thread(target=_thread, args=([40, 60],)).start() 
    thread4 = threading.Thread(target=_thread, args=([60, 80],)).start() 

Answer 1

試用Python 3.x Pool of threads。您可以定義有多少員工執行請求。使用比4更多（例如32），將會顯着加快代碼速度。

import requests 
from concurrent.futures import ThreadPoolExecutor 


NUM_OF_WORKERS=32 


def check(username): 
    try: 
     req = requests.get("http://kik.me/"+username, allow_redirects=False).status_code 

     if req == 302: 
      print(username, " does not exist.") 
     if req == 200: 
      print(username, "exists.") 
    except Exception as error: 
     print(error) 


usernames = _loadList(filename) 

with ThreadPoolExecutor(max_workers=NUM_OF_WORKERS) as pool: 
    pool.map(check, usernames) 

這使得您的代碼方式更具可讀性爲好。

編輯：現在注意到Python 2.7標記。

Python 2有一個線程池，可在multiprocessing模塊下使用。不幸的是，它沒有記錄，因爲沒有測試可用。

import requests 
from multiprocessing.pool import ThreadPool 


NUM_OF_WORKERS=32 


def check(username): 
    try: 
     req = requests.get("http://kik.me/"+username, allow_redirects=False).status_code 

     if req == 302: 
      print(username, " does not exist.") 
     if req == 200: 
      print(username, "exists.") 
    except Exception as error: 
     print(error) 


usernames = _loadList(filename) 


pool = ThreadPool(processes=NUM_OF_WORKERS) 
pool.map_async(check, usernames) 
pool.close() 
pool.join() 

如果你想線程的Python的2更好的游泳池，你可以嘗試Pebble module