請求與代理和線程

Question

我試圖使用請求檢查巨大的代理列表。爲了做到這一點，我使用線程。因爲我使用相同的代碼結構後，爲了使一個網站，我做1個線程= 1個IP上的許多要求，我真的需要這些線程。

所以我排序的代碼是：

def proxyList(proxies, nbThread): 
    with open('proxyList.txt', 'w') as f: 
     f.write('') 
     f.close() 
    proxies = list(set(proxies)) 
    prox = [] 
    lenS = len(proxies) 
    pas = int(lenS/nbThread) 
    subSeq = [proxies[i*pas:(i+1)*pas] for i in range(nbThread)] 
    subSeq[nbThread-1]+=proxies[nbThread*pas:] 
    threads = [0 for i in range(nbThread)] 

    for i in range(nbThread): 
     threads[i] = proxy(subSeq[i],) 
    for i in range(nbThread): 
     threads[i].start(); 
    for i in range(nbThread): 
     threads[i].join(); 

    return list(set(prox)) 

class proxy(Thread): 
    def __init__(self, proxies): 
     Thread.__init__(self) 
     self.proxies = proxies 

    def run(self): 
     k=0 
     prox = [] 
     for proxy in self.proxies: 
      k+=1 
      try: 
       requests.get("https://api.ipify.org/?format=json", timeout=15, proxies={"https":str(re.findall(r'[0-9]+(?:\.[0-9]+){3}:[0-9]+', proxy)[0])}) 
       try: 
        requests.get("https://api.ipify.org/?format=json", timeout=15, proxies={"https":str(re.findall(r'[0-9]+(?:\.[0-9]+){3}:[0-9]+', proxy)[0])}) 
        prox+=[str(proxy)] 
        print("Bon proxy : " + str(re.findall(r'[0-9]+(?:\.[0-9]+){3}:[0-9]+', proxy)[0])) 
        with open('proxyList.txt', 'a') as f: 
         f.writelines(str(re.findall(r'[0-9]+(?:\.[0-9]+){3}:[0-9]+', proxy)[0])+'\n'); 
         f.close() 
       except: 
        t = "a" 
      except: 
       print("Mauvais proxy : "+ str(re.findall(r'[0-9]+(?:\.[0-9]+){3}:[0-9]+', proxy)[0])) 
       print(sys.exc_info()[0]) 
     print("Terminé: "+str(k), prox) 

它的工作原理，但我並不總是有相同的輸出結果，它是高度相關的線程我設置的數量。

你們是否有一個想法，我已經看到了，也許是請求不是最好的選擇，在這裏，但我真的需要我的主題我的代理。

感謝， Djokx

Answer 1

我敢肯定，要求是最好的方式。以下是討論。

https://gist.github.com/kennethreitz/973705

但我試圖讓你的代碼的一些改進，它一直在做同樣的工作，並防止所謂的「get」方法兩次循環下降。

希望它可以幫助

def proxyList(proxies, nbThread): 
    with open('proxyList.txt', 'w') as f: 
     f.write('') 
     f.close() 
    proxies = list(set(proxies)) 
    prox = [] 
    lenS = len(proxies) 
    pas = int(lenS/nbThread) 
    subSeq = [proxies[i*pas:(i+1)*pas] for i in range(nbThread)] 
    subSeq[nbThread-1]+=proxies[nbThread*pas:] 
    threads = [0 for i in range(nbThread)] 

    for i in range(nbThread): 
     threads[i] = proxy(subSeq[i],) 
     threads[i].start() 
     threads[i].join() 

    return list(set(prox)) 

class proxy(Thread): 
    def __init__(self, proxies): 
     Thread.__init__(self) 
     self.proxies = proxies 

    def run(self): 
     k=0 
     prox = [] 
     for proxy in self.proxies: 
      k+=1 
      try: 
       s = requests.Session() 
       try: 
        s.get("https://api.ipify.org/?format=json", timeout=15, proxies={"https":str(re.findall(r'[0-9]+(?:\.[0-9]+){3}:[0-9]+', proxy)[0])}) 
        prox+=[str(proxy)] 
        print("Bon proxy : " + str(re.findall(r'[0-9]+(?:\.[0-9]+){3}:[0-9]+', proxy)[0])) 
        with open('proxyList.txt', 'a') as f: 
         f.writelines(str(re.findall(r'[0-9]+(?:\.[0-9]+){3}:[0-9]+', proxy)[0])+'\n'); 
         f.close() 
       except: 
        t = "a" 
      except: 
       print("Mauvais proxy : "+ str(re.findall(r'[0-9]+(?:\.[0-9]+){3}:[0-9]+', proxy)[0])) 
       print(sys.exc_info()[0]) 
     print("Terminé: "+str(k), prox)