Python + splinter + http：Error - httplib.ResponseNotReady

Question

使用splinter和Python，我有兩個線程正在運行，每個線程都訪問相同的主URL但路徑不同，例如，線程一個點擊：mainurl.com/threadone和線程兩支安打：用mainurl.com/threadtwo：

Traceback (most recent call last): 
    File "multi_thread_practice.py", line 299, in <module> 
    main() 
    File "multi_thread_practice.py", line 290, in main 
    first_method(r) 
    File "multi_thread_practice.py", line 195, in parser 
    second_method(title, name) 
    File "multi_thread_practice.py", line 208, in confirm_product 
    third_method(current_url) 
    File "multi_thread_practice.py", line 214, in buy_product 
    browser.visit(url) 
    File "/Users/joshua/anaconda/lib/python2.7/site-packages/splinter/driver/webdriver/__init__.py", line 184, in visit 
    self.driver.get(url) 
    File "/Users/joshua/anaconda/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 261, in get 
    self.execute(Command.GET, {'url': url}) 
    File "/Users/joshua/anaconda/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 247, in execute 
    response = self.command_executor.execute(driver_command, params) 
    File "/Users/joshua/anaconda/lib/python2.7/site-packages/selenium/webdriver/remote/remote_connection.py", line 464, in execute 
    return self._request(command_info[0], url, body=data) 
    File "/Users/joshua/anaconda/lib/python2.7/site-packages/selenium/webdriver/remote/remote_connection.py", line 488, in _request 
    resp = self._conn.getresponse() 
    File "/Users/joshua/anaconda/lib/python2.7/httplib.py", line 1108, in getresponse 
    raise ResponseNotReady() 
httplib.ResponseNotReady 

什麼是錯誤，我應該如何去處理這個問題：

from splinter import Browser 
browser = Browser('chrome') 

但是遇到下列錯誤來了？

謝謝你提前一定會給予好評/接受的答案

CODE ADDED

import time 
from splinter import Browser 
import threading 

browser = Browser('chrome') 

start_time = time.time() 

urlOne = 'http://www.practiceurl.com/one' 
urlTwo = 'http://www.practiceurl.com/two' 
baseUrl = 'http://practiceurl.com' 

browser.visit(baseUrl) 

def secondThread(url): 
    print 'STARTING 2ND REQUEST: ' + str(time.time() - start_time) 
    browser.visit(url) 
    print 'END 2ND REQUEST: ' + str(time.time() - start_time) 


def mainThread(url): 
    print 'STARTING 1ST REQUEST: ' + str(time.time() - start_time) 
    browser.visit(url) 
    print 'END 1ST REQUEST: ' + str(time.time() - start_time) 


def main(): 
    threadObj = threading.Thread(target=secondThread, args=[urlTwo]) 
    threadObj.daemon = True 

    threadObj.start() 

    mainThread(urlOne) 

main() 

Answer 1

據我所知，你想做什麼是不可能的在一個瀏覽器上。斯普林特正在對一個實際的瀏覽器進行操作，因此，同時傳入許多命令會導致問題。它的行爲就像人類與瀏覽器交互（當然是自動的）。可以打開許多瀏覽器窗口，但不能在不接收前一個請求的響應的情況下，在其他線程中發送請求。這會導致CannotSendRequest錯誤。所以，我建議（如果您需要使用線程）打開兩個瀏覽器，然後使用線程通過每個瀏覽器發送請求。否則，它不能完成。

此線程在硒上，但信息是可轉移的。 Selenium multiple tabs at once同樣，這說明你想要的（我假設）要做的事情是不可能的。綠色答覆的答題者給出了和我一樣的建議。

希望不會讓你走得太遠，並幫助你。

編輯：只是爲了證明：

import time 
from splinter import Browser 
import threading 

browser = Browser('firefox') 
browser2 = Browser('firefox') 

start_time = time.time() 

urlOne = 'http://www.practiceurl.com/one' 
urlTwo = 'http://www.practiceurl.com/two' 
baseUrl = 'http://practiceurl.com' 

browser.visit(baseUrl) 


def secondThread(url): 
    print 'STARTING 2ND REQUEST: ' + str(time.time() - start_time) 
    browser2.visit(url) 
    print 'END 2ND REQUEST: ' + str(time.time() - start_time) 


def mainThread(url): 
    print 'STARTING 1ST REQUEST: ' + str(time.time() - start_time) 
    browser.visit(url) 
    print 'END 1ST REQUEST: ' + str(time.time() - start_time) 


def main(): 
    threadObj = threading.Thread(target=secondThread, args=[urlTwo]) 
    threadObj.daemon = True 

    threadObj.start() 

    mainThread(urlOne) 

main() 

請注意，我用的Firefox，因爲我一直沒有得到chromedriver安裝。

在定時器開始之前，在瀏覽器打開後設置一個等待，以確保它們完全準備就緒可能是一個好主意。

Answer 2

@GenericSnake在這個問題上是正確的。爲了一點點添加到它，我會強烈建議您重構代碼使用multiprocessing library，主要是因爲線程實現使用GIL：

In CPython, due to the Global Interpreter Lock, only one thread can execute Python code at once (even though certain performance-oriented libraries might overcome this limitation). If you want your application to make better use of the computational resources of multi-core machines, you are advised to use multiprocessing. However, threading is still an appropriate model if you want to run multiple I/O-bound tasks simultaneously.

實際使用多一個好處是，你可以重構你的代碼爲了避免重複的方法secondThread和mainThread，例如這種方式（一個過去的事情，不要忘記清理你使用的資源，像browser.quit()關閉瀏覽器一旦你完成）：

import time 
from splinter import Browser 
from multiprocessing import Process 
import os 

os.environ['PATH'] = os.environ[ 
         'PATH'] + "path/to/geckodriver" + "path/to/firefox/binary" 

start_time = time.time() 

urlOne = 'http://pythoncarsecurity.com/Support/FAQ.aspx' 
urlTwo = 'http://pythoncarsecurity.com/Products/' 



def url_visitor(url): 
    print("url called: " + url) 
    browser = Browser('firefox') 
    print('STARTING REQUEST TO: ' + url + " at "+ str(time.time() - start_time)) 
    browser.visit(url) 
    print('END REQUEST TO: ' + url + " at "+ str(time.time() - start_time)) 

def main(): 
    p1 = Process(target=url_visitor, args=[urlTwo]) 
    p2 = Process(target=url_visitor, args=[urlOne]) 
    p1.start() 
    p2.start() 
    p1.join() #join processes to the main process to see the output 
    p2.join() 

if __name__=="__main__": 
    main() 

那給我們下面的輸出（定時w生病取決於您的系統雖然）：

url called: http://pythoncarsecurity.com/Support/FAQ.aspx 
url called: http://pythoncarsecurity.com/Products/ 
STARTING REQUEST TO: http://pythoncarsecurity.com/Support/FAQ.aspx at 10.763000011444092 
STARTING REQUEST TO: http://pythoncarsecurity.com/Products/ at 11.764999866485596 
END REQUEST TO: http://pythoncarsecurity.com/Support/FAQ.aspx at 16.20199990272522 
END REQUEST TO: http://pythoncarsecurity.com/Products/ at 16.625999927520752 

編輯：多線程和硒的問題是，一個瀏覽器實例不是線程安全的，我發現了繞過這個問題的唯一辦法就是獲得鎖在url_visitor，但是，在這種情況下，你失去了多線程的優勢。這就是爲什麼我相信，使用多個瀏覽器是更有益的（雖然我猜你有一些非常具體的要求），請參閱下面的代碼：

import time 
from splinter import Browser 
import threading 
from threading import Lock 
import os 

os.environ['PATH'] = os.environ[ 
         'PATH'] + "/path/to/chromedriver" 

start_time = time.time() 

urlOne = 'http://pythoncarsecurity.com/Support/FAQ.aspx' 
urlTwo = 'http://pythoncarsecurity.com/Products/' 
browser = Browser('chrome') 
lock = threading.Lock()#create a lock for the url_visitor method 

def init(): 
    browser.visit("https://www.google.fr") 
    driver = browser.driver 
    driver.execute_script("window.open('{0}', '_blank');") #create a new tab 
    tabs = driver.window_handles 


def url_visitor(url, tabs): 
    with lock: 
     if tabs != 0: 
      browser.driver.switch_to_window(browser.driver.window_handles[tabs]) 
     print("url called: " + url) 
     print('STARTING REQUEST TO: ' + url + " at "+ str(time.time() - start_time)) 
     browser.visit(url) 
     print('END REQUEST TO: ' + url + " at "+ str(time.time() - start_time)) 
     browser.quit() 


def main(): 
    p1 = threading.Thread(target=url_visitor, args=[urlTwo, 0]) 
    p2 = threading.Thread(target=url_visitor, args=[urlOne, 1]) 
    p1.start() 
    p2.start() 

if __name__=="__main__": 
    init() #create a browser with two tabs 
    main()