Python的scrapy工作（只有一半的時間）

我創建了一個python scrapy項目來提取一些谷歌航班的價格。Python的scrapy工作（只有一半的時間）

我配置中間件使用PhantomJS而不是普通的瀏覽器。

class JSMiddleware(object): 
    def process_request(self, request, spider): 
     driver = webdriver.PhantomJS() 
     try: 
      driver.get(request.url) 
      time.sleep(1.5) 
     except e: 
      raise ValueError("request url failed - \n url: {},\n error: 
        {}").format(request.url, e) 
     body = driver.page_source 
     #encoding='utf-8' - add to html response if necessary 
     return HtmlResponse(driver.current_url, body=body,encoding='utf-8', 
       request=request)

在settings.py我添加：

DOWNLOADER_MIDDLEWARES = { 
# key path intermediate class, order value of middleware 
'scraper_module.middlewares.middleware.JSMiddleware' : 543 , 
# prohibit the built-in middleware 
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware' : None , } `

我還創建以下蜘蛛類：

import scrapy 
from scrapy import Selector 

class Gspider(scrapy.Spider): 
    name = "google_spider" 

    def __init__(self): 
     self.start_urls = ["https://www.google.pt/flights/#search;f=LIS;t=POR;d=2017-06-18;r=2017-06-22"] 
     self.prices = [] 
     self.links = [] 

    def clean_price(self, part): 
     #part received as a list 
     #the encoding is utf-8 
     part = part[0] 
     part = part.encode('utf-8') 
     part = filter(str.isdigit, part) 
     return part 

    def clean_link(self, part): 
     part = part[0] 
     part = part.encode('utf-8') 
     return part 

    def get_part(self, var_holder, response, marker, inner_marker, amount = 1): 
     selector = Selector(response) 
     divs = selector.css(marker) 
     for n, div in enumerate(divs): 
      if n < amount: 
       part = div.css(inner_marker).extract() 
       if inner_marker == '::text': 
        part = self.clean_price(part) 
       else: 
        part = self.clean_link(part) 
       var_holder.append(part) 
      else: 
       break 
     return var_holder 

    def parse(self, response): 
     prices, links = [], [] 
     prices = self.get_part(prices, response, 'div.OMOBOQD-d-Ab', '::text') 
     print prices 
     links = self.get_part(links, response, 'a.OMOBOQD-d-X', 'a::attr(href)') 
     print links

的問題是，我運行在殼中的代碼，並大約一半的時間我成功地獲得了要求的價格和鏈接，但另一半時間，應該包含提取數據的最終矢量是空的。

我在執行過程中沒有收到任何錯誤。

有沒有人有任何想法，爲什麼發生這種情況？這裏是命令行日誌：