Scrapy登錄認證不起作用

Question

我剛開始玩scrapy。我正在嘗試抓取需要登錄的網站。我爲github工作得很好。我找到了表單ID，添加了必填字段，並按計劃繼續進行。

但是，當我在investopedia網站上嘗試相同的時候，我陷入了困境。我附上代碼。

class Investo_spider(InitSpider): 
    name = 'investo_spider' 
    allowed_domains = ['investopedia.com'] 
    login_page = 'http://www.investopedia.com/accounts/login.aspx' 
    start_urls = ['http://www.investopedia.com'] 

    def init_request(self): 
     return Request(url=self.login_page, callback=self.login) 

    def login(self, response): 
      return FormRequest.from_response(response, 
           formdata={'email': 'mymail','password': 'mypass'}, 
           callback=self.check_login_response) 

    def check_login_response(self, response): 
     if "myname" in response.body: 
      self.log("Successfully logged in. Let's start crawling!") 
      self.initialized() 
     else: 
      self.log("Login was unsuccessful") 


    def parse_item(self, response): 
     print 'I got in here, finally!!!!' 
     pass 

我曾嘗試添加formnumber = 0，clickdata = { 'NR'：0}和改變方法（POST或GET）儘管默認已經選擇合適的形式和點擊。

令人驚訝的是，我得到它在機械化瀏覽器上工作，使用相同的參數。我可以將html轉換爲scrapy可以處理的HtmlResponse對象。

br = mechanize.Browser() 
br.open("http://www.investopedia.com/accounts/login.aspx") 
br.select_form(nr=0) 
br.form["email"] = 'mymail' 
br.form["password"] = 'mypass' 
br.submit() 
br.open('http://www.investopedia.com') 
response = HtmlResponse(url="some_url"),body=br.response().read()) 

然而，這將意味着我將不得不隨身攜帶的機械化瀏覽器，我以爲是不是最好的解決方案。我想我可能會錯過一些東西。我真的很感激你對此的意見。謝謝！

Answer 1

你將不得不處理重定向。這將爲你工作。

class Investo_spider(scrapy.Spider): 
name = 'investo_spider' 
allowed_domains = ['investopedia.com'] 
login_page = 'http://www.investopedia.com/accounts/login.aspx' 
start_urls = ['http://www.investopedia.com'] 

def init_request(self): 
    return scrapy.Request(url=self.login_page, callback=self.login) 

def parse(self, response): 
    return scrapy.FormRequest('http://www.investopedia.com/accounts/login.aspx', 
            formdata={'email': 'you_email', 'password': 'your_password', 
               'form_build_id': 'form - v14V92zFkSSVFSerfvWyH1WEUoxrV2khjfhAETJZydk', 
               'form_id': 'account_api_form', 
               'op': 'Sign in' 
               }, 
            meta = {'dont_redirect': True, 'handle_httpstatus_list':[302]}, 
            callback=self.check_login_response) 

def check_login_response(self, response): 
    return scrapy.Request('http://www.investopedia.com/accounts/manageprofile.aspx', self.validate_login) 

def validate_login(self, response): 
    if "myname" in response.body: 
     self.log("Successfully logged in. Let's start crawling!") 
     self.initialized() 
    else: 
     self.log("Login was unsuccessful") 

def parse_item(self, response): 
    print 'I got in here, finally!!!!' 
    pass