使用其他網址登錄後對網頁進行爬網

Question

我對scrapy的知識有限。有了這段代碼，我可以在特定的論壇上進行登錄。現在我需要登錄之後另設網址：

https://forum.xxx.com/threads/topic-name/page-300

我想與自動化300-360之間的頁面範圍內的爬行。具體而言，所有元素messageText

我該怎麼做？

import scrapy 

class LoginSpider(scrapy.Spider): 
    name = 'xxx.com' 
    start_urls = ['https://forum.xxx.com/login/login'] 

    def parse(self, response): 
     return scrapy.FormRequest.from_response(
      response, 
      formdata={'login': 'xxx', 'register': '0', 'password': 'xxxxx', 'cookie_check': '0'}, 
      callback=self.after_login 
     ) 

    def after_login(self, response): 
     # check login succeed before going on 
     if "authentication failed" in response.body: 
      self.logger.error("Login failed") 
      return 

     # continue scraping with authenticated session... 

Answer 1

一旦登錄，你只需要得到儘可能多的要求：

from scrapy import Request 
def after_login(self, response): 
    # check login succeed before going on 
    if "authentication failed" in response.body: 
     self.logger.error("Login failed") 
     return 
    for i in range(300, 360): 
     url = 'https://forum.xxx.com/threads/topic-name/page-{}'.format(i) 
     yield Request(url, self.parse_page) 


def parse_page(self, response): 
    # parse page here