問題得到過去的HTML表單與Scrapy

Question

網址我試圖刮：https://www.uvic.ca/BAN1P/bwckschd.p_disp_dyn_sched

有3頁共，第一頁選擇長期，第二頁選擇主題，頁面與實際課程信息。

我遇到的問題是，一旦主題（）調用課程（）回調在被寫入文件response.body的HTML是主題頁面，而不是課程的HTML頁。我怎麼能告訴我發送正確的表格數據，以便我收到正確的答覆？

# term(): 
# Selects the school term to use. Clicks submit 

def term(self, response): 
    return scrapy.FormRequest.from_response(
    response, 
    formxpath="/html/body/div[3]/form", 
    formdata={ 
     "p_term" : "201705" }, 
    clickdata = { "type": "submit" }, 
    callback=self.subject 
    ) 

# subject(): 
# Selects the subject to query. Clicks submit 

def subject(self, response): 
    return scrapy.FormRequest.from_response(
    response, 
    formxpath="/html/body/div[3]/form", 
    formdata={ 
     "sel_subj" : "ART" }, 
    clickdata = { "type": "submit" }, 
    callback=self.courses 
    ) 

# courses(): 
# Currently just saves all the html on the page. 

def courses(self, response): 
    page = response.url.split("/")[-1] 
    filename = 'uvic-%s.html' % page 
    with open(filename, 'wb') as f: 
     f.write(response.body) 
    self.log('Saved file %s' % filename) 

調試輸出

2017-04-02 01:15:28 [scrapy.utils.log] INFO: Scrapy 1.3.3 started (bot: scrapy4uvic) 
2017-04-02 01:15:28 [scrapy.utils.log] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'scrapy4uvic.spiders', 'SPIDER_MODULES': ['scrapy4uvic.spiders'], 'ROBOTSTXT_OBEY': True, 'BOT_NAME': 'scrapy4uvic'} 
2017-04-02 01:15:28 [scrapy.middleware] INFO: Enabled extensions: 
['scrapy.extensions.logstats.LogStats', 
'scrapy.extensions.telnet.TelnetConsole', 
'scrapy.extensions.corestats.CoreStats'] 
2017-04-02 01:15:28 [scrapy.middleware] INFO: Enabled downloader middlewares: 
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware', 
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 
'scrapy.downloadermiddlewares.retry.RetryMiddleware', 
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 
'scrapy.downloadermiddlewares.stats.DownloaderStats'] 
2017-04-02 01:15:28 [scrapy.middleware] INFO: Enabled spider middlewares: 
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware', 
'scrapy.spidermiddlewares.referer.RefererMiddleware', 
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 
'scrapy.spidermiddlewares.depth.DepthMiddleware'] 
2017-04-02 01:15:28 [scrapy.middleware] INFO: Enabled item pipelines: 
[] 
2017-04-02 01:15:28 [scrapy.core.engine] INFO: Spider opened 
2017-04-02 01:15:28 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 
2017-04-02 01:15:28 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023 
2017-04-02 01:15:29 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.uvic.ca/robots.txt> (referer: None) 
2017-04-02 01:15:29 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.uvic.ca/BAN1P/bwckschd.p_disp_dyn_sched> (referer: None) 
2017-04-02 01:15:29 [scrapy.core.engine] DEBUG: Crawled (200) <POST https://www.uvic.ca/BAN1P/bwckgens.p_proc_term_date> (referer: https://www.uvic.ca/BAN1P/bwckschd.p_disp_dyn_sched) 
2017-04-02 01:15:29 [scrapy.core.engine] DEBUG: Crawled (200) <POST https://www.uvic.ca/BAN1P/bwckschd.p_get_crse_unsec> (referer: https://www.uvic.ca/BAN1P/bwckgens.p_proc_term_date) 
2017-04-02 01:15:30 [uvic] DEBUG: Saved file uvic-bwckschd.p_get_crse_unsec.html 
2017-04-02 01:15:30 [scrapy.core.engine] INFO: Closing spider (finished) 
2017-04-02 01:15:30 [scrapy.statscollectors] INFO: Dumping Scrapy stats: 
{'downloader/request_bytes': 2335, 
'downloader/request_count': 4, 
'downloader/request_method_count/GET': 2, 
'downloader/request_method_count/POST': 2, 
'downloader/response_bytes': 105499, 
'downloader/response_count': 4, 
'downloader/response_status_count/200': 4, 
'finish_reason': 'finished', 
'finish_time': datetime.datetime(2017, 4, 2, 8, 15, 30, 103536), 
'log_count/DEBUG': 6, 
'log_count/INFO': 7, 
'request_depth_max': 2, 
'response_received_count': 4, 
'scheduler/dequeued': 3, 
'scheduler/dequeued/memory': 3, 
'scheduler/enqueued': 3, 
'scheduler/enqueued/memory': 3, 
'start_time': datetime.datetime(2017, 4, 2, 8, 15, 28, 987034)} 
2017-04-02 01:15:30 [scrapy.core.engine] INFO: Spider closed (finished) 

Answer 1

好像你在subject()缺少某種形式的數據。

我設法得到它的工作：

formdata={ 
    "sel_subj": ["dummy", "ART"], 
} 

我如何去有關調試它。
首先，你不必保存到文件，你可以inspect_response抓取期間：

def courses(self, response): 
    from scrapy.shell import inspect_response 
    inspect_response(response, self) 

這將打開一個外殼，response和request對象，你甚至可以調用view(response)在瀏覽器中打開的HTML 。如果可用，它也將使用ipython或bpython shell，在下面的示例中，我使用ipython進行方便的格式化。

其次，我檢查了我的瀏覽器（Firefox）什麼樣的形式是，當我點擊該按鈕將它發送和複製它在變量bar殼和它相比，請求主體scrapy贊成派：

bar = '''term_in=201705&sel_subj=dummy&sel_day=dummy&sel_schd=dummy&sel_insm=dummy& 
     sel_camp=dummy&sel_levl=dummy              
     &sel_sess=dummy&sel_instr=dummy&sel_ptrm=dummy&sel_attr=dummy&sel_subj=ART&sel_crse 
     =&sel_title=&sel_schd                
     =%25&sel_insm=%25&sel_from_cred=&sel_to_cred=&sel_camp=%25&sel_levl=%25&sel_ptrm=%2 
     5&sel_instr=%25&begin_hh               
     =0&begin_mi=0&begin_ap=a&end_hh=0&end_mi=0&end_ap=a''' 
# split into arguments 
bar = sorted(bar.split('&')) 
# do the same with the request body that was sent by scrapy 
foo =sorted(request.body.split('&')) 
# now join these together and find the differences! 
zip(foo, bar) 
[('begin_ap=a', 'begin_ap=a'), 
('begin_hh=0', 'begin_hh\n=0'), 
('begin_mi=0', 'begin_mi=0'), 
('end_ap=a', 'end_ap=a'), 
('end_hh=0', 'end_hh=0'), 
('end_mi=0', 'end_mi=0'), 
('sel_attr=dummy', 'sel_attr=dummy'), 
('sel_camp=%25', 'sel_camp=%25'), 
('sel_camp=dummy', 'sel_camp=dummy'), 
('sel_crse=', 'sel_crse='), 
('sel_day=dummy', 'sel_day=dummy'), 
('sel_from_cred=', 'sel_from_cred='), 
('sel_insm=%25', 'sel_insm=%25'), 
('sel_insm=dummy', 'sel_insm=dummy'), 
('sel_instr=%25', 'sel_instr=%25'), 
('sel_instr=dummy', 'sel_instr=dummy'), 
('sel_levl=%25', 'sel_levl=%25'), 
('sel_levl=dummy', 'sel_levl=dummy\n'), 
('sel_ptrm=%25', 'sel_ptrm=%25'), 
('sel_ptrm=dummy', 'sel_ptrm=dummy'), 
('sel_schd=%25', 'sel_schd\n=%25'), 
('sel_schd=dummy', 'sel_schd=dummy'), 
('sel_sess=dummy', 'sel_sess=dummy'), 
('sel_subj=ART', 'sel_subj=ART'), 
('sel_title=', 'sel_subj=dummy'), 
('sel_to_cred=', 'sel_title='), 
('term_in=201705', 'sel_to_cred=')] 

正如你可以看到你在sel_subj中缺少「虛擬」，'term_in'在那裏不應該但它看起來沒有效果:)