使用R從aspx網站上刮取

Question

我正在嘗試使用R來抓取網站上的數據。

1. 我想去貫穿於各個環節下頁： http://capitol.hawaii.gov/advreports/advreport.aspx?year=2013&report=deadline&rpt_type=&measuretype=hb&title=House票據

2. 與當前狀態顯示「發送到省長」只選項目。例如，http://capitol.hawaii.gov/measure_indiv.aspx?billtype=HB&billnumber=17&year=2013

3. 然後在STATUS TEXT中爲以下子句「通過最終讀數」取消單元格。例如：通過SD 2中與代表Fale，Jordan，Tsuji投票贊成並保留的最終解讀;代表（s）Cabanilla，森川，大城，Tokioka投票否（4）並且沒有原諒（0）。

我一直在使用前面的例子中使用包Rcurl和XML（以R）試過了，但我不知道如何正確地使用它們的aspx網站。所以我喜歡的是：1.關於如何構建這樣的代碼的一些建議。 2.建議如何學習執行此類任務所需的知識。

感謝您的幫助，

湯姆

Answer 1

require(httr) 
require(XML) 

basePage <- "http://capitol.hawaii.gov" 

h <- handle(basePage) 

GET(handle = h) 

res <- GET(handle = h, path = "/advreports/advreport.aspx?year=2013&report=deadline&rpt_type=&measuretype=hb&title=House") 

# parse content for "Transmitted to Governor" text 
resXML <- htmlParse(content(res, as = "text")) 
resTable <- getNodeSet(resXML, '//*/table[@id ="GridViewReports"]/tr/td[3]') 
appRows <-sapply(resTable, xmlValue) 
include <- grepl("Transmitted to Governor", appRows) 
resUrls <- xpathSApply(resXML, '//*/table[@id ="GridViewReports"]/tr/td[2]//@href') 

appUrls <- resUrls[include] 

# look at just the first 

res <- GET(handle = h, path = appUrls[1]) 

resXML <- htmlParse(content(res, as = "text")) 


xpathSApply(resXML, '//*[text()[contains(.,"Passed Final Reading")]]', xmlValue) 

[1] "Passed Final Reading as amended in SD 2 with Representative(s) Fale, Jordan, 
Tsuji voting aye with reservations; Representative(s) Cabanilla, Morikawa, Oshiro, 
Tokioka voting no (4) and none excused (0)." 

讓包httr處理通過建立handle所有的後臺工作。

如果你想運行在所有92個鏈接：

# get all the links returned as a list (will take sometime) 
# print statement included for sanity 
res <- lapply(appUrls, function(x){print(sprintf("Got url no. %d",which(appUrls%in%x))); 
            GET(handle = h, path = x)}) 
resXML <- lapply(res, function(x){htmlParse(content(x, as = "text"))}) 
appString <- sapply(resXML, function(x){ 
        xpathSApply(x, '//*[text()[contains(.,"Passed Final Reading")]]', xmlValue) 
         }) 


head(appString) 

> head(appString) 
$href 
[1] "Passed Final Reading as amended in SD 2 with Representative(s) Fale, Jordan, Tsuji voting aye with reservations; Representative(s) Cabanilla, Morikawa, Oshiro, Tokioka voting no (4) and none excused (0)." 

$href 
[1] "Passed Final Reading, as amended (CD 1). 25 Aye(s); Aye(s) with reservations: none . 0 No(es): none. 0 Excused: none."             
[2] "Passed Final Reading as amended in CD 1 with Representative(s) Cullen, Har voting aye with reservations; Representative(s) McDermott voting no (1) and none excused (0)." 

$href 
[1] "Passed Final Reading, as amended (CD 1). 25 Aye(s); Aye(s) with reservations: none . 0 No(es): none. 0 Excused: none."         
[2] "Passed Final Reading as amended in CD 1 with none voting aye with reservations; Representative(s) Hashem, McDermott voting no (2) and none excused (0)." 

$href 
[1] "Passed Final Reading, as amended (CD 1). 24 Aye(s); Aye(s) with reservations: none . 0 No(es): none. 1 Excused: Ige."      
[2] "Passed Final Reading as amended in CD 1 with none voting aye with reservations; none voting no (0) and Representative(s) Say excused (1)." 

$href 
[1] "Passed Final Reading, as amended (CD 1). 25 Aye(s); Aye(s) with reservations: none . 0 No(es): none. 0 Excused: none."       
[2] "Passed Final Reading as amended in CD 1 with Representative(s) Johanson voting aye with reservations; none voting no (0) and none excused (0)." 

$href 
[1] "Passed Final Reading, as amended (CD 1). 25 Aye(s); Aye(s) with reservations: none . 0 No(es): none. 0 Excused: none." 
[2] "Passed Final Reading as amended in CD 1 with none voting aye with reservations; none voting no (0) and none excused (0)."