使用JavaScript從R刮取頁面

Question

我是新來的網絡抓取在R和最近碰到一個問題，參考JavaScript的網站。我正在嘗試從下面的網頁中抓取數據，但未成功。我相信JavaScript鏈接阻止我訪問表。結果帶有函數「readHTMLTable」的R包「XML」變爲空。

library(XML) 
library(RCurl) 
url <- "http://votingrights.news21.com/interactive/movement-voter-id/index.html" 
tabs <- getURL(url) 
tabs <- htmlParse(url) 
tabs <- readHTMLTable(tabs, stringsAsFactors = FALSE) 

如何訪問JavaScript鏈接以獲取數據？或者這甚至有可能？當使用數據的直接鏈接（下面）和R包「rjson」時，我仍然無法讀取數據。

library("rjson") 
json_file <- "http://votingrights.news21.com/static/interactives/movement/data/fulldata.js" 
lines <- readLines(json_file) 
json_data <- fromJSON(lines, collapse="") 

Answer 1

您引用的文件是包含JSON而非JSON的JavaScript文件。在這種情況下，您可以手動擦洗內容來獲取數據：

library("rjson") 
json_file <- "http://votingrights.news21.com/static/interactives/movement/data/fulldata.js" 
lines <- readLines(json_file) 
lines[1] <- sub(".* = (.*)", "\\1", lines[1]) 
lines[length(lines)] <- sub(";", "", lines[length(lines)]) 
json_data <- fromJSON(paste(lines, collapse="\n")) 
> head(json_data[[1]][[1]]) 
$state 
[1] "Alabama" 

$bill 
[1] "HB 19" 

$category 
[1] "Strict photo ID" 

$introduced 
[1] "Mar 1, 2011" 

$house 
[1] "Yes" 

$senate 
[1] "Yes" 

如果你想在網頁上使用JavaScript數據交互，你可以使用Selenium：

library(RSelenium) 
appURL <- "http://votingrights.news21.com/static/interactives/movement/index.html" 
pJS <- phantom() 
remDr <- remoteDriver(browserName = "phantom") 
remDr$open() 
remDr$navigate(appURL) 
fullData <- remDr$executeScript("return fullData;") 
pJS$stop() 
> head(fullData[[1]][[1]]) 
$state 
[1] "Alabama" 

$bill 
[1] "HB 19" 

$category 
[1] "Strict photo ID" 

$introduced 
[1] "Mar 1, 2011" 

$house 
[1] "Yes" 

$senate 
[1] "Yes"