無法使用R XML包將文本從刮出的HTML頁面中拉出

Question

我試圖提取紐約時報電影評論的正文，以便對它們進行一些語義分析。不幸的是，我的HTML + R + XML包技能還不足以完成工作。我可以使用NYT電影API的XML輸出來獲取電影細節，但我無法弄清楚如何使用文章API或直接網頁抓取，以便了解評論的正文。

工作代碼爲電影細節：

library(RCurl) 
nyt.x.url<-'http://api.nytimes.com/svc/movies/v2/reviews/search.xml?query=The+Hangover&api-key=YOUR-OWN-FREE-API-KEY-GOES-HERE' 
nyt.x.out<-getURLContent(nyt.x.url,curl=getCurlHandle()) 
library(XML) 
a <- xmlTreeParse(nyt.x.url) 
r <- xmlRoot(a) 
# need to put the separate list items together into a mtrix, before they can be turned to a dataframe 
nyt.df <- as.data.frame(stringsAsFactors=FALSE, 
        matrix(c(as.character(r[[4]][[1]][[1]][[1]])[6], # display name 
          as.character(r[[4]][[1]][[3]][[1]])[6], # rating - agrees with rotten tomatoes, but not imdb 
          as.character(r[[4]][[1]][[4]][[1]])[6], # is it a critics pick 
          as.character(r[[4]][[1]][[5]][[1]])[6], # is it a thousand best 
          as.character(r[[4]][[1]][[11]][[1]])[6], # opening date 
          as.character(r[[4]][[1]][[15]][[1]][[1]])[6]), # this is really the URL.... 
          nrow=1, 
          ncol=6)) 

# now apply the right names 
colnames(nyt.df) <- c("Title","MPAA-Rating", "Critics.Pick", "Thousand.Best", "Release.Date", "Article.URL") 

我會再使用的電影細節這個數據幀，抓取的審查網頁，並試圖抓住的評論文章：

nyt.review.out<-getURLContent(as.character(nyt.df[6]),curl=getCurlHandle()) 
a2 <- htmlTreeParse(nyt.review.url) 

但我無法弄清楚如何得到評論的全文。當我嘗試使用json API進行文章時，我遇到了同樣的問題（url調用api在下面）

nyt.review.url < - 'http://api.nytimes.com/svc/搜索/ V1 /條？格式= JSON &查詢=審查+在+宿醉& BEGIN_DATE = 20090605 & END_DATE = 20090606 & API密鑰= YOUR-OTHER-FREE-API-KEY-GOES-HERE」

任何幫助非常感謝，但您需要註冊您自己的API密鑰（我從代碼中刪除了我的密碼）

Answer 1

I 認爲 this做你想做的事。可能有一種方法可以直接從API執行您想要的操作，但我沒有對此進行調查。

# load package 
library(XML) 

# grabs text from new york times movie page. 
grab_nyt_text <- function(u) { 
    doc <- htmlParse(u) 
    txt <- xpathSApply(doc, '//div[@class="articleBody"]//p', xmlValue) 
    txt <- paste(txt, collapse = "\n") 
    free(doc) 
    return(txt) 
} 


###--- Main ---### 

# Step 1: api URL 
nyt.x.url <- 'http://api.nytimes.com/svc/movies/v2/reviews/search.xml?query=The+Hangover&api-key=YOUR-OWN-FREE-API-KEY-GOES-HERE' 

# Step 2: Parse XML of webpage pointed to by URL 
doc <- xmlParse(nyt.x.url) 

# Step 3: Parse XML and extract some values using XPath expressions 
df <- data.frame(display.title = xpathSApply(doc, "//results//display_title", xmlValue), 
       critics.pick = xpathSApply(doc, "//results//critics_pick", xmlValue), 
       thousand.best = xpathSApply(doc, "//results//thousand_best", xmlValue), 
       opening.date = xpathSApply(doc, "//results//opening_date", xmlValue), 
       url = xpathSApply(doc, "//results//link[@type='article']/url", xmlValue), 
       stringsAsFactors=FALSE) 

df 
#   display.title critics.pick thousand.best opening.date                       url 
#1   The Hangover   0    0 2009-06-05          http://movies.nytimes.com/2009/06/05/movies/05hang.html 
#2 The Hangover Part II   0    0 2011-05-26 http://movies.nytimes.com/2011/05/26/movies/the-hangover-part-ii-3-men-and-a-monkey-baby.html 

# Step 4: clean up - remove doc from memory 
free(doc) 

# Step 5: crawl article links and grab text 
df$text <- sapply(df$url, grab_nyt_text) 

# Step 6: inspect txt 
cat(df$text[1]) 

HTH

託尼Breyal

附：還有一個R包http://www.omegahat.org/RNYTimes，但目前網站已關閉，所以我不知道它的能力。