從R網頁中用JavaScript抓取鏈接

我試圖從http://www.childrenshospital.org/directory?state=%7B%22showLandingContent%22%3Afalse%2C%22model%22%3A%7B%22search_specialist%22%3Afalse%2C%22search_type%22%3A%5B%22directoryphysician%22%2C%22directorynurse%22%5D%7D%2C%22customModel%22%3A%7B%22nurses%22%3Atrue%7D%7D颳去個別提供商的網址。從R網頁中用JavaScript抓取鏈接

我查看了頁面源和確定的感興趣的URL。例如，我想刮「http://www.childrenshospital.org/doctors/mirna-aeschlimann」形成如下節點

<a data-layer-event="searchClick" data-bind="attr: {href: model.Url}" href="http://www.childrenshospital.org/doctors/mirna-aeschlimann"><!--ko text: model.FirstName-->Mirna<!--/ko--><!--ko text: ' ' + model.LastName--> Aeschlimann<!--/ko--><!--ko if: model.Suffix-->, <!--ko text: model.Suffix-->MD<!--/ko--><!--/ko--></a>

我嘗試下面的代碼。但是，由於某些原因，它沒有返回上面的節點。

base_html <- "http://www.childrenshospital.org/directory?state=%7B%22showLandingContent%22%3Afalse%2C%22model%22%3A%7B%22search_specialist%22%3Afalse%2C%22search_type%22%3A%5B%22directoryphysician%22%2C%22directorynurse%22%5D%7D%2C%22customModel%22%3A%7B%22nurses%22%3Atrue%7D%7D" 
doc <- htmlTreeParse(base_html, useInternal = TRUE)

任何幫助將不勝感激。請讓我知道是否應該提供更多信息。

來源

2017-01-03 M Lin

查看'rvest'包[這裏]（https://github.com/hadley/rvest）。 – Jeff

，你在調用'htmlTreeParse'後發現你沒有得到目標節點？ – hrbrmstr

您是否試過針對網站製作的XHR請求獲取數據？

library(httr) 
library(purrr) 
library(xml2) 
library(dplyr) 
library(jsonlite) 

map_df(1:17, function(i) { 

    POST("http://www.childrenshospital.org/searchdirectory.ajax", 
     body = list(search_query = "", 
        search_specialties = "", 
        search_languages = "", 
        search_gender = "", search_departments = "", 
        search_programs = "", search_userlocation = "", 
        search_radius = "10", search_pcp = "true", 
        search_specialist = "false", 
        search_type = "directorynurse|directoryphysician", 
        search_letter = "", search_querygroup = "dirnametext", 
        search_page = "10"), 
     encode = "form") -> res 

    content(res, as="text") %>% 
    fromJSON() %>% 
    .$Records %>% 
    mutate(Address=xml2::xml_text(xml2::read_html(paste0("<x>", Address, "</x>")))) %>% 
    tbl_df() 

}) -> tmp_df 

glimpse(tmp_df) 
## Observations: 408 
## Variables: 21 
## $ ID    <chr> "{E8ECAF3B-B49C-4CD8-AB16-6CE63F0379C0}", "{1E1... 
## $ FirstName  <chr> "Jonathan", "Barbara", "Mark", "Maura", "Sarah"... 
## $ LastName  <chr> "Schwab", "Seagle", "Shapira", "Shea", "Sheldon... 
## $ Image   <chr> "/~/media/directory/physicians/schwab_jonathan.... 
## $ Suffix   <chr> "MD", "MD", "MD", "MD", "MD", "MD", "MD", "MD",... 
## $ Url   <chr> "http://www.childrenshospital.org/doctors/jonat... 
## $ Gender   <chr> "male", "female", "male", "female", "female", "... 
## $ Latitude  <chr> "42.3344382", "42.326435", "41.559642", "42.423... 
## $ Longitude  <chr> "-72.6618324", "-71.149499", "-70.939315", "-71... 
## $ Address  <chr> "{\"practice_name\":\"Northampton Area Pediatri... 
## $ Distance  <chr> "", "", "", "", "", "", "", "", "", "", "", "",... 
## $ OtherLocations <chr> "", "", "", "", "", "", "", "", "", "", "Westwo... 
## $ AcademicTitle <chr> "", "", "", "", "", "", "", "", "", "", "", "",... 
## $ HospitalTitle <chr> "Pediatrician", "Pediatrician", "Pediatrician",... 
## $ Specialties <chr> "Pediatrics", "General Pediatrics, Pediatrics, ... 
## $ Departments <chr> "", "General Pediatrics", "General Pediatrics",... 
## $ Languages  <chr> "", "English", "", "English", "English", "", "E... 
## $ PPOCLink  <chr> "http://www.childrenshospital.org/patient-resou... 
## $ Gallery  <chr> "", "", "", "", "", "", "", "", "", "", "", "",... 
## $ Phone   <chr> "(413) 584-8700", "(617) 731-0200", "(508) 996-... 
## $ Fax   <chr> "(413) 584-1714", "(617) 731-0289", "(508) 992-...

來源

2017-01-03 02:49:29 hrbrmstr

從R網頁中用JavaScript抓取鏈接

回答

相關問題