2016-11-06 86 views
1

我試圖刮掉從以下站點找到該表從網站刮網表:,我已經尋找幾種不同的方法解答https://finance.yahoo.com/gainers?e=us怎麼做,使用R

但是湊在這裏從現場表和沒有任何方法爲我工作。

我曾嘗試:

library(xml2) 
url <- "https://finance.yahoo.com/gainers?e=us" 
tbl <- read_html(url) 

也:

library(XML) 
url <- "https://finance.yahoo.com/gainers?e=us" 
tbl <- readHTMLList(url) 

和其它軟件包如rvest但我不能讓表顯示!

回答

4

它們將該數據存儲在頁面本身的JavaScript塊中。你有兩個選擇:要麼使用RSelenium,然後對付真正複雜的表格,你必須放鬆或從V8包得到一些幫助後,要進行一些字符串手術的一些JSON的mangling相結合:

library(V8) 
library(xml2) 
library(stringi) 
library(jsonlite) 
library(ndjson) 

pg <- read_html("https://finance.yahoo.com/gainers?e=us") 

這下位從顯着的<script>標記中提取數據。我正在按位置做這件事,這意味着如果Yahoo!曾經改變它的格式(比如,如果VZ曾經完成購買,那麼插入無用的Verizon無線廣告通過javascript)。你可以改變這個尋找文字指標,但這會導致第二個問題...

「刮」這種方式的第二個問題是,如果JavaScript的數據有變化,這也將打破。 但是對於基於HTML的抓取也是如此。此方法不需要啓動用於導航Web服務器的Web服務器和控制服務器。反正...

html_nodes(pg, "script")[13] %>% 
    html_text() %>% 
    stri_replace_first_fixed("(function (root) {", "var root = { App : {}};\n") %>% 
    stri_replace_last_fixed("}(this));", "") -> js 

我們不得不這樣做,因爲^^在<script>標籤javascript代碼期待是在瀏覽器(我們沒有)。現在,我們評估的是通過V8 JavaScript的大塊,並提取主要數據元素:

ctx <- v8() 
ctx$eval(JS(js)) 
root <- ctx$get("root", flatten=TRUE) 

中的數據元素包含所有該頁面的數據(這實在是一個單頁的應用程序)的。因此,我們必須找到我們關心的數據,實在是遠了嵌套的JavaScript漏洞:

quotes <- root$App$main$context$dispatcher$stores$`QuoteDataStore-Immutable`$quoteData 

有許多方法把這一嵌套表數據轉換成一個漂亮的,長方形的數據幀。下面的方法選擇「將其轉換爲JSON,然後將其從JSON中以完全」平坦「的格式返回。隨意使用其他方法,您可以在SO上找到它。

代碼首先告訴R忽略非-stocks(因爲你,想必,只想要在美麗的表頁面上的引號):

discard(names(quotes), ~grepl("[\\^\\=]", .)) %>% 
    map_df(~ndjson::flatten(toJSON(quotes[[.]]))) %>% 
    glimpse() 

## Observations: 30 
## Variables: 53 
## $ averageDailyVolume3Month.fmt.0  <chr> "437,939", "541,801", "1.033M", "992,278", "1.40... 
## $ averageDailyVolume3Month.longFmt.0 <chr> "437,939", "541,801", "1,033,453", "992,278", "1... 
## $ averageDailyVolume3Month.raw.0  <dbl> 437939, 541801, 1033453, 992278, 1402175, 537906... 
## $ exchange.0       <chr> "NYQ", "NMS", "NYQ", "NGM", "NGM", "NMS", "NGM",... 
## $ exchangeTimezoneName.0    <chr> "America/New_York", "America/New_York", "America... 
## $ exchangeTimezoneShortName.0   <chr> "EST", "EST", "EST", "EST", "EST", "EST", "EST",... 
## $ fiftyTwoWeekHigh.fmt.0    <chr> "15.36", "46.12", "15.82", "7.64", "5.97", "9.86... 
## $ fiftyTwoWeekHigh.raw.0    <dbl> 15.360, 46.120, 15.820, 7.640, 5.970, 9.860, 5.0... 
## $ fiftyTwoWeekHighChange.fmt.0  <chr> "-6.96", "-17.72", "-4.19", "-2.83", "-4.29", "-... 
## $ fiftyTwoWeekHighChange.raw.0  <dbl> -6.960, -17.720, -4.190, -2.830, -4.290, -3.640,... 
## $ fiftyTwoWeekHighChangePercent.fmt.0 <chr> "-45.31%", "-38.42%", "-26.49%", "-37.04%", "-71... 
## $ fiftyTwoWeekHighChangePercent.raw.0 <dbl> -0.4531, -0.3842, -0.2649, -0.3704, -0.7186, -0.... 
## $ fiftyTwoWeekLow.fmt.0    <chr> "6.59", "23.75", "9.46", "4.22", "0.25", "4.97",... 
## $ fiftyTwoWeekLow.raw.0    <dbl> 6.590, 23.750, 9.460, 4.220, 0.251, 4.970, 0.970... 
## $ fiftyTwoWeekLowChange.fmt.0   <chr> "1.81", "4.65", "2.17", "0.59", "1.43", "1.25", ... 
## $ fiftyTwoWeekLowChange.raw.0   <dbl> 1.810, 4.650, 2.170, 0.590, 1.429, 1.250, 0.190,... 
## $ fiftyTwoWeekLowChangePercent.fmt.0 <chr> "27.47%", "19.58%", "22.94%", "13.98%", "569.32%... 
## $ fiftyTwoWeekLowChangePercent.raw.0 <dbl> 0.2747, 0.1958, 0.2294, 0.1398, 5.6932, 0.2515, ... 
## $ fullExchangeName.0     <chr> "NYSE", "NasdaqGS", "NYSE", "NasdaqGM", "NasdaqG... 
## $ gmtOffSetMilliseconds.0    <dbl> -1.8e+07, -1.8e+07, -1.8e+07, -1.8e+07, -1.8e+07... 
## $ invalid.0       <lgl> FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE,... 
## $ isLoading.0       <lgl> FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE,... 
## $ language.0       <chr> "en-US", "en-US", "en-US", "en-US", "en-US", "en... 
## $ longName.0       <chr> "Bankrate, Inc.", "Air Methods Corp.", "Bristow ... 
## $ market.0       <chr> "us_market", "us_market", "us_market", "us_marke... 
## $ marketCap.fmt.0      <chr> "758.268M", "1.082B", "407.786M", "494.03M", "33... 
## $ marketCap.longFmt.0     <chr> "758,267,968", "1,081,625,344", "407,786,176", "... 
## $ marketCap.raw.0      <dbl> 758267968, 1081625344, 407786176, 494030272, 338... 
## $ marketState.0      <chr> "CLOSED", "CLOSED", "CLOSED", "CLOSED", "CLOSED"... 
## $ messageBoardId.0     <chr> "finmb_30061", "finmb_24494", "finmb_292980", "f... 
## $ quoteType.0       <chr> "EQUITY", "EQUITY", "EQUITY", "EQUITY", "EQUITY"... 
## $ regularMarketChange.fmt.0   <chr> "1.20", "3.65", "1.44", "0.59", "0.27", "0.80", ... 
## $ regularMarketChange.raw.0   <dbl> 1.20, 3.65, 1.44, 0.59, 0.27, 0.80, 0.15, 0.04, ... 
## $ regularMarketChangePercent.fmt.0 <chr> "16.67%", "14.75%", "14.13%", "13.98%", "19.15%"... 
## $ regularMarketChangePercent.raw.0 <dbl> 16.6667, 14.7475, 14.1315, 13.9810, 19.1489, 14.... 
## $ regularMarketDayHigh.fmt.0   <chr> "9.90", "30.25", "12.60", "5.29", "1.85", "6.30"... 
## $ regularMarketDayHigh.raw.0   <dbl> 9.9000, 30.2500, 12.6000, 5.2900, 1.8500, 6.2950... 
## $ regularMarketDayLow.fmt.0   <chr> "7.90", "25.30", "9.46", "4.30", "1.33", "5.79",... 
## $ regularMarketDayLow.raw.0   <dbl> 7.9000, 25.3000, 9.4600, 4.3000, 1.3300, 5.7900,... 
## $ regularMarketPrice.fmt.0   <chr> "8.40", "28.40", "11.63", "4.81", "1.68", "6.22"... 
## $ regularMarketPrice.raw.0   <dbl> 8.40, 28.40, 11.63, 4.81, 1.68, 6.22, 1.16, 1.20... 
## $ regularMarketTime.fmt.0    <chr> "4:02PM EDT", "4:00PM EDT", "4:00PM EDT", "4:00P... 
## $ regularMarketTime.raw.0    <dbl> 1478289722, 1478289600, 1478289614, 1478289600, ... 
## $ regularMarketVolume.fmt.0   <chr> "2.383M", "3.32M", "3.411M", "4.567M", "4.232M",... 
## $ regularMarketVolume.longFmt.0  <chr> "2,382,856", "3,320,072", "3,411,052", "4,566,79... 
## $ regularMarketVolume.raw.0   <dbl> 2382856, 3320072, 3411052, 4566790, 4232135, 185... 
## $ sharesOutstanding.fmt.0    <chr> "90.27M", "38.085M", "35.063M", "102.709M", "20.... 
## $ sharesOutstanding.longFmt.0   <chr> "90,270,000", "38,085,400", "35,063,300", "102,7... 
## $ sharesOutstanding.raw.0    <dbl> 90270000, 38085400, 35063300, 102709000, 2016800... 
## $ shortName.0       <chr> "Bankrate, Inc. Common Stock", "Air Methods Corp... 
## $ sourceInterval.0     <dbl> 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, ... 
## $ symbol.0       <chr> "RATE", "AIRM", "BRS", "CERS", "EBIO", "ELNK", "... 
## $ uuid.0        <chr> "79521cde-a3ef-383f-917d-31c49f9082f5", "f0432c1... 

我覺得你還是有一些數據轉換(取決於你以後)做的,但你有數據,你正在尋找。

+0

非常感謝你! – Rime