它們將該數據存儲在頁面本身的JavaScript塊中。你有兩個選擇:要麼使用RSelenium,然後對付真正複雜的表格,你必須放鬆或從V8
包得到一些幫助後,要進行一些字符串手術的一些JSON的mangling相結合:
library(V8)
library(xml2)
library(stringi)
library(jsonlite)
library(ndjson)
pg <- read_html("https://finance.yahoo.com/gainers?e=us")
這下位從顯着的<script>
標記中提取數據。我正在按位置做這件事,這意味着如果Yahoo!曾經改變它的格式(比如,如果VZ曾經完成購買,那麼插入無用的Verizon無線廣告通過javascript)。你可以改變這個尋找文字指標,但這會導致第二個問題...
「刮」這種方式的第二個問題是,如果JavaScript的數據有變化,這也將打破。 但是對於基於HTML的抓取也是如此。此方法不需要啓動用於導航Web服務器的Web服務器和控制服務器。反正...
html_nodes(pg, "script")[13] %>%
html_text() %>%
stri_replace_first_fixed("(function (root) {", "var root = { App : {}};\n") %>%
stri_replace_last_fixed("}(this));", "") -> js
我們不得不這樣做,因爲^^在<script>
標籤javascript代碼期待是在瀏覽器(我們沒有)。現在,我們評估的是通過V8 JavaScript的大塊,並提取主要數據元素:
ctx <- v8()
ctx$eval(JS(js))
root <- ctx$get("root", flatten=TRUE)
中的數據元素包含所有該頁面的數據(這實在是一個單頁的應用程序)的。因此,我們必須找到我們關心的數據,實在是遠了嵌套的JavaScript漏洞:
quotes <- root$App$main$context$dispatcher$stores$`QuoteDataStore-Immutable`$quoteData
有許多方法把這一嵌套表數據轉換成一個漂亮的,長方形的數據幀。下面的方法選擇「將其轉換爲JSON,然後將其從JSON中以完全」平坦「的格式返回。隨意使用其他方法,您可以在SO上找到它。
代碼首先告訴R忽略非-stocks(因爲你,想必,只想要在美麗的表頁面上的引號):
discard(names(quotes), ~grepl("[\\^\\=]", .)) %>%
map_df(~ndjson::flatten(toJSON(quotes[[.]]))) %>%
glimpse()
## Observations: 30
## Variables: 53
## $ averageDailyVolume3Month.fmt.0 <chr> "437,939", "541,801", "1.033M", "992,278", "1.40...
## $ averageDailyVolume3Month.longFmt.0 <chr> "437,939", "541,801", "1,033,453", "992,278", "1...
## $ averageDailyVolume3Month.raw.0 <dbl> 437939, 541801, 1033453, 992278, 1402175, 537906...
## $ exchange.0 <chr> "NYQ", "NMS", "NYQ", "NGM", "NGM", "NMS", "NGM",...
## $ exchangeTimezoneName.0 <chr> "America/New_York", "America/New_York", "America...
## $ exchangeTimezoneShortName.0 <chr> "EST", "EST", "EST", "EST", "EST", "EST", "EST",...
## $ fiftyTwoWeekHigh.fmt.0 <chr> "15.36", "46.12", "15.82", "7.64", "5.97", "9.86...
## $ fiftyTwoWeekHigh.raw.0 <dbl> 15.360, 46.120, 15.820, 7.640, 5.970, 9.860, 5.0...
## $ fiftyTwoWeekHighChange.fmt.0 <chr> "-6.96", "-17.72", "-4.19", "-2.83", "-4.29", "-...
## $ fiftyTwoWeekHighChange.raw.0 <dbl> -6.960, -17.720, -4.190, -2.830, -4.290, -3.640,...
## $ fiftyTwoWeekHighChangePercent.fmt.0 <chr> "-45.31%", "-38.42%", "-26.49%", "-37.04%", "-71...
## $ fiftyTwoWeekHighChangePercent.raw.0 <dbl> -0.4531, -0.3842, -0.2649, -0.3704, -0.7186, -0....
## $ fiftyTwoWeekLow.fmt.0 <chr> "6.59", "23.75", "9.46", "4.22", "0.25", "4.97",...
## $ fiftyTwoWeekLow.raw.0 <dbl> 6.590, 23.750, 9.460, 4.220, 0.251, 4.970, 0.970...
## $ fiftyTwoWeekLowChange.fmt.0 <chr> "1.81", "4.65", "2.17", "0.59", "1.43", "1.25", ...
## $ fiftyTwoWeekLowChange.raw.0 <dbl> 1.810, 4.650, 2.170, 0.590, 1.429, 1.250, 0.190,...
## $ fiftyTwoWeekLowChangePercent.fmt.0 <chr> "27.47%", "19.58%", "22.94%", "13.98%", "569.32%...
## $ fiftyTwoWeekLowChangePercent.raw.0 <dbl> 0.2747, 0.1958, 0.2294, 0.1398, 5.6932, 0.2515, ...
## $ fullExchangeName.0 <chr> "NYSE", "NasdaqGS", "NYSE", "NasdaqGM", "NasdaqG...
## $ gmtOffSetMilliseconds.0 <dbl> -1.8e+07, -1.8e+07, -1.8e+07, -1.8e+07, -1.8e+07...
## $ invalid.0 <lgl> FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE,...
## $ isLoading.0 <lgl> FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE,...
## $ language.0 <chr> "en-US", "en-US", "en-US", "en-US", "en-US", "en...
## $ longName.0 <chr> "Bankrate, Inc.", "Air Methods Corp.", "Bristow ...
## $ market.0 <chr> "us_market", "us_market", "us_market", "us_marke...
## $ marketCap.fmt.0 <chr> "758.268M", "1.082B", "407.786M", "494.03M", "33...
## $ marketCap.longFmt.0 <chr> "758,267,968", "1,081,625,344", "407,786,176", "...
## $ marketCap.raw.0 <dbl> 758267968, 1081625344, 407786176, 494030272, 338...
## $ marketState.0 <chr> "CLOSED", "CLOSED", "CLOSED", "CLOSED", "CLOSED"...
## $ messageBoardId.0 <chr> "finmb_30061", "finmb_24494", "finmb_292980", "f...
## $ quoteType.0 <chr> "EQUITY", "EQUITY", "EQUITY", "EQUITY", "EQUITY"...
## $ regularMarketChange.fmt.0 <chr> "1.20", "3.65", "1.44", "0.59", "0.27", "0.80", ...
## $ regularMarketChange.raw.0 <dbl> 1.20, 3.65, 1.44, 0.59, 0.27, 0.80, 0.15, 0.04, ...
## $ regularMarketChangePercent.fmt.0 <chr> "16.67%", "14.75%", "14.13%", "13.98%", "19.15%"...
## $ regularMarketChangePercent.raw.0 <dbl> 16.6667, 14.7475, 14.1315, 13.9810, 19.1489, 14....
## $ regularMarketDayHigh.fmt.0 <chr> "9.90", "30.25", "12.60", "5.29", "1.85", "6.30"...
## $ regularMarketDayHigh.raw.0 <dbl> 9.9000, 30.2500, 12.6000, 5.2900, 1.8500, 6.2950...
## $ regularMarketDayLow.fmt.0 <chr> "7.90", "25.30", "9.46", "4.30", "1.33", "5.79",...
## $ regularMarketDayLow.raw.0 <dbl> 7.9000, 25.3000, 9.4600, 4.3000, 1.3300, 5.7900,...
## $ regularMarketPrice.fmt.0 <chr> "8.40", "28.40", "11.63", "4.81", "1.68", "6.22"...
## $ regularMarketPrice.raw.0 <dbl> 8.40, 28.40, 11.63, 4.81, 1.68, 6.22, 1.16, 1.20...
## $ regularMarketTime.fmt.0 <chr> "4:02PM EDT", "4:00PM EDT", "4:00PM EDT", "4:00P...
## $ regularMarketTime.raw.0 <dbl> 1478289722, 1478289600, 1478289614, 1478289600, ...
## $ regularMarketVolume.fmt.0 <chr> "2.383M", "3.32M", "3.411M", "4.567M", "4.232M",...
## $ regularMarketVolume.longFmt.0 <chr> "2,382,856", "3,320,072", "3,411,052", "4,566,79...
## $ regularMarketVolume.raw.0 <dbl> 2382856, 3320072, 3411052, 4566790, 4232135, 185...
## $ sharesOutstanding.fmt.0 <chr> "90.27M", "38.085M", "35.063M", "102.709M", "20....
## $ sharesOutstanding.longFmt.0 <chr> "90,270,000", "38,085,400", "35,063,300", "102,7...
## $ sharesOutstanding.raw.0 <dbl> 90270000, 38085400, 35063300, 102709000, 2016800...
## $ shortName.0 <chr> "Bankrate, Inc. Common Stock", "Air Methods Corp...
## $ sourceInterval.0 <dbl> 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, ...
## $ symbol.0 <chr> "RATE", "AIRM", "BRS", "CERS", "EBIO", "ELNK", "...
## $ uuid.0 <chr> "79521cde-a3ef-383f-917d-31c49f9082f5", "f0432c1...
我覺得你還是有一些數據轉換(取決於你以後)做的,但你有數據,你正在尋找。
非常感謝你! – Rime