Python從網站刮表？

Question

我想刮掉treasury.gov網站上的每個國庫券收益率。

https://www.treasury.gov/resource-center/data-chart-center/interest-rates/Pages/TextView.aspx?data=yieldAll

我怎麼會去考慮這些信息？我假設我不得不使用BeautifulSoup或Selenium或類似的東西（最好是BS4）。我最終想將這些數據放入Pandas DataFrame中。

Answer 1

這裏有一種方法可以使用請求獲取數據以表格和beautifulsoup

import pandas as pd 
import requests 
from bs4 import BeautifulSoup 

url = 'https://www.treasury.gov/resource-center/data-chart-center/interest-rates/Pages/TextView.aspx?data=yieldAll' 

r = requests.get(url) 
html = r.text 

soup = BeautifulSoup(html) 
table = soup.find('table', {"class": "t-chart"}) 
rows = table.find_all('tr') 
data = [] 
for row in rows[1:]: 
    cols = row.find_all('td') 
    cols = [ele.text.strip() for ele in cols] 
    data.append([ele for ele in cols if ele]) 

result = pd.DataFrame(data, columns=['Date', '1 Mo', '3 Mo', '6 Mo', '1 Yr', '2 Yr', '3 Yr', '5 Yr', '7 Yr', '10 Yr', '20 Yr', '30 Yr']) 

print(result)