用美麗的湯刮到熊貓的HTML表格

我想刮一張使用美麗的湯的html表，並將其導入到熊貓 - http://www.baseball-reference.com/teams/NYM/2017.shtml - 「團隊擊球」表。用美麗的湯刮到熊貓的HTML表格

查找表是沒有問題的：

table = soup.find('div', attrs={'class': 'overthrow table_container'}) 
table_body = table.find('tbody')

查找數據行是不是一個問題或者：

for i in table.findAll('tr')[2]: #increase to 3 to get next row in table... 
    print(i.get_text())

而且我甚至可以找到頭名：

table_head = table.find('thead') 

for i in table_head.findAll('th'): 
    print(i.get_text())

現在我很難將所有東西放在一起放入數據框中。這是我到目前爲止：

header = []  
for th in table_head.findAll('th'): 
     key = th.get_text() 
     header.append(key) 

row= [] 
for tr in table.findAll('tr')[2]: 
    value = tr.get_text() 
    row.append(value) 

od = OrderedDict(zip(head, row)) 
df = pd.DataFrame(d1, index=[0])

這隻適用於一次一行。我的問題是如何在同一時間對錶格中的每一行執行此操作？

來源

2017-04-20 e9e9s

我已經測試過，下面的內容適用於你的目的。基本上你需要創建一個列表，循環遍歷播放器，使用該列表填充DataFrame。建議不要逐行創建DataFrame，因爲這可能會明顯變慢。

import collections as co 
import pandas as pd 

from bs4 import BeautifulSoup 

with open('team_batting.html','r') as fin: 
    soup = BeautifulSoup(fin.read(),'lxml') 

table = soup.find('div', attrs={'class': 'overthrow table_container'}) 
table_body = table.find('tbody') 

table_head = table.find('thead') 
header = []  
for th in table_head.findAll('th'): 
    key = th.get_text() 
    header.append(key) 

# loop over table to find number of rows with '' in first column 
endrows = 0 
for tr in table.findAll('tr'): 
    if tr.findAll('th')[0].get_text() in (''): 
     endrows += 1 

rows = len(table.findAll('tr')) 
rows -= endrows + 1 # there is a pernicious final row that begins with 'Rk' 

list_of_dicts = [] 
for row in range(rows): 
    the_row = [] 
    try: 
     table_row = table.findAll('tr')[row] 
     for tr in table_row: 
      value = tr.get_text() 
      the_row.append(value) 
     od = co.OrderedDict(zip(header,the_row)) 
     list_of_dicts.append(od) 
    except AttributeError: 
     continue 

df = pd.DataFrame(list_of_dicts)

來源

2017-04-20 17:10:30 bernie

感謝您的好評。在這一行中是否有「[row]」的名稱：table_row = table.findAll（'tr'）[row] ---我從來沒有見過像這樣在範圍之前使用它。 – e9e9s

你非常歡迎。這只是這種情況下的索引。等於'table_row = table.findAll（'tr'）[0]'或'table_row = table.findAll（'tr'）[1]' – bernie

那麼如果'[row]'被排除在這一行之外，迭代'table_row'你將無法做到？ – e9e9s

該解決方案使用僅pandas，但通過提前知道球隊擊球表是第十表欺騙了一點。有了這些知識，下面通過pandas的read_html功能和返回DataFrame對象列表抓住第十DataFrame。其餘的只是一些數據清理：

import pandas as pd 


url = 'http://www.baseball-reference.com/teams/NYM/2017.shtml' 

# Take 10th dataframe 
team_batting = pd.read_html(url)[9] 

# Take columns whose names don't contain "Unnamed" 
team_batting.drop([x for x in team_batting.columns if 'Unnamed' in x], axis=1, inplace=True) 

# Remove the rows that are just a copy of the headers/columns 
team_batting = team_batting.ix[team_batting.apply(lambda x: x != team_batting.columns,axis=1).all(axis=1),:] 

# Take out the Totals rows 
team_batting = team_batting.ix[~team_batting.Rk.isnull(),:] 

# Get a glimpse of the data 
print(team_batting.head(5)) 

# Rk Pos    Name Age G PA AB R H 2B ... OBP SLG OPS OPS+ TB GDP HBP SH SF IBB 
# 0 1 C Travis d'Arnaud 28 12 42 37 6 10 2 ... .357 .541 .898 144 20 1 1 0 0 1 
# 1 2 1B  Lucas Duda* 31 13 50 42 4 10 2 ... .360 .571 .931 153 24 1 0 0 0 2 
# 2 3 2B  Neil Walker# 31 14 62 54 5 12 3 ... .306 .278 .584 64 15 2 0 0 1 0 
# 3 4 SS Asdrubal Cabrera# 31 15 67 63 10 17 2 ... .313 .397 .710 96 25 0 0 0 0 0 
# 4 5 3B  Jose Reyes# 34 15 59 53 3 5 2 ... .186 .132 .319 -9 7 0 0 0 0 0

我希望這有助於。

來源

2017-04-20 18:24:02 Abdou

用美麗的湯刮到熊貓的HTML表格

回答

相關問題