Python的網頁抓取特殊字符提取數據

Question

cityURL='https://en.wikipedia.org/wiki/Elko,_Nevada' 

def createObj(url): 
    html = urlopen(url) 
    bsObj = BeautifulSoup(html, 'lxml') 
    return bsObj 

bsObj1 = createObj(cityURL) 

table1 = bsObj1.find("table", {"class":"infobox geography vcard"}) 
incorporated = table1.find("th", text='Incorporated (city)').findNext('td').get_text() 

table1.find("th", text='. Total') # Problem here, due to the special dot, I cannot identify the "th" 

我希望得到以下數據：

總，17.6 土地，17.6 水，0.0

Answer 1

的 「。」 在頁面不是「點」。這是一個unicode字符BULLET（\ u2022）。

你可以使用python的regex（re）模塊來實現這一點。

更新的代碼會是這個樣子：

import re 
cityURL='https://en.wikipedia.org/wiki/Elko,_Nevada' 

def createObj(url): 
    html = urlopen(url) 
    bsObj = BeautifulSoup(html, 'lxml') 
    return bsObj 

bsObj1 = createObj(cityURL) 

table1 = bsObj1.find("table", {"class":"infobox geography vcard"}) 
incorporated = table1.find("th", text='Incorporated (city)').findNext('td').get_text() 

pattern = re.compile(r'Total') 
table1.find("th", text=pattern) 

或者，你可以使用lxml的模塊，它是很多比beautifulsoup快。

import requests 
from lxml import html 

cityURL='https://en.wikipedia.org/wiki/Elko,_Nevada' 
r = requests.get(cityURL) 
root = html.fromstring(r.content) 

def normalize(text) : 
    return ''.join([i if ord(i) < 128 else ' ' for i in text]).strip().split()[0] 

val_list = [(normalize(root.xpath('//table[@class="infobox geography vcard"]//tr[./th/text()="Area"]/following-sibling::tr[{}]//text()'.format(str(val)))[1]), normalize(root.xpath('//table[@class="infobox geography vcard"]//tr[./th/text()="Area"]/following-sibling::tr[{}]//text()'.format(str(val)))[3])) for val in xrange(1,4)] 
print(val_list) 

上面的代碼將輸出：

[(u'Total', u'17.6'), (u'Land', u'17.6'), (u'Water', u'0.0')]