Python使用lxml遍歷節

Question

我有一個我正在使用BeautifulSoup解析的網頁，但速度很慢，所以我決定嘗試使用lxml，因爲我讀的速度非常快。

無論如何，我努力讓我的代碼迭代我想要的部分，不知道如何使用lxml，我找不到明確的文檔。

總之，這裏是我的代碼：

import urllib, urllib2 
from lxml import etree 

def wgetUrl(target): 
    try: 
     req = urllib2.Request(target) 
     req.add_header('User-Agent', 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.0.3 Gecko/2008092417 Firefox/3.0.3') 
     response = urllib2.urlopen(req) 
     outtxt = response.read() 
     response.close() 
    except: 
     return '' 
    return outtxt 

newUrl = 'http://www.tv3.ie/3player' 

data = wgetUrl(newUrl) 
parser = etree.HTMLParser() 
tree = etree.fromstring(data, parser) 

for elem in tree.iter("div"): 
    print elem.tag, elem.attrib, elem.text 

這將返回所有的DIV的，但我怎麼指定只能通過機號=「slider1」循環？

div {'style': 'position: relative;', 'id': 'slider1'} None 

這不起作用：

for elem in tree.iter("slider1"): 

我知道這可能是一個愚蠢的問題，但我不出來..

謝謝！

*編輯* *

在您的幫助將這個代碼我現在有下面的輸出：

for elem in tree.xpath("//div[@id='slider1']//div[@id='gridshow']"): 
    print elem[0].tag, elem[0].attrib, elem[0].text 
    print elem[1].tag, elem[1].attrib, elem[1].text 
    print elem[2].tag, elem[2].attrib, elem[2].text 
    print elem[3].tag, elem[3].attrib, elem[3].text 
    print elem[4].tag, elem[4].attrib, elem[4].text 

輸出：

a {'href': '/3player/show/392/57922/1/Tallafornia', 'title': '3player | Tallafornia, 11/01/2013. The Tallafornia crew are back, living in a beachside villa in Santa Ponsa, Majorca. As the crew settle in, the egos grow bigger than ever and cause tension'} None 
h3 {} None 
span {'id': 'gridcaption'} The Tallafornia crew are back, living in a beachside vill... 
span {'id': 'griddate'} 11/01/2013 
span {'id': 'gridduration'} 00:27:52 

這是所有精彩，但我缺少上面的標籤的一部分。解析器不能正確處理代碼？

我沒有收到如下：

<img alt="3player | Tallafornia, 11/01/2013. The Tallafornia crew are back, living in a beachside villa in Santa Ponsa, Majorca. As the crew settle in, the egos grow bigger than ever and cause tension" src='http://content.tv3.ie/content/videos/0378/tallaforniaep2_fri11jan2013_3player_1_57922_180x102.jpg' class='shadow smallroundcorner'></img> 

任何想法，爲什麼它不拉呢？

再次感謝，非常有幫助的帖子..

Answer 1

可以按如下方式使用XPath表達式：

for elem in tree.xpath("//div[@id='slider1']"): 

例子：

>>> import urllib2 
>>> import lxml.etree 
>>> url = 'http://www.tv3.ie/3player' 
>>> data = urllib2.urlopen(url) 
>>> parser = lxml.etree.HTMLParser() 
>>> tree = lxml.etree.parse(data,parser) 
>>> elem = tree.xpath("//div[@id='slider1']") 
>>> elem[0].attrib 
{'style': 'position: relative;', 'id': 'slider1'} 

---

你需要更好的分析你正在處理的頁面內容（一個很好的方法是使用Firefox和Firebug插件）。

的<img>標籤你試圖獲取實際上是<a>標籤的孩子：

>>> for elem in tree.xpath("//div[@id='slider1']//div[@id='gridshow']"): 
... for elem_a in elem.xpath("./a"): 
...  for elem_img in elem_a.xpath("./img"): 
...   print '<A> HREF=%s'%(elem_a.attrib['href']) 
...   print '<IMG> ALT="%s"'%(elem_img.attrib['alt']) 
<A> HREF=/3player/show/392/58784/1/Tallafornia 
<IMG> ALT="3player | Tallafornia, 01/02/2013. A fresh romance blossoms in the Tallafornia house. Marc challenges Cormac to a 'bench off' in the gym" 
<A> HREF=/3player/show/46/58765/1/Coronation-Street 
<IMG> ALT="3player | Coronation Street, 01/02/2013. Tyrone bumps into Kirsty in the street and tries to take Ruby from her pram" 
../.. 

Answer 2

這是我得到了它的工作爲我自己，我不知道這是否是最好的方法，使歡迎評論：

import urllib2, re 
from lxml import etree 
from datetime import datetime 

def wgetUrl(target): 
    try: 
     req = urllib2.Request(target) 
     req.add_header('User-Agent', 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.0.3 Gecko/2008092417 Firefox/3.0.3') 
     response = urllib2.urlopen(req) 
     outtxt = response.read() 
     response.close() 
    except: 
     return '' 
    return outtxt 

start = datetime.now() 

newUrl = 'http://www.tv3.ie/3player' # homepage 

data = wgetUrl(newUrl) 
parser = etree.HTMLParser() 
tree = etree.fromstring(data, parser) 

for elem in tree.xpath("//div[@id='slider1']//div[@id='gridshow'] | //div[@id='slider1']//div[@id='gridshow']//img[@class='shadow smallroundcorner']"): 
    if elem.tag == 'img': 
     img = elem.attrib.get('src') 
     print 'img: ', img 

    if elem.tag == 'div': 
     show = elem[0].attrib.get('href') 
     print 'show: ', show 
     titleData = elem[0].attrib.get('title') 

     match=re.search("3player\s+\|\s+(.+),\s+(\d\d/\d\d/\d\d\d\d)\.\s*(.*)", titleData) 
     title=match.group(1) 
     print 'title: ', title 

     description = match.group(3) 
     print 'description: ', description 

     date = elem[3].text 
     duration = elem[4].text 
     print 'date: ', date 
     print 'duration: ', duration 

end = datetime.now() 
print 'time took was ', (end-start) 

的時機是相當不錯的，雖然沒有大的區別，我期待在BeautifulSoup ..