Scrapy通過兄弟節點迭代 - xpath生成空列表

Question

我遍歷元素的兄弟節點。但是，我無法使用xpath函數來提取其中的數據。我似乎總是獲得一個空的列表。

# Determine if the cast members are listed in a table or unordered list. 
cast_siblings = response.xpath('//h2/span[starts-with(@id,"Cast")]/../following-sibling::*') 
for sibling in cast_siblings: 
    desc_str = "".join(sibling.xpath('./descendant::text()').extract()) 
    if "Trailer" in desc_str or "Film Festival" in desc_str or "Comment" in desc_str: 
     break 

    # Extract the name to get the type of the node. 
    node_type = sibling.xpath('name()').extract_first() 
    if node_type == 'ul': 
     cast_list = sibling.xpath('./ul[1]/li') 
     self.parse_list_into_cast_table(cast_list, _id, conn) 
    elif node_type == 'table': 
     cast_tables = sibling.xpath('./table') 
     self.parse_table_into_cast_table(cast_tables, _id, conn) 
    else: 
     print "Unknown cast element type! " + _id 

在上面的代碼，我一直在經歷的兄弟節點，直到我打它包含「拖車」，「電影節」或「評論」的地方文本的元素。否則，對於每個兄弟節點，我首先得到它的名字來找出它是什麼樣的節點 - 我對錶和無序列表感興趣。

但是，失敗的部分是當我調用sibling.xpath（'./ table'）和/或sibling.xpath（'ul [1]/li'）時。

生成的對象始終是一個空列表。

我已經使用scrapy shell並檢查了從cast_siblings獲得的選擇器列表。每個選擇器似乎都是有效的，當我提取它們時，我可以看到內部元素。我只是無法使用xpath到達他們。

下面是我的控制檯顯示空表結果的摘錄：

>>> a = response.xpath('//h2/span[starts-with(@id,"Cas")]/../following-sibling::*') 
>>> a 
[<Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<h3> <span class="mw-headline" id="Sudo_'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<table border="0" cellspacing="3" cellpa'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<table border="0" cellspacing="3" cellpa'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<h3> <span class="mw-headline" id="Maruy'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<table border="0" cellspacing="3" cellpa'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<h3> <span class="mw-headline" id="Azumi'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<table border="0" cellspacing="3" cellpa'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<table border="0" cellspacing="3" cellpa'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<table border="0" cellspacing="3" cellpa'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<h3> <span class="mw-headline" id="Prese'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<table border="0" cellspacing="3" cellpa'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<p><b>Additional Cast Members:</b>\n</p>'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<ul><li> <a href="/Yukari_Ito" title="Yu'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<h2> <span class="mw-headline" id="Trail'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<p>\n<script type="text/javascript" src="'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<div id="mediaplayer"></div>'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<script type="text/javascript">\n jwplay'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<h2> <span class="mw-headline" id="Comme'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<br>'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<div id="comment-outer"><hr><div id="com'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<div align="center"></div>'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<table cellpadding="2" style="border:0px'>] 
>>> a[0] 
<Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<h3> <span class="mw-headline" id="Sudo_'> 
>>> a[1] 
<Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<table border="0" cellspacing="3" cellpa'> 
>>> a[1].xpath('./table') 
[] 

這是很奇怪的，因爲A [1]在上面的例子清楚地表明內的表。

Answer 1

好像你正在尋找已經是一個表節點的表兒童：

node_type = sibling.xpath('name()').extract_first() 
if node_type == 'ul': 
    ... 
elif node_type == 'table': 
    cast_tables = sibling.xpath('./table') <--- 

這裏你已經知道，當前節點表，你正在試圖尋找它的下面另一個表。所以在這種情況下，它應該僅僅是：

cast_tables = sibling 

另外值得一提的是，當你測試scrapy選擇的結果包括根節點的標籤，即'//h1'將返回<h1>...</h1>