URL解析錯誤[BeautifulSoup]

Question

我試圖從網站頁面獲取href鏈接列表;但是我的代碼工作不正常。該代碼在不應該加入urlList時附加。它也在複製href鏈接。

import urllib2 
from BeautifulSoup import BeautifulSoup 

response = urllib2.urlopen("http://www.gamefaqs.com") 
html = response.read() 
soup = BeautifulSoup(html) 

doNotProcessList = ["gamespot.com", "cnet.com", "gamefaqs.com"] 

urlList = [] 

for link in soup.findAll('a'): 
    for bad in doNotProcessList: 
     if bad not in link['href']: 
      urlList.append(link['href']) 

print urlList 

典型的錯誤：

[u'http://cbsiprivacy.custhelp.com/app/answers/detail/a_id/1272/', u'http://cbsiprivacy 
.custhelp.com/app/answers/detail/a_id/1272/', u'http://www.cbsinteractive.com/terms_of_use.php?tag=ft', u'http://www 
.cbsinteractive.com/terms_of_use.php?tag=ft', u'http://www.cbsinteractive.com/terms_of_use.php?tag=ft', u'http://m.g 
amefaqs.com/?mob_on=1', u'http://m.gamefaqs.com/?mob_on=1'] 

錯誤與「不是」的if語句去掉了不只會造成不好的項目被存儲在列表中，像這樣的事：

[u'http://membership.gamefaqs.com/1328-4-46.html', u'http://www.gamefaqs.com/user/register.html', u'http://www.games 
pot.com/6316274', u'http://www.gamespot.com/6316274', u'http://www.gamespot.com/6316489', u'http://www.gamespot.com/ 
6316489', u'http://www.gamespot.com/6316225', u'http://www.gamespot.com/6316225', u'http://www.gamespot.com/features 
/index.html', u'http://www.gamespot.com/news/6322016.html', u'http://www.gamespot.com/news/6322019.html', u'http://w 
ww.gamespot.com/news/6322017.html', u'http://www.gamespot.com/news/6322010.html', u'http://www.gamespot.com/news/632 
1996.html', u'http://www.gamespot.com/news/index.html', u'http://www.gamespot.com/features/6314339/index.html', u'ht 
tp://www.gamespot.com/features/6313939/index.html', u'http://www.gamespot.com/features/6309202/index.html', u'http:/ 
/www.gamespot.com/features/6320393/index.html', u'http://www.gamespot.com/features/6162248/index.html', u'http://www 
.gamespot.com/gameguides.html', u'http://www.gamespot.com/downloads/index.html', u'http://www.gamespot.com/news/inde 
x.html', u'http://www.gamespot.com/pc/index.html', u'http://www.gamespot.com/xbox360/index.html', u'http://www.games 
pot.com/wii/index.html', u'http://www.gamespot.com/ps3/index.html', u'http://www.gamespot.com/psp/index.html', u'htt 
p://www.gamespot.com/ds/index.html', u'http://www.gamespot.com/ps2/index.html', u'http://www.gamespot.com/gba/index. 
html', u'http://www.gamespot.com/mobile/index.html', u'http://www.gamespot.com/cheats.html', u'http://www.gamespot.c 
om/forums/index.html', u'http://www.gamespot.com/', u'http://www.gamefaqs.com/features/help/', u'http://sitemap.game 
faqs.com/', u'http://www.gamefaqs.com/features/aboutus.html', u'http://reviews.cnet.com/Music/2001-6450_7-0.html', u 
'http://reviews.cnet.com/Cell_phones/2001-3504_7-0.html', u'http://reviews.cnet.com/Digital_cameras/2001-6501_7-0.ht 
ml', u'http://reviews.cnet.com/Notebooks/2001-3121_7-0.html', u'http://reviews.cnet.com/Handhelds/2001-3127_7-0.html 
', u'http://reviews.cnet.com/4521-6531_7-5021436-3.html', u'http://reviews.cnet.com/Web_hosting/2001-6540_7-0.html', 
u'http://clearance.cnet.com', u'http://shopper.cnet.com/4520-5-6276184.html', u'http://www.cnet.com', u'http://www. 
gamespot.com', u'http://www.gamespot.com/cheats.html', u'http://www.cnet.com/apple-iphone.html', u'http://www.gamesp 
ot.com/reviews.html', u'http://reviews.cnet.com/laptops', u'http://download.cnet.com/windows/antivirus-software/', u 
'http://m.gamefaqs.com/?mob_on=1'] 

Answer 1

列表理解FTW：

[link['href'] for link in soup.findAll('a') 
if not any(bad in link['href'] for bad in doNotProcessList)] 

而且，爲便於閱讀...

def condition(x): 
    return not any((bad in x) for bad in doNotProcessList) 

[link['href'] for link in soup.findAll('a') if condition(link['href'])]