使用列表中的URL刮擦文本（BeautifulSoup4）

Question

我從一個URL獲得了工作文本刮除器。問題是我需要再刮25個網址。這些網址幾乎相同，唯一不同的是最後一封信。這裏的代碼更清晰：

import urllib2 
from bs4 import BeautifulSoup 

f = open('(path to file)/names', 'a') 
links = ['http://guardsmanbob.com/media/playlist.php?char='+ chr(i) for i in range(97,123)] 

response = urllib2.urlopen(links[0]).read() 
soup = BeautifulSoup(response) 

for tr in soup.findAll('tr'): 
    if not tr.find('td'): continue 
    for td in tr.find('td').findAll('a'): 
     f.write(td.contents[0] + '\n') 

我不能讓這個腳本一次運行列表中的所有URL。所有我設法得到的是每個網址的第一首歌曲名稱。對不起我的英語不好。我希望你能理解我。

Answer 1

I can't make this script to run all urls from list in one time.

一個參數，*args保存你的代碼的方法（或任何你想要的名字，只是不要忘了*）。 *將自動解包您的列表。 *沒有正式名稱，但有些人（包括我）喜歡將它稱爲splat operator。

def start_download(*args): 
    for value in args: 
     ##for debugging purposes 
     ##print value 

     response = urllib2.urlopen(value).read() 
     ##put the rest of your code here 

if __name__ == '__main__': 
    links = ['http://guardsmanbob.com/media/playlist.php?char='+ 
       chr(i) for i in range(97,123)] 

    start_download(links) 

編輯： 或者您也可以直接循環在你的鏈接列表，並下載每個。

links = ['http://guardsmanbob.com/media/playlist.php?char='+ 
       chr(i) for i in range(97,123)] 
    for link in links: 
     response = urllib2.urlopen(link).read() 
     ##put the rest of your code here 

編輯2：

爲了讓所有的鏈接，然後保存這些文件，這裏的具體意見的完整代碼：

import urllib2 
from bs4 import BeautifulSoup, SoupStrainer 

links = ['http://guardsmanbob.com/media/playlist.php?char='+ 
      chr(i) for i in range(97,123)] 

    for link in links: 
     response = urllib2.urlopen(link).read() 
     ## gets all <a> tags 
     soup = BeautifulSoup(response, parse_only=SoupStrainer('a')) 
     ## unnecessary link texts to be removed 
     not_included = ['News', 'FAQ', 'Stream', 'Chat', 'Media', 
        'League of Legends', 'Forum', 'Latest', 'Wallpapers', 
        'Links', 'Playlist', 'Sessions', 'BobRadio', 'All', 
        'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 
        'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 
        'U', 'V', 'W', 'X', 'Y', 'Z', 'Misc', 'Play', 
        'Learn more about me', 'Chat info', 'Boblights', 
        'Music Playlist', 'Official Facebook', 
        'Latest Music Played', 'Muppets - Closing Theme', 
        'Billy Joel - The River Of Dreams', 
        'Manic Street Preachers - If You Tolerate This 
        Your Children Will Be Next', 
        'The Bravery - An Honest Mistake', 
        'The Black Keys - Strange Times', 
        'View whole playlist', 'View latest sessions', 
        'Referral Link', 'Donate to BoB', 
        'Guardsman Bob', 'Website template', 
        'Arcsin'] 

     ## create a file named "test.txt" 
     ## write to file and close afterwards 
     with open("test.txt", 'w') as output: 
      for hyperlink in soup: 
       if hyperlink.text: 
        if hyperlink.text not in not_included: 
         ##print hyperlink.text 
         output.write("%s\n" % hyperlink.text.encode('utf-8')) 

這裏是保存在test.txt輸出：

我建議您將test.txt更改爲不同的文件名（前S標題），每次循環鏈接列表時都會覆蓋前一個鏈接。