0
我一直在嘗試使用一些代碼，我發現in this answer遞歸找到一個給定的URL的所有鏈接：如何遞歸查找來自網頁與美麗的所有鏈接？
import urllib2
from bs4 import BeautifulSoup
url = "http://francaisauthentique.libsyn.com/"
def recursiveUrl(url,depth):
if depth == 5:
return url
else:
page=urllib2.urlopen(url)
soup = BeautifulSoup(page.read())
newlink = soup.find('a') #find just the first one
if len(newlink) == 0:
return url
else:
return url, recursiveUrl(newlink,depth+1)
def getLinks(url):
page=urllib2.urlopen(url)
soup = BeautifulSoup(page.read())
links = soup.find_all('a')
for link in links:
links.append(recursiveUrl(link,0))
return links
links = getLinks(url)
print(links)
再說警告
/usr/local/lib/python2.7/dist-packages/bs4/__init__.py:181: UserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system ("lxml"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.
The code that caused this warning is on line 28 of the file downloader.py. To get rid of this warning, change code that looks like this:
BeautifulSoup(YOUR_MARKUP})
to this:
BeautifulSoup(YOUR_MARKUP, "lxml")
我收到以下錯誤：
Traceback (most recent call last):
File "downloader.py", line 28, in <module>
links = getLinks(url)
File "downloader.py", line 25, in getLinks
links.append(recursiveUrl(link,0))
File "downloader.py", line 11, in recursiveUrl
page=urllib2.urlopen(url)
File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 396, in open
protocol = req.get_type()
TypeError: 'NoneType' object is not callable
問題是什麼？
我想你傳遞一個BeautifulSoup對象'urlopen'，而不是URL。試試類似'link ['href']'，但一定要檢查它是否在第一位。 – Thomas
謝謝托馬斯，但現在我收到一個錯誤「ValueError：unknown url type：/ webpage/categery/general」。也許是因爲這是一個相對的鏈接而不是絕對的鏈接？ – Alex
@Alex正確：） –