從網頁提取所有鏈接

Question

我想寫一個功能，需要一個網頁的URL，下載網頁，並返回該網頁的URL列表（使用urllib模塊） 任何幫助，將不勝感激

Answer 1

在這裏你去：

import sys 
import urllib2 
import lxml.html 

try: 
    url = sys.argv[1] 
except IndexError: 
    print "Specify a url to scrape" 
    sys.exit(1) 

if not url.startswith("http://"): 
    print "Please include the http:// at the beginning of the url" 
    sys.exit(1) 

html = urllib2.urlopen(url).read() 
etree = lxml.html.fromstring(html) 

for href in etree.xpath("//a/@href"): 
    print href 

---

 
C:\Programming>getlinks.py http://example.com 
/
/domains/ 
/numbers/ 
/protocols/ 
/about/ 
/go/rfc2606 
/about/ 
/about/presentations/ 
/about/performance/ 
/reports/ 
/domains/ 
/domains/root/ 
/domains/int/ 
/domains/arpa/ 
/domains/idn-tables/ 
/protocols/ 
/numbers/ 
/abuse/ 
http://www.icann.org/ 
mailto:iana@iana.org?subject=General%20website%20feedback