修復追溯（最近一次通話最後）錯誤？

我寫了一個程序來提取網頁中PDF文件的所有鏈接。該項目工程的完美，與一些網站沒有錯誤，例如像：修復追溯（最近一次通話最後）錯誤？

Hussam# python extractPDF.py http://www.cs.odu.edu/~mln/teaching/cs532-s17/test/pdfs.html

輸出：

Entered URL: 
http://www.cs.odu.edu/~mln/teaching/cs532-s17/test/pdfs.html 
Final URL: 
http://www.cs.odu.edu/~mln/teaching/cs532-s17/test/pdfs.html 
http://www.cs.odu.edu/~mln/pubs/ht-2015/hypertext-2015-temporal-violations.pdf 
Size: 2184076 
http://www.cs.odu.edu/~mln/pubs/tpdl-2015/tpdl-2015-annotations.pdf 
Size: 622981 
http://arxiv.org/pdf/1512.06195 
Size: 1748961 
http://www.cs.odu.edu/~mln/pubs/tpdl-2015/tpdl-2015-off-topic.pdf 
Size: 4308768 
http://www.cs.odu.edu/~mln/pubs/tpdl-2015/tpdl-2015-stories.pdf 
Size: 1274604 
http://www.cs.odu.edu/~mln/pubs/tpdl-2015/tpdl-2015-profiling.pdf 
Size: 639001 
http://www.cs.odu.edu/~mln/pubs/jcdl-2014/jcdl-2014-brunelle-damage.pdf 
Size: 2205546 
http://www.cs.odu.edu/~mln/pubs/jcdl-2015/jcdl-2015-mink.pdf 
Size: 1254605 
http://www.cs.odu.edu/~mln/pubs/jcdl-2015/jcdl-2015-arabic-sites.pdf 
Size: 709420 
http://www.cs.odu.edu/~mln/pubs/jcdl-2015/jcdl-2015-dictionary.pdf 
Size: 2350603

在另一方面，如果我嘗試此鏈接：

Hussam# python extractPDF.py http://www.cs.odu.edu/~mln/pubs/all.html

我得到的輸出是正確的，但最後有一個錯誤。

Entered URL: 
http://www.cs.odu.edu/~mln/pubs/all.html 
Final URL: 
http://www.cs.odu.edu/~mln/pubs/all.html 
http://www.cs.odu.edu/~mln/pubs/tpdl-2016/tpdl-2016-kelly.pdf 
Size: 953454 
http://www.cs.odu.edu/~mln/pubs/tpdl-2016/tpdl-2016-alam.pdf 
Size: 928749 
http://www.cs.odu.edu/~mln/pubs/jcdl-2016/jcdl-2016-alam-ipfs.pdf 
Size: 516538 
http://www.cs.odu.edu/~mln/pubs/jcdl-2016/jcdl-2016-alam-memgator.pdf 
Size: 345028 
http://www.cs.odu.edu/~mln/pubs/jcdl-2016/jcdl-2016-nwala.pdf 
Size: 640173 
http://www.cs.odu.edu/~mln/pubs/ht-2015/hypertext-2015-temporal-violations.pdf 
Size: 2184076 
http://www.cs.odu.edu/~mln/pubs/tpdl-2015/tpdl-2015-annotations.pdf 
Size: 622981 
http://www.cs.odu.edu/~mln/pubs/tpdl-2015/tpdl-2015-off-topic.pdf 
Size: 4308768 
http://www.cs.odu.edu/~mln/pubs/tpdl-2015/tpdl-2015-stories.pdf 
Size: 1274604 
http://www.cs.odu.edu/~mln/pubs/tpdl-2015/tpdl-2015-profiling.pdf 
Size: 639001 
http://www.cs.odu.edu/~mln/pubs/jcdl-2015/jcdl-2015-temporal-intention.pdf 
Size: 720476 
http://www.cs.odu.edu/~mln/pubs/jcdl-2015/jcdl-2015-mink.pdf 
Size: 1254605 
http://www.cs.odu.edu/~mln/pubs/jcdl-2015/jcdl-2015-arabic-sites.pdf 
Size: 709420 
http://www.cs.odu.edu/~mln/pubs/jcdl-2015/jcdl-2015-dictionary.pdf 
Size: 2350603 
http://www.cs.odu.edu/~mln/pubs/jcdl-2014/jcdl-2014-kelly-acid.pdf 
Size: 541843 
http://www.cs.odu.edu/~mln/pubs/jcdl-2014/jcdl-2014-kelly-mink.pdf 
Size: 556863 
http://www.cs.odu.edu/~mln/pubs/jcdl-2014/jcdl-2014-brunelle-damage.pdf 
Size: 2205546 
http://www.cs.odu.edu/~mln/pubs/jcdl-2014/jcdl-2014-cartledge-copies.pdf 
Size: 1199511 
http://www.cs.odu.edu/~mln/pubs/sigcse-2014/web-science-sigcse-2014.pdf 
Size: 158242 
http://www.cs.odu.edu/~mln/pubs/ecir-2014/ecir-2014.pdf 
Size: 902825 
http://www.cs.odu.edu/~mln/pubs/ieee-vis-2013/2013-ieee-vis-boxoffice.pdf 
Size: 122738 
Traceback (most recent call last): 
    File "extractPDF.py", line 21, in <module> 
    r = urllib2.urlopen(link) 
    File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen 
    return _opener.open(url, data, timeout) 
    File "/usr/lib/python2.7/urllib2.py", line 397, in open 
    response = meth(req, response) 
    File "/usr/lib/python2.7/urllib2.py", line 510, in http_response 
    'http', request, response, code, msg, hdrs) 
    File "/usr/lib/python2.7/urllib2.py", line 429, in error 
    result = self._call_chain(*args) 
    File "/usr/lib/python2.7/urllib2.py", line 369, in _call_chain 
    result = func(*args) 
    File "/usr/lib/python2.7/urllib2.py", line 605, in http_error_302 
    return self.parent.open(new, timeout=req.timeout) 
    File "/usr/lib/python2.7/urllib2.py", line 397, in open 
    response = meth(req, response) 
    File "/usr/lib/python2.7/urllib2.py", line 510, in http_response 
    'http', request, response, code, msg, hdrs) 
    File "/usr/lib/python2.7/urllib2.py", line 435, in error 
    return self._call_chain(*args) 
    File "/usr/lib/python2.7/urllib2.py", line 369, in _call_chain 
    result = func(*args) 
    File "/usr/lib/python2.7/urllib2.py", line 518, in http_error_default 
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) 
urllib2.HTTPError: HTTP Error 403: Forbidden

這裏是程序的代碼。

import sys 
from bs4 import * 
import urllib2 
import re 

if len(sys.argv) != 2: 
     print "USAGE:" 
     print "Python extracrPDF.py http://example.com/page.html" 
else: 
     url = sys.argv[1] 
     print "Entered URL:" 
     print url 
     html_page = urllib2.urlopen(url) 
     print "Final URL:" 
     print html_page.geturl() 
     soup = BeautifulSoup(html_page, "html.parser") 
     links = [] 
     for link in soup.findAll('a', attrs={'href': re.compile("^http://")}): 
       links.append(link.get('href')) 
     for link in links: 
       r = urllib2.urlopen(link) 
       if r.headers['content-type'] == "application/pdf": 
         print link 
         print "Size: " + r.headers['Content-Length']

來源

2017-01-24 Hussam Hallak

403禁止。您無權訪問'req.get_furll_url（）'中的任何值。你不明白什麼？ –

問題出在@WayneWerner指出的URL上。嘗試檢查** HttpError **異常。檢查此[鏈接]（http://stackoverflow.com/questions/13303449/urllib2-httperror-http-error-403-forbidden）的更多信息。 –

您的爬蟲程序是否被阻止，因爲它已被檢測到？ – mmenschig

urllib2.HTTPError: HTTP Error 403: Forbidden

您的代碼獲取頁面中所有的鏈接。至少有一個鏈接（不一定是PDF的鏈接）不適用於您。 403 Forbidden表示「服務器理解請求但拒絕授權請求。」該網址可能需要您擁有允許您訪問的憑據。

urllib2引發錯誤條件異常。你的代碼將需要處理其中的一些。

如果你只是想你的代碼繼續而不死亡，與更換相關章節：

for link in links: 
     r = None 
     try: 
      r = urllib2.urlopen(link) 
     except urllib2.HTTPError as e: 
      print link 
      print "Error: " + e.code + " " + e.reason 
      continue 

     if r.headers['content-type'] == "application/pdf": 
      print link 
      print "Size: " + r.headers['Content-Length']

來源

2017-01-24 21:33:26 Ouroborus

修復追溯（最近一次通話最後）錯誤？

回答

相關問題