網站刮板不會刮我的一個鏈接

Question

我可以刮一個網站容易，但其他我得到錯誤？林不知道如果因爲網站有某種塊或東西

import random 
from bs4 import BeautifulSoup 
import urllib2 
import re 
from urlparse import urljoin 

user_input = raw_input ("Search for Team = "); 


resp = urllib2.urlopen("http://idimsports.eu/football.html") ###working 
soup = BeautifulSoup(resp, from_encoding=resp.info().getparam('charset')) 

base_url = "http://idimsports.eu" 
links = soup.find_all('a', href=re.compile(''+user_input)) 
if len(links) == 0: 
    print "No Streams Available" 
else: 
    for link in links: 
     print urljoin(base_url, link['href']) 

resp = urllib2.urlopen("http://cricfree.tv/football-live-stream") ###not working 
soup = BeautifulSoup(resp, from_encoding=resp.info().getparam('charset')) 

links = soup.find_all('a', href=re.compile(''+user_input)) 
if len(links) == 0: 
    print "No Streams Available" 
else: 
    for link in links: 
     print urljoin(base_url, link['href']) 

Answer 1

設置你的請求的用戶代理頭

headers = { 'User-Agent' : 'Mozilla/5.0' } 
req = urllib2.Request("http://cricfree.tv/football-live-stream", None, headers) 
resp = urllib2.urlopen(req) 

還對你的第二個循環中，您打算重用base_url你可能不想這樣做。