美麗的湯刮圖案？

Question

我試圖刮包含以下HTML代碼網站：

<div class="content-sidebar-wrap"><main class="content"><article 
class="post-773 post type-post status-publish format-standard has-post- 
thumbnail category-money entry" itemscope 
itemtype="http://schema.org/CreativeWork"> 

這包含數據我感興趣......我一直在使用BeautifulSoup解析它嘗試過，但以下回報：

<div class="content-sidebar-wrap"><main class="content"><article 
class="entry"> 
<h1 class="entry-title">Not found, error 404</h1><div class="entry-content 
"><p>"The page you are looking for no longer exists. Perhaps you can return 
back to the site's "<a href="http://www.totalsportek.com/">homepage</a> and 
see if you can find what you are looking for. Or, you can try finding it 
by using the search form below.</p><form 
action="http://www.totalsportek.com/" class="search-form" 
itemprop="potentialAction" itemscope="" 
itemtype="http://schema.org/SearchAction" method="get" role="search"> 

# I've made small modifications to make it readable 

美麗的湯元素不包含我想要的代碼。我不太熟悉html，但我假設這會調用一些外部服務來返回數據..？我讀過這個與Schema有關的東西。

無論如何我可以訪問這些數據嗎？

Answer 1

您在發出請求時需要指定User-Agent標頭。打印文章標題和內容的工作示例：

import requests 
from bs4 import BeautifulSoup 

url = "http://www.totalsportek.com/money/barcelona-player-salaries/" 

response = requests.get(url, headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.78 Safari/537.36"}) 
soup = BeautifulSoup(response.content, "html.parser") 

article = soup.select_one(".content article.post.entry.status-publish") 
header = article.header.get_text(strip=True) 
content = article.select_one(".entry-content").get_text(strip=True) 

print(header) 
print(content)