從維基百科解析網站信息框數據

Question

我使用維基百科API獲取信息框數據。我想從這個信息框數據解析website url。我嘗試使用mwparserfromhell解析網站網址，但不同的關鍵字有不同的格式。

這裏有幾個模式的網站 -

url     = <!-- {{URL|www.example.com}} --> 
| url = [https://www.TheGuardian.com/ TheGuardian.com] 
| url = <span class="plainlinks">[https://www.naver.com/ www.naver.com]</span> 
|url    = [https://www.tmall.com/ tmall.com] 
|url   = [http://www.ustream.tv/ ustream.tv] 

我需要在解析official website link由維基百科所支持的所有模式的幫助？

編輯 -

碼 -

# get infobox data 
import requests 
# keyword 
keyword = 'stackoverflow.com' 
# wikipedia api url 
api_url = (
    'https://en.wikipedia.org/w/api.php?action=query&prop=revisions&' 
    'rvprop=content&titles=%s&rvsection=0&format=json' % keyword) 
# api request 
resp = requests.get(api_url).json() 
page_one = next(iter(resp['query']['pages'].values())) 
revisions = page_one.get('revisions', []) 
# infobox daa 
infobox_data = next(iter(revisions[0].values())) 

# parse website url 
import mwparserfromhell 
wikicode = mwparserfromhell.parse(infobox_data) 
templates = wikicode.filter_templates() 
website_url_1 = '' 
website_url_2 = '' 
for template in templates: 
    # Pattern - `URL|http://x.com` 
    if template.name == "URL": 
     website_url_1 = str(template.get(1).value) 
     break 
    if not website_url_1: 
     # Pattern - `website = http://x.com` 
     try: 
      website_url_2 = str(template.get("website").value) 
     except ValueError: 
      pass 
    if not website_url_1: 
     # Pattern - `homepage = http://x.com` 
     try: 
      website_url_2 = str(template.get("homepage").value) 
     except ValueError: 
      pass 
if website_url_1: 
    website_url = website_url_1 
elif website_url_2: 
    website_url = website_url_2 

Answer 1

一個可以分析你已經使用正則表達式和BeautifulSoup提到的模式。可以想象，可以通過擴展這種方法來解析附加模式。

我從行的開頭刪除包含'url ='的內容，然後使用BeautifulSoup處理餘數。由於BeautifulSoup封裝了它形成完整頁面的內容，原始內容可以作爲body元素的文本獲得。

>>> import re 
>>> patterns = '''\ 
... url     = <!-- {{URL|www.example.com}} --> 
... | url = [https://www.TheGuardian.com/ TheGuardian.com] 
... | url = <span class="plainlinks">[https://www.naver.com/ www.naver.com]</span> 
... |url    = [https://www.tmall.com/ tmall.com] 
... |url   = [http://www.ustream.tv/ ustream.tv]''' 
>>> import bs4 
>>> regex = re.compile(r'\s*\|?\s*url\s*=\s*', re.I) 
>>> for pattern in patterns.split('\n'): 
...  soup = bs4.BeautifulSoup(re.sub(regex, '', pattern), 'lxml') 
...  if str(soup).startswith('<!--'): 
...   'just a comment' 
...  else: 
...   soup.find('body').getText() 
... 
'just a comment' 
'[https://www.TheGuardian.com/ TheGuardian.com]' 
'[https://www.naver.com/ www.naver.com]' 
'[https://www.tmall.com/ tmall.com]' 
'[http://www.ustream.tv/ ustream.tv]' 

Answer 2

mwparserfromhell是一個很好的工具：

import mwclient 
import mwparserfromhell 

site = mwclient.Site('en.wikipedia.org') 
text = site.pages[pagename].text() 
wikicode = mwparserfromhell.parse(text) 
templates = wikicode.filter_templates(matches='infobox .*') 
url = templates[0].get('url').value 

url_template = url.filter_templates(matches='url') 
url_link = url.filter_external_links() 
if url_template: 
    print url_template[0].get(1) 
elif url_link: 
    print url_link.url 
else: 
    print url