HTML到可讀文本

我正在編寫一個程序以在Google App Engine上運行。它只需獲取URL並通過從HTML源代碼中刪除標記，腳本和任何其他不可讀的東西（類似於nltk.clear_html）來返回文本。HTML到可讀文本

HtmlTool by Eike。

import urllib 

class HtmlTool(object): 
    import HTMLParser 
    import re 
    """ 
    Algorithms to process HTML. 
    """ 
    #Regular expressions to recognize different parts of HTML. 
    #Internal style sheets or JavaScript 
    script_sheet = re.compile(r"<(script|style).*?>.*?(</\1>)", 
           re.IGNORECASE | re.DOTALL) 
    #HTML comments - can contain ">" 
    comment = re.compile(r"<!--(.*?)-->", re.DOTALL) 
    #HTML tags: <any-text> 
    tag = re.compile(r"<.*?>", re.DOTALL) 
    #Consecutive whitespace characters 
    nwhites = re.compile(r"[\s]+") 
    #<p>, <div>, <br> tags and associated closing tags 
    p_div = re.compile(r"</?(p|div|br).*?>", 
         re.IGNORECASE | re.DOTALL) 
    #Consecutive whitespace, but no newlines 
    nspace = re.compile("[^\S\n]+", re.UNICODE) 
    #At least two consecutive newlines 
    n2ret = re.compile("\n\n+") 
    #A return followed by a space 
    retspace = re.compile("(\n)") 

    #For converting HTML entities to unicode 
    html_parser = HTMLParser.HTMLParser() 

    @staticmethod 
    def to_nice_text(html): 
     """Remove all HTML tags, but produce a nicely formatted text.""" 
     if html is None: 
      return u"" 
     text = html 
     text = HtmlTool.script_sheet.sub(" ", text) 
     text = HtmlTool.comment.sub(" ", text) 
     text = HtmlTool.nwhites.sub(" ", text) 
     text = HtmlTool.p_div.sub("\n", text) #convert <p>, <div>, <br> to "\n" 
     text = HtmlTool.tag.sub(" ", text)  #remove all tags 
     text = HtmlTool.html_parser.unescape(text) 
     #Get whitespace right 
     text = HtmlTool.nspace.sub(" ", text) 
     text = HtmlTool.retspace.sub("\n", text) 
     text = HtmlTool.n2ret.sub("\n\n", text) 
     text = text.strip() 
     return text

它適用於 'http://google.com'

text = HtmlTool.to_nice_text(urllib.urlopen('http://google.com').read())

但是，它拋出一個錯誤 'http://yahoo.com'

text = HtmlTool.to_nice_text(urllib.urlopen('http://yahoo.com').read())

錯誤：

Traceback (most recent call last): 
    File "C:\Users\BK\Desktop\Working Folder\AppEngine\crawlnsearch\-test.py", line 51, in <module> 
    text = HtmlTool.to_nice_text(urllib.urlopen('http://yahoo.com').read()) 
    File "C:\Users\BK\Desktop\Working Folder\AppEngine\crawlnsearch\-test.py", line 43, in to_nice_text 
    text = HtmlTool.html_parser.unescape(text) 
    File "C:\Python27\lib\HTMLParser.py", line 472, in unescape 
    return re.sub(r"&(#?[xX]?(?:[0-9a-fA-F]+|\w{1,8}));", replaceEntities, s) 
    File "C:\Python27\lib\re.py", line 151, in sub 
    return _compile(pattern, flags).sub(repl, string, count) 
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe0 in position 1531: ordinal not in range(128)

那麼，任何人都可以解釋這個代碼有什麼問題，併爲此發佈修復或請告訴我如何使用nltk.clean_html。

來源

2013-08-31 bkmagnetron

這是因爲unicode和bytestrings之間有混合。

如果您在模塊中使用與HtmlTool

from __future__ import unicode_literals

爲了確保每" "樣塊是unicode的，

text = HtmlTool.to_nice_text(urllib.urlopen(url).read().decode("utf-8"))

要發送一個UTF-8字符串的方法，可以解決你的問題。

有關Unicode的更多信息，請閱讀以下內容： http://www.joelonsoftware.com/articles/Unicode.html

來源

2013-08-31 04:09:26

HTML到可讀文本

回答

相關問題