Tokenize基於空格和尾部標點符號？

Question

我想要拿出正則表達式來將字符串分成基於空格或尾標點的列表。

例如

s = 'hel-lo this has whi(.)te, space. very \n good' 

我要的是

['hel-lo', 'this', 'has', 'whi(.)te', ',', 'space', '.', 'very', 'good'] 

s.split()得到最深的方式存在，只是它不採取尾隨空白的照顧。

Answer 1

import re 
s = 'hel-lo this has whi(.)te, space. very \n good' 
[x for x in re.split(r"([.,!?]+)?\s+", s) if x] 
# => ['hel-lo', 'this', 'has', 'whi(.)te', ',', 'space', '.', 'very', 'good'] 

您可能需要調整什麼是 「標點」 是。

Answer 2

使用spacy的粗略解決方案。它已經很好地使用詞彙化詞彙。

import spacy 
s = 'hel-lo this has whi(.)te, space. very \n good' 
nlp = spacy.load('en') 
ls = [t.text for t in nlp(s) if t.text.strip()] 

>> ['hel', '-', 'lo', 'this', 'has', 'whi(.)te', ',', 'space', '.', 'very', 'good'] 

然而，-之間也令牌化的話，所以我借的解決方案，從here合併-之間的話重新走到一起。

merge = [(i-1, i+2) for i, s in enumerate(ls) if i >= 1 and s == '-'] 
for t in merge[::-1]: 
    merged = ''.join(ls[t[0]:t[1]]) 
    ls[t[0]:t[1]] = [merged] 

>> ['hel-lo', 'this', 'has', 'whi(.)te', ',', 'space', '.', 'very', 'good'] 

Answer 3

我正在使用Python 3.6.1。

import re 

s = 'hel-lo this has whi(.)te, space. very \n good' 
a = [] # this list stores the items 
for i in s.split(): # split on whitespaces 
    j = re.split('(\,|\.)$',i) # split on your definition of trailing punctuation marks 
    if len(j) > 1: 
     a.extend(j[:-1]) 
    else: 
     a.append(i) 
# a -> ['hel-lo', 'this', 'has', 'whi(.)te', ',', 'space', '.', 'very', 'good']