無法從NLTK庫導入Bigrams

Question

快速問題令我困惑。我有NLTK安裝，它一直工作正常。然而，我試圖讓一個語料庫的bigrams，並希望基本上使用bigrams（語料庫）..但它說，當我「來自nltk import bigrams」時，bigrams沒有定義。

與trigrams相同。我錯過了什麼嗎？另外，我怎麼能手動從語料庫中獲取bigrams。

我也在計算bigrams trigrams和quad的頻率，但我不確定如何去做這件事。

我在開頭和結尾處都適當地使用了"<s>"和"</s>"的語料庫。計劃至今這裏：

#!/usr/bin/env python 
import re 
import nltk 
import nltk.corpus as corpus 
import tokenize 
from nltk.corpus import brown 

def alter_list(row): 
    if row[-1] == '.': 
     row[-1] = '</s>' 
    else: 
     row.append('</s>') 
    return ['<s>'] + row 

news = corpus.brown.sents(categories = 'editorial') 
print len(news),'\n' 

x = len(news) 
for row in news[:x]: 
    print(alter_list(row)) 

Answer 1

我在測試的virtualenv這和它的作品：

In [20]: from nltk import bigrams 

In [21]: bigrams('This is a test') 
Out[21]: 
[('T', 'h'), 
('h', 'i'), 
('i', 's'), 
('s', ' '), 
(' ', 'i'), 
('i', 's'), 
('s', ' '), 
(' ', 'a'), 
('a', ' '), 
(' ', 't'), 
('t', 'e'), 
('e', 's'), 
('s', 't')] 

這是唯一的錯誤你得到？

順便說一句，作爲你的第二個問題：

from collections import Counter 
In [44]: b = bigrams('This is a test') 

In [45]: Counter(b) 
Out[45]: Counter({('i', 's'): 2, ('s', ' '): 2, ('a', ' '): 1, (' ', 't'): 1, ('e', 's'): 1, ('h', 'i'): 1, ('t', 'e'): 1, ('T', 'h'): 1, (' ', 'i'): 1, (' ', 'a'): 1, ('s', 't'): 1}) 

對於話：

In [49]: b = bigrams("This is a test".split(' ')) 

In [50]: b 
Out[50]: [('This', 'is'), ('is', 'a'), ('a', 'test')] 

In [51]: Counter(b) 
Out[51]: Counter({('is', 'a'): 1, ('a', 'test'): 1, ('This', 'is'): 1}) 

這種分裂的話顯然是很膚淺的，但取決於你的應用程序可能就足夠了。顯然，你可以使用nltk的標記化，這是非常複雜的。

爲了實現自己的最終目標，你可以做這樣的事情：

In [56]: d = "Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged. It was popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum passages, and more recently with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum." 

In [56]: from nltk import trigrams 
In [57]: tri = trigrams(d.split(' ')) 

In [60]: counter = Counter(tri) 

In [61]: import random 

In [62]: random.sample(counter, 5) 
Out[62]: 
[('Ipsum', 'has', 'been'), 
('industry.', 'Lorem', 'Ipsum'), 
('Ipsum', 'passages,', 'and'), 
('was', 'popularised', 'in'), 
('galley', 'of', 'type')] 

我修剪的輸出，因爲它是不必要的大，但你的想法。