我想用更多的n-gram來製作word2vec模型。正如我發現的,gensim.models.phrase中的短語類可以找到我想要的短語,並且可以在語料庫上使用短語並將其用於word2vec訓練函數的結果模型。文字處理 - 短語檢測後的Word2Vec訓練(bigram模型)
因此,首先我要做一些類似於下面的事情,完全像gensim documentation中的示例代碼。
class MySentences(object):
def __init__(self, dirname):
self.dirname = dirname
def __iter__(self):
for fname in os.listdir(self.dirname):
for line in open(os.path.join(self.dirname, fname)):
yield word_tokenize(line)
sentences = MySentences('sentences_directory')
bigram = gensim.models.Phrases(sentences)
model = gensim.models.Word2Vec(bigram['sentences'], size=300, window=5, workers=8)
模型已經建立,但沒有任何效果良好評價,並警告:我搜索了它,我發現https://groups.google.com/forum/#!topic/gensim/XWQ8fPMFSi0,改變了我的代碼
WARNING : train() called with an empty iterator (if not intended, be sure to provide a corpus that offers restartable iteration = an iterable)
:
class MySentences(object):
def __init__(self, dirname):
self.dirname = dirname
def __iter__(self):
for fname in os.listdir(self.dirname):
for line in open(os.path.join(self.dirname, fname)):
yield word_tokenize(line)
class PhraseItertor(object):
def __init__(self, my_phraser, data):
self.my_phraser, self.data = my_phraser, data
def __iter__(self):
yield self.my_phraser[self.data]
sentences = MySentences('sentences_directory')
bigram_transformer = gensim.models.Phrases(sentences)
bigram = gensim.models.phrases.Phraser(bigram_transformer)
corpus = PhraseItertor(bigram, sentences)
model = gensim.models.Word2Vec(corpus, size=300, window=5, workers=8)
我獲得錯誤:
Traceback (most recent call last):
File "/home/fatemeh/Desktop/Thesis/bigramModeler.py", line 36, in <module>
model = gensim.models.Word2Vec(corpus, size=300, window=5, workers=8)
File "/home/fatemeh/.local/lib/python3.4/site-packages/gensim/models/word2vec.py", line 478, in init
self.build_vocab(sentences, trim_rule=trim_rule)
File "/home/fatemeh/.local/lib/python3.4/site-packages/gensim/models/word2vec.py", line 553, in build_vocab
self.scan_vocab(sentences, progress_per=progress_per, trim_rule=trim_rule) # initial survey
File "/home/fatemeh/.local/lib/python3.4/site-packages/gensim/models/word2vec.py", line 575, in scan_vocab
vocab[word] += 1
TypeError: unhashable type: 'list'
現在我想知道我的代碼有什麼問題。