Error in Computing the Coherence Score – AttributeError: 'dict' object has no attribute 'id2token'

Question

我是NLP初学者，第一次做Topic Modeling。我能够生成我的模型，但是我无法生成一致性指标。

将term-document矩阵转换成新的gensim格式，from df --> sparse matrix --> gensim corpus

sparse_counts = scipy.sparse.csr_matrix(data_dtm)
corpus = matutils.Sparse2Corpus(sparse_counts)
corpus

df_lemmatized.head()

# Gensim also requires dictionary of the all terms and their respective location in the term-document matrix
tfidfv = pickle.load(open("tfidf.pkl", "rb"))
id2word = dict((v, k) for k, v in tfidfv.vocabulary_.items())
id2word

这是我的模型：

lda = models.LdaModel(corpus=corpus, id2word=id2word, num_topics=15, passes=10, random_state=43)
lda.print_topics()

最后，这是我尝试使用一致性模型获得一致性分数的地方：

# Compute Perplexity
print('\nPerplexity: ', lda.log_perplexity(corpus))  

# Compute Coherence Score
coherence_model_lda = CoherenceModel(model=lda, texts=df_lemmatized.long_title, dictionary=id2word, coherence='c_v')
coherence_lda = coherence_model_lda.get_coherence()
print('\nCoherence Score: ', coherence_lda)

这是错误：

---> 57 if not dictionary.id2token: # 可能没有在标准中初始化 gensim.corpora.Dictionary 58 setattr(字典, 'id2token', {v: k for k, v in dictionary.token2id.items()}) 59 AttributeError: 'dict' 对象没有属性 'id2token'

Answer 1

我没有你的数据，所以我无法重现错误。所以，我来猜一猜！问题出在你的 id2word 内，它应该是 corpora.dictionary.Dictionary 而不仅仅是 dict。因此，您需要执行以下操作：

>>> from gensim import corpora
>>>
>>> word2id = dict((k, v) for k, v in tfidfv.vocabulary_.items())
>>> d = corpora.Dictionary()
>>> d.id2token = id2word
>>> d.token2id = word2id
>>> #...
>>> # change `id2word` to `d`
>>> coherence_model_lda = CoherenceModel(model=lda, texts=df_lemmatized.long_title, dictionary=d, coherence='c_v')

而且我认为它现在应该可以正常工作了！

Error in Computing the Coherence Score – AttributeError: 'dict' object has no attribute 'id2token'

Error in Computing the Coherence Score – AttributeError: 'dict' object has no attribute 'id2token'

python

nlp

scipy

gensim

topic-modeling