我现在将推文中的单词限制为实词我想将单词转换为小写并添加带有下划线的 POS

Question

我写了下面的代码，将推文中的单词限制为实词，即名词、动词和形容词，现在我想将单词转换为小写，并在 POS 中添加下划线。例如：

love_VERB old-fashioneds_NOUN 但我不知道怎么办，谁能帮帮我？


! pip install wget
import wget
url = 'https://raw.githubusercontent.com/dirkhovy/NLPclass/master/data/reviews.full.tsv.zip'
wget.download(url, 'reviews.full.tsv.zip')


from zipfile import ZipFile
with ZipFile('reviews.full.tsv.zip', 'r') as zf:
    zf.extractall()


import pandas as pd
df = pd.read_csv('reviews.full.tsv', sep='\t', nrows=100000) # nrows , max amount of rows 
documents = df.text.values.tolist()
print(documents[:4])


import spacy

nlp = spacy.load('en_core_web_sm') #you can use other methods
# excluded tags
included_tags = {"NOUN", "VERB", "ADJ"}
#document = [line.strip() for line in open('moby_dick.txt', encoding='utf8').readlines()]

sentences = documents[:103] #first 10 sentences
new_sentences = []
for sentence in sentences:
    new_sentence = []
    for token in nlp(sentence):
        if token.pos_  in included_tags:
            new_sentence.append(token.text)
    new_sentences.append(" ".join(new_sentence))

#Creates a list of lists of tokens
tokens = [[token.text for token in nlp(new_sentence)] for new_sentence in documents[:200]]
tokens

# import itertools
# tok = itertools.chain.from_iterable(
#    [[token.text for token in nlp(new_sentence)] for new_sentence in documents[:200]])

# tok

Answer 1

我相信如果你改变

        new_sentence.append(token.text)

到

        new_sentence.append(token.text.lower()+'_'+token.POS)

你会得到你想要的。

我现在将推文中的单词限制为实词我想将单词转换为小写并添加带有下划线的 POS

I restricted the words in the tweets to content words now I want to Transform the words to lower case and add the POS with an underderscore

python

nlp

nltk

pandas

spacy