在逻辑条件中包含更多停用词列表以过滤单词

Question

我需要在清理数据时添加更多条件，包括删除停用词、星期几和月份。对于星期几和月份，我创建了一个单独的列表（我不知道 python 中是否有一些已经内置的包来包含它们）。对于数字，我会考虑 isdigit。所以像这样：

days=['Monday','Tuesday','Wednesday','Thursday','Friday','Saturday','Sunday']
# need to put into lower case
months=['January','February','March', 'April','May','June','July','August','September','October','November','December']
# need to put into lower case

cleaned = [w for w in remove_punc.split() if w.lower() not in stopwords.words('english')]

我怎样才能包含在上面的代码中？我知道要考虑额外的 if 语句，但我正在为此苦苦挣扎。

Answer 1

您可以将所有列表转换为集合，并将它们的并集作为最终集合。然后它只是检查你的单词在集合中的成员资格。像下面这样的东西会起作用：

# existing code
from nltk.corpus import stopwords

days=['Monday','Tuesday','Wednesday','Thursday','Friday','Saturday','Sunday']
# need to put into lower case
months=['January','February','March', 'April','May','June','July','August','September','October','November','December']
# need to put into lower case

# add these lines
stop_words = set(stopwords.words('english'))
lowercase_days = {item.lower() for item in days}
lowercase_months = {item.lower() for item in months}

exclusion_set = lowercase_days.union(lowercase_months).union(stop_words)

# now do the final check
cleaned = [w for w in remove_punc.split() if w.lower() not in exclusion_set and not w.isdigit()]

在逻辑条件中包含更多停用词列表以过滤单词

Include more lists of stopwords in logical condition to filter words

python

nlp

list

nltk