在 pandas 中执行 nltk.stem.SnowballStemmer

Question

我有一个四列的 DataFrame，其中有两列标记化的单词，这些单词已删除停用词并转换为小写，现在正试图阻止。

我不确定 apply() 方法是否访问系列及其各个单元格，或者我是否需要另一种进入每条记录的方法，所以尝试了两种方法（我认为！）

from nltk.stem import SnowballStemmer
stemmer = nltk.stem.SnowballStemmer('english')

我试过了：

df_2['Headline'] = df_2['Headline'].apply(lambda x: stemmer.stem(item) for item in x)

--------------------------------------------------------------------------- TypeError Traceback (most recent call last) in () ----> 1 df_2['Headline__'] = df_2['Headline'].apply(lambda x: stemmer.stem(item) for item in x)

~\AppData\Local\Continuum\anaconda3\envs\learn-env\lib\site-packages\pandas\core\series.py in apply(self, func, convert_dtype, args, **kwds) 3192
else: 3193 values = self.astype(object).values -> 3194 mapped = lib.map_infer(values, f, convert=convert_dtype) 3195 3196 if len(mapped) and isinstance(mapped[0], Series):

pandas/_libs/src\inference.pyx in pandas._libs.lib.map_infer()

TypeError: 'generator' object is not callable

我相信这个 TypeError 类似于说 'List' object is not callable 的那个，并用 apply() 方法修复了那个，这里没有想法。

df_2['Headline'] = df_2['Headline'].apply(lambda x: stemmer.stem(x))

--------------------------------------------------------------------------- AttributeError Traceback (most recent call last) in () ----> 1 df_2['Headline'] = df_2['Headline'].apply(lambda x: stemmer.stem(x)) 2 3 df_2.head()

~\AppData\Local\Continuum\anaconda3\envs\learn-env\lib\site-packages\pandas\core\series.py in apply(self, func, convert_dtype, args, **kwds) 3192
else: 3193 values = self.astype(object).values -> 3194 mapped = lib.map_infer(values, f, convert=convert_dtype) 3195 3196 if len(mapped) and isinstance(mapped[0], Series):

pandas/_libs/src\inference.pyx in pandas._libs.lib.map_infer()

in (x) ----> 1 df_2['Headline'] = df_2['Headline'].apply(lambda x: stemmer.stem(x)) 2 3 df_2.head()

~\AppData\Local\Continuum\anaconda3\envs\learn-env\lib\site-packages\nltk\stem\snowball.py in stem(self, word) 1415 1416 """ -> 1417 word = word.lower() 1418 1419 if word in self.stopwords or len(word) <= 2:

AttributeError: 'list' object has no attribute 'lower'

Answer 1

您需要为 apply 指定 axis。

这是一个完整的工作示例：

import pandas as pd

df = pd.DataFrame({
    'col_1' : [['ducks'], ['dogs']],
    'col_2' : [['he', 'eats', 'apples'], ['she', 'has', 'cats', 'dogs']],
    'col_3' : ['some data 1', 'some data 2'],
    'col_4' : ['another data 1', 'another data 2']
})
df.head()

输出

    col_1   col_2                   col_3       col_4
0   [ducks] [he, eats, apples]      some data 1 another data 1
1   [dogs]  [she, has, cats, dogs]  some data 2 another data 2

现在让我们为标记化的列应用词干提取：

import nltk
from nltk.stem import SnowballStemmer
stemmer = nltk.stem.SnowballStemmer('english')

df.col_1 = df.apply(lambda row: [stemmer.stem(item) for item in row.col_1], axis=1)
df.col_2 = df.apply(lambda row: [stemmer.stem(item) for item in row.col_2], axis=1)

检查数据框的新内容。

df.head()

输出

    col_1   col_2                   col_3       col_4
0   [duck]  [he, eat, appl]         some data 1 another data 1
1   [dog]   [she, has, cat, dog]    some data 2 another data 2

在 pandas 中执行 nltk.stem.SnowballStemmer

Execute nltk.stem.SnowballStemmer in pandas

python

nlp

nltk

pandas