在字符串的 lambda 函数中省略 Nans

Question

这个例子有点复杂。我在 lambda 函数的代码中使用了之前创建的函数 (document_path_similarity())。

import numpy as np
import pandas as pd
import nltk

from nltk.corpus import wordnet as wn
...<some code>

def similarity_score(s1, s2):

# where s1, s2 are the list of synsets.

    lst = []
    # For each synset in s1
    for x in s1:
        # finds the synset in s2 with the largest similarity value
        lst.append(max([y.path_similarity(x) for y in s2 if y.path_similarity(x)])) #so its not None
  
    return sum(lst)/float(len(lst)) 

def document_path_similarity(doc1, doc2):
    # Finds the symmetrical similarity between doc1 and doc2
    synsets1 = doc_to_synsets(doc1)
    synsets2 = doc_to_synsets(doc2)

    return (similarity_score(synsets1, synsets2) + similarity_score(synsets2, synsets1)) / 2

现在我正在尝试向我的数据框 df 添加一个新列 s_scores，它将显示 D1 列和 D2 列中的字符串之间的相似性分数。

    Q   D1                                                  D2
1   1   After more than two years' detention under the...   After more than two years in detention by the ...
2   1   "It still remains to be seen whether the reven...   "It remains to be seen whether the revenue rec...
8   0   "It's a major victory for Maine, and it's a ma...   The Maine program could be a model for other s...
9   1   Microsoft said Friday that it is halting devel...   Microsoft will stop developing versions of its...
10  0   New legit download service launches with PC us...   BuyMusic is the first subscription-free paid d...

我已尝试按以下方式处理此问题。

df['s_scores'] = df.apply(lambda x: document_path_similarity(x['D1'], x['D2']), axis=1)

这给出了

ValueError: ('max() arg is an empty sequence', 'occurred at index 8')

因为在应用 lambda 表达式后，索引 8 的 s_score 是 NaN。这种情况发生在我的 df 中的几行。

8   0   "It's a major victory for Maine, and it's a ma...   The Maine program could be a model for other s...   NaN

如果我尝试应用 similarity_score() 而不是 document_path_similarity() 函数，则不会出现此错误。它运行正常，因为我有条件确保 'if y.path_similarity(x)' 没有 NaN 值。

我试过像这样添加'if x is not None'或'np.isnan(x)'。

df['s_scores'] = df.apply(lambda x: document_path_similarity(x.D1, x.D2),axis=1 if x is not None)
SyntaxError: invalid syntax

我什至试过这个：

df['s_scores'] = df.apply(lambda x: (similarity_score(x.D1, x.D2) + similarity_score(x.D2, x.D1)) / 2,axis=1)
AttributeError: ("'str' object has no attribute 'path_similarity'", 'occurred at index 0')

所以我不知道如何在我的函数中添加 NaN 异常？

我也很疑惑为什么document_path_similarity()不会像similarity_score()那样跳过NaN，如果前者是从后者派生出来的？

对不起，如果我试图解释我的功能是如何工作的，时间太长了。非常感谢您的帮助。

Answer 1

这与您在另一个问题中提出的问题完全相同。 similarity 发布了包含错误的代码。你必须修补 similarity_score()

df = pd.read_csv(io.StringIO("""    Q   D1                                                  D2
1   1   After more than two years' detention under the...   After more than two years in detention by the ...
2   1   "It still remains to be seen whether the reven...   "It remains to be seen whether the revenue rec...
8   0   "It's a major victory for Maine, and it's a ma...   The Maine program could be a model for other s...
9   1   Microsoft said Friday that it is halting devel...   Microsoft will stop developing versions of its...
10  0   New legit download service launches with PC us...   BuyMusic is the first subscription-free paid d..."""), 
           sep="\s\s+", engine="python")

def similarity_score(s1, s2):
    list1 = []
    for a in s1:
        # patch +[0] at end so never finding max of empty list
        list1.append(max([i.path_similarity(a) for i in s2 if i.path_similarity(a) is not None]+[0]))
    output = sum(list1)/len(list1)
    return output



df = df.assign(
    s_scores=lambda x: x.apply(lambda r: document_path_similarity(r.D1, r.D2), axis=1),
    s_scores2=lambda x: x.apply(lambda r: (similarity_score(doc_to_synsets(r.D1), 
                                                            doc_to_synsets(r.D2)) + 
                                           similarity_score(doc_to_synsets(r.D2), 
                                                            doc_to_synsets(r.D1))) / 2,axis=1)
)

print(df.to_string(index=False))

输出

 Q                                                 D1                                                 D2  s_scores  s_scores2
 1  After more than two years' detention under the...  After more than two years in detention by the ...  0.782738   0.782738
 1  "It still remains to be seen whether the reven...  "It remains to be seen whether the revenue rec...  0.844444   0.844444
 0  "It's a major victory for Maine, and it's a ma...  The Maine program could be a model for other s...  0.407526   0.407526
 1  Microsoft said Friday that it is halting devel...  Microsoft will stop developing versions of its...  0.371869   0.371869
 0  New legit download service launches with PC us...  BuyMusic is the first subscription-free paid d...  0.048678   0.048678

Answer 2

我找到了另一种更改 similarity_score() 的方法，因此它会忽略空列表。

def similarity_score(s1, s2):

lst = []

for x in s1:
    s = [x.path_similarity(y) for y in s2 if x.path_similarity(y) is not None]

    if len(s)>0:
        lst.append(max(s))
output = sum(lst)/len(lst)
return output

在字符串的 lambda 函数中省略 Nans

Leaving out the Nans within the lambda function for strings

lambda

nan

nltk

python-3.x

pandas