Pandas 自定义分组聚合

Question

我有一个关于 pandas 和自定义组聚合的问题，以找到计算我的值的最有效方法。这是我的代码片段：

import pandas as pd

listA = list('abcdefghijklmnopqrstuvwxyz') * 2
listB = listA[::-1]
listC = listA[::2] * 2
listD = "Won"
data1 = range(52) 
data2 = range(52,104) 
data3 = range(104,156)

rawStructure = [('A', listA),
                ('B', listB),
                ('C', listC),
                ('D', listD),
                ('Data1', data1),
                ('Data2', data2),
                ('Data3', data3)]
df = pd.DataFrame.from_items(rawStructure, orient='columns')

df.loc[40:,"D"] = "Lost" 

def customfct(x,y,z):
    print('x',x)
    data = round(((x.sum() + y.sum())/z.sum()) * 100,2)
    return  data

def f(row): 
    val1 = row.loc[(row['D'] == "Won"), 'Data1'].sum()
    val2 = row.loc[(row['D'] == "Won"), 'Data2'].sum()
    val3 = row.loc[(row['D'] == "Won"), 'Data3'].sum()
    val4 = customfct(row.loc[(row['D'] == "Won"), 'Data1'], row.loc[(row['D'] == "Won"), 'Data2'], row.loc[(row['D'] == "Won"), 'Data3'])
    return val1, val2, val3, val4

groupByCriteria = "C"
agg = df[:].groupby(by=groupByCriteria).apply(f)
print(agg)

我想知道是否有更有效的方法来进行分组和应用自定义计算（如函数 "customfct"，它使用不同的列（数据 1、数据 2、数据 3））。我的第一种方法就像你在这里看到的那样：http://www.shanelynn.ie/summarising-aggregation-and-grouping-data-in-python-pandas/ 但创建一个不受一列约束的公式似乎是不可行的（例如 lambda x: max(x) - min(x)）。此外，您如何 return pandas 数据框而不是 pandas 系列（带有元组）？提前致谢！

这是我当前的输出（这是正确的，但我想还有更有效的方法）：

Pandas output

Answer 1

考虑在一个 groupby() 调用中聚合所有 Data 列，然后为 val4 创建一个新列。然后将聚合合并回原始数据框。

# EQUIVALENT EXAMPLE DATA
listA = list('abcdefghijklmnopqrstuvwxyz') * 2
df = pd.DataFrame({'A': listA, 'B': listA[::-1], 'C': listA[::2] * 2,
                   'D': ["Won" for i in range(40)] + ["Lost" for i in range(40,52)],
                   'Data1': range(52), 'Data2': range(52,104), 'Data3': range(104,156)})

# ADJUSTED METHOD
groupByCriteria = "C"
grp = df[df['D']=="Won"].groupby(by=groupByCriteria).sum().reset_index()\
                              .rename(columns={'Data1':'val1','Data2':'val2','Data3':'val3'})
grp['val4'] = round(((grp['val1'] + grp['val2'])/grp['val3']) * 100,2)

agg = df.merge(grp, on='C').sort_values('Data1').reset_index(drop=True)

在时序比较中，调整后的代码明显更快。请注意：您的方法已调整为 return 数据框而不是系列。

def origfct():
    def customfct(x,y,z):
        #print('x',x)
        data = round(((x.sum() + y.sum())/z.sum()) * 100,2)
        return data

    def f(row): 
        row['val1'] = row.loc[(row['D'] == "Won"), 'Data1'].sum()
        row['val2'] = row.loc[(row['D'] == "Won"), 'Data2'].sum()
        row['val3'] = row.loc[(row['D'] == "Won"), 'Data3'].sum()
        row['val4'] = customfct(row.loc[(row['D'] == "Won"), 'Data1'],
                                row.loc[(row['D'] == "Won"), 'Data2'],
                                row.loc[(row['D'] == "Won"), 'Data3'])
        return row

    groupByCriteria = "C"
    agg = df[:].groupby(by=groupByCriteria).apply(f)
    return agg

def newsetup():
    groupByCriteria = "C"
    grp = df[df['D']=="Won"].groupby(by=groupByCriteria).sum().reset_index()\
                           .rename(columns={'Data1':'val1','Data2':'val2','Data3':'val3'})
    grp['val4'] = round(((grp['val1'] + grp['val2'])/grp['val3']) * 100,2)

    agg = df.merge(grp, on='C').sort_values('Data1').reset_index(drop=True)
    return agg


python -mtimeit -n'100' -s'import pyscript as test' 'test.origfct()'
# 100 loops, best of 3: 198 msec per loop

python -mtimeit -n'100' -s'import pyscript as test' 'test.newsetup()'
# 100 loops, best of 3: 16 msec per loop

Pandas 自定义分组聚合

Pandas customized group aggregation

customization

grouping

aggregation

pandas