在 groupby 对象内的列列表上生成滚动计算的更快方法

Question

我创建了这个函数来计算我的 df 中的专长列表的滚动统计数据。此功能按预期工作，但在我的 df 上运行大约需要 30 分钟，它有大约 100 万行。在 python/pandas 中有更快的方法吗？

def add_rolling_vars(df, feats, amounts, group):
#creates rolling stats for a list of feats(columns) over a list of amounts[12,48](window sizes)
#grouped by a group like $gvkey or $sector
orig_feats = feats.copy()
new_feats= []
for amount in amounts:
    for name in feats:
        df[group+'_'+name+f'_{amount}_sma'] = df.groupby(group)[name].rolling(amount,1).mean().values
        df[group+'_'+name+f'_{amount}_std'] = df.groupby(group)[name].rolling(amount,1).std().values
        df[group+'_'+name+f'_{amount}_min'] = df.groupby(group)[name].rolling(amount,1).min().values
        df[group+'_'+name+f'_{amount}_max'] = df.groupby(group)[name].rolling(amount,1).max().values
        df[group+'_'+name+f'_{amount}_med'] = df.groupby(group)[name].rolling(amount,1).median().values
        df[group+'_'+name+f'_{amount}_25Q'] = df.groupby(group)[name].rolling(amount,1).quantile(.25).values
        df[group+'_'+name+f'_{amount}_75Q'] = df.groupby(group)[name].rolling(amount,1).quantile(.75).values

例如，我还创建了这个运行在同一数据集上用了大约 1 分钟的函数。显然，它是不同的，因为它不必遍历 windows 行，但我仍然可以传递一个 feats 列表而不是 feats 中的名称，然后使用列表理解命名将整个转换后的输出添加到我的数据框中方案：

def add_cat_stats(df,feats,group):
    #feats is a list of continuous feats to compute the monthly stats of       
    df[[group+'_'+name+'_avg' for name in feats]] = df.groupby([group,'Date'])[feats].transform('mean')
    df[[group+'_'+name+'_std' for name in feats]] = df.groupby([group,'Date'])[feats].transform('std')
    df[[group+'_'+name+'_min' for name in feats]] = df.groupby([group,'Date'])[feats].transform('min')
    df[[group+'_'+name+'_max' for name in feats]] = df.groupby([group,'Date'])[feats].transform('max')
    df[[group+'_'+name+'_med' for name in feats]] = df.groupby([group,'Date'])[feats].transform('median')

更新

len(数量) = 2

len(专长)= 16

Answer 1

我无法让@John Zwinck 代码工作，但它确实给了我重新格式化代码的想法，将时间从 30 分钟减少到 4 分 45 秒，这太棒了！最好进一步降低它，但这是一个可行的解决方案：

def add_rolling_vars(df, feats, amounts, group):
    for amount in amounts:
        grouped = df.groupby(group)[feats].rolling(amount,1)
        prefix = ['_'.join([group, name, str(amount)]) for name in feats]
        df[[pre+'_sma' for pre in prefix]] = grouped.mean().reset_index(0,drop=True)
        df[[pre+'_std' for pre in prefix]] = grouped.std().reset_index(0,drop=True)
        df[[pre+'_min' for pre in prefix]] = grouped.min().reset_index(0,drop=True)
        df[[pre+'_max' for pre in prefix]] = grouped.max().reset_index(0,drop=True)
        df[[pre+'_med' for pre in prefix]] = grouped.median().reset_index(0,drop=True)
        df[[pre+'_25Q' for pre in prefix]] = grouped.quantile(.25).reset_index(0,drop=True)
        df[[pre+'_75Q' for pre in prefix]] = grouped.quantile(.75).reset_index(0,drop=True)

在 groupby 对象内的列列表上生成滚动计算的更快方法

Faster Way to Generate Rolling Calculations on a list of columns within a groupby object

python

performance

cython

python-3.x

pandas