使用另一个数据框中的日期范围聚合值

Question

我需要使用 master_records 对 maindata 中的所有值求和。即使这些列有 timestamp 和值，ids 的许多值也不会求和。

import pandas as pd
 
#Proxy reference dataframe
master_records = [['site a', '2021-03-05 02:00:00', '2021-03-05 03:00:00'], 
        ['site a', '2021-03-05 06:00:00', '2021-03-05 08:00:00'], 
        ['site b', '2021-04-08 10:00:00', '2021-04-08 13:00:00']] 
 
mst_df = pd.DataFrame(master_records, columns = ['id', 'start', 'end'])
mst_df['start'] = pd.to_datetime(mst_df['start'], infer_datetime_format=True)
mst_df['end'] = pd.to_datetime(mst_df['end'], infer_datetime_format=True)


#Proxy main high frequency dataframe
main_data = [['id a','2021-03-05 00:00:00', 10], #not aggregated
        ['id a','2021-03-05 01:00:00', 19], #not aggregated
        ['id a','2021-03-05 02:00:00', 9], 
        ['id a','2021-03-05 03:00:00', 16],
        ['id a','2021-03-05 04:00:00', 16], #not aggregated
        ['id a','2021-03-05 05:00:00', 11], #not aggregated
        ['id a','2021-03-05 06:00:00', 16], 
        ['id a','2021-03-05 07:00:00', 12], 
        ['id a','2021-03-05 08:00:00', 9], 
        ['id b','2021-04-08 10:00:00', 11], 
        ['id b','2021-04-08 11:00:00', 10], 
        ['id b','2021-04-08 12:00:00', 19], 
        ['id b','2021-04-08 13:00:00', 10], 
        ['id b','2021-04-08 14:00:00', 16]] #not aggregated
 
# Create the pandas DataFrame
maindata = pd.DataFrame(main_data, columns = ['id', 'timestamp', 'value'])
maindata['timestamp'] = pd.to_datetime(maindata['timestamp'], infer_datetime_format=True)

所需的 DataFrame 如下所示：

print(mst_df)
id  start   end                                      sum(value)
0   site a  2021-03-05 02:00:00 2021-03-05 03:00:00  25
1   site a  2021-03-05 06:00:00 2021-03-05 08:00:00  37
2   site b  2021-04-08 10:00:00 2021-04-08 13:00:00  50

Answer 1

“id”不匹配；所以首先我们在两个 DataFrame 中创建一个列来获得匹配的 ID；然后 merge 在匹配的“id”上；然后在时间戳位于“开始”和“结束”之间的行上过滤合并的 DataFrame。最后 groupby + sum 将获取所需的结果：

maindata['id_letter'] = maindata['id'].str.split().str[-1]
mst_df['id_letter'] = mst_df['id'].str.split().str[-1]
merged = mst_df.merge(maindata, on='id_letter', suffixes=('','_'))
out = (merged[merged['timestamp'].between(merged['start'], merged['end'])]
       .groupby(['id','start','end'], as_index=False)['value'].sum())

输出：

       id               start                 end  value
0  site a 2021-03-05 02:00:00 2021-03-05 03:00:00     25
1  site a 2021-03-05 06:00:00 2021-03-05 08:00:00     37
2  site b 2021-04-08 10:00:00 2021-04-08 13:00:00     50

使用另一个数据框中的日期范围聚合值

Aggregating Values using Date Ranges in Another Dataframe

python

dataframe

pandas

pandas-groupby

pandas-merge