构建非重叠日期时间记录（开始、结束日期时间）数据框

Question

我需要为多个 ids 创建一个删除重叠 start 和 end 日期时间的数据框。我将使用 start 和 end 日期时间来聚合高频 pandas 数据框中的值，因此我需要删除 mst_df.

中那些重叠的日期时间

import pandas as pd
 
#Proxy reference dataframe
master = [['site a', '2021-07-08 00:00:00', '2021-07-08 10:56:00'], 
        ['site a', '2021-07-08 06:00:00',   '2021-07-08 12:00:00'], #slightly overlapping
        ['site a', '2021-07-08 17:36:00',   '2021-07-09 11:40:00'],
        ['site a', '2021-07-08 18:00:00',   '2021-07-09 11:40:00'], #overlapping
        ['site a', '2021-07-09 00:00:00',   '2021-07-09 05:40:00'], #overlapping
        ['site b', '2021-07-08 00:00:00',   '2021-07-08 10:24:00'],
        ['site b', '2021-07-08 06:00:00',   '2021-07-08 10:24:00'], #overlapping
        ['site b', '2021-07-08 17:32:00',   '2021-07-09 11:12:00'],
        ['site b', '2021-07-08 18:00:00',   '2021-07-09 11:12:00'], #overlapping
        ['site b', '2021-07-09 00:00:00',   '2021-07-09 13:00:00']] #slightly overlapping

 
mst_df = pd.DataFrame(master, columns = ['id', 'start', 'end'])
mst_df['start'] = pd.to_datetime(mst_df['start'], infer_datetime_format=True)
mst_df['end'] = pd.to_datetime(mst_df['end'], infer_datetime_format=True)

所需的数据帧：

    id      start               end
    site a  2021-07-08 00:00:00 2021-07-08 12:00:00
    site a  2021-07-08 17:36:00 2021-07-09 11:40:00
    site b  2021-07-08 00:00:00 2021-07-08 10:24:00
    site b  2021-07-08 17:32:00 2021-07-09 13:00:00

Answer 1

不知道pandas有没有这方面的特殊功能。它有 Interval.overlaping() 来检查两个范围是否重叠（它甚至可以与 datetime 一起工作）但我没有看到合并这两个范围的功能，所以它仍然需要自己的代码来合并。幸运的是这很容易。

行按 start 排序，因此当 previous_end < next_start 时行不重叠，我在 for-loop.

中使用它

但首先我按 site 分组以单独处理每个站点。

接下来我得到第一行（如 previous）和运行与其他行循环（如 next) 并检查 previous_end < next_start。

如果它是 True 那么我可以将 previous 放在结果列表中并得到 next 作为 previous 以处理其余行。

如果它是 False，那么我从两行创建新范围并使用它来处理其余行。

最后我将 previous 添加到列表中。

处理所有组后，我将它们全部转换为 DataFrame。

import pandas as pd
 
#Proxy reference dataframe
master = [
    ['site a', '2021-07-08 00:00:00',   '2021-07-08 10:56:00'], 
    ['site a', '2021-07-08 06:00:00',   '2021-07-08 12:00:00'], # slightly overlapping
    ['site a', '2021-07-08 17:36:00',   '2021-07-09 11:40:00'],
    ['site a', '2021-07-08 18:00:00',   '2021-07-09 11:40:00'], # overlapping
    ['site a', '2021-07-09 00:00:00',   '2021-07-09 05:40:00'], # overlapping
    ['site b', '2021-07-08 00:00:00',   '2021-07-08 10:24:00'],
    ['site b', '2021-07-08 06:00:00',   '2021-07-08 10:24:00'], # overlapping
    ['site b', '2021-07-08 17:32:00',   '2021-07-09 11:12:00'],
    ['site b', '2021-07-08 18:00:00',   '2021-07-09 11:12:00'], # overlapping
    ['site b', '2021-07-09 00:00:00',   '2021-07-09 13:00:00']  # slightly overlapping
]

mst_df = pd.DataFrame(master, columns = ['id', 'start', 'end'])

mst_df['start'] = pd.to_datetime(mst_df['start'], infer_datetime_format=True)
mst_df['end']   = pd.to_datetime(mst_df['end'], infer_datetime_format=True)

result = []

for val, group in mst_df.groupby('id'):
    
    # get first
    prev = group.iloc[0]
    
    for idx, item in group[1:].iterrows():
        if prev['end'] < item['start']:
            # not overlapping - put previous to results and use next as previous
            result.append(prev)
            prev = item
        else:
            # overlappig - create on range start, end
            prev['start'] = min(prev['start'], item['start'])
            prev['end']   = max(prev['end'], item['end'])
    
    # add when there is no next item
    result.append(prev)

print(pd.DataFrame(result))

结果：

       id               start                 end
0  site a 2021-07-08 00:00:00 2021-07-08 12:00:00
2  site a 2021-07-08 17:36:00 2021-07-09 11:40:00
5  site b 2021-07-08 00:00:00 2021-07-08 10:24:00
7  site b 2021-07-08 17:32:00 2021-07-09 13:00:00

构建非重叠日期时间记录（开始、结束日期时间）数据框

Construct non-overlapping datetime record (start, end datetime) dataframe

python

datetime

pandas