Pandas Dataframe 查找间隔并计算出现次数

Question

我得到了一个混合发生的不同事件的列表。例如，事件 1 可能发生三次，然后另一个事件和稍后事件 1 将再次发生。

我需要的是每个事件的时间间隔以及该事件在这些时间间隔内发生的次数。

values = {
        '2017-11-28 11:00': 'event1',
        '2017-11-28 11:01': 'event1',
        '2017-11-28 11:02': 'event1',
        '2017-11-28 11:03': 'event2',
        '2017-11-28 11:04': 'event2',
        '2017-11-28 11:05': 'event1',
        '2017-11-28 11:06': 'event1',
        '2017-11-28 11:07': 'event1',
        '2017-11-28 11:08': 'event3',
        '2017-11-28 11:09': 'event3',
        '2017-11-28 11:10': 'event2',
        }

import pandas as pd
df = pd.DataFrame.from_dict(values, orient='index').reset_index()
df.columns = ['time', 'event']
df['time'] = df['time'].apply(pd.to_datetime)
df.set_index('time', inplace=True)
df.sort_index(inplace=True)
df.head()

预期结果是：

occurrences = [
        {'start':'2017-11-28 11:00',
         'end':'2017-11-28 11:02',
         'event':'event1',
         'count':3},
        {'start':'2017-11-28 11:03',
         'end':'2017-11-28 11:04',
         'event':'event2',
         'count':2},
        {'start':'2017-11-28 11:05',
         'end':'2017-11-28 11:07',
         'event':'event1',
         'count':3},
        {'start':'2017-11-28 11:08',
         'end':'2017-11-28 11:09',
         'event':'event3',
         'count':2},
        {'start':'2017-11-28 11:10',
         'end':'2017-11-28 11:10',
         'event':'event2',
         'count':1},
        ]

我正在考虑使用 pd.merge_asof 来查找间隔的 start/end 次并使用 pd.cut (as explained here) for groupby 和 count。但不知何故我被困住了。感谢任何帮助。

Answer 1

尝试以下方法：

In [68]: x = df.reset_index()

In [69]: (x.groupby(x.event.ne(x.event.shift()).cumsum())
    ...:   .apply(lambda x:
    ...:             pd.DataFrame({
    ...:                 'start':[x['time'].min()],
    ...:                 'end':[x['time'].min()],
    ...:                 'event':[x['event'].iloc[0]],
    ...:                 'count':[len(x)]})
    ...:         )
    ...:   .reset_index(drop=True)
    ...:   .to_dict('r')
    ...: )
Out[69]:
[{'count': 3,
  'end': Timestamp('2017-11-28 11:00:00'),
  'event': 'event1',
  'start': Timestamp('2017-11-28 11:00:00')},
 {'count': 2,
  'end': Timestamp('2017-11-28 11:03:00'),
  'event': 'event2',
  'start': Timestamp('2017-11-28 11:03:00')},
 {'count': 3,
  'end': Timestamp('2017-11-28 11:05:00'),
  'event': 'event1',
  'start': Timestamp('2017-11-28 11:05:00')},
 {'count': 2,
  'end': Timestamp('2017-11-28 11:08:00'),
  'event': 'event3',
  'start': Timestamp('2017-11-28 11:08:00')},
 {'count': 1,
  'end': Timestamp('2017-11-28 11:10:00'),
  'event': 'event2',
  'start': Timestamp('2017-11-28 11:10:00')}]

或以下内容，如果您希望将 time 列作为字符串：

In [75]: (x.groupby(x.event.ne(x.event.shift()).cumsum())
    ...:   .apply(lambda x:
    ...:             pd.DataFrame({
    ...:                 'start':[x['time'].min().strftime('%Y-%m-%d %H:%M:%S')],
    ...:                 'end':[x['time'].min().strftime('%Y-%m-%d %H:%M:%S')],
    ...:                 'event':[x['event'].iloc[0]],
    ...:                 'count':[len(x)]})
    ...:         )
    ...:   .reset_index(drop=True)
    ...:   .to_dict('r')
    ...: )
Out[75]:
[{'count': 3,
  'end': '2017-11-28 11:00:00',
  'event': 'event1',
  'start': '2017-11-28 11:00:00'},
 {'count': 2,
  'end': '2017-11-28 11:03:00',
  'event': 'event2',
  'start': '2017-11-28 11:03:00'},
 {'count': 3,
  'end': '2017-11-28 11:05:00',
  'event': 'event1',
  'start': '2017-11-28 11:05:00'},
 {'count': 2,
  'end': '2017-11-28 11:08:00',
  'event': 'event3',
  'start': '2017-11-28 11:08:00'},
 {'count': 1,
  'end': '2017-11-28 11:10:00',
  'event': 'event2',
  'start': '2017-11-28 11:10:00'}]

Answer 2

这里有两种解决方法。第一个是基于vivek-harikrishnan and explained here提供的link。它为间隔创建连续数字并累积计算此类间隔内的出现次数。

#%% first solution

# create intervals and count occurrences per interval
df['interval'] = (df['event'] != df['event'].shift(1)).astype(int).cumsum()
df['count'] = df.groupby(['event', 'interval']).cumcount() + 1

# now group by intervals
df.groupby('interval').last()

第二种解决方案是基于maxu上面给出的答案。与第一个想法类似，它也创建了间隔号，但也找到了此类间隔的 start/end 时间戳。

#%% second solution

df = df.reset_index()
# create intervals
df = df.groupby(df['event'].ne(df['event'].shift()).cumsum())
# calc start/end times and count occurances at the same time
df.apply(lambda x: pd.DataFrame({
                    'start':[x['time'].min()], 
                    'end':[x['time'].max()],
                    'event':[x['event'].iloc[0]],
                    'count':[len(x)]})).reset_index(drop=True)

Pandas Dataframe 查找间隔并计算出现次数

Pandas Dataframe find intervals and count occurances

python

dataframe

pandas

pandas-groupby