Pandas 在日期时间之间左联接

Question

我必须数据帧 - df 和 gdf

from datetime import datetime
import pandas as pd

data = [['foo', datetime(2020,1,1,0,0,0) ], ['foo', datetime(2020,2,1,0,0,0)], ['foo', datetime(2020,3,1,0,0,0)],
       ['bar', datetime(2020,4,1,0,0,0)],['bar', datetime(2020,5,1,0,0,0)],['bar', datetime(2020,6,1,0,0,0)]]
df = pd.DataFrame(data, columns = ['id', 'timestamp'])

data = [['A', datetime(2020,1,15,0,0,0), datetime(2020,3,15,0,0,0) ], ['B', datetime(2020,4,15,0,0,0),datetime(2020,6,15,0,0,0)]]
gdf = pd.DataFrame(data, columns = ['geoid', 'starttime', 'endtime'])


df
    id  timestamp
0   foo 2020-01-01
1   foo 2020-02-01
2   foo 2020-03-01
3   bar 2020-04-01
4   bar 2020-05-01
5   bar 2020-06-01

gdf
    geoid starttime     endtime
0   A     2020-01-15    2020-03-15
1   B     2020-04-15    2020-06-15

我的目标是在 df 上左连接 gdf，其中 timestamp 在 starttime 和 endtime 之间，这样输出看起来像：

res
    id  timestamp   geoid
0   foo 2020-01-01  None
1   foo 2020-02-01  A
2   foo 2020-03-01  A
3   bar 2020-04-01  None
4   bar 2020-05-01  B
5   bar 2020-06-01  B

据我研究，pandas 中唯一存在于 pandas 中的时间连接方法是 pandas.merge_asof()，它不适合这个用例，因为目标是在时间戳之间合并，而不是在最接近的时间戳之间合并。

pandas（不使用 sqllite）根据重叠时间戳将一个 table 与另一个（左连接）合并的正确方法是什么？

Answer 1

如果可能，使用由 gdf 列创建的 IntervalIndex，然后通过 Index.get_indexer 获取位置并通过在 numpy 中使用 None 进行索引获取 geoid if -1（不匹配）：

s = pd.IntervalIndex.from_arrays(gdf['starttime'], gdf['endtime'], closed='both')

arr = gdf['geoid'].to_numpy()
pos = s.get_indexer(df['timestamp'])

df['new'] = np.where(pos != -1, arr[pos], None)
print (df)
    id  timestamp   new
0  foo 2020-01-01  None
1  foo 2020-02-01     A
2  foo 2020-03-01     A
3  bar 2020-04-01  None
4  bar 2020-05-01     B
5  bar 2020-06-01     B

或者交叉连接的解决方案，将 df 的索引转换为 reset_index 的列以避免丢失索引值，并在 Series.between with DataFrame.loc, last add new column by DataFrame.set_index 中过滤以匹配 index 列df.index:

df1 = df.reset_index().assign(a=1).merge(gdf.assign(a=1), on='a')
df1 = df1.loc[df1['timestamp'].between(df1['starttime'], df1['endtime']), ['index','geoid']]

df['geoid'] = df1.set_index('index')['geoid']
print (df)
    id  timestamp geoid
0  foo 2020-01-01   NaN
1  foo 2020-02-01     A
2  foo 2020-03-01     A
3  bar 2020-04-01   NaN
4  bar 2020-05-01     B
5  bar 2020-06-01     B

Answer 2

您可以创建一个虚拟列并使用 df.merge:

In [1460]: df['tmp'] = 1
In [1461]: gdf['tmp'] = 1

In [1463]: x = df.merge(gdf) # merge on `tmp` column.

# assign None to geoid where timestamp is not in range
In [1465]: import numpy as np
In [1466]: x['geoid'] = np.where(x['timestamp'].between(x.starttime, x.endtime), x.geoid, None) 

# groupby and pick the correct geoid
In [1477]: ans = x.groupby(['id', 'timestamp'])['geoid'].first().reset_index()

In [1478]: ans
Out[1478]: 
    id  timestamp geoid
0  bar 2020-04-01  None
1  bar 2020-05-01     B
2  bar 2020-06-01     B
3  foo 2020-01-01  None
4  foo 2020-02-01     A
5  foo 2020-03-01     A

Answer 3

non-equi 加入的一个选项是 conditional_join from pyjanitor；在引擎盖下，它使用二进制搜索来避免笛卡尔积；它还可以处理重叠间隔：

# pip install pyjanitor
import pandas as pd
import janitor

(
df
.conditional_join(
    gdf, 
   ("timestamp", "starttime", ">="), 
   ("timestamp", "endtime", "<="), 
    how="left")
.loc[:, ['id', 'timestamp', 'geoid']]
)

    id  timestamp geoid
0  foo 2020-01-01   NaN
1  foo 2020-02-01     A
2  foo 2020-03-01     A
3  bar 2020-04-01   NaN
4  bar 2020-05-01     B
5  bar 2020-06-01     B

Pandas 在日期时间之间左联接

Pandas left join between datetimes

python

merge

left-join

dataframe

pandas