递归地对匹配行的数据帧求和

Question

我想通过对具有匹配变量的列求和（而不是追加列），将一组数据框组合成一个数据框。

例如给定

df1 <- data.frame(A = c(0,0,1,1,1,2,2), B = c(1,2,1,2,3,1,5), x = c(2,3,1,5,3,7,0))
df2 <- data.frame(A = c(0,1,1,2,2,2), B = c(1,1,3,2,4,5), x = c(4,8,4,1,0,3))
df3 <- data.frame(A = c(0,1,2), B = c(5,4,2), x = c(5,3,1))

我想匹配 "A" 和 "B" 并对 "x" 的值求和。对于这个例子，我可以得到想要的结果如下：

library(plyr)
library(dplyr)
# rename columns so that join_all preserves them all:
colnames(df1)[3] <- "x1"
colnames(df2)[3] <- "x2"
colnames(df3)[3] <- "x3"
# join the data frames by matching "A" and "B" values:
res <- join_all(list(df1, df2, df3), by = c("A", "B"), type = "full")
# get the sums and drop superfluous columns:
arrange(res, A, B) %>% 
  rowwise() %>% 
  mutate(x = sum(x1, x2, x3, na.rm = TRUE)) %>% 
  select(A, B, x)

结果：

       A     B     x
   <dbl> <dbl> <dbl>
 1     0     1     6
 2     0     2     3
 3     0     5     5
 4     1     1     9
 5     1     2     5
 6     1     3     7
 7     1     4     3
 8     2     1     7
 9     2     2     2
10     2     4     0
11     2     5     3

更通用的解决方案是

library(dplyr)
# function to get the desired result for two data frames:
my_merge <- function(df1, df2)
{
  m1 <- merge(df1, df2, by = c("A", "B"), all = TRUE)
  m1 <- rowwise(res) %>% 
    mutate(x = sum(x.x, x.y, na.rm = TRUE)) %>%
    select(A, B, x)
  return(m1)
}
l1 <- list(df2, df3) # omit the first data frame
res <- df1 # initial value of the result
for(df in l1) res <- my_merge(res, df) # call the function repeatedly

是否有更有效的方法来组合大量数据框？理想情况下它应该是递归的（即最好不要在计算总和之前将所有数据帧连接到一个庞大的数据帧中）。

Answer 1

一个更简单的选择是绑定数据集的行，然后按感兴趣的列分组并通过获取 'x'

的 sum 来获得汇总输出

library(tidyverse)
bind_rows(df1, df2, df3) %>% 
        group_by(A, B) %>%
        summarise(x = sum(x))
# A tibble: 11 x 3
# Groups:   A [?]
#       A     B     x
#   <dbl> <dbl> <dbl>
# 1     0     1     6
# 2     0     2     3
# 3     0     5     5
# 4     1     1     9
# 5     1     2     5
# 6     1     3     7
# 7     1     4     3
# 8     2     1     7
# 9     2     2     2
#10     2     4     0
#11     2     5     3

如果全局环境中有许多对象具有 "df" 后跟一些数字

的模式

mget(ls(pattern= "^df\d+")) %>%
        bind_rows %>%
        group_by(A, B) %>% 
        summarise(x = sum(x))

正如 OP 提到的关于 memory 约束，如果我们先执行 join 然后使用 rowSums 或 + 和 reduce，它会更有效率

mget(ls(pattern= "^df\d+")) %>% 
      reduce(full_join, by = c("A", "B")) %>%
      transmute(A, B, x = rowSums(.[3:5], na.rm = TRUE)) %>%
      arrange(A, B)
#   A B x
#1  0 1 6
#2  0 2 3
#3  0 5 5
#4  1 1 9
#5  1 2 5
#6  1 3 7
#7  1 4 3
#8  2 1 7
#9  2 2 2
#10 2 4 0
#11 2 5 3

这也可以用 data.table

来完成

library(data.table)
rbindlist(mget(ls(pattern= "^df\d+")))[, .(x = sum(x)), by = .(A, B)]

Answer 2

Ideally it should be recursive (i.e. it's better not to join all data frames into one massive data frame before calculating the sums).

如果您的内存有限并且愿意牺牲速度（与@akrun 的 data.table 方法相比），请在循环中一次使用一个 table：

library(data.table)
tabs = c("df1", "df2", "df3")

# enumerate all combos for the results table
# initializing sum to 0
res = CJ(A = 0:2, B = 1:5, x = 0)
# loop over tabs, adding on
for (i in seq_along(tabs)){
  tab = get(tabs[[i]])
  res[tab, on=.(A, B), x := x + i.x][]
  rm(tab)
}

如果您需要从磁盘读取 tables，请将 tabs 更改为文件名，将 get 更改为 fread 或任何函数。

我怀疑您能否将所有 table 放入内存中，但不能将它们的 rbind 编辑副本放在一起。

同样（感谢@akrun 的评论），成对使用他的方法：

res = data.table(get(tabs[[1]]))[0L]

for (i in seq_along(tabs)){
  tab = get(tabs[[i]])
  res = rbind(res, tab)[, .(x = sum(x)), by=.(A,B)]
  rm(tab)
}

递归地对匹配行的数据帧求和

Recursively sum data frames for matching rows

r

plyr

dplyr