r data.table 语言上的函数式编程/元编程/计算

Question

我正在探索使用 data.table 包装聚合函数（但实际上它可以是任何类型的函数）的不同方法（还提供了一个 dplyr 示例）并且想知道函数式编程的最佳实践 /关于

的元编程

性能（关于 data.table 可能适用的潜在优化，实施是否重要）
可读性（是否有一个普遍认可的标准，例如在大多数使用 data.table 的包中）
易于泛化（元编程的方式是否存在差异"generalizable"）

基本应用是灵活聚合一个table，即参数化聚合的变量、聚合的维度、两者各自的结果变量名和聚合函数。我已经（几乎）以三种 data.table 和一种 dplyr 方式实现了相同的功能：

fn_dt_agg1（这里我想不通如何参数化聚合函数）
fn_dt_agg2（灵感来自@jangorecki 的回答，他称之为"computing on the language"）
fn_dt_agg3（灵感来自@Arun 的回答here，这似乎是元编程的另一种方法）
fn_df_agg1（我在 dplyr 中的谦虚方法）

图书馆

library(data.table)
library(dplyr)

数据

n_size <- 1*10^6
sample_metrics <- sample(seq(from = 1, to = 100, by = 1), n_size, rep = T)
sample_dimensions <- sample(letters[10:12], n_size, rep = T)
df <- 
  data.frame(
    a = sample_metrics,
    b = sample_metrics,
    c = sample_dimensions,
    d = sample_dimensions,
    x = sample_metrics,
    y = sample_dimensions,
    stringsAsFactors = F)

dt <- as.data.table(df)

实现

1. fn_dt_agg1

fn_dt_agg1 <- 
  function(dt, metric, metric_name, dimension, dimension_name) {

  temp <- dt[, setNames(lapply(.SD, function(x) {sum(x, na.rm = T)}), 
                        metric_name), 
             keyby = dimension, .SDcols = metric]
  temp[]
}

res_dt1 <- 
  fn_dt_agg1(
    dt = dt, metric = c("a", "b"), metric_name = c("a", "b"),
    dimension = c("c", "d"), dimension_name = c("c", "d"))

2。 fn_dt_agg2

fn_dt_agg2 <- 
  function(dt, metric, metric_name, dimension, dimension_name,
           agg_type) {

  j_call = as.call(c(
    as.name("."),
    sapply(setNames(metric, metric_name), 
           function(var) as.call(list(as.name(agg_type), 
                                      as.name(var), na.rm = T)), 
           simplify = F)
    ))

  dt[, eval(j_call), keyby = dimension][]
}

res_dt2 <- 
  fn_dt_agg2(
    dt = dt, metric = c("a", "b"), metric_name = c("a", "b"),
    dimension = c("c", "d"), dimension_name = c("c", "d"),
    agg_type = c("sum"))

all.equal(res_dt1, res_dt2)
#TRUE

3。 fn_dt_agg3

fn_dt_agg3 <- 
  function(dt, metric, metric_name, dimension, dimension_name, agg_type) {

  e <- eval(parse(text=paste0("function(x) {", 
                              agg_type, "(", "x, na.rm = T)}"))) 

  temp <- dt[, setNames(lapply(.SD, e), 
                        metric_name), 
             keyby = dimension, .SDcols = metric]
  temp[]
}

res_dt3 <- 
  fn_dt_agg3(
    dt = dt, metric = c("a", "b"), metric_name = c("a", "b"),
    dimension = c("c", "d"), dimension_name = c("c", "d"), 
    agg_type = "sum")

all.equal(res_dt1, res_dt3)
#TRUE

4. fn_df_agg1

fn_df_agg1 <-
  function(df, metric, metric_name, dimension, dimension_name, agg_type) {

    all_vars <- c(dimension, metric)
    all_vars_new <- c(dimension_name, metric_name)
    dots_group <- lapply(dimension, as.name)

    e <- eval(parse(text=paste0("function(x) {", 
                                agg_type, "(", "x, na.rm = T)}")))

    df %>%
      select_(.dots = all_vars) %>%
      group_by_(.dots = dots_group) %>%
      summarise_each_(funs(e), metric) %>%
      rename_(.dots = setNames(all_vars, all_vars_new))
}

res_df1 <- 
  fn_df_agg1(
    df = df, metric = c("a", "b"), metric_name = c("a", "b"),
    dimension = c("c", "d"), dimension_name = c("c", "d"),
    agg_type = "sum")

all.equal(res_dt1, as.data.table(res_df1))
#"Datasets has different keys. 'target': c, d. 'current' has no key."

基准测试

出于好奇以及为了我未来的自己和其他感兴趣的各方，我运行对所有 4 种实现进行了基准测试，这可能已经阐明了性能问题（尽管我不是基准测试专家，所以如果我没有应用公认的最佳实践，请原谅）。我原以为 fn_dt_agg1 是最快的，因为它少了一个参数（聚合函数），但这似乎没有太大的影响。我也对相对较慢的 dplyr 功能感到惊讶，但这可能是由于我的设计选择不当造成的。

library(microbenchmark)
bench_res <- 
  microbenchmark(
    fn_dt_agg1 = 
      fn_dt_agg1(
      dt = dt, metric = c("a", "b"), 
      metric_name = c("a", "b"), 
      dimension = c("c", "d"), 
      dimension_name = c("c", "d")), 
    fn_dt_agg2 = 
      fn_dt_agg2(
        dt = dt, metric = c("a", "b"), 
        metric_name = c("a", "b"), 
        dimension = c("c", "d"), 
        dimension_name = c("c", "d"),
        agg_type = c("sum")),
    fn_dt_agg3 =
      fn_dt_agg3(
        dt = dt, metric = c("a", "b"), 
        metric_name = c("a", "b"),
        dimension = c("c", "d"), 
        dimension_name = c("c", "d"),
        agg_type = c("sum")),
    fn_df_agg1 =
      fn_df_agg1(
        df = df, metric = c("a", "b"), metric_name = c("a", "b"),
        dimension = c("c", "d"), dimension_name = c("c", "d"),
        agg_type = "sum"),
    times = 100L)

bench_res

# Unit: milliseconds
#       expr      min       lq     mean   median       uq       max neval
# fn_dt_agg1 28.96324 30.49507 35.60988 32.62860 37.43578 140.32975   100
# fn_dt_agg2 27.51993 28.41329 31.80023 28.93523 33.17064  84.56375   100
# fn_dt_agg3 25.46765 26.04711 30.11860 26.64817 30.28980 153.09715   100
# fn_df_agg1 88.33516 90.23776 97.84826 94.28843 97.97154 172.87838   100

其他资源

Advanced R by Hadley Wickham: Expressions
Advanced R by Hadley Wickham: Functions
CRAN R Language Definition: Computing on the language
CRAN Non-standard evaluation
Data.table FAQ: Programmatically passing expressions in j
Data.table meta-programming
How can one work fully generically in data.table in R with column names in variables
Using get inside lapply, inside a function

Answer 1

我不推荐eval(parse())。如果没有它，您可以实现与方法三相同的效果：

fn_dt_agg4 <- 
  function(dt, metric, metric_name, dimension, dimension_name, agg_type) {

    e <- function(x) getFunction(agg_type)(x, na.rm = T)

    temp <- dt[, setNames(lapply(.SD, e), 
                          metric_name), 
               keyby = dimension, .SDcols = metric]
    temp[]
  }

这也避免了一些安全风险。

PS：您可以通过设置 options("datatable.verbose" = TRUE).

检查 data.table 在优化方面做了什么

r data.table 语言上的函数式编程/元编程/计算

r data.table functional programming / metaprogramming / computing on the language

functional-programming

r

metaprogramming

dplyr

data.table