Elixir Benchfella 和 Benchwarmer 的结果截然不同

Question

我正在尝试对快速排序算法与 Enum.sort 进行基准测试。我从 benchfella 和 benchwarmer.

得到了截然不同的结果

总结：

               Benchfella        Benchwarmer 
Enum.sort       8920.47 µs/op    2418767.00 μs/op
QuickSort      16660.45 µs/op      15745.04 μs/op

详情：

这是我的基准测试：

defmodule Thing do
  defstruct [:key]
end

defmodule QuickSort do
  def qsort([]), do: []
  def qsort([pivot | rest]) do
    {left, right} = Enum.partition(rest, fn(x) -> x.key < pivot.key end)
    qsort(left) ++ [pivot] ++ qsort(right)
  end
end

defmodule Bench do
  use Benchfella
  Benchfella.start

  bench "QuickSort", [list: gen()] do
    QuickSort.qsort(list)
  end

  bench "Enum.sort", [list: gen()] do
    Enum.sort(list, fn(x, y) -> x.key > y.key end)
  end

 def gen, do: for _ <- 1..10000, do: %Thing{key: :rand.uniform}

  # Tests
  list = for _ <- 1..10000, do: %Thing{key: :rand.uniform}
  sorted = Enum.sort_by(list, &(&1.key))
  true = sorted == QuickSort.qsort(list)
end

这是 Benchfella 的输出：

$ mix bench

Settings:
  duration:      1.0 s

## Bench
[20:57:40] 1/2: Enum.sort
[20:57:43] 2/2: QuickSort

Finished in 4.71 seconds

## Bench
benchmark  iterations   average time
Enum.sort         200   8920.47 µs/op
QuickSort         100   16660.45 µs/op

这是 Benchwarmer 的输出：

$ iex -S mix
Erlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]

Interactive Elixir (1.3.3) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> list = Bench.gen
[%Thing{key: 0.29976857621751096}, %Thing{key: 0.42956956935048163},
 %Thing{key: 0.8682735084348573}, %Thing{key: 0.13149039866062429},
 %Thing{key: 0.5315758481143932}, %Thing{...}, ...]

iex(2)> Benchwarmer.benchmark fn -> QuickSort.qsort list end
*** #Function<20.52032458/0 in :erl_eval.expr/5> ***
1.9 sec    127 iterations   15745.04 μs/op

[%Benchwarmer.Results{args: [], duration: 1999619,
  function: #Function<20.52032458/0 in :erl_eval.expr/5>, n: 127, prev_n: 64}]

iex(3)> Benchwarmer.benchmark fn -> Enum.sort(list, fn(x, y) -> x.key > y.key end) end
*** #Function<20.52032458/0 in :erl_eval.expr/5> ***
2.4 sec      1 iterations   2418767.0 μs/op

[%Benchwarmer.Results{args: [], duration: 2418767,
  function: #Function<20.52032458/0 in :erl_eval.expr/5>, n: 1, prev_n: 1}]

Answer 1

这很可能是由于测试 shell 中定义的函数造成的。在 shell 中，代码被评估，而不是被编译。除了已编译的模块，不要对任何东西进行基准测试。在已编译模块之外进行的任何基准测试或多或少都是无用的。

Elixir Benchfella 和 Benchwarmer 的结果截然不同

Elixir Benchfella & Benchwarmer have drastically different results

benchmarking

elixir