像 Guava Splitter 一样快速地搜索大型 CSV

Question

自从 Java 8 发布后，我发现我的项目中不需要超过 2 MB Google 番石榴，因为我可以用普通的 Java 替换其中的大部分。但是我真的很喜欢 nice Splitter API，它同时都非常快。最重要的是 - 确实 懒惰地 拆分了。它似乎可以用 Pattern.splitAsStream 代替。所以我准备了快速测试 - 在长字符串的中间找到一个值（即拆分整个字符串没有意义）。

package splitstream;


import com.google.common.base.Splitter;
import org.junit.Assert;
import org.junit.Test;

import java.util.StringTokenizer;
import java.util.regex.Pattern;
import java.util.stream.Collectors;
import java.util.stream.IntStream;

public class SplitStreamPerfTest {

    private static final int TIMES = 1000;
    private static final String FIND = "10000";

    @Test
    public void go() throws Exception {
        final String longString = IntStream.rangeClosed(1,20000).boxed()
                .map(Object::toString)
                .collect(Collectors.joining(" ,"));

        IntStream.rangeClosed(1,3).forEach((i) -> {
            measureTime("Test " + i + " with regex", () -> doWithRegex(longString));
            measureTime("Test " + i + " with string tokenizer", () -> doWithStringTokenizer(longString));
            measureTime("Test " + i + " with guava", () -> doWithGuava(longString));
        });

    }

    private void measureTime(String name, Runnable r) {
        long s = System.currentTimeMillis();
        r.run();
        long elapsed = System.currentTimeMillis() - s;
        System.out.println("Check " + name +" took " + elapsed + " ms");
    }

    private void doWithStringTokenizer(String longString) {

        String f = null;
        for (int i = 0; i < TIMES; i++) {
            StringTokenizer st = new StringTokenizer(longString,",",false);
            while (st.hasMoreTokens()) {
                String t = st.nextToken().trim();
                if (FIND.equals(t)) {
                    f = t;
                    break;
                }
            }
        }
        Assert.assertEquals(FIND, f);
    }


    private void doWithRegex(String longString) {
        final Pattern pattern = Pattern.compile(",");
        String f = null;
        for (int i = 0; i < TIMES; i++) {
            f = pattern.splitAsStream(longString)
                    .map(String::trim)
                    .filter(FIND::equals)
                    .findFirst().orElse("");
        }
        Assert.assertEquals(FIND, f);
    }


    private void doWithGuava(String longString) {
        final Splitter splitter = Splitter.on(',').trimResults();
        String f = null;
        for (int i = 0; i < TIMES; i++) {
            Iterable<String> iterable = splitter.split(longString);
            for (String s : iterable) {
                if (FIND.equals(s)) {
                    f = s;
                    break;
                }
            }
        }
        Assert.assertEquals(FIND, f);
    }
}

结果是（热身后）

Check Test 3 with regex took 1359 ms
Check Test 3 with string tokenizer took 750 ms
Check Test 3 with guava took 594 ms

如何使 Java 实现与 Guava 一样快？也许我做错了？

或者您可能知道任何 tool/library 与 Guava Splitter 一样快，并且不涉及为了这个而拉出大量未使用的类？

Answer 1

这可能会有用，您可以在番石榴中只导入您需要的部分： https://github.com/google/guava/wiki/UsingProGuardWithGuava

Answer 2

你能给出 pattern.split(text) 并在正常的 for 循环中迭代结果吗？试试看。它可能比流更快。虽然我不确定它是否会打败 Guava。

我是这个意思..

private void doWithRegexAndSplit(String longString) {
        final Pattern pattern = Pattern.compile(",");
        for (int i = 0; i < TIMES; i++) {
         String f = "";
         String[] arr = pattern.split(longString);
            for (int i = 0; i < arr.length; i++){
                String t= arr[i].trim();
                if (FIND.equals(t)) {
                f = t;
                break;
                }
            }       
        }
        Assert.assertEquals(FIND, f);
    }

请检查此案例的完成时间。

Answer 3

首先，番石榴 比 Splitter、Predicate 和 Function 多 - 您可能没有使用它所提供的一切；我们使用它是铁杆，只是听到它让我颤抖。无论如何，您的测试被破坏了——可能以多种方式。我使用 JMH 来测试这两种方法只是为了好玩：

    @BenchmarkMode(org.openjdk.jmh.annotations.Mode.AverageTime) 
    @OutputTimeUnit(TimeUnit.NANOSECONDS) 
    @Warmup(iterations = 5, time = 2, timeUnit = TimeUnit.SECONDS)   
    @Measurement(iterations = 5, time = 2, timeUnit = TimeUnit.SECONDS) 
    @State(Scope.Thread) public class GuavaTest {

    public static void main(String[] args) throws RunnerException {
        Options opt = new OptionsBuilder().include(GuavaTest.class.getSimpleName())
                .jvmArgs("-ea", "-Xms10g", "-Xmx10g")
                .shouldFailOnError(true)
                .build();
        new Runner(opt).run();
    }

    @Param(value = { "300", "1000" })
    public String tokenToSearchFor;

    @State(Scope.Benchmark)
    public static class ThreadState {
        String longString = IntStream.range(1, 20000).boxed().map(Object::toString).collect(Collectors.joining(" ,"));

        StringTokenizer st = null;

        Pattern pattern = null;

        Splitter splitter = null;

        @Setup(Level.Invocation)
        public void setUp() {
            st = new StringTokenizer(longString, ",", false);
            pattern = Pattern.compile(",");
            splitter = Splitter.on(',').trimResults();
        }
    }

    @Benchmark
    @Fork(1)
    public boolean doWithStringTokenizer(ThreadState ts) {
        while (ts.st.hasMoreTokens()) {
            String t = ts.st.nextToken().trim();
            if (t.equals(tokenToSearchFor)) {
                return true;
            }
        }
        return false;
    }

    @Benchmark
    @Fork(1)
    public boolean doWithRegex(ThreadState ts) {
        return ts.pattern.splitAsStream(ts.longString)
                .map(String::trim)
                .anyMatch(tokenToSearchFor::equals);
    }

    @Benchmark
    @Fork(1)
    public boolean doWithGuava(ThreadState ts) {
        Iterable<String> iterable = ts.splitter.split(ts.longString);
        for (String s : iterable) {
            if (s.equals(tokenToSearchFor)) {
                return true;
            }
        }
        return false;
    }

}

结果：

Benchmark                        (tokenToSearchFor)  Mode  Cnt       Score        Error  Units
GuavaTest.doWithGuava                           300  avgt    5   19284.192 ±  23536.321  ns/op
GuavaTest.doWithGuava                          1000  avgt    5   67182.531 ±  93242.266  ns/op
GuavaTest.doWithRegex                           300  avgt    5   65780.954 ± 169044.641  ns/op
GuavaTest.doWithRegex                          1000  avgt    5  182530.069 ± 409571.222  ns/op
GuavaTest.doWithStringTokenizer                 300  avgt    5   34111.030 ±  61014.332  ns/op
GuavaTest.doWithStringTokenizer                1000  avgt    5  118963.048 ± 165510.183  ns/op

这使得番石榴确实是最快的。

如果在splitAsStream后面加上parallel就会变得有趣，必读

Answer 4

您正在比较Pattern.splitAsStream(CharSequence) to Splitter.split(CharSequence) on a Splitter.on(char) instead of on a Splitter.onPattern(String)。查找与字符的匹配在计算上比查找与模式 (regex) 的匹配要简单得多。

如果你使用 Splitter.onPattern(",").trimResults() 那么你会得到如下结果：

Check Test 3 with regex took 608 ms
Check Test 3 with string tokenizer took 403 ms
Check Test 3 with guava took 306 ms
Check Test 3 with guava pattern took 689 ms

在这种情况下 Pattern.splitAsStrimg(CharSequence) 实际上比 Guava 的实现表现得更好（假设这是一个有效的基准，这总是值得怀疑的，因为我们没有使用 jmh）。

我不知道有任何类似于 Guava Splitter.on(char).split(CharSequence) 的 JDK char 定界拆分解决方案。您可以自己推出，但 Guava 的解决方案似乎非常优化。

像 Guava Splitter 一样快速地搜索大型 CSV

Search in large CSV as fast as Guava Splitter

java

stream

guava