NoSuchMethodError : Spark + Kafka : Java

Question

用例

使用 Spark 和 Java 作为编程语言从 Kafka 主题获取和打印简单的消息

背景

有Kafka Storm Integration处理经验，开发和维护kafka集群和storm拓扑一年多。
没有使用 Apache Spark 和 Scala 的经验
使用独立 spark 集群构建并成功测试的简单字数统计应用程序。

问题

Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
    at org.apache.spark.streaming.kafka.KafkaUtils$.createStream(KafkaUtils.scala:64)
    at org.apache.spark.streaming.kafka.KafkaUtils$.createStream(KafkaUtils.scala:110)
    at org.apache.spark.streaming.kafka.KafkaUtils.createStream(KafkaUtils.scala)
    at com.random.spark.EventsToFileAggregator.main(EventsToFileAggregator.java:54)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain(SparkSubmit.scala:181)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

在 EventsToFileAggregator.java:54

JavaPairReceiverInputDStream<String, String> messages =
 KafkaUtils.createStream(jsc, args[0], args[1], topicMap,
 StorageLevel.MEMORY_AND_DISK_SER());

pom.xml

<dependencies>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.11</artifactId>
        <version>1.6.1</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming_2.11</artifactId>
        <version>1.6.1</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming-kafka_2.11</artifactId>
        <version>1.6.1</version>
    </dependency>
</dependencies>

建造

成功，没有任何警告

命令

./bin/spark-submit --class com.random.spark.EventsToFileAggregator --master spark://host:7077 /usr/local/spark/stats/target/stats-1.0-SNAPSHOT-jar-with-dependencies.jar localhost:2181 test topic 2

Answer 1

NoSuchMethodError 几乎总是表明两个库的版本不兼容。在这种情况下，Spark-Streaming Kafka 试图使用不存在的 Scala 语言功能。检查 Spark-Streaming Kafka 的版本是否与您使用的 Scala 版本兼容。确保您实际上不是运行 Scala 而不是 Java.

NoSuchMethodError : Spark + Kafka : Java

NoSuchMethodError : Spark + Kafka : Java

java

streaming

apache-kafka

apache-spark