实例化 JavaStreamingContext 时出现 AbstractMethodError 异常

Question

我在创建 JavaStreamingContext 时收到 AbstractMethodError 异常。我的依赖 pom 如下；无法找到线索，任何人都可以建议这里出了什么问题吗？

    <dependency> <!-- Spark dependency -->
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.11</artifactId>
        <version>2.3.1</version>
        <scope>provided</scope>
    </dependency>
    <dependency> <!-- Spark dependency -->
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.11</artifactId>
        <version>2.3.1</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>commons-io</groupId>
        <artifactId>commons-io</artifactId>
        <version>2.5</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming -->
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming_2.11</artifactId>
        <version>2.2.0</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kafka-0-10 -->
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
        <version>2.0.0</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka -->
    <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka_2.10</artifactId>
        <version>0.8.2.2</version>
    </dependency>

线程异常 "main" java.lang.AbstractMethodError 在 org.apache.spark.util.ListenerBus$class.$init$(ListenerBus.scala:35) 在 org.apache.spark.streaming.scheduler.StreamingListenerBus.(流媒体ListenerBus.scala:30) 在 org.apache.spark.streaming.scheduler.JobScheduler.(JobScheduler.scala:57) 在 org.apache.spark.streaming.StreamingContext.(StreamingContext.scala:184) 在 org.apache.spark.streaming.StreamingContext.(StreamingContext.scala:76) 在 org.apache.spark.streaming.api.java.JavaStreamingContext.(JavaStreamingContext.scala:130)

Answer 1

您在这里混用了多个版本的 Spark
首先，如果您使用的是 Apache Spark 2.3.1 和 kafka 0.10+

我建议如下：

<dependency> <!-- Spark dependency -->
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.11</artifactId>
        <version>2.3.1</version>
        <scope>provided</scope>
    </dependency>
    <dependency> <!-- Spark dependency -->
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.11</artifactId>
        <version>2.3.1</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>commons-io</groupId>
        <artifactId>commons-io</artifactId>
        <version>2.5</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming -->
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming_2.11</artifactId>
        <!-- Keep the same Spark version as before -->
        <version>2.3.1</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kafka-0-10 -->
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
        <!-- Keep the same Spark version as before -->
        <version>2.3.1</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka -->
    <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka_2.10</artifactId>
        <!-- Support for kafka version 0.8 is deprecated as of Spark 2.3.1 and you add the dependencies for kafka 0.10+ above -->
        <version><your_kafka_version_0.10+></version>
    </dependency>

很高兴知道您 build/deploy 您的应用程序如何？根据您的运行时环境，您可能希望添加一些提供的范围，以防止构建包与现有环境之间发生冲突。

希望对您有所帮助

实例化 JavaStreamingContext 时出现 AbstractMethodError 异常

AbstractMethodError exception while instantiating JavaStreamingContext

apache-spark

spark-streaming