Spark Structured Streaming 错误读取字段时出错 'topic_metadata'

Question

我正在尝试运行一个非常简单的示例。我有一个从 Kafka 主题读取的 Kafka readStream。我是运行ning spark 2.4.0 和 Kafka 0.10.2

var streamingInputDF =
  spark.readStream
    .format("kafka")
    .option("kafka.bootstrap.servers", "localhost:9092")
    .option("subscribe", "twitter-topic")
    .load()

一台主机writeStream:

val activityQuery = streamingInputDF.writeStream
  .format("console")
  .outputMode("append")
  .start()

activityQuery.awaitTermination()

但是当我启动控制台时 writeStream 我得到以下异常

org.apache.spark.sql.streaming.StreamingQueryException: Query [id = d21cd9b4-7f51-4f5f-acbf-943dfaaeb7e5, runId = c2b2c58d-7afe-4ca5-bc36-6a3f496c19b3] terminated with exception: Error reading field 'topic_metadata': Error reading array of size 881783, only 41 bytes available
  at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:295)
  at org.apache.spark.sql.execution.streaming.StreamExecution$$anon.run(StreamExecution.scala:189)
Caused by: org.apache.kafka.common.protocol.types.SchemaException: Error reading field 'topic_metadata': Error reading array of size 881783, only 41 bytes available
  at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:73)
  at org.apache.kafka.clients.NetworkClient.parseResponse(NetworkClient.java:380)
  at org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:449)
  at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:269)
  at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:360)

Answer 1

我将 kafka-clients-0.10.2.2.jar 添加到 spark-submit 命令行，此错误消失了

Spark Structured Streaming 错误读取字段时出错 'topic_metadata'

Spark Structured Streaming error Error reading field 'topic_metadata'

apache-kafka

apache-spark

spark-structured-streaming