流式传输选项卡未显示结构化流式传输

Question

我正在使用结构化流来读取 csvs 和写入 kafka。流媒体选项卡未在 Spark UI 中显示（未使用流媒体上下文）。

val userSchema = new StructType().add("name", "string").add("age", "integer")
val csvDF = spark
  .readStream
  .option("sep", ";")
  .schema(userSchema)      // Specify schema of the csv files
  .csv("/path/to/directory")

如何在 UI 中获取流媒体指标？

Answer 1

要查看一些指标（在控制台中），您需要添加一个侦听器

spark.streams.addListener(new StreamingQueryListener {
  override def onQueryStarted(event: StreamingQueryListener.QueryStartedEvent): Unit = logger.debug(s"QueryStarted [id = ${event.id}, name = ${event.name}, runId = ${event.runId}]")

  override def onQueryProgress(event: StreamingQueryListener.QueryProgressEvent): Unit = logger.warn(s"QueryProgress ${event.progress}")

  override def onQueryTerminated(event: StreamingQueryListener.QueryTerminatedEvent): Unit = logger.debug(s"QueryTerminated [id = ${event.id}, runId = ${event.runId}, error = ${event.exception}]")
})

QueryProgressEvent，显示偏移量、水印、源、汇等信息

这个视频可以帮到你：Monitoring Structured Streaming Applications

流式传输选项卡未显示结构化流式传输

Streaming tab is not showing for structured streaming

scala

apache-spark

spark-streaming

spark-structured-streaming

spark-ui