如何检查点火花流的许多来源

Question

我在不同的位置有很多 CSV spark.readStream，我必须用 scala 检查所有这些，我为每个流指定了一个查询但是当我运行工作时，我得到了这个留言

java.lang.IllegalArgumentException: Cannot start query with name "query1" as a query with that name is already active

我通过像这样创建许多流式查询解决了我的问题：

val spark = SparkSession
    .builder
    .appName("test")
    .config("spark.local", "local[*]")
    .getOrCreate()
spark.sparkContext.setCheckpointDir(path_checkpoint)
val event1 = spark  
.readStream //  
.schema(schema_a)  
.option("header", "true")    
.option("sep", ",")    
.csv(path_a) 

val query = event1.writeStream
  .outputMode("append")
  .format("console")
  .start()
   spark.streams.awaitAnyTermination()

Answer 1

val spark = SparkSession
    .builder
    .appName("test")
    .config("spark.local", "local[*]")
    .getOrCreate()
spark.sparkContext.setCheckpointDir(path_checkpoint)
val event1 = spark  
.readStream //  
.schema(schema_a)  
.option("header", "true")    
.option("sep", ",")    
.csv(path_a) 

val query = event1.writeStream
  .outputMode("append")
  .format("console")
  .start()
   spark.streams.awaitAnyTermination()

如何检查点火花流的许多来源

How to checkpoint many source of spark streaming

hadoop

scala

spark-streaming

apache-spark-sql