ForeachWriter 的 "method process in class ForeachWriter of type (value: org.apache.spark.sql.Row)Unit is not defined" 是什么意思？

Question

我在将数据帧写入配置单元时尝试使用 foreach 接收器 table:

// dp is my dataframe (aggregated streaming data)
dp.writeStream.foreach(
  new ForeachWriter[Row] {
    def open(partitionId: Long, version: Long): Boolean = true
    def process(record: String): Unit = {
      //dp.createOrReplaceTempView("tableA")
      // i need to do insert into tablename (select * from tableA)
    }
    def close(errorOrNull: Throwable): Unit = {}
  }
).start()

我收到以下错误

error: object creation impossible, since method process in class ForeachWriter of type (value: org.apache.spark.sql.Row)Unit is not defined

可能是什么问题？

Answer 1

你应该更加注意类型，即比较ForeachWriter[Row]和def process(record: String): Unit中的类型。它们不兼容，因此出现错误。

来自 org.apache.spark.sql.ForeachWriter 的 scaladoc:

abstract class ForeachWriter[T]

和

abstract def process(value: T): Unit

T类型是这里的关键。使用 def process(record: Row): Unit 应该可以修复编译错误。