如何使这些函数定义 D.R.Y.?

Question

我有多个函数定义来编写 parquet 文件，但我宁愿有一个通用函数。我怎样才能使一个通用功能起作用？问题是一个通用函数 returns error
could not find implicit value for parameter writerFactory: com.github.mjakubowski84.parquet4s.ParquetWriter.ParquetWriterFactory[Product]

多项功能：

import com.github.mjakubowski84.parquet4s.ParquetWriter
import org.apache.parquet.hadoop.metadata.CompressionCodecName
import org.apache.parquet.hadoop.ParquetFileWriter
import java.time.format.DateTimeFormatter
import java.time.LocalDate

def writeParquetUnsegmented(rows: List[ACaseClass], date: String) = {
    val outPath = s"s3a://redacted/path-$date.parquet"
    ParquetWriter.writeAndClose(outPath,
                                rows,
                                    ParquetWriter.Options(
                                                        writeMode = ParquetFileWriter.Mode.OVERWRITE,
                                                        compressionCodecName = CompressionCodecName.SNAPPY
                                                        ))
}

def writeParquetSegmented(rows: List[BCaseClass], date: String) = {
    val outPath = s"s3a://redacted/path-$date.parquet"
    ParquetWriter.writeAndClose(outPath,
                                rows,
                                    ParquetWriter.Options(
                                                        writeMode = ParquetFileWriter.Mode.OVERWRITE,
                                                        compressionCodecName = CompressionCodecName.SNAPPY
                                                        ))
}

一个函数：

def writeParquetSegmented(rows: List[Product], date: String) = {
    val outPath = s"s3a://redacted/path-$date.parquet"
    ParquetWriter.writeAndClose(outPath,
                                rows,
                                    ParquetWriter.Options(
                                                        writeMode = ParquetFileWriter.Mode.OVERWRITE,
                                                        compressionCodecName = CompressionCodecName.SNAPPY
                                                        ))
}

我也尝试过不同的函数签名，但得到了同样的错误。

def writeParquetSegmented[A](rows: List[A], date: String)

def writeParquetSegmented[A :< RowParent](rows: List[A], date: String)

Answer 1

只需使函数通用：

def writeParquetUnsegmented[A](
  rows: List[A],
  date: String)(
  implicit writerFactory: ParquetWriterFactory[A]) = {
    val outPath = s"s3a://redacted/path-$date.parquet"
    ParquetWriter.writeAndClose(
      outPath,
      rows,
      ParquetWriter.Options(
        writeMode = ParquetFileWriter.Mode.OVERWRITE,
        compressionCodecName = CompressionCodecName.SNAPPY))
}

Answer 2

我会试试这个签名：

def writeParquetSegmented[A : ParquetWriterFactory](rows: List[A], date: String)

编译器会将 context bound [A : ParquetWriterFactory] 转换为错误消息所抱怨的隐式参数。

def writeParquetSegmented[A](rows: List[A], date: String)(implicit pwf: ParquetWriterFactory)

如何使这些函数定义 D.R.Y.?

How do I make these function definitions D.R.Y.?

hadoop

scala

amazon-web-services

parquet