从 spark dataframe 或 RDD 生成键值对，键中存在列名

Question

我有一个 spark 数据框，我需要如下所示的键值对。我特别需要键中的列名。我想使用单个映射器传递来执行此操作。

原始数据集：

期望键值对:(Attribute_Name,Attribute_Value,Class),1

单次映射后的预期结果：

预期数据集

Answer 1

这应该有帮助：

import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.sql.functions.{explode, udf, typedLit}
import org.apache.spark.sql.SparkSession

object test extends App {


  val conf: SparkConf = new SparkConf().setAppName("test").setMaster("local[*]")

  val sc: SparkContext = new SparkContext(conf)

    val spark = SparkSession
      .builder()
      .appName("test")
      .master("local[*]")
      .getOrCreate()

    import spark.implicits._

  val df = spark.read.format("csv").option("header", true).load("file:///Users/test/Desktop/file2.csv")

  val header: Seq[String] = df.columns.toSeq.map(x => x.trim)

  val df1 = df.withColumn("header", typedLit(header))

  val transform = udf((col0: String, col1: String, col2: String, col3: String, header: Seq[String]) => {
    Array(
      ((header(0), col0.trim, col3.trim),1),
      ((header(1), col1.trim, col3.trim),1),
      ((header(2), col2.trim, col3.trim),1)
    )
  })

  val df2 = df1.withColumn("transformed",transform($"A1", $" A2", $" A3", $" Class", $"header"))
    .withColumn("exploded", explode($"transformed"))
    .select($"exploded")

  df2.take(1).foreach(println)
}

输出：https://imgur.com/a/Je1M3Dx

从 spark dataframe 或 RDD 生成键值对，键中存在列名

Generate key value pairs from spark dataframe or RDD with column name present in key

hadoop

scala

decision-tree

apache-spark

apache-spark-sql