Spark group by - 猪转换

Question

我正试图在 spark 中实现这样的目标。以下代码片段来自 Pig Latin。无论如何，我可以用 Spark 做同样的事情吗？

    A = load 'student' AS (name:chararray,age:int,gpa:float);
    DESCRIBE A; 

    A: {name: chararray,age: int,gpa: float} DUMP A; (John,18,4.0F)
    (Mary,19,3.8F) (Bill,20,3.9F) (Joe,18,3.8F)

    B = GROUP A BY age;

    Result:  (18,{(John,18,4.0F),(Joe,18,3.8F)}) (19,{(Mary,19,3.8F)})
    (20,{(Bill,20,3.9F)})

谢谢。

Answer 1

按年龄列名单很容易。我相信 Spark API 不允许您以相同的方式收集完整的行并获得完整的行列表。

// Input data
val df = {
    import org.apache.spark.sql._
    import org.apache.spark.sql.types._
    import scala.collection.JavaConverters._
    import java.time.LocalDate

    val simpleSchema = StructType(
        StructField("name", StringType) ::
        StructField("age", IntegerType) ::
        StructField("gpa", FloatType) :: Nil)

    val data = List(
        Row("John", 18, 4.0f),
        Row("Mary", 19, 3.8f),
        Row("Bill", 20, 3.9f),
        Row("Joe", 18, 3.8f)
    )    

    spark.createDataFrame(data.asJava, simpleSchema)
}
df.show()
val df2 = df.groupBy(col("age")).agg(collect_list(col("name")))
df2.show()

Spark group by - 猪转换

Spark group by - Pig conversion

apache-pig

apache-spark