将数组数组的rdd保存到文本文件spark

Question

我有一个像这样的名为 tmp 的 RDD。

"org.apache.spark.rdd.RDD[(String, List[(String, String, Double)])]"

值如下。

Array[(String, List[(String, String, Double)])] = Array((1076486,List((1076486,1076486,0.0), (1076486,431000,0.7438727490345501), (1076486,351632,3.139055446043724), (1076486,431611,6.173095256463185))), (430067,List((430067,430067,0.0), (430067,1037380,4.0390818750047535), (430067,431611,6.396930255172381), (430067,824889,7.265222659014164))))

我的输出应该是列表的内部内容，如下所示...

1076486,1076486,0.0
1076486,431000,0.7438727490345501
.
.
430067,1037380,4.0390818750047535

我试过了..

.mapValues(_.toList).saveAsTextFile

在文件中显示如下。

(1076486,List((1076486,1076486,0.0), (1076486,431000,0.7438727490345501), (1076486,351632,3.139055446043724), (1076486,431611,6.173095256463185)))
(430067,List((430067,430067,0.0), (430067,1037380,4.0390818750047535), (430067,431611,6.396930255172381), (430067,824889,7.265222659014164)))

我可以通过下面的代码打印出想要的数据

tmp.collect().foreach(a=> {a.foreach(e=>print(e+" "))})

但是无法保存到文件中。

怎样才能得到想要的结果？

Answer 1

只需手动创建输出字符串：

tmp.values.flatMap(_.map{case (x, y, z) => s"$x,$y,$z"})

将数组数组的rdd保存到文本文件spark

save rdd of array of array to text file spark

arrays

scala

list

apache-spark

rdd