spark.debug.maxToStringFields 无效

Question

我尝试按照消息 WARN Utils: Truncated the string representation of a plan since it was too large. This behavior can be adjusted by setting 'spark.debug.maxToStringFields' in SparkEnv.conf. 中的描述设置 "spark.debug.maxToStringFields"。请在下面找到代码

val sparkConf= new SparkConf()
//here
sparkConf.set("spark.debug.maxToStringFields", "100000")
sparkConf.set("spark.sql.debug.maxToStringFields", "100000")
val spark = SparkSession.builder.config(sparkConf).getOrCreate() 
//here
spark.conf.set("spark.debug.maxToStringFields", 100000)
spark.conf.set("spark.sql.debug.maxToStringFields", 100000)

val data = spark.read
        .option("header", "true")
        .option("delimiter", "|")
        .format("csv")
        .csv(path_to_csv_file)
        .toDF()
        .repartition(col("country"))

data.rdd.toDebugString

我只得到带有上述警告消息的 toDebugString 的部分输出。如您所见，我已经尝试了两种选择。为什么它不打印完整的 RDD 沿袭？

Answer 1

你能在这里查看吗:

https://www.programcreek.com/scala/org.apache.spark.SparkEnv

我认为你必须像这样设置值

val sparkenv = SparkEnv.get sparkenv.conf.set("spark.oap.cache.strategy", "not_support_cache")

spark.debug.maxToStringFields 无效

spark.debug.maxToStringFields doesn't work

scala

apache-spark

rdd