pyspark select/filter 语句都不起作用

Question

我正在通过《Spark：权威指南》这本书学习 pyspark。我正在学习 window 函数 section.My 代码是：

**dfWithDate.where("CustomerId IS NOT NULL").orderBy("CustomerId").select(col("CustomerId"),col("date"),col("Quantity"),purchaseRank.alias("quantityRank"),purchaseDenseRank.alias("quantityDenseRank"),maxPurchaseQuantity.alias("maxPurchaseQuantity")).show()**

我收到一个错误：

我也试试：

df2=dfWithDate.filter(" CustomerId IS NOT NULL").orderBy("CustomerId").show()

我收到了相同类型的错误。我想知道是什么原因造成的，以及解决方法。

Answer 1

错误与 select/filter 无关。在你得到的异常中很清楚

Fail to parse '6/12/2011 13:001 in the new parser. You can set spark.sal. legacy. timeParserpolicy to LEGACY to restore the behavior before Spark 3.0, or set to CORRECTED and treat it as an invalid datetime string

pyspark select/filter 语句都不起作用

pyspark select/filter statement both not working

select

filter

pyspark