sparksql - 将带时区的字符串转换为 UTC

Question

val someDF = Seq(
        (8, "2019-12-11T00:00:00.451086451+09"),
        (64, "2017-08-09T13:12:35.283018201-04"),
        (-27, "2019-12-01T23:00:00.484072842Z")
      ).toDF("number", "tt")
someDF.createOrReplaceTempView("t")
spark.sql("SELECT * FROM t").show(20, false)

忽略第一列。如何使用纯 sparksql 将第二列值转换为时间戳数据类型（UTC）？我在下面尝试过，但它返回了 null，我找不到用引号将 'T' 括起来的方法：

spark.sql("SELECT tt, to_timestamp(tt,'yyyy-MM-ddTHH:mm:ss.SSSX') n FROM t").show(20, false)

Answer 1

您可以省略格式字符串。

spark.sql("SELECT tt, to_timestamp(tt) n FROM t").show(20, false) returns

+--------------------------------+--------------------------+
|tt                              |n                         |
+--------------------------------+--------------------------+
|2019-12-11T00:00:00.451086451+09|2019-12-10 15:00:00.451086|
|2017-08-09T13:12:35.283018201-04|2017-08-09 17:12:35.283018|
|2019-12-01T23:00:00.484072842Z  |2019-12-01 23:00:00.484072|
+--------------------------------+--------------------------+

sparksql - 将带时区的字符串转换为 UTC

sparksql - convert string with timezone to UTC

scala

apache-spark

apache-spark-sql

pyspark-sql