[jira] [Created] (SPARK-27450) Timestamp cast fails when ISO8601 string omits zero minutes or seconds

["Leandro Rosa (JIRA)" <ji...@apache.org>]

Leandro Rosa created SPARK-27450:
------------------------------------

             Summary: Timestamp cast fails when ISO8601 string omits zero minutes or seconds
                 Key: SPARK-27450
                 URL: https://issues.apache.org/jira/browse/SPARK-27450
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 2.3.0
         Environment: Spark 2.3.x
            Reporter: Leandro Rosa


ISO8601 allows to omit zero minutes, seconds and milliseconds.
{quote}
|hh:mm:ss.sss|_or_|hhmmss.sss|
|hh:mm:ss|_or_|hhmmss|
|hh:mm|_or_|hhmm|
| |hh|
{quote}
{quote}Either the seconds, or the minutes and seconds, may be omitted from the basic or extended time formats for greater brevity but decreased accuracy: [hh]:[mm], [hh][mm] and [hh] are the resulting reduced accuracy time formats
{quote}
Source: [Wikipedia ISO8601|https://en.wikipedia.org/wiki/ISO_8601]

Popular libs, such as [ZonedDateTime|[https://docs.oracle.com/javase/8/docs/api/java/time/ZonedDateTime.html]], respect that. However, Timestamp cast fails silently.

 
{code:java}
import org.apache.spark.sql.types._
val df1 = Seq(("2017-08-01T02:33")).toDF("eventTimeString") // NON-ISO8601 (missing TZ offset) [OK]
val new_df1 = df1
.withColumn("eventTimeTS", col("eventTimeString").cast(TimestampType))

new_df1.show(false)

+----------------+-------------------+
|eventTimeString |eventTimeTS |
+----------------+-------------------+
|2017-08-01T02:33|2017-08-01 02:33:00|
+----------------+-------------------+
{code}
{code:java}
val df2 = Seq(("2017-08-01T02:33Z")).toDF("eventTimeString") // ISO8601 [FAIL]
val new_df2 = df2
.withColumn("eventTimeTS", col("eventTimeString").cast(TimestampType))

new_df2.show(false)

+-----------------+-----------+
|eventTimeString |eventTimeTS|
+-----------------+-----------+
|2017-08-01T02:33Z|null |
+-----------------+-----------+
{code}
 
{code:java}
val df3 = Seq(("2017-08-01T02:33-03:00")).toDF("eventTimeString") // ISO8601 [FAIL]
val new_df3 = df3
.withColumn("eventTimeTS", col("eventTimeString").cast(TimestampType))

new_df3.show(false)

+----------------------+-----------+
|eventTimeString |eventTimeTS|
+----------------------+-----------+
|2017-08-01T02:33-03:00|null |
+----------------------+-----------+
{code}
 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org