[jira] [Created] (SPARK-60199) XML reader silently changes microsecond timestamps under LEGACY timeParserPolicy, even in FAILFAST mode

["Matthew Bayer (Jira)" <ji...@apache.org>]

Matthew Bayer created SPARK-60199:
-------------------------------------

             Summary: XML reader silently changes microsecond timestamps under LEGACY timeParserPolicy, even in FAILFAST mode
                 Key: SPARK-60199
                 URL: https://issues.apache.org/jira/browse/SPARK-60199
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 4.2.0
            Reporter: Matthew Bayer


When {{spark.sql.legacy.timeParserPolicy=LEGACY}} is explicitly configured, Spark’s built-in XML reader misinterprets fractional seconds in otherwise valid ISO-8601 timestamps.

For example:

 

{{Input:    2024-02-29T12:34:56.123456Z
Expected: 2024-02-29T12:34:56.123456Z
Actual:   2024-02-29T12:36:59.456Z}}

This affects both {{PERMISSIVE}} and {{{}FAILFAST{}}}: parsing succeeds, so {{FAILFAST}} does not protect against the incorrect value.

This is a migration concern for pipelines moving from Spark 3 with {{spark-xml}} to Spark 4’s built-in XML reader while retaining {{LEGACY}} for compatibility with other timestamp operations. The old connector parsed these ISO timestamps independently of Spark’s legacy parser and preserved the fractional seconds.

 
h3. Reproduction

Run in {{{}spark-shell{}}}; no external connector or input files are required:

 

{{import spark.implicits._

println(spark.version)

spark.conf.set("spark.sql.session.timeZone", "UTC")
spark.conf.set("spark.sql.legacy.timeParserPolicy", "LEGACY")

val xml = Seq(
  "<ROW><stamp>2024-02-29T12:34:56.123456Z</stamp></ROW>"
).toDS()

Seq("PERMISSIVE", "FAILFAST").foreach \{ mode =>
  println(s"mode=$mode")

  spark.read
    .schema("stamp TIMESTAMP")
    .option("rowTag", "ROW")
    .option("mode", mode)
    .xml(xml)
    .selectExpr(
      "CAST(stamp AS STRING) AS parsed",
      "unix_micros(stamp) AS epoch_micros"
    )
    .show(false)
}}}

Both modes produce:

 

{{parsed                    epoch_micros
2024-02-29 12:36:59.456     1709210219456000}}

Expected:

 

{{parsed                       epoch_micros
2024-02-29 12:34:56.123456     1709210096123456}}

Changing the policy to {{CORRECTED}} produces the expected value for this example. However, globally changing that policy also changes non-XML parsing behavior.

 
h3. Apparent cause

Under {{{}LEGACY{}}}, [XmlOptions|https://github.com/apache/spark/blob/v4.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/XmlOptions.scala#L161-L167] supplies the default pattern:

 

{{yyyy-MM-dd'T'HH:mm:ss.SSSXXX}}

[StaxXmlParser|https://github.com/apache/spark/blob/v4.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/StaxXmlParser.scala#L60-L65] selects {{{}FAST_DATE_FORMAT{}}}. [LegacyFastTimestampFormatter|https://github.com/apache/spark/blob/v4.2.0/sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala#L414-L471] accepts the six-digit fraction but scales it using the pattern’s three {{S}} characters:

 

{{123456 × 1,000,000 / 10³ = 123,456,000 microseconds
                        = 123.456 seconds}}

That value is added to the whole-second timestamp instead of being interpreted as {{0.123456}} seconds.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org