[jira] [Created] (SPARK-60201) Spark 4 XML silently changes ISO date interpretation when a custom dateFormat is configured

["Matthew Bayer (Jira)" <ji...@apache.org>]

Matthew Bayer created SPARK-60201:
-------------------------------------

             Summary: Spark 4 XML silently changes ISO date interpretation when a custom dateFormat is configured
                 Key: SPARK-60201
                 URL: https://issues.apache.org/jira/browse/SPARK-60201
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 4.2.0
            Reporter: Matthew Bayer


h3. Description

Migrating from \{{spark-xml}} to Spark 4’s built-in XML reader can silently change date values when an input matches both ISO format and a configured \{{dateFormat}}.

The previous connector tries ISO parsing first, falling back to the custom format. Spark 4 uses the configured format directly.

For example, \{{2024-03-04}} with \{{dateFormat=yyyy-dd-MM}} changes from *March 4* to *April 3*, without an error.

April 3 is a valid interpretation of the configured format. This report concerns the backward-incompatible change in parsing precedence, rather than an incorrectly implemented format.

h3. Reproduction

Compare \{{spark-xml}} 0.17.0 with Spark 4’s built-in XML reader, using the same input and options.

{\{input.xml}}:
{code:xml}
<ROW><event_date>2024-03-04</event_date></ROW>
{code}

{code:python}
for policy in ["LEGACY", "CORRECTED", "EXCEPTION"]:
    spark.conf.set("spark.sql.legacy.timeParserPolicy", policy)

    result = (
        spark.read.format("xml")
        .option("rowTag", "ROW")
        .option("dateFormat", "yyyy-dd-MM")
        .option("mode", "FAILFAST")
        .schema("event_date DATE")
        .load("/path/to/input.xml")
        .first()
    )

    print(policy, result.event_date)
{code}

h3. Actual behavior

|| Reader || LEGACY || CORRECTED || EXCEPTION ||
| \{{spark-xml}} 0.17.0 | \{{2024-03-04}} | \{{2024-03-04}} | \{{2024-03-04}} |
| Spark 4 built-in XML | \{{2024-04-03}} | \{{2024-04-03}} | \{{2024-04-03}} |

{\{FAILFAST}} does not protect against this change because parsing succeeds. Retaining \{{spark.sql.legacy.timeParserPolicy=LEGACY}} also does not preserve the previous result.

h3. Expected behavior

Provide a compatibility path that preserves the previous interpretation, or fails explicitly instead of silently returning a different date.

A narrowly scoped solution would restore the old XML date-parsing precedence under \{{LEGACY}}: try ISO first, then the explicitly configured custom format. Reading and schema inference should use the same parsing rules, without changing shared date formatters or other data sources.

h3. Impact

Existing pipelines containing ISO dates alongside a custom date-format option can produce different dates after upgrading. Ambiguous day/month values can affect date filters, joins, partitioning, and aggregations without failing the build.

For ISO-only inputs, removing \{{dateFormat}} or setting it to \{{yyyy-MM-dd}} is a workaround. That does not preserve the old behavior for inputs mixing ISO and custom formats.

h3. Reference

The previous ISO-first behavior is implemented in [spark-xml 0.17.0’s parseXmlDate|https://github.com/databricks/spark-xml/blob/v0.17.0/src/main/scala/com/databricks/spark/xml/util/TypeCast.scala].



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org