You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "matthewbayer (via GitHub)" <gi...@apache.org> on 2026/10/12 18:36:57 UTC
[PR] [SPARK-60199][SQL] Preserve XML timestamps under the LEGACY parser policy [spark]
matthewbayer opened a new pull request, #59404:
URL: https://github.com/apache/spark/pull/59404
### What changes were proposed in this pull request?
Restore the `spark-xml` timestamp parser for XML `TIMESTAMP` reads and inference under `spark.sql.legacy.timeParserPolicy=LEGACY`.
The XML-specific helper tries ISO-8601 formats before an optional custom format, as `spark-xml` did. `CORRECTED`, `EXCEPTION`, `TIMESTAMP_NTZ`, and nanosecond timestamp parsing continue to use their existing formatters. No shared timestamp formatter or session configuration is changed.
### Why are the changes needed?
[SPARK-60199](https://issues.apache.org/jira/browse/SPARK-60199): the built-in XML reader applies a legacy `SSS` pattern to longer fractional seconds, accepting the input but changing its value. For example, `2024-02-29T12:34:56.123456Z` becomes `2024-02-29T12:36:59.456Z`. Both `PERMISSIVE` and `FAILFAST` return the wrong value because parsing succeeds.
The previous `spark-xml` connector parsed these timestamps independently of Spark's legacy formatter. Changing the session-wide policy is not an XML-only workaround.
### Does this PR introduce _any_ user-facing change?
Yes. XML reads under `LEGACY` preserve fractional seconds and restore the connector's ISO timestamp handling: optional fractions, offsets, UTC for unzoned ISO timestamps, and ISO-before-custom-format precedence. Historical timestamps retain the connector's `java.sql.Timestamp` conversion. Malformed ISO input is rejected instead of normalized (null in `PERMISSIVE`, an error in `FAILFAST`).
### How was this patch tested?
Added regression coverage for file reads with explicit and inferred schemas, both XML parser implementations, `PERMISSIVE`/`FAILFAST`, custom formats, historical dates, and `from_xml`. Added checks that modern and nanosecond parser behavior remains unchanged.
The regression suite reproduces the corruption on unpatched upstream master. All 40 focused tests pass with the fix: `XmlLegacyTimestampSuite`, `XmlTimestampParserSuite`, `XmlInferSchemaTypeCastingSuite`, and the timestamp tests in `XmlSuite` and `XmlSuiteWithLegacyParser`. Catalyst production/test and SQL test Scalastyle checks pass. The full Spark suite was not run locally.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-5)
This contribution is licensed under the Apache License, Version 2.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60199][SQL] Preserve XML timestamps under the LEGACY parser policy [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #59404:
URL: https://github.com/apache/spark/pull/59404#discussion_r4277892710
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/StaxXmlParser.scala:
##########
@@ -1172,6 +1189,73 @@ class XmlTokenizer(
}
object StaxXmlParser {
+ private val legacyZonedTimestampFormatters = Seq(
+ DateTimeFormatter.ISO_OFFSET_DATE_TIME,
+ DateTimeFormatter.ISO_INSTANT)
+
+ private val legacyLocalTimestampFormatters = Seq(
+ new DateTimeFormatterBuilder()
+ .parseCaseInsensitive()
+ .append(DateTimeFormatter.ISO_LOCAL_DATE)
+ .appendLiteral(' ')
+ .append(DateTimeFormatter.ISO_LOCAL_TIME)
+ .toFormatter()
Review Comment:
**Nit (P3):** `DateTimeFormatterBuilder.toFormatter()` uses `ResolverStyle.SMART`, and appending `ISO_LOCAL_DATE` does not carry over its STRICT resolver. This space-separated formatter therefore clamps impossible dates instead of rejecting them, while its `T`-separated sibling `ISO_LOCAL_DATE_TIME` stays STRICT. Checked with jshell:
- `2024-02-30 12:34:56` resolves to `2024-02-29T12:34:56Z`.
- `2023-04-31 00:00:00` resolves to `2023-04-30T00:00:00Z`.
- `2024-02-29 24:00:00` resolves to `2024-03-01T00:00:00Z`.
The `T` forms of all three are rejected. Under LEGACY, such values arrive here when there is no `timestampFormat`, or when the value does not match it. They now read as a different valid instant in both PERMISSIVE and FAILFAST, and inference reports TIMESTAMP. Before this PR they were rejected, and the PR description says malformed ISO values are rejected. The rejection test only uses `T` values with an offset, so it does not catch this.
Adding `.withResolverStyle(ResolverStyle.STRICT)` here matches the `T` path. Valid values such as `2024-02-29 12:34:56.123456` still parse.
**Verification:**
- **Regression:** LEGACY XML reads and inference reject space-separated impossible dates without a matching timestampFormat, in PERMISSIVE and FAILFAST.
- **Compatibility:** Existing LEGACY expectations for valid space-separated and 'T'-separated zone-less ISO values, including fraction preservation, still hold.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60199][SQL] Preserve XML timestamps under the LEGACY parser policy [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #59404:
URL: https://github.com/apache/spark/pull/59404#discussion_r4273345884
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/XmlInferSchema.scala:
##########
@@ -52,8 +52,9 @@ class XmlInferSchema(private val options: XmlOptions, private val caseSensitive:
import org.apache.spark.sql.catalyst.xml.XmlInferSchema._
private val decimalParser = ExprUtils.getDecimalParser(options.locale)
+ private val legacyTimeParser = SQLConf.get.legacyTimeParserPolicy == LegacyBehaviorPolicy.LEGACY
- private val timestampFormatter = TimestampFormatter(
+ @transient private lazy val timestampFormatter = TimestampFormatter(
Review Comment:
**Non-blocking (P2):** Making this a `lazy val` moves `TimestampFormatter`'s pattern validation out of the `XmlInferSchema` constructor and into the first `tryParseTimestamp` call, where per-record error handling hides it. This happens under every policy, including the default CORRECTED.
For example, with `timestampFormat` set to `YYYY-MM-dd HH:mm:ss` (week-based year), inference used to fail right away with `INCONSISTENT_BEHAVIOR_CROSS_VERSION.DATETIME_PATTERN_RECOGNITION`, which tells the user to fix the pattern or switch to LEGACY. Now:
- The `SparkUpgradeException` escapes `tryParseTimestamp`'s `IllegalArgumentException` catch and reaches the `NonFatal` handler in `infer()`.
- In PERMISSIVE, every record with a string field infers as `_corrupt_record: string`. FAILFAST reports the generic malformed-records error instead.
- Patterns rejected with a `SparkIllegalArgumentException`, such as an unsupported letter (`INVALID_DATETIME_PATTERN.ILLEGAL_CHARACTER`), are caught as "not a timestamp". The column then silently infers as `StringType` with no error.
`schema_of_xml` behaves the same way. Keeping this a plain `private val`, as before this PR and as in `JsonInferSchema`/`CSVInferSchema`, restores the construction-time error.
**Verification:**
- **Regression:** XML schema inference with an invalid or legacy-only timestampFormat raises the pattern or upgrade error under CORRECTED, rather than returning a corrupt-record or StringType schema.
- **Compatibility:** Existing LEGACY and CORRECTED XML timestamp inference expectations, including the LEGACY nanosecond-type error, still hold.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/xml/XmlLegacyTimestampSuite.scala:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.xml
+
+import java.sql.Timestamp
+import java.time.{Instant, LocalDateTime, ZoneId}
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.{SparkConf, SparkException}
+import org.apache.spark.sql.{QueryTest, Row}
+import org.apache.spark.sql.catalyst.util.DateTimeTestUtils.withDefaultTimeZone
+import org.apache.spark.sql.catalyst.util.DateTimeUtils
+import org.apache.spark.sql.functions.{from_xml, struct, to_xml}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types._
+
+class XmlLegacyTimestampSuite extends QueryTest with SharedSparkSession {
+ import testImplicits._
+
+ override protected def sparkConf: SparkConf = super.sparkConf
+ .set(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED, false)
+
+ private val schema = new StructType().add("stamp", TimestampType)
+
+ test("SPARK-60199: LEGACY XML preserves spark-xml timestamps in files and inference") {
+ val timestamps = Seq(
+ "2024-02-29T12:34:56Z" -> "2024-02-29T12:34:56Z",
+ "2024-02-29T12:34:56.1Z" -> "2024-02-29T12:34:56.100Z",
+ "2024-02-29T12:34:56.123Z" -> "2024-02-29T12:34:56.123Z",
+ "2024-02-29T12:34:56.123456Z" -> "2024-02-29T12:34:56.123456Z",
+ "2024-02-29T12:34:56.000001Z" -> "2024-02-29T12:34:56.000001Z",
+ "2024-02-29T12:34:56.123456789Z" -> "2024-02-29T12:34:56.123456Z",
+ "2024-02-29T12:34:56.123456+02:00" -> "2024-02-29T10:34:56.123456Z",
+ "2024-02-29T12:34:56.123456" -> "2024-02-29T12:34:56.123456Z",
+ "2024-02-29 12:34:56.123456" -> "2024-02-29T12:34:56.123456Z",
+ "1500-01-01T12:34:56.123456Z" -> "1500-01-01T12:34:56.123456Z")
+
+ withSQLConf(
+ SQLConf.LEGACY_TIME_PARSER_POLICY.key -> "LEGACY",
+ SQLConf.SESSION_LOCAL_TIMEZONE.key -> "America/Los_Angeles") {
+ Seq(false, true).foreach { multiLine =>
+ withTempPath { path =>
+ val rows = timestamps.map { case (input, _) => s"<ROW><stamp>$input</stamp></ROW>" }
+ val records = if (multiLine) Seq(rows.mkString("<ROWS>\n", "\n", "\n</ROWS>")) else rows
+ records.toDS().coalesce(1).write.text(path.getAbsolutePath)
+ for {
+ mode <- Seq("PERMISSIVE", "FAILFAST")
+ inferSchema <- Seq(false, true)
+ legacyXmlParser <- Seq(false, true)
+ } {
+ withClue(
+ s"$mode, infer=$inferSchema, multiLine=$multiLine, legacy=$legacyXmlParser") {
+ val reader = spark.read
+ .option("rowTag", "ROW")
+ .option("mode", mode)
+ .option("multiLine", multiLine)
+ .option("useLegacyXMLParser", legacyXmlParser)
+ if (!inferSchema) {
+ reader.schema(schema)
+ }
+ val result = reader.xml(path.getAbsolutePath)
+ assert(result.schema == schema)
+ checkAnswer(
+ result.selectExpr("unix_micros(stamp)"),
+ timestamps.map { case (_, expected) =>
+ Row(DateTimeUtils.instantToMicros(Instant.parse(expected)))
+ })
+ }
+ }
+ }
+ }
+ }
+ }
+
+ test("SPARK-60199: LEGACY XML tries ISO timestamps before custom formats") {
+ val cases = Seq(
+ ("dd/MM/yyyy HH:mm:ss.SSSSSS", "29/02/2024 12:34:56.123456", "2024-02-29T20:34:56.123456Z"),
+ (
+ "dd/MM/yyyy HH:mm:ss.SSSSSS XXX",
+ "29/02/2024 12:34:56.123456 +02:00",
+ "2024-02-29T10:34:56.123456Z"),
+ (
+ "yyyy-MM-dd'T'HH:mm:ss.SSSXXX",
+ "2024-02-29T12:34:56.123456Z",
+ "2024-02-29T12:34:56.123456Z"))
+ withSQLConf(SQLConf.LEGACY_TIME_PARSER_POLICY.key -> "LEGACY") {
+ cases.foreach { case (format, input, expected) =>
+ val xml = Seq(s"<ROW><stamp>$input</stamp></ROW>").toDS()
+ val reader = spark.read
+ .option("rowTag", "ROW")
+ .option("mode", "FAILFAST")
+ .option("timestampFormat", format)
+ .option("timeZone", "America/Los_Angeles")
+ checkAnswer(reader.xml(xml), Row(Timestamp.from(Instant.parse(expected))))
+ checkAnswer(reader.schema(schema).xml(xml), Row(Timestamp.from(Instant.parse(expected))))
+ }
+ }
+ }
+
+ test("SPARK-60199: LEGACY XML rejects invalid ISO timestamps instead of normalizing them") {
+ withSQLConf(SQLConf.LEGACY_TIME_PARSER_POLICY.key -> "LEGACY") {
+ Seq("2024-02-30T12:34:56.123Z", "2024-02-29T25:34:56.123Z", "2024-02-29T12:34:56.123Zjunk")
+ .foreach { value =>
+ val xml = Seq(s"<ROW><stamp>$value</stamp></ROW>").toDS()
+ val reader = spark.read.option("rowTag", "ROW").schema(schema)
Review Comment:
**Nit (P3):** This malformed-value test reads with an explicit schema, so it never runs inference. Two declared LEGACY branches have no assertion that would fail if they regressed:
- **No-format inference guard.** Inference without `timestampFormat` must not fall back to the lenient legacy default pattern. If it did, `2024-02-30T12:34:56.123Z` or a trailing-text value would infer as TimestampType and then read as null.
- **UTC fallback after a format miss.** A zone-less ISO value that does not match an explicit `timestampFormat`, for example `2024-02-29T12:34:56` under `dd/MM/yyyy HH:mm:ss`, should still read as UTC. None of the explicit-format cases uses such a value. Inference cannot cover this either, because `tryParseTimestampNTZ` claims zone-less values first under LEGACY.
An inference case for the first and a mixed-format read for the second would cover both.
**Verification:**
- **Behavior:** LEGACY XML inference without timestampFormat reports STRING for impossible-date and trailing-text ISO-like values.
- **Behavior:** LEGACY XML reads with a non-matching explicit timestampFormat return the UTC instant for a zone-less ISO value in PERMISSIVE and FAILFAST.
##########
docs/sql-migration-guide.md:
##########
@@ -35,6 +35,7 @@ license: |
- Since Spark 4.4, when an already-analyzed Data Source V2 query is refreshed after a compatible schema change, connectors can return more data columns from the current table schema in `Scan.readSchema()` than requested by `SupportsPushDownRequiredColumns.pruneColumns`. Previously, this partial pruning could fail planning because the scan reported columns absent from the analyzed relation output.
- Since Spark 4.4, when an already-analyzed Data Source V2 query is refreshed, if rebinding an expanded struct used as a map key causes distinct current keys to collide under the analyzed schema, the query fails with `DUPLICATED_MAP_KEY` by default instead of returning duplicate keys. With `spark.sql.mapKeyDedupPolicy=LAST_WIN`, the last value is kept.
- Since Spark 4.4, the Oracle JDBC datasource maps bare `NUMBER` columns (declared without explicit precision or scale) to `DecimalType(38, 18)` instead of `DecimalType(38, 10)`. This uses `DecimalType.DEFAULT_SCALE` and preserves up to 18 fractional digits instead of 10, reducing silent precision loss. However, the integer range shrinks from 28 to 20 digits; bare `NUMBER` values with more than 20 integer digits that previously read correctly will raise `NUMERIC_VALUE_OUT_OF_RANGE`. The same change applies to Oracle `FLOAT` columns, which JDBC reports as `NUMERIC` with `scale = -127`. Columns with explicit `NUMBER(p,s)` are not affected. To restore the previous `DecimalType(38, 10)` mapping, set `spark.sql.legacy.oracle.numberMapping.enabled` to `true`.
+- Since Spark 4.4, XML `TIMESTAMP` reads under `spark.sql.legacy.timeParserPolicy=LEGACY` preserve fractional seconds in ISO timestamps carrying an offset. Explicit `timestampFormat` values retain legacy pattern, locale, and time-zone handling; otherwise zone-less ISO timestamps are interpreted as UTC for spark-xml compatibility. `CORRECTED`, `EXCEPTION`, `TIMESTAMP_NTZ`, and nanosecond timestamp parsing are unchanged.
Review Comment:
**Nit (P3):** This entry says explicit `timestampFormat` values "retain legacy pattern, locale, and time-zone handling", and that zone-less ISO values are UTC "otherwise". Under LEGACY, `parseLegacyTimestamp` actually tries offset-bearing ISO values before the explicit format, and tries the UTC zone-less formatters after an explicit format that does not match. For upgrading LEGACY users, that means:
- **Shifted instants.** With `timestampFormat` set to `yyyy-MM-dd'T'HH:mm:ss'Z'` and session zone `America/Los_Angeles`, `2024-02-29T12:34:56Z` used to read as `2024-02-29T20:34:56Z` and now reads as `2024-02-29T12:34:56Z`.
- **Newly parsed values.** With `timestampFormat` set to `dd/MM/yyyy HH:mm:ss`, the same value used to be null, or StringType in inference, and now parses.
- **Newly rejected values.** Without `timestampFormat`, values the old lenient parser normalized or partially consumed, such as `2024-02-30T12:34:56.123Z` or a value with trailing text, now become null or a FAILFAST error, and may infer as STRING.
The option table in `sql-data-sources-xml.md` already describes the order correctly. Aligning this entry with it and listing the newly rejected inputs would fix this.
**Verification:**
- **Inspection:** The migration-guide entry describes the same precedence and UTC fallback as the timestampFormat option documentation and parseLegacyTimestamp, and names the newly rejected input classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60199][SQL] Preserve XML timestamps under the LEGACY parser policy [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #59404:
URL: https://github.com/apache/spark/pull/59404#discussion_r4269543975
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/StaxXmlParser.scala:
##########
@@ -1172,6 +1189,48 @@ class XmlTokenizer(
}
object StaxXmlParser {
+ private val legacyTimestampFormatters = Seq(
+ new DateTimeFormatterBuilder()
+ .parseCaseInsensitive()
+ .append(DateTimeFormatter.ISO_LOCAL_DATE)
+ .appendLiteral(' ')
+ .append(DateTimeFormatter.ISO_LOCAL_TIME)
+ .toFormatter()
+ .withZone(ZoneOffset.UTC),
+ DateTimeFormatter.ISO_LOCAL_DATE_TIME.withZone(ZoneOffset.UTC),
+ DateTimeFormatter.ISO_OFFSET_DATE_TIME,
+ DateTimeFormatter.ISO_INSTANT)
+
+ private[xml] def customLegacyTimestampFormatter(
+ options: XmlOptions): Option[DateTimeFormatter] = {
+ options.parameters.get(XmlOptions.TIMESTAMP_FORMAT).map { pattern =>
+ val format = DateTimeFormatter.ofPattern(pattern)
+ if (pattern.exists("VzOXxZ".contains(_))) {
+ format
+ } else {
+ format.withZone(options.timezone.map(ZoneId.of).orNull)
Review Comment:
**Blocking (P1):** `options.timezone` is the raw `timeZone` option, so a zone-less custom `timestampFormat` gets `withZone(null)` unless the user sets `timeZone` explicitly. `Instant.from` then always fails, and the `Try` turns that into `None`.
Example under LEGACY: `.option("timestampFormat", "dd/MM/yyyy HH:mm:ss")` with `29/02/2024 12:34:56` and no `timeZone`. Before this PR the value was parsed in the session time zone (`options.zoneId`). Now it gives null in PERMISSIVE, `_LEGACY_ERROR_TEMP_3260` in FAILFAST, and StringType in inference.
`ZoneId.of` also rejects IDs that `DateTimeUtils.getZoneId`, which `options.zoneId` uses, accepts, such as `PST` or `+8:00`. The resulting `ZoneRulesException` or `DateTimeException` is thrown outside the `Try` and is not an `IllegalArgumentException`. During inference, a non-digit string field skips the ISO formatters and forces this lazy formatter, so every record containing one becomes `_corrupt_record` in PERMISSIVE, or the read fails in FAILFAST.
The `VzOXxZ` check also matches quoted literals such as `'Z'`, which leaves those patterns without a zone. None of the new tests use a custom format without an explicit region `timeZone`.
See **Shared repair plan 1** in the review body.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/StaxXmlParser.scala:
##########
@@ -1172,6 +1189,48 @@ class XmlTokenizer(
}
object StaxXmlParser {
+ private val legacyTimestampFormatters = Seq(
+ new DateTimeFormatterBuilder()
+ .parseCaseInsensitive()
+ .append(DateTimeFormatter.ISO_LOCAL_DATE)
+ .appendLiteral(' ')
+ .append(DateTimeFormatter.ISO_LOCAL_TIME)
+ .toFormatter()
+ .withZone(ZoneOffset.UTC),
+ DateTimeFormatter.ISO_LOCAL_DATE_TIME.withZone(ZoneOffset.UTC),
+ DateTimeFormatter.ISO_OFFSET_DATE_TIME,
+ DateTimeFormatter.ISO_INSTANT)
+
+ private[xml] def customLegacyTimestampFormatter(
+ options: XmlOptions): Option[DateTimeFormatter] = {
+ options.parameters.get(XmlOptions.TIMESTAMP_FORMAT).map { pattern =>
+ val format = DateTimeFormatter.ofPattern(pattern)
+ if (pattern.exists("VzOXxZ".contains(_))) {
+ format
+ } else {
+ format.withZone(options.timezone.map(ZoneId.of).orNull)
+ }
+ }
+ }
+
+ private[xml] def parseLegacyTimestamp(
+ value: String,
+ customFormatter: => Option[DateTimeFormatter]): Option[Long] = {
+ def parseWithFormatter(format: DateTimeFormatter): Option[Long] = {
+ // Preserve spark-xml's java.sql.Timestamp conversion, including historical calendar rebasing.
+ Try(
+ DateTimeUtils.fromJavaTimestamp(Timestamp.from(Instant.from(format.parse(value))))).toOption
+ }
+
+ val isoTimestamp = if (value.nonEmpty && Character.isDigit(value.head)) {
+ legacyTimestampFormatters.iterator.map(parseWithFormatter)
+ .collectFirst { case Some(timestamp) => timestamp }
+ } else {
+ None
+ }
+ isoTimestamp.orElse(customFormatter.flatMap(parseWithFormatter))
Review Comment:
**Blocking (P1):** Zone-less ISO-shaped values are tried against the UTC-pinned `ISO_LOCAL_DATE_TIME` and space-separated formatters before the user's `timestampFormat`. An overlapping custom format, the `timeZone` option, and the session zone are therefore all silently ignored for those values.
Example under LEGACY, with the session zone (or `timeZone`) set to `America/Los_Angeles` and `timestampFormat` set to `yyyy-MM-dd HH:mm:ss`: `2024-02-29 12:34:56` was read as `2024-02-29T20:34:56Z` before this PR and is now read as `2024-02-29T12:34:56Z`. Inference still reports TimestampType, so nothing signals the shift. Formats that reorder fields, such as `yyyy-dd-MM HH:mm:ss`, are also misread.
The `tries ISO timestamps before custom formats` test doesn't catch this. Its only overlapping case carries an explicit offset, and its `SSSXXX` pattern cannot parse the six-digit input at all, so the test passes in either order.
See **Shared repair plan 1** in the review body.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/StaxXmlParser.scala:
##########
@@ -1172,6 +1189,48 @@ class XmlTokenizer(
}
object StaxXmlParser {
+ private val legacyTimestampFormatters = Seq(
+ new DateTimeFormatterBuilder()
+ .parseCaseInsensitive()
+ .append(DateTimeFormatter.ISO_LOCAL_DATE)
+ .appendLiteral(' ')
+ .append(DateTimeFormatter.ISO_LOCAL_TIME)
+ .toFormatter()
+ .withZone(ZoneOffset.UTC),
+ DateTimeFormatter.ISO_LOCAL_DATE_TIME.withZone(ZoneOffset.UTC),
+ DateTimeFormatter.ISO_OFFSET_DATE_TIME,
+ DateTimeFormatter.ISO_INSTANT)
+
+ private[xml] def customLegacyTimestampFormatter(
+ options: XmlOptions): Option[DateTimeFormatter] = {
+ options.parameters.get(XmlOptions.TIMESTAMP_FORMAT).map { pattern =>
+ val format = DateTimeFormatter.ofPattern(pattern)
+ if (pattern.exists("VzOXxZ".contains(_))) {
+ format
+ } else {
+ format.withZone(options.timezone.map(ZoneId.of).orNull)
+ }
+ }
+ }
+
+ private[xml] def parseLegacyTimestamp(
+ value: String,
+ customFormatter: => Option[DateTimeFormatter]): Option[Long] = {
+ def parseWithFormatter(format: DateTimeFormatter): Option[Long] = {
+ // Preserve spark-xml's java.sql.Timestamp conversion, including historical calendar rebasing.
+ Try(
+ DateTimeUtils.fromJavaTimestamp(Timestamp.from(Instant.from(format.parse(value))))).toOption
Review Comment:
**Nit (P3):** `Try(... format.parse(value) ...)` builds and throws a `DateTimeParseException`, with a stack trace, for every formatter that doesn't match. Under LEGACY that is:
- four exceptions for every digit-leading string that is not a timestamp during schema inference (for example `2020-01-01` or `123 Main St`), or five with a `timestampFormat`;
- two exceptions for every offset-bearing value on the read path.
The previous LEGACY path used the non-throwing `parseOptional`; SPARK-39281 made legacy-format inference exception-free for exactly this cost. Pre-screening with `format.parseUnresolved(value, new ParsePosition(0))` and checking that the whole input was consumed avoids the throw without changing results.
**Verification:**
- **Behavior:** The existing LEGACY accept and reject expectations, including impossible dates and trailing text, still hold.
- **Inspection:** The probing path for a non-matching formatter performs no throwing parse.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/StaxXmlParser.scala:
##########
@@ -1172,6 +1189,48 @@ class XmlTokenizer(
}
object StaxXmlParser {
+ private val legacyTimestampFormatters = Seq(
+ new DateTimeFormatterBuilder()
+ .parseCaseInsensitive()
+ .append(DateTimeFormatter.ISO_LOCAL_DATE)
+ .appendLiteral(' ')
+ .append(DateTimeFormatter.ISO_LOCAL_TIME)
+ .toFormatter()
+ .withZone(ZoneOffset.UTC),
+ DateTimeFormatter.ISO_LOCAL_DATE_TIME.withZone(ZoneOffset.UTC),
+ DateTimeFormatter.ISO_OFFSET_DATE_TIME,
+ DateTimeFormatter.ISO_INSTANT)
+
+ private[xml] def customLegacyTimestampFormatter(
+ options: XmlOptions): Option[DateTimeFormatter] = {
+ options.parameters.get(XmlOptions.TIMESTAMP_FORMAT).map { pattern =>
+ val format = DateTimeFormatter.ofPattern(pattern)
Review Comment:
**Non-blocking (P2):** Under `spark.sql.legacy.timeParserPolicy=LEGACY`, an explicit `timestampFormat` used to go through `TimestampFormatter.getLegacyFormatter`, i.e. `LegacyFastTimestampFormatter(pattern, options.zoneId, options.locale)`: SimpleDateFormat semantics with the `locale` option, which defaults to en-US. `DateTimeFormatter.ofPattern(pattern)` uses java.time letter semantics and the JVM default locale instead. As a result:
- `locale=fr-FR` with month names, or English month names on a non-English JVM, no longer parse. The values become null, a FAILFAST error, or StringType in inference.
- Letters whose meaning differs between the two APIs change behavior. For example, `dd/MM/yy HH:mm:ss` now reads `01/01/70 00:00:00` as 2070-01-01 instead of 1970-01-01 (checked with jshell).
This contradicts the documented LEGACY policy (SimpleDateFormat, locale-sensitive) and the XML `locale` option, which is "used while parsing dates and timestamps". CORRECTED and EXCEPTION are not affected.
See **Shared repair plan 1** in the review body.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/StaxXmlParser.scala:
##########
@@ -1172,6 +1189,48 @@ class XmlTokenizer(
}
object StaxXmlParser {
+ private val legacyTimestampFormatters = Seq(
+ new DateTimeFormatterBuilder()
+ .parseCaseInsensitive()
+ .append(DateTimeFormatter.ISO_LOCAL_DATE)
+ .appendLiteral(' ')
+ .append(DateTimeFormatter.ISO_LOCAL_TIME)
+ .toFormatter()
+ .withZone(ZoneOffset.UTC),
+ DateTimeFormatter.ISO_LOCAL_DATE_TIME.withZone(ZoneOffset.UTC),
+ DateTimeFormatter.ISO_OFFSET_DATE_TIME,
+ DateTimeFormatter.ISO_INSTANT)
+
+ private[xml] def customLegacyTimestampFormatter(
+ options: XmlOptions): Option[DateTimeFormatter] = {
+ options.parameters.get(XmlOptions.TIMESTAMP_FORMAT).map { pattern =>
+ val format = DateTimeFormatter.ofPattern(pattern)
+ if (pattern.exists("VzOXxZ".contains(_))) {
+ format
+ } else {
+ format.withZone(options.timezone.map(ZoneId.of).orNull)
+ }
+ }
+ }
+
+ private[xml] def parseLegacyTimestamp(
+ value: String,
+ customFormatter: => Option[DateTimeFormatter]): Option[Long] = {
+ def parseWithFormatter(format: DateTimeFormatter): Option[Long] = {
+ // Preserve spark-xml's java.sql.Timestamp conversion, including historical calendar rebasing.
+ Try(
+ DateTimeUtils.fromJavaTimestamp(Timestamp.from(Instant.from(format.parse(value))))).toOption
Review Comment:
**Non-blocking (P2):** `Instant.from(...)` is already a proleptic-Gregorian instant. `Timestamp.from` plus `DateTimeUtils.fromJavaTimestamp` then applies `rebaseJulianToGregorianMicros` in the executor's JVM default time zone.
For `1500-01-01T12:34:56.123Z`, the stored value becomes `1499-12-23T12:34:56.123Z` on a UTC JVM (`Timestamp.from(Instant.parse("1500-01-01T12:34:56Z"))` renders as `1499-12-23 12:34:56.0`), and it is off by a few more minutes on, for example, an `America/Los_Angeles` JVM. CAST to string, `show()`, and Parquet/ORC writes all expose the shifted date. CORRECTED and the previous LEGACY reader both keep 1500-01-01, and pre-1582 values written by the LEGACY XML writer no longer round-trip.
The new 1500-01-01 test case cannot catch this, because `collect()` applies the inverse rebase in `toJavaTimestamp`. Converting with `DateTimeUtils.instantToMicros`, and asserting on the stored or string-cast value, would fix both.
**Verification:**
- **Regression:** A pre-1582 LEGACY XML timestamp is checked on its stored or string-cast value, not through java.sql.Timestamp collection, and equals the CORRECTED result.
- **Compatibility:** Post-1582 LEGACY ISO results remain unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org