You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Max Gekk (Jira)" <ji...@apache.org> on 2026/06/19 12:08:00 UTC
[jira] [Updated] (SPARK-57572) Infer the TIME type during CSV and JSON schema inference
[ https://issues.apache.org/jira/browse/SPARK-57572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Max Gekk updated SPARK-57572:
-----------------------------
Description:
h2. What
Infer {{TimeType}} for time-only string values during CSV and JSON schema inference (when no
explicit schema is provided), analogous to the existing Date/Timestamp inference.
h2. Gap
{{CSVInferSchema}} (and {{JsonInferSchema}}) infer DateType/TimestampNTZType/TimestampType but
never {{TimeType}}. With an explicit schema, TIME read/write already works; only auto-inference
is missing.
h2. Scope
* Add a tryParseTime step to the inference type ladder, ordered to avoid regressions (a
time-only string like 12:13:14 must not be misclassified, and existing date/timestamp
inference must be unchanged).
* Consider gating behind the timeFormat option / a config and the spark.sql.timeType.enabled
flag, given inference ambiguity.
h2. Acceptance criteria
* CSV/JSON files with time-only columns infer TimeType (when enabled); existing inference is
unchanged.
* Tests in CSVInferSchemaSuite / JsonInferSchemaSuite and the CSV/JSON file suites.
> Infer the TIME type during CSV and JSON schema inference
> --------------------------------------------------------
>
> Key: SPARK-57572
> URL: https://issues.apache.org/jira/browse/SPARK-57572
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: Max Gekk
> Priority: Major
>
> h2. What
> Infer {{TimeType}} for time-only string values during CSV and JSON schema inference (when no
> explicit schema is provided), analogous to the existing Date/Timestamp inference.
> h2. Gap
> {{CSVInferSchema}} (and {{JsonInferSchema}}) infer DateType/TimestampNTZType/TimestampType but
> never {{TimeType}}. With an explicit schema, TIME read/write already works; only auto-inference
> is missing.
> h2. Scope
> * Add a tryParseTime step to the inference type ladder, ordered to avoid regressions (a
> time-only string like 12:13:14 must not be misclassified, and existing date/timestamp
> inference must be unchanged).
> * Consider gating behind the timeFormat option / a config and the spark.sql.timeType.enabled
> flag, given inference ambiguity.
> h2. Acceptance criteria
> * CSV/JSON files with time-only columns infer TimeType (when enabled); existing inference is
> unchanged.
> * Tests in CSVInferSchemaSuite / JsonInferSchemaSuite and the CSV/JSON file suites.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org