You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Shrirang Mhalgi (Jira)" <ji...@apache.org> on 2026/06/20 00:09:00 UTC
[jira] [Commented] (SPARK-57572) Infer the TIME type during CSV and JSON schema inference
[ https://issues.apache.org/jira/browse/SPARK-57572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18090324#comment-18090324 ]
Shrirang Mhalgi commented on SPARK-57572:
-----------------------------------------
Hi [~maxgekk] I'd like to work on this one if it's available.
I'll follow the existing DateType/TimestampType inference pattern in CSVInferSchema and JsonInferSchema, gate it behind spark.sql.timeType.enabled, and add tests in the inference suites. Let me know if you'd prefer to handle it yourself or if there are any dependencies I should be aware of.
> Infer the TIME type during CSV and JSON schema inference
> --------------------------------------------------------
>
> Key: SPARK-57572
> URL: https://issues.apache.org/jira/browse/SPARK-57572
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: Max Gekk
> Priority: Major
>
> h2. What
> Infer {{TimeType}} for time-only string values during CSV and JSON schema inference (when no
> explicit schema is provided), analogous to the existing Date/Timestamp inference.
> h2. Gap
> {{CSVInferSchema}} (and {{JsonInferSchema}}) infer DateType/TimestampNTZType/TimestampType but
> never {{TimeType}}. With an explicit schema, TIME read/write already works; only auto-inference
> is missing.
> h2. Scope
> * Add a tryParseTime step to the inference type ladder, ordered to avoid regressions (a
> time-only string like 12:13:14 must not be misclassified, and existing date/timestamp
> inference must be unchanged).
> * Consider gating behind the timeFormat option / a config and the spark.sql.timeType.enabled
> flag, given inference ambiguity.
> h2. Acceptance criteria
> * CSV/JSON files with time-only columns infer TimeType (when enabled); existing inference is
> unchanged.
> * Tests in CSVInferSchemaSuite / JsonInferSchemaSuite and the CSV/JSON file suites.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org