You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Wenchen Fan (Jira)" <ji...@apache.org> on 2024/01/20 13:01:00 UTC
[jira] [Assigned] (SPARK-46769) Refine timestamp related schema inference
[ https://issues.apache.org/jira/browse/SPARK-46769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wenchen Fan reassigned SPARK-46769:
-----------------------------------
Assignee: Wenchen Fan (was: Max Gekk)
> Refine timestamp related schema inference
> -----------------------------------------
>
> Key: SPARK-46769
> URL: https://issues.apache.org/jira/browse/SPARK-46769
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.0.0
> Reporter: Max Gekk
> Assignee: Wenchen Fan
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0, 3.5.1
>
>
> After the PR https://github.com/apache/spark/pull/43243, the TIMESTAMP_NTZ type inference in CSV/JSON datasource got 2 new guards which means TIMESTAMP_NTZ should be inferred either if:
> 1. the SQL config `spark.sql.legacy.timeParserPolicy` is set to `LEGACY` or
> 2. `spark.sql.timestampType` is set to `TIMESTAMP_NTZ`.
> otherwise CSV/JSON should try to infer `TIMESTAMP_LTZ`.
> Both guards are unnecessary because:
> 1. when `spark.sql.legacy.timeParserPolicy` is `LEGACY` that only means Spark should use a legacy Java 7- parser: `FastDateFormat` or `SimpleDateFormat`. Both parser are applicable for parsing `TIMESTAMP_NTZ`.
> 2. when `spark.sql.timestampType` is set to `TIMESTAMP_LTZ`, it doesn't mean that we should skip inferring of `TIMESTAMP_NTZ` types in CSV/JSON, and try to parse the timestamp string value w/o time zone like `2024-01-19T09:10:11.123` using a LTZ format **with timezone** like `yyyy-MM-dd'T'HH:mm:ss.SSSXXX`. _The last one cannot match any NTZ values for sure._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org