[jira] [Commented] (SPARK-40474) Infer columns with mixed date and timestamp as String in CSV schema inference

["Apache Spark (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-40474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17606683#comment-17606683 ] 

Apache Spark commented on SPARK-40474:
--------------------------------------

User 'xiaonanyang-db' has created a pull request for this issue:
https://github.com/apache/spark/pull/37933

> Infer columns with mixed date and timestamp as String in CSV schema inference
> -----------------------------------------------------------------------------
>
>                 Key: SPARK-40474
>                 URL: https://issues.apache.org/jira/browse/SPARK-40474
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 3.4.0
>            Reporter: Xiaonan Yang
>            Priority: Major
>
> In this ticket https://issues.apache.org/jira/browse/SPARK-39469, we introduced the support of date type in CSV schema inference. The schema inference behavior on date time columns now is:
>  * For a column only containing dates, we will infer it as Date type
>  * For a column only containing timestamps, we will infer it as Timestamp type
>  * For a column containing a mixture of dates and timestamps, we will infer it as Timestamp type
> However, we found that we are too ambitious on the last scenario, to support which we have introduced much complexity in code and caused a lot of performance concerns. Thus, we want to simplify the behavior of the last scenario as:
>  * For a column containing a mixture of dates and timestamps, we will infer it as String type



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org