[jira] [Assigned] (SPARK-39012) SparkSQL parse partition value does not support all data types

["Wenchen Fan (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-39012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Wenchen Fan reassigned SPARK-39012:
-----------------------------------

    Assignee: Rui Wang

> SparkSQL parse partition value does not support all data types
> --------------------------------------------------------------
>
>                 Key: SPARK-39012
>                 URL: https://issues.apache.org/jira/browse/SPARK-39012
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 3.3.0
>            Reporter: Rui Wang
>            Assignee: Rui Wang
>            Priority: Major
>
> When Spark needs to infer schema, it needs to parse string to a type. Not all data types are supported so far in this path. For example, binary is known to not be supported. If a user uses binary column, and if the user does not use a metastore, then SparkSQL could fall back to schema inference thus fail to execute during table scan. This should be a bug as schema inference is supported but some types are missing.
> string might be converted to all types except ARRAY, MAP, STRUCT, etc. Also because when converting from a string, small scale type won't be identified if there is a larger scale type. For example, short and long 
> Based on Spark SQL data types: https://spark.apache.org/docs/latest/sql-ref-datatypes.html, we can support the following types:
> BINARY
> BOOLEAN
> And there are two types that I am not sure if SparkSQL is supporting:
> YearMonthIntervalType
> DayTimeIntervalType



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org