[jira] [Created] (SPARK-4293) Make Cast be able to handle complex types.

["Takuya Ueshin (JIRA)" <ji...@apache.org>]

Takuya Ueshin created SPARK-4293:
------------------------------------

             Summary: Make Cast be able to handle complex types.
                 Key: SPARK-4293
                 URL: https://issues.apache.org/jira/browse/SPARK-4293
             Project: Spark
          Issue Type: Improvement
          Components: SQL
            Reporter: Takuya Ueshin


Inserting data of type including {{ArrayType.containsNull == false}} or {{MapType.valueContainsNull == false}} or {{StructType.fields.exists(_.nullable == false)}} into Hive table will fail because {{Cast}} inserted by {{HiveMetastoreCatalog.PreInsertionCasts}} rule of {{Analyzer}} can't handle these types correctly.

Complex type cast rule proposal:

* Cast for non-complex types should be able to cast the same as before.
* Cast for {{ArrayType}} can evaluate if
** Element type can cast
** Nullability rule doesn't break
* Cast for {{MapType}} can evaluate if
** Key type can cast
** Nullability for casted key type is {{false}}
** Value type can cast
** Nullability rule for value type doesn't break
* Cast for {{StructType}} can evaluate if
** The field size is the same
** Each field can cast
** Nullability rule for each field doesn't break
* The nested structure should be the same.

Nullability rule:

* If the casted type is {{nullable == true}}, the target nullability should be {{true}}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org