[jira] [Updated] (SPARK-10186) Inconsistent handling of complex column types in external databases

["Michael Armbrust (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-10186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael Armbrust updated SPARK-10186:
-------------------------------------
    Issue Type: New Feature  (was: Bug)

> Inconsistent handling of complex column types in external databases
> -------------------------------------------------------------------
>
>                 Key: SPARK-10186
>                 URL: https://issues.apache.org/jira/browse/SPARK-10186
>             Project: Spark
>          Issue Type: New Feature
>          Components: SQL
>    Affects Versions: 1.4.1
>         Environment: Ubuntu on AWS
>            Reporter: Simeon Simeonov
>              Labels: array, json, postgres, sql, struct
>
> The specific observations below are based on Postgres 9.4 tables accessed via the postgresql-9.4-1201.jdbc41.jar driver. However, based on the behavior, I would expect the problem to exists for all external SQL databases.
> - *json and jsonb columns generate {{java.sql.SQLException: Unsupported type 1111}}*. While it is reasonable to not support dynamic schema discovery of JSON columns automatically (it requires two passes over the data), a better behavior would be to create a String column and return the JSON.
> - *Array columns generate {{java.sql.SQLException: Unsupported type 2003}}*. This is true even for simple types, e.g., {{text[]}}. A better behavior would be be create an Array column. 
> - *Custom type columns are mapped to a String column.* This behavior is harder to understand as the schema of a custom type is fixed and therefore mappable to a Struct column. The automatic conversion to a string is also inconsistent when compared to json and array column handling.
> The exceptions are throw by {{org.apache.spark.sql.jdbc.JDBCRDD$.org$apache$spark$sql$jdbc$JDBCRDD$$getCatalystType(JDBCRDD.scala:100)}} so this definitely looks like a Spark SQL and not a JDBC problem.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org