[jira] [Issue Comment Deleted] (SPARK-6923) Spark SQL CLI does not read Data Source schema correctly

["pin_zhang (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-6923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

pin_zhang updated SPARK-6923:
-----------------------------
    Comment: was deleted

(was: Hi, Cheng Hao
   Thanks for your reply!
   Do you mean if provide a wrapper for datasource api, the Hive Storage Handler can get the "data sourced" table schema correctly for the external application via Hive API?

    If so, can it be fixed in Spark 1.3.x?

)

> Spark SQL CLI does not read Data Source schema correctly
> --------------------------------------------------------
>
>                 Key: SPARK-6923
>                 URL: https://issues.apache.org/jira/browse/SPARK-6923
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.3.0
>            Reporter: pin_zhang
>            Priority: Blocker
>
> {code:java}
> HiveContext hctx = new HiveContext(sc);
> List<String> sample = new ArrayList<String>();
> sample.add( "{\"id\": \"id_1\", \"age\":1}" );
> RDD<String> sampleRDD = new JavaSparkContext(sc).parallelize(sample).rdd();	
> DataFrame df = hctx.jsonRDD(sampleRDD);
> String table="test";
> df.saveAsTable(table, "json",SaveMode.Overwrite);
> Table t = hctx.catalog().client().getTable(table);
> System.out.println( t.getCols());
> {code}
> --------------------------------------------------------------
> With the code above to save DataFrame to hive table,
> Get table cols returns one column named 'col'
> [FieldSchema(name:col, type:array<string>, comment:from deserializer)]
> Expected return fields schema id, age.
> This results in the jdbc API cannot retrieves the table columns via ResultSet DatabaseMetaData.getColumns(String catalog, String schemaPattern,String tableNamePattern, String columnNamePattern)
> But resultset metadata for query " select * from test "  contains fields id, age.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org