[jira] [Assigned] (SPARK-16926) Partition columns are present in columns metadata for partition but not table

["Apache Spark (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-16926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Apache Spark reassigned SPARK-16926:
------------------------------------

    Assignee:     (was: Apache Spark)

> Partition columns are present in columns metadata for partition but not table
> -----------------------------------------------------------------------------
>
>                 Key: SPARK-16926
>                 URL: https://issues.apache.org/jira/browse/SPARK-16926
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>            Reporter: Brian Cho
>
> A change introduced in SPARK-14388 removes partition columns from the column metadata of tables, but not for partitions. This causes TableReader to believe that the schema is different and create an unnecessary conversion object inspector, taking the else codepath in TableReader below:
> {code}
>     val soi = if (rawDeser.getObjectInspector.equals(tableDeser.getObjectInspector)) {
>       rawDeser.getObjectInspector.asInstanceOf[StructObjectInspector]
>     } else {
>       ObjectInspectorConverters.getConvertedOI(
>         rawDeser.getObjectInspector,
>         tableDeser.getObjectInspector).asInstanceOf[StructObjectInspector]
>     }
> {code}
> Printing the properties as debug output confirms the difference for the Hive table.
> Table properties (tableDesc.getProperties):
> {code}
> 16/08/04 20:36:58 DEBUG HadoopTableReader: columns.types, string:bigint:string:bigint:bigint:array<string>
> {code}
> Partition properties (partProps):
> {code}
> 16/08/04 20:36:58 DEBUG HadoopTableReader: columns.types, string:bigint:string:bigint:bigint:array<string>:string:string:string
> {code}
> Where the final three string columns are partition columns



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org