[jira] [Commented] (SPARK-3559) appendReadColumnIDs and appendReadColumnNames introduce unnecessary columns in the lists of needed column ids and column names stored in hiveConf

["Venkata Ramana G (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-3559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14163218#comment-14163218 ] 

Venkata Ramana G commented on SPARK-3559:
-----------------------------------------

As same hiveConf is used across queries columns get appended, and cannot be controlled to send only required columns.
HiveConf can be cloned at TableScanOperator and configure required properties.
deserializers are expecting this property to be set in HiveConf but not in table-specific properties.

> appendReadColumnIDs and appendReadColumnNames introduce unnecessary columns in the lists of needed column ids and column names stored in hiveConf
> -------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-3559
>                 URL: https://issues.apache.org/jira/browse/SPARK-3559
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>            Reporter: Yin Huai
>            Priority: Blocker
>
> Because we are using the same hiveConf and we are currently using ColumnProjectionUtils.appendReadColumnIDs ColumnProjectionUtils.appendReadColumnNames to append needed column ids and names for a table, lists of needed column ids and names can have unnecessary columns.
> Also, for a join operation, TableScanOperators for both tables are sharing the same hiveConf and they may need to set table-specific properties.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org