You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Ferdinand Xu (JIRA)" <ji...@apache.org> on 2015/03/20 09:40:38 UTC
[jira] [Commented] (HIVE-10031) Modify the using of jobConf
variable in ParquetRecordReaderWrapper constructor
[ https://issues.apache.org/jira/browse/HIVE-10031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14370961#comment-14370961 ]
Ferdinand Xu commented on HIVE-10031:
-------------------------------------
Hi Dong,
AFAIK, the class <noformat>Utilities</noformat> will copy the table configurations into this conf. If we don't do a clone operation on this configuration, it will pollute the original configuration if you get the parquet input split next time. In the same time, do you have any data about the time saving without clone?
Any thoughts?
> Modify the using of jobConf variable in ParquetRecordReaderWrapper constructor
> ------------------------------------------------------------------------------
>
> Key: HIVE-10031
> URL: https://issues.apache.org/jira/browse/HIVE-10031
> Project: Hive
> Issue Type: Sub-task
> Reporter: Dong Chen
> Assignee: Dong Chen
> Attachments: HIVE-10031-parquet.patch
>
>
> In {{ParquetRecordReaderWrapper}} constructor, it create splits, set projections and filters in conf, create task context, and then create Parquet record reader. In this procedure, we could improve the logic of conf usage:
> 1. the clone of jobConf is not necessary. This could speed up getRecordReader a little.
> 2. the updated jobConf is not passed to Parquet in one case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)