[jira] [Commented] (HIVE-12682) Reducers in dynamic partitioning job spend a lot of time running hadoop.conf.Configuration.getOverlay

["Prasanth Jayachandran (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-12682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15092921#comment-15092921 ] 

Prasanth Jayachandran commented on HIVE-12682:
----------------------------------------------

[~gopalv] Could you take a look at the patch? The patch just reuses the taskId instead of getting it in inner loop from the configuration object.

> Reducers in dynamic partitioning job spend a lot of time running hadoop.conf.Configuration.getOverlay
> -----------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-12682
>                 URL: https://issues.apache.org/jira/browse/HIVE-12682
>             Project: Hive
>          Issue Type: Bug
>          Components: Hive
>    Affects Versions: 1.2.1
>            Reporter: Carter Shanklin
>            Assignee: Prasanth Jayachandran
>         Attachments: HIVE-12682.1.patch, HIVE-12682.2.patch, reducer.png
>
>
> I tested this on Hive 1.2.1 but looks like it's still applicable to 2.0.
> I ran this query:
> {code}
> create table flights (
> …
> )
> PARTITIONED BY (Year int)
> CLUSTERED BY (Month)
> SORTED BY (DayofMonth) into 12 buckets
> STORED AS ORC
> TBLPROPERTIES("orc.bloom.filter.columns"="*")
> ;
> {code}
> (Taken from here: https://github.com/t3rmin4t0r/all-airlines-data/blob/master/ddl/orc.sql)
> I profiled just the reduce phase and noticed something odd, the attached graph shows where time was spent during the reducer phase.
> !reducer.png!
> Problem seems to relate to https://github.com/apache/hive/blob/branch-2.0/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java#L903
> /cc [~gopalv]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)