You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Carter Shanklin (JIRA)" <ji...@apache.org> on 2015/12/15 23:14:46 UTC
[jira] [Created] (HIVE-12682) Reducers in dynamic partitioning job
spend a lot of time running hadoop.conf.Configuration.getOverlay
Carter Shanklin created HIVE-12682:
--------------------------------------
Summary: Reducers in dynamic partitioning job spend a lot of time running hadoop.conf.Configuration.getOverlay
Key: HIVE-12682
URL: https://issues.apache.org/jira/browse/HIVE-12682
Project: Hive
Issue Type: Bug
Components: Hive
Affects Versions: 1.2.1
Reporter: Carter Shanklin
Attachments: reducer.png
I tested this on Hive 1.2.1 but looks like it's still applicable to 2.0.
I ran this query:
{code}
create table flights (
…
)
PARTITIONED BY (Year int)
CLUSTERED BY (Month)
SORTED BY (DayofMonth) into 12 buckets
STORED AS ORC
TBLPROPERTIES("orc.bloom.filter.columns"="*")
;
{code}
(Taken from here: https://github.com/t3rmin4t0r/all-airlines-data/blob/master/ddl/orc.sql)
I profiled just the reduce phase and noticed something odd, the attached graph shows where time was spent during the reducer phase.
Problem seems to relate to https://github.com/apache/hive/blob/branch-2.0/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java#L903
/cc [~gopalv]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)