You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "okumin (Jira)" <ji...@apache.org> on 2020/07/10 05:15:00 UTC
[jira] [Assigned] (HIVE-23831) Enforce auto-parallelism of Tez even
if the number of reducers can be 1
[ https://issues.apache.org/jira/browse/HIVE-23831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
okumin reassigned HIVE-23831:
-----------------------------
> Enforce auto-parallelism of Tez even if the number of reducers can be 1
> -----------------------------------------------------------------------
>
> Key: HIVE-23831
> URL: https://issues.apache.org/jira/browse/HIVE-23831
> Project: Hive
> Issue Type: Improvement
> Components: Hive
> Reporter: okumin
> Assignee: okumin
> Priority: Major
>
> Hive on Tez disables auto-reduce and uses the maximum number of partitions when the minimum number of reducers, `estimated number * min.partition.factor`, is less than or equal to 1.
> [https://github.com/apache/hive/blob/f98e136bdd5642e3de10d2fd1a4c14d1d6762113/ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java#L121-L131]
> https://issues.apache.org/jira/browse/HIVE-14200
>
> This can make Tez lose chance to optimize # of reducers especially in an environment with small `hive.tez.min.partition.factor`.
> I'd say it would be fine to introduce an option to enforce auto-reduce.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)