[jira] [Commented] (TEZ-4130) Config for max task parallelism in shuffle - tez.shuffle-vertex-manager.max-task-parallelism

["David Mollitor (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-4130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17051547#comment-17051547 ] 

David Mollitor commented on TEZ-4130:
-------------------------------------

Hive has {{hive.exec.reducers.max}} to cap the number of Reducers.

Is there some way to pin the maximum number of tasks to the bucket number limitations (4k) so that this will work out-of-the-box?

> Config for max task parallelism in shuffle - tez.shuffle-vertex-manager.max-task-parallelism
> --------------------------------------------------------------------------------------------
>
>                 Key: TEZ-4130
>                 URL: https://issues.apache.org/jira/browse/TEZ-4130
>             Project: Apache Tez
>          Issue Type: Improvement
>            Reporter: László Bodor
>            Assignee: László Bodor
>            Priority: Major
>
> During the investigation of a customer issue, I found that tez generated a dag plan containing >4k tasks. It failed for hive because of bucket number limitations (4k). It can be configured properly, e.g. bigger splits (tez.grouping.min-size), but maybe it would be more convenient for users to config a hard limit for shuffle vertex manager.
> However, I'm not really sure if it's correct to force changing the max task parallelism after split generation already happened (e.g. [HiveSplitGenerator|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HiveSplitGenerator.java#L192-L244]):
> https://github.com/apache/tez/blob/master/tez-runtime-library/src/main/java/org/apache/tez/dag/library/vertexmanager/ShuffleVertexManager.java#L477



--
This message was sent by Atlassian Jira
(v8.3.4#803005)