[jira] [Commented] (TEZ-3126) Auto-Reduce Parallelism: Vertex not re-configured when reduced by less than half.

["Bikas Saha (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-3126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15153229#comment-15153229 ] 

Bikas Saha commented on TEZ-3126:
---------------------------------

Yes. That was the intent. This was in a merged partition there will be 2 original partitions merged together. The idea is to create balanced partitions rather than having a chunk of partitions with much more data than the rest. Of course this is assuming that partition data is equally distributed. But essentially the idea is to maintain balanced partitions (with only the last partition having something smaller).

> Auto-Reduce Parallelism: Vertex not re-configured when reduced by less than half.
> ---------------------------------------------------------------------------------
>
>                 Key: TEZ-3126
>                 URL: https://issues.apache.org/jira/browse/TEZ-3126
>             Project: Apache Tez
>          Issue Type: Bug
>            Reporter: Jonathan Eagles
>            Assignee: Jonathan Eagles
>            Priority: Critical
>
> For example, when reducing parallelism from 36 to 22. The basePartitionRange will be 1 and will not re-configure the vertex.
> {code:java|title=ShuffleVertexManager#determineParallelismAndApply|borderStyle=dashed|bgColor=lightgrey}
>     int desiredTaskParallelism = 
>         (int)(
>             (expectedTotalSourceTasksOutputSize+desiredTaskInputDataSize-1)/
>             desiredTaskInputDataSize);
>     if(desiredTaskParallelism < minTaskParallelism) {
>       desiredTaskParallelism = minTaskParallelism;
>     }
>     
>     if(desiredTaskParallelism >= currentParallelism) {
>       return true;
>     }
>     
>     // most shufflers will be assigned this range
>     basePartitionRange = currentParallelism/desiredTaskParallelism;
>     
>     if (basePartitionRange <= 1) {
>       // nothing to do if range is equal 1 partition. shuffler does it by default
>       return true;
>     }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)