[jira] [Created] (TEZ-3126) Auto-Reduce Parallelism: Vertex not re-configured when reduced by less than half.

["Jonathan Eagles (JIRA)" <ji...@apache.org>]

Jonathan Eagles created TEZ-3126:
------------------------------------

             Summary: Auto-Reduce Parallelism: Vertex not re-configured when reduced by less than half.
                 Key: TEZ-3126
                 URL: https://issues.apache.org/jira/browse/TEZ-3126
             Project: Apache Tez
          Issue Type: Bug
            Reporter: Jonathan Eagles
            Assignee: Jonathan Eagles
            Priority: Critical


For example, when reducing parallelism from 36 to 22. The basePartitionRange will be 1 and will not re-configure the vertex.

{code:java|title=ShuffleVertexManager#determineParallelismAndApply|borderStyle=dashed|bgColor=lightgrey}
    int desiredTaskParallelism = 
        (int)(
            (expectedTotalSourceTasksOutputSize+desiredTaskInputDataSize-1)/
            desiredTaskInputDataSize);
    if(desiredTaskParallelism < minTaskParallelism) {
      desiredTaskParallelism = minTaskParallelism;
    }
    
    if(desiredTaskParallelism >= currentParallelism) {
      return true;
    }
    
    // most shufflers will be assigned this range
    basePartitionRange = currentParallelism/desiredTaskParallelism;
    
    if (basePartitionRange <= 1) {
      // nothing to do if range is equal 1 partition. shuffler does it by default
      return true;
    }
{code}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)