[jira] [Created] (YARN-10380) Import logic of multi-node allocation in CapacityScheduler

["Wangda Tan (Jira)" <ji...@apache.org>]

Wangda Tan created YARN-10380:
---------------------------------

             Summary: Import logic of multi-node allocation in CapacityScheduler
                 Key: YARN-10380
                 URL: https://issues.apache.org/jira/browse/YARN-10380
             Project: Hadoop YARN
          Issue Type: Improvement
            Reporter: Wangda Tan


*1) Entry point:* 
When we do multi-node allocation, we're using the same logic of async scheduling:
{code:java}
// Allocate containers of node [start, end)
 for (FiCaSchedulerNode node : nodes) {
  if (current++ >= start) {
     if (shouldSkipNodeSchedule(node, cs, printSkipedNodeLogging)) {
        continue;
     }
     cs.allocateContainersToNode(node.getNodeID(), false);
  }
 } {code}
Is it the most effective way to do multi-node scheduling? Should we allocate based on partitions? In above logic, if we have thousands of node in one partition, we will repeatly access all nodes of the partition thousands of times.

I would suggest looking at making entry-point for node-heartbeat, async-scheduling (single node), and async-scheduling (multi-node) to be different.

Node-heartbeat and async-scheduling (single node) can be still similar and share most of the code. 

async-scheduling (multi-node): should iterate partition first, using pseudo code like: 
{code:java}
for (partition : all partitions) {
  allocateContainersOnMultiNodes(getCandidate(partition))
} {code}
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org