[jira] [Created] (YARN-2380) The normalizeRequests method in SchedulerUtils always resets the vCore to 1

["Jian Fang (JIRA)" <ji...@apache.org>]

Jian Fang created YARN-2380:
-------------------------------

             Summary: The normalizeRequests method in SchedulerUtils always resets the vCore to 1
                 Key: YARN-2380
                 URL: https://issues.apache.org/jira/browse/YARN-2380
             Project: Hadoop YARN
          Issue Type: Bug
          Components: resourcemanager
    Affects Versions: 2.4.0
            Reporter: Jian Fang
            Priority: Critical


I added some log info to the method normalizeRequest() as follows.

  public static void normalizeRequest(
      ResourceRequest ask, 
      ResourceCalculator resourceCalculator, 
      Resource clusterResource,
      Resource minimumResource,
      Resource maximumResource,
      Resource incrementResource) {
    LOG.info("Before request normalization, the ask capacity: " + ask.getCapability());
    Resource normalized = 
        Resources.normalize(
            resourceCalculator, ask.getCapability(), minimumResource,
            maximumResource, incrementResource);
    LOG.info("After request normalization, the ask capacity: " + normalized);
    ask.setCapability(normalized);
  }

The resulted log showed that the vcore in ask was changed from 2 to 1.

2014-08-01 20:54:15,537 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils (IPC Server handler 4 on 9024): Before request normalization, the ask capacity: <memory:1536, vCores:2>
2014-08-01 20:54:15,537 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils (IPC Server handler 4 on 9024): After request normalization, the ask capacity: <memory:1536, vCores:1>

The root cause is the DefaultResourceCalculator calls Resources.createResource(normalizedMemory) to regenerate a new resource with vcore = 1.

This bug is critical and it leads to the mismatch of the request resource and the container resource and many other potential issues if the user requests containers with vcore > 1.



--
This message was sent by Atlassian JIRA
(v6.2#6252)