[jira] [Created] (GOBBLIN-1823) Improving Container Calculation and Allocation Methodology

["Zihan Li (Jira)" <ji...@apache.org>]

Zihan Li created GOBBLIN-1823:
---------------------------------

             Summary: Improving Container Calculation and Allocation Methodology
                 Key: GOBBLIN-1823
                 URL: https://issues.apache.org/jira/browse/GOBBLIN-1823
             Project: Apache Gobblin
          Issue Type: Improvement
            Reporter: Zihan Li


When Yarn allocates "ghost containers" without calling the onContainerAllocated() method and when the container is eventually released, onContainersCompleted() is called, container numbers mismatches can occur. 

In the onContainerAllocated() method, we add the container to the containerMap using the container ID as the key, and increase the count for the specific tag.

In the onContainersCompleted() method, we remove the container from the containerMap and decrease the count. However, in some cases, we find that the containerMap does not contain the ID, and we ignore this while still decreasing the number of the allocated tag. We do this because sometimes onContainersCompleted() is called before onContainerAllocated() for the same container.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)