[jira] [Resolved] (SAMZA-1334) When host-affinity is turned off, ContainerAllocator should ignore any previous container locality info

["Jake Maes (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SAMZA-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jake Maes resolved SAMZA-1334.
------------------------------
    Resolution: Fixed

PR merged and closed

> When host-affinity is turned off, ContainerAllocator should ignore any previous container locality info
> -------------------------------------------------------------------------------------------------------
>
>                 Key: SAMZA-1334
>                 URL: https://issues.apache.org/jira/browse/SAMZA-1334
>             Project: Samza
>          Issue Type: Bug
>            Reporter: Yi Pan (Data Infrastructure)
>            Priority: Major
>
> Consider a case where host affinity is turned on once for a job, and the locality info is written to the coordinator stream. Then the user may turn off the host affinity feature. 
> That triggers a bug in ContainerAllocator:
> 1) it gets the locality map from JobModel which has the list of preferred hosts from the coordinator stream. Hence, ContainerAllocator is making preferred host resource requests.
> 2) At the end, ContainerAllocator finishes launching all containers and tries to release all extra containers mapping to ANY_HOST. However, all preferred host resource responses are kept under the specific host's entry. Hence, it failed to release those containers.
> The end result is: the job is still successfully launched. However, YARN RM reports a lot of reserved memory/containers not released by the job. In some extreme cases, the reserved memory/container can be huge and affects the availability of the whole cluster.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)