You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@samza.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2017/06/17 00:22:00 UTC
[jira] [Commented] (SAMZA-1334) When host-affinity is turned off,
ContainerAllocator should ignore any previous container locality info
[ https://issues.apache.org/jira/browse/SAMZA-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16052569#comment-16052569 ]
ASF GitHub Bot commented on SAMZA-1334:
---------------------------------------
GitHub user nickpan47 opened a pull request:
https://github.com/apache/samza/pull/228
SAMZA-1334: fix pre-condition for ContainerAllocator to work properly
We have observed issues when the LocalityManager reports the container locality mapping while the host-affinity is disabled in ContainerAllocator, in which the ContainerAllocator failed to release extra containers.
Hence, fix is in the form of make sure the pre-condition is met for the ContainerAllocator w/o host-affinity: the localityMap from the JobModel should contain no preferred host info.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickpan47/samza SAMZA-1334
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/samza/pull/228.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #228
----
commit f76fff14d17f9c99dcfaa3d065f90bb5a656c705
Author: Yi Pan (Data Infrastructure) <ni...@gmail.com>
Date: 2017-06-15T17:26:15Z
WIP: SAMZA-1334 fix
commit ad3320fc66f49c5c2637b8e0dd38cf129b486f66
Author: Yi Pan (Data Infrastructure) <ni...@gmail.com>
Date: 2017-06-17T00:16:13Z
SAMZA-1334: fix the pre-conditions for ContainerAllocator to work properly. Make sure JobModel is generated w/o LocalityManager if host-affinity is disabled
----
> When host-affinity is turned off, ContainerAllocator should ignore any previous container locality info
> -------------------------------------------------------------------------------------------------------
>
> Key: SAMZA-1334
> URL: https://issues.apache.org/jira/browse/SAMZA-1334
> Project: Samza
> Issue Type: Bug
> Reporter: Yi Pan (Data Infrastructure)
>
> Consider a case where host affinity is turned on once for a job, and the locality info is written to the coordinator stream. Then the user may turn off the host affinity feature.
> That triggers a bug in ContainerAllocator:
> 1) it gets the locality map from JobModel which has the list of preferred hosts from the coordinator stream. Hence, ContainerAllocator is making preferred host resource requests.
> 2) At the end, ContainerAllocator finishes launching all containers and tries to release all extra containers mapping to ANY_HOST. However, all preferred host resource responses are kept under the specific host's entry. Hence, it failed to release those containers.
> The end result is: the job is still successfully launched. However, YARN RM reports a lot of reserved memory/containers not released by the job. In some extreme cases, the reserved memory/container can be huge and affects the availability of the whole cluster.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)