[jira] [Updated] (SAMZA-906) Host Affinity - Minimize task reassignment when container count changes

["Jake Maes (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SAMZA-906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jake Maes updated SAMZA-906:
----------------------------
    Attachment: SAMZA-906_5.patch

Patch 5 is just a rebase.

> Host Affinity - Minimize task reassignment when container count changes
> -----------------------------------------------------------------------
>
>                 Key: SAMZA-906
>                 URL: https://issues.apache.org/jira/browse/SAMZA-906
>             Project: Samza
>          Issue Type: Bug
>            Reporter: Jake Maes
>            Assignee: Jake Maes
>             Fix For: 0.10.1
>
>         Attachments: SAMZA-906.patch, SAMZA-906_2.patch, SAMZA-906_3.patch, SAMZA-906_3.patch, SAMZA-906_4.patch, SAMZA-906_5.patch
>
>
> When the job container count changes the default taskname grouper (GroupByContainerCount) will reassign all the tasks among the new container list in a round-robin fashion. This causes many of the tasks to shift to new containers. The shifted tasks will be unable to restore state from local disk because the new containers may not be assigned to the same host that the task's original container was assigned to.
> This ticket is to implement a task-to-container affinity, which complements the container-to-host affinity in the current implementation. The implementation will include a mapping which is persisted to the coordinator stream and is used as the basis for the new task-to-container mapping (ContainerModel). 
> If the container count doesn't change, the old task mapping will be used. (Note that this will allow tools to inject custom mappings by writing to the coordinator stream). 
> If the container count changes, a minimal number of tasks will be reassigned from the persisted mapping, in order to "balance" the containers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)