You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@samza.apache.org by "Jagadish (JIRA)" <ji...@apache.org> on 2016/02/29 20:41:18 UTC

[jira] [Updated] (SAMZA-881) Re-think the Samza Job Coordinator

     [ https://issues.apache.org/jira/browse/SAMZA-881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jagadish updated SAMZA-881:
---------------------------
    Attachment: SamzaJobCoordinatorRe-designProposal.pdf

> Re-think the Samza Job Coordinator
> ----------------------------------
>
>                 Key: SAMZA-881
>                 URL: https://issues.apache.org/jira/browse/SAMZA-881
>             Project: Samza
>          Issue Type: Improvement
>            Reporter: Jagadish
>            Assignee: Jagadish
>         Attachments: SamzaJobCoordinatorRe-designProposal.pdf
>
>
> Currently, the only way to run Samza containers in distributed mode is using Yarn. However, there has been interest to run Samza on top of other resource managers with the recent explosion in the # of such systems. Users have also requested us to run Samza as a library, and to run Samza on Docker containers managed by Kubernetes.
> We must re-think the JobCoordinator functionality as follows:
> 1. ID assignment: Provide an ID to each SamzaContainer.
> 2. JobModel agreement: Ensure containers agree on a JobModel.
> 3. Re-start the SamzaContainer when the job model changes. 
> This will arguably require some leader election (depending on how users choose to run Samza).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)