You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@samza.apache.org by "Jagadish (JIRA)" <ji...@apache.org> on 2016/02/29 20:30:18 UTC

[jira] [Created] (SAMZA-881) Re-think the Samza Job Coordinator

Jagadish created SAMZA-881:
------------------------------

             Summary: Re-think the Samza Job Coordinator
                 Key: SAMZA-881
                 URL: https://issues.apache.org/jira/browse/SAMZA-881
             Project: Samza
          Issue Type: Improvement
            Reporter: Jagadish
            Assignee: Jagadish


Currently, the only way to run Samza containers in distributed mode is using Yarn. However, there has been interest to run Samza on top of other resource managers with the recent explosion in the # of such systems. Users have also requested us to run Samza as a library, and to run Samza on Docker containers managed by Kubernetes.

We must re-think the JobCoordinator functionality as follows:
1. ID assignment: Provide an ID to each SamzaContainer.
2. JobModel agreement: Ensure containers agree on a JobModel.
3. Re-start the SamzaContainer when the job model changes. 

This will arguably require some leader election (depending on how users choose to run Samza).




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)