[jira] [Commented] (SAMZA-881) Re-think the Samza Job Coordinator

["Yi Pan (Data Infrastructure) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SAMZA-881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15181405#comment-15181405 ] 

Yi Pan (Data Infrastructure) commented on SAMZA-881:
----------------------------------------------------

Hi, [~jagadish1989@gmail.com], the proposal looks good to me. I am attaching a few comments below:

# When we describe mutual exclusiveness in partition assignment, please exclude broadcast stream in the discussion
# Be consistent w/ terms: in page 2, “leader container” vs “leader process”
# In the architecture graph, it would be nice to label the text on the edges w/ execution order
# Where is the container liveness management module in the design of JobCoordinator? W/ SAMZA-871, requesting for direct heart beat between the containers and AM (i.e. essentially followers and leaders in the new design), I think that we should have a separate pluggable module for this, in addition to ContainerProcessManager, which is just interface to allocate/request processes.
# Case 3.2 is the prototype implemented in SAMZA-516, right? We should call it out.

Thanks a lot for the design!

> Re-think the Samza Job Coordinator
> ----------------------------------
>
>                 Key: SAMZA-881
>                 URL: https://issues.apache.org/jira/browse/SAMZA-881
>             Project: Samza
>          Issue Type: Improvement
>            Reporter: Jagadish
>            Assignee: Jagadish
>         Attachments: SamzaJobCoordinatorRe-designProposal.pdf
>
>
> Currently, the only way to run Samza containers in distributed mode is using Yarn. However, there has been interest to run Samza on top of other resource managers with the recent explosion in the # of such systems. Users have also requested us to run Samza as a library, and to run Samza on Docker containers managed by Kubernetes.
> We must re-think the JobCoordinator functionality as follows:
> 1. ID assignment: Provide an ID to each SamzaContainer.
> 2. JobModel agreement: Ensure containers agree on a JobModel.
> 3. Re-start the SamzaContainer when the job model changes. 
> This will arguably require some leader election (depending on how users choose to run Samza).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)