[DISCUSS] Re-thinking the Samza Job Coordinator

[Jagadish Venkatraman <ja...@gmail.com>]

Hi all,

Currently, the only way to run distributed Samza containers is using Yarn.
There have been requests to run Samza with other resource managers like
Mesos, to run Samza as an embedded library, and to run in Docker containers
managed by ECS/Kubernetes.

I created SAMZA-881 for this proposing a re-design of the Samza
coordinator.


   - The new re-design views the JobCoordinator as a component that can
   provide the containerId, and JobModel to the SamzaContainer.
   - There are pluggable interfaces defined for Leader Election, and
   interacting with a Resource Manager like Yarn/Mesos.
   - Primitives for dynamically refreshing the job model.

I've attached a design doc with the Jira ticket, and I'd really appreciate
your feedback on the ticket.

Thanks,
Jagadish

-- 
Jagadish V,
Graduate Student,
Department of Computer Science,
Stanford University

Re: [DISCUSS] Re-thinking the Samza Job Coordinator

[xinyu liu <xi...@gmail.com>]

This is awesome! Samza will be able to run standalone or on different kinds
of clusters, which can serve a wide range of applications. Glad to see this
is happening.

Thanks,
Xinyu

On Tue, Mar 1, 2016 at 8:48 AM, Jagadish Venkatraman <jagadish1989@gmail.com
> wrote:

> Hi all,
>
> Currently, the only way to run distributed Samza containers is using Yarn.
> There have been requests to run Samza with other resource managers like
> Mesos, to run Samza as an embedded library, and to run in Docker containers
> managed by ECS/Kubernetes.
>
> I created SAMZA-881 for this proposing a re-design of the Samza
> coordinator.
>
>
>    - The new re-design views the JobCoordinator as a component that can
>    provide the containerId, and JobModel to the SamzaContainer.
>    - There are pluggable interfaces defined for Leader Election, and
>    interacting with a Resource Manager like Yarn/Mesos.
>    - Primitives for dynamically refreshing the job model.
>
> I've attached a design doc with the Jira ticket, and I'd really appreciate
> your feedback on the ticket.
>
> Thanks,
> Jagadish
>
> --
> Jagadish V,
> Graduate Student,
> Department of Computer Science,
> Stanford University
>