[jira] [Commented] (SAMZA-1064) Standalone Samza with Zookeeper for Coordination

["Navina Ramesh (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SAMZA-1064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15768557#comment-15768557 ] 

Navina Ramesh commented on SAMZA-1064:
--------------------------------------

We had a round of discussions based on this initial design. Here are some notes:
* Action Items (or things not yet addressed in the design):
** Split config from JobModel message in the Coordination Stream
** Job Model distribution mechanism should be defined clearly
** Add a note about Yarn: In yarn , the leader starts the processors. Hence, it can tell the container what version of job model to pick up from the CS. 

* Design document should call out the following:
** rolling bounce is not supported in this design
** what defines the lifetime of a Samza job attempt?
** Debounce time should be larger than sessionTimeout (debounceTime >> sessionTimeout)
** JobModel (and its version) should be treated as a resource that can be updated by only one processor. Hence, we should use conditional updates to fence off JobModel resource
** Figure out how to clean up the incomplete and/ old barriers

* Questions to resolve:
** Is coordination stream a suitable option for job model distribution? What kind of guarantees do we want wrt jobmodel distribution (atomicity/durability?)
** Should JM be written as a single message to the coordinator stream ? or multiple small container model messages? (1 msg / multipart message in Coordinator Stream). If it is the latter, how do we guarantee atomicity of writes? Do we need marker messages to indicate the  start/stop of the batch of job model messages?
** Do we need job attempts in the tree? If yes, who creates the persistent subtrees 
** Apart from performance, what is the benefit of having a /leader znode as a lock? Does it really provide fencing for the leader processor?

> Standalone Samza with Zookeeper for Coordination
> ------------------------------------------------
>
>                 Key: SAMZA-1064
>                 URL: https://issues.apache.org/jira/browse/SAMZA-1064
>             Project: Samza
>          Issue Type: New Feature
>            Reporter: Navina Ramesh
>         Attachments: Samza Standalone-0.md, Samza Standalone-0.pdf, image_0.png, image_1.png
>
>
> In this use-case, we propose using Zookeeper for coordinating processor liveness and task distribution in a Samza job. Additionally, it opens up the possibility of allowing a flexible number of participating processors (no fixed container count).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)