[jira] [Commented] (SAMZA-42) Add a job setup phase to Samza

["Martin Kleppmann (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SAMZA-42?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942569#comment-13942569 ] 

Martin Kleppmann commented on SAMZA-42:
---------------------------------------

I noticed that KafkaCheckpointManager checks whether it is responsible for partition 0, and only creates the topic if that is the case. Using the fact that partition 0 always exists and is always assigned to exactly one container, that provides a simple way of performing a startup task in just one container.

Is that approach sufficient, or are there use cases that are not covered by the partition-0 trick?

> Add a job setup phase to Samza
> ------------------------------
>
>                 Key: SAMZA-42
>                 URL: https://issues.apache.org/jira/browse/SAMZA-42
>             Project: Samza
>          Issue Type: Bug
>          Components: container
>    Affects Versions: 0.6.0
>            Reporter: Chris Riccomini
>
> We have several use cases for doing things once at the beginning of a Samza job's execution (before containers start). Examples:
> * Validate or create checkpoint topic (if using KafkaCheckpointManager)
> * Validate or create state topic (if using LoggedStore)
> Right now, we have to do this in the container, which means that there's a race condition when running on YARN, as each container will try to create the same topic.
> Initially, I thought this logic could be put in the YARN AM, but then we'd have to put corresponding logic in the LocalJobFactory. This gets problematic if we implement SAMZA-41, since there would no longer be a central place to do a "before job starts" operation with the LocalJobFactory. If we don't do SAMZA-41, then we should be fine putting this logic in the YARN AM and LocalJobFactory.
> Alternatively, we could put this logic in JobRunner. One downside to this is that it would mean the JobRunner would need full access to the grid that it was trying to execute on (not just the RM) so that it could talk to Kafka/ZooKeeper (for example). I think this is actually fine, since we always execute our jobs from a spot that has access to the full grid.



--
This message was sent by Atlassian JIRA
(v6.2#6252)