[jira] [Commented] (SAMZA-465) Use coordinator stream and eliminate CheckpointManager

["Chris Riccomini (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SAMZA-465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14393739#comment-14393739 ] 

Chris Riccomini commented on SAMZA-465:
---------------------------------------

Posted comments on RB. Also, was going to run some tests with this patch, but it doesn't apply to master:

{noformat}
$ git apply SAMZA-465.patch 
SAMZA-465.patch:117: trailing whitespace.
   * 
SAMZA-465.patch:128: trailing whitespace.
   * 
SAMZA-465.patch:194: trailing whitespace.
 * 
SAMZA-465.patch:198: trailing whitespace.
 * 
SAMZA-465.patch:200: trailing whitespace.
 * key => [1, "set-config", "job.name"] 
error: patch failed: samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemFactory.scala:28
error: samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemFactory.scala: patch does not apply
error: patch failed: samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManagerFactory.scala:1
error: samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManagerFactory.scala: patch does not apply
error: patch failed: samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemFactory.scala:92
error: samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemFactory.scala: patch does not apply
{noformat}

> Use coordinator stream and eliminate CheckpointManager
> ------------------------------------------------------
>
>                 Key: SAMZA-465
>                 URL: https://issues.apache.org/jira/browse/SAMZA-465
>             Project: Samza
>          Issue Type: Sub-task
>          Components: container, kafka
>    Affects Versions: 0.10.0
>            Reporter: Chris Riccomini
>            Assignee: Naveen
>             Fix For: 0.10.0
>
>         Attachments: SAMZA-465.patch
>
>
> SAMZA-448 introduced a coordinator stream into Samza, but still kept CheckpointManagers. This means that master currently has two streams, which is totally unnecessary. We should eliminate the CheckpointManager, and use the coordinator stream instead.
> This work will involve modifying the JobCoordinator to continue reading messages from the coordinator stream even after it starts up. This is required because a container may fail and restart. When it does so, the offset that it receives from the JobCoordinator (via the HTTP JSON server) must be up to date.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)