You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@samza.apache.org by "Chris Riccomini (JIRA)" <ji...@apache.org> on 2014/09/06 01:23:28 UTC

[jira] [Commented] (SAMZA-405) Trying for deterministic behavior on recovery and rewind

    [ https://issues.apache.org/jira/browse/SAMZA-405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14123784#comment-14123784 ] 

Chris Riccomini commented on SAMZA-405:
---------------------------------------

bq. Ability to store and replay message chooser history. Samza could have a configuration option to save a history of the messages a task has processed. This log could be used during recovery or rewind to replay messages in a deterministic order.

If we have Kafka transactionality, and atomically commit all changelog messages, output messages, and checkpoint messages as a single transaction, I'm not entirely sure that we need this. In such a case, we will only ever recover on transaction boundaries, so we shouldn't care about what the MessageChooser picked before the failure, because nothing was output by the prior container (due to a failed transaction). The one area where this could be useful is if you're writing to something other than Kafka, and therefore don't have transactions. In such a case, you could mimic the idempotent producer if your inputs were deterministic and your processing were also deterministic. Is this along the lines of what you were thinking?

> Trying for deterministic behavior on recovery and rewind
> --------------------------------------------------------
>
>                 Key: SAMZA-405
>                 URL: https://issues.apache.org/jira/browse/SAMZA-405
>             Project: Samza
>          Issue Type: Improvement
>            Reporter: Roger Hoover
>
> Ideally, we want streaming tasks to produce the exact same output on recovery or rewind as they did/would during normal operation.  After thinking harder on this, I don't believe it's possible with at-least-once semantics.  I think duplicates break ordering guarantees.  For any message that updates local state, it can always be surrounded on both sides by duplicate of another message which negates it.  Nonetheless, we can get closer now and if idempotent producers later are supported by Kafka, we'll have what we want.
> See discussion here: http://mail-archives.apache.org/mod_mbox/incubator-samza-dev/201409.mbox/%3CCAPOm=TPseVPXLUDaxiNTXj9Z54gYEANOCV5dk7NbotSGPD-5tw@mail.gmail.com%3E
> Here are three changes that seem to make sense for Samza to support in order to achieve this.
> 1) Bootstrapping is only appropriate on cold start, not when restoring saved state.  On recovery, local state will be restored from the change log.
> 2) Local state should be saved and restored atomically with checkpoint state.  This may require support for transactions in Kafka.
> 3) Ability to store and replay message chooser history.  Samza could have a configuration option to save a history of the messages a task has processed.  This log could be used during recovery or rewind to replay messages in a deterministic order. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)