[jira] [Created] (SAMZA-405) Trying for deterministic behavior on recovery and rewind

["Roger Hoover (JIRA)" <ji...@apache.org>]

Roger Hoover created SAMZA-405:
----------------------------------

             Summary: Trying for deterministic behavior on recovery and rewind
                 Key: SAMZA-405
                 URL: https://issues.apache.org/jira/browse/SAMZA-405
             Project: Samza
          Issue Type: Improvement
            Reporter: Roger Hoover


Ideally, we want streaming tasks to produce the exact same output on recovery or rewind as they did/would during normal operation.  After thinking harder on this, I don't believe it's possible with at-least-once semantics.  I think duplicates break ordering guarantees.  For any message that updates local state, it can always be surrounded on both sides by duplicate of another message which negates it.  Nonetheless, we can get closer now and if idempotent producers later are supported by Kafka, we'll have what we want.

See discussion here: http://mail-archives.apache.org/mod_mbox/incubator-samza-dev/201409.mbox/%3CCAPOm=TPseVPXLUDaxiNTXj9Z54gYEANOCV5dk7NbotSGPD-5tw@mail.gmail.com%3E

Here are three changes that seem to make sense for Samza to support in order to achieve this.

1) Bootstrapping is only appropriate on cold start, not when restoring saved state.  On recovery, local state will be restored from the change log.
2) Local state should be saved and restored atomically with checkpoint state.  This may require support for transactions in Kafka.
3) Ability to store and replay message chooser history.  Samza could have a configuration option to save a history of the messages a task has processed.  This log could be used during recovery or rewind to replay messages in a deterministic order. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)