[jira] [Created] (SAMZA-273) Make offset-related configuration more intuitive

["Martin Kleppmann (JIRA)" <ji...@apache.org>]

Martin Kleppmann created SAMZA-273:
--------------------------------------

             Summary: Make offset-related configuration more intuitive
                 Key: SAMZA-273
                 URL: https://issues.apache.org/jira/browse/SAMZA-273
             Project: Samza
          Issue Type: Improvement
    Affects Versions: 0.7.0
            Reporter: Martin Kleppmann


There are a few configuration parameters which control the offset at which a consumer starts:

* systems.\*.samza.reset.offset (whether to ignore checkpoints on container startup)
* systems.\*.samza.offset.default (what to do if there is no checkpoint)
* systems.\*.consumer.auto.offset.reset (what to do if the requested offset is out of range of the broker's stream history)
* CheckpointTool isn't a config per se, but is also related to consumer offsets

Although they are all valid, they are not really great. The parameter names are a bit obscure (I still don't remember them, even though I've been staring at them for some time), there are subtle interactions between them, and generally I feel they are set up from the framework's internals' point of view, rather than the "what is the job trying to accomplish" point of view. (Put another way, you need to understand how Samza works internally in order to make sense of them.)

I don't have an answer of what a better design would look like. This ticket is just a place to discuss how we could make offset-related configuration easier for job authors to understand and use.



--
This message was sent by Atlassian JIRA
(v6.2#6252)