[jira] [Commented] (SAMZA-255) Rewinding Streams within a StreamTask

["Martin Kleppmann (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SAMZA-255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985539#comment-13985539 ] 

Martin Kleppmann commented on SAMZA-255:
----------------------------------------

Thanks for this idea. Trying to dig into it a bit further:

For use case 1, it sounds like what you really need is for the checkpoint to happen at a point in time determined by the task (namely the start of your window). Is that right? I think you can already achieve that by setting task.commit.ms to a very large value, and calling coordinator.commit() when you want to write a checkpoint.

For use case 2, are you trying to synchronize several input streams/partitions based on some timestamp within the messages themselves? You might be able to achieve some of what you want by using a custom MessageChooser; if one stream is getting ahead of the others, you can hold off consuming it until the other streams have got further along.

I'm trying to understand whether you really need this feature, because so far we have been keen to keep Samza always consuming messages in the order they appear in streams. Allowing tasks to seek within streams would break that assumption. It's very nice to be able to assume a strict order of messages per partition, because it makes the processing model much easier to reason about, makes fault tolerance semantics much more tractable, and allows us to make performance optimizations.

For those reasons, we'd have to think very carefully about whether we want to allow rewinding streams. It's clearly a powerful feature, perhaps _too_ powerful.

> Rewinding Streams within a StreamTask
> -------------------------------------
>
>                 Key: SAMZA-255
>                 URL: https://issues.apache.org/jira/browse/SAMZA-255
>             Project: Samza
>          Issue Type: Wish
>            Reporter: Nicolas Bär
>            Priority: Minor
>
> The many benefits of Kafka include persistent storage and its resulting possibility to rewind streams to a specific offset. Samza does currently not support rewinding of streams within a StreamTask. I'd like to place this functionality as a feature request and provide two use cases to further describe the benefits of such a feature. Let's consider a general use case to aggregate values within sliding windows.
> 1. Offline-Processing
> In case of offline-processing the sliding window does not correlate to the system time. In this case any node failure will result in samza restoring from a checkpointed offset that most probably does not match the beginning of the most recent sliding window. But in order to gain precise results, one could rewind to the specific offset and process the missing events of the sliding window. The same holds for any use case where the data has to be processed in small batches and these batches do not correspond to the system time.
> 2. Late Arrival
> Messages might get delayed before they are stored into Kafka. In this case one could rewind the offset in order to process older messages corresponding to the same sliding window.
> I'd be happy to further discuss these cases and the proposed feature request.



--
This message was sent by Atlassian JIRA
(v6.2#6252)