You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@samza.apache.org by "Martin Kleppmann (JIRA)" <ji...@apache.org> on 2014/05/02 00:41:16 UTC
[jira] [Commented] (SAMZA-252) Document stream reprocessing
[ https://issues.apache.org/jira/browse/SAMZA-252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13987106#comment-13987106 ]
Martin Kleppmann commented on SAMZA-252:
----------------------------------------
So far it doesn't talk about the details of implementing a two-job system (where one job processes real-time updates and the other reprocesses history). We could explain the "A/B streams" approach that was discussed offline, and how a downstream job can use timestamps of jobs and/or version numbers of jobs to ensure that old data doesn't overwrite new data.
It would also be nice not to have to mess with checkpoints, and instead have a simple config for resetting to offset 0 when a new version of a job is deployed (e.g. by including the job version number/UUID in the checkpoint, and ignoring the checkpoint on mismatch). However, we don't have that implemented yet, so I didn't want to mention it in the docs.
If we include everything, this is going to get quite long — which is maybe a good thing, but it becomes a lot of work for people to read. Anyone have thoughts on where we should strike the balance between comprehensiveness and conciseness in this doc?
> Document stream reprocessing
> ----------------------------
>
> Key: SAMZA-252
> URL: https://issues.apache.org/jira/browse/SAMZA-252
> Project: Samza
> Issue Type: Improvement
> Components: docs
> Affects Versions: 0.6.0
> Reporter: Chris Riccomini
> Assignee: Martin Kleppmann
> Fix For: 0.7.0, 0.8.0
>
> Attachments: SAMZA-252.1.patch
>
>
> A need with stream processing is to want to re-process prior messages at some later date. An example of this is having a stream processing job that is classifying messages in some way using a machine learning algorithm. At some point, the algorithm will be updated with a more accurate vector of weights. When this happens, usually you wish to re-process past messages to get more accurate results. Usually this is solved by running a parallel pipeline from Hadoop.
> We have thought extensively about this use case, and should document how to use Samza in a re-processing use case.
--
This message was sent by Atlassian JIRA
(v6.2#6252)