You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by Jagadish Venkatraman <ja...@apache.org> on 2019/07/24 18:45:45 UTC
At-least once processing guarantee
Dear Samza users,
We recently discovered an issue with the way we handle state in Samza Beam
and Samza High-Level API Window operators. Under certain situations, at
least once processing guarantee is violated.
*Details on the issue*
The Samza high-level API includes operators such as windows which can hold
messages and emit them at a later time. eg: a time based window will buffer
its messages and emit results only at periodic time intervals. Here’s the
sequence of operations when a window operator is ready to emit results:
1.
Obtain the results ready to be emitted for the window
2.
Remove the operator state corresponding to those results from its
state-store
3.
Propagate the window results to down-stream operators in the pipeline.
The results eventually makes it to a terminal operator, which emits the
final output.
4.
At some future point, issue a commit operation, which flushes the
producers, the state stores and persists the input offsets.
Scenarios
-
An exception in a downstream operator in step 3.
-
An unclean shutdown, e.g., due to a “kill -9” to the container before
the outputs have been flushed in step 4
Both these examples violate Samza’s at-least once processing guarantee,
since they cause state to be modified even though the processed outputs may
not have been emitted.
*Solution*
Here's a proposal
<https://docs.google.com/document/d/1wtSAUMrns14cWCf5Wdf4YwXd7K5Hf-bm7W3SQ3sXZsQ/edit>
to fix the above issue. We are working on addressing this with the highest
priority.
Thanks,
Jagadish
Re: At-least once processing guarantee
Posted by Jagadish Venkatraman <ja...@apache.org>.
Hello Nicholas,
This is a more comprehensive doc, that also includes rejected alternatives.
https://docs.google.com/document/d/1OEbGZlnCcYVj8zBF-pUgLL69SFxOgkEDVzRpqK7bG_g/edit?usp=sharing
Thanks,
Jagadish
On Thu, Jul 25, 2019 at 8:51 AM Nicholas Audo <na...@naudo.de> wrote:
> Hello Jagadish,
> Is the rejected alternatives document meant to be private (link at the
> bottom of the document)?
>
> On Wed, Jul 24, 2019 at 11:46 AM Jagadish Venkatraman <jagadish@apache.org
> >
> wrote:
>
> > Dear Samza users,
> >
> > We recently discovered an issue with the way we handle state in Samza
> Beam
> > and Samza High-Level API Window operators. Under certain situations, at
> > least once processing guarantee is violated.
> >
> >
> > *Details on the issue*
> >
> > The Samza high-level API includes operators such as windows which can
> hold
> > messages and emit them at a later time. eg: a time based window will
> buffer
> > its messages and emit results only at periodic time intervals. Here’s the
> > sequence of operations when a window operator is ready to emit results:
> >
> > 1.
> >
> > Obtain the results ready to be emitted for the window
> > 2.
> >
> > Remove the operator state corresponding to those results from its
> > state-store
> > 3.
> >
> > Propagate the window results to down-stream operators in the pipeline.
> > The results eventually makes it to a terminal operator, which emits
> the
> > final output.
> > 4.
> >
> > At some future point, issue a commit operation, which flushes the
> > producers, the state stores and persists the input offsets.
> >
> >
> > Scenarios
> >
> > -
> >
> > An exception in a downstream operator in step 3.
> > -
> >
> > An unclean shutdown, e.g., due to a “kill -9” to the container before
> > the outputs have been flushed in step 4
> >
> >
> > Both these examples violate Samza’s at-least once processing guarantee,
> > since they cause state to be modified even though the processed outputs
> may
> > not have been emitted.
> >
> >
> > *Solution*
> >
> > Here's a proposal
> > <
> >
> https://docs.google.com/document/d/1wtSAUMrns14cWCf5Wdf4YwXd7K5Hf-bm7W3SQ3sXZsQ/edit
> > >
> > to fix the above issue. We are working on addressing this with the
> highest
> > priority.
> >
> >
> > Thanks,
> >
> > Jagadish
> >
>
Re: At-least once processing guarantee
Posted by Nicholas Audo <na...@naudo.de>.
Hello Jagadish,
Is the rejected alternatives document meant to be private (link at the
bottom of the document)?
On Wed, Jul 24, 2019 at 11:46 AM Jagadish Venkatraman <ja...@apache.org>
wrote:
> Dear Samza users,
>
> We recently discovered an issue with the way we handle state in Samza Beam
> and Samza High-Level API Window operators. Under certain situations, at
> least once processing guarantee is violated.
>
>
> *Details on the issue*
>
> The Samza high-level API includes operators such as windows which can hold
> messages and emit them at a later time. eg: a time based window will buffer
> its messages and emit results only at periodic time intervals. Here’s the
> sequence of operations when a window operator is ready to emit results:
>
> 1.
>
> Obtain the results ready to be emitted for the window
> 2.
>
> Remove the operator state corresponding to those results from its
> state-store
> 3.
>
> Propagate the window results to down-stream operators in the pipeline.
> The results eventually makes it to a terminal operator, which emits the
> final output.
> 4.
>
> At some future point, issue a commit operation, which flushes the
> producers, the state stores and persists the input offsets.
>
>
> Scenarios
>
> -
>
> An exception in a downstream operator in step 3.
> -
>
> An unclean shutdown, e.g., due to a “kill -9” to the container before
> the outputs have been flushed in step 4
>
>
> Both these examples violate Samza’s at-least once processing guarantee,
> since they cause state to be modified even though the processed outputs may
> not have been emitted.
>
>
> *Solution*
>
> Here's a proposal
> <
> https://docs.google.com/document/d/1wtSAUMrns14cWCf5Wdf4YwXd7K5Hf-bm7W3SQ3sXZsQ/edit
> >
> to fix the above issue. We are working on addressing this with the highest
> priority.
>
>
> Thanks,
>
> Jagadish
>