You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by Bhupesh Chawda <bh...@datatorrent.com> on 2015/11/09 07:57:43 UTC
Managing state within an application window
Hi All,
I have a question regarding maintaining state of a variable within a single
application window.
If I have a variable which is just needed for a single application window
(just between beginWindow and endWindow calls), do I need to serialize and
checkpoint it? or can I make it transient?
I understand that in case of checkpoint windows being aligned with the
application windows, we can do so. Since in this case, the variable will
start afresh immediately after the checkpoint. However in case of
checkpoints happening in between application windows, this might not be the
case.
In the second case, is there any mechanism using which I can conditionally
make it transient and non-transient based on when the checkpoints happen
with respect to the application windows?
Thanks.
Bhupesh
Re: Managing state within an application window
Posted by Bhupesh Chawda <bh...@datatorrent.com>.
Thanks Gaurav, that helps.
-Bhupesh
On Mon, Nov 9, 2015 at 1:57 PM, Gaurav Gupta <ga...@datatorrent.com> wrote:
> Bhupesh,
>
> Platform supports sliding window feature natively. You can refer to
> SLIDE_BY_WINDOW_COUNT attribute (
> https://github.com/apache/incubator-apex-core/blob/master/api/src/main/java/com/datatorrent/api/Context.java
> <
> https://github.com/apache/incubator-apex-core/blob/master/api/src/main/java/com/datatorrent/api/Context.java>)
> and you can also look at SliderTest
> https://github.com/apache/incubator-apex-core/blob/master/engine/src/test/java/com/datatorrent/stram/engine/SliderTest.java
>
> Thanks
> - Gaurav
>
> > On Nov 9, 2015, at 12:00 AM, Bhupesh Chawda <bh...@datatorrent.com>
> wrote:
> >
> > Yes, it will be reset in the endWindow call. So nothing should be needed
> to
> > be done explicitly.
> > Also, after the crash, the operator is restored to the checkpoint which
> had
> > happened after beginWindow but before endWindow. In this case the
> > beginWindow() call will be skipped when the operator restarts after
> > recovery.
> > It is easy to wrongly assume (which I did) that checkpoints always happen
> > "after endWindow" (in which case, the variable can be made transient).
> >
> > I also had a concern regarding *sliding application windows*. It is not
> > clear as to how the beginWindow and endWindow calls would be structured.
> In
> > addition to the application window size, we would also need some *delta*
> > which is the amount by which the window would "slide" by. How is this
> > specified? Is this per tuple? Also how would checkpointing and recovery
> act
> > in such a case?
> >
> > Thanks.
> > Bhupesh
> >
> > On Mon, Nov 9, 2015 at 12:39 PM, Milind Barve <mi...@gmail.com> wrote:
> >
> >> You might want to initialize it in beginwidow() as well. Should keep it
> non
> >> transient so for recovery.
> >> On Nov 9, 2015 12:34 PM, "Amol Kekre" <am...@datatorrent.com> wrote:
> >>
> >>> Bhupesh.
> >>> I would recommend reseting the variable in endWindow call and leaving
> it
> >>> non-transient. Usually an empty variable should not add too much to
> >>> serialization.
> >>>
> >>> Thks,
> >>> Amol
> >>>
> >>>
> >>> On Sun, Nov 8, 2015 at 10:57 PM, Bhupesh Chawda <
> bhupesh@datatorrent.com
> >>>
> >>> wrote:
> >>>
> >>>> Hi All,
> >>>>
> >>>> I have a question regarding maintaining state of a variable within a
> >>> single
> >>>> application window.
> >>>> If I have a variable which is just needed for a single application
> >> window
> >>>> (just between beginWindow and endWindow calls), do I need to serialize
> >>> and
> >>>> checkpoint it? or can I make it transient?
> >>>>
> >>>> I understand that in case of checkpoint windows being aligned with the
> >>>> application windows, we can do so. Since in this case, the variable
> >> will
> >>>> start afresh immediately after the checkpoint. However in case of
> >>>> checkpoints happening in between application windows, this might not
> be
> >>> the
> >>>> case.
> >>>>
> >>>> In the second case, is there any mechanism using which I can
> >>> conditionally
> >>>> make it transient and non-transient based on when the checkpoints
> >> happen
> >>>> with respect to the application windows?
> >>>>
> >>>> Thanks.
> >>>> Bhupesh
> >>>>
> >>>
> >>
>
>
Re: Managing state within an application window
Posted by Gaurav Gupta <ga...@datatorrent.com>.
Bhupesh,
Platform supports sliding window feature natively. You can refer to SLIDE_BY_WINDOW_COUNT attribute (https://github.com/apache/incubator-apex-core/blob/master/api/src/main/java/com/datatorrent/api/Context.java <https://github.com/apache/incubator-apex-core/blob/master/api/src/main/java/com/datatorrent/api/Context.java>) and you can also look at SliderTest https://github.com/apache/incubator-apex-core/blob/master/engine/src/test/java/com/datatorrent/stram/engine/SliderTest.java
Thanks
- Gaurav
> On Nov 9, 2015, at 12:00 AM, Bhupesh Chawda <bh...@datatorrent.com> wrote:
>
> Yes, it will be reset in the endWindow call. So nothing should be needed to
> be done explicitly.
> Also, after the crash, the operator is restored to the checkpoint which had
> happened after beginWindow but before endWindow. In this case the
> beginWindow() call will be skipped when the operator restarts after
> recovery.
> It is easy to wrongly assume (which I did) that checkpoints always happen
> "after endWindow" (in which case, the variable can be made transient).
>
> I also had a concern regarding *sliding application windows*. It is not
> clear as to how the beginWindow and endWindow calls would be structured. In
> addition to the application window size, we would also need some *delta*
> which is the amount by which the window would "slide" by. How is this
> specified? Is this per tuple? Also how would checkpointing and recovery act
> in such a case?
>
> Thanks.
> Bhupesh
>
> On Mon, Nov 9, 2015 at 12:39 PM, Milind Barve <mi...@gmail.com> wrote:
>
>> You might want to initialize it in beginwidow() as well. Should keep it non
>> transient so for recovery.
>> On Nov 9, 2015 12:34 PM, "Amol Kekre" <am...@datatorrent.com> wrote:
>>
>>> Bhupesh.
>>> I would recommend reseting the variable in endWindow call and leaving it
>>> non-transient. Usually an empty variable should not add too much to
>>> serialization.
>>>
>>> Thks,
>>> Amol
>>>
>>>
>>> On Sun, Nov 8, 2015 at 10:57 PM, Bhupesh Chawda <bhupesh@datatorrent.com
>>>
>>> wrote:
>>>
>>>> Hi All,
>>>>
>>>> I have a question regarding maintaining state of a variable within a
>>> single
>>>> application window.
>>>> If I have a variable which is just needed for a single application
>> window
>>>> (just between beginWindow and endWindow calls), do I need to serialize
>>> and
>>>> checkpoint it? or can I make it transient?
>>>>
>>>> I understand that in case of checkpoint windows being aligned with the
>>>> application windows, we can do so. Since in this case, the variable
>> will
>>>> start afresh immediately after the checkpoint. However in case of
>>>> checkpoints happening in between application windows, this might not be
>>> the
>>>> case.
>>>>
>>>> In the second case, is there any mechanism using which I can
>>> conditionally
>>>> make it transient and non-transient based on when the checkpoints
>> happen
>>>> with respect to the application windows?
>>>>
>>>> Thanks.
>>>> Bhupesh
>>>>
>>>
>>
Re: Managing state within an application window
Posted by Bhupesh Chawda <bh...@datatorrent.com>.
Yes, it will be reset in the endWindow call. So nothing should be needed to
be done explicitly.
Also, after the crash, the operator is restored to the checkpoint which had
happened after beginWindow but before endWindow. In this case the
beginWindow() call will be skipped when the operator restarts after
recovery.
It is easy to wrongly assume (which I did) that checkpoints always happen
"after endWindow" (in which case, the variable can be made transient).
I also had a concern regarding *sliding application windows*. It is not
clear as to how the beginWindow and endWindow calls would be structured. In
addition to the application window size, we would also need some *delta*
which is the amount by which the window would "slide" by. How is this
specified? Is this per tuple? Also how would checkpointing and recovery act
in such a case?
Thanks.
Bhupesh
On Mon, Nov 9, 2015 at 12:39 PM, Milind Barve <mi...@gmail.com> wrote:
> You might want to initialize it in beginwidow() as well. Should keep it non
> transient so for recovery.
> On Nov 9, 2015 12:34 PM, "Amol Kekre" <am...@datatorrent.com> wrote:
>
> > Bhupesh.
> > I would recommend reseting the variable in endWindow call and leaving it
> > non-transient. Usually an empty variable should not add too much to
> > serialization.
> >
> > Thks,
> > Amol
> >
> >
> > On Sun, Nov 8, 2015 at 10:57 PM, Bhupesh Chawda <bhupesh@datatorrent.com
> >
> > wrote:
> >
> > > Hi All,
> > >
> > > I have a question regarding maintaining state of a variable within a
> > single
> > > application window.
> > > If I have a variable which is just needed for a single application
> window
> > > (just between beginWindow and endWindow calls), do I need to serialize
> > and
> > > checkpoint it? or can I make it transient?
> > >
> > > I understand that in case of checkpoint windows being aligned with the
> > > application windows, we can do so. Since in this case, the variable
> will
> > > start afresh immediately after the checkpoint. However in case of
> > > checkpoints happening in between application windows, this might not be
> > the
> > > case.
> > >
> > > In the second case, is there any mechanism using which I can
> > conditionally
> > > make it transient and non-transient based on when the checkpoints
> happen
> > > with respect to the application windows?
> > >
> > > Thanks.
> > > Bhupesh
> > >
> >
>
Re: Managing state within an application window
Posted by Milind Barve <mi...@gmail.com>.
You might want to initialize it in beginwidow() as well. Should keep it non
transient so for recovery.
On Nov 9, 2015 12:34 PM, "Amol Kekre" <am...@datatorrent.com> wrote:
> Bhupesh.
> I would recommend reseting the variable in endWindow call and leaving it
> non-transient. Usually an empty variable should not add too much to
> serialization.
>
> Thks,
> Amol
>
>
> On Sun, Nov 8, 2015 at 10:57 PM, Bhupesh Chawda <bh...@datatorrent.com>
> wrote:
>
> > Hi All,
> >
> > I have a question regarding maintaining state of a variable within a
> single
> > application window.
> > If I have a variable which is just needed for a single application window
> > (just between beginWindow and endWindow calls), do I need to serialize
> and
> > checkpoint it? or can I make it transient?
> >
> > I understand that in case of checkpoint windows being aligned with the
> > application windows, we can do so. Since in this case, the variable will
> > start afresh immediately after the checkpoint. However in case of
> > checkpoints happening in between application windows, this might not be
> the
> > case.
> >
> > In the second case, is there any mechanism using which I can
> conditionally
> > make it transient and non-transient based on when the checkpoints happen
> > with respect to the application windows?
> >
> > Thanks.
> > Bhupesh
> >
>
Re: Managing state within an application window
Posted by Amol Kekre <am...@datatorrent.com>.
Bhupesh.
I would recommend reseting the variable in endWindow call and leaving it
non-transient. Usually an empty variable should not add too much to
serialization.
Thks,
Amol
On Sun, Nov 8, 2015 at 10:57 PM, Bhupesh Chawda <bh...@datatorrent.com>
wrote:
> Hi All,
>
> I have a question regarding maintaining state of a variable within a single
> application window.
> If I have a variable which is just needed for a single application window
> (just between beginWindow and endWindow calls), do I need to serialize and
> checkpoint it? or can I make it transient?
>
> I understand that in case of checkpoint windows being aligned with the
> application windows, we can do so. Since in this case, the variable will
> start afresh immediately after the checkpoint. However in case of
> checkpoints happening in between application windows, this might not be the
> case.
>
> In the second case, is there any mechanism using which I can conditionally
> make it transient and non-transient based on when the checkpoints happen
> with respect to the application windows?
>
> Thanks.
> Bhupesh
>