You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Sowmya Vallabhajosyula <so...@zemosolabs.com> on 2016/04/25 11:22:34 UTC
Clear irrelevant state values
Hi,
Scenario: Health care where a list of patient events are flowing in. We
would like to keep a Value / List state holding all events and updating the
state based on a set of business rules. For e.g. if 4 vitals exceed range
in 24 hours, state is sirs. If the patient is in sirs state and a source of
infection is reported now, update state to sepsis. Attached flow for your
reference.
Considering the example of RideSource, if I use a RichFlatMapFunction to
maintain this state (please let me know if this doesn't make sense),
(a) out.collect will return the current patient state to a sink so at the
end of every event, we know the state of the patient at this instance. Is
my understanding right?
(b) Let's say after x days, these events that we recorded are not valid
anymore. How do we clear the state?
--
Thanks and Regards,
Sowmya Vallabhajosyula
Re: Clear irrelevant state values
Posted by Sowmya Vallabhajosyula <so...@zemosolabs.com>.
Thanks Gyula.
Yes, I am using state only in RichFlatMapFunction. Will try to evaluate
generating events for removal of state.
Regards,
Sowmya
On Mon, Apr 25, 2016 at 5:44 PM, Gyula Fóra <gy...@gmail.com> wrote:
> Hi,
>
> The removal markers are just something I made up :) What I meant is that
> you can generate events in a custom source for instance that will trigger
> the removal of the state. This might be easy or hard to do depending on
> your use-case.
>
> What do you mean by custom state maintenance? As long as you are using the
> state interfaces correctly in your functions you should be fine in terms of
> scalability.
>
> Gyula
>
> Sowmya Vallabhajosyula <so...@zemosolabs.com> ezt írta (időpont: 2016.
> ápr. 25., H, 13:29):
>
>> Hi Gyula,
>>
>> Thank you so much.
>>
>> 1. Can you point me to any documentation on removal markers?
>> 2. My understanding is this implementation of custom state maintenance
>> does not impact scalabiity. Is that right?
>>
>> Thanks,
>> Sowmya
>>
>> On Mon, Apr 25, 2016 at 3:06 PM, Gyula Fóra <gy...@apache.org> wrote:
>>
>>> Hi,
>>>
>>> (a) I think your understanding is correct, one consideration might be
>>> that if you are always sending the state to the sink, it might make sense
>>> to build it there directly using a RichSinkFunction.
>>>
>>> (b) There is no built-in support for this at the moment. What you can do
>>> yourself is to generate removal markers for the patients automatically. We
>>> could probably add this feature later which might be easier to implement
>>> for some state backends. For instance in RocksDB we could use a
>>> time-to-live database to remove states after a given period.
>>>
>>> Cheers,
>>> Gyula
>>>
>>> Sowmya Vallabhajosyula <so...@zemosolabs.com> ezt írta (időpont:
>>> 2016. ápr. 25., H, 11:22):
>>>
>>>> Hi,
>>>>
>>>> Scenario: Health care where a list of patient events are flowing in. We
>>>> would like to keep a Value / List state holding all events and updating the
>>>> state based on a set of business rules. For e.g. if 4 vitals exceed range
>>>> in 24 hours, state is sirs. If the patient is in sirs state and a source of
>>>> infection is reported now, update state to sepsis. Attached flow for your
>>>> reference.
>>>>
>>>> Considering the example of RideSource, if I use a RichFlatMapFunction
>>>> to maintain this state (please let me know if this doesn't make sense),
>>>>
>>>> (a) out.collect will return the current patient state to a sink so at
>>>> the end of every event, we know the state of the patient at this instance.
>>>> Is my understanding right?
>>>> (b) Let's say after x days, these events that we recorded are not valid
>>>> anymore. How do we clear the state?
>>>>
>>>> --
>>>> Thanks and Regards,
>>>> Sowmya Vallabhajosyula
>>>>
>>>
>>
>>
>> --
>> Thanks and Regards,
>> Sowmya Vallabhajosyula
>>
>
--
Thanks and Regards,
Sowmya Vallabhajosyula
Re: Clear irrelevant state values
Posted by Gyula Fóra <gy...@gmail.com>.
Hi,
The removal markers are just something I made up :) What I meant is that
you can generate events in a custom source for instance that will trigger
the removal of the state. This might be easy or hard to do depending on
your use-case.
What do you mean by custom state maintenance? As long as you are using the
state interfaces correctly in your functions you should be fine in terms of
scalability.
Gyula
Sowmya Vallabhajosyula <so...@zemosolabs.com> ezt írta (időpont: 2016.
ápr. 25., H, 13:29):
> Hi Gyula,
>
> Thank you so much.
>
> 1. Can you point me to any documentation on removal markers?
> 2. My understanding is this implementation of custom state maintenance
> does not impact scalabiity. Is that right?
>
> Thanks,
> Sowmya
>
> On Mon, Apr 25, 2016 at 3:06 PM, Gyula Fóra <gy...@apache.org> wrote:
>
>> Hi,
>>
>> (a) I think your understanding is correct, one consideration might be
>> that if you are always sending the state to the sink, it might make sense
>> to build it there directly using a RichSinkFunction.
>>
>> (b) There is no built-in support for this at the moment. What you can do
>> yourself is to generate removal markers for the patients automatically. We
>> could probably add this feature later which might be easier to implement
>> for some state backends. For instance in RocksDB we could use a
>> time-to-live database to remove states after a given period.
>>
>> Cheers,
>> Gyula
>>
>> Sowmya Vallabhajosyula <so...@zemosolabs.com> ezt írta (időpont:
>> 2016. ápr. 25., H, 11:22):
>>
>>> Hi,
>>>
>>> Scenario: Health care where a list of patient events are flowing in. We
>>> would like to keep a Value / List state holding all events and updating the
>>> state based on a set of business rules. For e.g. if 4 vitals exceed range
>>> in 24 hours, state is sirs. If the patient is in sirs state and a source of
>>> infection is reported now, update state to sepsis. Attached flow for your
>>> reference.
>>>
>>> Considering the example of RideSource, if I use a RichFlatMapFunction to
>>> maintain this state (please let me know if this doesn't make sense),
>>>
>>> (a) out.collect will return the current patient state to a sink so at
>>> the end of every event, we know the state of the patient at this instance.
>>> Is my understanding right?
>>> (b) Let's say after x days, these events that we recorded are not valid
>>> anymore. How do we clear the state?
>>>
>>> --
>>> Thanks and Regards,
>>> Sowmya Vallabhajosyula
>>>
>>
>
>
> --
> Thanks and Regards,
> Sowmya Vallabhajosyula
>
Re: Clear irrelevant state values
Posted by Sowmya Vallabhajosyula <so...@zemosolabs.com>.
Hi Gyula,
Thank you so much.
1. Can you point me to any documentation on removal markers?
2. My understanding is this implementation of custom state maintenance does
not impact scalabiity. Is that right?
Thanks,
Sowmya
On Mon, Apr 25, 2016 at 3:06 PM, Gyula Fóra <gy...@apache.org> wrote:
> Hi,
>
> (a) I think your understanding is correct, one consideration might be that
> if you are always sending the state to the sink, it might make sense to
> build it there directly using a RichSinkFunction.
>
> (b) There is no built-in support for this at the moment. What you can do
> yourself is to generate removal markers for the patients automatically. We
> could probably add this feature later which might be easier to implement
> for some state backends. For instance in RocksDB we could use a
> time-to-live database to remove states after a given period.
>
> Cheers,
> Gyula
>
> Sowmya Vallabhajosyula <so...@zemosolabs.com> ezt írta (időpont: 2016.
> ápr. 25., H, 11:22):
>
>> Hi,
>>
>> Scenario: Health care where a list of patient events are flowing in. We
>> would like to keep a Value / List state holding all events and updating the
>> state based on a set of business rules. For e.g. if 4 vitals exceed range
>> in 24 hours, state is sirs. If the patient is in sirs state and a source of
>> infection is reported now, update state to sepsis. Attached flow for your
>> reference.
>>
>> Considering the example of RideSource, if I use a RichFlatMapFunction to
>> maintain this state (please let me know if this doesn't make sense),
>>
>> (a) out.collect will return the current patient state to a sink so at the
>> end of every event, we know the state of the patient at this instance. Is
>> my understanding right?
>> (b) Let's say after x days, these events that we recorded are not valid
>> anymore. How do we clear the state?
>>
>> --
>> Thanks and Regards,
>> Sowmya Vallabhajosyula
>>
>
--
Thanks and Regards,
Sowmya Vallabhajosyula
Re: Clear irrelevant state values
Posted by Gyula Fóra <gy...@apache.org>.
Hi,
(a) I think your understanding is correct, one consideration might be that
if you are always sending the state to the sink, it might make sense to
build it there directly using a RichSinkFunction.
(b) There is no built-in support for this at the moment. What you can do
yourself is to generate removal markers for the patients automatically. We
could probably add this feature later which might be easier to implement
for some state backends. For instance in RocksDB we could use a
time-to-live database to remove states after a given period.
Cheers,
Gyula
Sowmya Vallabhajosyula <so...@zemosolabs.com> ezt írta (időpont: 2016.
ápr. 25., H, 11:22):
> Hi,
>
> Scenario: Health care where a list of patient events are flowing in. We
> would like to keep a Value / List state holding all events and updating the
> state based on a set of business rules. For e.g. if 4 vitals exceed range
> in 24 hours, state is sirs. If the patient is in sirs state and a source of
> infection is reported now, update state to sepsis. Attached flow for your
> reference.
>
> Considering the example of RideSource, if I use a RichFlatMapFunction to
> maintain this state (please let me know if this doesn't make sense),
>
> (a) out.collect will return the current patient state to a sink so at the
> end of every event, we know the state of the patient at this instance. Is
> my understanding right?
> (b) Let's say after x days, these events that we recorded are not valid
> anymore. How do we clear the state?
>
> --
> Thanks and Regards,
> Sowmya Vallabhajosyula
>