How does Flink handle shorted lived keyed streams

[narasimha <sw...@gmail.com>]

Hi,

Belos is the use case.

Have a stream of transaction events, success/failure of a transaction can
be determined by those events.
Partitioning stream by transaction id and applying CEP to determine the
success/failure of a transaction.
Each transaction keyed stream is valid only until the final status is
found. Which can end up having large inactive keyed streams in the system.

Know that using keygroup flink distributes the keyedstream to tasks based
on it, but still there will be a large set of inactive keys.

Does this have any side effects? If so what has to be done to overcome
humongous keyed streams?

-- 
A.Narasimha Swamy

Re: How does Flink handle shorted lived keyed streams

[narasimha <sw...@gmail.com>]

Thanks Matthias, I think it will help to find out what all live keys are
present.

Let me check and revert back on the thread.

On Wed, Feb 10, 2021 at 10:46 PM Matthias Pohl <ma...@ververica.com>
wrote:

> Hi narashima,
> not sure whether this fits your use case, but have you considered creating
> a savepoint and analyzing it using the State Processor API [1]?
>
> Best,
> Matthias
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html#state-processor-api
>
> On Wed, Feb 10, 2021 at 6:08 PM narasimha <sw...@gmail.com> wrote:
>
>> It is not solving the problem.
>>
>> I could see the memory keep increasing, resulting in a lot of high GCs.
>>
>> There could be a memory leak, just want to know how to know if older keps
>> are skill alive, even after the pattern has been satisfied or within range
>> of the pattern has expired.
>>
>> Can someone suggest how to proceed further.
>>
>
>

-- 
A.Narasimha Swamy

Re: How does Flink handle shorted lived keyed streams

[Matthias Pohl <ma...@ververica.com>]

Hi narashima,
not sure whether this fits your use case, but have you considered creating
a savepoint and analyzing it using the State Processor API [1]?

Best,
Matthias

[1]
https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html#state-processor-api

On Wed, Feb 10, 2021 at 6:08 PM narasimha <sw...@gmail.com> wrote:

> It is not solving the problem.
>
> I could see the memory keep increasing, resulting in a lot of high GCs.
>
> There could be a memory leak, just want to know how to know if older keps
> are skill alive, even after the pattern has been satisfied or within range
> of the pattern has expired.
>
> Can someone suggest how to proceed further.
>

Re: How does Flink handle shorted lived keyed streams

[narasimha <sw...@gmail.com>]

It is not solving the problem.

I could see the memory keep increasing, resulting in a lot of high GCs.

There could be a memory leak, just want to know how to know if older keps
are skill alive, even after the pattern has been satisfied or within range
of the pattern has expired.

Can someone suggest how to proceed further.

Re: How does Flink handle shorted lived keyed streams

[narasimha <sw...@gmail.com>]

Thanks Xintong.

I'll check it out and get back to you.

On Thu, Dec 24, 2020 at 1:30 PM Xintong Song <to...@gmail.com> wrote:

> I believe what you are looking for is the State TTL [1][2].
>
>
> Thank you~
>
> Xintong Song
>
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/state.html#state-time-to-live-ttl
>
> [2]
> https://ci.apache.org/projects/flink/flink-docs-stabledev/table/config.html#table-exec-state-ttl
>
>
>
> On Wed, Dec 23, 2020 at 11:57 PM narasimha <sw...@gmail.com> wrote:
>
>> Hi,
>>
>> Belos is the use case.
>>
>> Have a stream of transaction events, success/failure of a transaction can
>> be determined by those events.
>> Partitioning stream by transaction id and applying CEP to determine the
>> success/failure of a transaction.
>> Each transaction keyed stream is valid only until the final status is
>> found. Which can end up having large inactive keyed streams in the system.
>>
>> Know that using keygroup flink distributes the keyedstream to tasks based
>> on it, but still there will be a large set of inactive keys.
>>
>> Does this have any side effects? If so what has to be done to overcome
>> humongous keyed streams?
>>
>> --
>> A.Narasimha Swamy
>>
>

-- 
A.Narasimha Swamy

Re: How does Flink handle shorted lived keyed streams

[Xintong Song <to...@gmail.com>]

I believe what you are looking for is the State TTL [1][2].


Thank you~

Xintong Song


[1]
https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/state.html#state-time-to-live-ttl

[2]
https://ci.apache.org/projects/flink/flink-docs-stabledev/table/config.html#table-exec-state-ttl



On Wed, Dec 23, 2020 at 11:57 PM narasimha <sw...@gmail.com> wrote:

> Hi,
>
> Belos is the use case.
>
> Have a stream of transaction events, success/failure of a transaction can
> be determined by those events.
> Partitioning stream by transaction id and applying CEP to determine the
> success/failure of a transaction.
> Each transaction keyed stream is valid only until the final status is
> found. Which can end up having large inactive keyed streams in the system.
>
> Know that using keygroup flink distributes the keyedstream to tasks based
> on it, but still there will be a large set of inactive keys.
>
> Does this have any side effects? If so what has to be done to overcome
> humongous keyed streams?
>
> --
> A.Narasimha Swamy
>