Re: # of active session windows of a streaming job

[Dongwon Kim <ea...@gmail.com>]

Hi Fabian,

Thanks for the suggestion!

Currently we are just measuring finished sessions for a quick prototype:

I'll try your suggestion and let you know :-)

Thanks,

- Dongwon

On Tue, Feb 20, 2018 at 6:53 PM, Fabian Hueske <fh...@gmail.com> wrote:

> Hi Dongwon Kim,
>
> That's an interesting question.
>
> I don't have a solution blueprint for you, but a few ideas that should
> help to solve the problem.
>
> I would start with a separate job first and later try to integrate it with
> the other job.
> You could implement a Trigger that fires when a new window is created and
> when the window is closed. A ProcessWindowFunction would emit a +1 if the
> window was created and a -1 when the window is closes.
> Session windows are a bit special, because you also need to handle the
> case of merging windows, i.e., two opened windows can be merged and only
> one (the merged) window is closed. So would need to emit a -2 if a merged
> window was closes (assuming only two windows were merged).
> In order to do that, you'd need to carry the merging information forward.
> The Trigger.onMerge method cannot trigger the window function, but it could
> store the merging information in state that is later accessed.
>
> Hope this helps,
> Fabian
>
> 2018-02-20 9:54 GMT+01:00 Dongwon Kim <ea...@gmail.com>:
>
>> Hi,
>>
>> It could be a totally stupid question but I currently have no idea how to
>> get the number of active session windows from a running job.
>>
>> Our traffic trajectory application (which handles up to 10,000 tps) uses
>> event-time session window on KeyedStream (keyed by userID).
>>
>> Should I write another Flink job for the purpose?
>>
>> Cheers,
>>
>> Dongwon Kim
>
>
>