You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Andrew Roberts <ar...@fuze.com> on 2019/02/19 20:14:18 UTC
Metrics for number of "open windows"?
Hello,
I’m trying to track the number of currently-in-state windows in a keyed, windowed stream (stream.keyBy(…).window(…).trigger(…).process(…)) using Flink metrics. Are there any built in? Or any good approaches for collecting this data?
Thanks,
Andrew
--
*Confidentiality Notice: The information contained in this e-mail and any
attachments may be confidential. If you are not an intended recipient, you
are hereby notified that any dissemination, distribution or copying of this
e-mail is strictly prohibited. If you have received this e-mail in error,
please notify the sender and permanently delete the e-mail and any
attachments immediately. You should not retain, copy or use this e-mail or
any attachment for any purpose, nor disclose all or any part of the
contents to any other person. Thank you.*
Re: Metrics for number of "open windows"?
Posted by Konstantin Knauf <ko...@ververica.com>.
Hi Andrew,
when you implement your own Trigger or customize an existing Trigger you
get access to the TriggerContext in all callbacks of the Trigger interface.
The TriggerContext allows you to register custom metrics via
TriggerContext:getMetricGroup() and you can use partitioned state scoped
to key and window of the current element via the
TriggerContext:getPartitionedState(). It should be possible to use those
two ingredients to add a custom metric with the number of open windows.
Cheers,
Konstantin
On Mon, Feb 25, 2019 at 10:58 AM Andrey Zagrebin <an...@ververica.com>
wrote:
> Hi Andrew,
>
> Just to add the Rong's answer, if you use RocksDB state backend, you can
> activate state metrics forwarded from RocksDB [1].
>
> Best,
> Andrey
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#rocksdb
>
> On Thu, Feb 21, 2019 at 11:22 PM Rong Rong <wa...@gmail.com> wrote:
>
>> Hi Andrew,
>>
>> I am assuming you are actually using customized windowAssigner, trigger
>> and process function.
>> I think the best way for you to keep in-flight, not-yet-triggered windows
>> is to emit metrics in these 3 pieces.
>>
>> Upon looking at the window operator, I don't think there's a a metrics
>> (guage) that keeps how many windows are not yet fired.
>> This information is available in the KeyedStateBackend, but I dont think
>> the KeyedStateBackend is emitting any metrics related to what you want.
>>
>> Thanks,
>> Rong
>>
>> On Tue, Feb 19, 2019 at 12:14 PM Andrew Roberts <ar...@fuze.com>
>> wrote:
>>
>>> Hello,
>>>
>>> I’m trying to track the number of currently-in-state windows in a keyed,
>>> windowed stream (stream.keyBy(…).window(…).trigger(…).process(…)) using
>>> Flink metrics. Are there any built in? Or any good approaches for
>>> collecting this data?
>>>
>>> Thanks,
>>>
>>> Andrew
>>> --
>>> *Confidentiality Notice: The information contained in this e-mail and any
>>>
>>> attachments may be confidential. If you are not an intended recipient,
>>> you
>>>
>>> are hereby notified that any dissemination, distribution or copying of
>>> this
>>>
>>> e-mail is strictly prohibited. If you have received this e-mail in error,
>>>
>>> please notify the sender and permanently delete the e-mail and any
>>>
>>> attachments immediately. You should not retain, copy or use this e-mail
>>> or
>>>
>>> any attachment for any purpose, nor disclose all or any part of the
>>>
>>> contents to any other person. Thank you.*
>>>
>>
--
Konstantin Knauf | Solutions Architect
+49 160 91394525
<https://www.ververica.com/>
Follow us @VervericaData
--
Join Flink Forward <https://flink-forward.org/> - The Apache Flink
Conference
Stream Processing | Event Driven | Real Time
--
Data Artisans GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
--
Data Artisans GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen
Re: Metrics for number of "open windows"?
Posted by Andrey Zagrebin <an...@ververica.com>.
Hi Andrew,
Just to add the Rong's answer, if you use RocksDB state backend, you can
activate state metrics forwarded from RocksDB [1].
Best,
Andrey
[1]
https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#rocksdb
On Thu, Feb 21, 2019 at 11:22 PM Rong Rong <wa...@gmail.com> wrote:
> Hi Andrew,
>
> I am assuming you are actually using customized windowAssigner, trigger
> and process function.
> I think the best way for you to keep in-flight, not-yet-triggered windows
> is to emit metrics in these 3 pieces.
>
> Upon looking at the window operator, I don't think there's a a metrics
> (guage) that keeps how many windows are not yet fired.
> This information is available in the KeyedStateBackend, but I dont think
> the KeyedStateBackend is emitting any metrics related to what you want.
>
> Thanks,
> Rong
>
> On Tue, Feb 19, 2019 at 12:14 PM Andrew Roberts <ar...@fuze.com> wrote:
>
>> Hello,
>>
>> I’m trying to track the number of currently-in-state windows in a keyed,
>> windowed stream (stream.keyBy(…).window(…).trigger(…).process(…)) using
>> Flink metrics. Are there any built in? Or any good approaches for
>> collecting this data?
>>
>> Thanks,
>>
>> Andrew
>> --
>> *Confidentiality Notice: The information contained in this e-mail and any
>>
>> attachments may be confidential. If you are not an intended recipient, you
>>
>> are hereby notified that any dissemination, distribution or copying of
>> this
>>
>> e-mail is strictly prohibited. If you have received this e-mail in error,
>>
>> please notify the sender and permanently delete the e-mail and any
>>
>> attachments immediately. You should not retain, copy or use this e-mail or
>>
>> any attachment for any purpose, nor disclose all or any part of the
>>
>> contents to any other person. Thank you.*
>>
>
Re: Metrics for number of "open windows"?
Posted by Rong Rong <wa...@gmail.com>.
Hi Andrew,
I am assuming you are actually using customized windowAssigner, trigger and
process function.
I think the best way for you to keep in-flight, not-yet-triggered windows
is to emit metrics in these 3 pieces.
Upon looking at the window operator, I don't think there's a a metrics
(guage) that keeps how many windows are not yet fired.
This information is available in the KeyedStateBackend, but I dont think
the KeyedStateBackend is emitting any metrics related to what you want.
Thanks,
Rong
On Tue, Feb 19, 2019 at 12:14 PM Andrew Roberts <ar...@fuze.com> wrote:
> Hello,
>
> I’m trying to track the number of currently-in-state windows in a keyed,
> windowed stream (stream.keyBy(…).window(…).trigger(…).process(…)) using
> Flink metrics. Are there any built in? Or any good approaches for
> collecting this data?
>
> Thanks,
>
> Andrew
> --
> *Confidentiality Notice: The information contained in this e-mail and any
>
> attachments may be confidential. If you are not an intended recipient, you
>
> are hereby notified that any dissemination, distribution or copying of this
>
> e-mail is strictly prohibited. If you have received this e-mail in error,
>
> please notify the sender and permanently delete the e-mail and any
>
> attachments immediately. You should not retain, copy or use this e-mail or
>
> any attachment for any purpose, nor disclose all or any part of the
>
> contents to any other person. Thank you.*
>