Way to get accumulators values *during* job execution ?

["LINZ, Arnaud" <AL...@bouyguestelecom.fr>]

Hello,

I use Grafana/Graphite to monitor my applications. The Flink GUI is really nice, but it disappears after the job completes and consequently is not suitable to long-term monitoring.

For batch applications, I simply send the accumulator’s values at the end of the job to my Graphite base.
For streaming applications, it’s more complex as the job never ends. It would be nice to have a way of getting current accumulator values (like in the GUI)  to push it periodically to Graphite in a monitoring thread. Is there any API to get the values during execution ?

Best regards,
Arnaud

________________________________

L'intégrité de ce message n'étant pas assurée sur internet, la société expéditrice ne peut être tenue responsable de son contenu ni de ses pièces jointes. Toute utilisation ou diffusion non autorisée est interdite. Si vous n'êtes pas destinataire de ce message, merci de le détruire et d'avertir l'expéditeur.

The integrity of this message cannot be guaranteed on the Internet. The company that sent this message cannot therefore be held liable for its content nor attachments. Any unauthorized use or dissemination is prohibited. If you are not the intended recipient of this message, then please delete it and notify the sender.

Re: Way to get accumulators values *during* job execution ?

[Stephan Ewen <se...@apache.org>]

Hi Arnaud!

One thing you can do is to periodically retrieve them by querying the
monitoring API:
https://ci.apache.org/projects/flink/flink-docs-release-0.10/internals/monitoring_rest_api.html

A nice approach would be to let the JobManager eagerly publish the
metrics.  I think that Christian Kreizfeld was working on an extension that
periodically publishes the accumulators to some metrics store. Maybe he can
comment on that.

Greetings,
Stephan


On Wed, Dec 2, 2015 at 5:21 PM, LINZ, Arnaud <AL...@bouyguestelecom.fr>
wrote:

> Hello,
>
>
>
> I use Grafana/Graphite to monitor my applications. The Flink GUI is really
> nice, but it disappears after the job completes and consequently is not
> suitable to long-term monitoring.
>
>
>
> For batch applications, I simply send the accumulator’s values at the end
> of the job to my Graphite base.
>
> For streaming applications, it’s more complex as the job never ends. It
> would be nice to have a way of getting current accumulator values (like in
> the GUI)  to push it periodically to Graphite in a monitoring thread. Is
> there any API to get the values during execution ?
>
>
>
> Best regards,
>
> Arnaud
>
> ------------------------------
>
> L'intégrité de ce message n'étant pas assurée sur internet, la société
> expéditrice ne peut être tenue responsable de son contenu ni de ses pièces
> jointes. Toute utilisation ou diffusion non autorisée est interdite. Si
> vous n'êtes pas destinataire de ce message, merci de le détruire et
> d'avertir l'expéditeur.
>
> The integrity of this message cannot be guaranteed on the Internet. The
> company that sent this message cannot therefore be held liable for its
> content nor attachments. Any unauthorized use or dissemination is
> prohibited. If you are not the intended recipient of this message, then
> please delete it and notify the sender.
>