You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Phil Franklin <ph...@gmail.com> on 2018/11/26 21:57:22 UTC
Beam Metrics questions
All of the discussion I’ve seen says that Flink and Spark only provided attempted metric values, but when I use MetricsHttpSink and look at the JSON it has both attempted and committed values (albeit, both the same for my simple testing). Has the metrics processing been updated recently, and I’m just missing the change updates? Or are the committed values being defaulted to the attempted values?
Also, I’ve seen it mentioned that Flink doesn’t report metrics when in detached mode. Is this still the case?
Thanks for your help!
Re: Beam Metrics questions
Posted by Etienne Chauchot <ec...@apache.org>.
Hi Phil,
Thanks for using MetricsPusher and Beam in general !
- MetricsHttpSink works that way: it filters out committed metrics from the json output when committed metrics are not
supported. I checked, Flink runner still does not support committed metrics. So there should be no committed metrics
values in the output json.There might be a bug. I'll open a ticket: thx for pointing out ! You tested on flink and spark
right? and both output committed metrics values right?
- there is no default mechanism to fallback committed metrics values on attempted ones
- Apache Flink does no make flink Accumulators available in detached mode, so indeed, metrics are not available in this
mode.
CCing dev list.
Etienne
Le lundi 26 novembre 2018 à 15:57 -0600, Phil Franklin a écrit :
> All of the discussion I’ve seen says that Flink and Spark only provided attempted metric values, but when I use
> MetricsHttpSink and look at the JSON it has both attempted and committed values (albeit, both the same for my simple
> testing). Has the metrics processing been updated recently, and I’m just missing the change updates? Or are the
> committed values being defaulted to the attempted values?
>
> Also, I’ve seen it mentioned that Flink doesn’t report metrics when in detached mode. Is this still the case?
>
>
> Thanks for your help!
Re: Beam Metrics questions
Posted by Etienne Chauchot <ec...@apache.org>.
Hi Phil,
Thanks for using MetricsPusher and Beam in general !
- MetricsHttpSink works that way: it filters out committed metrics from the json output when committed metrics are not
supported. I checked, Flink runner still does not support committed metrics. So there should be no committed metrics
values in the output json.There might be a bug. I'll open a ticket: thx for pointing out ! You tested on flink and spark
right? and both output committed metrics values right?
- there is no default mechanism to fallback committed metrics values on attempted ones
- Apache Flink does no make flink Accumulators available in detached mode, so indeed, metrics are not available in this
mode.
CCing dev list.
Etienne
Le lundi 26 novembre 2018 à 15:57 -0600, Phil Franklin a écrit :
> All of the discussion I’ve seen says that Flink and Spark only provided attempted metric values, but when I use
> MetricsHttpSink and look at the JSON it has both attempted and committed values (albeit, both the same for my simple
> testing). Has the metrics processing been updated recently, and I’m just missing the change updates? Or are the
> committed values being defaulted to the attempted values?
>
> Also, I’ve seen it mentioned that Flink doesn’t report metrics when in detached mode. Is this still the case?
>
>
> Thanks for your help!