You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by sohimankotia <so...@gmail.com> on 2017/06/20 10:12:14 UTC
Custom Gauge Metric is not getting updated on Job Dashboard
I was testing flink Gauge metric in my flat map function . I was just sending
a value (end time -start time ) of a DB Op in Flatmap . But on dashboard is
only showing first value instead of updated value .
I am using following code in my flat map .
getRuntimeContext()
.getMetricGroup()
.gauge("DBOpInSec", (Gauge<String>) () -> String.valueOf(
(endTime-startTime)/1000 ));
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Custom-Gauge-Metric-is-not-getting-updated-on-Job-Dashboard-tp13842.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: Custom Gauge Metric is not getting updated on Job Dashboard
Posted by sohimankotia <so...@gmail.com>.
Cool. Thanks Closing thread .
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Custom-Gauge-Metric-is-not-getting-updated-on-Job-Dashboard-tp13842p13895.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: Custom Gauge Metric is not getting updated on Job Dashboard
Posted by Chesnay Schepler <ch...@apache.org>.
Exactly, you register the gauge once in open(), and modify the code so
that this gauge returns
different values.
On 21.06.2017 12:04, sohimankotia wrote:
> Basically Every time I am calling add metric method it is just registering
> the gauge .
>
> I can register this gauge in open method and then in flatmap update the
> value of gauge .
>
> Right ?
>
>
>
> --
> View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Custom-Gauge-Metric-is-not-getting-updated-on-Job-Dashboard-tp13842p13891.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
>
Re: Custom Gauge Metric is not getting updated on Job Dashboard
Posted by sohimankotia <so...@gmail.com>.
Basically Every time I am calling add metric method it is just registering
the gauge .
I can register this gauge in open method and then in flatmap update the
value of gauge .
Right ?
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Custom-Gauge-Metric-is-not-getting-updated-on-Job-Dashboard-tp13842p13891.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: Custom Gauge Metric is not getting updated on Job Dashboard
Posted by Chesnay Schepler <ch...@apache.org>.
The reason why the gauge value is not updating is because you are not
actually updating the gauge,
but register a new gauge under the same name. The subsequent
registration are ignored, and should've
logged a warning.
I suggest to make your gauge stateful by adding a field for the
opTimeInSec with a setter which you call
in addMetricData(...).
On 21.06.2017 11:48, sohimankotia wrote:
> Here it is :
>
>
> import com.codahale.metrics.SlidingWindowReservoir;
> import in.dailyhunt.cis.enrichments.datatype.BasicInfoTuple;
> import in.dailyhunt.cis.enrichments.datatype.SinkTuple;
> import org.apache.flink.api.common.accumulators.LongCounter;
> import org.apache.flink.api.common.functions.RichFlatMapFunction;
> import org.apache.flink.api.common.functions.RuntimeContext;
> import org.apache.flink.configuration.Configuration;
> import org.apache.flink.dropwizard.metrics.DropwizardHistogramWrapper;
> import org.apache.flink.dropwizard.metrics.DropwizardMeterWrapper;
> import org.apache.flink.metrics.Gauge;
> import org.apache.flink.metrics.Meter;
> import org.apache.flink.util.Collector;
>
>
> public class DBFlatMap extends RichFlatMapFunction<BasicInfoTuple,
> SinkTuple> {
>
> private transient Meter meter;
> private transient org.apache.flink.metrics.Histogram histogram;
>
>
> @Override
> public void open(Configuration parameters) throws Exception {
>
> /*
> some app specific code
> */
>
> com.codahale.metrics.Meter meter1 = new com.codahale.metrics.Meter();
> this.meter = getRuntimeContext()
> .getMetricGroup()
> .meter("myMeter", new DropwizardMeterWrapper(meter1));
>
> com.codahale.metrics.Histogram histogram1 =
> new com.codahale.metrics.Histogram(new SlidingWindowReservoir(500));
>
> this.histogram = getRuntimeContext()
> .getMetricGroup()
> .histogram("myHistogram", new DropwizardHistogramWrapper(histogram1));
>
> }
>
> @Override
> public void flatMap(BasicInfoTuple in, Collector<SinkTuple> out) throws
> Exception {
> long start = System.currentTimeMillis();
> incrementCounter("input-in-group-and-KeyElect-Flow",
> this.getRuntimeContext());
> this.dbOperations();
> addMetricData(start, System.currentTimeMillis());
> this.meter.markEvent();
> }
>
> private void dbOperations() {
> // Db Operation and some app logic
> }
>
> public void incrementCounter(String counterName, RuntimeContext
> runtimeContext) {
> if (runtimeContext == null) {
> return;
> }
>
> LongCounter lc = runtimeContext.getLongCounter(counterName);
> if (lc == null) {
> lc = new LongCounter();
> runtimeContext.addAccumulator(counterName, lc);
> }
> lc.add(1L);
> }
>
>
> private void addMetricData(long startTime, long endTime) {
> final long opTimeInSec = (endTime - startTime) / 1000;
> this.histogram.update(opTimeInSec);
> getRuntimeContext().getMetricGroup()
> .gauge("DbOpGauge", (Gauge<String>) () -> String.valueOf(opTimeInSec));
>
> }
>
>
> }
>
>
>
>
> --
> View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Custom-Gauge-Metric-is-not-getting-updated-on-Job-Dashboard-tp13842p13888.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
>
Re: Custom Gauge Metric is not getting updated on Job Dashboard
Posted by sohimankotia <so...@gmail.com>.
Here it is :
import com.codahale.metrics.SlidingWindowReservoir;
import in.dailyhunt.cis.enrichments.datatype.BasicInfoTuple;
import in.dailyhunt.cis.enrichments.datatype.SinkTuple;
import org.apache.flink.api.common.accumulators.LongCounter;
import org.apache.flink.api.common.functions.RichFlatMapFunction;
import org.apache.flink.api.common.functions.RuntimeContext;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.dropwizard.metrics.DropwizardHistogramWrapper;
import org.apache.flink.dropwizard.metrics.DropwizardMeterWrapper;
import org.apache.flink.metrics.Gauge;
import org.apache.flink.metrics.Meter;
import org.apache.flink.util.Collector;
public class DBFlatMap extends RichFlatMapFunction<BasicInfoTuple,
SinkTuple> {
private transient Meter meter;
private transient org.apache.flink.metrics.Histogram histogram;
@Override
public void open(Configuration parameters) throws Exception {
/*
some app specific code
*/
com.codahale.metrics.Meter meter1 = new com.codahale.metrics.Meter();
this.meter = getRuntimeContext()
.getMetricGroup()
.meter("myMeter", new DropwizardMeterWrapper(meter1));
com.codahale.metrics.Histogram histogram1 =
new com.codahale.metrics.Histogram(new SlidingWindowReservoir(500));
this.histogram = getRuntimeContext()
.getMetricGroup()
.histogram("myHistogram", new DropwizardHistogramWrapper(histogram1));
}
@Override
public void flatMap(BasicInfoTuple in, Collector<SinkTuple> out) throws
Exception {
long start = System.currentTimeMillis();
incrementCounter("input-in-group-and-KeyElect-Flow",
this.getRuntimeContext());
this.dbOperations();
addMetricData(start, System.currentTimeMillis());
this.meter.markEvent();
}
private void dbOperations() {
// Db Operation and some app logic
}
public void incrementCounter(String counterName, RuntimeContext
runtimeContext) {
if (runtimeContext == null) {
return;
}
LongCounter lc = runtimeContext.getLongCounter(counterName);
if (lc == null) {
lc = new LongCounter();
runtimeContext.addAccumulator(counterName, lc);
}
lc.add(1L);
}
private void addMetricData(long startTime, long endTime) {
final long opTimeInSec = (endTime - startTime) / 1000;
this.histogram.update(opTimeInSec);
getRuntimeContext().getMetricGroup()
.gauge("DbOpGauge", (Gauge<String>) () -> String.valueOf(opTimeInSec));
}
}
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Custom-Gauge-Metric-is-not-getting-updated-on-Job-Dashboard-tp13842p13888.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: Custom Gauge Metric is not getting updated on Job Dashboard
Posted by Chesnay Schepler <ch...@apache.org>.
Can you provide more of your code (you can also send it to me directly)?
I'm interested in where the startTime/endTime arguments are defined.
On 21.06.2017 10:47, sohimankotia wrote:
> I ran job and monitored for approx 20 mins .
>
> I tried with meter,accumulators,histogram,gauge .
>
> Out of those only meter and accumulators were updating values, other were
> only only showing constant value all the time .
>
>
>
>
>
> --
> View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Custom-Gauge-Metric-is-not-getting-updated-on-Job-Dashboard-tp13842p13886.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
>
Re: Custom Gauge Metric is not getting updated on Job Dashboard
Posted by sohimankotia <so...@gmail.com>.
I ran job and monitored for approx 20 mins .
I tried with meter,accumulators,histogram,gauge .
Out of those only meter and accumulators were updating values, other were
only only showing constant value all the time .
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Custom-Gauge-Metric-is-not-getting-updated-on-Job-Dashboard-tp13842p13886.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: Custom Gauge Metric is not getting updated on Job Dashboard
Posted by Chesnay Schepler <ch...@apache.org>.
how long does the job run? The metrics are only updated every 10
seconds; and are not updated when the job finishes.
On 20.06.2017 12:12, sohimankotia wrote:
> I was testing flink Gauge metric in my flat map function . I was just sending
> a value (end time -start time ) of a DB Op in Flatmap . But on dashboard is
> only showing first value instead of updated value .
>
> I am using following code in my flat map .
>
> getRuntimeContext()
> .getMetricGroup()
> .gauge("DBOpInSec", (Gauge<String>) () -> String.valueOf(
> (endTime-startTime)/1000 ));
>
>
>
>
>
> --
> View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Custom-Gauge-Metric-is-not-getting-updated-on-Job-Dashboard-tp13842.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
>