You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by bastien dine <ba...@gmail.com> on 2018/11/27 16:00:19 UTC
Flink - Metric are not reported
Hello everyone,
Once again I require your help !
I am trying to report custom metric (see my code below)
Yet, I do not see them anywhere.. nor in the metric tab from my tasks, nor
in the rest API, nor in the declared slf4j reporter..
Can someone help me to debug this ..
Here is my RichMap function :
public class MetricGaugeRichMap<T, E extends MetricTuple<T>> extends
RichMapFunction<E, E> {
private transient T valueToExpose;
private final String metricGroup;
private final String metricName;
public MetricGaugeRichMap(String metricGroup, String metricName) {
this.metricGroup = metricGroup;
this.metricName = metricName;
}
@Override
public void open(Configuration config) {
getRuntimeContext()
.getMetricGroup()
.addGroup(metricGroup)
.gauge(metricName, (Gauge<T>) () -> valueToExpose);
}
@Override
public E map(E metricTuple) throws Exception {
valueToExpose = metricTuple.getMetricValue();
return metricTuple;
}
}
calling from topology :
env.fromElements(new MetricTuple<>(metricGroup, metricName, metricValue))
.map(new MetricGaugeRichMap<>(metricGroup, metricName))
.output(new MetricGaugeOutputFormat<>()); // dummy output
------------------
Bastien DINE
Data Architect / Software Engineer / Sysadmin
bastiendine.io
Re: Flink - Metric are not reported
Posted by Chesnay Schepler <ch...@apache.org>.
Yes, you can override notifyOnAddedMetric/notifyOnRemovedMetric to
ensure that a metric is logged at least once.
On 28.11.2018 16:01, bastien dine wrote:
> Yea, that was I was thinking..
> Batch can be quick
> Can I report metric on "added" action ? (i should override the
> notifyOnAddedMetric to report ?)
>
> ------------------
>
> Bastien DINE
> Data Architect / Software Engineer / Sysadmin
> bastiendine.io <http://bastiendine.io>
>
>
> Le mer. 28 nov. 2018 à 15:54, Chesnay Schepler <chesnay@apache.org
> <ma...@apache.org>> a écrit :
>
> How quick does job batch terminate? Metrics are unregistered once
> the job ends; if the job duration is shorter than the report
> interval they may never be exposed.
>
> On 28.11.2018 15:18, bastien dine wrote:
>> Hello Chesnay,
>>
>> Thanks for your response !
>> I have logs enable (info), slf4jReporter is working, I can see :
>>
>> 15:16:00.112 [Flink-MetricRegistry-thread-1] INFO
>> org.apache.flink.metrics.slf4j.Slf4jReporter -
>> =========================== Starting metrics report
>> ===========================
>>
>> -- Counters
>> -------------------------------------------------------------------
>>
>> -- Gauges
>> ---------------------------------------------------------------------
>> ....
>>
>> On both jobmanager & taskmanager,
>> *BUT* i see only system metrics, not my custom one..
>> Am i missing something in declaration in my topology ?
>>
>> _Note_ : I am using DataSet API (so my program is batch, and not
>> continuous)
>>
>> Regards,
>> Bastien
>>
>> ------------------
>>
>> Bastien DINE
>> Data Architect / Software Engineer / Sysadmin
>> bastiendine.io <http://bastiendine.io>
>>
>>
>> Le mar. 27 nov. 2018 à 17:07, Chesnay Schepler
>> <chesnay@apache.org <ma...@apache.org>> a écrit :
>>
>> Please enable WARN logging and check for warnings by the
>> SLF4JReporter and/or MetricQueryService.
>>
>> On 27.11.2018 17:00, bastien dine wrote:
>>> Hello everyone,
>>>
>>> Once again I require your help !
>>> I am trying to report custom metric (see my code below)
>>> Yet, I do not see them anywhere.. nor in the metric tab from
>>> my tasks, nor in the rest API, nor in the declared slf4j
>>> reporter..
>>> Can someone help me to debug this ..
>>>
>>> Here is my RichMap function :
>>> public class MetricGaugeRichMap<T,E extends MetricTuple<T>> extends RichMapFunction<E,E> {
>>>
>>> private transient T valueToExpose;
>>> private final StringmetricGroup;
>>> private final StringmetricName;
>>>
>>>
>>> public MetricGaugeRichMap(String metricGroup, String metricName) {
>>> this.metricGroup = metricGroup;
>>> this.metricName = metricName;
>>> }
>>>
>>> @Override public void open(Configuration config) {
>>> getRuntimeContext()
>>> .getMetricGroup()
>>> .addGroup(metricGroup)
>>> .gauge(metricName, (Gauge<T>) () -> valueToExpose);
>>> }
>>>
>>> @Override public E map(E metricTuple)throws Exception {
>>> valueToExpose = metricTuple.getMetricValue();
>>> return metricTuple;
>>> }
>>> }
>>> calling from topology :
>>> env.fromElements(new MetricTuple<>(metricGroup,metricName,metricValue))
>>> .map(new MetricGaugeRichMap<>(metricGroup,metricName))
>>> .output(new MetricGaugeOutputFormat<>()); // dummy output
>>> ------------------
>>>
>>> Bastien DINE
>>> Data Architect / Software Engineer / Sysadmin
>>> bastiendine.io <http://bastiendine.io>
>>
>>
>
Re: Flink - Metric are not reported
Posted by bastien dine <ba...@gmail.com>.
Yea, that was I was thinking..
Batch can be quick
Can I report metric on "added" action ? (i should override the
notifyOnAddedMetric to report ?)
------------------
Bastien DINE
Data Architect / Software Engineer / Sysadmin
bastiendine.io
Le mer. 28 nov. 2018 à 15:54, Chesnay Schepler <ch...@apache.org> a
écrit :
> How quick does job batch terminate? Metrics are unregistered once the job
> ends; if the job duration is shorter than the report interval they may
> never be exposed.
>
> On 28.11.2018 15:18, bastien dine wrote:
>
> Hello Chesnay,
>
> Thanks for your response !
> I have logs enable (info), slf4jReporter is working, I can see :
>
> 15:16:00.112 [Flink-MetricRegistry-thread-1] INFO
> org.apache.flink.metrics.slf4j.Slf4jReporter -
> =========================== Starting metrics report
> ===========================
>
> -- Counters
> -------------------------------------------------------------------
>
> -- Gauges
> ---------------------------------------------------------------------
> ....
>
> On both jobmanager & taskmanager,
> *BUT* i see only system metrics, not my custom one..
> Am i missing something in declaration in my topology ?
>
> *Note* : I am using DataSet API (so my program is batch, and not
> continuous)
>
> Regards,
> Bastien
>
> ------------------
>
> Bastien DINE
> Data Architect / Software Engineer / Sysadmin
> bastiendine.io
>
>
> Le mar. 27 nov. 2018 à 17:07, Chesnay Schepler <ch...@apache.org> a
> écrit :
>
>> Please enable WARN logging and check for warnings by the SLF4JReporter
>> and/or MetricQueryService.
>>
>> On 27.11.2018 17:00, bastien dine wrote:
>>
>> Hello everyone,
>>
>> Once again I require your help !
>> I am trying to report custom metric (see my code below)
>> Yet, I do not see them anywhere.. nor in the metric tab from my tasks,
>> nor in the rest API, nor in the declared slf4j reporter..
>> Can someone help me to debug this ..
>>
>> Here is my RichMap function :
>>
>> public class MetricGaugeRichMap<T, E extends MetricTuple<T>> extends RichMapFunction<E, E> {
>>
>> private transient T valueToExpose;
>> private final String metricGroup;
>> private final String metricName;
>>
>>
>> public MetricGaugeRichMap(String metricGroup, String metricName) {
>> this.metricGroup = metricGroup;
>> this.metricName = metricName;
>> }
>>
>> @Override public void open(Configuration config) {
>> getRuntimeContext()
>> .getMetricGroup()
>> .addGroup(metricGroup)
>> .gauge(metricName, (Gauge<T>) () -> valueToExpose);
>> }
>>
>> @Override public E map(E metricTuple) throws Exception {
>> valueToExpose = metricTuple.getMetricValue();
>> return metricTuple;
>> }
>> }
>>
>> calling from topology :
>>
>> env.fromElements(new MetricTuple<>(metricGroup, metricName, metricValue))
>> .map(new MetricGaugeRichMap<>(metricGroup, metricName))
>> .output(new MetricGaugeOutputFormat<>()); // dummy output
>>
>> ------------------
>>
>>
>> Bastien DINE
>> Data Architect / Software Engineer / Sysadmin
>> bastiendine.io
>>
>>
>>
>
Re: Flink - Metric are not reported
Posted by Chesnay Schepler <ch...@apache.org>.
How quick does job batch terminate? Metrics are unregistered once the
job ends; if the job duration is shorter than the report interval they
may never be exposed.
On 28.11.2018 15:18, bastien dine wrote:
> Hello Chesnay,
>
> Thanks for your response !
> I have logs enable (info), slf4jReporter is working, I can see :
>
> 15:16:00.112 [Flink-MetricRegistry-thread-1] INFO
> org.apache.flink.metrics.slf4j.Slf4jReporter -
> =========================== Starting metrics report
> ===========================
>
> -- Counters
> -------------------------------------------------------------------
>
> -- Gauges
> ---------------------------------------------------------------------
> ....
>
> On both jobmanager & taskmanager,
> *BUT* i see only system metrics, not my custom one..
> Am i missing something in declaration in my topology ?
>
> _Note_ : I am using DataSet API (so my program is batch, and not
> continuous)
>
> Regards,
> Bastien
>
> ------------------
>
> Bastien DINE
> Data Architect / Software Engineer / Sysadmin
> bastiendine.io <http://bastiendine.io>
>
>
> Le mar. 27 nov. 2018 à 17:07, Chesnay Schepler <chesnay@apache.org
> <ma...@apache.org>> a écrit :
>
> Please enable WARN logging and check for warnings by the
> SLF4JReporter and/or MetricQueryService.
>
> On 27.11.2018 17:00, bastien dine wrote:
>> Hello everyone,
>>
>> Once again I require your help !
>> I am trying to report custom metric (see my code below)
>> Yet, I do not see them anywhere.. nor in the metric tab from my
>> tasks, nor in the rest API, nor in the declared slf4j reporter..
>> Can someone help me to debug this ..
>>
>> Here is my RichMap function :
>> public class MetricGaugeRichMap<T,E extends MetricTuple<T>> extends RichMapFunction<E,E> {
>>
>> private transient T valueToExpose;
>> private final StringmetricGroup;
>> private final StringmetricName;
>>
>>
>> public MetricGaugeRichMap(String metricGroup, String metricName) {
>> this.metricGroup = metricGroup;
>> this.metricName = metricName;
>> }
>>
>> @Override public void open(Configuration config) {
>> getRuntimeContext()
>> .getMetricGroup()
>> .addGroup(metricGroup)
>> .gauge(metricName, (Gauge<T>) () -> valueToExpose);
>> }
>>
>> @Override public E map(E metricTuple)throws Exception {
>> valueToExpose = metricTuple.getMetricValue();
>> return metricTuple;
>> }
>> }
>> calling from topology :
>> env.fromElements(new MetricTuple<>(metricGroup,metricName,metricValue))
>> .map(new MetricGaugeRichMap<>(metricGroup,metricName))
>> .output(new MetricGaugeOutputFormat<>()); // dummy output
>> ------------------
>>
>> Bastien DINE
>> Data Architect / Software Engineer / Sysadmin
>> bastiendine.io <http://bastiendine.io>
>
>
Re: Flink - Metric are not reported
Posted by bastien dine <ba...@gmail.com>.
Hello Chesnay,
Thanks for your response !
I have logs enable (info), slf4jReporter is working, I can see :
15:16:00.112 [Flink-MetricRegistry-thread-1] INFO
org.apache.flink.metrics.slf4j.Slf4jReporter -
=========================== Starting metrics report
===========================
-- Counters
-------------------------------------------------------------------
-- Gauges
---------------------------------------------------------------------
....
On both jobmanager & taskmanager,
*BUT* i see only system metrics, not my custom one..
Am i missing something in declaration in my topology ?
*Note* : I am using DataSet API (so my program is batch, and not continuous)
Regards,
Bastien
------------------
Bastien DINE
Data Architect / Software Engineer / Sysadmin
bastiendine.io
Le mar. 27 nov. 2018 à 17:07, Chesnay Schepler <ch...@apache.org> a
écrit :
> Please enable WARN logging and check for warnings by the SLF4JReporter
> and/or MetricQueryService.
>
> On 27.11.2018 17:00, bastien dine wrote:
>
> Hello everyone,
>
> Once again I require your help !
> I am trying to report custom metric (see my code below)
> Yet, I do not see them anywhere.. nor in the metric tab from my tasks, nor
> in the rest API, nor in the declared slf4j reporter..
> Can someone help me to debug this ..
>
> Here is my RichMap function :
>
> public class MetricGaugeRichMap<T, E extends MetricTuple<T>> extends RichMapFunction<E, E> {
>
> private transient T valueToExpose;
> private final String metricGroup;
> private final String metricName;
>
>
> public MetricGaugeRichMap(String metricGroup, String metricName) {
> this.metricGroup = metricGroup;
> this.metricName = metricName;
> }
>
> @Override public void open(Configuration config) {
> getRuntimeContext()
> .getMetricGroup()
> .addGroup(metricGroup)
> .gauge(metricName, (Gauge<T>) () -> valueToExpose);
> }
>
> @Override public E map(E metricTuple) throws Exception {
> valueToExpose = metricTuple.getMetricValue();
> return metricTuple;
> }
> }
>
> calling from topology :
>
> env.fromElements(new MetricTuple<>(metricGroup, metricName, metricValue))
> .map(new MetricGaugeRichMap<>(metricGroup, metricName))
> .output(new MetricGaugeOutputFormat<>()); // dummy output
>
> ------------------
>
>
> Bastien DINE
> Data Architect / Software Engineer / Sysadmin
> bastiendine.io
>
>
>
Re: Flink - Metric are not reported
Posted by Chesnay Schepler <ch...@apache.org>.
Please enable WARN logging and check for warnings by the SLF4JReporter
and/or MetricQueryService.
On 27.11.2018 17:00, bastien dine wrote:
> Hello everyone,
>
> Once again I require your help !
> I am trying to report custom metric (see my code below)
> Yet, I do not see them anywhere.. nor in the metric tab from my tasks,
> nor in the rest API, nor in the declared slf4j reporter..
> Can someone help me to debug this ..
>
> Here is my RichMap function :
> public class MetricGaugeRichMap<T,E extends MetricTuple<T>> extends RichMapFunction<E,E> {
>
> private transient T valueToExpose;
> private final StringmetricGroup;
> private final StringmetricName;
>
>
> public MetricGaugeRichMap(String metricGroup, String metricName) {
> this.metricGroup = metricGroup;
> this.metricName = metricName;
> }
>
> @Override public void open(Configuration config) {
> getRuntimeContext()
> .getMetricGroup()
> .addGroup(metricGroup)
> .gauge(metricName, (Gauge<T>) () -> valueToExpose);
> }
>
> @Override public E map(E metricTuple)throws Exception {
> valueToExpose = metricTuple.getMetricValue();
> return metricTuple;
> }
> }
> calling from topology :
> env.fromElements(new MetricTuple<>(metricGroup,metricName,metricValue))
> .map(new MetricGaugeRichMap<>(metricGroup,metricName))
> .output(new MetricGaugeOutputFormat<>()); // dummy output
> ------------------
>
> Bastien DINE
> Data Architect / Software Engineer / Sysadmin
> bastiendine.io <http://bastiendine.io>