You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by aitozi <gj...@gmail.com> on 2017/10/22 14:36:54 UTC
ResultPartitionMetrics
Hi,
i see in version 1.3, it add the ResultPartitionMetrics with
issue:https://issues.apache.org/jira/browse/FLINK-5090
but i am doubt what is the difference between totalQueueLen and
inputQueueLength in
https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/metrics.html#network
i read the code where register the both metric in Task.java:
this.metrics.getIOMetricGroup().initializeBufferMetrics(this);
// register detailed network metrics, if configured
if
(taskManagerConfig.getConfiguration().getBoolean(TaskManagerOptions.NETWORK_DETAILED_METRICS))
{
// similar to MetricUtils.instantiateNetworkMetrics() but inside this
IOMetricGroup
MetricGroup networkGroup =
this.metrics.getIOMetricGroup().addGroup("Network");
MetricGroup outputGroup = networkGroup.addGroup("Output");
MetricGroup inputGroup = networkGroup.addGroup("Input");
// output metrics
for (int i = 0; i < producedPartitions.length; i++) {
ResultPartitionMetrics.registerQueueLengthMetrics(
outputGroup.addGroup(i), producedPartitions[i]);
}
for (int i = 0; i < inputGates.length; i++) {
InputGateMetrics.registerQueueLengthMetrics(
inputGroup.addGroup(i), inputGates[i]);
}
}
i think the first :initializeBufferMetrics#InputBuffersGauge will get all
the buffers in AllInputGate of the Task and i think the method in
InputGateMetric
"group.gauge("totalQueueLen",metrics.getTotalQueueLenGauge());" does the
same thing , if i understand wrong , please tell me,.
thanks,
Aitozi
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Re: ResultPartitionMetrics
Posted by Nico Kruber <ni...@data-artisans.com>.
Hi Aitozi,
the difference is the scope: the normal metrics (without
taskmanager.net.detailed-metrics) reflect _all_ buffers of a task while the
detailed statistics are more fine-grained and give you statistics per input (or
output) gate - the "total" there reflects the fact that each gate has multiple
channels and the metrics offered are the sum among all of them.
Nico
On Monday, 23 October 2017 09:31:04 CET Timo Walther wrote:
> Hi Aitozi,
>
> I will loop in people that are more familar with the network stack and
> metrics. Maybe this is a bug?
>
> Regards,
> Timo
>
> Am 10/22/17 um 4:36 PM schrieb aitozi:
> > Hi,
> >
> > i see in version 1.3, it add the ResultPartitionMetrics with
> > issue:https://issues.apache.org/jira/browse/FLINK-5090
> >
> > but i am doubt what is the difference between totalQueueLen and
> > inputQueueLength in
> > https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/met
> > rics.html#network
> >
> > i read the code where register the both metric in Task.java:
> >
> > this.metrics.getIOMetricGroup().initializeBufferMetrics(this);
> >
> > // register detailed network metrics, if configured
> > if
> > (taskManagerConfig.getConfiguration().getBoolean(TaskManagerOptions.NETWOR
> > K_DETAILED_METRICS)) {
> >
> > // similar to MetricUtils.instantiateNetworkMetrics() but inside
this
> >
> > IOMetricGroup
> >
> > MetricGroup networkGroup =
> >
> > this.metrics.getIOMetricGroup().addGroup("Network");
> >
> > MetricGroup outputGroup = networkGroup.addGroup("Output");
> > MetricGroup inputGroup = networkGroup.addGroup("Input");
> >
> > // output metrics
> > for (int i = 0; i < producedPartitions.length; i++) {
> >
> > ResultPartitionMetrics.registerQueueLengthMetrics(
> >
> > outputGroup.addGroup(i), producedPartitions[i]);
> >
> > }
> >
> > for (int i = 0; i < inputGates.length; i++) {
> >
> > InputGateMetrics.registerQueueLengthMetrics(
> >
> > inputGroup.addGroup(i), inputGates[i]);
> >
> > }
> >
> > }
> >
> > i think the first :initializeBufferMetrics#InputBuffersGauge will get all
> > the buffers in AllInputGate of the Task and i think the method in
> > InputGateMetric
> > "group.gauge("totalQueueLen",metrics.getTotalQueueLenGauge());" does the
> > same thing , if i understand wrong , please tell me,.
> >
> > thanks,
> > Aitozi
> >
> >
> >
> > --
> > Sent from:
> > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Re: ResultPartitionMetrics
Posted by aitozi <gj...@gmail.com>.
hi,
i have understood it.
thanks,
aitozi
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Re: ResultPartitionMetrics
Posted by Chesnay Schepler <ch...@apache.org>.
The metrics registered in initializeBufferMetrics aggregate across all
InputGates, whereas the metrics registered in the InputGateMetrics are
separate for each InputGate.
As an example, let's say a task has 2 input gates, with each having 2
input buffers queued:
// IOMetricGroup#initializeBufferMetrics
task.buffers.inputQueueLength= 4
// detailed metrics
task.input.0.totalQueueLen = 2
task.input.1.totalQueueLen = 2
On 23.10.2017 09:31, Timo Walther wrote:
> Hi Aitozi,
>
> I will loop in people that are more familar with the network stack and
> metrics. Maybe this is a bug?
>
> Regards,
> Timo
>
>
>
> Am 10/22/17 um 4:36 PM schrieb aitozi:
>> Hi,
>>
>> i see in version 1.3, it add the ResultPartitionMetrics with
>> issue:https://issues.apache.org/jira/browse/FLINK-5090
>>
>> but i am doubt what is the difference between totalQueueLen and
>> inputQueueLength in
>> https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/metrics.html#network
>>
>>
>> i read the code where register the both metric in Task.java:
>>
>> this.metrics.getIOMetricGroup().initializeBufferMetrics(this);
>>
>> // register detailed network metrics, if configured
>> if
>> (taskManagerConfig.getConfiguration().getBoolean(TaskManagerOptions.NETWORK_DETAILED_METRICS))
>>
>> {
>> // similar to MetricUtils.instantiateNetworkMetrics()
>> but inside this
>> IOMetricGroup
>> MetricGroup networkGroup =
>> this.metrics.getIOMetricGroup().addGroup("Network");
>> MetricGroup outputGroup =
>> networkGroup.addGroup("Output");
>> MetricGroup inputGroup = networkGroup.addGroup("Input");
>>
>> // output metrics
>> for (int i = 0; i < producedPartitions.length; i++) {
>> ResultPartitionMetrics.registerQueueLengthMetrics(
>> outputGroup.addGroup(i), producedPartitions[i]);
>> }
>>
>> for (int i = 0; i < inputGates.length; i++) {
>> InputGateMetrics.registerQueueLengthMetrics(
>> inputGroup.addGroup(i), inputGates[i]);
>> }
>> }
>>
>> i think the first :initializeBufferMetrics#InputBuffersGauge will get
>> all
>> the buffers in AllInputGate of the Task and i think the method in
>> InputGateMetric
>> "group.gauge("totalQueueLen",metrics.getTotalQueueLenGauge());" does the
>> same thing , if i understand wrong , please tell me,.
>>
>> thanks,
>> Aitozi
>>
>>
>>
>> --
>> Sent from:
>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>
>
>
Re: ResultPartitionMetrics
Posted by Timo Walther <tw...@apache.org>.
Hi Aitozi,
I will loop in people that are more familar with the network stack and
metrics. Maybe this is a bug?
Regards,
Timo
Am 10/22/17 um 4:36 PM schrieb aitozi:
> Hi,
>
> i see in version 1.3, it add the ResultPartitionMetrics with
> issue:https://issues.apache.org/jira/browse/FLINK-5090
>
> but i am doubt what is the difference between totalQueueLen and
> inputQueueLength in
> https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/metrics.html#network
>
> i read the code where register the both metric in Task.java:
>
> this.metrics.getIOMetricGroup().initializeBufferMetrics(this);
>
> // register detailed network metrics, if configured
> if
> (taskManagerConfig.getConfiguration().getBoolean(TaskManagerOptions.NETWORK_DETAILED_METRICS))
> {
> // similar to MetricUtils.instantiateNetworkMetrics() but inside this
> IOMetricGroup
> MetricGroup networkGroup =
> this.metrics.getIOMetricGroup().addGroup("Network");
> MetricGroup outputGroup = networkGroup.addGroup("Output");
> MetricGroup inputGroup = networkGroup.addGroup("Input");
>
> // output metrics
> for (int i = 0; i < producedPartitions.length; i++) {
> ResultPartitionMetrics.registerQueueLengthMetrics(
> outputGroup.addGroup(i), producedPartitions[i]);
> }
>
> for (int i = 0; i < inputGates.length; i++) {
> InputGateMetrics.registerQueueLengthMetrics(
> inputGroup.addGroup(i), inputGates[i]);
> }
> }
>
> i think the first :initializeBufferMetrics#InputBuffersGauge will get all
> the buffers in AllInputGate of the Task and i think the method in
> InputGateMetric
> "group.gauge("totalQueueLen",metrics.getTotalQueueLenGauge());" does the
> same thing , if i understand wrong , please tell me,.
>
> thanks,
> Aitozi
>
>
>
> --
> Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/