You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@accumulo.apache.org by Arvind Shyamsundar <ar...@microsoft.com> on 2019/08/27 00:38:44 UTC
Uniquely identifying metrics with multiple tservers per node.
Hello!
We are testing Apache Accumulo 1.9.3 on Microsoft Azure. For centrally gathering metrics we are sending Accumulo and Hadoop metrics to collectd's statsd plugin (which acts as a statsd sink), which in turn forwards those metrics to an Azure monitoring service. The hadoop-metrics2-accumulo.properties looks like this:
*.sink.statsd.class=org.apache.hadoop.metrics2.sink.StatsDSink
accumulo.sink.statsd.server.host=127.0.0.1
accumulo.sink.statsd.server.port=8125
accumulo.sink.statsd.skip.hostname=false # we have tried setting this to true and also to false, but that still does not include the service instance.
accumulo.sink.statsd.service.name=master
Everything works great when we have the default of 1 tserver per node. However, when running 2 tservers per node (Accumulo 1.9.3), we find that the metrics being reported (TabletServer.tserver.general.ingestRate for example) are not qualified / distinguished with the service instance ID (1_ or 2_) in the name. It appears as though only the values for one instance are being sent. As a result, we are unable to accurately measure aggregate metrics (as an example, the value of ingestRate recorded through this route is ~ 50% of what is shown on the Monitor UI).
If someone has an example of how to configure hadoop-metrics2-accumulo.properties and multiple tservers per node in a way that the counter names would be distinguished by their service instance ID, that would be really helpful.
Thank you!
- Arvind.
RE: Uniquely identifying metrics with multiple tservers per node.
Posted by Arvind Shyamsundar <ar...@microsoft.com>.
Thank you very much, Christopher! That worked (details in GitHub issue).
p.s. n0w 1 kn0w wh47 l337 15! 😊
-----Original Message-----
From: Christopher <ct...@apache.org>
Sent: Monday, August 26, 2019 8:27 PM
To: accumulo-user <us...@accumulo.apache.org>
Subject: Re: Uniquely identifying metrics with multiple tservers per node.
I've responded to your "Leet" issue
(https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Faccumulo%2Fissues%2F1337&data=02%7C01%7Carvindsh%40microsoft.com%7Cbf9076c7a0394ba5d1a408d72a9e7429%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637024732350688416&sdata=zO47sAGUM%2FGOL1e%2FqXR8AYN%2BP8wEzVTzByhr3LXIzwE%3D&reserved=0)
;)
On Mon, Aug 26, 2019 at 9:35 PM <dl...@comcast.net> wrote:
>
> IIRC the ‘service.name’ property in the metrics properties file is used to note from which service the metric was emitted. I would suggest testing this by setting the service.name property to tserver1 and see if your metrics show up with that name. If they do, then you might be able to modify the start scripts to have the different tserver instances use different metrics properties files.
>
>
>
> From: Arvind Shyamsundar <ar...@microsoft.com>
> Sent: Monday, August 26, 2019 8:39 PM
> To: user@accumulo.apache.org
> Subject: Uniquely identifying metrics with multiple tservers per node.
>
>
>
> Hello!
>
> We are testing Apache Accumulo 1.9.3 on Microsoft Azure. For centrally gathering metrics we are sending Accumulo and Hadoop metrics to collectd’s statsd plugin (which acts as a statsd sink), which in turn forwards those metrics to an Azure monitoring service. The hadoop-metrics2-accumulo.properties looks like this:
>
>
>
> *.sink.statsd.class=org.apache.hadoop.metrics2.sink.StatsDSink
>
> accumulo.sink.statsd.server.host=127.0.0.1
>
> accumulo.sink.statsd.server.port=8125
>
> accumulo.sink.statsd.skip.hostname=false # we have tried setting this to true and also to false, but that still does not include the service instance.
>
> accumulo.sink.statsd.service.name=master
>
>
>
> Everything works great when we have the default of 1 tserver per node. However, when running 2 tservers per node (Accumulo 1.9.3), we find that the metrics being reported (TabletServer.tserver.general.ingestRate for example) are not qualified / distinguished with the service instance ID (1_ or 2_) in the name. It appears as though only the values for one instance are being sent. As a result, we are unable to accurately measure aggregate metrics (as an example, the value of ingestRate recorded through this route is ~ 50% of what is shown on the Monitor UI).
>
>
>
> If someone has an example of how to configure hadoop-metrics2-accumulo.properties and multiple tservers per node in a way that the counter names would be distinguished by their service instance ID, that would be really helpful.
>
>
>
> Thank you!
>
>
>
> - Arvind.
Re: Uniquely identifying metrics with multiple tservers per node.
Posted by Christopher <ct...@apache.org>.
I've responded to your "Leet" issue
(https://github.com/apache/accumulo/issues/1337)
;)
On Mon, Aug 26, 2019 at 9:35 PM <dl...@comcast.net> wrote:
>
> IIRC the ‘service.name’ property in the metrics properties file is used to note from which service the metric was emitted. I would suggest testing this by setting the service.name property to tserver1 and see if your metrics show up with that name. If they do, then you might be able to modify the start scripts to have the different tserver instances use different metrics properties files.
>
>
>
> From: Arvind Shyamsundar <ar...@microsoft.com>
> Sent: Monday, August 26, 2019 8:39 PM
> To: user@accumulo.apache.org
> Subject: Uniquely identifying metrics with multiple tservers per node.
>
>
>
> Hello!
>
> We are testing Apache Accumulo 1.9.3 on Microsoft Azure. For centrally gathering metrics we are sending Accumulo and Hadoop metrics to collectd’s statsd plugin (which acts as a statsd sink), which in turn forwards those metrics to an Azure monitoring service. The hadoop-metrics2-accumulo.properties looks like this:
>
>
>
> *.sink.statsd.class=org.apache.hadoop.metrics2.sink.StatsDSink
>
> accumulo.sink.statsd.server.host=127.0.0.1
>
> accumulo.sink.statsd.server.port=8125
>
> accumulo.sink.statsd.skip.hostname=false # we have tried setting this to true and also to false, but that still does not include the service instance.
>
> accumulo.sink.statsd.service.name=master
>
>
>
> Everything works great when we have the default of 1 tserver per node. However, when running 2 tservers per node (Accumulo 1.9.3), we find that the metrics being reported (TabletServer.tserver.general.ingestRate for example) are not qualified / distinguished with the service instance ID (1_ or 2_) in the name. It appears as though only the values for one instance are being sent. As a result, we are unable to accurately measure aggregate metrics (as an example, the value of ingestRate recorded through this route is ~ 50% of what is shown on the Monitor UI).
>
>
>
> If someone has an example of how to configure hadoop-metrics2-accumulo.properties and multiple tservers per node in a way that the counter names would be distinguished by their service instance ID, that would be really helpful.
>
>
>
> Thank you!
>
>
>
> - Arvind.
RE: Uniquely identifying metrics with multiple tservers per node.
Posted by dl...@comcast.net.
IIRC the 'service.name' property in the metrics properties file is used to
note from which service the metric was emitted. I would suggest testing this
by setting the service.name property to tserver1 and see if your metrics
show up with that name. If they do, then you might be able to modify the
start scripts to have the different tserver instances use different metrics
properties files.
From: Arvind Shyamsundar <ar...@microsoft.com>
Sent: Monday, August 26, 2019 8:39 PM
To: user@accumulo.apache.org
Subject: Uniquely identifying metrics with multiple tservers per node.
Hello!
We are testing Apache Accumulo 1.9.3 on Microsoft Azure. For centrally
gathering metrics we are sending Accumulo and Hadoop metrics to collectd's
statsd plugin (which acts as a statsd sink), which in turn forwards those
metrics to an Azure monitoring service. The
hadoop-metrics2-accumulo.properties looks like this:
*.sink.statsd.class=org.apache.hadoop.metrics2.sink.StatsDSink
accumulo.sink.statsd.server.host=127.0.0.1
accumulo.sink.statsd.server.port=8125
accumulo.sink.statsd.skip.hostname=false # we have tried setting
this to true and also to false, but that still does not include the service
instance.
accumulo.sink.statsd.service.name=master
Everything works great when we have the default of 1 tserver per node.
However, when running 2 tservers per node (Accumulo 1.9.3), we find that the
metrics being reported (TabletServer.tserver.general.ingestRate for example)
are not qualified / distinguished with the service instance ID (1_ or 2_) in
the name. It appears as though only the values for one instance are being
sent. As a result, we are unable to accurately measure aggregate metrics (as
an example, the value of ingestRate recorded through this route is ~ 50% of
what is shown on the Monitor UI).
If someone has an example of how to configure
hadoop-metrics2-accumulo.properties and multiple tservers per node in a way
that the counter names would be distinguished by their service instance ID,
that would be really helpful.
Thank you!
- Arvind.