You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Sutanu Das <sd...@att.com> on 2015/08/05 23:39:19 UTC
How can i get flume performance metrix? Jmx
Hi Team,
Can you please advise how can I see the Flume jmx performance metrix in our production?
Our management is interested to know Flume performance usage as far as cpu, mem, gc, events, etc.....
We use Hortonworks Flume 1.5.2.2
Re: How can i get flume performance metrix? Jmx
Posted by Roshan Naik <ro...@hortonworks.com>.
If you are using Hortonworks distro and installed Flume via Ambari .. Then you should be able to monitor much of that from Ambari.
If you used command line installation, then you can do what Iain suggested below.
-roshan
From: iain wright <ia...@gmail.com>>
Reply-To: "user@flume.apache.org<ma...@flume.apache.org>" <us...@flume.apache.org>>
Date: Wednesday, August 5, 2015 3:42 PM
To: "user@flume.apache.org<ma...@flume.apache.org>" <us...@flume.apache.org>>
Subject: Re: How can i get flume performance metrix? Jmx
For CPU/mem/GC/etc. you should be able to add this to JAVA_OPTS in flume-env.sh (im not 100% sure hortonworks has this, or that it hasn't been abstracted/pulled into an init script or something)
-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=54321 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false
IE:
JAVA_OPTS="-Xms1024m -Xmx2048m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=54321 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"
For tracking events through the pipeline, im not sure if those are in JMX, i don't believe they are, we use the /metrics endpoint flume serves up:
[root@ip-172-16-1-119 ec2-user]# curl -s http://localhost:16742/metrics | python -mjson.tool
{
"CHANNEL.channel1": {
"ChannelCapacity": "100000000",
"ChannelFillPercentage": "0.0",
"ChannelSize": "0",
"EventPutAttemptCount": "42355657",
"EventPutSuccessCount": "42355657",
"EventTakeAttemptCount": "42391346",
"EventTakeSuccessCount": "42355657",
"StartTime": "1438627774367",
"StopTime": "0",
"Type": "CHANNEL"
},
"SINK.sink1": {
"BatchCompleteCount": "423556",
"BatchEmptyCount": "35688",
"BatchUnderflowCount": "1",
"ConnectionClosedCount": "1634",
"ConnectionCreatedCount": "1639",
"ConnectionFailedCount": "20",
"EventDrainAttemptCount": "42355657",
"EventDrainSuccessCount": "42355657",
"StartTime": "1438627774370",
"StopTime": "0",
"Type": "SINK"
},
"SOURCE.source1": {
"AppendAcceptedCount": "0",
"AppendBatchAcceptedCount": "423562",
"AppendBatchReceivedCount": "423562",
"AppendReceivedCount": "0",
"EventAcceptedCount": "42355657",
"EventReceivedCount": "42355657",
"OpenConnectionCount": "0",
"StartTime": "1438627774387",
"StopTime": "0",
"Type": "SOURCE"
}
}
You can poll this, awk/grep/parse/etc. & store values of channelsize or take deltas of event drained success to extrapolate an average throughput
Note that flume would need to be started with these properties for /metrics to work:
-Dflume.monitoring.type=HTTP -Dflume.monitoring.port=16742
Hope that helps
--
Iain Wright
This email message is confidential, intended only for the recipient(s) named above and may contain information that is privileged, exempt from disclosure under applicable law. If you are not the intended recipient, do not disclose or disseminate the message to anyone except the intended recipient. If you have received this message in error, or are not the named recipient(s), please immediately notify the sender by return email, and delete all copies of this message.
On Wed, Aug 5, 2015 at 2:39 PM, Sutanu Das <sd...@att.com>> wrote:
Hi Team,
Can you please advise how can I see the Flume jmx performance metrix in our production?
Our management is interested to know Flume performance usage as far as cpu, mem, gc, events, etc.....
We use Hortonworks Flume 1.5.2.2
Re: How can i get flume performance metrix? Jmx
Posted by iain wright <ia...@gmail.com>.
For CPU/mem/GC/etc. you should be able to add this to JAVA_OPTS in
flume-env.sh (im not 100% sure hortonworks has this, or that it hasn't been
abstracted/pulled into an init script or something)
-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=54321
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
IE:
JAVA_OPTS="-Xms1024m -Xmx2048m -Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=54321
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false"
For tracking events through the pipeline, im not sure if those are in JMX,
i don't believe they are, we use the /metrics endpoint flume serves up:
[root@ip-172-16-1-119 ec2-user]# curl -s http://localhost:16742/metrics |
python -mjson.tool
{
"CHANNEL.channel1": {
"ChannelCapacity": "100000000",
"ChannelFillPercentage": "0.0",
"ChannelSize": "0",
"EventPutAttemptCount": "42355657",
"EventPutSuccessCount": "42355657",
"EventTakeAttemptCount": "42391346",
"EventTakeSuccessCount": "42355657",
"StartTime": "1438627774367",
"StopTime": "0",
"Type": "CHANNEL"
},
"SINK.sink1": {
"BatchCompleteCount": "423556",
"BatchEmptyCount": "35688",
"BatchUnderflowCount": "1",
"ConnectionClosedCount": "1634",
"ConnectionCreatedCount": "1639",
"ConnectionFailedCount": "20",
"EventDrainAttemptCount": "42355657",
"EventDrainSuccessCount": "42355657",
"StartTime": "1438627774370",
"StopTime": "0",
"Type": "SINK"
},
"SOURCE.source1": {
"AppendAcceptedCount": "0",
"AppendBatchAcceptedCount": "423562",
"AppendBatchReceivedCount": "423562",
"AppendReceivedCount": "0",
"EventAcceptedCount": "42355657",
"EventReceivedCount": "42355657",
"OpenConnectionCount": "0",
"StartTime": "1438627774387",
"StopTime": "0",
"Type": "SOURCE"
}
}
You can poll this, awk/grep/parse/etc. & store values of channelsize or
take deltas of event drained success to extrapolate an average throughput
Note that flume would need to be started with these properties for /metrics
to work:
-Dflume.monitoring.type=HTTP -Dflume.monitoring.port=16742
Hope that helps
--
Iain Wright
This email message is confidential, intended only for the recipient(s)
named above and may contain information that is privileged, exempt from
disclosure under applicable law. If you are not the intended recipient, do
not disclose or disseminate the message to anyone except the intended
recipient. If you have received this message in error, or are not the named
recipient(s), please immediately notify the sender by return email, and
delete all copies of this message.
On Wed, Aug 5, 2015 at 2:39 PM, Sutanu Das <sd...@att.com> wrote:
> Hi Team,
>
>
>
> Can you please advise how can I see the Flume jmx performance metrix in
> our production?
>
>
>
> Our management is interested to know Flume performance usage as far as
> cpu, mem, gc, events, etc…..
>
>
>
> We use Hortonworks Flume 1.5.2.2
>
RE: How can i get flume performance metrix? Jmx
Posted by Robert B Hamilton <ro...@gm.com>.
There are some command line jmx tools available that might be worth looking at.
Also if you need your own instrumentation then it is straightforward to get to the JMX metrics via java, using standard javax.management.* classes. We did something like that to make our own jmx command line tool and found it to be extremely useful, especially when instrumenting performance tests or interfacing with monitoring tools.
From: Sutanu Das [mailto:sd2302@att.com]
Sent: Wednesday, August 05, 2015 4:39 PM
To: user@flume.apache.org
Subject: How can i get flume performance metrix? Jmx
Hi Team,
Can you please advise how can I see the Flume jmx performance metrix in our production?
Our management is interested to know Flume performance usage as far as cpu, mem, gc, events, etc…..
We use Hortonworks Flume 1.5.2.2
Nothing in this message is intended to constitute an electronic signature unless a specific statement to the contrary is included in this message.
Confidentiality Note: This message is intended only for the person or entity to which it is addressed. It may contain confidential and/or privileged material. Any review, transmission, dissemination or other use, or taking of any action in reliance upon this message by persons or entities other than the intended recipient is prohibited and may be unlawful. If you received this message in error, please contact the sender and delete it from your computer.