You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Guillermo Ortiz <ko...@gmail.com> on 2016/01/21 11:54:34 UTC
Monitoring Flume
Hello,
I've looking for solutions to monitor a Flume Agent. Sometimes the process
is running but it isn't available to process any message because some error
happened. So, the process is running but Flume doesn't work.
How are people monitoring Flume in general?
We thought to create a JMX Source and send all data to some sink and do the
monitoring based on the JMX logs though Solr or whatever.
Any other solution or advise about monitoring Flume?
Re: Monitoring Flume
Posted by Ashish <pa...@gmail.com>.
See the section http://flume.apache.org/FlumeUserGuide.html#monitoring
IMHO look for Counters values, if they don't change means things may
be off track. Outside this I would really look for JVM level stats
like GC's. In case of Full GC's nothing from within the application
would respond.
I did some work on https://issues.apache.org/jira/browse/FLUME-2550
(test cases are missing). If you have free cycles can try that out, it
shall open up things you can do with codahale Metrics package. Expect
a bumpy ride with the patch.
On Thu, Jan 21, 2016 at 2:54 AM, Guillermo Ortiz <ko...@gmail.com> wrote:
> Hello,
>
> I've looking for solutions to monitor a Flume Agent. Sometimes the process
> is running but it isn't available to process any message because some error
> happened. So, the process is running but Flume doesn't work.
>
> How are people monitoring Flume in general?
>
> We thought to create a JMX Source and send all data to some sink and do the
> monitoring based on the JMX logs though Solr or whatever.
>
> Any other solution or advise about monitoring Flume?
--
thanks
ashish
Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal
Re: Monitoring Flume
Posted by Dmytro Popovych <dr...@gmail.com>.
We are pretty happy with LogStash for logs + JMX / DataDog for metrics.
I didn’t have a situation when wasn’t able to explain behaviour by those monitors.
> On Jan 21, 2016, at 12:54 PM, Guillermo Ortiz <ko...@gmail.com> wrote:
>
> Hello,
>
> I've looking for solutions to monitor a Flume Agent. Sometimes the process is running but it isn't available to process any message because some error happened. So, the process is running but Flume doesn't work.
>
> How are people monitoring Flume in general?
>
> We thought to create a JMX Source and send all data to some sink and do the monitoring based on the JMX logs though Solr or whatever.
>
> Any other solution or advise about monitoring Flume?
Re: Monitoring Flume
Posted by Simone Roselli <si...@plista.com>.
I mean to develop a Nagios check which is creating a connection obj with JMX and verify a metric to be as expected. eg: if the "kafka unreplicated partition value" is != 0, sys.exit(1), which means release a "WARNING" in the Nagios slang
This about the logic I would adopt
The problem is that I could not find a way to do it using python/ruby, because I didn't find valid examples how to use "jython" or "jruby" for JMX, or other similar things. To be clear, I wanted to use python or ruby just for lazyness; it would be nice in general to find a way to query JMX using something that already works
Simone Roselli
ITE Sysadmin
simone.roselli@plista.com
http://www.plista.com
----- Original Message -----
From: "Guillermo Ortiz" <ko...@gmail.com>
To: "user" <us...@flume.apache.org>
Sent: Thursday, January 21, 2016 2:44:55 PM
Subject: Re: Monitoring Flume
When you talk about ruby or python, you mean reading the log and waiting
some pathers and if they don't happen to launch an "action"?
2016-01-21 12:11 GMT+01:00 Simone Roselli <si...@plista.com>:
> Hi,
>
> in the case of "process up but not working" which can be considered a
> particular case, I would setup a custom script (Bash? Python?) and use
> Nagios/Icinga for the alerting
>
> For the collection of the metrics, I would connect Collectd to JMX, and
> then link Collectd directly on Grafana (http://grafana.org/). The usual
> annoying part is to build the Collectd configuration for the interested
> MBeans; but probably someone has done this already
>
> I also would be interested in a generic good practice to query JMX, since
> I didn't find a human way to do it using Python or Ruby ..
>
>
> Simone Roselli
> ITE Sysadmin
> simone.roselli@plista.com
> http://www.plista.com
>
> ----- Original Message -----
> From: "Guillermo Ortiz" <ko...@gmail.com>
> To: "user" <us...@flume.apache.org>
> Sent: Thursday, January 21, 2016 11:54:34 AM
> Subject: Monitoring Flume
>
> Hello,
>
> I've looking for solutions to monitor a Flume Agent. Sometimes the process
> is running but it isn't available to process any message because some error
> happened. So, the process is running but Flume doesn't work.
>
> How are people monitoring Flume in general?
>
> We thought to create a JMX Source and send all data to some sink and do the
> monitoring based on the JMX logs though Solr or whatever.
>
> Any other solution or advise about monitoring Flume?
>
RE: Monitoring Flume
Posted by Ravi Kiran Aita <ra...@eiqnetworks.com>.
Hi,
We can enable monitoring for Flume Nodes by starting the agent with below configuration
Invoke the agent with setting --> -Dflume.monitoring.port=34545
Access the stats from url --> http://<ip>:34545/metrics<http://%3cip%3e:34545/metrics>
We get the stats in json format which can be further processed. Probably this may help you.
Regards,
Ravikiran A
From: Guillermo Ortiz [mailto:konstt2000@gmail.com]
Sent: Thursday, January 21, 2016 7:15 PM
To: user@flume.apache.org
Subject: Re: Monitoring Flume
When you talk about ruby or python, you mean reading the log and waiting some pathers and if they don't happen to launch an "action"?
2016-01-21 12:11 GMT+01:00 Simone Roselli <si...@plista.com>>:
Hi,
in the case of "process up but not working" which can be considered a particular case, I would setup a custom script (Bash? Python?) and use Nagios/Icinga for the alerting
For the collection of the metrics, I would connect Collectd to JMX, and then link Collectd directly on Grafana (http://grafana.org/). The usual annoying part is to build the Collectd configuration for the interested MBeans; but probably someone has done this already
I also would be interested in a generic good practice to query JMX, since I didn't find a human way to do it using Python or Ruby ..
Simone Roselli
ITE Sysadmin
simone.roselli@plista.com<ma...@plista.com>
http://www.plista.com
----- Original Message -----
From: "Guillermo Ortiz" <ko...@gmail.com>>
To: "user" <us...@flume.apache.org>>
Sent: Thursday, January 21, 2016 11:54:34 AM
Subject: Monitoring Flume
Hello,
I've looking for solutions to monitor a Flume Agent. Sometimes the process
is running but it isn't available to process any message because some error
happened. So, the process is running but Flume doesn't work.
How are people monitoring Flume in general?
We thought to create a JMX Source and send all data to some sink and do the
monitoring based on the JMX logs though Solr or whatever.
Any other solution or advise about monitoring Flume?
Re: Monitoring Flume
Posted by Guillermo Ortiz <ko...@gmail.com>.
When you talk about ruby or python, you mean reading the log and waiting
some pathers and if they don't happen to launch an "action"?
2016-01-21 12:11 GMT+01:00 Simone Roselli <si...@plista.com>:
> Hi,
>
> in the case of "process up but not working" which can be considered a
> particular case, I would setup a custom script (Bash? Python?) and use
> Nagios/Icinga for the alerting
>
> For the collection of the metrics, I would connect Collectd to JMX, and
> then link Collectd directly on Grafana (http://grafana.org/). The usual
> annoying part is to build the Collectd configuration for the interested
> MBeans; but probably someone has done this already
>
> I also would be interested in a generic good practice to query JMX, since
> I didn't find a human way to do it using Python or Ruby ..
>
>
> Simone Roselli
> ITE Sysadmin
> simone.roselli@plista.com
> http://www.plista.com
>
> ----- Original Message -----
> From: "Guillermo Ortiz" <ko...@gmail.com>
> To: "user" <us...@flume.apache.org>
> Sent: Thursday, January 21, 2016 11:54:34 AM
> Subject: Monitoring Flume
>
> Hello,
>
> I've looking for solutions to monitor a Flume Agent. Sometimes the process
> is running but it isn't available to process any message because some error
> happened. So, the process is running but Flume doesn't work.
>
> How are people monitoring Flume in general?
>
> We thought to create a JMX Source and send all data to some sink and do the
> monitoring based on the JMX logs though Solr or whatever.
>
> Any other solution or advise about monitoring Flume?
>
Re: Monitoring Flume
Posted by Simone Roselli <si...@plista.com>.
Hi,
in the case of "process up but not working" which can be considered a particular case, I would setup a custom script (Bash? Python?) and use Nagios/Icinga for the alerting
For the collection of the metrics, I would connect Collectd to JMX, and then link Collectd directly on Grafana (http://grafana.org/). The usual annoying part is to build the Collectd configuration for the interested MBeans; but probably someone has done this already
I also would be interested in a generic good practice to query JMX, since I didn't find a human way to do it using Python or Ruby ..
Simone Roselli
ITE Sysadmin
simone.roselli@plista.com
http://www.plista.com
----- Original Message -----
From: "Guillermo Ortiz" <ko...@gmail.com>
To: "user" <us...@flume.apache.org>
Sent: Thursday, January 21, 2016 11:54:34 AM
Subject: Monitoring Flume
Hello,
I've looking for solutions to monitor a Flume Agent. Sometimes the process
is running but it isn't available to process any message because some error
happened. So, the process is running but Flume doesn't work.
How are people monitoring Flume in general?
We thought to create a JMX Source and send all data to some sink and do the
monitoring based on the JMX logs though Solr or whatever.
Any other solution or advise about monitoring Flume?