You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Mingjie Lai <mj...@gmail.com> on 2011/08/19 01:20:59 UTC
Performance monitoring by Ganglia
Hi.
Has anyone used Ganglia to monitor flume?
I'd like to have a way to monitor performance data (e.g., throughput) of
our customized decorator. I saw there is Ganglia sink in the source
code, but don't think it's for flume itself.
Can anyone point me how to use ganglia to monitor flume metrics?
Thanks,
Mingjie
Re: Performance monitoring by Ganglia
Posted by Torsten Curdt <tc...@vafer.org>.
> Torsten, Is there possibility you guys will share the python script? :)
We have a couple of things I would like to see contributed back.
Will try to get permission. Might take a couple of days or so.
cheers,
Torsten
Re: Performance monitoring by Ganglia
Posted by Shuang <sh...@open42.com>.
Torsten, Is there possibility you guys will share the python script? :)
Shuang
On Fri, Aug 19, 2011 at 12:43 AM, Torsten Curdt <tc...@vafer.org> wrote:
> The current reporting is quite broken and does not support the Ganglia
> suggested multicast setup at all. We've got some code where we fixed
> the source and the sink but still had some trouble getting it working
> reliably (on the Ganglia sink side of things). But it didn't felt
> worth spending the time as the reporting source only polls the status
> anyway ...so we ended up having a python script polling the json
> metrics and pushing that into Ganglia. Was easier than the awkward
> source sink setup for reporting.
>
> cheers,
> Torsten
>
> On Fri, Aug 19, 2011 at 1:20 AM, Mingjie Lai <mj...@gmail.com> wrote:
> > Hi.
> >
> > Has anyone used Ganglia to monitor flume?
> >
> > I'd like to have a way to monitor performance data (e.g., throughput) of
> our
> > customized decorator. I saw there is Ganglia sink in the source code, but
> > don't think it's for flume itself.
> >
> > Can anyone point me how to use ganglia to monitor flume metrics?
> >
> > Thanks,
> > Mingjie
> >
>
Re: Performance monitoring by Ganglia
Posted by Eric Hauser <ew...@gmail.com>.
FWIW, there is a Ruby script in the ostrich repository for parsing
JSON stats and sending them to Ganglia that could be slightly modified
to work with Flume:
https://github.com/twitter/ostrich/blob/master/src/scripts/json_stats_fetcher.rb
On Fri, Aug 19, 2011 at 6:19 PM, Mingjie Lai <mj...@gmail.com> wrote:
> Jon.
>
> Have you considered to bring something like hadoop metrics framework into
> flume? So we can decide where the metrics data goes by configuration.
>
> Otherwise we have to have another piece of code that polls and parses the
> json and pushes to ganglia (as Torsten mentioned).
>
> The current html and json reporting is pretty powerful. If flume can push
> metrics to ganglia out-of-box, it would be even better especially for
> operations.
>
> Thanks,
> Mingjie
>
> On 08/19/2011 12:53 AM, Jonathan Hsieh wrote:
>>
>> I tend to agree with Torsten here about the complexity of the ganglia
>> sink. I think it easier to ask the service for metrics data. I
>> started documenting the metrics fields reported by the json -- here's a
>> link:
>>
>> https://cwiki.apache.org/FLUME/monitoring-flume.html
>>
>> Its a work in progress -- I'll eventually document more of it in the wiki.
>>
>> Jon.
>>
>> On Fri, Aug 19, 2011 at 12:43 AM, Torsten Curdt <tcurdt@vafer.org
>> <ma...@vafer.org>> wrote:
>>
>> The current reporting is quite broken and does not support the Ganglia
>> suggested multicast setup at all. We've got some code where we fixed
>> the source and the sink but still had some trouble getting it working
>> reliably (on the Ganglia sink side of things). But it didn't felt
>> worth spending the time as the reporting source only polls the status
>> anyway ...so we ended up having a python script polling the json
>> metrics and pushing that into Ganglia. Was easier than the awkward
>> source sink setup for reporting.
>>
>> cheers,
>> Torsten
>>
>> On Fri, Aug 19, 2011 at 1:20 AM, Mingjie Lai <mjlai09@gmail.com
>> <ma...@gmail.com>> wrote:
>> > Hi.
>> >
>> > Has anyone used Ganglia to monitor flume?
>> >
>> > I'd like to have a way to monitor performance data (e.g.,
>> throughput) of our
>> > customized decorator. I saw there is Ganglia sink in the source
>> code, but
>> > don't think it's for flume itself.
>> >
>> > Can anyone point me how to use ganglia to monitor flume metrics?
>> >
>> > Thanks,
>> > Mingjie
>> >
>>
>>
>>
>>
>> --
>> // Jonathan Hsieh (shay)
>> // Software Engineer, Cloudera
>> // jon@cloudera.com <ma...@cloudera.com>
>>
>
Re: Performance monitoring by Ganglia
Posted by Mingjie Lai <mj...@gmail.com>.
Jon.
Regarding metrics, hbase utilizes hadoop-metrics directly:
http://hbase.apache.org/metrics.html
However I'm not sure how difficult it is to port hadoop-metrics with
current flume reporting framework. Need to look into source code to
understand.
This blog lists all hbase metrics:
http://blog.kenweiner.com/2010/10/monitor-hbase-hadoop-with-ganglia-on.html
-mingjie
On 08/19/2011 04:45 PM, Jonathan Hsieh wrote:
> Mingjie,
>
> I don't know much about Hadoop-metrics currently but that sounds like a
> great idea. Do you know if other hadoop-ish (hbase?) projects use it as
> well? If they do then I think this makes even more sense.
>
> Right now the the ganglia stuff in flume is very old and really needs to
> be re-thought. It was essentially "borrowed" from a 0.18 version of
> hadoop I believe.
>
> Jon.
>
> On Fri, Aug 19, 2011 at 3:19 PM, Mingjie Lai <mjlai09@gmail.com
> <ma...@gmail.com>> wrote:
>
> Jon.
>
> Have you considered to bring something like hadoop metrics framework
> into flume? So we can decide where the metrics data goes by
> configuration.
>
> Otherwise we have to have another piece of code that polls and
> parses the json and pushes to ganglia (as Torsten mentioned).
>
> The current html and json reporting is pretty powerful. If flume can
> push metrics to ganglia out-of-box, it would be even better
> especially for operations.
>
> Thanks,
> Mingjie
>
>
> On 08/19/2011 12:53 AM, Jonathan Hsieh wrote:
>
> I tend to agree with Torsten here about the complexity of the
> ganglia
> sink. I think it easier to ask the service for metrics data. I
> started documenting the metrics fields reported by the json --
> here's a
> link:
>
> https://cwiki.apache.org/__FLUME/monitoring-flume.html
> <https://cwiki.apache.org/FLUME/monitoring-flume.html>
>
> Its a work in progress -- I'll eventually document more of it in
> the wiki.
>
> Jon.
>
> On Fri, Aug 19, 2011 at 12:43 AM, Torsten Curdt
> <tcurdt@vafer.org <ma...@vafer.org>
> <mailto:tcurdt@vafer.org <ma...@vafer.org>>> wrote:
>
> The current reporting is quite broken and does not support
> the Ganglia
> suggested multicast setup at all. We've got some code where
> we fixed
> the source and the sink but still had some trouble getting
> it working
> reliably (on the Ganglia sink side of things). But it didn't
> felt
> worth spending the time as the reporting source only polls
> the status
> anyway ...so we ended up having a python script polling the json
> metrics and pushing that into Ganglia. Was easier than the
> awkward
> source sink setup for reporting.
>
> cheers,
> Torsten
>
> On Fri, Aug 19, 2011 at 1:20 AM, Mingjie Lai
> <mjlai09@gmail.com <ma...@gmail.com>
> <mailto:mjlai09@gmail.com <ma...@gmail.com>>> wrote:
> > Hi.
> >
> > Has anyone used Ganglia to monitor flume?
> >
> > I'd like to have a way to monitor performance data (e.g.,
> throughput) of our
> > customized decorator. I saw there is Ganglia sink in the source
> code, but
> > don't think it's for flume itself.
> >
> > Can anyone point me how to use ganglia to monitor flume metrics?
> >
> > Thanks,
> > Mingjie
> >
>
>
>
>
> --
> // Jonathan Hsieh (shay)
> // Software Engineer, Cloudera
> // jon@cloudera.com <ma...@cloudera.com>
> <mailto:jon@cloudera.com <ma...@cloudera.com>>
>
>
>
>
> --
> // Jonathan Hsieh (shay)
> // Software Engineer, Cloudera
> // jon@cloudera.com <ma...@cloudera.com>
>
Re: Performance monitoring by Ganglia
Posted by Jonathan Hsieh <jo...@cloudera.com>.
Mingjie,
I don't know much about Hadoop-metrics currently but that sounds like a
great idea. Do you know if other hadoop-ish (hbase?) projects use it as
well? If they do then I think this makes even more sense.
Right now the the ganglia stuff in flume is very old and really needs to be
re-thought. It was essentially "borrowed" from a 0.18 version of hadoop I
believe.
Jon.
On Fri, Aug 19, 2011 at 3:19 PM, Mingjie Lai <mj...@gmail.com> wrote:
> Jon.
>
> Have you considered to bring something like hadoop metrics framework into
> flume? So we can decide where the metrics data goes by configuration.
>
> Otherwise we have to have another piece of code that polls and parses the
> json and pushes to ganglia (as Torsten mentioned).
>
> The current html and json reporting is pretty powerful. If flume can push
> metrics to ganglia out-of-box, it would be even better especially for
> operations.
>
> Thanks,
> Mingjie
>
>
> On 08/19/2011 12:53 AM, Jonathan Hsieh wrote:
>
>> I tend to agree with Torsten here about the complexity of the ganglia
>> sink. I think it easier to ask the service for metrics data. I
>> started documenting the metrics fields reported by the json -- here's a
>> link:
>>
>> https://cwiki.apache.org/**FLUME/monitoring-flume.html<https://cwiki.apache.org/FLUME/monitoring-flume.html>
>>
>> Its a work in progress -- I'll eventually document more of it in the wiki.
>>
>> Jon.
>>
>> On Fri, Aug 19, 2011 at 12:43 AM, Torsten Curdt <tcurdt@vafer.org
>> <ma...@vafer.org>> wrote:
>>
>> The current reporting is quite broken and does not support the Ganglia
>> suggested multicast setup at all. We've got some code where we fixed
>> the source and the sink but still had some trouble getting it working
>> reliably (on the Ganglia sink side of things). But it didn't felt
>> worth spending the time as the reporting source only polls the status
>> anyway ...so we ended up having a python script polling the json
>> metrics and pushing that into Ganglia. Was easier than the awkward
>> source sink setup for reporting.
>>
>> cheers,
>> Torsten
>>
>> On Fri, Aug 19, 2011 at 1:20 AM, Mingjie Lai <mjlai09@gmail.com
>> <ma...@gmail.com>> wrote:
>> > Hi.
>> >
>> > Has anyone used Ganglia to monitor flume?
>> >
>> > I'd like to have a way to monitor performance data (e.g.,
>> throughput) of our
>> > customized decorator. I saw there is Ganglia sink in the source
>> code, but
>> > don't think it's for flume itself.
>> >
>> > Can anyone point me how to use ganglia to monitor flume metrics?
>> >
>> > Thanks,
>> > Mingjie
>> >
>>
>>
>>
>>
>> --
>> // Jonathan Hsieh (shay)
>> // Software Engineer, Cloudera
>> // jon@cloudera.com <ma...@cloudera.com>
>>
>>
--
// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// jon@cloudera.com
Re: Performance monitoring by Ganglia
Posted by Mingjie Lai <mj...@gmail.com>.
Jon.
Have you considered to bring something like hadoop metrics framework
into flume? So we can decide where the metrics data goes by configuration.
Otherwise we have to have another piece of code that polls and parses
the json and pushes to ganglia (as Torsten mentioned).
The current html and json reporting is pretty powerful. If flume can
push metrics to ganglia out-of-box, it would be even better especially
for operations.
Thanks,
Mingjie
On 08/19/2011 12:53 AM, Jonathan Hsieh wrote:
> I tend to agree with Torsten here about the complexity of the ganglia
> sink. I think it easier to ask the service for metrics data. I
> started documenting the metrics fields reported by the json -- here's a
> link:
>
> https://cwiki.apache.org/FLUME/monitoring-flume.html
>
> Its a work in progress -- I'll eventually document more of it in the wiki.
>
> Jon.
>
> On Fri, Aug 19, 2011 at 12:43 AM, Torsten Curdt <tcurdt@vafer.org
> <ma...@vafer.org>> wrote:
>
> The current reporting is quite broken and does not support the Ganglia
> suggested multicast setup at all. We've got some code where we fixed
> the source and the sink but still had some trouble getting it working
> reliably (on the Ganglia sink side of things). But it didn't felt
> worth spending the time as the reporting source only polls the status
> anyway ...so we ended up having a python script polling the json
> metrics and pushing that into Ganglia. Was easier than the awkward
> source sink setup for reporting.
>
> cheers,
> Torsten
>
> On Fri, Aug 19, 2011 at 1:20 AM, Mingjie Lai <mjlai09@gmail.com
> <ma...@gmail.com>> wrote:
> > Hi.
> >
> > Has anyone used Ganglia to monitor flume?
> >
> > I'd like to have a way to monitor performance data (e.g.,
> throughput) of our
> > customized decorator. I saw there is Ganglia sink in the source
> code, but
> > don't think it's for flume itself.
> >
> > Can anyone point me how to use ganglia to monitor flume metrics?
> >
> > Thanks,
> > Mingjie
> >
>
>
>
>
> --
> // Jonathan Hsieh (shay)
> // Software Engineer, Cloudera
> // jon@cloudera.com <ma...@cloudera.com>
>
Re: Performance monitoring by Ganglia
Posted by Jonathan Hsieh <jo...@cloudera.com>.
I tend to agree with Torsten here about the complexity of the ganglia sink.
I think it easier to ask the service for metrics data. I started
documenting the metrics fields reported by the json -- here's a link:
https://cwiki.apache.org/FLUME/monitoring-flume.html
Its a work in progress -- I'll eventually document more of it in the wiki.
Jon.
On Fri, Aug 19, 2011 at 12:43 AM, Torsten Curdt <tc...@vafer.org> wrote:
> The current reporting is quite broken and does not support the Ganglia
> suggested multicast setup at all. We've got some code where we fixed
> the source and the sink but still had some trouble getting it working
> reliably (on the Ganglia sink side of things). But it didn't felt
> worth spending the time as the reporting source only polls the status
> anyway ...so we ended up having a python script polling the json
> metrics and pushing that into Ganglia. Was easier than the awkward
> source sink setup for reporting.
>
> cheers,
> Torsten
>
> On Fri, Aug 19, 2011 at 1:20 AM, Mingjie Lai <mj...@gmail.com> wrote:
> > Hi.
> >
> > Has anyone used Ganglia to monitor flume?
> >
> > I'd like to have a way to monitor performance data (e.g., throughput) of
> our
> > customized decorator. I saw there is Ganglia sink in the source code, but
> > don't think it's for flume itself.
> >
> > Can anyone point me how to use ganglia to monitor flume metrics?
> >
> > Thanks,
> > Mingjie
> >
>
--
// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// jon@cloudera.com
Re: Performance monitoring by Ganglia
Posted by Torsten Curdt <tc...@vafer.org>.
The current reporting is quite broken and does not support the Ganglia
suggested multicast setup at all. We've got some code where we fixed
the source and the sink but still had some trouble getting it working
reliably (on the Ganglia sink side of things). But it didn't felt
worth spending the time as the reporting source only polls the status
anyway ...so we ended up having a python script polling the json
metrics and pushing that into Ganglia. Was easier than the awkward
source sink setup for reporting.
cheers,
Torsten
On Fri, Aug 19, 2011 at 1:20 AM, Mingjie Lai <mj...@gmail.com> wrote:
> Hi.
>
> Has anyone used Ganglia to monitor flume?
>
> I'd like to have a way to monitor performance data (e.g., throughput) of our
> customized decorator. I saw there is Ganglia sink in the source code, but
> don't think it's for flume itself.
>
> Can anyone point me how to use ganglia to monitor flume metrics?
>
> Thanks,
> Mingjie
>