You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Gautam <ga...@gmail.com> on 2010/08/24 15:27:26 UTC
Ganglia 3.1 on Hadoop 0.20.2 ...
I was trying to get Ganglia 3.1 to work with the stable hadoop-0.20.2
version from Apache. I patched this release from HADOOP-4675 using
HADOOP-4675-v7.patch as suggested by CDH3 release notes [1] I am unable to
see any hadoop metrics on the Ganglia monitoring UI. The other metrics that
gmond spews (system CPU/Memory etc) seem to work.
When I switch to FileContext the metrics are written properly to the log
file. Once I moved to GangliaContext31 it doesn't show anything. I tried
pointing the "servers" param to localhost:8649 while listening on that port
using netcat on that machine... nothing comes up on netcat. Has anyone faced
this issue?
This is what most of my hadoop-metrics looks like:
dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
dfs.period=10
dfs.fileName=/tmp/dfsmetrics.log
dfs.servers=$Ganglia_Server:9988
# Configuration of the "mapred" context for null
mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
mapred.period=10
mapred.fileName=/tmp/mrmetrics.log
mapred.servers=$Ganglia_Server:9988
# Configuration of the "jvm" context for null
jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
jvm.period=10
jvm.fileName=/tmp/jvmmetrics.log
jvm.servers=$GANGLIA_SERVER:9988
-G.
[1] - http://archive.cloudera.com/cdh/3/hadoop-0.20.2+228.CHANGES.txt
Re: Ganglia 3.1 on Hadoop 0.20.2 ...
Posted by Brian Bockelman <bb...@cse.unl.edu>.
Hi Gautam,
Yup - that's one possible way to configure Ganglia and is common at many sites. That's why I usually recommend the "telnet" trick to determine what IP address your configuration is using.
Brian
On Aug 25, 2010, at 5:53 AM, Gautam wrote:
> Brian,
>
> Works for me now.. one should point the "servers" param to the multicast
> address that gmond writes to and listens on... and not the ganglia server.
> Started working once I did this.
>
> thanks for you inputs,
> -G.
>
> On Tue, Aug 24, 2010 at 7:12 PM, Brian Bockelman <bb...@cse.unl.edu>wrote:
>
>>
>> On Aug 24, 2010, at 8:27 AM, Gautam wrote:
>>
>>> I was trying to get Ganglia 3.1 to work with the stable hadoop-0.20.2
>>> version from Apache. I patched this release from HADOOP-4675 using
>>> HADOOP-4675-v7.patch as suggested by CDH3 release notes [1] I am unable
>> to
>>> see any hadoop metrics on the Ganglia monitoring UI. The other metrics
>> that
>>> gmond spews (system CPU/Memory etc) seem to work.
>>>
>>> When I switch to FileContext the metrics are written properly to the log
>>> file. Once I moved to GangliaContext31 it doesn't show anything. I tried
>>> pointing the "servers" param to localhost:8649 while listening on that
>> port
>>> using netcat on that machine... nothing comes up on netcat. Has anyone
>> faced
>>> this issue?
>>
>> This is possibly misleading - netcat won't work if Hadoop is using UDP.
>>
>> My advice is to do:
>>
>> telnet $Ganglia_Server 9988
>>
>> and see if it spits out a bunch of XML. In the typical Ganglia
>> configuration, it is set up to listen on UDP and write on TCP of the same
>> port.
>>
>> A third thing to test is to switch the hadoop-metrics back to the file
>> output, and make sure something gets written to the log file. The issue
>> might be upstream.
>>
>> Brian
>>
>>>
>>> This is what most of my hadoop-metrics looks like:
>>>
>>> dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
>>> dfs.period=10
>>> dfs.fileName=/tmp/dfsmetrics.log
>>> dfs.servers=$Ganglia_Server:9988
>>>
>>> # Configuration of the "mapred" context for null
>>> mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
>>> mapred.period=10
>>> mapred.fileName=/tmp/mrmetrics.log
>>> mapred.servers=$Ganglia_Server:9988
>>>
>>> # Configuration of the "jvm" context for null
>>> jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
>>> jvm.period=10
>>> jvm.fileName=/tmp/jvmmetrics.log
>>> jvm.servers=$GANGLIA_SERVER:9988
>>>
>>> -G.
>>>
>>> [1] - http://archive.cloudera.com/cdh/3/hadoop-0.20.2+228.CHANGES.txt
>>
>>
>
>
> --
> "If you really want something in this life, you have to work for it. Now,
> quiet! They're about to announce the lottery numbers..."
Re: Ganglia 3.1 on Hadoop 0.20.2 ...
Posted by Gautam <ga...@gmail.com>.
Brian,
Works for me now.. one should point the "servers" param to the multicast
address that gmond writes to and listens on... and not the ganglia server.
Started working once I did this.
thanks for you inputs,
-G.
On Tue, Aug 24, 2010 at 7:12 PM, Brian Bockelman <bb...@cse.unl.edu>wrote:
>
> On Aug 24, 2010, at 8:27 AM, Gautam wrote:
>
> > I was trying to get Ganglia 3.1 to work with the stable hadoop-0.20.2
> > version from Apache. I patched this release from HADOOP-4675 using
> > HADOOP-4675-v7.patch as suggested by CDH3 release notes [1] I am unable
> to
> > see any hadoop metrics on the Ganglia monitoring UI. The other metrics
> that
> > gmond spews (system CPU/Memory etc) seem to work.
> >
> > When I switch to FileContext the metrics are written properly to the log
> > file. Once I moved to GangliaContext31 it doesn't show anything. I tried
> > pointing the "servers" param to localhost:8649 while listening on that
> port
> > using netcat on that machine... nothing comes up on netcat. Has anyone
> faced
> > this issue?
>
> This is possibly misleading - netcat won't work if Hadoop is using UDP.
>
> My advice is to do:
>
> telnet $Ganglia_Server 9988
>
> and see if it spits out a bunch of XML. In the typical Ganglia
> configuration, it is set up to listen on UDP and write on TCP of the same
> port.
>
> A third thing to test is to switch the hadoop-metrics back to the file
> output, and make sure something gets written to the log file. The issue
> might be upstream.
>
> Brian
>
> >
> > This is what most of my hadoop-metrics looks like:
> >
> > dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
> > dfs.period=10
> > dfs.fileName=/tmp/dfsmetrics.log
> > dfs.servers=$Ganglia_Server:9988
> >
> > # Configuration of the "mapred" context for null
> > mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
> > mapred.period=10
> > mapred.fileName=/tmp/mrmetrics.log
> > mapred.servers=$Ganglia_Server:9988
> >
> > # Configuration of the "jvm" context for null
> > jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
> > jvm.period=10
> > jvm.fileName=/tmp/jvmmetrics.log
> > jvm.servers=$GANGLIA_SERVER:9988
> >
> > -G.
> >
> > [1] - http://archive.cloudera.com/cdh/3/hadoop-0.20.2+228.CHANGES.txt
>
>
--
"If you really want something in this life, you have to work for it. Now,
quiet! They're about to announce the lottery numbers..."
Re: Ganglia 3.1 on Hadoop 0.20.2 ...
Posted by Brian Bockelman <bb...@cse.unl.edu>.
On Aug 24, 2010, at 8:27 AM, Gautam wrote:
> I was trying to get Ganglia 3.1 to work with the stable hadoop-0.20.2
> version from Apache. I patched this release from HADOOP-4675 using
> HADOOP-4675-v7.patch as suggested by CDH3 release notes [1] I am unable to
> see any hadoop metrics on the Ganglia monitoring UI. The other metrics that
> gmond spews (system CPU/Memory etc) seem to work.
>
> When I switch to FileContext the metrics are written properly to the log
> file. Once I moved to GangliaContext31 it doesn't show anything. I tried
> pointing the "servers" param to localhost:8649 while listening on that port
> using netcat on that machine... nothing comes up on netcat. Has anyone faced
> this issue?
This is possibly misleading - netcat won't work if Hadoop is using UDP.
My advice is to do:
telnet $Ganglia_Server 9988
and see if it spits out a bunch of XML. In the typical Ganglia configuration, it is set up to listen on UDP and write on TCP of the same port.
A third thing to test is to switch the hadoop-metrics back to the file output, and make sure something gets written to the log file. The issue might be upstream.
Brian
>
> This is what most of my hadoop-metrics looks like:
>
> dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
> dfs.period=10
> dfs.fileName=/tmp/dfsmetrics.log
> dfs.servers=$Ganglia_Server:9988
>
> # Configuration of the "mapred" context for null
> mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
> mapred.period=10
> mapred.fileName=/tmp/mrmetrics.log
> mapred.servers=$Ganglia_Server:9988
>
> # Configuration of the "jvm" context for null
> jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
> jvm.period=10
> jvm.fileName=/tmp/jvmmetrics.log
> jvm.servers=$GANGLIA_SERVER:9988
>
> -G.
>
> [1] - http://archive.cloudera.com/cdh/3/hadoop-0.20.2+228.CHANGES.txt