You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Hari Sreekumar <hs...@clickable.com> on 2010/11/22 13:55:21 UTC
Using ganglia to monitor hadoop
Hi,
I have set up ganglia for my cluster, and it works fine. What are
the changes I need to make to make ganglia show hadoop related parameters?
My gmond/gmetad config is default except for one change in gmond.conf for
all nodes:
/* You can specify as many tcp_accept_channels as you like to share
an xml description of the state of the cluster */
tcp_accept_channel {
port = 8650
}
I have changed the hadoop-metrics.properties file as follows:
# Configuration of the "dfs" context for ganglia
dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext
dfs.period=10
dfs.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649
# Configuration of the "mapred" context for null
#mapred.class=org.apache.hadoop.metrics.spi.NullContext
# Configuration of the "mapred" context for file
#mapred.class=org.apache.hadoop.metrics.file.FileContext
#mapred.period=10
#mapred.fileName=/tmp/mrmetrics.log
# Configuration of the "mapred" context for ganglia
mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext
mapred.period=10
mapred.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649
# Configuration of the "jvm" context for null
#jvm.class=org.apache.hadoop.metrics.spi.NullContext
# Configuration of the "jvm" context for file
#jvm.class=org.apache.hadoop.metrics.file.FileContext
#jvm.period=10
#jvm.fileName=/tmp/jvmmetrics.log
# Configuration of the "jvm" context for ganglia
jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext
jvm.period=10
jvm.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649
rpc.class=org.apache.hadoop.metrics.ganglia.GangliaContext
rpc.period=10
rpc.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649
hadoop1,2,3 and ejabber are my nodes.
But I am unable to see any new metrics in ganglia's drop down menu. Is there
something else I need to do as well? I have restarted gmond on all nodes.
thanks,
Hari
Re: Using ganglia to monitor hadoop
Posted by Hari Sreekumar <hs...@clickable.com>.
Hey Matthew,
Got it working now.... had to restart the cluster. I am not sure
if restarting the cluster is what did the trick though. Even after
restarting, I had no metrics for about half an hour and then they started
coming. Thanks a lot for helping me set it up!
Cheers,
Hari
On Tue, Nov 23, 2010 at 1:40 AM, Matthew E. Kennedy <matt.kennedy@spadac.com
> wrote:
> I started to get mapred and jvm metrics without restarting the cluster, but
> it did require a restart to start getting dfs stats. Yes, you should see
> the new metrics appear in the dropdown.
>
> On Nov 22, 2010, at 12:21 PM, Hari Sreekumar wrote:
>
> > Hi Matt,
> > Tried using the specified(default) mcast join ip address. Still no
> > new metrics. Do I have to restart the hadoop cluster for the changes to
> take
> > effect? The new metrics will appear in the "Metrics" dropdown menu of
> > ganglia UI home page right? Or is there some other link for custom
> metrics?
> >
> > thanks,
> > Hari
> >
> > On Mon, Nov 22, 2010 at 9:08 PM, Matthew E. Kennedy <
> matt.kennedy@spadac.com
> >> wrote:
> >
> >> Hari,
> >>
> >> I have this set up and working and it looks to me like your *.servers
> >> properties are set incorrectly. Point that property to the multicast
> >> address that is listed in your gmond.conf file on port 8649.
> >>
> >> You can find the multicast address with:
> >> cat /etc/gmond.conf | grep mcast
> >>
> >> an example of the config follows:
> >> dfs.servers=<mcast_join_ip_addr>:8649
> >>
> >> Good luck,
> >> Matt
> >>
> >> On Nov 22, 2010, at 8:20 AM, Hari Sreekumar wrote:
> >>
> >>> Hi Sagar,
> >>>
> >>> I followed that article. But I am not clear about some things:
> >>> 1. Should I use port 8650 or 8649? I have tried both actually, and 8649
> >>> makes sense, but just to confirm. (I have set tcp_accept_channel as
> 8650)
> >>> 2. In hadoop-metrics.properties, should I mention all my nodes as I
> have
> >>> done or only the local node? e.g, localhost:8649 everywhere?
> >>>
> >>> I have configured gmetad.conf, and I am able to see metrics from all my
> >>> nodes in my ganglia UI. I just don't see any hadoop specific metrics.
> >>>
> >>> Thanks,
> >>> Hari
> >>>
> >>> On Mon, Nov 22, 2010 at 6:42 PM, Sagar Shukla <
> >> sagar_shukla@persistent.co.in
> >>>> wrote:
> >>>
> >>>> Hi Hari,
> >>>> Not sure if you have already followed article -
> >>>> http://wiki.apache.org/hadoop/GangliaMetrics to configure Ganglia for
> >>>> Hadoop Monitoring. Also in the details given by you, I do not see any
> >>>> configuration settings done in gmetad config. You will need to define
> >> the
> >>>> hosts from whom metrics should be collected. Once this configuration,
> >> you
> >>>> should see the host in gmetad UI interface.
> >>>>
> >>>> Thanks,
> >>>> Sagar
> >>>>
> >>>> -----Original Message-----
> >>>> From: Hari Sreekumar [mailto:hsreekumar@clickable.com]
> >>>> Sent: Monday, November 22, 2010 6:25 PM
> >>>> To: common-user@hadoop.apache.org
> >>>> Subject: Using ganglia to monitor hadoop
> >>>>
> >>>> Hi,
> >>>>
> >>>> I have set up ganglia for my cluster, and it works fine. What are
> >>>> the changes I need to make to make ganglia show hadoop related
> >> parameters?
> >>>> My gmond/gmetad config is default except for one change in gmond.conf
> >> for
> >>>> all nodes:
> >>>>
> >>>> /* You can specify as many tcp_accept_channels as you like to share
> >>>> an xml description of the state of the cluster */
> >>>> tcp_accept_channel {
> >>>> port = 8650
> >>>> }
> >>>>
> >>>> I have changed the hadoop-metrics.properties file as follows:
> >>>>
> >>>> # Configuration of the "dfs" context for ganglia
> >>>> dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext
> >>>> dfs.period=10
> >>>> dfs.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649
> >>>>
> >>>>
> >>>> # Configuration of the "mapred" context for null
> >>>> #mapred.class=org.apache.hadoop.metrics.spi.NullContext
> >>>>
> >>>> # Configuration of the "mapred" context for file
> >>>> #mapred.class=org.apache.hadoop.metrics.file.FileContext
> >>>> #mapred.period=10
> >>>> #mapred.fileName=/tmp/mrmetrics.log
> >>>>
> >>>> # Configuration of the "mapred" context for ganglia
> >>>> mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext
> >>>> mapred.period=10
> >>>> mapred.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649
> >>>>
> >>>>
> >>>> # Configuration of the "jvm" context for null
> >>>> #jvm.class=org.apache.hadoop.metrics.spi.NullContext
> >>>>
> >>>> # Configuration of the "jvm" context for file
> >>>> #jvm.class=org.apache.hadoop.metrics.file.FileContext
> >>>> #jvm.period=10
> >>>> #jvm.fileName=/tmp/jvmmetrics.log
> >>>>
> >>>> # Configuration of the "jvm" context for ganglia
> >>>> jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext
> >>>> jvm.period=10
> >>>> jvm.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649
> >>>>
> >>>> rpc.class=org.apache.hadoop.metrics.ganglia.GangliaContext
> >>>> rpc.period=10
> >>>> rpc.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649
> >>>>
> >>>> hadoop1,2,3 and ejabber are my nodes.
> >>>>
> >>>> But I am unable to see any new metrics in ganglia's drop down menu. Is
> >>>> there
> >>>> something else I need to do as well? I have restarted gmond on all
> >> nodes.
> >>>>
> >>>> thanks,
> >>>> Hari
> >>>>
> >>>> DISCLAIMER
> >>>> ==========
> >>>> This e-mail may contain privileged and confidential information which
> is
> >>>> the property of Persistent Systems Ltd. It is intended only for the
> use
> >> of
> >>>> the individual or entity to which it is addressed. If you are not the
> >>>> intended recipient, you are not authorized to read, retain, copy,
> print,
> >>>> distribute or use this message. If you have received this
> communication
> >> in
> >>>> error, please notify the sender and delete all copies of this message.
> >>>> Persistent Systems Ltd. does not accept any liability for virus
> infected
> >>>> mails.
> >>>>
> >>
> >>
>
>
Re: Using ganglia to monitor hadoop
Posted by "Matthew E. Kennedy" <ma...@spadac.com>.
I started to get mapred and jvm metrics without restarting the cluster, but it did require a restart to start getting dfs stats. Yes, you should see the new metrics appear in the dropdown.
On Nov 22, 2010, at 12:21 PM, Hari Sreekumar wrote:
> Hi Matt,
> Tried using the specified(default) mcast join ip address. Still no
> new metrics. Do I have to restart the hadoop cluster for the changes to take
> effect? The new metrics will appear in the "Metrics" dropdown menu of
> ganglia UI home page right? Or is there some other link for custom metrics?
>
> thanks,
> Hari
>
> On Mon, Nov 22, 2010 at 9:08 PM, Matthew E. Kennedy <matt.kennedy@spadac.com
>> wrote:
>
>> Hari,
>>
>> I have this set up and working and it looks to me like your *.servers
>> properties are set incorrectly. Point that property to the multicast
>> address that is listed in your gmond.conf file on port 8649.
>>
>> You can find the multicast address with:
>> cat /etc/gmond.conf | grep mcast
>>
>> an example of the config follows:
>> dfs.servers=<mcast_join_ip_addr>:8649
>>
>> Good luck,
>> Matt
>>
>> On Nov 22, 2010, at 8:20 AM, Hari Sreekumar wrote:
>>
>>> Hi Sagar,
>>>
>>> I followed that article. But I am not clear about some things:
>>> 1. Should I use port 8650 or 8649? I have tried both actually, and 8649
>>> makes sense, but just to confirm. (I have set tcp_accept_channel as 8650)
>>> 2. In hadoop-metrics.properties, should I mention all my nodes as I have
>>> done or only the local node? e.g, localhost:8649 everywhere?
>>>
>>> I have configured gmetad.conf, and I am able to see metrics from all my
>>> nodes in my ganglia UI. I just don't see any hadoop specific metrics.
>>>
>>> Thanks,
>>> Hari
>>>
>>> On Mon, Nov 22, 2010 at 6:42 PM, Sagar Shukla <
>> sagar_shukla@persistent.co.in
>>>> wrote:
>>>
>>>> Hi Hari,
>>>> Not sure if you have already followed article -
>>>> http://wiki.apache.org/hadoop/GangliaMetrics to configure Ganglia for
>>>> Hadoop Monitoring. Also in the details given by you, I do not see any
>>>> configuration settings done in gmetad config. You will need to define
>> the
>>>> hosts from whom metrics should be collected. Once this configuration,
>> you
>>>> should see the host in gmetad UI interface.
>>>>
>>>> Thanks,
>>>> Sagar
>>>>
>>>> -----Original Message-----
>>>> From: Hari Sreekumar [mailto:hsreekumar@clickable.com]
>>>> Sent: Monday, November 22, 2010 6:25 PM
>>>> To: common-user@hadoop.apache.org
>>>> Subject: Using ganglia to monitor hadoop
>>>>
>>>> Hi,
>>>>
>>>> I have set up ganglia for my cluster, and it works fine. What are
>>>> the changes I need to make to make ganglia show hadoop related
>> parameters?
>>>> My gmond/gmetad config is default except for one change in gmond.conf
>> for
>>>> all nodes:
>>>>
>>>> /* You can specify as many tcp_accept_channels as you like to share
>>>> an xml description of the state of the cluster */
>>>> tcp_accept_channel {
>>>> port = 8650
>>>> }
>>>>
>>>> I have changed the hadoop-metrics.properties file as follows:
>>>>
>>>> # Configuration of the "dfs" context for ganglia
>>>> dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext
>>>> dfs.period=10
>>>> dfs.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649
>>>>
>>>>
>>>> # Configuration of the "mapred" context for null
>>>> #mapred.class=org.apache.hadoop.metrics.spi.NullContext
>>>>
>>>> # Configuration of the "mapred" context for file
>>>> #mapred.class=org.apache.hadoop.metrics.file.FileContext
>>>> #mapred.period=10
>>>> #mapred.fileName=/tmp/mrmetrics.log
>>>>
>>>> # Configuration of the "mapred" context for ganglia
>>>> mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext
>>>> mapred.period=10
>>>> mapred.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649
>>>>
>>>>
>>>> # Configuration of the "jvm" context for null
>>>> #jvm.class=org.apache.hadoop.metrics.spi.NullContext
>>>>
>>>> # Configuration of the "jvm" context for file
>>>> #jvm.class=org.apache.hadoop.metrics.file.FileContext
>>>> #jvm.period=10
>>>> #jvm.fileName=/tmp/jvmmetrics.log
>>>>
>>>> # Configuration of the "jvm" context for ganglia
>>>> jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext
>>>> jvm.period=10
>>>> jvm.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649
>>>>
>>>> rpc.class=org.apache.hadoop.metrics.ganglia.GangliaContext
>>>> rpc.period=10
>>>> rpc.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649
>>>>
>>>> hadoop1,2,3 and ejabber are my nodes.
>>>>
>>>> But I am unable to see any new metrics in ganglia's drop down menu. Is
>>>> there
>>>> something else I need to do as well? I have restarted gmond on all
>> nodes.
>>>>
>>>> thanks,
>>>> Hari
>>>>
>>>> DISCLAIMER
>>>> ==========
>>>> This e-mail may contain privileged and confidential information which is
>>>> the property of Persistent Systems Ltd. It is intended only for the use
>> of
>>>> the individual or entity to which it is addressed. If you are not the
>>>> intended recipient, you are not authorized to read, retain, copy, print,
>>>> distribute or use this message. If you have received this communication
>> in
>>>> error, please notify the sender and delete all copies of this message.
>>>> Persistent Systems Ltd. does not accept any liability for virus infected
>>>> mails.
>>>>
>>
>>
gc setting for hadoop
Posted by Jinsong Hu <ji...@hotmail.com>.
Hi, There:
I have been searching good gc setting for hadoop namenode and datanode,
and I use this setting for
namenode and data node:
-XX:NewSize=18m -XX:MaxNewSize=18m -XX:+
HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70
-XX:+CMSClassUnloadingEnabled -XX:+DisableExplicitGC -XX:+UseCompressedOops
-XX:+DoEscapeAnalysis -XX:+AggressiveOpts -verbose:gc -XX:+PrintGCDetails -
XX:+PrintGCTimeStamps -Xmx3G -Dcom.sun.management.jmxremote.port=8004 -Xloggc:/usr/lib/hadoop/logs/gc-namenode.log
-XX:ParallelGCThreads=8 -XX:PermSize=256m -XX
:MaxPermSize=256m
The problem , however, is that I see frequent CMS gc that happens every 100
second. and 2/3 of the GC says things like this:
CMS: abort preclean due to time 795.999:
[CMS-concurrent-abortable-preclean: 0.
188/5.082 secs] [Times: user=0.15 sys=0.00, real=5.08 secs]
I searched internet and some people says that is ok. However, I would like
to see if anybody else has a good GC setting for hadoop that produces less
frequent GC and works well.
The advantage of the above setting is that it controls memory growth well,
and there is no sudden full GC.
the default setting without above tunning produces very bad gc profile. If
anybody else can share their gc tunning experience, that is really
appreciated.
Jimmy
Re: Using ganglia to monitor hadoop
Posted by Hari Sreekumar <hs...@clickable.com>.
Hi Matt,
Tried using the specified(default) mcast join ip address. Still no
new metrics. Do I have to restart the hadoop cluster for the changes to take
effect? The new metrics will appear in the "Metrics" dropdown menu of
ganglia UI home page right? Or is there some other link for custom metrics?
thanks,
Hari
On Mon, Nov 22, 2010 at 9:08 PM, Matthew E. Kennedy <matt.kennedy@spadac.com
> wrote:
> Hari,
>
> I have this set up and working and it looks to me like your *.servers
> properties are set incorrectly. Point that property to the multicast
> address that is listed in your gmond.conf file on port 8649.
>
> You can find the multicast address with:
> cat /etc/gmond.conf | grep mcast
>
> an example of the config follows:
> dfs.servers=<mcast_join_ip_addr>:8649
>
> Good luck,
> Matt
>
> On Nov 22, 2010, at 8:20 AM, Hari Sreekumar wrote:
>
> > Hi Sagar,
> >
> > I followed that article. But I am not clear about some things:
> > 1. Should I use port 8650 or 8649? I have tried both actually, and 8649
> > makes sense, but just to confirm. (I have set tcp_accept_channel as 8650)
> > 2. In hadoop-metrics.properties, should I mention all my nodes as I have
> > done or only the local node? e.g, localhost:8649 everywhere?
> >
> > I have configured gmetad.conf, and I am able to see metrics from all my
> > nodes in my ganglia UI. I just don't see any hadoop specific metrics.
> >
> > Thanks,
> > Hari
> >
> > On Mon, Nov 22, 2010 at 6:42 PM, Sagar Shukla <
> sagar_shukla@persistent.co.in
> >> wrote:
> >
> >> Hi Hari,
> >> Not sure if you have already followed article -
> >> http://wiki.apache.org/hadoop/GangliaMetrics to configure Ganglia for
> >> Hadoop Monitoring. Also in the details given by you, I do not see any
> >> configuration settings done in gmetad config. You will need to define
> the
> >> hosts from whom metrics should be collected. Once this configuration,
> you
> >> should see the host in gmetad UI interface.
> >>
> >> Thanks,
> >> Sagar
> >>
> >> -----Original Message-----
> >> From: Hari Sreekumar [mailto:hsreekumar@clickable.com]
> >> Sent: Monday, November 22, 2010 6:25 PM
> >> To: common-user@hadoop.apache.org
> >> Subject: Using ganglia to monitor hadoop
> >>
> >> Hi,
> >>
> >> I have set up ganglia for my cluster, and it works fine. What are
> >> the changes I need to make to make ganglia show hadoop related
> parameters?
> >> My gmond/gmetad config is default except for one change in gmond.conf
> for
> >> all nodes:
> >>
> >> /* You can specify as many tcp_accept_channels as you like to share
> >> an xml description of the state of the cluster */
> >> tcp_accept_channel {
> >> port = 8650
> >> }
> >>
> >> I have changed the hadoop-metrics.properties file as follows:
> >>
> >> # Configuration of the "dfs" context for ganglia
> >> dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext
> >> dfs.period=10
> >> dfs.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649
> >>
> >>
> >> # Configuration of the "mapred" context for null
> >> #mapred.class=org.apache.hadoop.metrics.spi.NullContext
> >>
> >> # Configuration of the "mapred" context for file
> >> #mapred.class=org.apache.hadoop.metrics.file.FileContext
> >> #mapred.period=10
> >> #mapred.fileName=/tmp/mrmetrics.log
> >>
> >> # Configuration of the "mapred" context for ganglia
> >> mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext
> >> mapred.period=10
> >> mapred.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649
> >>
> >>
> >> # Configuration of the "jvm" context for null
> >> #jvm.class=org.apache.hadoop.metrics.spi.NullContext
> >>
> >> # Configuration of the "jvm" context for file
> >> #jvm.class=org.apache.hadoop.metrics.file.FileContext
> >> #jvm.period=10
> >> #jvm.fileName=/tmp/jvmmetrics.log
> >>
> >> # Configuration of the "jvm" context for ganglia
> >> jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext
> >> jvm.period=10
> >> jvm.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649
> >>
> >> rpc.class=org.apache.hadoop.metrics.ganglia.GangliaContext
> >> rpc.period=10
> >> rpc.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649
> >>
> >> hadoop1,2,3 and ejabber are my nodes.
> >>
> >> But I am unable to see any new metrics in ganglia's drop down menu. Is
> >> there
> >> something else I need to do as well? I have restarted gmond on all
> nodes.
> >>
> >> thanks,
> >> Hari
> >>
> >> DISCLAIMER
> >> ==========
> >> This e-mail may contain privileged and confidential information which is
> >> the property of Persistent Systems Ltd. It is intended only for the use
> of
> >> the individual or entity to which it is addressed. If you are not the
> >> intended recipient, you are not authorized to read, retain, copy, print,
> >> distribute or use this message. If you have received this communication
> in
> >> error, please notify the sender and delete all copies of this message.
> >> Persistent Systems Ltd. does not accept any liability for virus infected
> >> mails.
> >>
>
>
Re: Using ganglia to monitor hadoop
Posted by "Matthew E. Kennedy" <ma...@spadac.com>.
Hari,
I have this set up and working and it looks to me like your *.servers properties are set incorrectly. Point that property to the multicast address that is listed in your gmond.conf file on port 8649.
You can find the multicast address with:
cat /etc/gmond.conf | grep mcast
an example of the config follows:
dfs.servers=<mcast_join_ip_addr>:8649
Good luck,
Matt
On Nov 22, 2010, at 8:20 AM, Hari Sreekumar wrote:
> Hi Sagar,
>
> I followed that article. But I am not clear about some things:
> 1. Should I use port 8650 or 8649? I have tried both actually, and 8649
> makes sense, but just to confirm. (I have set tcp_accept_channel as 8650)
> 2. In hadoop-metrics.properties, should I mention all my nodes as I have
> done or only the local node? e.g, localhost:8649 everywhere?
>
> I have configured gmetad.conf, and I am able to see metrics from all my
> nodes in my ganglia UI. I just don't see any hadoop specific metrics.
>
> Thanks,
> Hari
>
> On Mon, Nov 22, 2010 at 6:42 PM, Sagar Shukla <sagar_shukla@persistent.co.in
>> wrote:
>
>> Hi Hari,
>> Not sure if you have already followed article -
>> http://wiki.apache.org/hadoop/GangliaMetrics to configure Ganglia for
>> Hadoop Monitoring. Also in the details given by you, I do not see any
>> configuration settings done in gmetad config. You will need to define the
>> hosts from whom metrics should be collected. Once this configuration, you
>> should see the host in gmetad UI interface.
>>
>> Thanks,
>> Sagar
>>
>> -----Original Message-----
>> From: Hari Sreekumar [mailto:hsreekumar@clickable.com]
>> Sent: Monday, November 22, 2010 6:25 PM
>> To: common-user@hadoop.apache.org
>> Subject: Using ganglia to monitor hadoop
>>
>> Hi,
>>
>> I have set up ganglia for my cluster, and it works fine. What are
>> the changes I need to make to make ganglia show hadoop related parameters?
>> My gmond/gmetad config is default except for one change in gmond.conf for
>> all nodes:
>>
>> /* You can specify as many tcp_accept_channels as you like to share
>> an xml description of the state of the cluster */
>> tcp_accept_channel {
>> port = 8650
>> }
>>
>> I have changed the hadoop-metrics.properties file as follows:
>>
>> # Configuration of the "dfs" context for ganglia
>> dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext
>> dfs.period=10
>> dfs.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649
>>
>>
>> # Configuration of the "mapred" context for null
>> #mapred.class=org.apache.hadoop.metrics.spi.NullContext
>>
>> # Configuration of the "mapred" context for file
>> #mapred.class=org.apache.hadoop.metrics.file.FileContext
>> #mapred.period=10
>> #mapred.fileName=/tmp/mrmetrics.log
>>
>> # Configuration of the "mapred" context for ganglia
>> mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext
>> mapred.period=10
>> mapred.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649
>>
>>
>> # Configuration of the "jvm" context for null
>> #jvm.class=org.apache.hadoop.metrics.spi.NullContext
>>
>> # Configuration of the "jvm" context for file
>> #jvm.class=org.apache.hadoop.metrics.file.FileContext
>> #jvm.period=10
>> #jvm.fileName=/tmp/jvmmetrics.log
>>
>> # Configuration of the "jvm" context for ganglia
>> jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext
>> jvm.period=10
>> jvm.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649
>>
>> rpc.class=org.apache.hadoop.metrics.ganglia.GangliaContext
>> rpc.period=10
>> rpc.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649
>>
>> hadoop1,2,3 and ejabber are my nodes.
>>
>> But I am unable to see any new metrics in ganglia's drop down menu. Is
>> there
>> something else I need to do as well? I have restarted gmond on all nodes.
>>
>> thanks,
>> Hari
>>
>> DISCLAIMER
>> ==========
>> This e-mail may contain privileged and confidential information which is
>> the property of Persistent Systems Ltd. It is intended only for the use of
>> the individual or entity to which it is addressed. If you are not the
>> intended recipient, you are not authorized to read, retain, copy, print,
>> distribute or use this message. If you have received this communication in
>> error, please notify the sender and delete all copies of this message.
>> Persistent Systems Ltd. does not accept any liability for virus infected
>> mails.
>>
Re: Using ganglia to monitor hadoop
Posted by Hari Sreekumar <hs...@clickable.com>.
Hi Sagar,
I followed that article. But I am not clear about some things:
1. Should I use port 8650 or 8649? I have tried both actually, and 8649
makes sense, but just to confirm. (I have set tcp_accept_channel as 8650)
2. In hadoop-metrics.properties, should I mention all my nodes as I have
done or only the local node? e.g, localhost:8649 everywhere?
I have configured gmetad.conf, and I am able to see metrics from all my
nodes in my ganglia UI. I just don't see any hadoop specific metrics.
Thanks,
Hari
On Mon, Nov 22, 2010 at 6:42 PM, Sagar Shukla <sagar_shukla@persistent.co.in
> wrote:
> Hi Hari,
> Not sure if you have already followed article -
> http://wiki.apache.org/hadoop/GangliaMetrics to configure Ganglia for
> Hadoop Monitoring. Also in the details given by you, I do not see any
> configuration settings done in gmetad config. You will need to define the
> hosts from whom metrics should be collected. Once this configuration, you
> should see the host in gmetad UI interface.
>
> Thanks,
> Sagar
>
> -----Original Message-----
> From: Hari Sreekumar [mailto:hsreekumar@clickable.com]
> Sent: Monday, November 22, 2010 6:25 PM
> To: common-user@hadoop.apache.org
> Subject: Using ganglia to monitor hadoop
>
> Hi,
>
> I have set up ganglia for my cluster, and it works fine. What are
> the changes I need to make to make ganglia show hadoop related parameters?
> My gmond/gmetad config is default except for one change in gmond.conf for
> all nodes:
>
> /* You can specify as many tcp_accept_channels as you like to share
> an xml description of the state of the cluster */
> tcp_accept_channel {
> port = 8650
> }
>
> I have changed the hadoop-metrics.properties file as follows:
>
> # Configuration of the "dfs" context for ganglia
> dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext
> dfs.period=10
> dfs.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649
>
>
> # Configuration of the "mapred" context for null
> #mapred.class=org.apache.hadoop.metrics.spi.NullContext
>
> # Configuration of the "mapred" context for file
> #mapred.class=org.apache.hadoop.metrics.file.FileContext
> #mapred.period=10
> #mapred.fileName=/tmp/mrmetrics.log
>
> # Configuration of the "mapred" context for ganglia
> mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext
> mapred.period=10
> mapred.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649
>
>
> # Configuration of the "jvm" context for null
> #jvm.class=org.apache.hadoop.metrics.spi.NullContext
>
> # Configuration of the "jvm" context for file
> #jvm.class=org.apache.hadoop.metrics.file.FileContext
> #jvm.period=10
> #jvm.fileName=/tmp/jvmmetrics.log
>
> # Configuration of the "jvm" context for ganglia
> jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext
> jvm.period=10
> jvm.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649
>
> rpc.class=org.apache.hadoop.metrics.ganglia.GangliaContext
> rpc.period=10
> rpc.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649
>
> hadoop1,2,3 and ejabber are my nodes.
>
> But I am unable to see any new metrics in ganglia's drop down menu. Is
> there
> something else I need to do as well? I have restarted gmond on all nodes.
>
> thanks,
> Hari
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is
> the property of Persistent Systems Ltd. It is intended only for the use of
> the individual or entity to which it is addressed. If you are not the
> intended recipient, you are not authorized to read, retain, copy, print,
> distribute or use this message. If you have received this communication in
> error, please notify the sender and delete all copies of this message.
> Persistent Systems Ltd. does not accept any liability for virus infected
> mails.
>
RE: Using ganglia to monitor hadoop
Posted by Sagar Shukla <sa...@persistent.co.in>.
Hi Hari,
Not sure if you have already followed article - http://wiki.apache.org/hadoop/GangliaMetrics to configure Ganglia for Hadoop Monitoring. Also in the details given by you, I do not see any configuration settings done in gmetad config. You will need to define the hosts from whom metrics should be collected. Once this configuration, you should see the host in gmetad UI interface.
Thanks,
Sagar
-----Original Message-----
From: Hari Sreekumar [mailto:hsreekumar@clickable.com]
Sent: Monday, November 22, 2010 6:25 PM
To: common-user@hadoop.apache.org
Subject: Using ganglia to monitor hadoop
Hi,
I have set up ganglia for my cluster, and it works fine. What are
the changes I need to make to make ganglia show hadoop related parameters?
My gmond/gmetad config is default except for one change in gmond.conf for
all nodes:
/* You can specify as many tcp_accept_channels as you like to share
an xml description of the state of the cluster */
tcp_accept_channel {
port = 8650
}
I have changed the hadoop-metrics.properties file as follows:
# Configuration of the "dfs" context for ganglia
dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext
dfs.period=10
dfs.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649
# Configuration of the "mapred" context for null
#mapred.class=org.apache.hadoop.metrics.spi.NullContext
# Configuration of the "mapred" context for file
#mapred.class=org.apache.hadoop.metrics.file.FileContext
#mapred.period=10
#mapred.fileName=/tmp/mrmetrics.log
# Configuration of the "mapred" context for ganglia
mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext
mapred.period=10
mapred.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649
# Configuration of the "jvm" context for null
#jvm.class=org.apache.hadoop.metrics.spi.NullContext
# Configuration of the "jvm" context for file
#jvm.class=org.apache.hadoop.metrics.file.FileContext
#jvm.period=10
#jvm.fileName=/tmp/jvmmetrics.log
# Configuration of the "jvm" context for ganglia
jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext
jvm.period=10
jvm.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649
rpc.class=org.apache.hadoop.metrics.ganglia.GangliaContext
rpc.period=10
rpc.servers=hadoop1:8649 hadoop2:8649 hadoop3:8649 ejabber:8649
hadoop1,2,3 and ejabber are my nodes.
But I am unable to see any new metrics in ganglia's drop down menu. Is there
something else I need to do as well? I have restarted gmond on all nodes.
thanks,
Hari
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.