hadoop and ganglia without UDP multicast

[Mark McCorkle <ma...@medialets.com>]

Since I couldn't get my hosted network to work with UDP multicasting, I've
configured my ganglia boxes to talk tcp to one another.  Now that I've done
that, I don't know how to get hadoop to publish data to my ganglia.  I'm
running Hadoop 0.20 and gmond 3.07.  Below is the important part of the
config on one of my data nodes, and though I've changed the IPs, the box is
the first of my datanodes, which the others report to, its IP is 192.168.1.222
and the IP for the gmond/gmetad server is 192.168.1.72.  This works for all
other ganglia metrics (I've got a few, non hadoop things manually placed in
via gmetric), but I don't see any of the new hadoop ganglia metrics:

cluster {
  name = "hadoop"
  owner = "unspecified"
  latlong = "unspecified"
  url = "unspecified"
}

udp_send_channel {
  host = 192.168.1.222
  port = 8649
  ttl = 1
}

udp_recv_channel {
  bind = 192.168.1.222
  port = 8649
}

tcp_accept_channel {
  port = 8649
  acl {
    default = "deny"
    access {
        ip = 192.168.1.72 /* the ip of Monitoring Server */
        mask = 32
        action = "allow"
    }
    access {
        ip = 192.168.1.222
        mask = 32
        action = "allow"
    }
 }
}

And here is what I put into my hadoop setup
(in /usr/lib/hadoop-0.20/conf/hadoop-metrics.properties):

dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext
dfs.period=10
dfs.servers=localhost:8649
mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext
mapred.period=10
mapred.servers=localhost:8649
jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext
jvm.period=10
jvm.servers=localhost:8649
rpc.class=org.apache.hadoop.metrics.ganglia.GangliaContext
rpc.period=10
rpc.servers=localhost:8649

There have been at least 2 restarts of all of the parts of my hadoop
(datanodes, namenode, and secondary name node) since we've put this config
in place.

Any help would be greatly appreciated.

::Mark McCorkle

Re: hadoop and ganglia without UDP multicast

[Eric Fiala <er...@fiala.ca>]

Mark,
You might want to try changing your [dfs|mapred|jvm|rpc].servers in
hadoop-metrics.properties to point to your monitoring IP address (
192.168.1.72?) rather than localhost.
If you are relaying each node from local gmond than try to use the IP
address to which gmond is bound (netstat -an | grep 8649) in this same
config file.
Both should work.

EF

On 26 November 2010 17:15, Mark McCorkle <ma...@medialets.com>wrote:

> Since I couldn't get my hosted network to work with UDP multicasting, I've
> configured my ganglia boxes to talk tcp to one another.  Now that I've done
> that, I don't know how to get hadoop to publish data to my ganglia.  I'm
> running Hadoop 0.20 and gmond 3.07.  Below is the important part of the
> config on one of my data nodes, and though I've changed the IPs, the box is
> the first of my datanodes, which the others report to, its IP is
> 192.168.1.222
> and the IP for the gmond/gmetad server is 192.168.1.72.  This works for all
> other ganglia metrics (I've got a few, non hadoop things manually placed in
> via gmetric), but I don't see any of the new hadoop ganglia metrics:
>
> cluster {
>  name = "hadoop"
>  owner = "unspecified"
>  latlong = "unspecified"
>  url = "unspecified"
> }
>
> udp_send_channel {
>  host = 192.168.1.222
>  port = 8649
>  ttl = 1
> }
>
> udp_recv_channel {
>  bind = 192.168.1.222
>  port = 8649
> }
>
> tcp_accept_channel {
>  port = 8649
>  acl {
>    default = "deny"
>    access {
>        ip = 192.168.1.72 /* the ip of Monitoring Server */
>        mask = 32
>        action = "allow"
>    }
>    access {
>        ip = 192.168.1.222
>        mask = 32
>        action = "allow"
>    }
>  }
> }
>
> And here is what I put into my hadoop setup
> (in /usr/lib/hadoop-0.20/conf/hadoop-metrics.properties):
>
> dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext
> dfs.period=10
> dfs.servers=localhost:8649
> mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext
> mapred.period=10
> mapred.servers=localhost:8649
> jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext
> jvm.period=10
> jvm.servers=localhost:8649
> rpc.class=org.apache.hadoop.metrics.ganglia.GangliaContext
> rpc.period=10
> rpc.servers=localhost:8649
>
> There have been at least 2 restarts of all of the parts of my hadoop
> (datanodes, namenode, and secondary name node) since we've put this config
> in place.
>
> Any help would be greatly appreciated.
>
> ::Mark McCorkle
>