something fishy about a Metrics+Ganglia integration

[Mitch Gitman <mg...@gmail.com>]

I've inherited an integration of Cassandra's Codahale Metrics reporting
with Ganglia that looks sensible enough on the Metrics side. The
metrics-reporter-config.yaml points to a gmond.conf on the node. Excerpt:
ganglia:
  -
    period: 60
    timeunit: 'SECONDS'
    gmondConf: '/etc/ganglia/gmond.conf'

Now if I look at the gmond.conf on one of the nodes in the Cassandra
cluster, I see that there's a UDP entry for EVERY node in the cluster, plus
OpsCenter:

udp_send_channel {
  host = sandbox-cas00
  port = 8649
  ttl = 1
}

udp_send_channel {
  host = sandbox-cas01
  port = 8649
  ttl = 1
}

udp_send_channel {
  host = sandbox-cas02
  port = 8649
  ttl = 1
}

udp_send_channel {
  host = sandbox-cas03
  port = 8649
  ttl = 1
}

udp_send_channel {
  host = sandbox-cas04
  port = 8649
  ttl = 1
}

udp_send_channel {
  host = sandbox-opscenter
  port = 8649
  ttl = 1
}

What strikes me funny about this configuration is that every node in the
cluster is referenced. I was expecting that each node's gmond.conf would
only define a udp_send_channel for itself. This is fine now with a
five-node cluster, and we have enough deployment automation in place to
mitigate the fact that every time we add a Cassandra node every other
Cassandra node's gmond.conf would have to be updated. But is it really the
intention that every Ganglia host needs to be aware of every other host? I
was expecting that it would only be the Ganglia server these hosts are
reporting to that would need to be aware of the full membership of the
cluster.

OK, I realize at this point I'm asking more of a Ganglia question than a
Cassandra or Metrics question. Apparently, we're using Ganglia in unicast
mode rather than multicast.