[Hadoop Wiki] Update of "ManagementTools" by SteveLoughran

[Apache Wiki <wi...@apache.org>]

Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.

The "ManagementTools" page has been changed by SteveLoughran.
The comment on this change is: New page on cluster management tools.
http://wiki.apache.org/hadoop/ManagementTools

--------------------------------------------------

New page:
= Hadoop Cluster Management Tools =

On a big cluster you don't want to have your phone page you every time a node goes down. The only invididual machines you care about are the NameNode, the Secondary NameNode and the JobTracker. Worker nodes come and go. What matters there is the total cluster availability, the availability of the live data, and whether the rate of node failure is too high to get useful work done.

The other thing to be aware of is that the troublesome workers are not the dead ones; they are easy to detect; they don't report for duty. The troublesome ones are the nodes where the disk is playing up so badly that the system is really slow, so their work takes too long. Or their RAM isn't working properly so only 1GB of it appears there, and every job fails with memory problems. Or some strange motherboard/CPU/OS combination causes a machine to find race conditions in code where none surface elsewhere. That's what you need to identify: the troublemakers. Once found, you can set up Hadoop to blacklist nodes.

== Nagios ==

There is support for Nagios in Hadoop.

== Ganglia ==

There is support for Ganglia in Hadoop.

== JMX Support ==

Hadoop has JMX support, so with the right JMX bridge for your chosen management tools, it should be possible to keep an eye on Hadoop from your favorite management console.

=== JMX Bridging to Zenoss ===

Allen at LinkedIn says
"We've working on getting our stats into Zenoss via the JMX connector and
SNMP because Ganglia seems to have some fundamental issues (like grouping of
hosts is a *client* side config).  Note that Zenoss is available in both
open source and commercial forms.  We're using the commercial version, but
the open source version would probably be just as good.

But that aside:

We're taking the approach of grid health by watching and monitoring the
dead/live node count by scraping the NN and JT web pages.  We also do daily
fsck's, lsr's, and run a cut-down version of gridmix.

While monitoring individual nodes is useful in a pro-active sense, the
bigger your grid gets, the less important it becomes"