Hadoop idioms for reporting cluster and counter stats.

[Jay Vyas <ja...@gmail.com>]

Hi guys : I want to start automating the output of  counter stats, cluster
size, etc... at the end of the main map reduce jobs which we run.  Is there
a simple way to do this ?

Here is my current thought :

1) Run all jobs from a driver class (we already do this).

2) At the end of each job, intercept the global counters and write them out
to a text file.  This would
presumably be on the local fs.

3) Export the local filesystem.

4) Maybe the NameNode also has access to such data , maybe via an API
(clearly, the hadoop web ui gets this
data from somewhere, re in the "cluster summary" header..


-- 
Jay Vyas
MMSB/UCHC