state of HOD

[Stijn De Weirdt <st...@ugent.be>]

hi all,

i am a system administrator/user support person/... for the HPC team at 
Ghent University (Ghent, Flanders, Belgium).

recently we have been asked to look into support for hadoop. for the 
moment we are holding off on a dedicated cluster (esp dedicated hdfs setup).

but as all our systems are torque/pbs based, we looked into HOD to help 
out our users.
we have started from the HOD code that was part of the hadoop 1.0.0 
release (in the contrib part).
at first it was not working, but we have been patching and cleaning up 
the code for a a few weeks and now have a version that works for us (we 
had to add some features besides fixing a few things).
it looks sufficient for now, although we will add some more features 
soon to get the users started.


my question is the following: what is the state of HOD atm? is it still 
maintained/supported? are there forks somewhere that have more 
up-to-date code?
what we are now missing most is the documentation (eg 
http://hadoop.apache.org/common/docs/r0.16.4/hod.html) so we can update 
this with our extra features. is the source available somewhere?

i could contribute back all patches, but a few of them are identation 
fixes (to use 4 space indentation throughout the code) and other 
cosmetic changes, so this messes up patches a lot.
i have also shuffled a bit with the options (rename and/or move to other 
sections) so no 100% backwards compatibility with the current HOD code.

current main improvements:
- works with python 2.5 and up (we have been testing with 2.7.2)
- set options through environment variables
- better default values (we can now run with empty hodrc file)
- support for mail and nodes:ppn for pbs
- no deprecation warnings from hadoop (nearly finished)
- host-mask to bind xrs addr on non-default ip (in case you have 
non-standard network on the compute nodes)
- more debug statements
- gradual code cleanup (using pylint)

on the todo list:
- further tuning of hadoop parameters (i'm not a hadoop user myself, so 
this will take some time)
- 0.23.X support



many thanks,

stijn