Nutch hadoop and Torque integration

[rishi pathak <ma...@gmail.com>]

Hello,
           Sorry for cross posting. We have a compute cluster running Torque
resource manager and Maui scheduler.
Compute cluster is almost full but at times (early mornings, late night,
holidays), resources are available in pockets( 2-10 nodes for 2-5 hrs).
Our idea is setup nutch(hadoop) in a way to utilize these pockets i.e. an
automated  system wherein a long crawling job is broken down in to smaller
map/red jobs
. The system would be constantly monitoring the availability of resources
and would request, execute and finalize these smaller tasks using resource
manager interface. We had a look at HOD but to the extent of my knowledge
about it, it does not serve the purpose.

In a way it is too much to ask and may be a complete solution is not
available but any pointers/links are more than welcomed.

We are also looking at JobStream.py available at
http://wiki.apache.org/nutch/Automating_Fetches_with_Python

Thanks

-- 
Rishi Pathak
National PARAM Supercomputing Facility
C-DAC