Load Balancing?

[Tim Patton <tp...@dealcatcher.com>]

Sorry to post this on the Dev list, but the user list doesn't seem to get
any traffic.  I've been going through the code trying to figure out how
Hadoop would load balance among machines.  Specifically, if I had two types
of tasks, one high CPU and one low CPU, how can I make sure machines aren't
getting too bogged down by being assigned too many high CPU  tasks or aren't
sitting idle when they could be running more low CPU tasks?  I suppose the
same question could be asked about high and low RAM usage as well.   From
looking at the Nutch source code, it appears it fetches/processes first,
then indexes, when it could be done in parallel.  Is this why?

 

Tim