Capacity scheduler for dividing cluster resousrces

[Amit Sela <am...@infolinks.com>]

Hi everyone,

I'm running Hadoop 1.0.4 on a modest cluster (~20 machines) and I would
like to divide my cluster resources by job's process time.
The jobs running on the cluster can be divided as follows:

1. Very short jobs: less then 1 minute.
2. Normal jobs: 2-3 minutes up to an hour or two.
3. Very long jobs: days of processing. (still not active and the reason for
my inquiries here).

I was thinking of using the Capacity Scheduler and divide the cluster
resources such that the long jobs can run without disturbing the other jobs
and the very short jobs won't be delayed.

>From what I understand, if a very long job is running and the cluster
resources are free, it will use them all (unless queue is upper bound) but
once a job from another queue starts, will it claim it's resources back
even though the very long job is still running ? or the lack of preemption
will prevent that ?

Any other advise about using the CapacityScheduler in that use case ?

Thanks,

Amit.