job priorities

[nitin reddy <ch...@gmail.com>]

Hi

I want to induce dynamic priority to the jobs being submitted like
rather than the default FIFO i want to set a variable that decides the
priority of it a situation may be even the job may be first in the
queue it might have less priority than the job that is behind in the
queue . just wanted to confirm if jobConf is the place where it should
be done.

Thanks
Nitin

Re: job priorities

[Sriram Ramachandrasekaran <sr...@gmail.com>]

Nitin,
You should read about the queues concept in hadoop job scheduler. Each
scheduler works with a certain type of Queue. The default scheduler uses a
FIFO queue. However, hadoop is quite flexible, in that, you can plug-in any
scheduler that's been contributed. Right now, capacity scheduler & fair
scheduler are the ones which are stable & used(afaik). There are related
config files like mapred-queues.xml which will allow to configure queue
related details. Below are the related links.
http://hadoop.apache.org/common/docs/r0.20.2/fair_scheduler.html
http://hadoop.apache.org/common/docs/r0.20.2/capacity_scheduler.html

Thanks,
Sriram

On Sun, Dec 12, 2010 at 11:31 PM, Harsh J <qw...@gmail.com> wrote:

> Hi,
>
> On Sun, Dec 12, 2010 at 11:21 PM, nitin reddy <ch...@gmail.com>
> wrote:
> > Hi
> >
> > I want to induce dynamic priority to the jobs being submitted like
> > rather than the default FIFO i want to set a variable that decides the
> > priority of it a situation may be even the job may be first in the
> > queue it might have less priority than the job that is behind in the
> > queue . just wanted to confirm if jobConf is the place where it should
> > be done.
>
> I'm not quite sure I got your query right, but if you are looking to:
>
> 1. Set (permanently) a job's priority in code:
> Use JobConf.setJobPriority() to set it while configuring your job's
> submission.
>
> 2. Change a queued/running job's priority dynamically:
> Use either `hadoop job -set-priority` or use JobClient to retrieve a
> RunningJob object for a JobID, and use RunningJob.setJobPriority().
>
> --
> Harsh J
> www.harshj.com
>

Re: job priorities

[Harsh J <qw...@gmail.com>]

Hi,

On Sun, Dec 12, 2010 at 11:21 PM, nitin reddy <ch...@gmail.com> wrote:
> Hi
>
> I want to induce dynamic priority to the jobs being submitted like
> rather than the default FIFO i want to set a variable that decides the
> priority of it a situation may be even the job may be first in the
> queue it might have less priority than the job that is behind in the
> queue . just wanted to confirm if jobConf is the place where it should
> be done.

I'm not quite sure I got your query right, but if you are looking to:

1. Set (permanently) a job's priority in code:
Use JobConf.setJobPriority() to set it while configuring your job's submission.

2. Change a queued/running job's priority dynamically:
Use either `hadoop job -set-priority` or use JobClient to retrieve a
RunningJob object for a JobID, and use RunningJob.setJobPriority().

-- 
Harsh J
www.harshj.com