You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Neo Anderson <ja...@yahoo.co.uk> on 2010/03/10 16:38:12 UTC
Fair scheduler fairness question
I am learning how fair scheduler manage the jobs to allow each job share resource over time; but don't know if my understanding is correct or not.
My scenario is that I have 3 data nodes and the cluster is configured using fair scheduler with three pools launched (e.g. A, B, C). Each pool is configured with '<maxRunningJobs>1</maxRunningJobs>.' Now the clients try to submit 4 jobs (e.g. submitjob()) to 3 differt pools. For instance,
the first job is submitted to pool A
the second job is submitted to pool B
the third job is submitted to pool B
the fourth job is submitted to pool C
So I expect that the first 3 jobs will occupy the free slots (the slots should be fool now.) Then the fourth job is submitted. But since the slots are full, and the fourth job should also have a slot executing its job; therefore, the third job will be terminated (or kill) so that the fourth job can be launched.
Is my scenario correct?
And if I am right, is there any key word searchable in the log to observe such activites (the job that is being killed e.g. the third job)?
Thanks for help.
I apprecaite any advice.
Re: Fair scheduler fairness question
Posted by Todd Lipcon <to...@cloudera.com>.
On Wed, Mar 10, 2010 at 9:18 AM, Allen Wittenauer
<aw...@linkedin.com>wrote:
>
>
>
> On 3/10/10 9:14 AM, "Neo Anderson" <ja...@yahoo.co.uk> wrote:
> > At the moment I use hadoop 0.20.2 and I can not find code that relates to
> > 'preempt' function; however, I read the jira MAPREDUCE-551 saying preempt
> > function is already been fixed at version 0.20.0.
>
> MR-551 says fixed in 0.21 at the top. Reading the text shows that patches
> are available if you want to patch your own build of 0.20.
>
>
If you'd rather not patch your own build of Hadoop, the fair scheduler
preemption feature is also available in CDH2:
http://archive.cloudera.com/cdh/2/hadoop-0.20.1+169.56.tar.gz
-Todd
--
Todd Lipcon
Software Engineer, Cloudera
Re: Fair scheduler fairness question
Posted by Allen Wittenauer <aw...@linkedin.com>.
On 3/10/10 9:14 AM, "Neo Anderson" <ja...@yahoo.co.uk> wrote:
> At the moment I use hadoop 0.20.2 and I can not find code that relates to
> 'preempt' function; however, I read the jira MAPREDUCE-551 saying preempt
> function is already been fixed at version 0.20.0.
MR-551 says fixed in 0.21 at the top. Reading the text shows that patches
are available if you want to patch your own build of 0.20.
Re: Fair scheduler fairness question
Posted by Neo Anderson <ja...@yahoo.co.uk>.
--- On Wed, 10/3/10, Allen Wittenauer <aw...@linkedin.com> wrote:
> From: Allen Wittenauer <aw...@linkedin.com>
> Subject: Re: Fair scheduler fairness question
> To: common-user@hadoop.apache.org
> Date: Wednesday, 10 March, 2010, 16:06
> On 3/10/10 7:38 AM, "Neo Anderson"
> <ja...@yahoo.co.uk>
> wrote:
>
> > I am learning how fair scheduler manage the jobs to
> allow each job share
> > resource over time; but don't know if my understanding
> is correct or not.
> >
> > My scenario is that I have 3 data nodes and the
> cluster is configured using
> > fair scheduler with three pools launched (e.g. A, B,
> C). Each pool is
> > configured with
> '<maxRunningJobs>1</maxRunningJobs>.' Now the
> clients try to
> > submit 4 jobs (e.g. submitjob()) to 3 differt pools.
> For instance,
> >
> > the first job is submitted to pool A
> > the second job is submitted to pool B
> > the third job is submitted to pool B
> > the fourth job is submitted to pool C
> >
> > So I expect that the first 3 jobs will occupy the free
> slots (the slots should
> > be fool now.) Then the fourth job is submitted. But
> since the slots are full,
> > and the fourth job should also have a slot executing
> its job; therefore, the
> > third job will be terminated (or kill) so that the
> fourth job can be launched.
> >
> > Is my scenario correct?
> > And if I am right, is there any key word searchable in
> the log to observe such
> > activites (the job that is being killed e.g. the third
> job)?
>
>
> A lot of it depends upon timing. If there is a long
> enough pause between
> job 1 and job 2, job 1 will take every slot available to
> it. As job 1's
> slots finish, job 2 and 4 would get those
> slots. As job 2 finishes, job 3
> will get its slots.
>
> Slots are only freed by force if the scheduler you are
> using has
> pre-emption. I think some versions of fair share may
> have it. Entire jobs
> are never killed.
>
>
>
>
At the moment I use hadoop 0.20.2 and I can not find code that relates to 'preempt' function; however, I read the jira MAPREDUCE-551 saying preempt function is already been fixed at version 0.20.0. Also, I can find some functons that relates to 'preemption' e.g. 'protected void preemptTasksIfNecessary()' in the patch. I am confused now - which function in version 0.20.2 (or 0.20.1) is used to preempt unnecessary tasks (so that slots can be freed for other tasks/ jobs to run)?
Thanks you for your help.
Re: Fair scheduler fairness question
Posted by Allen Wittenauer <aw...@linkedin.com>.
On 3/10/10 7:38 AM, "Neo Anderson" <ja...@yahoo.co.uk> wrote:
> I am learning how fair scheduler manage the jobs to allow each job share
> resource over time; but don't know if my understanding is correct or not.
>
> My scenario is that I have 3 data nodes and the cluster is configured using
> fair scheduler with three pools launched (e.g. A, B, C). Each pool is
> configured with '<maxRunningJobs>1</maxRunningJobs>.' Now the clients try to
> submit 4 jobs (e.g. submitjob()) to 3 differt pools. For instance,
>
> the first job is submitted to pool A
> the second job is submitted to pool B
> the third job is submitted to pool B
> the fourth job is submitted to pool C
>
> So I expect that the first 3 jobs will occupy the free slots (the slots should
> be fool now.) Then the fourth job is submitted. But since the slots are full,
> and the fourth job should also have a slot executing its job; therefore, the
> third job will be terminated (or kill) so that the fourth job can be launched.
>
> Is my scenario correct?
> And if I am right, is there any key word searchable in the log to observe such
> activites (the job that is being killed e.g. the third job)?
A lot of it depends upon timing. If there is a long enough pause between
job 1 and job 2, job 1 will take every slot available to it. As job 1's
slots finish, job 2 and 4 would get those slots. As job 2 finishes, job 3
will get its slots.
Slots are only freed by force if the scheduler you are using has
pre-emption. I think some versions of fair share may have it. Entire jobs
are never killed.