You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Du Lam <de...@gmail.com> on 2014/05/20 08:53:43 UTC
question on yarn and fairscheduler
Hi
some questions on yarn+fairscheduler:
1. is the maxMaps and maxReduces in allocations.xml actually work? notice
it is not working in my setup.
2. my job always fail with diagnostics such as: Task
task_1400033851458_4824_m_000006 failed 4 times .
is it possible that this is due to preempted too many times? or any other
issue. At the same job, there are also tasks get killed with note: Attmpt
state missing from History : marked as KILLED
any help would be appreciated. Thanks.
Re: question on yarn and fairscheduler
Posted by Du Lam <de...@gmail.com>.
Thanks for the feedback on the yarn.
regarding preemption is killed or failed, the Web UI really place the task
preemption as 'FAILED' and placed in the 'FAILED' column, instead of KILLED.
On Tue, May 20, 2014 at 3:32 PM, Harsh J <ha...@cloudera.com> wrote:
> Hello,
>
> YARN's FairScheduler does not have maxMaps/maxReducers fields like its
> MR1 counterpart did, given YARN's generic Container based
> architecture. Please see
>
> http://archive.cloudera.com/cdh5/cdh/5/hadoop/hadoop-yarn/hadoop-yarn-site/FairScheduler.html#Configuration
> ,
> sub-section "Allocation file format" for the actual configurable
> elements in it.
>
> Preemption marks the tasks as killed, and not failed, so they do not
> count towards your job failure thresholds. If your task attempts are
> truly failing, their task attempt logs (syslog, stderr, stdout) will
> have the reason they failed.
>
> On Tue, May 20, 2014 at 12:23 PM, Du Lam <de...@gmail.com> wrote:
> > Hi
> > some questions on yarn+fairscheduler:
> > 1. is the maxMaps and maxReduces in allocations.xml actually work?
> notice it
> > is not working in my setup.
> > 2. my job always fail with diagnostics such as: Task
> > task_1400033851458_4824_m_000006 failed 4 times .
> > is it possible that this is due to preempted too many times? or any
> other
> > issue. At the same job, there are also tasks get killed with note: Attmpt
> > state missing from History : marked as KILLED
> >
> > any help would be appreciated. Thanks.
>
>
>
> --
> Harsh J
>
Re: question on yarn and fairscheduler
Posted by Du Lam <de...@gmail.com>.
Thanks for the feedback on the yarn.
regarding preemption is killed or failed, the Web UI really place the task
preemption as 'FAILED' and placed in the 'FAILED' column, instead of KILLED.
On Tue, May 20, 2014 at 3:32 PM, Harsh J <ha...@cloudera.com> wrote:
> Hello,
>
> YARN's FairScheduler does not have maxMaps/maxReducers fields like its
> MR1 counterpart did, given YARN's generic Container based
> architecture. Please see
>
> http://archive.cloudera.com/cdh5/cdh/5/hadoop/hadoop-yarn/hadoop-yarn-site/FairScheduler.html#Configuration
> ,
> sub-section "Allocation file format" for the actual configurable
> elements in it.
>
> Preemption marks the tasks as killed, and not failed, so they do not
> count towards your job failure thresholds. If your task attempts are
> truly failing, their task attempt logs (syslog, stderr, stdout) will
> have the reason they failed.
>
> On Tue, May 20, 2014 at 12:23 PM, Du Lam <de...@gmail.com> wrote:
> > Hi
> > some questions on yarn+fairscheduler:
> > 1. is the maxMaps and maxReduces in allocations.xml actually work?
> notice it
> > is not working in my setup.
> > 2. my job always fail with diagnostics such as: Task
> > task_1400033851458_4824_m_000006 failed 4 times .
> > is it possible that this is due to preempted too many times? or any
> other
> > issue. At the same job, there are also tasks get killed with note: Attmpt
> > state missing from History : marked as KILLED
> >
> > any help would be appreciated. Thanks.
>
>
>
> --
> Harsh J
>
Re: question on yarn and fairscheduler
Posted by Du Lam <de...@gmail.com>.
Thanks for the feedback on the yarn.
regarding preemption is killed or failed, the Web UI really place the task
preemption as 'FAILED' and placed in the 'FAILED' column, instead of KILLED.
On Tue, May 20, 2014 at 3:32 PM, Harsh J <ha...@cloudera.com> wrote:
> Hello,
>
> YARN's FairScheduler does not have maxMaps/maxReducers fields like its
> MR1 counterpart did, given YARN's generic Container based
> architecture. Please see
>
> http://archive.cloudera.com/cdh5/cdh/5/hadoop/hadoop-yarn/hadoop-yarn-site/FairScheduler.html#Configuration
> ,
> sub-section "Allocation file format" for the actual configurable
> elements in it.
>
> Preemption marks the tasks as killed, and not failed, so they do not
> count towards your job failure thresholds. If your task attempts are
> truly failing, their task attempt logs (syslog, stderr, stdout) will
> have the reason they failed.
>
> On Tue, May 20, 2014 at 12:23 PM, Du Lam <de...@gmail.com> wrote:
> > Hi
> > some questions on yarn+fairscheduler:
> > 1. is the maxMaps and maxReduces in allocations.xml actually work?
> notice it
> > is not working in my setup.
> > 2. my job always fail with diagnostics such as: Task
> > task_1400033851458_4824_m_000006 failed 4 times .
> > is it possible that this is due to preempted too many times? or any
> other
> > issue. At the same job, there are also tasks get killed with note: Attmpt
> > state missing from History : marked as KILLED
> >
> > any help would be appreciated. Thanks.
>
>
>
> --
> Harsh J
>
Re: question on yarn and fairscheduler
Posted by Du Lam <de...@gmail.com>.
Thanks for the feedback on the yarn.
regarding preemption is killed or failed, the Web UI really place the task
preemption as 'FAILED' and placed in the 'FAILED' column, instead of KILLED.
On Tue, May 20, 2014 at 3:32 PM, Harsh J <ha...@cloudera.com> wrote:
> Hello,
>
> YARN's FairScheduler does not have maxMaps/maxReducers fields like its
> MR1 counterpart did, given YARN's generic Container based
> architecture. Please see
>
> http://archive.cloudera.com/cdh5/cdh/5/hadoop/hadoop-yarn/hadoop-yarn-site/FairScheduler.html#Configuration
> ,
> sub-section "Allocation file format" for the actual configurable
> elements in it.
>
> Preemption marks the tasks as killed, and not failed, so they do not
> count towards your job failure thresholds. If your task attempts are
> truly failing, their task attempt logs (syslog, stderr, stdout) will
> have the reason they failed.
>
> On Tue, May 20, 2014 at 12:23 PM, Du Lam <de...@gmail.com> wrote:
> > Hi
> > some questions on yarn+fairscheduler:
> > 1. is the maxMaps and maxReduces in allocations.xml actually work?
> notice it
> > is not working in my setup.
> > 2. my job always fail with diagnostics such as: Task
> > task_1400033851458_4824_m_000006 failed 4 times .
> > is it possible that this is due to preempted too many times? or any
> other
> > issue. At the same job, there are also tasks get killed with note: Attmpt
> > state missing from History : marked as KILLED
> >
> > any help would be appreciated. Thanks.
>
>
>
> --
> Harsh J
>
Re: question on yarn and fairscheduler
Posted by Harsh J <ha...@cloudera.com>.
Hello,
YARN's FairScheduler does not have maxMaps/maxReducers fields like its
MR1 counterpart did, given YARN's generic Container based
architecture. Please see
http://archive.cloudera.com/cdh5/cdh/5/hadoop/hadoop-yarn/hadoop-yarn-site/FairScheduler.html#Configuration,
sub-section "Allocation file format" for the actual configurable
elements in it.
Preemption marks the tasks as killed, and not failed, so they do not
count towards your job failure thresholds. If your task attempts are
truly failing, their task attempt logs (syslog, stderr, stdout) will
have the reason they failed.
On Tue, May 20, 2014 at 12:23 PM, Du Lam <de...@gmail.com> wrote:
> Hi
> some questions on yarn+fairscheduler:
> 1. is the maxMaps and maxReduces in allocations.xml actually work? notice it
> is not working in my setup.
> 2. my job always fail with diagnostics such as: Task
> task_1400033851458_4824_m_000006 failed 4 times .
> is it possible that this is due to preempted too many times? or any other
> issue. At the same job, there are also tasks get killed with note: Attmpt
> state missing from History : marked as KILLED
>
> any help would be appreciated. Thanks.
--
Harsh J
Re: question on yarn and fairscheduler
Posted by Harsh J <ha...@cloudera.com>.
Hello,
YARN's FairScheduler does not have maxMaps/maxReducers fields like its
MR1 counterpart did, given YARN's generic Container based
architecture. Please see
http://archive.cloudera.com/cdh5/cdh/5/hadoop/hadoop-yarn/hadoop-yarn-site/FairScheduler.html#Configuration,
sub-section "Allocation file format" for the actual configurable
elements in it.
Preemption marks the tasks as killed, and not failed, so they do not
count towards your job failure thresholds. If your task attempts are
truly failing, their task attempt logs (syslog, stderr, stdout) will
have the reason they failed.
On Tue, May 20, 2014 at 12:23 PM, Du Lam <de...@gmail.com> wrote:
> Hi
> some questions on yarn+fairscheduler:
> 1. is the maxMaps and maxReduces in allocations.xml actually work? notice it
> is not working in my setup.
> 2. my job always fail with diagnostics such as: Task
> task_1400033851458_4824_m_000006 failed 4 times .
> is it possible that this is due to preempted too many times? or any other
> issue. At the same job, there are also tasks get killed with note: Attmpt
> state missing from History : marked as KILLED
>
> any help would be appreciated. Thanks.
--
Harsh J
Re: question on yarn and fairscheduler
Posted by Harsh J <ha...@cloudera.com>.
Hello,
YARN's FairScheduler does not have maxMaps/maxReducers fields like its
MR1 counterpart did, given YARN's generic Container based
architecture. Please see
http://archive.cloudera.com/cdh5/cdh/5/hadoop/hadoop-yarn/hadoop-yarn-site/FairScheduler.html#Configuration,
sub-section "Allocation file format" for the actual configurable
elements in it.
Preemption marks the tasks as killed, and not failed, so they do not
count towards your job failure thresholds. If your task attempts are
truly failing, their task attempt logs (syslog, stderr, stdout) will
have the reason they failed.
On Tue, May 20, 2014 at 12:23 PM, Du Lam <de...@gmail.com> wrote:
> Hi
> some questions on yarn+fairscheduler:
> 1. is the maxMaps and maxReduces in allocations.xml actually work? notice it
> is not working in my setup.
> 2. my job always fail with diagnostics such as: Task
> task_1400033851458_4824_m_000006 failed 4 times .
> is it possible that this is due to preempted too many times? or any other
> issue. At the same job, there are also tasks get killed with note: Attmpt
> state missing from History : marked as KILLED
>
> any help would be appreciated. Thanks.
--
Harsh J
Re: question on yarn and fairscheduler
Posted by Harsh J <ha...@cloudera.com>.
Hello,
YARN's FairScheduler does not have maxMaps/maxReducers fields like its
MR1 counterpart did, given YARN's generic Container based
architecture. Please see
http://archive.cloudera.com/cdh5/cdh/5/hadoop/hadoop-yarn/hadoop-yarn-site/FairScheduler.html#Configuration,
sub-section "Allocation file format" for the actual configurable
elements in it.
Preemption marks the tasks as killed, and not failed, so they do not
count towards your job failure thresholds. If your task attempts are
truly failing, their task attempt logs (syslog, stderr, stdout) will
have the reason they failed.
On Tue, May 20, 2014 at 12:23 PM, Du Lam <de...@gmail.com> wrote:
> Hi
> some questions on yarn+fairscheduler:
> 1. is the maxMaps and maxReduces in allocations.xml actually work? notice it
> is not working in my setup.
> 2. my job always fail with diagnostics such as: Task
> task_1400033851458_4824_m_000006 failed 4 times .
> is it possible that this is due to preempted too many times? or any other
> issue. At the same job, there are also tasks get killed with note: Attmpt
> state missing from History : marked as KILLED
>
> any help would be appreciated. Thanks.
--
Harsh J