You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Clay McDonald <st...@bateswhite.com> on 2014/03/27 19:08:48 UTC
Maps stuck on Pending
Hi all, I have a job running with 1750 maps and 1 reduce and the status has been the same for the last two hours. Any thoughts?
Thanks, Clay
Re: Maps stuck on Pending
Posted by Dieter De Witte <dr...@gmail.com>.
There's is a big chance that your map output is being copied to your
reducer, this could take quite some time if you have a lot of data and
could be resolved by:
1) having more reducers
2) adjust the slowstart parameter so that the copying can start while the
map tasks are still running
Regards, Dieter
2014-03-27 20:42 GMT+01:00 Clay McDonald <st...@bateswhite.com>:
> Thanks Serge, looks like I need to at memory to my datanodes.
>
> Clay McDonald
> Cell: 202.560.4101
> Direct: 202.747.5962
>
> -----Original Message-----
> From: Serge Blazhievsky [mailto:hadoop.ca@gmail.com]
> Sent: Thursday, March 27, 2014 2:16 PM
> To: user@hadoop.apache.org
> Cc: <us...@hadoop.apache.org>
> Subject: Re: Maps stuck on Pending
>
> Next step would be to look in the logs under userlog directory for that job
>
> Sent from my iPhone
>
> > On Mar 27, 2014, at 11:08 AM, Clay McDonald <
> stuart.mcdonald@bateswhite.com> wrote:
> >
> > Hi all, I have a job running with 1750 maps and 1 reduce and the status
> has been the same for the last two hours. Any thoughts?
> >
> > Thanks, Clay
>
Re: Maps stuck on Pending
Posted by Dieter De Witte <dr...@gmail.com>.
There's is a big chance that your map output is being copied to your
reducer, this could take quite some time if you have a lot of data and
could be resolved by:
1) having more reducers
2) adjust the slowstart parameter so that the copying can start while the
map tasks are still running
Regards, Dieter
2014-03-27 20:42 GMT+01:00 Clay McDonald <st...@bateswhite.com>:
> Thanks Serge, looks like I need to at memory to my datanodes.
>
> Clay McDonald
> Cell: 202.560.4101
> Direct: 202.747.5962
>
> -----Original Message-----
> From: Serge Blazhievsky [mailto:hadoop.ca@gmail.com]
> Sent: Thursday, March 27, 2014 2:16 PM
> To: user@hadoop.apache.org
> Cc: <us...@hadoop.apache.org>
> Subject: Re: Maps stuck on Pending
>
> Next step would be to look in the logs under userlog directory for that job
>
> Sent from my iPhone
>
> > On Mar 27, 2014, at 11:08 AM, Clay McDonald <
> stuart.mcdonald@bateswhite.com> wrote:
> >
> > Hi all, I have a job running with 1750 maps and 1 reduce and the status
> has been the same for the last two hours. Any thoughts?
> >
> > Thanks, Clay
>
Re: Maps stuck on Pending
Posted by Dieter De Witte <dr...@gmail.com>.
There's is a big chance that your map output is being copied to your
reducer, this could take quite some time if you have a lot of data and
could be resolved by:
1) having more reducers
2) adjust the slowstart parameter so that the copying can start while the
map tasks are still running
Regards, Dieter
2014-03-27 20:42 GMT+01:00 Clay McDonald <st...@bateswhite.com>:
> Thanks Serge, looks like I need to at memory to my datanodes.
>
> Clay McDonald
> Cell: 202.560.4101
> Direct: 202.747.5962
>
> -----Original Message-----
> From: Serge Blazhievsky [mailto:hadoop.ca@gmail.com]
> Sent: Thursday, March 27, 2014 2:16 PM
> To: user@hadoop.apache.org
> Cc: <us...@hadoop.apache.org>
> Subject: Re: Maps stuck on Pending
>
> Next step would be to look in the logs under userlog directory for that job
>
> Sent from my iPhone
>
> > On Mar 27, 2014, at 11:08 AM, Clay McDonald <
> stuart.mcdonald@bateswhite.com> wrote:
> >
> > Hi all, I have a job running with 1750 maps and 1 reduce and the status
> has been the same for the last two hours. Any thoughts?
> >
> > Thanks, Clay
>
Re: Maps stuck on Pending
Posted by Dieter De Witte <dr...@gmail.com>.
There's is a big chance that your map output is being copied to your
reducer, this could take quite some time if you have a lot of data and
could be resolved by:
1) having more reducers
2) adjust the slowstart parameter so that the copying can start while the
map tasks are still running
Regards, Dieter
2014-03-27 20:42 GMT+01:00 Clay McDonald <st...@bateswhite.com>:
> Thanks Serge, looks like I need to at memory to my datanodes.
>
> Clay McDonald
> Cell: 202.560.4101
> Direct: 202.747.5962
>
> -----Original Message-----
> From: Serge Blazhievsky [mailto:hadoop.ca@gmail.com]
> Sent: Thursday, March 27, 2014 2:16 PM
> To: user@hadoop.apache.org
> Cc: <us...@hadoop.apache.org>
> Subject: Re: Maps stuck on Pending
>
> Next step would be to look in the logs under userlog directory for that job
>
> Sent from my iPhone
>
> > On Mar 27, 2014, at 11:08 AM, Clay McDonald <
> stuart.mcdonald@bateswhite.com> wrote:
> >
> > Hi all, I have a job running with 1750 maps and 1 reduce and the status
> has been the same for the last two hours. Any thoughts?
> >
> > Thanks, Clay
>
RE: Maps stuck on Pending
Posted by Clay McDonald <st...@bateswhite.com>.
Thanks Serge, looks like I need to at memory to my datanodes.
Clay McDonald
Cell: 202.560.4101
Direct: 202.747.5962
-----Original Message-----
From: Serge Blazhievsky [mailto:hadoop.ca@gmail.com]
Sent: Thursday, March 27, 2014 2:16 PM
To: user@hadoop.apache.org
Cc: <us...@hadoop.apache.org>
Subject: Re: Maps stuck on Pending
Next step would be to look in the logs under userlog directory for that job
Sent from my iPhone
> On Mar 27, 2014, at 11:08 AM, Clay McDonald <st...@bateswhite.com> wrote:
>
> Hi all, I have a job running with 1750 maps and 1 reduce and the status has been the same for the last two hours. Any thoughts?
>
> Thanks, Clay
RE: Maps stuck on Pending
Posted by Clay McDonald <st...@bateswhite.com>.
Thanks Serge, looks like I need to at memory to my datanodes.
Clay McDonald
Cell: 202.560.4101
Direct: 202.747.5962
-----Original Message-----
From: Serge Blazhievsky [mailto:hadoop.ca@gmail.com]
Sent: Thursday, March 27, 2014 2:16 PM
To: user@hadoop.apache.org
Cc: <us...@hadoop.apache.org>
Subject: Re: Maps stuck on Pending
Next step would be to look in the logs under userlog directory for that job
Sent from my iPhone
> On Mar 27, 2014, at 11:08 AM, Clay McDonald <st...@bateswhite.com> wrote:
>
> Hi all, I have a job running with 1750 maps and 1 reduce and the status has been the same for the last two hours. Any thoughts?
>
> Thanks, Clay
RE: Maps stuck on Pending
Posted by Clay McDonald <st...@bateswhite.com>.
Thanks Serge, looks like I need to at memory to my datanodes.
Clay McDonald
Cell: 202.560.4101
Direct: 202.747.5962
-----Original Message-----
From: Serge Blazhievsky [mailto:hadoop.ca@gmail.com]
Sent: Thursday, March 27, 2014 2:16 PM
To: user@hadoop.apache.org
Cc: <us...@hadoop.apache.org>
Subject: Re: Maps stuck on Pending
Next step would be to look in the logs under userlog directory for that job
Sent from my iPhone
> On Mar 27, 2014, at 11:08 AM, Clay McDonald <st...@bateswhite.com> wrote:
>
> Hi all, I have a job running with 1750 maps and 1 reduce and the status has been the same for the last two hours. Any thoughts?
>
> Thanks, Clay
RE: Maps stuck on Pending
Posted by Clay McDonald <st...@bateswhite.com>.
Thanks Serge, looks like I need to at memory to my datanodes.
Clay McDonald
Cell: 202.560.4101
Direct: 202.747.5962
-----Original Message-----
From: Serge Blazhievsky [mailto:hadoop.ca@gmail.com]
Sent: Thursday, March 27, 2014 2:16 PM
To: user@hadoop.apache.org
Cc: <us...@hadoop.apache.org>
Subject: Re: Maps stuck on Pending
Next step would be to look in the logs under userlog directory for that job
Sent from my iPhone
> On Mar 27, 2014, at 11:08 AM, Clay McDonald <st...@bateswhite.com> wrote:
>
> Hi all, I have a job running with 1750 maps and 1 reduce and the status has been the same for the last two hours. Any thoughts?
>
> Thanks, Clay
Re: Maps stuck on Pending
Posted by Serge Blazhievsky <ha...@gmail.com>.
Next step would be to look in the logs under userlog directory for that job
Sent from my iPhone
> On Mar 27, 2014, at 11:08 AM, Clay McDonald <st...@bateswhite.com> wrote:
>
> Hi all, I have a job running with 1750 maps and 1 reduce and the status has been the same for the last two hours. Any thoughts?
>
> Thanks, Clay
Re: Maps stuck on Pending
Posted by Serge Blazhievsky <ha...@gmail.com>.
Next step would be to look in the logs under userlog directory for that job
Sent from my iPhone
> On Mar 27, 2014, at 11:08 AM, Clay McDonald <st...@bateswhite.com> wrote:
>
> Hi all, I have a job running with 1750 maps and 1 reduce and the status has been the same for the last two hours. Any thoughts?
>
> Thanks, Clay
Re: Maps stuck on Pending
Posted by Serge Blazhievsky <ha...@gmail.com>.
Next step would be to look in the logs under userlog directory for that job
Sent from my iPhone
> On Mar 27, 2014, at 11:08 AM, Clay McDonald <st...@bateswhite.com> wrote:
>
> Hi all, I have a job running with 1750 maps and 1 reduce and the status has been the same for the last two hours. Any thoughts?
>
> Thanks, Clay
Re: Maps stuck on Pending
Posted by Serge Blazhievsky <ha...@gmail.com>.
Next step would be to look in the logs under userlog directory for that job
Sent from my iPhone
> On Mar 27, 2014, at 11:08 AM, Clay McDonald <st...@bateswhite.com> wrote:
>
> Hi all, I have a job running with 1750 maps and 1 reduce and the status has been the same for the last two hours. Any thoughts?
>
> Thanks, Clay