You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by ch huang <ju...@gmail.com> on 2013/12/02 13:08:24 UTC
how to prevent JAVA HEAP OOM happen in shuffle process in a MR job?
hi,maillist:
i recent get a problem,when i run MR job, it happened OOM in shuffle
process,the options about MR is default,not changed,which option should i
tuning? thanks
Re: how to prevent JAVA HEAP OOM happen in shuffle process in a MR job?
Posted by Azuryy Yu <az...@gmail.com>.
Hi Jay,
Can you point out a detail about 1.x that job tracker having an
ever-expanding hash map. I want fix it.
On Tue, Dec 3, 2013 at 3:08 AM, Jay Vyas <ja...@gmail.com> wrote:
> version is rewally important here..
>
> - If 1.x, then Where (NN , JT , TT ?)
> - if 2.x, then where? (AM, NM, ... ?) <-- probably less likely here, since
> the resources are ephemeral.
>
> I know that some older 1x versions had an issue with the jobtracker having
> an ever-expanding hashmap or something like that, so that if you ran 100s
> of jobs, you could get OOM erros on the JobTracker.
>
Re: how to prevent JAVA HEAP OOM happen in shuffle process in a MR job?
Posted by Azuryy Yu <az...@gmail.com>.
Hi Jay,
Can you point out a detail about 1.x that job tracker having an
ever-expanding hash map. I want fix it.
On Tue, Dec 3, 2013 at 3:08 AM, Jay Vyas <ja...@gmail.com> wrote:
> version is rewally important here..
>
> - If 1.x, then Where (NN , JT , TT ?)
> - if 2.x, then where? (AM, NM, ... ?) <-- probably less likely here, since
> the resources are ephemeral.
>
> I know that some older 1x versions had an issue with the jobtracker having
> an ever-expanding hashmap or something like that, so that if you ran 100s
> of jobs, you could get OOM erros on the JobTracker.
>
Re: how to prevent JAVA HEAP OOM happen in shuffle process in a MR job?
Posted by Azuryy Yu <az...@gmail.com>.
Hi Jay,
Can you point out a detail about 1.x that job tracker having an
ever-expanding hash map. I want fix it.
On Tue, Dec 3, 2013 at 3:08 AM, Jay Vyas <ja...@gmail.com> wrote:
> version is rewally important here..
>
> - If 1.x, then Where (NN , JT , TT ?)
> - if 2.x, then where? (AM, NM, ... ?) <-- probably less likely here, since
> the resources are ephemeral.
>
> I know that some older 1x versions had an issue with the jobtracker having
> an ever-expanding hashmap or something like that, so that if you ran 100s
> of jobs, you could get OOM erros on the JobTracker.
>
Re: how to prevent JAVA HEAP OOM happen in shuffle process in a MR job?
Posted by Azuryy Yu <az...@gmail.com>.
Hi Jay,
Can you point out a detail about 1.x that job tracker having an
ever-expanding hash map. I want fix it.
On Tue, Dec 3, 2013 at 3:08 AM, Jay Vyas <ja...@gmail.com> wrote:
> version is rewally important here..
>
> - If 1.x, then Where (NN , JT , TT ?)
> - if 2.x, then where? (AM, NM, ... ?) <-- probably less likely here, since
> the resources are ephemeral.
>
> I know that some older 1x versions had an issue with the jobtracker having
> an ever-expanding hashmap or something like that, so that if you ran 100s
> of jobs, you could get OOM erros on the JobTracker.
>
Re: how to prevent JAVA HEAP OOM happen in shuffle process in a MR job?
Posted by Jay Vyas <ja...@gmail.com>.
version is rewally important here..
- If 1.x, then Where (NN , JT , TT ?)
- if 2.x, then where? (AM, NM, ... ?) <-- probably less likely here, since
the resources are ephemeral.
I know that some older 1x versions had an issue with the jobtracker having
an ever-expanding hashmap or something like that, so that if you ran 100s
of jobs, you could get OOM erros on the JobTracker.
Re: how to prevent JAVA HEAP OOM happen in shuffle process in a MR job?
Posted by Jay Vyas <ja...@gmail.com>.
version is rewally important here..
- If 1.x, then Where (NN , JT , TT ?)
- if 2.x, then where? (AM, NM, ... ?) <-- probably less likely here, since
the resources are ephemeral.
I know that some older 1x versions had an issue with the jobtracker having
an ever-expanding hashmap or something like that, so that if you ran 100s
of jobs, you could get OOM erros on the JobTracker.
Re: how to prevent JAVA HEAP OOM happen in shuffle process in a MR job?
Posted by Jay Vyas <ja...@gmail.com>.
version is rewally important here..
- If 1.x, then Where (NN , JT , TT ?)
- if 2.x, then where? (AM, NM, ... ?) <-- probably less likely here, since
the resources are ephemeral.
I know that some older 1x versions had an issue with the jobtracker having
an ever-expanding hashmap or something like that, so that if you ran 100s
of jobs, you could get OOM erros on the JobTracker.
Re: how to prevent JAVA HEAP OOM happen in shuffle process in a MR job?
Posted by Jay Vyas <ja...@gmail.com>.
version is rewally important here..
- If 1.x, then Where (NN , JT , TT ?)
- if 2.x, then where? (AM, NM, ... ?) <-- probably less likely here, since
the resources are ephemeral.
I know that some older 1x versions had an issue with the jobtracker having
an ever-expanding hashmap or something like that, so that if you ran 100s
of jobs, you could get OOM erros on the JobTracker.