You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-user@hadoop.apache.org by Randy Fox <rf...@connexity.com> on 2016/01/23 02:49:13 UTC

NodeManager High CPU due to high GC

Hi,

We just upgraded to using Yarn on Hadoop 2.6.0 – CDH5.4.5
We are running a large job – 200K mappers, 100K reducers and we can’t get through the shuffle phase.  The node managers are 800% cpu and high GC.  The reducers get socket timouts after 1.5 hours of running and only getting a few percent of the data from the mappers.  This job took about 30 hours total 12 in mappers on MRv1 with no issues.

I have looked for configs that might help or issues filed and anyone that has seen this and I have come up with nothing.
Anyone have ideas on things to try or explain why the node managers are in GC hell and why the data is just not flowing from mappers to reducers?

Thanks in advanced,

Randy

Re: NodeManager High CPU due to high GC

Posted by sudhakara st <su...@gmail.com>.

Hello Randy,
It is too many mappers and reducers(200K mappers, 100K reducers) for any
cluster, it has indirect effects.  It seems reducer not able
get enough memory to perform processing or not able to reducer container.
Check  values for these two parameters
*yarn.nodemanager.resource.memory-mb and
yarn.nodemanager.resource.cpu-vcores,*
 It could be better if look into
http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas

How many nodes cluster do you have ?

Regard,
sudhakara

On Tue, Jan 26, 2016 at 9:55 PM, Randy Fox <rf...@connexity.com> wrote:

> What configs control the shuffle phase?
>
> From: Randy Fox
> Date: Saturday, January 23, 2016 at 9:53 AM
> To: Daniel Haviv
> Cc: "user@hadoop.apache.org"
> Subject: Re: NodeManager High CPU due to high GC
>
> 24 virtual cores and we allocated 22 for Yarn
>
> From: Daniel Haviv
> Date: Saturday, January 23, 2016 at 4:00 AM
> To: Randy Fox
> Cc: "user@hadoop.apache.org"
> Subject: Re: NodeManager High CPU due to high GC
>
> Hi Randy,
> How much cores do you have on your machines and how much did you allocate
> for Yarn?
>
> Daniel
>
> On Saturday, 23 January 2016, Randy Fox <rf...@connexity.com> wrote:
>
>> Hi,
>>
>> We just upgraded to using Yarn on Hadoop 2.6.0 – CDH5.4.5
>> We are running a large job – 200K mappers, 100K reducers and we can’t get
>> through the shuffle phase.  The node managers are 800% cpu and high GC.
>> The reducers get socket timouts after 1.5 hours of running and only getting
>> a few percent of the data from the mappers.  This job took about 30 hours
>> total 12 in mappers on MRv1 with no issues.
>>
>> I have looked for configs that might help or issues filed and anyone that
>> has seen this and I have come up with nothing.
>> Anyone have ideas on things to try or explain why the node managers are
>> in GC hell and why the data is just not flowing from mappers to reducers?
>>
>> Thanks in advanced,
>>
>> Randy
>>
>


-- 

Regards,
...sudhakara

Re: NodeManager High CPU due to high GC

Posted by sudhakara st <su...@gmail.com>.

Hello Randy,
It is too many mappers and reducers(200K mappers, 100K reducers) for any
cluster, it has indirect effects.  It seems reducer not able
get enough memory to perform processing or not able to reducer container.
Check  values for these two parameters
*yarn.nodemanager.resource.memory-mb and
yarn.nodemanager.resource.cpu-vcores,*
 It could be better if look into
http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas

How many nodes cluster do you have ?

Regard,
sudhakara

On Tue, Jan 26, 2016 at 9:55 PM, Randy Fox <rf...@connexity.com> wrote:

> What configs control the shuffle phase?
>
> From: Randy Fox
> Date: Saturday, January 23, 2016 at 9:53 AM
> To: Daniel Haviv
> Cc: "user@hadoop.apache.org"
> Subject: Re: NodeManager High CPU due to high GC
>
> 24 virtual cores and we allocated 22 for Yarn
>
> From: Daniel Haviv
> Date: Saturday, January 23, 2016 at 4:00 AM
> To: Randy Fox
> Cc: "user@hadoop.apache.org"
> Subject: Re: NodeManager High CPU due to high GC
>
> Hi Randy,
> How much cores do you have on your machines and how much did you allocate
> for Yarn?
>
> Daniel
>
> On Saturday, 23 January 2016, Randy Fox <rf...@connexity.com> wrote:
>
>> Hi,
>>
>> We just upgraded to using Yarn on Hadoop 2.6.0 – CDH5.4.5
>> We are running a large job – 200K mappers, 100K reducers and we can’t get
>> through the shuffle phase.  The node managers are 800% cpu and high GC.
>> The reducers get socket timouts after 1.5 hours of running and only getting
>> a few percent of the data from the mappers.  This job took about 30 hours
>> total 12 in mappers on MRv1 with no issues.
>>
>> I have looked for configs that might help or issues filed and anyone that
>> has seen this and I have come up with nothing.
>> Anyone have ideas on things to try or explain why the node managers are
>> in GC hell and why the data is just not flowing from mappers to reducers?
>>
>> Thanks in advanced,
>>
>> Randy
>>
>


-- 

Regards,
...sudhakara

Re: NodeManager High CPU due to high GC

Posted by sudhakara st <su...@gmail.com>.

Hello Randy,
It is too many mappers and reducers(200K mappers, 100K reducers) for any
cluster, it has indirect effects.  It seems reducer not able
get enough memory to perform processing or not able to reducer container.
Check  values for these two parameters
*yarn.nodemanager.resource.memory-mb and
yarn.nodemanager.resource.cpu-vcores,*
 It could be better if look into
http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas

How many nodes cluster do you have ?

Regard,
sudhakara

On Tue, Jan 26, 2016 at 9:55 PM, Randy Fox <rf...@connexity.com> wrote:

> What configs control the shuffle phase?
>
> From: Randy Fox
> Date: Saturday, January 23, 2016 at 9:53 AM
> To: Daniel Haviv
> Cc: "user@hadoop.apache.org"
> Subject: Re: NodeManager High CPU due to high GC
>
> 24 virtual cores and we allocated 22 for Yarn
>
> From: Daniel Haviv
> Date: Saturday, January 23, 2016 at 4:00 AM
> To: Randy Fox
> Cc: "user@hadoop.apache.org"
> Subject: Re: NodeManager High CPU due to high GC
>
> Hi Randy,
> How much cores do you have on your machines and how much did you allocate
> for Yarn?
>
> Daniel
>
> On Saturday, 23 January 2016, Randy Fox <rf...@connexity.com> wrote:
>
>> Hi,
>>
>> We just upgraded to using Yarn on Hadoop 2.6.0 – CDH5.4.5
>> We are running a large job – 200K mappers, 100K reducers and we can’t get
>> through the shuffle phase.  The node managers are 800% cpu and high GC.
>> The reducers get socket timouts after 1.5 hours of running and only getting
>> a few percent of the data from the mappers.  This job took about 30 hours
>> total 12 in mappers on MRv1 with no issues.
>>
>> I have looked for configs that might help or issues filed and anyone that
>> has seen this and I have come up with nothing.
>> Anyone have ideas on things to try or explain why the node managers are
>> in GC hell and why the data is just not flowing from mappers to reducers?
>>
>> Thanks in advanced,
>>
>> Randy
>>
>


-- 

Regards,
...sudhakara