You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by "Ravuri, Venkata Puneet" <vr...@ea.com> on 2017/08/23 16:23:48 UTC
Recommendation for Resourcemanager GC configuration
Hello,
I wanted to know if there is any recommendation for ResourceManager GC settings.
Full GC (with Parallel GC, 8 threads) is sometimes taking more than 30 sec due to which state store sessions to Zookeeper time out resulting in FATAL errors.
The YARN cluster is heavily used with 1000’s of applications launched per hour.
Could you please share any documentation related to best practices for tuning resourcemanager GC?
Thanks,
Puneet
Re: Recommendation for Resourcemanager GC configuration
Posted by "Ravuri, Venkata Puneet" <vr...@ea.com>.
Hi Naga, Ravi,
We have lots of small applications running on the cluster. We use Java 8 and Hadoop version 2.7.3.
Resourcemanager is running on 40GB heap with NewRatio set to 3. We store 100,000 completed apps in memory (max-completed-apps).
Tenured space occupies ~28 GB after a full GC. Is this footprint expected for 100,000 apps?
We did try CMS before with 70% occupancy fraction, there were ‘promotion failures’ that ended up in stop the world.
Regards,
Puneet
From: Naganarasimha Garla <na...@apache.org>
Date: Wednesday, August 23, 2017 at 5:23 PM
To: Ravi Prakash <ra...@gmail.com>
Cc: "Ravuri, Venkata Puneet" <vr...@ea.com>, "common-user@hadoop.apache.org" <us...@hadoop.apache.org>
Subject: Re: Recommendation for Resourcemanager GC configuration
Hi Puneet,
Along with the heap dump details, I would also like to know the version of the Hadoop-Yarn being used, size of the cluster, all Memory configurations, and JRE version.
Also if possible can you share the rational behind the choice for Parallel GC collector over others (CMS or G1) ?
Regards,
+ Naga
On Thu, Aug 24, 2017 at 2:54 AM, Ravi Prakash <ra...@gmail.com>> wrote:
Hi Puneet
Can you take a heap dump and see where most of the churn is? Is it lots of small applications / few really large applications with small containers etc. ?
Cheers
Ravi
On Wed, Aug 23, 2017 at 9:23 AM, Ravuri, Venkata Puneet <vr...@ea.com>> wrote:
Hello,
I wanted to know if there is any recommendation for ResourceManager GC settings.
Full GC (with Parallel GC, 8 threads) is sometimes taking more than 30 sec due to which state store sessions to Zookeeper time out resulting in FATAL errors.
The YARN cluster is heavily used with 1000’s of applications launched per hour.
Could you please share any documentation related to best practices for tuning resourcemanager GC?
Thanks,
Puneet
Re: Recommendation for Resourcemanager GC configuration
Posted by Naganarasimha Garla <na...@apache.org>.
Hi Puneet,
Along with the heap dump details, I would also like to know the version of
the Hadoop-Yarn being used, size of the cluster, all Memory configurations,
and JRE version.
Also if possible can you share the rational behind the choice for Parallel
GC collector over others (CMS or G1) ?
Regards,
+ Naga
On Thu, Aug 24, 2017 at 2:54 AM, Ravi Prakash <ra...@gmail.com> wrote:
> Hi Puneet
>
> Can you take a heap dump and see where most of the churn is? Is it lots of
> small applications / few really large applications with small containers
> etc. ?
>
> Cheers
> Ravi
>
> On Wed, Aug 23, 2017 at 9:23 AM, Ravuri, Venkata Puneet <vr...@ea.com>
> wrote:
>
>> Hello,
>>
>>
>>
>> I wanted to know if there is any recommendation for ResourceManager GC
>> settings.
>>
>> Full GC (with Parallel GC, 8 threads) is sometimes taking more than 30
>> sec due to which state store sessions to Zookeeper time out resulting in
>> FATAL errors.
>>
>> The YARN cluster is heavily used with 1000’s of applications launched per
>> hour.
>>
>>
>>
>> Could you please share any documentation related to best practices for
>> tuning resourcemanager GC?
>>
>>
>>
>> Thanks,
>>
>> Puneet
>>
>
>
Re: Recommendation for Resourcemanager GC configuration
Posted by Ravi Prakash <ra...@gmail.com>.
Hi Puneet
Can you take a heap dump and see where most of the churn is? Is it lots of
small applications / few really large applications with small containers
etc. ?
Cheers
Ravi
On Wed, Aug 23, 2017 at 9:23 AM, Ravuri, Venkata Puneet <vr...@ea.com>
wrote:
> Hello,
>
>
>
> I wanted to know if there is any recommendation for ResourceManager GC
> settings.
>
> Full GC (with Parallel GC, 8 threads) is sometimes taking more than 30 sec
> due to which state store sessions to Zookeeper time out resulting in FATAL
> errors.
>
> The YARN cluster is heavily used with 1000’s of applications launched per
> hour.
>
>
>
> Could you please share any documentation related to best practices for
> tuning resourcemanager GC?
>
>
>
> Thanks,
>
> Puneet
>
Re: Recommendation for Resourcemanager GC configuration
Posted by "Ravuri, Venkata Puneet" <vr...@ea.com>.
Hi Vinod,
The heap size is 40GB and NewRatio is set to 3. We have max completed applications set to 100000.
Regards,
Puneet
From: Vinod Kumar Vavilapalli <vi...@apache.org>
Date: Wednesday, August 23, 2017 at 5:47 PM
To: "Ravuri, Venkata Puneet" <vr...@ea.com>
Cc: "common-user@hadoop.apache.org" <us...@hadoop.apache.org>
Subject: Re: Recommendation for Resourcemanager GC configuration
What is the ResourceManager JVM’s heap size? What is the value for the configuration yarn.resourcemanager.max-completed-applications?
+Vinod
On Aug 23, 2017, at 9:23 AM, Ravuri, Venkata Puneet <vr...@ea.com>> wrote:
Hello,
I wanted to know if there is any recommendation for ResourceManager GC settings.
Full GC (with Parallel GC, 8 threads) is sometimes taking more than 30 sec due to which state store sessions to Zookeeper time out resulting in FATAL errors.
The YARN cluster is heavily used with 1000’s of applications launched per hour.
Could you please share any documentation related to best practices for tuning resourcemanager GC?
Thanks,
Puneet
Re: Recommendation for Resourcemanager GC configuration
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
What is the ResourceManager JVM’s heap size? What is the value for the configuration yarn.resourcemanager.max-completed-applications?
+Vinod
> On Aug 23, 2017, at 9:23 AM, Ravuri, Venkata Puneet <vr...@ea.com> wrote:
>
> Hello,
>
> I wanted to know if there is any recommendation for ResourceManager GC settings.
> Full GC (with Parallel GC, 8 threads) is sometimes taking more than 30 sec due to which state store sessions to Zookeeper time out resulting in FATAL errors.
> The YARN cluster is heavily used with 1000’s of applications launched per hour.
>
> Could you please share any documentation related to best practices for tuning resourcemanager GC?
>
> Thanks,
> Puneet