You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Renu Yadav <yr...@gmail.com> on 2015/11/13 15:31:17 UTC
spark 1.4 GC issue
am using spark 1.4 and my application is taking much time in GC around
60-70% of time for each task
I am using parallel GC.
please help somebody as soon as possible.
Thanks,
Renu
Re: spark 1.4 GC issue
Posted by Gaurav Kumar <ga...@gmail.com>.
Please have a look at http://spark.apache.org/docs/1.4.0/tuning.html
You may also want to use the latest build of JDK 7/8 and use G1GC instead.
I saw considerable reductions in GC time just by doing that.
Rest of the tuning parameters are better explained in the link above.
Best Regards,
Gaurav Kumar
Big Data • Data Science • Photography • Music
+91 9953294125
On Fri, Nov 13, 2015 at 8:01 PM, Renu Yadav <yr...@gmail.com> wrote:
> am using spark 1.4 and my application is taking much time in GC around
> 60-70% of time for each task
>
> I am using parallel GC.
> please help somebody as soon as possible.
>
> Thanks,
> Renu
>
Re: spark 1.4 GC issue
Posted by Ted Yu <yu...@gmail.com>.
Please take a look at http://www.infoq.com/articles/tuning-tips-G1-GC
Cheers
On Sat, Nov 14, 2015 at 10:03 PM, Renu Yadav <yr...@gmail.com> wrote:
> I have tried with G1 GC .Please if anyone can provide their setting for GC.
> At code level I am :
> 1.reading orc table usind dataframe
> 2.map df to rdd of my case class
> 3. changed that rdd to paired rdd
> 4.Applied combineByKey
> 5. saving the result to orc file
>
> Please suggest
>
> Regards,
> Renu Yadav
>
> On Fri, Nov 13, 2015 at 8:01 PM, Renu Yadav <yr...@gmail.com> wrote:
>
>> am using spark 1.4 and my application is taking much time in GC around
>> 60-70% of time for each task
>>
>> I am using parallel GC.
>> please help somebody as soon as possible.
>>
>> Thanks,
>> Renu
>>
>
>
Re: spark 1.4 GC issue
Posted by Ted Yu <yu...@gmail.com>.
Please take a look at http://www.infoq.com/articles/tuning-tips-G1-GC
Cheers
On Sat, Nov 14, 2015 at 10:03 PM, Renu Yadav <yr...@gmail.com> wrote:
> I have tried with G1 GC .Please if anyone can provide their setting for GC.
> At code level I am :
> 1.reading orc table usind dataframe
> 2.map df to rdd of my case class
> 3. changed that rdd to paired rdd
> 4.Applied combineByKey
> 5. saving the result to orc file
>
> Please suggest
>
> Regards,
> Renu Yadav
>
> On Fri, Nov 13, 2015 at 8:01 PM, Renu Yadav <yr...@gmail.com> wrote:
>
>> am using spark 1.4 and my application is taking much time in GC around
>> 60-70% of time for each task
>>
>> I am using parallel GC.
>> please help somebody as soon as possible.
>>
>> Thanks,
>> Renu
>>
>
>
Re: spark 1.4 GC issue
Posted by PhuDuc Nguyen <du...@gmail.com>.
You can try this for G1GC:
.../spark-submit --conf "spark.executor.extraJavaOptions=-XX:+UseG1GC
-XX:+UseCompressedOops -XX:-UseGCOverheadLimit" ...
However, I would suggest ensuring your job is properly tuned. If you're
experiencing 60% GC in a task it's likely garbage collection is not the
problem, i.e., you may be doing something that is causing GC to thrash
around. Yes G1GC can give better GC performance, but this may not be the
root problem of your job. There is no magical-fix-all configuration. Based
on the info you've outlined, I would suggest looking at the results in your
UI after the combineByKey and first ensure the data is still well balanced
across all partitions.
HTH,
Duc
On Sat, Nov 14, 2015 at 11:03 PM, Renu Yadav <yr...@gmail.com> wrote:
> I have tried with G1 GC .Please if anyone can provide their setting for GC.
> At code level I am :
> 1.reading orc table usind dataframe
> 2.map df to rdd of my case class
> 3. changed that rdd to paired rdd
> 4.Applied combineByKey
> 5. saving the result to orc file
>
> Please suggest
>
> Regards,
> Renu Yadav
>
> On Fri, Nov 13, 2015 at 8:01 PM, Renu Yadav <yr...@gmail.com> wrote:
>
>> am using spark 1.4 and my application is taking much time in GC around
>> 60-70% of time for each task
>>
>> I am using parallel GC.
>> please help somebody as soon as possible.
>>
>> Thanks,
>> Renu
>>
>
>
Re: spark 1.4 GC issue
Posted by Renu Yadav <yr...@gmail.com>.
I have tried with G1 GC .Please if anyone can provide their setting for GC.
At code level I am :
1.reading orc table usind dataframe
2.map df to rdd of my case class
3. changed that rdd to paired rdd
4.Applied combineByKey
5. saving the result to orc file
Please suggest
Regards,
Renu Yadav
On Fri, Nov 13, 2015 at 8:01 PM, Renu Yadav <yr...@gmail.com> wrote:
> am using spark 1.4 and my application is taking much time in GC around
> 60-70% of time for each task
>
> I am using parallel GC.
> please help somebody as soon as possible.
>
> Thanks,
> Renu
>