You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Shawn Heisey <ap...@elyograg.org> on 2015/01/01 20:10:13 UTC
Garbage Collection tuning - G1 is now a good option
I've been working with Oracle employees to find better GC tuning
options. The results are good enough to share with the community:
https://wiki.apache.org/solr/ShawnHeisey#GC_Tuning
With the latest Java 7 or Java 8 version, and a couple of tuning
options, G1GC has grown up enough to be a viable choice. Two of the
settings on that list were critical for making the performance
acceptable with my testing: ParallelRefProcEnabled and G1HeapRegionSize.
I've included some notes on the wiki about how you can size the G1 heap
regions appropriately for your own index.
Thanks,
Shawn
Re: Garbage Collection tuning - G1 is now a good option
Posted by William Bell <bi...@gmail.com>.
It looks like 32 bit is affected.
> On 2013-08-14 08:27, Dawid Weiss wrote:
>>
>> Hi everyone,
>>
>> I am a committer to the Lucene/Solr project. We've recently hit what
>> we believe is a JIT/GC bug -- it manifests itself only when G1GC is
>> used, on a 32-bit VM:
On Fri, Jan 9, 2015 at 7:10 PM, Shawn Heisey <ap...@elyograg.org> wrote:
> On 1/1/2015 12:10 PM, Shawn Heisey wrote:
> > I've been working with Oracle employees to find better GC tuning
> > options. The results are good enough to share with the community:
> >
> > https://wiki.apache.org/solr/ShawnHeisey#GC_Tuning
> >
> > With the latest Java 7 or Java 8 version, and a couple of tuning
> > options, G1GC has grown up enough to be a viable choice. Two of the
> > settings on that list were critical for making the performance
> > acceptable with my testing: ParallelRefProcEnabled and G1HeapRegionSize.
> >
> > I've included some notes on the wiki about how you can size the G1 heap
> > regions appropriately for your own index.
>
> A note was just recently added to the Lucene wiki, saying that the G1
> collector should never be used with Lucene, because there are bugs that
> might cause index corruption. Solr uses Lucene, so this would apply.
>
>
> https://wiki.apache.org/lucene-java/JavaBugs#Oracle_Java_.2F_Sun_Java_.2F_OpenJDK_Bugs
>
> I have never had any problems with it, but perhaps I spoke too soon when
> recommending G1. Does anyone else have anything to add?
>
> Thanks,
> Shawn
>
>
--
Bill Bell
billnbell@gmail.com
cell 720-256-8076
Re: Garbage Collection tuning - G1 is now a good option
Posted by Shawn Heisey <ap...@elyograg.org>.
On 1/1/2015 12:10 PM, Shawn Heisey wrote:
> I've been working with Oracle employees to find better GC tuning
> options. The results are good enough to share with the community:
>
> https://wiki.apache.org/solr/ShawnHeisey#GC_Tuning
>
> With the latest Java 7 or Java 8 version, and a couple of tuning
> options, G1GC has grown up enough to be a viable choice. Two of the
> settings on that list were critical for making the performance
> acceptable with my testing: ParallelRefProcEnabled and G1HeapRegionSize.
>
> I've included some notes on the wiki about how you can size the G1 heap
> regions appropriately for your own index.
A note was just recently added to the Lucene wiki, saying that the G1
collector should never be used with Lucene, because there are bugs that
might cause index corruption. Solr uses Lucene, so this would apply.
https://wiki.apache.org/lucene-java/JavaBugs#Oracle_Java_.2F_Sun_Java_.2F_OpenJDK_Bugs
I have never had any problems with it, but perhaps I spoke too soon when
recommending G1. Does anyone else have anything to add?
Thanks,
Shawn
Re: Garbage Collection tuning - G1 is now a good option
Posted by Otis Gospodnetic <ot...@gmail.com>.
Not sure about AggressiveOpts, but G1 has been working for us nicely.
We've successfully used it with HBase, Hadoop, Elasticsearch, and other
custom Java apps (all still Java 7, but Java 8 should be even better). Not
sure if we are using in on our Solr instances.
e.g. see http://blog.sematext.com/2013/06/24/g1-cms-java-garbage-collector/
Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
On Thu, Jan 1, 2015 at 8:35 PM, William Bell <bi...@gmail.com> wrote:
> But tons of people on this mailing list do not recommend AggressiveOpts
>
> Why do you recommend it?
>
> On Thu, Jan 1, 2015 at 12:10 PM, Shawn Heisey <ap...@elyograg.org> wrote:
>
> > I've been working with Oracle employees to find better GC tuning
> > options. The results are good enough to share with the community:
> >
> > https://wiki.apache.org/solr/ShawnHeisey#GC_Tuning
> >
> > With the latest Java 7 or Java 8 version, and a couple of tuning
> > options, G1GC has grown up enough to be a viable choice. Two of the
> > settings on that list were critical for making the performance
> > acceptable with my testing: ParallelRefProcEnabled and G1HeapRegionSize.
> >
> > I've included some notes on the wiki about how you can size the G1 heap
> > regions appropriately for your own index.
> >
> > Thanks,
> > Shawn
> >
> >
>
>
> --
> Bill Bell
> billnbell@gmail.com
> cell 720-256-8076
>
Re: Garbage Collection tuning - G1 is now a good option
Posted by Mark Miller <ma...@gmail.com>.
bq. But tons of people on this mailing list do not recommend AggressiveOpts
It's up to you to decide - that is why it's an option. It will enable more
aggressive options that will tend to perform better. On the other hand,
these more aggressive options and optimizations have a history of being
more buggy. Depending on your needs and risk tolerance, you might make a
different choice.
You probably want to research the specific issues it has been found to
cause and whether those issues are important to you or affect the JVM you
are using. You also have to keep in mind that probably all of the issues
are not known or documented and that updates may introduce new issues. You
face some risk like that no matter what though.
- Mark
On Fri Jan 02 2015 at 8:37:47 AM Shawn Heisey <ap...@elyograg.org> wrote:
> On 1/1/2015 6:35 PM, William Bell wrote:
> > But tons of people on this mailing list do not recommend AggressiveOpts
> >
> > Why do you recommend it?
>
> I haven't done any comparisons with and without it. To call it a
> "recommendation" is a little bit strong. I use it, and I am seeing good
> results.
>
> My reading indicates that AggressiveOpts basically enables settings that
> are being considered for defaults in a later Java version. If they are
> being seriously considered for new defaults, they are probably something
> that I want to be using. I have also heard that there are sometimes
> bugs with that option, but I have not personally had any problems.
>
> If you don't want to use that option, feel free to leave it out. I will
> update my wiki page with a note about AggressiveOpts.
>
> Thanks,
> Shawn
>
>
Re: Garbage Collection tuning - G1 is now a good option
Posted by Shawn Heisey <ap...@elyograg.org>.
On 1/1/2015 6:35 PM, William Bell wrote:
> But tons of people on this mailing list do not recommend AggressiveOpts
>
> Why do you recommend it?
I haven't done any comparisons with and without it. To call it a
"recommendation" is a little bit strong. I use it, and I am seeing good
results.
My reading indicates that AggressiveOpts basically enables settings that
are being considered for defaults in a later Java version. If they are
being seriously considered for new defaults, they are probably something
that I want to be using. I have also heard that there are sometimes
bugs with that option, but I have not personally had any problems.
If you don't want to use that option, feel free to leave it out. I will
update my wiki page with a note about AggressiveOpts.
Thanks,
Shawn
Re: Garbage Collection tuning - G1 is now a good option
Posted by William Bell <bi...@gmail.com>.
But tons of people on this mailing list do not recommend AggressiveOpts
Why do you recommend it?
On Thu, Jan 1, 2015 at 12:10 PM, Shawn Heisey <ap...@elyograg.org> wrote:
> I've been working with Oracle employees to find better GC tuning
> options. The results are good enough to share with the community:
>
> https://wiki.apache.org/solr/ShawnHeisey#GC_Tuning
>
> With the latest Java 7 or Java 8 version, and a couple of tuning
> options, G1GC has grown up enough to be a viable choice. Two of the
> settings on that list were critical for making the performance
> acceptable with my testing: ParallelRefProcEnabled and G1HeapRegionSize.
>
> I've included some notes on the wiki about how you can size the G1 heap
> regions appropriately for your own index.
>
> Thanks,
> Shawn
>
>
--
Bill Bell
billnbell@gmail.com
cell 720-256-8076