GC tuning question - can improving GC pauses cause indexing to slow down?

[Shawn Heisey <ap...@elyograg.org>]

Is it possible that tuning garbage collection to achieve much better
pause characteristics might actually *decrease* index performance?

Rebuilds that I did while still using a tuned CMS config would take
between 5.5 and 6 hours, sometimes going slightly over 6 hours.

A rebuild that I did recently with G1 took 6.82 hours.  A rebuild that I
did yesterday with further tuned G1 settings (which seemed to result in
much smaller pauses than the previous G1 settings) took 8.97 hours, and
that was on slightly faster hardware than the rebuild that took 6.82 hours.

These rebuilds are done with DIH from MySQL.

It seems completely counter-intuitive that settings which show better GC
pause characteristics would result in indexing performance going down
... so can anyone shed light on this, tell me whether I'm out of my mind?

Thanks,
Shawn

Re: GC tuning question - can improving GC pauses cause indexing to slow down?

[Walter Underwood <wu...@wunderwood.org>]

For throughput with G1, get rid of the pause time goal (-XX:MaxGCPauseMillis), so the GC can pause as long as it wants.

Beyond that, use a non-concurrent collector and make sure that everything is OK with pauses that last a few seconds.

This is a pretty detailed paper about balancing throughput and pause:

https://engineering.linkedin.com/garbage-collection/garbage-collection-optimization-high-throughput-and-low-latency-java-applications

wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/


On Jan 8, 2015, at 11:38 PM, Shawn Heisey <ap...@elyograg.org> wrote:

> On 1/8/2015 11:05 PM, Boogie Shafer wrote:
>> In the abstract, it sounds like you are seeing the difference between tuning for latency vs tuning for throughput 
>> 
>> My hunch would be you are seeing more (albeit individually quicker) GC events with your new settings during the rebuild
>> 
>> I imagine that in most cases a solr rebuild is relatively rare compared to the amount of times where a lower latency request is desired. If the rebuild times are problematic for you, use tunings specific to that workload during the times you need it and then switch back to your low latency settings after. If you are doing that you can probably run with a bigger heap temporarily during the rebuild as you aren't likely to be fielding queries and don't benefit from having a larger OS cache available
> 
> Full rebuilds are indeed relatively rare.  Avoiding long pauses and
> keeping query latency low are usually a lot more important than how
> quickly the index rebuilds.  Quick rebuilds are nice, but not strictly
> necessary.
> 
> We do incremental updates that start at the top of every minute, unless
> an update is already running.  Exactly how long those updates take is of
> little importance, unless that time is easier to measure in minutes
> rather than seconds.
> 
> If I ever find myself in a situation where completing a rebuild as fast
> as possible becomes extremely important, does anyone have suggestions
> for GC tuning options that will optimize for throughput?
> 
> Thanks,
> Shawn
> 

Re: GC tuning question - can improving GC pauses cause indexing to slow down?

[Shawn Heisey <ap...@elyograg.org>]

On 1/8/2015 11:05 PM, Boogie Shafer wrote:
> In the abstract, it sounds like you are seeing the difference between tuning for latency vs tuning for throughput 
> 
> My hunch would be you are seeing more (albeit individually quicker) GC events with your new settings during the rebuild
> 
> I imagine that in most cases a solr rebuild is relatively rare compared to the amount of times where a lower latency request is desired. If the rebuild times are problematic for you, use tunings specific to that workload during the times you need it and then switch back to your low latency settings after. If you are doing that you can probably run with a bigger heap temporarily during the rebuild as you aren't likely to be fielding queries and don't benefit from having a larger OS cache available

Full rebuilds are indeed relatively rare.  Avoiding long pauses and
keeping query latency low are usually a lot more important than how
quickly the index rebuilds.  Quick rebuilds are nice, but not strictly
necessary.

We do incremental updates that start at the top of every minute, unless
an update is already running.  Exactly how long those updates take is of
little importance, unless that time is easier to measure in minutes
rather than seconds.

If I ever find myself in a situation where completing a rebuild as fast
as possible becomes extremely important, does anyone have suggestions
for GC tuning options that will optimize for throughput?

Thanks,
Shawn

Re: GC tuning question - can improving GC pauses cause indexing to slow down?

[Boogie Shafer <Bo...@proquest.com>]

In the abstract, it sounds like you are seeing the difference between tuning for latency vs tuning for throughput 

My hunch would be you are seeing more (albeit individually quicker) GC events with your new settings during the rebuild

I imagine that in most cases a solr rebuild is relatively rare compared to the amount of times where a lower latency request is desired. If the rebuild times are problematic for you, use tunings specific to that workload during the times you need it and then switch back to your low latency settings after. If you are doing that you can probably run with a bigger heap temporarily during the rebuild as you aren't likely to be fielding queries and don't benefit from having a larger OS cache available



Sent from my iPhone

> On Jan 8, 2015, at 20:54, Shawn Heisey <ap...@elyograg.org> wrote:
> 
> Is it possible that tuning garbage collection to achieve much better
> pause characteristics might actually *decrease* index performance?
> 
> Rebuilds that I did while still using a tuned CMS config would take
> between 5.5 and 6 hours, sometimes going slightly over 6 hours.
> 
> A rebuild that I did recently with G1 took 6.82 hours.  A rebuild that I
> did yesterday with further tuned G1 settings (which seemed to result in
> much smaller pauses than the previous G1 settings) took 8.97 hours, and
> that was on slightly faster hardware than the rebuild that took 6.82 hours.
> 
> These rebuilds are done with DIH from MySQL.
> 
> It seems completely counter-intuitive that settings which show better GC
> pause characteristics would result in indexing performance going down
> ... so can anyone shed light on this, tell me whether I'm out of my mind?
> 
> Thanks,
> Shawn
> 

Re: GC tuning question - can improving GC pauses cause indexing to slow down?

[Walter Underwood <wu...@wunderwood.org>]

I would not be surprised at all. Optimizing for minimum pauses usually increases overhead that decreases overall throughput. This is a pretty common tradeoff.

For maximum throughput, when you don’t care about pauses, the simplest non-concurrent GC is often the best. That might be the right choice for running big map-reduce jobs, for example.

Low-pause GCs do lots of extra work in parallel. Some of that work is making guesses which get thrown away, or doing “just in case” analysis.

To quote Oracle:

"When you evaluate or tune any garbage collection, there is always a latency versus throughput trade-off. The G1 GC is an incremental garbage collector with uniform pauses, but also more overhead on the application threads. The throughput goal for the G1 GC is 90 percent application time and 10 percent garbage collection time. When you compare this to Java HotSpot VM's throughput collector, the goal there is 99 percent application time and 1 percent garbage collection time.”

http://www.oracle.com/technetwork/articles/java/g1gc-1984535.html

wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/


On Jan 8, 2015, at 8:53 PM, Shawn Heisey <ap...@elyograg.org> wrote:

> Is it possible that tuning garbage collection to achieve much better
> pause characteristics might actually *decrease* index performance?
> 
> Rebuilds that I did while still using a tuned CMS config would take
> between 5.5 and 6 hours, sometimes going slightly over 6 hours.
> 
> A rebuild that I did recently with G1 took 6.82 hours.  A rebuild that I
> did yesterday with further tuned G1 settings (which seemed to result in
> much smaller pauses than the previous G1 settings) took 8.97 hours, and
> that was on slightly faster hardware than the rebuild that took 6.82 hours.
> 
> These rebuilds are done with DIH from MySQL.
> 
> It seems completely counter-intuitive that settings which show better GC
> pause characteristics would result in indexing performance going down
> ... so can anyone shed light on this, tell me whether I'm out of my mind?
> 
> Thanks,
> Shawn
>