You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@solr.apache.org by Michael Gibney <mi...@michaelgibney.net> on 2024/08/19 18:32:00 UTC
BitDocSet humongous objects
We're encountering a relatively large amount of "humongous object"
allocation with larger cores. For just under 32G configured heap,
G1HeapRegionSize seems to default to 8M or 16M depending on jdk
version. For 100M docs in a shard (e.g.), the long[] backing a
FixedBitSet (BitDocSet) will be ~12M, comfortably above the "humongous
object" threshold for either default region size.
We suspect this is contributing to some GC issues (higher pause times,
full collections, etc.). We could experiment with raising
G1HeapRegionSize (setting explicitly to 32M), but this would end up
with 1024 regions, which could have its own drawbacks, iiuc.
I'm wondering if others have encountered this type of situation, and
if so what they've done about it.
For a more robust solution than fussing with G1HeapRegionSize, I'm
wondering if it might be appropriate to change the implementation of
BitDocSet so that larger instances will be backed by an array of
multiple smaller FixedBitSet instances. This would introduce some
extra complexity for DocSets over large indexes, but it shouldn't be
terrible; it could be cleanly implemented and would ensure that we
never allocate humongous objects in the service of BitDocSets ...
Michael
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: BitDocSet humongous objects
Posted by David Smiley <ds...@apache.org>.
While orthogonal, I'd rather we not write code that exists purely only
to solve this problem. Segment sizes are capped configurably in the
MergePolicy; you can lower it if the FixedBitSet would be too large
for the largest segments. For single-segment indexes (and the status
quo today), maybe the solution is GC tuning as this is really a GC
ergonomics matter.
On Tue, Aug 20, 2024 at 11:23 AM Michael Gibney
<mi...@michaelgibney.net> wrote:
>
> Interesting -- although certainly related, I think these are somewhat
> orthogonal questions. You could well have a merge strategy/heap and gc
> configuration/index size that would have the same "humongous object"
> problem even under a per-segment cache approach (certainly for cores
> optimized to a single segment, but also in other cases).
>
> On Mon, Aug 19, 2024 at 7:12 PM David Smiley <ds...@apache.org> wrote:
> >
> > On Mon, Aug 19, 2024 at 2:32 PM Michael Gibney
> > <mi...@michaelgibney.net> wrote:
> > > For a more robust solution than fussing with G1HeapRegionSize, I'm
> > > wondering if it might be appropriate to change the implementation of
> > > BitDocSet so that larger instances will be backed by an array of
> > > multiple smaller FixedBitSet instances. This would introduce some
> > > extra complexity for DocSets over large indexes, but it shouldn't be
> > > terrible; it could be cleanly implemented and would ensure that we
> > > never allocate humongous objects in the service of BitDocSets ...
> >
> > IMO I don't think this is quite the right direction to go in.
> > Instead, we should prioritize a FixedBitSet per segment -- basically a
> > segment level filterCache. This exists at the Lucene level
> > (IndexSearcher QueryCache); we could stop disabling it. It'd go with
> > the new multiThreaded query stuff wonderfully. But would then want to
> > do changes in BitDocSet so that it becomes an aggregate of
> > FixedBitSets sourced from that cache. At least this is a strawman
> > proposal; needs more thought!
> >
> > Note that Yonik resisted ending Solr's top level FixedBitSet on the
> > grounds that benchmarks are needed to show that a segment approach is
> > better. But that's challenging -- we will find the status quo is
> > faster for some situations and a segment level is faster for others.
> > Shrug.
> >
> > ~ David
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > For additional commands, e-mail: dev-help@solr.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: BitDocSet humongous objects
Posted by Michael Gibney <mi...@michaelgibney.net>.
Interesting -- although certainly related, I think these are somewhat
orthogonal questions. You could well have a merge strategy/heap and gc
configuration/index size that would have the same "humongous object"
problem even under a per-segment cache approach (certainly for cores
optimized to a single segment, but also in other cases).
On Mon, Aug 19, 2024 at 7:12 PM David Smiley <ds...@apache.org> wrote:
>
> On Mon, Aug 19, 2024 at 2:32 PM Michael Gibney
> <mi...@michaelgibney.net> wrote:
> > For a more robust solution than fussing with G1HeapRegionSize, I'm
> > wondering if it might be appropriate to change the implementation of
> > BitDocSet so that larger instances will be backed by an array of
> > multiple smaller FixedBitSet instances. This would introduce some
> > extra complexity for DocSets over large indexes, but it shouldn't be
> > terrible; it could be cleanly implemented and would ensure that we
> > never allocate humongous objects in the service of BitDocSets ...
>
> IMO I don't think this is quite the right direction to go in.
> Instead, we should prioritize a FixedBitSet per segment -- basically a
> segment level filterCache. This exists at the Lucene level
> (IndexSearcher QueryCache); we could stop disabling it. It'd go with
> the new multiThreaded query stuff wonderfully. But would then want to
> do changes in BitDocSet so that it becomes an aggregate of
> FixedBitSets sourced from that cache. At least this is a strawman
> proposal; needs more thought!
>
> Note that Yonik resisted ending Solr's top level FixedBitSet on the
> grounds that benchmarks are needed to show that a segment approach is
> better. But that's challenging -- we will find the status quo is
> faster for some situations and a segment level is faster for others.
> Shrug.
>
> ~ David
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: BitDocSet humongous objects
Posted by David Smiley <ds...@apache.org>.
On Mon, Aug 19, 2024 at 2:32 PM Michael Gibney
<mi...@michaelgibney.net> wrote:
> For a more robust solution than fussing with G1HeapRegionSize, I'm
> wondering if it might be appropriate to change the implementation of
> BitDocSet so that larger instances will be backed by an array of
> multiple smaller FixedBitSet instances. This would introduce some
> extra complexity for DocSets over large indexes, but it shouldn't be
> terrible; it could be cleanly implemented and would ensure that we
> never allocate humongous objects in the service of BitDocSets ...
IMO I don't think this is quite the right direction to go in.
Instead, we should prioritize a FixedBitSet per segment -- basically a
segment level filterCache. This exists at the Lucene level
(IndexSearcher QueryCache); we could stop disabling it. It'd go with
the new multiThreaded query stuff wonderfully. But would then want to
do changes in BitDocSet so that it becomes an aggregate of
FixedBitSets sourced from that cache. At least this is a strawman
proposal; needs more thought!
Note that Yonik resisted ending Solr's top level FixedBitSet on the
grounds that benchmarks are needed to show that a segment approach is
better. But that's challenging -- we will find the status quo is
faster for some situations and a segment level is faster for others.
Shrug.
~ David
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org