You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Erick Erickson <er...@gmail.com> on 2013/05/08 17:21:49 UTC
Compressed field reader and memory problems with atomic updates
I'm seeing a case where (reported to me, not on my personal machine)
where Solr's heap is being exhausted apparently by compressed field
reader. Here's a sample line from a summary of a heap dump:
CompressingStoredFieldReader 735 instances
where the CompressingStoredFieldReader instances are taking up 7.5G of
heap space (out of 8 allocated). The GC cycles are a pattern I've seen
before
gc for a long time, recover a few meg
run for a _very_ short time
gc again, recover a few meg
This app happens to be doing a lot of atomic updates, running with CMS
collector, java 1.7. There is very little querying going on, and it is
happening on several different machines. Solr 4.2.
Now, there is a bit of custom update code involved, we're testing whether
1> if not compressing the stored fields changes things
2> if disabling the custom code changes things
3> if Solr 4.3 changes things.
4> whether memory usage grows over time linearly or spikes (pretty
sure the former).
This is an index-heavy/query-light application, and we're hosing a
_bunch_ of documents at the indexing process. But still, assuming that
the memory reporting above is accurate, each
CompressingStoredFieldReader is apparently taking up 10M each and
there are over 700 of them. The documents are being batched (don't
quite know how many per batch, will find out).
Mainly, I'm asking if this rings any bells. I don't see anything on a
quick look at the JIRAs.
Thanks,
Erick
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Compressed field reader and memory problems with atomic updates
Posted by Erick Erickson <er...@gmail.com>.
Thanks to you both. I've pinged the people who can reproduce this
problem, I suspect they'll try the patch out. I'm out next week, far
far away from internet connections so I'll be silent for a while. But
don't be surprised if new messages appear.
Thanks again
Erick
On Fri, May 10, 2013 at 3:46 AM, Adrien Grand <jp...@gmail.com> wrote:
> Erick,
>
> On Thu, May 9, 2013 at 10:28 PM, Erick Erickson <er...@gmail.com> wrote:
>> Yeah, this is getting warmer. Some of the
>> CompressingStoredFieldsReader objects are 240M. The documents aren't
>> nearly that large I don't think (but I'll verify).
>
> Wow, 240M is a lot! Would you be able to know which member of these
> instances is making them so large? Since you were mentioning
> threading, I first thought that the culprit was the buffer used for
> decompression but it could be the fields index as well (which is
> stored in memory but should be shared across instances for the same
> segment, so threading shouldn't make the situation worse).
>
>> But still, over 700 of these objects live at once? I _think_ I'm
>> seeing the number go up significantly when the number of indexing
>> threads increases, but that's a bit of indirect evidence. My other
>> question would be whether you'd expect the number of these objects to
>> go up as the number of segments goes up, i.e. I assume they're
>> per-segment....
>
> Indeed, these objects are managed per-segment, so increasing the
> number of segments increases the number of stored fields reader
> instances.
>
>> So the pattern here is atomic updates on documents where some of the
>> fields get quite large. So the underlying reader has to decompress
>> these a lot. Do you have any suggestions how to mitigate this? Other
>> than "don't do that<G>"....
>
> As Savia suggests, lowering the number of indexing threads should
> help. If the merge factor is highish, lowering it could help as well.
> On my end, I'll fix the stored fields reader to not reuse the buffer
> for decompression (https://issues.apache.org/jira/browse/LUCENE-4995,
> even if we find out that it is not the problem here, it shouldn't
> hurt).
>
> --
> Adrien
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Compressed field reader and memory problems with atomic updates
Posted by Adrien Grand <jp...@gmail.com>.
Erick,
On Thu, May 9, 2013 at 10:28 PM, Erick Erickson <er...@gmail.com> wrote:
> Yeah, this is getting warmer. Some of the
> CompressingStoredFieldsReader objects are 240M. The documents aren't
> nearly that large I don't think (but I'll verify).
Wow, 240M is a lot! Would you be able to know which member of these
instances is making them so large? Since you were mentioning
threading, I first thought that the culprit was the buffer used for
decompression but it could be the fields index as well (which is
stored in memory but should be shared across instances for the same
segment, so threading shouldn't make the situation worse).
> But still, over 700 of these objects live at once? I _think_ I'm
> seeing the number go up significantly when the number of indexing
> threads increases, but that's a bit of indirect evidence. My other
> question would be whether you'd expect the number of these objects to
> go up as the number of segments goes up, i.e. I assume they're
> per-segment....
Indeed, these objects are managed per-segment, so increasing the
number of segments increases the number of stored fields reader
instances.
> So the pattern here is atomic updates on documents where some of the
> fields get quite large. So the underlying reader has to decompress
> these a lot. Do you have any suggestions how to mitigate this? Other
> than "don't do that<G>"....
As Savia suggests, lowering the number of indexing threads should
help. If the merge factor is highish, lowering it could help as well.
On my end, I'll fix the stored fields reader to not reuse the buffer
for decompression (https://issues.apache.org/jira/browse/LUCENE-4995,
even if we find out that it is not the problem here, it shouldn't
hurt).
--
Adrien
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Compressed field reader and memory problems with atomic updates
Posted by Savia Beson <ek...@googlemail.com>.
Erick,
I would try to limit the number of threads to aleviate the problem, honestly, gong much above #CPU cores rarely makes things better. Just let excessive update commands wait in some queue. We impose hard limits intercepting update chain and configure container thread pool to have smaller number of long liven threads ( longer TTL times to reduce churn). Is there a simple way to do it on jetty?
On May 9, 2013, at 10:28 PM, Erick Erickson <er...@gmail.com> wrote:
> Adrien:
>
> Yeah, this is getting warmer. Some of the
> CompressingStoredFieldsReader objects are 240M. The documents aren't
> nearly that large I don't think (but I'll verify).
>
> But still, over 700 of these objects live at once? I _think_ I'm
> seeing the number go up significantly when the number of indexing
> threads increases, but that's a bit of indirect evidence. My other
> question would be whether you'd expect the number of these objects to
> go up as the number of segments goes up, i.e. I assume they're
> per-segment....
>
> So the pattern here is atomic updates on documents where some of the
> fields get quite large. So the underlying reader has to decompress
> these a lot. Do you have any suggestions how to mitigate this? Other
> than "don't do that<G>"....
>
> Thanks,
> Erick
>
> On Thu, May 9, 2013 at 6:45 AM, Adrien Grand <jp...@gmail.com> wrote:
>> Hi Erick,
>>
>> The stored fields reader reuses the buffer used for decompression across
>> calls in the same thread, so I'm thinking that this kind of behaviour could
>> happen if some documents are very large. Is it the case?
>>
>> Adrien
>>
>> Le 8 mai 2013 17:22, "Erick Erickson" <er...@gmail.com> a écrit :
>>>
>>> I'm seeing a case where (reported to me, not on my personal machine)
>>> where Solr's heap is being exhausted apparently by compressed field
>>> reader. Here's a sample line from a summary of a heap dump:
>>>
>>> CompressingStoredFieldReader 735 instances
>>>
>>> where the CompressingStoredFieldReader instances are taking up 7.5G of
>>> heap space (out of 8 allocated). The GC cycles are a pattern I've seen
>>> before
>>> gc for a long time, recover a few meg
>>> run for a _very_ short time
>>> gc again, recover a few meg
>>>
>>> This app happens to be doing a lot of atomic updates, running with CMS
>>> collector, java 1.7. There is very little querying going on, and it is
>>> happening on several different machines. Solr 4.2.
>>>
>>> Now, there is a bit of custom update code involved, we're testing whether
>>> 1> if not compressing the stored fields changes things
>>> 2> if disabling the custom code changes things
>>> 3> if Solr 4.3 changes things.
>>> 4> whether memory usage grows over time linearly or spikes (pretty
>>> sure the former).
>>>
>>> This is an index-heavy/query-light application, and we're hosing a
>>> _bunch_ of documents at the indexing process. But still, assuming that
>>> the memory reporting above is accurate, each
>>> CompressingStoredFieldReader is apparently taking up 10M each and
>>> there are over 700 of them. The documents are being batched (don't
>>> quite know how many per batch, will find out).
>>>
>>> Mainly, I'm asking if this rings any bells. I don't see anything on a
>>> quick look at the JIRAs.
>>>
>>> Thanks,
>>> Erick
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Compressed field reader and memory problems with atomic updates
Posted by Erick Erickson <er...@gmail.com>.
Adrien:
Yeah, this is getting warmer. Some of the
CompressingStoredFieldsReader objects are 240M. The documents aren't
nearly that large I don't think (but I'll verify).
But still, over 700 of these objects live at once? I _think_ I'm
seeing the number go up significantly when the number of indexing
threads increases, but that's a bit of indirect evidence. My other
question would be whether you'd expect the number of these objects to
go up as the number of segments goes up, i.e. I assume they're
per-segment....
So the pattern here is atomic updates on documents where some of the
fields get quite large. So the underlying reader has to decompress
these a lot. Do you have any suggestions how to mitigate this? Other
than "don't do that<G>"....
Thanks,
Erick
On Thu, May 9, 2013 at 6:45 AM, Adrien Grand <jp...@gmail.com> wrote:
> Hi Erick,
>
> The stored fields reader reuses the buffer used for decompression across
> calls in the same thread, so I'm thinking that this kind of behaviour could
> happen if some documents are very large. Is it the case?
>
> Adrien
>
> Le 8 mai 2013 17:22, "Erick Erickson" <er...@gmail.com> a écrit :
>>
>> I'm seeing a case where (reported to me, not on my personal machine)
>> where Solr's heap is being exhausted apparently by compressed field
>> reader. Here's a sample line from a summary of a heap dump:
>>
>> CompressingStoredFieldReader 735 instances
>>
>> where the CompressingStoredFieldReader instances are taking up 7.5G of
>> heap space (out of 8 allocated). The GC cycles are a pattern I've seen
>> before
>> gc for a long time, recover a few meg
>> run for a _very_ short time
>> gc again, recover a few meg
>>
>> This app happens to be doing a lot of atomic updates, running with CMS
>> collector, java 1.7. There is very little querying going on, and it is
>> happening on several different machines. Solr 4.2.
>>
>> Now, there is a bit of custom update code involved, we're testing whether
>> 1> if not compressing the stored fields changes things
>> 2> if disabling the custom code changes things
>> 3> if Solr 4.3 changes things.
>> 4> whether memory usage grows over time linearly or spikes (pretty
>> sure the former).
>>
>> This is an index-heavy/query-light application, and we're hosing a
>> _bunch_ of documents at the indexing process. But still, assuming that
>> the memory reporting above is accurate, each
>> CompressingStoredFieldReader is apparently taking up 10M each and
>> there are over 700 of them. The documents are being batched (don't
>> quite know how many per batch, will find out).
>>
>> Mainly, I'm asking if this rings any bells. I don't see anything on a
>> quick look at the JIRAs.
>>
>> Thanks,
>> Erick
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Compressed field reader and memory problems with atomic updates
Posted by Savia Beson <ek...@googlemail.com>.
Maybe completely wrong here, but I would be ready to bet that you see high thread churn
We were bitten by it many times at other places. We have some memory heavy analyzers and once you experiance high thread-churn gc() cannot keep up anymore.
This was the reason to move to tomcat from jetty, because we know how to configure ThreadPools there to alleviate the problem (not perfect, just slightly better!)….
What Adrian said somehow points at similar pattern "reuses the buffer used for decompression across calls in the same thread"
solr should really do something OOTB to limit the number of used threads and reuse them, but this is container "turf" and increases dependancy …
On May 9, 2013, at 12:45 PM, Adrien Grand <jp...@gmail.com> wrote:
> Hi Erick,
>
> The stored fields reader reuses the buffer used for decompression across calls in the same thread, so I'm thinking that this kind of behaviour could happen if some documents are very large. Is it the case?
>
> Adrien
>
> Le 8 mai 2013 17:22, "Erick Erickson" <er...@gmail.com> a écrit :
> I'm seeing a case where (reported to me, not on my personal machine)
> where Solr's heap is being exhausted apparently by compressed field
> reader. Here's a sample line from a summary of a heap dump:
>
> CompressingStoredFieldReader 735 instances
>
> where the CompressingStoredFieldReader instances are taking up 7.5G of
> heap space (out of 8 allocated). The GC cycles are a pattern I've seen
> before
> gc for a long time, recover a few meg
> run for a _very_ short time
> gc again, recover a few meg
>
> This app happens to be doing a lot of atomic updates, running with CMS
> collector, java 1.7. There is very little querying going on, and it is
> happening on several different machines. Solr 4.2.
>
> Now, there is a bit of custom update code involved, we're testing whether
> 1> if not compressing the stored fields changes things
> 2> if disabling the custom code changes things
> 3> if Solr 4.3 changes things.
> 4> whether memory usage grows over time linearly or spikes (pretty
> sure the former).
>
> This is an index-heavy/query-light application, and we're hosing a
> _bunch_ of documents at the indexing process. But still, assuming that
> the memory reporting above is accurate, each
> CompressingStoredFieldReader is apparently taking up 10M each and
> there are over 700 of them. The documents are being batched (don't
> quite know how many per batch, will find out).
>
> Mainly, I'm asking if this rings any bells. I don't see anything on a
> quick look at the JIRAs.
>
> Thanks,
> Erick
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
Re: Compressed field reader and memory problems with atomic updates
Posted by Adrien Grand <jp...@gmail.com>.
Hi Erick,
The stored fields reader reuses the buffer used for decompression across
calls in the same thread, so I'm thinking that this kind of behaviour could
happen if some documents are very large. Is it the case?
Adrien
Le 8 mai 2013 17:22, "Erick Erickson" <er...@gmail.com> a écrit :
> I'm seeing a case where (reported to me, not on my personal machine)
> where Solr's heap is being exhausted apparently by compressed field
> reader. Here's a sample line from a summary of a heap dump:
>
> CompressingStoredFieldReader 735 instances
>
> where the CompressingStoredFieldReader instances are taking up 7.5G of
> heap space (out of 8 allocated). The GC cycles are a pattern I've seen
> before
> gc for a long time, recover a few meg
> run for a _very_ short time
> gc again, recover a few meg
>
> This app happens to be doing a lot of atomic updates, running with CMS
> collector, java 1.7. There is very little querying going on, and it is
> happening on several different machines. Solr 4.2.
>
> Now, there is a bit of custom update code involved, we're testing whether
> 1> if not compressing the stored fields changes things
> 2> if disabling the custom code changes things
> 3> if Solr 4.3 changes things.
> 4> whether memory usage grows over time linearly or spikes (pretty
> sure the former).
>
> This is an index-heavy/query-light application, and we're hosing a
> _bunch_ of documents at the indexing process. But still, assuming that
> the memory reporting above is accurate, each
> CompressingStoredFieldReader is apparently taking up 10M each and
> there are over 700 of them. The documents are being batched (don't
> quite know how many per batch, will find out).
>
> Mainly, I'm asking if this rings any bells. I don't see anything on a
> quick look at the JIRAs.
>
> Thanks,
> Erick
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
Re: Compressed field reader and memory problems with atomic updates
Posted by Erick Erickson <er...@gmail.com>.
Since what we're seeing is increasing heap usage, it doesn't quite
smell like SOLR-4802 or SOLR-4589, but it's possible....
On Wed, May 8, 2013 at 11:21 AM, Erick Erickson <er...@gmail.com> wrote:
> I'm seeing a case where (reported to me, not on my personal machine)
> where Solr's heap is being exhausted apparently by compressed field
> reader. Here's a sample line from a summary of a heap dump:
>
> CompressingStoredFieldReader 735 instances
>
> where the CompressingStoredFieldReader instances are taking up 7.5G of
> heap space (out of 8 allocated). The GC cycles are a pattern I've seen
> before
> gc for a long time, recover a few meg
> run for a _very_ short time
> gc again, recover a few meg
>
> This app happens to be doing a lot of atomic updates, running with CMS
> collector, java 1.7. There is very little querying going on, and it is
> happening on several different machines. Solr 4.2.
>
> Now, there is a bit of custom update code involved, we're testing whether
> 1> if not compressing the stored fields changes things
> 2> if disabling the custom code changes things
> 3> if Solr 4.3 changes things.
> 4> whether memory usage grows over time linearly or spikes (pretty
> sure the former).
>
> This is an index-heavy/query-light application, and we're hosing a
> _bunch_ of documents at the indexing process. But still, assuming that
> the memory reporting above is accurate, each
> CompressingStoredFieldReader is apparently taking up 10M each and
> there are over 700 of them. The documents are being batched (don't
> quite know how many per batch, will find out).
>
> Mainly, I'm asking if this rings any bells. I don't see anything on a
> quick look at the JIRAs.
>
> Thanks,
> Erick
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org