You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Rahul Chandwani <rc...@eGain.com> on 2016/10/12 05:58:13 UTC
Default LRUQueryCache causing OOO exception
Hello,
We are using lucene 5.5.2 for search .
We are getting out of memory exception as default LRUQueryCache is occupying more much more than default 32 MB memory.
Searcher Manager is used here to get an instance of index searcher to perform search. Default lru query cache and default query cache policy (UsageTrackingQueryCachingPolicy) is used.
Maximum memory usage/size is 3 GB for our application.
Observation from heap dump:
1. Lucene Codecs CompressingStoredFieldReader objects has consumed ~ 727 MB of heap
2. Lucene LRUQueryCache objects has consumed ~ 380 MB of heap. Cache instance is holding up large number of SegmentCoreReaders instances in Map.
Request you to please guide us.
Thanks
Rahul
___ Watch an eGain Customer Service Transformation<https://www.youtube.com/watch?v=dEwdFfWoyuE> Success Story
Re: Default LRUQueryCache causing OOO exception
Posted by Michael McCandless <lu...@mikemccandless.com>.
Thank you for bringing closure!
Mike McCandless
http://blog.mikemccandless.com
On Fri, Oct 14, 2016 at 2:41 AM, Rahul Chandwani <rc...@egain.com> wrote:
> Thanks Adrien, Micheal, Trejkaz.
>
>
> You guys were right.
>
> I was concentrating my energy at wrong place.
>
> We have two types of indexes:
>
> 1. File based directory: which we use for text search. Here we make use of searcher manager.
>
> 2. In memory directory: which we use for auto complete/auto suggestion. Here no searcher manager is used, we open new reader/searcher for every request. Same were not closed after use.
>
>
> Issue is resolved now.
>
>
> Thanks
>
> Rahul
>
> ________________________________
> From: Michael McCandless <lu...@mikemccandless.com>
> Sent: Thursday, October 13, 2016 2:38:46 PM
> To: Lucene Users
> Cc: Rahul Chandwani
> Subject: Re: Default LRUQueryCache causing OOO exception
>
> These sounds like good ideas Trejkaz ... maybe open an issue and make
> a starting patch so we can iterate?
>
> I agree a try-w-resources solution, and/or a lambda solution, could be better.
>
> Did you already share how you got it working for multiple indices?
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> On Wed, Oct 12, 2016 at 10:56 PM, Trejkaz <tr...@trypticon.org> wrote:
>> On Thu, Oct 13, 2016 at 6:32 AM, Michael McCandless
>> <lu...@mikemccandless.com> wrote:
>>> You must be calling SearcherManager.maybeRefresh periodically, which
>>> does open new NRT readers.
>>>
>>> Can you please triple check that you do in fact always release() after
>>> an acquire(), in a finally clause?
>>
>> For what it's worth, I found this API particularly hard to use.
>>
>> 1. I would prefer a withReader(Callback) kind of method to separate
>> methods to acquire and release. It makes it impossible to forget to
>> call the release method and now that lambdas are in the language, it
>> looks a hell of a lot tidier than try-finally.
>>
>> 2. If there has to be some kind of cleanup I'm supposed to perform on
>> an object, I prefer that to be done in close() so that I can use
>> try-with-resources like with any other object that I'm expected to
>> close.
>>
>> 2b. The API design is doubly bad, because the object it returns
>> *does* have a close() method, but "no, you're not allowed to call
>> that, you have to use this other method over here which almost every
>> developer on the team will get wrong every single time".
>>
>> 3. I wish there had been a version which could keep track of the same
>> stuff for multiple indexes, since getting that to work reliably has
>> been nearly impossible. (I think we're there right now, but I have no
>> way to prove it!)
>>
>> TX
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
> ___ Watch an eGain Customer Service Transformation<https://www.youtube.com/watch?v=dEwdFfWoyuE> Success Story
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Default LRUQueryCache causing OOO exception
Posted by Rahul Chandwani <rc...@eGain.com>.
Thanks Adrien, Micheal, Trejkaz.
You guys were right.
I was concentrating my energy at wrong place.
We have two types of indexes:
1. File based directory: which we use for text search. Here we make use of searcher manager.
2. In memory directory: which we use for auto complete/auto suggestion. Here no searcher manager is used, we open new reader/searcher for every request. Same were not closed after use.
Issue is resolved now.
Thanks
Rahul
________________________________
From: Michael McCandless <lu...@mikemccandless.com>
Sent: Thursday, October 13, 2016 2:38:46 PM
To: Lucene Users
Cc: Rahul Chandwani
Subject: Re: Default LRUQueryCache causing OOO exception
These sounds like good ideas Trejkaz ... maybe open an issue and make
a starting patch so we can iterate?
I agree a try-w-resources solution, and/or a lambda solution, could be better.
Did you already share how you got it working for multiple indices?
Mike McCandless
http://blog.mikemccandless.com
On Wed, Oct 12, 2016 at 10:56 PM, Trejkaz <tr...@trypticon.org> wrote:
> On Thu, Oct 13, 2016 at 6:32 AM, Michael McCandless
> <lu...@mikemccandless.com> wrote:
>> You must be calling SearcherManager.maybeRefresh periodically, which
>> does open new NRT readers.
>>
>> Can you please triple check that you do in fact always release() after
>> an acquire(), in a finally clause?
>
> For what it's worth, I found this API particularly hard to use.
>
> 1. I would prefer a withReader(Callback) kind of method to separate
> methods to acquire and release. It makes it impossible to forget to
> call the release method and now that lambdas are in the language, it
> looks a hell of a lot tidier than try-finally.
>
> 2. If there has to be some kind of cleanup I'm supposed to perform on
> an object, I prefer that to be done in close() so that I can use
> try-with-resources like with any other object that I'm expected to
> close.
>
> 2b. The API design is doubly bad, because the object it returns
> *does* have a close() method, but "no, you're not allowed to call
> that, you have to use this other method over here which almost every
> developer on the team will get wrong every single time".
>
> 3. I wish there had been a version which could keep track of the same
> stuff for multiple indexes, since getting that to work reliably has
> been nearly impossible. (I think we're there right now, but I have no
> way to prove it!)
>
> TX
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
___ Watch an eGain Customer Service Transformation<https://www.youtube.com/watch?v=dEwdFfWoyuE> Success Story
Re: Default LRUQueryCache causing OOO exception
Posted by Michael McCandless <lu...@mikemccandless.com>.
These sounds like good ideas Trejkaz ... maybe open an issue and make
a starting patch so we can iterate?
I agree a try-w-resources solution, and/or a lambda solution, could be better.
Did you already share how you got it working for multiple indices?
Mike McCandless
http://blog.mikemccandless.com
On Wed, Oct 12, 2016 at 10:56 PM, Trejkaz <tr...@trypticon.org> wrote:
> On Thu, Oct 13, 2016 at 6:32 AM, Michael McCandless
> <lu...@mikemccandless.com> wrote:
>> You must be calling SearcherManager.maybeRefresh periodically, which
>> does open new NRT readers.
>>
>> Can you please triple check that you do in fact always release() after
>> an acquire(), in a finally clause?
>
> For what it's worth, I found this API particularly hard to use.
>
> 1. I would prefer a withReader(Callback) kind of method to separate
> methods to acquire and release. It makes it impossible to forget to
> call the release method and now that lambdas are in the language, it
> looks a hell of a lot tidier than try-finally.
>
> 2. If there has to be some kind of cleanup I'm supposed to perform on
> an object, I prefer that to be done in close() so that I can use
> try-with-resources like with any other object that I'm expected to
> close.
>
> 2b. The API design is doubly bad, because the object it returns
> *does* have a close() method, but "no, you're not allowed to call
> that, you have to use this other method over here which almost every
> developer on the team will get wrong every single time".
>
> 3. I wish there had been a version which could keep track of the same
> stuff for multiple indexes, since getting that to work reliably has
> been nearly impossible. (I think we're there right now, but I have no
> way to prove it!)
>
> TX
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Default LRUQueryCache causing OOO exception
Posted by Trejkaz <tr...@trypticon.org>.
On Thu, Oct 13, 2016 at 6:32 AM, Michael McCandless
<lu...@mikemccandless.com> wrote:
> You must be calling SearcherManager.maybeRefresh periodically, which
> does open new NRT readers.
>
> Can you please triple check that you do in fact always release() after
> an acquire(), in a finally clause?
For what it's worth, I found this API particularly hard to use.
1. I would prefer a withReader(Callback) kind of method to separate
methods to acquire and release. It makes it impossible to forget to
call the release method and now that lambdas are in the language, it
looks a hell of a lot tidier than try-finally.
2. If there has to be some kind of cleanup I'm supposed to perform on
an object, I prefer that to be done in close() so that I can use
try-with-resources like with any other object that I'm expected to
close.
2b. The API design is doubly bad, because the object it returns
*does* have a close() method, but "no, you're not allowed to call
that, you have to use this other method over here which almost every
developer on the team will get wrong every single time".
3. I wish there had been a version which could keep track of the same
stuff for multiple indexes, since getting that to work reliably has
been nearly impossible. (I think we're there right now, but I have no
way to prove it!)
TX
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Default LRUQueryCache causing OOO exception
Posted by Michael McCandless <lu...@mikemccandless.com>.
You must be calling SearcherManager.maybeRefresh periodically, which
does open new NRT readers.
Can you please triple check that you do in fact always release() after
an acquire(), in a finally clause?
You have far far too many CompressingStoredFieldsReader instances in your JVM.
Mike McCandless
http://blog.mikemccandless.com
On Wed, Oct 12, 2016 at 11:16 AM, Rahul Chandwani <rc...@egain.com> wrote:
> We do not create readers on our own, that part is done/managed by searcher manager. Not sure if they are garbage collected.
>
> What do you mean by indices ?
>
> No, we are not using NoMergePolicy. Default merge policy is used.
>
>
> Thanks
>
> Rahul
>
> ________________________________
> From: Adrien Grand <jp...@gmail.com>
> Sent: Wednesday, October 12, 2016 8:22:01 PM
> To: Rahul Chandwani; java-user@lucene.apache.org
> Subject: Re: Default LRUQueryCache causing OOO exception
>
> After readers are released, are they also free to be garbage collected or
> is something holding references to them?
> How many indices do you have? Are you using something like the
> NoMergePolicy?
>
> Le mer. 12 oct. 2016 à 16:35, Rahul Chandwani <rc...@egain.com> a
> écrit :
>
>> Hi Adrien,
>>
>> Every acquire() call is followed by release() call.
>>
>> ------------------------------
>> *From:* Adrien Grand <jp...@gmail.com>
>> *Sent:* Wednesday, October 12, 2016 7:55:58 PM
>> *To:* Rahul Chandwani; java-user@lucene.apache.org
>>
>> *Subject:* Re: Default LRUQueryCache causing OOO exception
>> Hi Rahul,
>>
>> The most likely cause to your problem is that your are leaking IndexReader
>> instances. You say you are using SearcherManager, could you check that
>> there is a call to release matching every call to acquire?
>>
>> Le mer. 12 oct. 2016 à 15:02, Rahul Chandwani <rc...@egain.com> a
>> écrit :
>>
>> Hi Adrien,
>>
>> Total number of instances of CompressingStoredFieldsReader = 76,808
>> occupying 760 MB heap.
>> Total number of instances of SegmentCoreReaders = 38,099 occupying 368 MB
>> heap.
>>
>> Thanks
>>
>> -----Original Message-----
>> From: Adrien Grand [mailto:jpountz@gmail.com <jp...@gmail.com>]
>> Sent: Wednesday, October 12, 2016 12:51 PM
>> To: java-user@lucene.apache.org
>> Subject: Re: Default LRUQueryCache causing OOO exception
>>
>>
>> Hi Rahul,
>>
>> High memory from both these classes is unexpected. I am wondering that
>> maybe your application does not close index readers when it doesn't need
>> them anymore? Another possibility would be that you are dealing with lots
>> of IndexReader instances in the same JVM. Are you able to count how many
>> instance of CompressingStoredFieldReader were present in the heap?
>>
>>
>> Le mer. 12 oct. 2016 à 09:08, Rahul Chandwani <rc...@egain.com> a
>> écrit :
>>
>> > Hello,
>> >
>> >
>> > We are using lucene 5.5.2 for search .
>> >
>> > We are getting out of memory exception as default LRUQueryCache is
>> > occupying more much more than default 32 MB memory.
>> >
>> > Searcher Manager is used here to get an instance of index searcher to
>> > perform search. Default lru query cache and default query cache policy
>> > (UsageTrackingQueryCachingPolicy) is used.
>> >
>> > Maximum memory usage/size is 3 GB for our application.
>> >
>> > Observation from heap dump:
>> >
>> > 1. Lucene Codecs CompressingStoredFieldReader objects has consumed ~
>> > 727 MB of heap
>> >
>> > 2. Lucene LRUQueryCache objects has consumed ~ 380 MB of heap. Cache
>> > instance is holding up large number of SegmentCoreReaders instances in
>> Map.
>> >
>> >
>> > Request you to please guide us.
>> >
>> >
>> > Thanks
>> >
>> > Rahul
>> >
>> > ___ Watch an eGain Customer Service Transformation<
>> > https://www.youtube.com/watch?v=dEwdFfWoyuE> Success Story
>> >
>> ___ Watch an eGain Customer Service Transformation
>> <https://www.youtube.com/watch?v=dEwdFfWoyuE> Success Story
>>
>> ___ Watch an eGain Customer Service Transformation
>> <https://www.youtube.com/watch?v=dEwdFfWoyuE> Success Story
>>
> ___ Watch an eGain Customer Service Transformation<https://www.youtube.com/watch?v=dEwdFfWoyuE> Success Story
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Default LRUQueryCache causing OOO exception
Posted by Rahul Chandwani <rc...@eGain.com>.
We do not create readers on our own, that part is done/managed by searcher manager. Not sure if they are garbage collected.
What do you mean by indices ?
No, we are not using NoMergePolicy. Default merge policy is used.
Thanks
Rahul
________________________________
From: Adrien Grand <jp...@gmail.com>
Sent: Wednesday, October 12, 2016 8:22:01 PM
To: Rahul Chandwani; java-user@lucene.apache.org
Subject: Re: Default LRUQueryCache causing OOO exception
After readers are released, are they also free to be garbage collected or
is something holding references to them?
How many indices do you have? Are you using something like the
NoMergePolicy?
Le mer. 12 oct. 2016 à 16:35, Rahul Chandwani <rc...@egain.com> a
écrit :
> Hi Adrien,
>
> Every acquire() call is followed by release() call.
>
> ------------------------------
> *From:* Adrien Grand <jp...@gmail.com>
> *Sent:* Wednesday, October 12, 2016 7:55:58 PM
> *To:* Rahul Chandwani; java-user@lucene.apache.org
>
> *Subject:* Re: Default LRUQueryCache causing OOO exception
> Hi Rahul,
>
> The most likely cause to your problem is that your are leaking IndexReader
> instances. You say you are using SearcherManager, could you check that
> there is a call to release matching every call to acquire?
>
> Le mer. 12 oct. 2016 à 15:02, Rahul Chandwani <rc...@egain.com> a
> écrit :
>
> Hi Adrien,
>
> Total number of instances of CompressingStoredFieldsReader = 76,808
> occupying 760 MB heap.
> Total number of instances of SegmentCoreReaders = 38,099 occupying 368 MB
> heap.
>
> Thanks
>
> -----Original Message-----
> From: Adrien Grand [mailto:jpountz@gmail.com <jp...@gmail.com>]
> Sent: Wednesday, October 12, 2016 12:51 PM
> To: java-user@lucene.apache.org
> Subject: Re: Default LRUQueryCache causing OOO exception
>
>
> Hi Rahul,
>
> High memory from both these classes is unexpected. I am wondering that
> maybe your application does not close index readers when it doesn't need
> them anymore? Another possibility would be that you are dealing with lots
> of IndexReader instances in the same JVM. Are you able to count how many
> instance of CompressingStoredFieldReader were present in the heap?
>
>
> Le mer. 12 oct. 2016 à 09:08, Rahul Chandwani <rc...@egain.com> a
> écrit :
>
> > Hello,
> >
> >
> > We are using lucene 5.5.2 for search .
> >
> > We are getting out of memory exception as default LRUQueryCache is
> > occupying more much more than default 32 MB memory.
> >
> > Searcher Manager is used here to get an instance of index searcher to
> > perform search. Default lru query cache and default query cache policy
> > (UsageTrackingQueryCachingPolicy) is used.
> >
> > Maximum memory usage/size is 3 GB for our application.
> >
> > Observation from heap dump:
> >
> > 1. Lucene Codecs CompressingStoredFieldReader objects has consumed ~
> > 727 MB of heap
> >
> > 2. Lucene LRUQueryCache objects has consumed ~ 380 MB of heap. Cache
> > instance is holding up large number of SegmentCoreReaders instances in
> Map.
> >
> >
> > Request you to please guide us.
> >
> >
> > Thanks
> >
> > Rahul
> >
> > ___ Watch an eGain Customer Service Transformation<
> > https://www.youtube.com/watch?v=dEwdFfWoyuE> Success Story
> >
> ___ Watch an eGain Customer Service Transformation
> <https://www.youtube.com/watch?v=dEwdFfWoyuE> Success Story
>
> ___ Watch an eGain Customer Service Transformation
> <https://www.youtube.com/watch?v=dEwdFfWoyuE> Success Story
>
___ Watch an eGain Customer Service Transformation<https://www.youtube.com/watch?v=dEwdFfWoyuE> Success Story
Re: Default LRUQueryCache causing OOO exception
Posted by Adrien Grand <jp...@gmail.com>.
After readers are released, are they also free to be garbage collected or
is something holding references to them?
How many indices do you have? Are you using something like the
NoMergePolicy?
Le mer. 12 oct. 2016 à 16:35, Rahul Chandwani <rc...@egain.com> a
écrit :
> Hi Adrien,
>
> Every acquire() call is followed by release() call.
>
> ------------------------------
> *From:* Adrien Grand <jp...@gmail.com>
> *Sent:* Wednesday, October 12, 2016 7:55:58 PM
> *To:* Rahul Chandwani; java-user@lucene.apache.org
>
> *Subject:* Re: Default LRUQueryCache causing OOO exception
> Hi Rahul,
>
> The most likely cause to your problem is that your are leaking IndexReader
> instances. You say you are using SearcherManager, could you check that
> there is a call to release matching every call to acquire?
>
> Le mer. 12 oct. 2016 à 15:02, Rahul Chandwani <rc...@egain.com> a
> écrit :
>
> Hi Adrien,
>
> Total number of instances of CompressingStoredFieldsReader = 76,808
> occupying 760 MB heap.
> Total number of instances of SegmentCoreReaders = 38,099 occupying 368 MB
> heap.
>
> Thanks
>
> -----Original Message-----
> From: Adrien Grand [mailto:jpountz@gmail.com <jp...@gmail.com>]
> Sent: Wednesday, October 12, 2016 12:51 PM
> To: java-user@lucene.apache.org
> Subject: Re: Default LRUQueryCache causing OOO exception
>
>
> Hi Rahul,
>
> High memory from both these classes is unexpected. I am wondering that
> maybe your application does not close index readers when it doesn't need
> them anymore? Another possibility would be that you are dealing with lots
> of IndexReader instances in the same JVM. Are you able to count how many
> instance of CompressingStoredFieldReader were present in the heap?
>
>
> Le mer. 12 oct. 2016 à 09:08, Rahul Chandwani <rc...@egain.com> a
> écrit :
>
> > Hello,
> >
> >
> > We are using lucene 5.5.2 for search .
> >
> > We are getting out of memory exception as default LRUQueryCache is
> > occupying more much more than default 32 MB memory.
> >
> > Searcher Manager is used here to get an instance of index searcher to
> > perform search. Default lru query cache and default query cache policy
> > (UsageTrackingQueryCachingPolicy) is used.
> >
> > Maximum memory usage/size is 3 GB for our application.
> >
> > Observation from heap dump:
> >
> > 1. Lucene Codecs CompressingStoredFieldReader objects has consumed ~
> > 727 MB of heap
> >
> > 2. Lucene LRUQueryCache objects has consumed ~ 380 MB of heap. Cache
> > instance is holding up large number of SegmentCoreReaders instances in
> Map.
> >
> >
> > Request you to please guide us.
> >
> >
> > Thanks
> >
> > Rahul
> >
> > ___ Watch an eGain Customer Service Transformation<
> > https://www.youtube.com/watch?v=dEwdFfWoyuE> Success Story
> >
> ___ Watch an eGain Customer Service Transformation
> <https://www.youtube.com/watch?v=dEwdFfWoyuE> Success Story
>
> ___ Watch an eGain Customer Service Transformation
> <https://www.youtube.com/watch?v=dEwdFfWoyuE> Success Story
>
Re: Default LRUQueryCache causing OOO exception
Posted by Rahul Chandwani <rc...@eGain.com>.
Hi Adrien,
Every acquire() call is followed by release() call.
________________________________
From: Adrien Grand <jp...@gmail.com>
Sent: Wednesday, October 12, 2016 7:55:58 PM
To: Rahul Chandwani; java-user@lucene.apache.org
Subject: Re: Default LRUQueryCache causing OOO exception
Hi Rahul,
The most likely cause to your problem is that your are leaking IndexReader instances. You say you are using SearcherManager, could you check that there is a call to release matching every call to acquire?
Le mer. 12 oct. 2016 ? 15:02, Rahul Chandwani <rc...@egain.com>> a ?crit :
Hi Adrien,
Total number of instances of CompressingStoredFieldsReader = 76,808 occupying 760 MB heap.
Total number of instances of SegmentCoreReaders = 38,099 occupying 368 MB heap.
Thanks
-----Original Message-----
From: Adrien Grand [mailto:jpountz@gmail.com]
Sent: Wednesday, October 12, 2016 12:51 PM
To: java-user@lucene.apache.org<ma...@lucene.apache.org>
Subject: Re: Default LRUQueryCache causing OOO exception
Hi Rahul,
High memory from both these classes is unexpected. I am wondering that maybe your application does not close index readers when it doesn't need them anymore? Another possibility would be that you are dealing with lots of IndexReader instances in the same JVM. Are you able to count how many instance of CompressingStoredFieldReader were present in the heap?
Le mer. 12 oct. 2016 ? 09:08, Rahul Chandwani <rc...@egain.com>> a ?crit :
> Hello,
>
>
> We are using lucene 5.5.2 for search .
>
> We are getting out of memory exception as default LRUQueryCache is
> occupying more much more than default 32 MB memory.
>
> Searcher Manager is used here to get an instance of index searcher to
> perform search. Default lru query cache and default query cache policy
> (UsageTrackingQueryCachingPolicy) is used.
>
> Maximum memory usage/size is 3 GB for our application.
>
> Observation from heap dump:
>
> 1. Lucene Codecs CompressingStoredFieldReader objects has consumed ~
> 727 MB of heap
>
> 2. Lucene LRUQueryCache objects has consumed ~ 380 MB of heap. Cache
> instance is holding up large number of SegmentCoreReaders instances in Map.
>
>
> Request you to please guide us.
>
>
> Thanks
>
> Rahul
>
> ___ Watch an eGain Customer Service Transformation<
> https://www.youtube.com/watch?v=dEwdFfWoyuE> Success Story
>
___ Watch an eGain Customer Service Transformation<https://www.youtube.com/watch?v=dEwdFfWoyuE> Success Story
___ Watch an eGain Customer Service Transformation<https://www.youtube.com/watch?v=dEwdFfWoyuE> Success Story
Re: Default LRUQueryCache causing OOO exception
Posted by Adrien Grand <jp...@gmail.com>.
Hi Rahul,
The most likely cause to your problem is that your are leaking IndexReader
instances. You say you are using SearcherManager, could you check that
there is a call to release matching every call to acquire?
Le mer. 12 oct. 2016 à 15:02, Rahul Chandwani <rc...@egain.com> a
écrit :
> Hi Adrien,
>
> Total number of instances of CompressingStoredFieldsReader = 76,808
> occupying 760 MB heap.
> Total number of instances of SegmentCoreReaders = 38,099 occupying 368 MB
> heap.
>
> Thanks
>
> -----Original Message-----
> From: Adrien Grand [mailto:jpountz@gmail.com <jp...@gmail.com>]
> Sent: Wednesday, October 12, 2016 12:51 PM
> To: java-user@lucene.apache.org
> Subject: Re: Default LRUQueryCache causing OOO exception
>
>
> Hi Rahul,
>
> High memory from both these classes is unexpected. I am wondering that
> maybe your application does not close index readers when it doesn't need
> them anymore? Another possibility would be that you are dealing with lots
> of IndexReader instances in the same JVM. Are you able to count how many
> instance of CompressingStoredFieldReader were present in the heap?
>
>
> Le mer. 12 oct. 2016 à 09:08, Rahul Chandwani <rc...@egain.com> a
> écrit :
>
> > Hello,
> >
> >
> > We are using lucene 5.5.2 for search .
> >
> > We are getting out of memory exception as default LRUQueryCache is
> > occupying more much more than default 32 MB memory.
> >
> > Searcher Manager is used here to get an instance of index searcher to
> > perform search. Default lru query cache and default query cache policy
> > (UsageTrackingQueryCachingPolicy) is used.
> >
> > Maximum memory usage/size is 3 GB for our application.
> >
> > Observation from heap dump:
> >
> > 1. Lucene Codecs CompressingStoredFieldReader objects has consumed ~
> > 727 MB of heap
> >
> > 2. Lucene LRUQueryCache objects has consumed ~ 380 MB of heap. Cache
> > instance is holding up large number of SegmentCoreReaders instances in
> Map.
> >
> >
> > Request you to please guide us.
> >
> >
> > Thanks
> >
> > Rahul
> >
> > ___ Watch an eGain Customer Service Transformation<
> > https://www.youtube.com/watch?v=dEwdFfWoyuE> Success Story
> >
> ___ Watch an eGain Customer Service Transformation
> <https://www.youtube.com/watch?v=dEwdFfWoyuE> Success Story
>
Re: Default LRUQueryCache causing OOO exception
Posted by Rahul Chandwani <rc...@eGain.com>.
Hi Adrien,
Total number of instances of CompressingStoredFieldsReader = 76,808 occupying 760 MB heap.
Total number of instances of SegmentCoreReaders = 38,099 occupying 368 MB heap.
Thanks
-----Original Message-----
From: Adrien Grand [mailto:jpountz@gmail.com]
Sent: Wednesday, October 12, 2016 12:51 PM
To: java-user@lucene.apache.org
Subject: Re: Default LRUQueryCache causing OOO exception
Hi Rahul,
High memory from both these classes is unexpected. I am wondering that maybe your application does not close index readers when it doesn't need them anymore? Another possibility would be that you are dealing with lots of IndexReader instances in the same JVM. Are you able to count how many instance of CompressingStoredFieldReader were present in the heap?
Le mer. 12 oct. 2016 à 09:08, Rahul Chandwani <rc...@egain.com> a écrit :
> Hello,
>
>
> We are using lucene 5.5.2 for search .
>
> We are getting out of memory exception as default LRUQueryCache is
> occupying more much more than default 32 MB memory.
>
> Searcher Manager is used here to get an instance of index searcher to
> perform search. Default lru query cache and default query cache policy
> (UsageTrackingQueryCachingPolicy) is used.
>
> Maximum memory usage/size is 3 GB for our application.
>
> Observation from heap dump:
>
> 1. Lucene Codecs CompressingStoredFieldReader objects has consumed ~
> 727 MB of heap
>
> 2. Lucene LRUQueryCache objects has consumed ~ 380 MB of heap. Cache
> instance is holding up large number of SegmentCoreReaders instances in Map.
>
>
> Request you to please guide us.
>
>
> Thanks
>
> Rahul
>
> ___ Watch an eGain Customer Service Transformation<
> https://www.youtube.com/watch?v=dEwdFfWoyuE> Success Story
>
___ Watch an eGain Customer Service Transformation<https://www.youtube.com/watch?v=dEwdFfWoyuE> Success Story
Re: Default LRUQueryCache causing OOO exception
Posted by Adrien Grand <jp...@gmail.com>.
Hi Rahul,
High memory from both these classes is unexpected. I am wondering that
maybe your application does not close index readers when it doesn't need
them anymore? Another possibility would be that you are dealing with lots
of IndexReader instances in the same JVM. Are you able to count how many
instance of CompressingStoredFieldReader were present in the heap?
Le mer. 12 oct. 2016 à 09:08, Rahul Chandwani <rc...@egain.com> a
écrit :
> Hello,
>
>
> We are using lucene 5.5.2 for search .
>
> We are getting out of memory exception as default LRUQueryCache is
> occupying more much more than default 32 MB memory.
>
> Searcher Manager is used here to get an instance of index searcher to
> perform search. Default lru query cache and default query cache policy
> (UsageTrackingQueryCachingPolicy) is used.
>
> Maximum memory usage/size is 3 GB for our application.
>
> Observation from heap dump:
>
> 1. Lucene Codecs CompressingStoredFieldReader objects has consumed ~ 727
> MB of heap
>
> 2. Lucene LRUQueryCache objects has consumed ~ 380 MB of heap. Cache
> instance is holding up large number of SegmentCoreReaders instances in Map.
>
>
> Request you to please guide us.
>
>
> Thanks
>
> Rahul
>
> ___ Watch an eGain Customer Service Transformation<
> https://www.youtube.com/watch?v=dEwdFfWoyuE> Success Story
>