You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Mukul Ranjan <mr...@egain.com> on 2016/12/21 12:37:46 UTC
CompressingStoredFieldReader object taking lot of heap space
Hi,
We are using lucene 5.5.2 for search.
We ran load test for 8 hours on our setup and observed that "org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader" object is taking 210 MB out of 1.2 GB heap space. There are 8890 instances of this class present in the heap.
Could you please confirm if this is the normal behaviour or there is some issue with the code.
Thanks,
Mukul Ranjan
-----
Watch an eGain Try+Buy(tm) customer success video<https://www.youtube.com/watch?v=ewEeyeqJRCE&feature=youtu.be/>
Re: CompressingStoredFieldReader object taking lot of heap space
Posted by Adrien Grand <jp...@gmail.com>.
I don't know anything about JBoss but I am pretty sure this is something
that can be configured. A quick Google search suggests the default thread
pool size is 512 * number_of_processors, which is indeed likely too high
for Lucene: https://developer.jboss.org/thread/267570. Looks like it can be
configured to have a lower max number of threads.
Le ven. 23 déc. 2016 à 14:14, Mukul Ranjan <mr...@egain.com> a écrit :
> Thanks Adrien for quick reply!
>
>
>
> We use jboss app server to handle application level threads.
>
>
>
> Thanks,
>
> Mukul Ranjan
>
>
>
> *From:* Adrien Grand [mailto:jpountz@gmail.com]
> *Sent:* Friday, December 23, 2016 6:33 PM
> *To:* Mukul Ranjan <mr...@egain.com>; Lucene Users Mailing List <
> java-user@lucene.apache.org>
>
>
> *Subject:* Re: CompressingStoredFieldReader object taking lot of heap
> space
>
>
>
> Hi,
>
> SearcherManager only cares about managing readers/searchers in a
> thread-safe way, it does not know about the thread pool it runs in.
>
> How does your application handle threads? I'm thinking you might be
> running in a servlet container, or a similar framework that handles it for
> you?
>
>
>
> Le ven. 23 déc. 2016 à 13:57, Mukul Ranjan <mr...@egain.com> a écrit :
>
> Hi Adrien,
>
>
>
> We are using searcher manager for acquiring and releasing index searcher
> object. Is there any way we can control fixed thread pool using searcher
> Manager?
>
>
>
> Thanks,
> Mukul Ranjan
>
>
>
> *From:* Adrien Grand [mailto:jpountz@gmail.com]
> *Sent:* Thursday, December 22, 2016 11:25 PM
> *To:* Mukul Ranjan <mr...@egain.com>
> *Cc:* java-user@lucene.apache.org
>
>
> *Subject:* Re: CompressingStoredFieldReader object taking lot of heap
> space
>
>
>
> Hi Mukul,
>
> Lucene maintains a number of threadlocals alongside each segment. While
> these structures should remain small (a couple tens of KB at most), the
> fact we have them per segment and in threadlocals means that overall memory
> usage for these structures is linear with the number of threads that have
> access to the index and to the number of segments that your index has.
>
> So you plan on having access to an index from multiple thread, you should
> make sure it is a fixed thread pool that has a reasonable number of threads
> (in the order of the number of parallel threads that the host can run).
>
>
>
> Le jeu. 22 déc. 2016 à 07:25, Mukul Ranjan <mr...@egain.com> a écrit :
>
> Hi Adrien,
>
> Could you please explain the 2nd point in detail?
>
> Thanks,
> Mukul Ranjan
>
> -----Original Message-----
> From: Adrien Grand [mailto:jpountz@gmail.com]
> Sent: Wednesday, December 21, 2016 6:43 PM
> To: java-user@lucene.apache.org
> Subject: Re: CompressingStoredFieldReader object taking lot of heap space
>
> This issue has been reported a couple times and is usually due to one of
> these issues:
> - readers are not always closed
> - the index is accessed from a non fixed thread pool
> - there are too many indices open at the same time
>
> Le mer. 21 déc. 2016 à 13:37, Mukul Ranjan <mr...@egain.com> a écrit :
>
> > Hi,
> >
> > We are using lucene 5.5.2 for search.
> >
> > We ran load test for 8 hours on our setup and observed that
> > "org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader"
> > object is taking 210 MB out of 1.2 GB heap space. There are 8890
> > instances of this class present in the heap.
> >
> > Could you please confirm if this is the normal behaviour or there is
> > some issue with the code.
> >
> > Thanks,
> > Mukul Ranjan
> >
> > -----
> > Watch an eGain Try+Buy(tm) customer success video<
> > https://www.youtube.com/watch?v=ewEeyeqJRCE&feature=youtu.be/>
> >
>
>
RE: CompressingStoredFieldReader object taking lot of heap space
Posted by Mukul Ranjan <mr...@egain.com>.
Thanks Adrien for quick reply!
We use jboss app server to handle application level threads.
Thanks,
Mukul Ranjan
From: Adrien Grand [mailto:jpountz@gmail.com]
Sent: Friday, December 23, 2016 6:33 PM
To: Mukul Ranjan <mr...@egain.com>; Lucene Users Mailing List <ja...@lucene.apache.org>
Subject: Re: CompressingStoredFieldReader object taking lot of heap space
Hi,
SearcherManager only cares about managing readers/searchers in a thread-safe way, it does not know about the thread pool it runs in.
How does your application handle threads? I'm thinking you might be running in a servlet container, or a similar framework that handles it for you?
Le ven. 23 déc. 2016 à 13:57, Mukul Ranjan <mr...@egain.com>> a écrit :
Hi Adrien,
We are using searcher manager for acquiring and releasing index searcher object. Is there any way we can control fixed thread pool using searcher Manager?
Thanks,
Mukul Ranjan
From: Adrien Grand [mailto:jpountz@gmail.com<ma...@gmail.com>]
Sent: Thursday, December 22, 2016 11:25 PM
To: Mukul Ranjan <mr...@egain.com>>
Cc: java-user@lucene.apache.org<ma...@lucene.apache.org>
Subject: Re: CompressingStoredFieldReader object taking lot of heap space
Hi Mukul,
Lucene maintains a number of threadlocals alongside each segment. While these structures should remain small (a couple tens of KB at most), the fact we have them per segment and in threadlocals means that overall memory usage for these structures is linear with the number of threads that have access to the index and to the number of segments that your index has.
So you plan on having access to an index from multiple thread, you should make sure it is a fixed thread pool that has a reasonable number of threads (in the order of the number of parallel threads that the host can run).
Le jeu. 22 déc. 2016 à 07:25, Mukul Ranjan <mr...@egain.com>> a écrit :
Hi Adrien,
Could you please explain the 2nd point in detail?
Thanks,
Mukul Ranjan
-----Original Message-----
From: Adrien Grand [mailto:jpountz@gmail.com<ma...@gmail.com>]
Sent: Wednesday, December 21, 2016 6:43 PM
To: java-user@lucene.apache.org<ma...@lucene.apache.org>
Subject: Re: CompressingStoredFieldReader object taking lot of heap space
This issue has been reported a couple times and is usually due to one of these issues:
- readers are not always closed
- the index is accessed from a non fixed thread pool
- there are too many indices open at the same time
Le mer. 21 déc. 2016 à 13:37, Mukul Ranjan <mr...@egain.com>> a écrit :
> Hi,
>
> We are using lucene 5.5.2 for search.
>
> We ran load test for 8 hours on our setup and observed that
> "org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader"
> object is taking 210 MB out of 1.2 GB heap space. There are 8890
> instances of this class present in the heap.
>
> Could you please confirm if this is the normal behaviour or there is
> some issue with the code.
>
> Thanks,
> Mukul Ranjan
>
> -----
> Watch an eGain Try+Buy(tm) customer success video<
> https://www.youtube.com/watch?v=ewEeyeqJRCE&feature=youtu.be/>
>
Re: CompressingStoredFieldReader object taking lot of heap space
Posted by Adrien Grand <jp...@gmail.com>.
Hi,
SearcherManager only cares about managing readers/searchers in a
thread-safe way, it does not know about the thread pool it runs in.
How does your application handle threads? I'm thinking you might be running
in a servlet container, or a similar framework that handles it for you?
Le ven. 23 déc. 2016 à 13:57, Mukul Ranjan <mr...@egain.com> a écrit :
> Hi Adrien,
>
>
>
> We are using searcher manager for acquiring and releasing index searcher
> object. Is there any way we can control fixed thread pool using searcher
> Manager?
>
>
>
> Thanks,
> Mukul Ranjan
>
>
>
> *From:* Adrien Grand [mailto:jpountz@gmail.com]
> *Sent:* Thursday, December 22, 2016 11:25 PM
> *To:* Mukul Ranjan <mr...@egain.com>
> *Cc:* java-user@lucene.apache.org
>
>
> *Subject:* Re: CompressingStoredFieldReader object taking lot of heap
> space
>
>
>
> Hi Mukul,
>
> Lucene maintains a number of threadlocals alongside each segment. While
> these structures should remain small (a couple tens of KB at most), the
> fact we have them per segment and in threadlocals means that overall memory
> usage for these structures is linear with the number of threads that have
> access to the index and to the number of segments that your index has.
>
> So you plan on having access to an index from multiple thread, you should
> make sure it is a fixed thread pool that has a reasonable number of threads
> (in the order of the number of parallel threads that the host can run).
>
>
>
> Le jeu. 22 déc. 2016 à 07:25, Mukul Ranjan <mr...@egain.com> a écrit :
>
> Hi Adrien,
>
> Could you please explain the 2nd point in detail?
>
> Thanks,
> Mukul Ranjan
>
> -----Original Message-----
> From: Adrien Grand [mailto:jpountz@gmail.com]
> Sent: Wednesday, December 21, 2016 6:43 PM
> To: java-user@lucene.apache.org
> Subject: Re: CompressingStoredFieldReader object taking lot of heap space
>
> This issue has been reported a couple times and is usually due to one of
> these issues:
> - readers are not always closed
> - the index is accessed from a non fixed thread pool
> - there are too many indices open at the same time
>
> Le mer. 21 déc. 2016 à 13:37, Mukul Ranjan <mr...@egain.com> a écrit :
>
> > Hi,
> >
> > We are using lucene 5.5.2 for search.
> >
> > We ran load test for 8 hours on our setup and observed that
> > "org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader"
> > object is taking 210 MB out of 1.2 GB heap space. There are 8890
> > instances of this class present in the heap.
> >
> > Could you please confirm if this is the normal behaviour or there is
> > some issue with the code.
> >
> > Thanks,
> > Mukul Ranjan
> >
> > -----
> > Watch an eGain Try+Buy(tm) customer success video<
> > https://www.youtube.com/watch?v=ewEeyeqJRCE&feature=youtu.be/>
> >
>
>
Re: CompressingStoredFieldReader object taking lot of heap space
Posted by Adrien Grand <jp...@gmail.com>.
Hi Mukul,
Lucene maintains a number of threadlocals alongside each segment. While
these structures should remain small (a couple tens of KB at most), the
fact we have them per segment and in threadlocals means that overall memory
usage for these structures is linear with the number of threads that have
access to the index and to the number of segments that your index has.
So you plan on having access to an index from multiple thread, you should
make sure it is a fixed thread pool that has a reasonable number of threads
(in the order of the number of parallel threads that the host can run).
Le jeu. 22 déc. 2016 à 07:25, Mukul Ranjan <mr...@egain.com> a écrit :
> Hi Adrien,
>
> Could you please explain the 2nd point in detail?
>
> Thanks,
> Mukul Ranjan
>
> -----Original Message-----
> From: Adrien Grand [mailto:jpountz@gmail.com]
> Sent: Wednesday, December 21, 2016 6:43 PM
> To: java-user@lucene.apache.org
> Subject: Re: CompressingStoredFieldReader object taking lot of heap space
>
> This issue has been reported a couple times and is usually due to one of
> these issues:
> - readers are not always closed
> - the index is accessed from a non fixed thread pool
> - there are too many indices open at the same time
>
> Le mer. 21 déc. 2016 à 13:37, Mukul Ranjan <mr...@egain.com> a écrit :
>
> > Hi,
> >
> > We are using lucene 5.5.2 for search.
> >
> > We ran load test for 8 hours on our setup and observed that
> > "org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader"
> > object is taking 210 MB out of 1.2 GB heap space. There are 8890
> > instances of this class present in the heap.
> >
> > Could you please confirm if this is the normal behaviour or there is
> > some issue with the code.
> >
> > Thanks,
> > Mukul Ranjan
> >
> > -----
> > Watch an eGain Try+Buy(tm) customer success video<
> > https://www.youtube.com/watch?v=ewEeyeqJRCE&feature=youtu.be/>
> >
>
RE: CompressingStoredFieldReader object taking lot of heap space
Posted by Mukul Ranjan <mr...@egain.com>.
Hi Adrien,
Could you please explain the 2nd point in detail?
Thanks,
Mukul Ranjan
-----Original Message-----
From: Adrien Grand [mailto:jpountz@gmail.com]
Sent: Wednesday, December 21, 2016 6:43 PM
To: java-user@lucene.apache.org
Subject: Re: CompressingStoredFieldReader object taking lot of heap space
This issue has been reported a couple times and is usually due to one of these issues:
- readers are not always closed
- the index is accessed from a non fixed thread pool
- there are too many indices open at the same time
Le mer. 21 déc. 2016 à 13:37, Mukul Ranjan <mr...@egain.com> a écrit :
> Hi,
>
> We are using lucene 5.5.2 for search.
>
> We ran load test for 8 hours on our setup and observed that
> "org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader"
> object is taking 210 MB out of 1.2 GB heap space. There are 8890
> instances of this class present in the heap.
>
> Could you please confirm if this is the normal behaviour or there is
> some issue with the code.
>
> Thanks,
> Mukul Ranjan
>
> -----
> Watch an eGain Try+Buy(tm) customer success video<
> https://www.youtube.com/watch?v=ewEeyeqJRCE&feature=youtu.be/>
>
Re: CompressingStoredFieldReader object taking lot of heap space
Posted by Adrien Grand <jp...@gmail.com>.
This issue has been reported a couple times and is usually due to one of
these issues:
- readers are not always closed
- the index is accessed from a non fixed thread pool
- there are too many indices open at the same time
Le mer. 21 déc. 2016 à 13:37, Mukul Ranjan <mr...@egain.com> a écrit :
> Hi,
>
> We are using lucene 5.5.2 for search.
>
> We ran load test for 8 hours on our setup and observed that
> "org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader" object
> is taking 210 MB out of 1.2 GB heap space. There are 8890 instances of
> this class present in the heap.
>
> Could you please confirm if this is the normal behaviour or there is some
> issue with the code.
>
> Thanks,
> Mukul Ranjan
>
> -----
> Watch an eGain Try+Buy(tm) customer success video<
> https://www.youtube.com/watch?v=ewEeyeqJRCE&feature=youtu.be/>
>