You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Jamie <ja...@stimulussoft.com> on 2010/03/16 23:00:05 UTC
OutOfMemory ParallelMultisearcher
Hi There
I have an index which is 36 GB large. When I perform eight simultaneous
searches (performed by JMeter) on the index, an OutOfMemory error
occurs. Since I need to potentially search across multiple indexes and
those indexes can change from one search query to the next, each user
has their own ParallelMultiSearcher object. Before each search
operation, I reconstruct the ParrallelMultisearcher with the appropriate
Searchers to each of the indexes that need to be included for that
particular search query.
The problem is that requiring each user to have their own
ParallelMultisearcher seems to limit the number of users that can use
the system at the same time.
While experimenting, when I make the ParallelMultiSearcher static, the
same object used by all users, the OutOfMemory problem goes away and I
am able to execute 50 simultaneous searches. The problem I have is I
cannot make ParallelMultisearcher static, since the specific indexes
used are variable from one search query to the next. I initially thought
one could just cache the underlying Searchers and all would be okay, but
this does not appear to be the case.
My question: Will ParallelMultisearcher tend to consume a large amount
of memory by itself when used on large indices? If so, do you have any
suggestions on how I might support the above scenario (i.e. when the
indexes used change from one query to the next)
Thanks in advance
Jamie
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: OutOfMemory ParallelMultisearcher
Posted by Jamie <ja...@stimulussoft.com>.
Hi Ian
Thanks for the info. Its difficult to reuse searchers as my users are
performing realtime searches, so I need to open an IndexReader for every
live search query.
I've since tracked the OutOfMemory issue down to sort on date. I am
using too high a precision (down to the second) which is causing the
number of search terms to escalate. Also, I haven't since moved the date
field in the index over to using Numerics instead of a String value for
fear of breaking compatibility with my old index format.
I dont yet quite understand the implications of the precisionStep and
what it all means. If I change my date string to a Numeric integer in
the format yyyyMMddHHmm, what should the precisionStep value be?
Jamie
On 2010/03/17 01:20 PM, Ian Lea wrote:
> Hi
>
>
> Caching searchers at some level should help keep memory usage down -
> and will help performance too. Searchers themselves don't generally
> consume large amounts of memory, but if you've got loads of them then
> obviously things will add up.
>
> Unless you can change the whole design of your app (single index with
> a user field that you use as a query filter to restrict users to
> "their" data?) you may be stuck with giving the app more memory or
> restricting the number of concurrent searchers.
>
>
> ---
> Ian.
>
>
> On Tue, Mar 16, 2010 at 10:00 PM, Jamie<ja...@stimulussoft.com> wrote:
>
>> Hi There
>>
>> I have an index which is 36 GB large. When I perform eight simultaneous
>> searches (performed by JMeter) on the index, an OutOfMemory error occurs.
>> Since I need to potentially search across multiple indexes and those indexes
>> can change from one search query to the next, each user has their own
>> ParallelMultiSearcher object. Before each search operation, I reconstruct
>> the ParrallelMultisearcher with the appropriate Searchers to each of the
>> indexes that need to be included for that particular search query.
>>
>> The problem is that requiring each user to have their own
>> ParallelMultisearcher seems to limit the number of users that can use the
>> system at the same time.
>>
>> While experimenting, when I make the ParallelMultiSearcher static, the same
>> object used by all users, the OutOfMemory problem goes away and I am able to
>> execute 50 simultaneous searches. The problem I have is I cannot make
>> ParallelMultisearcher static, since the specific indexes used are variable
>> from one search query to the next. I initially thought one could just cache
>> the underlying Searchers and all would be okay, but this does not appear to
>> be the case.
>>
>> My question: Will ParallelMultisearcher tend to consume a large amount of
>> memory by itself when used on large indices? If so, do you have any
>> suggestions on how I might support the above scenario (i.e. when the indexes
>> used change from one query to the next)
>>
>> Thanks in advance
>>
>> Jamie
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: OutOfMemory ParallelMultisearcher
Posted by Ian Lea <ia...@gmail.com>.
Hi
Caching searchers at some level should help keep memory usage down -
and will help performance too. Searchers themselves don't generally
consume large amounts of memory, but if you've got loads of them then
obviously things will add up.
Unless you can change the whole design of your app (single index with
a user field that you use as a query filter to restrict users to
"their" data?) you may be stuck with giving the app more memory or
restricting the number of concurrent searchers.
---
Ian.
On Tue, Mar 16, 2010 at 10:00 PM, Jamie <ja...@stimulussoft.com> wrote:
> Hi There
>
> I have an index which is 36 GB large. When I perform eight simultaneous
> searches (performed by JMeter) on the index, an OutOfMemory error occurs.
> Since I need to potentially search across multiple indexes and those indexes
> can change from one search query to the next, each user has their own
> ParallelMultiSearcher object. Before each search operation, I reconstruct
> the ParrallelMultisearcher with the appropriate Searchers to each of the
> indexes that need to be included for that particular search query.
>
> The problem is that requiring each user to have their own
> ParallelMultisearcher seems to limit the number of users that can use the
> system at the same time.
>
> While experimenting, when I make the ParallelMultiSearcher static, the same
> object used by all users, the OutOfMemory problem goes away and I am able to
> execute 50 simultaneous searches. The problem I have is I cannot make
> ParallelMultisearcher static, since the specific indexes used are variable
> from one search query to the next. I initially thought one could just cache
> the underlying Searchers and all would be okay, but this does not appear to
> be the case.
>
> My question: Will ParallelMultisearcher tend to consume a large amount of
> memory by itself when used on large indices? If so, do you have any
> suggestions on how I might support the above scenario (i.e. when the indexes
> used change from one query to the next)
>
> Thanks in advance
>
> Jamie
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org