How to determine memory required for searching

["Trieu, Jason T" <Tr...@con-way.com>]

Hello,

I keep getting java.lang.OutOfMemoryError while doing search and would like to get some ideas to resolve the issue..

BACKGROUND:

IBM AIX server with 8 GIG of memory.
Lucene index contains 625 million documents with each having roughly 200 bytes of stored fields/data.
Running Lucene IndexSearcher  in JVM with -Xms3072m -Xmx3072m (more  memory than that then the system kills the JVM).

PROBLEM:

I can search successfully search the index if I do not user sort or filtering.   Although I need to invoke IndexReader with a termInfosIndexDivisor of at least 100 (IndexReader.open(dir,NoDeletionPolicy.INSTANCE,true,100)
If I include a sort ( of a numeric field) or a numeric range filter, I get OutOfMemoryError no matter how big the termInfosIndexDivisor is.  I've tried it as big as 100,000,000 and still got OutOfMemoryError.
It seems I am running against hard limits of resources.  Perhaps 8 GB of memory is just simply not enough to handle an index of 600 million documents.  But before telling management that they must get more memory, I'd to see if there might be other ways to accomplish this.
Thanks in advance.

Jason

Re: How to determine memory required for searching

[Ian Lea <ia...@gmail.com>]

> Running Lucene IndexSearcher  in JVM with -Xms3072m -Xmx3072m (more  memory than that then the system kills the JVM).

I'd try and fix that first.  Use a different JVM?  No point in buying
more memory if you can't even use all you've got now.

What version of lucene are you using?
http://blog.mikemccandless.com/2010/07/lucenes-ram-usage-for-searching.html
implies you might be better off with a bleeding edge version.

Or maybe you need to look at splitting your index.

--
Ian.


On Thu, Aug 4, 2011 at 10:41 PM, Trieu, Jason T <Tr...@con-way.com> wrote:
> Hello,
>
> I keep getting java.lang.OutOfMemoryError while doing search and would like to get some ideas to resolve the issue..
>
> BACKGROUND:
>
> IBM AIX server with 8 GIG of memory.
> Lucene index contains 625 million documents with each having roughly 200 bytes of stored fields/data.
> Running Lucene IndexSearcher  in JVM with -Xms3072m -Xmx3072m (more  memory than that then the system kills the JVM).
>
> PROBLEM:
>
> I can search successfully search the index if I do not user sort or filtering.   Although I need to invoke IndexReader with a termInfosIndexDivisor of at least 100 (IndexReader.open(dir,NoDeletionPolicy.INSTANCE,true,100)
> If I include a sort ( of a numeric field) or a numeric range filter, I get OutOfMemoryError no matter how big the termInfosIndexDivisor is.  I've tried it as big as 100,000,000 and still got OutOfMemoryError.
> It seems I am running against hard limits of resources.  Perhaps 8 GB of memory is just simply not enough to handle an index of 600 million documents.  But before telling management that they must get more memory, I'd to see if there might be other ways to accomplish this.
> Thanks in advance.
>
> Jason
>
>
>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org