You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Dennis Berger <de...@metaspinner.de> on 2007/03/14 13:26:52 UTC
memory consumption on large indices
Do I have to keep something in mind to do searching on large indices?
I actually have an index with a size of 1.8gb. I have indexed 1.5
million items from Amazon.
How much memory do I have to give to the jvm?
As a sidenote I have to tell you that I optimized the index so it's one
segment file.
Do I need to have 1.8gb memory available for the jvm?
regards,
-Dennis
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: memory consumption on large indices
Posted by Ian Lea <ia...@gmail.com>.
When your app gets a java.lang.OutOfMemory exception.
--
Ian.
On 3/14/07, Dennis Berger <db...@bsdsystems.de> wrote:
> Ian Lea schrieb:
> > No, you don't need 1.8Gb of memory. Start with default and raise if
> > you need to?
> how do I know when I need it?
> > Or jump straight in at about 512Mb.
> >
> >
> > --
> > Ian.
> >
> >
> > On 3/14/07, Dennis Berger <de...@metaspinner.de> wrote:
> >> Do I have to keep something in mind to do searching on large indices?
> >> I actually have an index with a size of 1.8gb. I have indexed 1.5
> >> million items from Amazon.
> >> How much memory do I have to give to the jvm?
> >> As a sidenote I have to tell you that I optimized the index so it's one
> >> segment file.
> >> Do I need to have 1.8gb memory available for the jvm?
> >>
> >> regards,
> >> -Dennis
> >>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: memory consumption on large indices
Posted by Dennis Berger <db...@bsdsystems.de>.
Ian Lea schrieb:
> No, you don't need 1.8Gb of memory. Start with default and raise if
> you need to?
how do I know when I need it?
> Or jump straight in at about 512Mb.
>
>
> --
> Ian.
>
>
> On 3/14/07, Dennis Berger <de...@metaspinner.de> wrote:
>> Do I have to keep something in mind to do searching on large indices?
>> I actually have an index with a size of 1.8gb. I have indexed 1.5
>> million items from Amazon.
>> How much memory do I have to give to the jvm?
>> As a sidenote I have to tell you that I optimized the index so it's one
>> segment file.
>> Do I need to have 1.8gb memory available for the jvm?
>>
>> regards,
>> -Dennis
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
--
Dennis Berger
BSDSystems
Eduardstrasse 43b
20257 Hamburg
Phone: +49 (0)40 54 00 18 17
Mobile: +49 (0) 179 123 15 09
E-Mail: db@bsdsystems.de
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: memory consumption on large indices
Posted by Ian Lea <ia...@gmail.com>.
No, you don't need 1.8Gb of memory. Start with default and raise if
you need to?
Or jump straight in at about 512Mb.
--
Ian.
On 3/14/07, Dennis Berger <de...@metaspinner.de> wrote:
> Do I have to keep something in mind to do searching on large indices?
> I actually have an index with a size of 1.8gb. I have indexed 1.5
> million items from Amazon.
> How much memory do I have to give to the jvm?
> As a sidenote I have to tell you that I optimized the index so it's one
> segment file.
> Do I need to have 1.8gb memory available for the jvm?
>
> regards,
> -Dennis
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: memory consumption on large indices
Posted by Tim Patton <tp...@dealcatcher.com>.
I'm searching a 20GB index and my searching JVM is allocated 1Gig.
However, my indexing app only had 384mb availible to it, which means you
can get away with far less. I believe certain index tables will need to
be swapped in and out of memory though so it may not search as quickly.
With a 1.8gig index you could try the jvm default (64megs) and see how
it works.
Tim
Dennis Berger wrote:
> Do I have to keep something in mind to do searching on large indices?
> I actually have an index with a size of 1.8gb. I have indexed 1.5
> million items from Amazon.
> How much memory do I have to give to the jvm?
> As a sidenote I have to tell you that I optimized the index so it's one
> segment file.
> Do I need to have 1.8gb memory available for the jvm?
>
> regards,
> -Dennis
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org