You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Gili Nachum <gi...@gmail.com> on 2013/03/11 10:08:04 UTC
Should heap size be proportionate to the size of the index I'm opening?
Hello.
I'm getting an OOME with a heap size of 512MB while trying to open an
IndexReader over a 70GB index (FSDirectory).
It didn't happen before when the index was smaller, and it doesn't happen
if Increase the heap size above 512MB.
Does this mean that the required heap size is always proportionate to the
size of the index?
Linux, Lucene 3.0.3 (yeah it's old...).
Re: Should heap size be proportionate to the size of the index I'm opening?
Posted by Gili Nachum <gi...@gmail.com>.
Great links. Thanks Ian.
Good to know that Lucene v4, has a smaller heap foot print.
On Mon, Mar 11, 2013 at 11:18 AM, Ian Lea <ia...@gmail.com> wrote:
> It's not that simple. More to do with number of terms than raw index
> size. Of course your large index may well have more terms than a
> smaller one.
>
> See
> http://blog.mikemccandless.com/2010/07/lucenes-ram-usage-for-searching.html
> and
> http://searchhub.org/2011/09/14/estimating-memory-and-storage-for-lucenesolr/
>
> They'll be about versions more recent than 3.0.3 but the underlying
> concepts may still be applicable.
>
> 512Mb for a 70Gb index sounds very conservative.
>
>
>
> --
> Ian.
>
>
> On Mon, Mar 11, 2013 at 9:08 AM, Gili Nachum <gi...@gmail.com> wrote:
> > Hello.
> >
> > I'm getting an OOME with a heap size of 512MB while trying to open an
> > IndexReader over a 70GB index (FSDirectory).
> > It didn't happen before when the index was smaller, and it doesn't happen
> > if Increase the heap size above 512MB.
> >
> > Does this mean that the required heap size is always proportionate to the
> > size of the index?
> >
> > Linux, Lucene 3.0.3 (yeah it's old...).
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Should heap size be proportionate to the size of the index I'm opening?
Posted by Ian Lea <ia...@gmail.com>.
It's not that simple. More to do with number of terms than raw index
size. Of course your large index may well have more terms than a
smaller one.
See http://blog.mikemccandless.com/2010/07/lucenes-ram-usage-for-searching.html
and http://searchhub.org/2011/09/14/estimating-memory-and-storage-for-lucenesolr/
They'll be about versions more recent than 3.0.3 but the underlying
concepts may still be applicable.
512Mb for a 70Gb index sounds very conservative.
--
Ian.
On Mon, Mar 11, 2013 at 9:08 AM, Gili Nachum <gi...@gmail.com> wrote:
> Hello.
>
> I'm getting an OOME with a heap size of 512MB while trying to open an
> IndexReader over a 70GB index (FSDirectory).
> It didn't happen before when the index was smaller, and it doesn't happen
> if Increase the heap size above 512MB.
>
> Does this mean that the required heap size is always proportionate to the
> size of the index?
>
> Linux, Lucene 3.0.3 (yeah it's old...).
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org