You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Niels Ott <no...@sfs.uni-tuebingen.de> on 2009/03/11 18:47:07 UTC
Memory during Indexing
Hi Lucene professionals!
This may sound like a dumb beginner's question, but anyways: Can Lucene
run out of memory during indexing?
Should I use IndexWriter.flush() or .commit(), and if so, how often?
Thank you for your support.
Niels
--
Niels Ott
Computational Linguist (B.A.)
http://www.drni.de/niels/
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Memory during Indexing
Posted by Grant Ingersoll <gs...@apache.org>.
On Mar 12, 2009, at 10:47 AM, Niels Ott wrote:
> Michael McCandless schrieb:
>> When RAM is full, IW flushes the pending changes to disk, but does
>> not commit them, meaning external (newly opened or reopened)
>> readers will not see the changes.
>
> Is there a built-in mechanism in the IndexReader to reload the index
> every now and then, after having checked that it has changed?
>
>> You must commit() or close() (which just calls commit() before
>> closing) for external readers to see the changes.
>
> So there is no means to do this automatically? I mean, I can count
> documents while adding them and do commit() every now and then, or I
> could commit every time one hour of indexing has passed, etc...
It's called Solr ;-)
Solr has controls for all of these things. I know that isn't
necessarily satisfactory and that you may not want to switch, but you
can always go look at the code for how it is implemented. Otherwise,
Lucene-wise, you have to manage the lifecycle of the readers,
committing, etc..
-Grant
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Memory during Indexing
Posted by Niels Ott <no...@sfs.uni-tuebingen.de>.
Michael McCandless schrieb:
> When RAM is full, IW flushes the pending changes to disk, but does not
> commit them, meaning external (newly opened or reopened) readers will
> not see the changes.
Is there a built-in mechanism in the IndexReader to reload the index
every now and then, after having checked that it has changed?
> You must commit() or close() (which just calls commit() before closing)
> for external readers to see the changes.
So there is no means to do this automatically? I mean, I can count
documents while adding them and do commit() every now and then, or I
could commit every time one hour of indexing has passed, etc...
Best,
Niels
--
Niels Ott
Computational Linguist (B.A.)
http://www.drni.de/niels/
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Memory during Indexing
Posted by Michael McCandless <lu...@mikemccandless.com>.
Niels Ott wrote:
> Hi Mark,
>
> markharw00d schrieb:
>> Hi Niels,
>> See the javadocs for IndexWriter.setRAMBufferSizeMB()
>
> I tried different settings. Apart from the fact that my memory issue
> seems to by my own fault, I'm wondering what Lucene does in the
> background. Apparently it does flush(), but not commit()?
>
> At least my index remains empty when I'm querying. If I call
> commit() every now and then (like every 500 documents), there are
> data I can query.
That's right.
When RAM is full, IW flushes the pending changes to disk, but does not
commit them, meaning external (newly opened or reopened) readers will
not see the changes.
You must commit() or close() (which just calls commit() before
closing) for external readers to see the changes.
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Memory during Indexing
Posted by Niels Ott <no...@sfs.uni-tuebingen.de>.
Hi Mark,
markharw00d schrieb:
> Hi Niels,
> See the javadocs for IndexWriter.setRAMBufferSizeMB()
I tried different settings. Apart from the fact that my memory issue
seems to by my own fault, I'm wondering what Lucene does in the
background. Apparently it does flush(), but not commit()?
At least my index remains empty when I'm querying. If I call commit()
every now and then (like every 500 documents), there are data I can query.
Best
Niels
--
Niels Ott
Computational Linguist (B.A.)
http://www.drni.de/niels/
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Memory during Indexing
Posted by markharw00d <ma...@yahoo.co.uk>.
Hi Niels,
See the javadocs for IndexWriter.setRAMBufferSizeMB()
Cheers
Mark
Niels Ott wrote:
> Hi Lucene professionals!
>
> This may sound like a dumb beginner's question, but anyways: Can
> Lucene run out of memory during indexing?
>
> Should I use IndexWriter.flush() or .commit(), and if so, how often?
>
> Thank you for your support.
>
> Niels
>
> ------------------------------------------------------------------------
>
>
> No virus found in this incoming message.
> Checked by AVG - www.avg.com
> Version: 8.0.237 / Virus Database: 270.11.10/1995 - Release Date: 03/11/09 08:28:00
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org