You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Heng Mei <he...@gmail.com> on 2006/07/06 01:58:03 UTC
IndexSearcher memory leak?
Hi experts,
There seems to be a strange memory leak with the IndexSearcher.
I get an OutOfMemoryException after a few iterations of the following loop:
LOOP:
ramdir = new RAMDirectory( "path/to/my/directory" );
searcher = new IndexSearcher( reader );
searcher.search(new TermQuery( new Term( "field", "keyword")));
searcher.close();
ramdir.close();
END LOOP:
Note, ramdir and searcher are both instance variables.
The index is about 100mb, and the process is run with -Xmx256m.
**Strangely enough, if I comment out the "searcher.search()" line,
then there is no error. Or, if I make searcher a local variable
(instead of an instance variable), then there also is no error. I'm
running this on windows xp.
Has anyone observed this type of behavior before?
Any idea what might be causing the memory leak?
much thanks,
~Heng
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: IndexSearcher memory leak?
Posted by "Rob Staveley (Tom)" <rs...@seseit.com>.
> i guess I needed to RTFC
I found that recently too. My only contribution to Lucene has been asking
for a Javadoc addition to prevent others from falling into a trap, which I
fell into. My issue was http://issues.apache.org/jira/browse/LUCENE-594.
Similarly, you could ask for a Javadoc comment for RAMDirectory.close(),
which I expect would be valued by others.
-----Original Message-----
From: Heng Mei [mailto:heng.mei@gmail.com]
Sent: 06 July 2006 09:28
To: java-user@lucene.apache.org
Subject: Re: IndexSearcher memory leak?
Thanks for the helpful tip, it makes sense now.
I had previously assumed (wrongly) that RAMDirectory.close() would free up
its memory buffers.. but i guess I needed to RTFC...
RAMDirectory.close() is just an empty method.
On 7/5/06, Rob Staveley (Tom) <rs...@seseit.com> wrote:
> My two bits...
>
> If you consider (say) the 2nd pass of the loop...
>
> Searcher searcher = null;
> for(int i = 0; i < 5; ++i){
> RAMDirectory ramdir = new RAMDirectory( db );
> // <- Consider this moment
> searcher = new IndexSearcher( ramdir );
> }
>
> ... you will have two references to RAMDirectory objects in existence.
> The Searcher retains a reference to the previously constructed
RAMDirectory.
>
> You could nullify the Searcher before contructing the new RAMDirectory
> to avoid this.
>
> Searcher searcher = null;
> for(int i = 0; i < 5; ++i){
> searcher = null; <- No searcher now to retain the
> reference
> RAMDirectory ramdir = new RAMDirectory( db );
> // <- Consider this moment
> searcher = new IndexSearcher( ramdir );
> }
>
> -----Original Message-----
> From: Heng Mei [mailto:heng.mei@gmail.com]
> Sent: 06 July 2006 06:32
> To: java-user@lucene.apache.org
> Subject: Re: IndexSearcher memory leak?
>
> import org.apache.lucene.search.Searcher;
> import org.apache.lucene.search.IndexSearcher;
> import org.apache.lucene.search.*;
> import org.apache.lucene.index.*;
> import org.apache.lucene.store.RAMDirectory;
>
> /**
> * Index is ~100mb.
> * Run with -Xmx256m on Windows XP
> *
> * OutOfMemoryException encountered after repeatedly
> * opening/searching/closing an index.
> * Seems to only occur when Searcher scope is outside the loop */
> public class TestOpenClose {
> public static void main(String[] args) throws Exception {
> String db = "C:/data/index";
> String field = "field";
> String term = "keywords";
>
> //Normal behavior - no memory exceptions
> //The Searcher's scope is inside the loop
> for(int i = 0; i < 5; ++i){
> System.out.println(i);
> RAMDirectory ramdir = new RAMDirectory( db );
> Searcher searcher = new IndexSearcher( ramdir );
> searcher.search(new TermQuery( new Term(field,
> term)));
> searcher.close();
> ramdir.close();
> }
>
> //OutOfMemoryException after a few iterations
> //Search scope is outside the loop
> Searcher searcher = null;
> for(int i = 0; i < 5; ++i){
> System.out.println(i);
> RAMDirectory ramdir = new RAMDirectory( db );
> searcher = new IndexSearcher( ramdir );
> searcher.search(new TermQuery( new Term(field,
> term)));
> searcher.close();
> ramdir.close();
> }
> }
> }
>
> On 7/5/06, Chris Hostetter <ho...@fucit.org> wrote:
> >
> > The mailing list server doesn't allow most attachments (i think the
> > magic incantation is to make sure your mail client indicates that
> > the attachments are plain text -- not some fancy type like
> > text/x-java-source or application/x-java)
> >
> > try including it inline in your email.
> >
> >
> >
> > : Date: Wed, 5 Jul 2006 21:50:35 -0700
> > : From: Heng Mei <he...@gmail.com>
> > : Reply-To: java-user@lucene.apache.org
> > : To: java-user@lucene.apache.org
> > : Subject: Re: IndexSearcher memory leak?
> > :
> > : Thanks for the fast response -- you're right, I had a typo in the
> > code
> snippet.
> > :
> > : I've attached the test code that reproduces the OutOfMemoryException.
> > : The code is a simple main method that repeatedly
> > opens/searches/closes
> > : an index using RAMDirectory. Maybe there's something I'm
> > overlooking,
> > : but it appears that when the Searcher is declared outside of the
> > loop,
> > : that's when the OutOfMemoryException occurs. If the searcher is
> > : declared inside of the loop, then there's no memory problem.
> > :
> > : The motivation for this test code is that I plan to write a
> > : long-running server that periodically reopens an index -- so
> > clearly
> > : I'd want to be able to close and open the index without leaking
> > any
> > : memory.
> > :
> > : Any insight on why the memory exception occurs and how to resolve
> > it
> > : would be greatly appreciated.
> > :
> > :
> > : Thanks!
> > : ~Heng
> > :
> > : On 7/5/06, Chris Hostetter <ho...@fucit.org> wrote:
> > : > : I get an OutOfMemoryException after a few iterations of the
> following loop:
> > : > : LOOP:
> > : > : ramdir = new RAMDirectory( "path/to/my/directory" );
> > : > : searcher = new IndexSearcher( reader );
> > : > : searcher.search(new TermQuery( new Term( "field",
> "keyword")));
> > : > : searcher.close();
> > : > : ramdir.close();
> > : > : END LOOP:
> > : > : Note, ramdir and searcher are both instance variables.
> > : >
> > : > But what is "reader" ? .. where is it initialized ?
> > : >
> > : > I'm guessing this isn't hte exact code that gives you an OOM
> > (since there
> > : > seems to be some code missing)
> > : >
> > : > if you can post a complete small example program (with main
> > method) that
> > : > demonstrates this problem -- or even better a self contained
> > JUNit test --
> > : > then people can try to reproduce it.
> > : >
> > : > (if the root of the issue is opening a RAMDirectory based on an
> > : > FSDirectory you can create the on disk directory using the
> > system tmpdir
> > : > in your JUnit setUp method)
> > : >
> > : > I suspect that somewhere in there you have another line that
> > looks
> like...
> > : >
> > : > reader = IndexReader.open( ramdir );
> > : >
> > : > ...and that may be causing your problem. If you open an
IndexReader
> > : > explicitly, then IndexSearcher.close() won't close it for you
> > : >
> > : >
> > : >
> > : >
> > : >
> > : > -Hoss
> > : >
> > : >
> > : >
> > --------------------------------------------------------------------
> > -
> > : > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > : > For additional commands, e-mail:
> > java-user-help@lucene.apache.org
> > : >
> > : >
> > :
> > :
> >
> >
> >
> > -Hoss
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: IndexSearcher memory leak?
Posted by Heng Mei <he...@gmail.com>.
Thanks for the helpful tip, it makes sense now.
I had previously assumed (wrongly) that RAMDirectory.close() would
free up its memory buffers.. but i guess I needed to RTFC...
RAMDirectory.close() is just an empty method.
On 7/5/06, Rob Staveley (Tom) <rs...@seseit.com> wrote:
> My two bits...
>
> If you consider (say) the 2nd pass of the loop...
>
> Searcher searcher = null;
> for(int i = 0; i < 5; ++i){
> RAMDirectory ramdir = new RAMDirectory( db );
> // <- Consider this moment
> searcher = new IndexSearcher( ramdir );
> }
>
> ... you will have two references to RAMDirectory objects in existence. The
> Searcher retains a reference to the previously constructed RAMDirectory.
>
> You could nullify the Searcher before contructing the new RAMDirectory to
> avoid this.
>
> Searcher searcher = null;
> for(int i = 0; i < 5; ++i){
> searcher = null; <- No searcher now to retain the
> reference
> RAMDirectory ramdir = new RAMDirectory( db );
> // <- Consider this moment
> searcher = new IndexSearcher( ramdir );
> }
>
> -----Original Message-----
> From: Heng Mei [mailto:heng.mei@gmail.com]
> Sent: 06 July 2006 06:32
> To: java-user@lucene.apache.org
> Subject: Re: IndexSearcher memory leak?
>
> import org.apache.lucene.search.Searcher;
> import org.apache.lucene.search.IndexSearcher;
> import org.apache.lucene.search.*;
> import org.apache.lucene.index.*;
> import org.apache.lucene.store.RAMDirectory;
>
> /**
> * Index is ~100mb.
> * Run with -Xmx256m on Windows XP
> *
> * OutOfMemoryException encountered after repeatedly
> * opening/searching/closing an index.
> * Seems to only occur when Searcher scope is outside the loop */ public
> class TestOpenClose {
> public static void main(String[] args) throws Exception {
> String db = "C:/data/index";
> String field = "field";
> String term = "keywords";
>
> //Normal behavior - no memory exceptions
> //The Searcher's scope is inside the loop
> for(int i = 0; i < 5; ++i){
> System.out.println(i);
> RAMDirectory ramdir = new RAMDirectory( db );
> Searcher searcher = new IndexSearcher( ramdir );
> searcher.search(new TermQuery( new Term(field,
> term)));
> searcher.close();
> ramdir.close();
> }
>
> //OutOfMemoryException after a few iterations
> //Search scope is outside the loop
> Searcher searcher = null;
> for(int i = 0; i < 5; ++i){
> System.out.println(i);
> RAMDirectory ramdir = new RAMDirectory( db );
> searcher = new IndexSearcher( ramdir );
> searcher.search(new TermQuery( new Term(field,
> term)));
> searcher.close();
> ramdir.close();
> }
> }
> }
>
> On 7/5/06, Chris Hostetter <ho...@fucit.org> wrote:
> >
> > The mailing list server doesn't allow most attachments (i think the
> > magic incantation is to make sure your mail client indicates that the
> > attachments are plain text -- not some fancy type like
> > text/x-java-source or application/x-java)
> >
> > try including it inline in your email.
> >
> >
> >
> > : Date: Wed, 5 Jul 2006 21:50:35 -0700
> > : From: Heng Mei <he...@gmail.com>
> > : Reply-To: java-user@lucene.apache.org
> > : To: java-user@lucene.apache.org
> > : Subject: Re: IndexSearcher memory leak?
> > :
> > : Thanks for the fast response -- you're right, I had a typo in the code
> snippet.
> > :
> > : I've attached the test code that reproduces the OutOfMemoryException.
> > : The code is a simple main method that repeatedly
> > opens/searches/closes
> > : an index using RAMDirectory. Maybe there's something I'm
> > overlooking,
> > : but it appears that when the Searcher is declared outside of the
> > loop,
> > : that's when the OutOfMemoryException occurs. If the searcher is
> > : declared inside of the loop, then there's no memory problem.
> > :
> > : The motivation for this test code is that I plan to write a
> > : long-running server that periodically reopens an index -- so clearly
> > : I'd want to be able to close and open the index without leaking any
> > : memory.
> > :
> > : Any insight on why the memory exception occurs and how to resolve it
> > : would be greatly appreciated.
> > :
> > :
> > : Thanks!
> > : ~Heng
> > :
> > : On 7/5/06, Chris Hostetter <ho...@fucit.org> wrote:
> > : > : I get an OutOfMemoryException after a few iterations of the
> following loop:
> > : > : LOOP:
> > : > : ramdir = new RAMDirectory( "path/to/my/directory" );
> > : > : searcher = new IndexSearcher( reader );
> > : > : searcher.search(new TermQuery( new Term( "field",
> "keyword")));
> > : > : searcher.close();
> > : > : ramdir.close();
> > : > : END LOOP:
> > : > : Note, ramdir and searcher are both instance variables.
> > : >
> > : > But what is "reader" ? .. where is it initialized ?
> > : >
> > : > I'm guessing this isn't hte exact code that gives you an OOM
> > (since there
> > : > seems to be some code missing)
> > : >
> > : > if you can post a complete small example program (with main
> > method) that
> > : > demonstrates this problem -- or even better a self contained JUNit
> > test --
> > : > then people can try to reproduce it.
> > : >
> > : > (if the root of the issue is opening a RAMDirectory based on an
> > : > FSDirectory you can create the on disk directory using the system
> > tmpdir
> > : > in your JUnit setUp method)
> > : >
> > : > I suspect that somewhere in there you have another line that looks
> like...
> > : >
> > : > reader = IndexReader.open( ramdir );
> > : >
> > : > ...and that may be causing your problem. If you open an IndexReader
> > : > explicitly, then IndexSearcher.close() won't close it for you
> > : >
> > : >
> > : >
> > : >
> > : >
> > : > -Hoss
> > : >
> > : >
> > : >
> > ---------------------------------------------------------------------
> > : > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > : > For additional commands, e-mail: java-user-help@lucene.apache.org
> > : >
> > : >
> > :
> > :
> >
> >
> >
> > -Hoss
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: IndexSearcher memory leak?
Posted by "Rob Staveley (Tom)" <rs...@seseit.com>.
My two bits...
If you consider (say) the 2nd pass of the loop...
Searcher searcher = null;
for(int i = 0; i < 5; ++i){
RAMDirectory ramdir = new RAMDirectory( db );
// <- Consider this moment
searcher = new IndexSearcher( ramdir );
}
... you will have two references to RAMDirectory objects in existence. The
Searcher retains a reference to the previously constructed RAMDirectory.
You could nullify the Searcher before contructing the new RAMDirectory to
avoid this.
Searcher searcher = null;
for(int i = 0; i < 5; ++i){
searcher = null; <- No searcher now to retain the
reference
RAMDirectory ramdir = new RAMDirectory( db );
// <- Consider this moment
searcher = new IndexSearcher( ramdir );
}
-----Original Message-----
From: Heng Mei [mailto:heng.mei@gmail.com]
Sent: 06 July 2006 06:32
To: java-user@lucene.apache.org
Subject: Re: IndexSearcher memory leak?
import org.apache.lucene.search.Searcher;
import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.search.*;
import org.apache.lucene.index.*;
import org.apache.lucene.store.RAMDirectory;
/**
* Index is ~100mb.
* Run with -Xmx256m on Windows XP
*
* OutOfMemoryException encountered after repeatedly
* opening/searching/closing an index.
* Seems to only occur when Searcher scope is outside the loop */ public
class TestOpenClose {
public static void main(String[] args) throws Exception {
String db = "C:/data/index";
String field = "field";
String term = "keywords";
//Normal behavior - no memory exceptions
//The Searcher's scope is inside the loop
for(int i = 0; i < 5; ++i){
System.out.println(i);
RAMDirectory ramdir = new RAMDirectory( db );
Searcher searcher = new IndexSearcher( ramdir );
searcher.search(new TermQuery( new Term(field,
term)));
searcher.close();
ramdir.close();
}
//OutOfMemoryException after a few iterations
//Search scope is outside the loop
Searcher searcher = null;
for(int i = 0; i < 5; ++i){
System.out.println(i);
RAMDirectory ramdir = new RAMDirectory( db );
searcher = new IndexSearcher( ramdir );
searcher.search(new TermQuery( new Term(field,
term)));
searcher.close();
ramdir.close();
}
}
}
On 7/5/06, Chris Hostetter <ho...@fucit.org> wrote:
>
> The mailing list server doesn't allow most attachments (i think the
> magic incantation is to make sure your mail client indicates that the
> attachments are plain text -- not some fancy type like
> text/x-java-source or application/x-java)
>
> try including it inline in your email.
>
>
>
> : Date: Wed, 5 Jul 2006 21:50:35 -0700
> : From: Heng Mei <he...@gmail.com>
> : Reply-To: java-user@lucene.apache.org
> : To: java-user@lucene.apache.org
> : Subject: Re: IndexSearcher memory leak?
> :
> : Thanks for the fast response -- you're right, I had a typo in the code
snippet.
> :
> : I've attached the test code that reproduces the OutOfMemoryException.
> : The code is a simple main method that repeatedly
> opens/searches/closes
> : an index using RAMDirectory. Maybe there's something I'm
> overlooking,
> : but it appears that when the Searcher is declared outside of the
> loop,
> : that's when the OutOfMemoryException occurs. If the searcher is
> : declared inside of the loop, then there's no memory problem.
> :
> : The motivation for this test code is that I plan to write a
> : long-running server that periodically reopens an index -- so clearly
> : I'd want to be able to close and open the index without leaking any
> : memory.
> :
> : Any insight on why the memory exception occurs and how to resolve it
> : would be greatly appreciated.
> :
> :
> : Thanks!
> : ~Heng
> :
> : On 7/5/06, Chris Hostetter <ho...@fucit.org> wrote:
> : > : I get an OutOfMemoryException after a few iterations of the
following loop:
> : > : LOOP:
> : > : ramdir = new RAMDirectory( "path/to/my/directory" );
> : > : searcher = new IndexSearcher( reader );
> : > : searcher.search(new TermQuery( new Term( "field",
"keyword")));
> : > : searcher.close();
> : > : ramdir.close();
> : > : END LOOP:
> : > : Note, ramdir and searcher are both instance variables.
> : >
> : > But what is "reader" ? .. where is it initialized ?
> : >
> : > I'm guessing this isn't hte exact code that gives you an OOM
> (since there
> : > seems to be some code missing)
> : >
> : > if you can post a complete small example program (with main
> method) that
> : > demonstrates this problem -- or even better a self contained JUNit
> test --
> : > then people can try to reproduce it.
> : >
> : > (if the root of the issue is opening a RAMDirectory based on an
> : > FSDirectory you can create the on disk directory using the system
> tmpdir
> : > in your JUnit setUp method)
> : >
> : > I suspect that somewhere in there you have another line that looks
like...
> : >
> : > reader = IndexReader.open( ramdir );
> : >
> : > ...and that may be causing your problem. If you open an IndexReader
> : > explicitly, then IndexSearcher.close() won't close it for you
> : >
> : >
> : >
> : >
> : >
> : > -Hoss
> : >
> : >
> : >
> ---------------------------------------------------------------------
> : > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> : > For additional commands, e-mail: java-user-help@lucene.apache.org
> : >
> : >
> :
> :
>
>
>
> -Hoss
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: IndexSearcher memory leak?
Posted by Heng Mei <he...@gmail.com>.
import org.apache.lucene.search.Searcher;
import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.search.*;
import org.apache.lucene.index.*;
import org.apache.lucene.store.RAMDirectory;
/**
* Index is ~100mb.
* Run with -Xmx256m on Windows XP
*
* OutOfMemoryException encountered after repeatedly
* opening/searching/closing an index.
* Seems to only occur when Searcher scope is outside the loop
*/
public class TestOpenClose {
public static void main(String[] args) throws Exception {
String db = "C:/data/index";
String field = "field";
String term = "keywords";
//Normal behavior - no memory exceptions
//The Searcher's scope is inside the loop
for(int i = 0; i < 5; ++i){
System.out.println(i);
RAMDirectory ramdir = new RAMDirectory( db );
Searcher searcher = new IndexSearcher( ramdir );
searcher.search(new TermQuery( new Term(field, term)));
searcher.close();
ramdir.close();
}
//OutOfMemoryException after a few iterations
//Search scope is outside the loop
Searcher searcher = null;
for(int i = 0; i < 5; ++i){
System.out.println(i);
RAMDirectory ramdir = new RAMDirectory( db );
searcher = new IndexSearcher( ramdir );
searcher.search(new TermQuery( new Term(field, term)));
searcher.close();
ramdir.close();
}
}
}
On 7/5/06, Chris Hostetter <ho...@fucit.org> wrote:
>
> The mailing list server doesn't allow most attachments (i think the magic
> incantation is to make sure your mail client indicates that the
> attachments are plain text -- not some fancy type like text/x-java-source
> or application/x-java)
>
> try including it inline in your email.
>
>
>
> : Date: Wed, 5 Jul 2006 21:50:35 -0700
> : From: Heng Mei <he...@gmail.com>
> : Reply-To: java-user@lucene.apache.org
> : To: java-user@lucene.apache.org
> : Subject: Re: IndexSearcher memory leak?
> :
> : Thanks for the fast response -- you're right, I had a typo in the code snippet.
> :
> : I've attached the test code that reproduces the OutOfMemoryException.
> : The code is a simple main method that repeatedly opens/searches/closes
> : an index using RAMDirectory. Maybe there's something I'm overlooking,
> : but it appears that when the Searcher is declared outside of the loop,
> : that's when the OutOfMemoryException occurs. If the searcher is
> : declared inside of the loop, then there's no memory problem.
> :
> : The motivation for this test code is that I plan to write a
> : long-running server that periodically reopens an index -- so clearly
> : I'd want to be able to close and open the index without leaking any
> : memory.
> :
> : Any insight on why the memory exception occurs and how to resolve it
> : would be greatly appreciated.
> :
> :
> : Thanks!
> : ~Heng
> :
> : On 7/5/06, Chris Hostetter <ho...@fucit.org> wrote:
> : > : I get an OutOfMemoryException after a few iterations of the following loop:
> : > : LOOP:
> : > : ramdir = new RAMDirectory( "path/to/my/directory" );
> : > : searcher = new IndexSearcher( reader );
> : > : searcher.search(new TermQuery( new Term( "field", "keyword")));
> : > : searcher.close();
> : > : ramdir.close();
> : > : END LOOP:
> : > : Note, ramdir and searcher are both instance variables.
> : >
> : > But what is "reader" ? .. where is it initialized ?
> : >
> : > I'm guessing this isn't hte exact code that gives you an OOM (since there
> : > seems to be some code missing)
> : >
> : > if you can post a complete small example program (with main method) that
> : > demonstrates this problem -- or even better a self contained JUNit test --
> : > then people can try to reproduce it.
> : >
> : > (if the root of the issue is opening a RAMDirectory based on an
> : > FSDirectory you can create the on disk directory using the system tmpdir
> : > in your JUnit setUp method)
> : >
> : > I suspect that somewhere in there you have another line that looks like...
> : >
> : > reader = IndexReader.open( ramdir );
> : >
> : > ...and that may be causing your problem. If you open an IndexReader
> : > explicitly, then IndexSearcher.close() won't close it for you
> : >
> : >
> : >
> : >
> : >
> : > -Hoss
> : >
> : >
> : > ---------------------------------------------------------------------
> : > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> : > For additional commands, e-mail: java-user-help@lucene.apache.org
> : >
> : >
> :
> :
>
>
>
> -Hoss
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: IndexSearcher memory leak?
Posted by Chris Hostetter <ho...@fucit.org>.
The mailing list server doesn't allow most attachments (i think the magic
incantation is to make sure your mail client indicates that the
attachments are plain text -- not some fancy type like text/x-java-source
or application/x-java)
try including it inline in your email.
: Date: Wed, 5 Jul 2006 21:50:35 -0700
: From: Heng Mei <he...@gmail.com>
: Reply-To: java-user@lucene.apache.org
: To: java-user@lucene.apache.org
: Subject: Re: IndexSearcher memory leak?
:
: Thanks for the fast response -- you're right, I had a typo in the code snippet.
:
: I've attached the test code that reproduces the OutOfMemoryException.
: The code is a simple main method that repeatedly opens/searches/closes
: an index using RAMDirectory. Maybe there's something I'm overlooking,
: but it appears that when the Searcher is declared outside of the loop,
: that's when the OutOfMemoryException occurs. If the searcher is
: declared inside of the loop, then there's no memory problem.
:
: The motivation for this test code is that I plan to write a
: long-running server that periodically reopens an index -- so clearly
: I'd want to be able to close and open the index without leaking any
: memory.
:
: Any insight on why the memory exception occurs and how to resolve it
: would be greatly appreciated.
:
:
: Thanks!
: ~Heng
:
: On 7/5/06, Chris Hostetter <ho...@fucit.org> wrote:
: > : I get an OutOfMemoryException after a few iterations of the following loop:
: > : LOOP:
: > : ramdir = new RAMDirectory( "path/to/my/directory" );
: > : searcher = new IndexSearcher( reader );
: > : searcher.search(new TermQuery( new Term( "field", "keyword")));
: > : searcher.close();
: > : ramdir.close();
: > : END LOOP:
: > : Note, ramdir and searcher are both instance variables.
: >
: > But what is "reader" ? .. where is it initialized ?
: >
: > I'm guessing this isn't hte exact code that gives you an OOM (since there
: > seems to be some code missing)
: >
: > if you can post a complete small example program (with main method) that
: > demonstrates this problem -- or even better a self contained JUNit test --
: > then people can try to reproduce it.
: >
: > (if the root of the issue is opening a RAMDirectory based on an
: > FSDirectory you can create the on disk directory using the system tmpdir
: > in your JUnit setUp method)
: >
: > I suspect that somewhere in there you have another line that looks like...
: >
: > reader = IndexReader.open( ramdir );
: >
: > ...and that may be causing your problem. If you open an IndexReader
: > explicitly, then IndexSearcher.close() won't close it for you
: >
: >
: >
: >
: >
: > -Hoss
: >
: >
: > ---------------------------------------------------------------------
: > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
: > For additional commands, e-mail: java-user-help@lucene.apache.org
: >
: >
:
:
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: IndexSearcher memory leak?
Posted by Heng Mei <he...@gmail.com>.
Thanks for the fast response -- you're right, I had a typo in the code snippet.
I've attached the test code that reproduces the OutOfMemoryException.
The code is a simple main method that repeatedly opens/searches/closes
an index using RAMDirectory. Maybe there's something I'm overlooking,
but it appears that when the Searcher is declared outside of the loop,
that's when the OutOfMemoryException occurs. If the searcher is
declared inside of the loop, then there's no memory problem.
The motivation for this test code is that I plan to write a
long-running server that periodically reopens an index -- so clearly
I'd want to be able to close and open the index without leaking any
memory.
Any insight on why the memory exception occurs and how to resolve it
would be greatly appreciated.
Thanks!
~Heng
On 7/5/06, Chris Hostetter <ho...@fucit.org> wrote:
> : I get an OutOfMemoryException after a few iterations of the following loop:
> : LOOP:
> : ramdir = new RAMDirectory( "path/to/my/directory" );
> : searcher = new IndexSearcher( reader );
> : searcher.search(new TermQuery( new Term( "field", "keyword")));
> : searcher.close();
> : ramdir.close();
> : END LOOP:
> : Note, ramdir and searcher are both instance variables.
>
> But what is "reader" ? .. where is it initialized ?
>
> I'm guessing this isn't hte exact code that gives you an OOM (since there
> seems to be some code missing)
>
> if you can post a complete small example program (with main method) that
> demonstrates this problem -- or even better a self contained JUNit test --
> then people can try to reproduce it.
>
> (if the root of the issue is opening a RAMDirectory based on an
> FSDirectory you can create the on disk directory using the system tmpdir
> in your JUnit setUp method)
>
> I suspect that somewhere in there you have another line that looks like...
>
> reader = IndexReader.open( ramdir );
>
> ...and that may be causing your problem. If you open an IndexReader
> explicitly, then IndexSearcher.close() won't close it for you
>
>
>
>
>
> -Hoss
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: IndexSearcher memory leak?
Posted by Chris Hostetter <ho...@fucit.org>.
: I get an OutOfMemoryException after a few iterations of the following loop:
: LOOP:
: ramdir = new RAMDirectory( "path/to/my/directory" );
: searcher = new IndexSearcher( reader );
: searcher.search(new TermQuery( new Term( "field", "keyword")));
: searcher.close();
: ramdir.close();
: END LOOP:
: Note, ramdir and searcher are both instance variables.
But what is "reader" ? .. where is it initialized ?
I'm guessing this isn't hte exact code that gives you an OOM (since there
seems to be some code missing)
if you can post a complete small example program (with main method) that
demonstrates this problem -- or even better a self contained JUNit test --
then people can try to reproduce it.
(if the root of the issue is opening a RAMDirectory based on an
FSDirectory you can create the on disk directory using the system tmpdir
in your JUnit setUp method)
I suspect that somewhere in there you have another line that looks like...
reader = IndexReader.open( ramdir );
...and that may be causing your problem. If you open an IndexReader
explicitly, then IndexSearcher.close() won't close it for you
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org