modifier.optimize() causes Java heap space (OutOfMemoryException)

[Marcel Morisse <Ma...@jobsintime.de>]

Hey,

I have a problem with Lucene and because I am little bit inexperienced,
I would like to ask you.

I have a database with ca. 2500 items in it. I store these items in a
RAMIndex and try to rebuild it every 10 minutes. I use the same
procedure like updating a FSDirectory - deleting and adding again.

Everything works fine for the first time, but after 30 minutes the
method IndexModifier.optimize() causes a OutofMemoryException: Java heap
space

I tried to increase the java heap space, which is probably not the best
solution. So the exception waits a little bit longer (let's say 60
minutes), but then returns always during rebuilding.

So my questions: Is it possible to rebuild a RAMIndex in the same way
like a FSDirectory and if it so, do I have to pay regard to something?

I am using Lucene nightly build 2007-01-09 with a 512 MB RAM PC.

Thanks for any help


++++++++++
protected synchronized void rebuildRAMIndex() {
 IndexModifier modifier = new IndexModifier(_directory,_analyzer,false);
    			    			
    			int count = countAllObjects();
		        int maxResults = 100;
		        Document doc = new Document();
		        // load maxResults from Database
		        for (int firstResult = 0; firstResult < count;
firstResult += maxResults) {
		  // get the items from a database with the method
findAllObjects
		 for (Iterator it = findAllObjects(firstResult,
maxResults)
		                 .iterator(); it.hasNext();){
             //convert the database items to Lucene Documents

		 doc = convertToDocument(it.next());
		 try{
     		 modifier.deleteDocuments(new Term("id",
doc.get("id")));	            
           	 }
           	 catch (Exception ignore)
           	 {
           	 }
           	 try{
             modifier.addDocument(doc);		            
           	 }
           	 catch (Exception ignore)
           	 {
          	 }	
}    
}
modifier.optimize();
modifier.close(); 
}
+++++++

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: modifier.optimize() causes Java heap space (OutOfMemoryException)

[karl wettin <ka...@gmail.com>]

24 jan 2007 kl. 16.16 skrev Michael McCandless:

> A couple ideas to verify / try:

You can also use a profiler to see what is hogging the resources. I  
personally prefere JProfiler as it plugs right in to my IDE.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: modifier.optimize() causes Java heap space (OutOfMemoryException)

[Michael McCandless <lu...@mikemccandless.com>]

Marcel Morisse wrote:

> I have a problem with Lucene and because I am little bit inexperienced,
> I would like to ask you.
> 
> I have a database with ca. 2500 items in it. I store these items in a
> RAMIndex and try to rebuild it every 10 minutes. I use the same
> procedure like updating a FSDirectory - deleting and adding again.
> 
> Everything works fine for the first time, but after 30 minutes the
> method IndexModifier.optimize() causes a OutofMemoryException: Java heap
> space
> 
> I tried to increase the java heap space, which is probably not the best
> solution. So the exception waits a little bit longer (let's say 60
> minutes), but then returns always during rebuilding.
> 
> So my questions: Is it possible to rebuild a RAMIndex in the same way
> like a FSDirectory and if it so, do I have to pay regard to something?
> 
> I am using Lucene nightly build 2007-01-09 with a 512 MB RAM PC.

This approach sounds fine to me, as long as you give the JVM enough heap
to store your index.

It does sound like there is an unexplained memory (reference) leak
somewhere.

A couple ideas to verify / try:

   * Is your overall set of documents increasing over time (during the
     30 or 60 minutes) or does it remain roughly around 2500?  If it's
     increasing with time it would be expected to require more RAM with
     time.

   * After each iteration try printing _directory.sizeInBytes() to see
     if the RAMDirectory itself is growing in its size (vs some other
     references somewhere not being freed).  This is only available in
     trunk.

   * Print the value you get back from the deleteDocuments() call.  This
     tells you how many docs were actually deleted.  If this is 0 then
     the delete call is doing nothing and that would explain increasing
     memory usage.

   * Where you are catching Exception & ignoring, try at least calling
     ignore.printStackTrace so we can see if you are hitting an
     exception?

   * Try this code on the last release (2.0.0) instead of the trunk
     build: does it still run out of memory?

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org