You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Stephen Gray <St...@anu.edu.au> on 2007/05/15 08:31:05 UTC
Memory leak (JVM 1.6 only)
Hi everyone,
I have an application that indexes/searches xml documents using Lucene.
I'm having a problem with what looks like a memory leak, which occurs
when indexing a large number of documents, but only when the application
is running under JVM 1.6. Under JVM 1.5 there is no problem. What
happens is that the memory allocated consistently rises during indexing
until the JVM crashes with an OutOfMemory exception.
I'm using Lucene 2.1, and am using Maik Schreiber's LuceneIndexAccess
API, which hands out references to cached IndexWriter/Reader/Searchers
to objects that need to use them, and handles closing and re-opening
IndexSearchers after documents are added to the index. The application
is running under Tomcat 6.
I'm a bit out of my depth determining the source of the leak - I've
tried using Netbeans profiler, which shows a large number of HashMap
instances that survive a long time, but these are created by many
different classes so it's difficult to pinpoint one source.
Has anyone found similar problems with Lucene indexing operations
running under JVM 1.6? Does anyone have any suggestions re how to deal
with this?
Any help much appreciated.
Thanks,
Steve
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Memory leak (JVM 1.6 only)
Posted by Stephen Gray <St...@anu.edu.au>.
Thanks, that narrows it down a bit.
Thanks for all the replies to my question.
Steve
Mark Miller wrote:
> I don't have much help to offer other than to say I am also using a
> tweaked version of the IndexAccess code you are, with java 1.6, with
> hundreds of thousands to millions of docs, at multiple locations, for
> months -- and I have not seen any memory leaks. Leads me to think the
> leak may be with your code and not any Lucene code...
>
> I will mention that the IndexAccess code does not handle
> multi-searchers very well without some adjustments, but even that
> should not cause a leak.
>
> Keep playing with the Netbeans profiler...if it really is a leak, you
> aught to be able to find it.
>
> - Mark
>
> Stephen Gray wrote:
>> Hi everyone,
>>
>> I have an application that indexes/searches xml documents using
>> Lucene. I'm having a problem with what looks like a memory leak,
>> which occurs when indexing a large number of documents, but only when
>> the application is running under JVM 1.6. Under JVM 1.5 there is no
>> problem. What happens is that the memory allocated consistently rises
>> during indexing until the JVM crashes with an OutOfMemory exception.
>>
>> I'm using Lucene 2.1, and am using Maik Schreiber's LuceneIndexAccess
>> API, which hands out references to cached
>> IndexWriter/Reader/Searchers to objects that need to use them, and
>> handles closing and re-opening IndexSearchers after documents are
>> added to the index. The application is running under Tomcat 6.
>>
>> I'm a bit out of my depth determining the source of the leak - I've
>> tried using Netbeans profiler, which shows a large number of HashMap
>> instances that survive a long time, but these are created by many
>> different classes so it's difficult to pinpoint one source.
>>
>> Has anyone found similar problems with Lucene indexing operations
>> running under JVM 1.6? Does anyone have any suggestions re how to
>> deal with this?
>>
>> Any help much appreciated.
>>
>> Thanks,
>> Steve
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
--
Stephen Gray
Archive IT Officer
Australian Social Science Data Archive
18 Balmain Crescent (Building #66)
The Australian National University
Canberra ACT 0200
Phone +61 2 6125 2185
Fax +61 2 6125 0627
Web http://assda.anu.edu.au/
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Memory leak (JVM 1.6 only)
Posted by Mark Miller <ma...@gmail.com>.
I don't have much help to offer other than to say I am also using a
tweaked version of the IndexAccess code you are, with java 1.6, with
hundreds of thousands to millions of docs, at multiple locations, for
months -- and I have not seen any memory leaks. Leads me to think the
leak may be with your code and not any Lucene code...
I will mention that the IndexAccess code does not handle multi-searchers
very well without some adjustments, but even that should not cause a leak.
Keep playing with the Netbeans profiler...if it really is a leak, you
aught to be able to find it.
- Mark
Stephen Gray wrote:
> Hi everyone,
>
> I have an application that indexes/searches xml documents using
> Lucene. I'm having a problem with what looks like a memory leak, which
> occurs when indexing a large number of documents, but only when the
> application is running under JVM 1.6. Under JVM 1.5 there is no
> problem. What happens is that the memory allocated consistently rises
> during indexing until the JVM crashes with an OutOfMemory exception.
>
> I'm using Lucene 2.1, and am using Maik Schreiber's LuceneIndexAccess
> API, which hands out references to cached IndexWriter/Reader/Searchers
> to objects that need to use them, and handles closing and re-opening
> IndexSearchers after documents are added to the index. The application
> is running under Tomcat 6.
>
> I'm a bit out of my depth determining the source of the leak - I've
> tried using Netbeans profiler, which shows a large number of HashMap
> instances that survive a long time, but these are created by many
> different classes so it's difficult to pinpoint one source.
>
> Has anyone found similar problems with Lucene indexing operations
> running under JVM 1.6? Does anyone have any suggestions re how to deal
> with this?
>
> Any help much appreciated.
>
> Thanks,
> Steve
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Memory leak (JVM 1.6 only)
Posted by Antony Bowesman <ad...@teamware.com>.
Daniel Noll wrote:
> On Tuesday 15 May 2007 21:59:31 Narednra Singh Panwar wrote:
>> try using -Xmx option with your Application. and specify maximum/ minimum
>> memory for your Application.
>
> It's funny how a lot of people instantly suggest this. What if it isn't
> possible? There was a situation a while back where I said I had allocated it
> 1.8GB, and someone *still* recommended this option. :-)
Yes, and if it is a memory leak, it only prolongs the terminal illness :)
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Memory leak (JVM 1.6 only)
Posted by Daniel Noll <da...@nuix.com>.
On Tuesday 15 May 2007 21:59:31 Narednra Singh Panwar wrote:
> try using -Xmx option with your Application. and specify maximum/ minimum
> memory for your Application.
It's funny how a lot of people instantly suggest this. What if it isn't
possible? There was a situation a while back where I said I had allocated it
1.8GB, and someone *still* recommended this option. :-)
Daniel
--
Daniel Noll
Nuix Pty Ltd
Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2 9280 0699
Web: http://nuix.com/ Fax: +61 2 9212 6902
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Memory leak (JVM 1.6 only)
Posted by Narednra Singh Panwar <np...@chambal.com>.
try using -Xmx option with your Application. and specify maximum/ minimum
memory for your Application.
Hope this will solve you problem.
On 5/15/07, Stephen Gray <St...@anu.edu.au> wrote:
>
> Hi everyone,
>
> I have an application that indexes/searches xml documents using Lucene.
> I'm having a problem with what looks like a memory leak, which occurs
> when indexing a large number of documents, but only when the application
> is running under JVM 1.6. Under JVM 1.5 there is no problem. What
> happens is that the memory allocated consistently rises during indexing
> until the JVM crashes with an OutOfMemory exception.
>
> I'm using Lucene 2.1, and am using Maik Schreiber's LuceneIndexAccess
> API, which hands out references to cached IndexWriter/Reader/Searchers
> to objects that need to use them, and handles closing and re-opening
> IndexSearchers after documents are added to the index. The application
> is running under Tomcat 6.
>
> I'm a bit out of my depth determining the source of the leak - I've
> tried using Netbeans profiler, which shows a large number of HashMap
> instances that survive a long time, but these are created by many
> different classes so it's difficult to pinpoint one source.
>
> Has anyone found similar problems with Lucene indexing operations
> running under JVM 1.6? Does anyone have any suggestions re how to deal
> with this?
>
> Any help much appreciated.
>
> Thanks,
> Steve
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>