You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucenenet.apache.org by Trevor Watson <po...@gmail.com> on 2011/11/29 02:59:08 UTC
[Lucene.Net] Memory Leak in 2.9.2.2
I'm attempting to use Lucene.Net v2.9.2.2 in a Visual Studio 2005 (.NET
2.0) environment. We had a piece of software that WAS working. I'm not
sure what has changed however, the following code results in a memory leak
in the Lucene.Net component (or a failure to clean up used memory).
The code in issue is here:
private void SaveFileToFileInfo(Lucene.Net.Index.IndexWriter iw, bool
delayCommit, string sDataPath)
{
Document doc = getFileInfoDoc(sDataPath);
Analyzer analyzer = clsLuceneFunctions.getAnalyzer();
if (this.FileID == 0)
{
string s = "";
}
iw.UpdateDocument(new Lucene.Net.Index.Term("FileId",
this.fileID.ToString("000000000")), doc, analyzer);
analyzer = null;
doc = null;
if (!delayCommit)
iw.Commit();
}
Commenting out the line iw.UpdateDocument resulted in no memory increase.
I also tried replacing it with a deleteDocument and AddDocument and the
memory increased the same as using the UpdateDocument function
The getAnalyzer() function returns a ExtendedStandardAnalyzer, but it's the
UpdateDocument line specifically that gives me the issue.
Any assistance would be greatly appreciated.
Trevor Watson
[Lucene.Net] Re: Memory Leak in 2.9.2.2
Posted by Trevor Watson <po...@gmail.com>.
You said "pre 2.9.3" I checked the apache lucene.net page to try to see if
I could get a copy of 2.9.3, but it was never on the site, just 2.9.2.2 and
2.9.4(g). Was this an un-released version? Or am I looking in the wrong
spot for updates to lucene.net?
Thanks for all your help
On Tue, Nov 29, 2011 at 2:59 PM, Trevor Watson <
powersearchsoftware@gmail.com> wrote:
> I can send you the dll that I am using if you would like. The documents
> are _mostly_ small documents. Emails and office docs size of plain text
>
>
> On Tuesday, November 29, 2011, Christopher Currens <
> currens.chris@gmail.com> wrote:
> > Do you know how big the documents are that you are trying to
> delete/update?
> > I'm trying to find a copy of 2.9.2 to see if I can reproduce it.
> >
> >
> > Thanks,
> > Christopher
> >
> > On Tue, Nov 29, 2011 at 9:11 AM, Trevor Watson <
> > powersearchsoftware@gmail.com> wrote:
> >
> >> Sorry for the duplicate post. I was on the road and posted both via my
> web
> >> mail and office mail by mistake
> >>
> >> The increase is a very gradual, the program starts at about 160,000k
> >> according to task manager (I know that's not entirely accurate, but it
> was
> >> the best I had at the time) and would, after adding 25,000-40,000
> result in
> >> an out of memory exception (800,000k according to taskmanager). I tried
> >> building a copy of 2.9.4 to test, but could not find one that worked in
> >> visual studio 2005
> >>
> >> I did notice using Ants memory profiler that there were a number of
> >> byte[32789] arrays that I didn't know where they came from in memory.
> >>
> >> On Monday, November 28, 2011, Christopher Currens <
> currens.chris@gmail.com
> >> >
> >> wrote:
> >> > Hi Trevor,
> >> >
> >> > What kind of memory increase are we talking about? Also, how big are
> the
> >> > documents that you are indexing, the ones returned from
> getFileInfoDoc()?
> >> > Is it putting an entire file into the index? Pre 2.9.3 versions had
> >> > issues with holding onto allocated byte arrays far beyond when they
> were
> >> > used. The memory could only be freed via closing the IndexWriter.
> >> >
> >> > I'm a little unclear on exactly what's happening. Are you noticing
> >> memory
> >> > spike and stay constant at that level or is it a gradual increase?
> Is it
> >> > causing your application to error, (ie OutOfMemory exception, etc)?
> >> >
> >> >
> >> > Thanks,
> >> > Christopher
> >> >
> >> > On Mon, Nov 28, 2011 at 5:59 PM, Trevor Watson <
> >> > powersearchsoftware@gmail.com> wrote:
> >> >
> >> >> I'm attempting to use Lucene.Net v2.9.2.2 in a Visual Studio 2005
> (.NET
> >> >> 2.0) environment. We had a piece of software that WAS working. I'm
> not
> >> >> sure what has changed however, the following code results in a memory
> >> leak
> >> >> in the Lucene.Net component (or a failure to clean up used memory).
> >> >>
> >> >> The code in issue is here:
> >> >>
> >> >> private void SaveFileToFileInfo(Lucene.Net.Index.IndexWriter iw,
> bool
> >> >> delayCommit, string sDataPath)
> >> >> {
> >> >> Document doc = getFileInfoDoc(sDataPath);
> >> >> Analyzer analyzer = clsLuceneFunctions.getAnalyzer();
> >> >> if (this.FileID == 0)
> >> >> {
> >> >> string s = "";
> >> >> }
> >> >> iw.UpdateDocument(new Lucene.Net.Index.Term("FileId",
> >> >> this.fileID.ToString("000000000")), doc, analyzer);
> >> >>
> >> >> analyzer = null;
> >> >> doc = null;
> >> >> if (!delayCommit)
> >> >> iw.Commit();
> >> >> }
> >> >>
> >> >> Commenting out the line iw.UpdateDocument resulted in no memory
> >> increase.
> >> >> I also tried replacing it with a deleteDocument and AddDocument and
> the
> >> >> memory increased the same as using the UpdateDocument function
> >> >>
> >> >> The getAnalyzer() function returns a ExtendedStandardAnalyzer, but
> it's
> >> the
> >> >> UpdateDocument line specifically that gives me the issue.
> >> >>
> >> >> Any assistance would be greatly appreciated.
> >> >>
> >> >> Trevor Watson
> >> >>
> >> >
> >>
> >
>
[Lucene.Net] Re: Memory Leak in 2.9.2.2
Posted by Trevor Watson <po...@gmail.com>.
I can send you the dll that I am using if you would like. The documents
are _mostly_ small documents. Emails and office docs size of plain text
On Tuesday, November 29, 2011, Christopher Currens <cu...@gmail.com>
wrote:
> Do you know how big the documents are that you are trying to
delete/update?
> I'm trying to find a copy of 2.9.2 to see if I can reproduce it.
>
>
> Thanks,
> Christopher
>
> On Tue, Nov 29, 2011 at 9:11 AM, Trevor Watson <
> powersearchsoftware@gmail.com> wrote:
>
>> Sorry for the duplicate post. I was on the road and posted both via my
web
>> mail and office mail by mistake
>>
>> The increase is a very gradual, the program starts at about 160,000k
>> according to task manager (I know that's not entirely accurate, but it
was
>> the best I had at the time) and would, after adding 25,000-40,000 result
in
>> an out of memory exception (800,000k according to taskmanager). I tried
>> building a copy of 2.9.4 to test, but could not find one that worked in
>> visual studio 2005
>>
>> I did notice using Ants memory profiler that there were a number of
>> byte[32789] arrays that I didn't know where they came from in memory.
>>
>> On Monday, November 28, 2011, Christopher Currens <
currens.chris@gmail.com
>> >
>> wrote:
>> > Hi Trevor,
>> >
>> > What kind of memory increase are we talking about? Also, how big are
the
>> > documents that you are indexing, the ones returned from
getFileInfoDoc()?
>> > Is it putting an entire file into the index? Pre 2.9.3 versions had
>> > issues with holding onto allocated byte arrays far beyond when they
were
>> > used. The memory could only be freed via closing the IndexWriter.
>> >
>> > I'm a little unclear on exactly what's happening. Are you noticing
>> memory
>> > spike and stay constant at that level or is it a gradual increase? Is
it
>> > causing your application to error, (ie OutOfMemory exception, etc)?
>> >
>> >
>> > Thanks,
>> > Christopher
>> >
>> > On Mon, Nov 28, 2011 at 5:59 PM, Trevor Watson <
>> > powersearchsoftware@gmail.com> wrote:
>> >
>> >> I'm attempting to use Lucene.Net v2.9.2.2 in a Visual Studio 2005
(.NET
>> >> 2.0) environment. We had a piece of software that WAS working. I'm
not
>> >> sure what has changed however, the following code results in a memory
>> leak
>> >> in the Lucene.Net component (or a failure to clean up used memory).
>> >>
>> >> The code in issue is here:
>> >>
>> >> private void SaveFileToFileInfo(Lucene.Net.Index.IndexWriter iw, bool
>> >> delayCommit, string sDataPath)
>> >> {
>> >> Document doc = getFileInfoDoc(sDataPath);
>> >> Analyzer analyzer = clsLuceneFunctions.getAnalyzer();
>> >> if (this.FileID == 0)
>> >> {
>> >> string s = "";
>> >> }
>> >> iw.UpdateDocument(new Lucene.Net.Index.Term("FileId",
>> >> this.fileID.ToString("000000000")), doc, analyzer);
>> >>
>> >> analyzer = null;
>> >> doc = null;
>> >> if (!delayCommit)
>> >> iw.Commit();
>> >> }
>> >>
>> >> Commenting out the line iw.UpdateDocument resulted in no memory
>> increase.
>> >> I also tried replacing it with a deleteDocument and AddDocument and
the
>> >> memory increased the same as using the UpdateDocument function
>> >>
>> >> The getAnalyzer() function returns a ExtendedStandardAnalyzer, but
it's
>> the
>> >> UpdateDocument line specifically that gives me the issue.
>> >>
>> >> Any assistance would be greatly appreciated.
>> >>
>> >> Trevor Watson
>> >>
>> >
>>
>
Re: [Lucene.Net] Re: Memory Leak in 2.9.2.2
Posted by Christopher Currens <cu...@gmail.com>.
Do you know how big the documents are that you are trying to delete/update?
I'm trying to find a copy of 2.9.2 to see if I can reproduce it.
Thanks,
Christopher
On Tue, Nov 29, 2011 at 9:11 AM, Trevor Watson <
powersearchsoftware@gmail.com> wrote:
> Sorry for the duplicate post. I was on the road and posted both via my web
> mail and office mail by mistake
>
> The increase is a very gradual, the program starts at about 160,000k
> according to task manager (I know that's not entirely accurate, but it was
> the best I had at the time) and would, after adding 25,000-40,000 result in
> an out of memory exception (800,000k according to taskmanager). I tried
> building a copy of 2.9.4 to test, but could not find one that worked in
> visual studio 2005
>
> I did notice using Ants memory profiler that there were a number of
> byte[32789] arrays that I didn't know where they came from in memory.
>
> On Monday, November 28, 2011, Christopher Currens <currens.chris@gmail.com
> >
> wrote:
> > Hi Trevor,
> >
> > What kind of memory increase are we talking about? Also, how big are the
> > documents that you are indexing, the ones returned from getFileInfoDoc()?
> > Is it putting an entire file into the index? Pre 2.9.3 versions had
> > issues with holding onto allocated byte arrays far beyond when they were
> > used. The memory could only be freed via closing the IndexWriter.
> >
> > I'm a little unclear on exactly what's happening. Are you noticing
> memory
> > spike and stay constant at that level or is it a gradual increase? Is it
> > causing your application to error, (ie OutOfMemory exception, etc)?
> >
> >
> > Thanks,
> > Christopher
> >
> > On Mon, Nov 28, 2011 at 5:59 PM, Trevor Watson <
> > powersearchsoftware@gmail.com> wrote:
> >
> >> I'm attempting to use Lucene.Net v2.9.2.2 in a Visual Studio 2005 (.NET
> >> 2.0) environment. We had a piece of software that WAS working. I'm not
> >> sure what has changed however, the following code results in a memory
> leak
> >> in the Lucene.Net component (or a failure to clean up used memory).
> >>
> >> The code in issue is here:
> >>
> >> private void SaveFileToFileInfo(Lucene.Net.Index.IndexWriter iw, bool
> >> delayCommit, string sDataPath)
> >> {
> >> Document doc = getFileInfoDoc(sDataPath);
> >> Analyzer analyzer = clsLuceneFunctions.getAnalyzer();
> >> if (this.FileID == 0)
> >> {
> >> string s = "";
> >> }
> >> iw.UpdateDocument(new Lucene.Net.Index.Term("FileId",
> >> this.fileID.ToString("000000000")), doc, analyzer);
> >>
> >> analyzer = null;
> >> doc = null;
> >> if (!delayCommit)
> >> iw.Commit();
> >> }
> >>
> >> Commenting out the line iw.UpdateDocument resulted in no memory
> increase.
> >> I also tried replacing it with a deleteDocument and AddDocument and the
> >> memory increased the same as using the UpdateDocument function
> >>
> >> The getAnalyzer() function returns a ExtendedStandardAnalyzer, but it's
> the
> >> UpdateDocument line specifically that gives me the issue.
> >>
> >> Any assistance would be greatly appreciated.
> >>
> >> Trevor Watson
> >>
> >
>
[Lucene.Net] Re: Memory Leak in 2.9.2.2
Posted by Trevor Watson <po...@gmail.com>.
Sorry for the duplicate post. I was on the road and posted both via my web
mail and office mail by mistake
The increase is a very gradual, the program starts at about 160,000k
according to task manager (I know that's not entirely accurate, but it was
the best I had at the time) and would, after adding 25,000-40,000 result in
an out of memory exception (800,000k according to taskmanager). I tried
building a copy of 2.9.4 to test, but could not find one that worked in
visual studio 2005
I did notice using Ants memory profiler that there were a number of
byte[32789] arrays that I didn't know where they came from in memory.
On Monday, November 28, 2011, Christopher Currens <cu...@gmail.com>
wrote:
> Hi Trevor,
>
> What kind of memory increase are we talking about? Also, how big are the
> documents that you are indexing, the ones returned from getFileInfoDoc()?
> Is it putting an entire file into the index? Pre 2.9.3 versions had
> issues with holding onto allocated byte arrays far beyond when they were
> used. The memory could only be freed via closing the IndexWriter.
>
> I'm a little unclear on exactly what's happening. Are you noticing memory
> spike and stay constant at that level or is it a gradual increase? Is it
> causing your application to error, (ie OutOfMemory exception, etc)?
>
>
> Thanks,
> Christopher
>
> On Mon, Nov 28, 2011 at 5:59 PM, Trevor Watson <
> powersearchsoftware@gmail.com> wrote:
>
>> I'm attempting to use Lucene.Net v2.9.2.2 in a Visual Studio 2005 (.NET
>> 2.0) environment. We had a piece of software that WAS working. I'm not
>> sure what has changed however, the following code results in a memory
leak
>> in the Lucene.Net component (or a failure to clean up used memory).
>>
>> The code in issue is here:
>>
>> private void SaveFileToFileInfo(Lucene.Net.Index.IndexWriter iw, bool
>> delayCommit, string sDataPath)
>> {
>> Document doc = getFileInfoDoc(sDataPath);
>> Analyzer analyzer = clsLuceneFunctions.getAnalyzer();
>> if (this.FileID == 0)
>> {
>> string s = "";
>> }
>> iw.UpdateDocument(new Lucene.Net.Index.Term("FileId",
>> this.fileID.ToString("000000000")), doc, analyzer);
>>
>> analyzer = null;
>> doc = null;
>> if (!delayCommit)
>> iw.Commit();
>> }
>>
>> Commenting out the line iw.UpdateDocument resulted in no memory increase.
>> I also tried replacing it with a deleteDocument and AddDocument and the
>> memory increased the same as using the UpdateDocument function
>>
>> The getAnalyzer() function returns a ExtendedStandardAnalyzer, but it's
the
>> UpdateDocument line specifically that gives me the issue.
>>
>> Any assistance would be greatly appreciated.
>>
>> Trevor Watson
>>
>
Re: [Lucene.Net] Memory Leak in 2.9.2.2
Posted by Christopher Currens <cu...@gmail.com>.
Hi Trevor,
What kind of memory increase are we talking about? Also, how big are the
documents that you are indexing, the ones returned from getFileInfoDoc()?
Is it putting an entire file into the index? Pre 2.9.3 versions had
issues with holding onto allocated byte arrays far beyond when they were
used. The memory could only be freed via closing the IndexWriter.
I'm a little unclear on exactly what's happening. Are you noticing memory
spike and stay constant at that level or is it a gradual increase? Is it
causing your application to error, (ie OutOfMemory exception, etc)?
Thanks,
Christopher
On Mon, Nov 28, 2011 at 5:59 PM, Trevor Watson <
powersearchsoftware@gmail.com> wrote:
> I'm attempting to use Lucene.Net v2.9.2.2 in a Visual Studio 2005 (.NET
> 2.0) environment. We had a piece of software that WAS working. I'm not
> sure what has changed however, the following code results in a memory leak
> in the Lucene.Net component (or a failure to clean up used memory).
>
> The code in issue is here:
>
> private void SaveFileToFileInfo(Lucene.Net.Index.IndexWriter iw, bool
> delayCommit, string sDataPath)
> {
> Document doc = getFileInfoDoc(sDataPath);
> Analyzer analyzer = clsLuceneFunctions.getAnalyzer();
> if (this.FileID == 0)
> {
> string s = "";
> }
> iw.UpdateDocument(new Lucene.Net.Index.Term("FileId",
> this.fileID.ToString("000000000")), doc, analyzer);
>
> analyzer = null;
> doc = null;
> if (!delayCommit)
> iw.Commit();
> }
>
> Commenting out the line iw.UpdateDocument resulted in no memory increase.
> I also tried replacing it with a deleteDocument and AddDocument and the
> memory increased the same as using the UpdateDocument function
>
> The getAnalyzer() function returns a ExtendedStandardAnalyzer, but it's the
> UpdateDocument line specifically that gives me the issue.
>
> Any assistance would be greatly appreciated.
>
> Trevor Watson
>