You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucenenet.apache.org by Trevor Watson <po...@gmail.com> on 2011/11/29 18:11:29 UTC
[Lucene.Net] Re: Memory Leak in 2.9.2.2
Sorry for the duplicate post. I was on the road and posted both via my web
mail and office mail by mistake
The increase is a very gradual, the program starts at about 160,000k
according to task manager (I know that's not entirely accurate, but it was
the best I had at the time) and would, after adding 25,000-40,000 result in
an out of memory exception (800,000k according to taskmanager). I tried
building a copy of 2.9.4 to test, but could not find one that worked in
visual studio 2005
I did notice using Ants memory profiler that there were a number of
byte[32789] arrays that I didn't know where they came from in memory.
On Monday, November 28, 2011, Christopher Currens <cu...@gmail.com>
wrote:
> Hi Trevor,
>
> What kind of memory increase are we talking about? Also, how big are the
> documents that you are indexing, the ones returned from getFileInfoDoc()?
> Is it putting an entire file into the index? Pre 2.9.3 versions had
> issues with holding onto allocated byte arrays far beyond when they were
> used. The memory could only be freed via closing the IndexWriter.
>
> I'm a little unclear on exactly what's happening. Are you noticing memory
> spike and stay constant at that level or is it a gradual increase? Is it
> causing your application to error, (ie OutOfMemory exception, etc)?
>
>
> Thanks,
> Christopher
>
> On Mon, Nov 28, 2011 at 5:59 PM, Trevor Watson <
> powersearchsoftware@gmail.com> wrote:
>
>> I'm attempting to use Lucene.Net v2.9.2.2 in a Visual Studio 2005 (.NET
>> 2.0) environment. We had a piece of software that WAS working. I'm not
>> sure what has changed however, the following code results in a memory
leak
>> in the Lucene.Net component (or a failure to clean up used memory).
>>
>> The code in issue is here:
>>
>> private void SaveFileToFileInfo(Lucene.Net.Index.IndexWriter iw, bool
>> delayCommit, string sDataPath)
>> {
>> Document doc = getFileInfoDoc(sDataPath);
>> Analyzer analyzer = clsLuceneFunctions.getAnalyzer();
>> if (this.FileID == 0)
>> {
>> string s = "";
>> }
>> iw.UpdateDocument(new Lucene.Net.Index.Term("FileId",
>> this.fileID.ToString("000000000")), doc, analyzer);
>>
>> analyzer = null;
>> doc = null;
>> if (!delayCommit)
>> iw.Commit();
>> }
>>
>> Commenting out the line iw.UpdateDocument resulted in no memory increase.
>> I also tried replacing it with a deleteDocument and AddDocument and the
>> memory increased the same as using the UpdateDocument function
>>
>> The getAnalyzer() function returns a ExtendedStandardAnalyzer, but it's
the
>> UpdateDocument line specifically that gives me the issue.
>>
>> Any assistance would be greatly appreciated.
>>
>> Trevor Watson
>>
>
[Lucene.Net] Re: Memory Leak in 2.9.2.2
Posted by Trevor Watson <po...@gmail.com>.
You said "pre 2.9.3" I checked the apache lucene.net page to try to see if
I could get a copy of 2.9.3, but it was never on the site, just 2.9.2.2 and
2.9.4(g). Was this an un-released version? Or am I looking in the wrong
spot for updates to lucene.net?
Thanks for all your help
On Tue, Nov 29, 2011 at 2:59 PM, Trevor Watson <
powersearchsoftware@gmail.com> wrote:
> I can send you the dll that I am using if you would like. The documents
> are _mostly_ small documents. Emails and office docs size of plain text
>
>
> On Tuesday, November 29, 2011, Christopher Currens <
> currens.chris@gmail.com> wrote:
> > Do you know how big the documents are that you are trying to
> delete/update?
> > I'm trying to find a copy of 2.9.2 to see if I can reproduce it.
> >
> >
> > Thanks,
> > Christopher
> >
> > On Tue, Nov 29, 2011 at 9:11 AM, Trevor Watson <
> > powersearchsoftware@gmail.com> wrote:
> >
> >> Sorry for the duplicate post. I was on the road and posted both via my
> web
> >> mail and office mail by mistake
> >>
> >> The increase is a very gradual, the program starts at about 160,000k
> >> according to task manager (I know that's not entirely accurate, but it
> was
> >> the best I had at the time) and would, after adding 25,000-40,000
> result in
> >> an out of memory exception (800,000k according to taskmanager). I tried
> >> building a copy of 2.9.4 to test, but could not find one that worked in
> >> visual studio 2005
> >>
> >> I did notice using Ants memory profiler that there were a number of
> >> byte[32789] arrays that I didn't know where they came from in memory.
> >>
> >> On Monday, November 28, 2011, Christopher Currens <
> currens.chris@gmail.com
> >> >
> >> wrote:
> >> > Hi Trevor,
> >> >
> >> > What kind of memory increase are we talking about? Also, how big are
> the
> >> > documents that you are indexing, the ones returned from
> getFileInfoDoc()?
> >> > Is it putting an entire file into the index? Pre 2.9.3 versions had
> >> > issues with holding onto allocated byte arrays far beyond when they
> were
> >> > used. The memory could only be freed via closing the IndexWriter.
> >> >
> >> > I'm a little unclear on exactly what's happening. Are you noticing
> >> memory
> >> > spike and stay constant at that level or is it a gradual increase?
> Is it
> >> > causing your application to error, (ie OutOfMemory exception, etc)?
> >> >
> >> >
> >> > Thanks,
> >> > Christopher
> >> >
> >> > On Mon, Nov 28, 2011 at 5:59 PM, Trevor Watson <
> >> > powersearchsoftware@gmail.com> wrote:
> >> >
> >> >> I'm attempting to use Lucene.Net v2.9.2.2 in a Visual Studio 2005
> (.NET
> >> >> 2.0) environment. We had a piece of software that WAS working. I'm
> not
> >> >> sure what has changed however, the following code results in a memory
> >> leak
> >> >> in the Lucene.Net component (or a failure to clean up used memory).
> >> >>
> >> >> The code in issue is here:
> >> >>
> >> >> private void SaveFileToFileInfo(Lucene.Net.Index.IndexWriter iw,
> bool
> >> >> delayCommit, string sDataPath)
> >> >> {
> >> >> Document doc = getFileInfoDoc(sDataPath);
> >> >> Analyzer analyzer = clsLuceneFunctions.getAnalyzer();
> >> >> if (this.FileID == 0)
> >> >> {
> >> >> string s = "";
> >> >> }
> >> >> iw.UpdateDocument(new Lucene.Net.Index.Term("FileId",
> >> >> this.fileID.ToString("000000000")), doc, analyzer);
> >> >>
> >> >> analyzer = null;
> >> >> doc = null;
> >> >> if (!delayCommit)
> >> >> iw.Commit();
> >> >> }
> >> >>
> >> >> Commenting out the line iw.UpdateDocument resulted in no memory
> >> increase.
> >> >> I also tried replacing it with a deleteDocument and AddDocument and
> the
> >> >> memory increased the same as using the UpdateDocument function
> >> >>
> >> >> The getAnalyzer() function returns a ExtendedStandardAnalyzer, but
> it's
> >> the
> >> >> UpdateDocument line specifically that gives me the issue.
> >> >>
> >> >> Any assistance would be greatly appreciated.
> >> >>
> >> >> Trevor Watson
> >> >>
> >> >
> >>
> >
>
[Lucene.Net] Re: Memory Leak in 2.9.2.2
Posted by Trevor Watson <po...@gmail.com>.
I can send you the dll that I am using if you would like. The documents
are _mostly_ small documents. Emails and office docs size of plain text
On Tuesday, November 29, 2011, Christopher Currens <cu...@gmail.com>
wrote:
> Do you know how big the documents are that you are trying to
delete/update?
> I'm trying to find a copy of 2.9.2 to see if I can reproduce it.
>
>
> Thanks,
> Christopher
>
> On Tue, Nov 29, 2011 at 9:11 AM, Trevor Watson <
> powersearchsoftware@gmail.com> wrote:
>
>> Sorry for the duplicate post. I was on the road and posted both via my
web
>> mail and office mail by mistake
>>
>> The increase is a very gradual, the program starts at about 160,000k
>> according to task manager (I know that's not entirely accurate, but it
was
>> the best I had at the time) and would, after adding 25,000-40,000 result
in
>> an out of memory exception (800,000k according to taskmanager). I tried
>> building a copy of 2.9.4 to test, but could not find one that worked in
>> visual studio 2005
>>
>> I did notice using Ants memory profiler that there were a number of
>> byte[32789] arrays that I didn't know where they came from in memory.
>>
>> On Monday, November 28, 2011, Christopher Currens <
currens.chris@gmail.com
>> >
>> wrote:
>> > Hi Trevor,
>> >
>> > What kind of memory increase are we talking about? Also, how big are
the
>> > documents that you are indexing, the ones returned from
getFileInfoDoc()?
>> > Is it putting an entire file into the index? Pre 2.9.3 versions had
>> > issues with holding onto allocated byte arrays far beyond when they
were
>> > used. The memory could only be freed via closing the IndexWriter.
>> >
>> > I'm a little unclear on exactly what's happening. Are you noticing
>> memory
>> > spike and stay constant at that level or is it a gradual increase? Is
it
>> > causing your application to error, (ie OutOfMemory exception, etc)?
>> >
>> >
>> > Thanks,
>> > Christopher
>> >
>> > On Mon, Nov 28, 2011 at 5:59 PM, Trevor Watson <
>> > powersearchsoftware@gmail.com> wrote:
>> >
>> >> I'm attempting to use Lucene.Net v2.9.2.2 in a Visual Studio 2005
(.NET
>> >> 2.0) environment. We had a piece of software that WAS working. I'm
not
>> >> sure what has changed however, the following code results in a memory
>> leak
>> >> in the Lucene.Net component (or a failure to clean up used memory).
>> >>
>> >> The code in issue is here:
>> >>
>> >> private void SaveFileToFileInfo(Lucene.Net.Index.IndexWriter iw, bool
>> >> delayCommit, string sDataPath)
>> >> {
>> >> Document doc = getFileInfoDoc(sDataPath);
>> >> Analyzer analyzer = clsLuceneFunctions.getAnalyzer();
>> >> if (this.FileID == 0)
>> >> {
>> >> string s = "";
>> >> }
>> >> iw.UpdateDocument(new Lucene.Net.Index.Term("FileId",
>> >> this.fileID.ToString("000000000")), doc, analyzer);
>> >>
>> >> analyzer = null;
>> >> doc = null;
>> >> if (!delayCommit)
>> >> iw.Commit();
>> >> }
>> >>
>> >> Commenting out the line iw.UpdateDocument resulted in no memory
>> increase.
>> >> I also tried replacing it with a deleteDocument and AddDocument and
the
>> >> memory increased the same as using the UpdateDocument function
>> >>
>> >> The getAnalyzer() function returns a ExtendedStandardAnalyzer, but
it's
>> the
>> >> UpdateDocument line specifically that gives me the issue.
>> >>
>> >> Any assistance would be greatly appreciated.
>> >>
>> >> Trevor Watson
>> >>
>> >
>>
>
Re: [Lucene.Net] Re: Memory Leak in 2.9.2.2
Posted by Christopher Currens <cu...@gmail.com>.
Do you know how big the documents are that you are trying to delete/update?
I'm trying to find a copy of 2.9.2 to see if I can reproduce it.
Thanks,
Christopher
On Tue, Nov 29, 2011 at 9:11 AM, Trevor Watson <
powersearchsoftware@gmail.com> wrote:
> Sorry for the duplicate post. I was on the road and posted both via my web
> mail and office mail by mistake
>
> The increase is a very gradual, the program starts at about 160,000k
> according to task manager (I know that's not entirely accurate, but it was
> the best I had at the time) and would, after adding 25,000-40,000 result in
> an out of memory exception (800,000k according to taskmanager). I tried
> building a copy of 2.9.4 to test, but could not find one that worked in
> visual studio 2005
>
> I did notice using Ants memory profiler that there were a number of
> byte[32789] arrays that I didn't know where they came from in memory.
>
> On Monday, November 28, 2011, Christopher Currens <currens.chris@gmail.com
> >
> wrote:
> > Hi Trevor,
> >
> > What kind of memory increase are we talking about? Also, how big are the
> > documents that you are indexing, the ones returned from getFileInfoDoc()?
> > Is it putting an entire file into the index? Pre 2.9.3 versions had
> > issues with holding onto allocated byte arrays far beyond when they were
> > used. The memory could only be freed via closing the IndexWriter.
> >
> > I'm a little unclear on exactly what's happening. Are you noticing
> memory
> > spike and stay constant at that level or is it a gradual increase? Is it
> > causing your application to error, (ie OutOfMemory exception, etc)?
> >
> >
> > Thanks,
> > Christopher
> >
> > On Mon, Nov 28, 2011 at 5:59 PM, Trevor Watson <
> > powersearchsoftware@gmail.com> wrote:
> >
> >> I'm attempting to use Lucene.Net v2.9.2.2 in a Visual Studio 2005 (.NET
> >> 2.0) environment. We had a piece of software that WAS working. I'm not
> >> sure what has changed however, the following code results in a memory
> leak
> >> in the Lucene.Net component (or a failure to clean up used memory).
> >>
> >> The code in issue is here:
> >>
> >> private void SaveFileToFileInfo(Lucene.Net.Index.IndexWriter iw, bool
> >> delayCommit, string sDataPath)
> >> {
> >> Document doc = getFileInfoDoc(sDataPath);
> >> Analyzer analyzer = clsLuceneFunctions.getAnalyzer();
> >> if (this.FileID == 0)
> >> {
> >> string s = "";
> >> }
> >> iw.UpdateDocument(new Lucene.Net.Index.Term("FileId",
> >> this.fileID.ToString("000000000")), doc, analyzer);
> >>
> >> analyzer = null;
> >> doc = null;
> >> if (!delayCommit)
> >> iw.Commit();
> >> }
> >>
> >> Commenting out the line iw.UpdateDocument resulted in no memory
> increase.
> >> I also tried replacing it with a deleteDocument and AddDocument and the
> >> memory increased the same as using the UpdateDocument function
> >>
> >> The getAnalyzer() function returns a ExtendedStandardAnalyzer, but it's
> the
> >> UpdateDocument line specifically that gives me the issue.
> >>
> >> Any assistance would be greatly appreciated.
> >>
> >> Trevor Watson
> >>
> >
>