You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucenenet.apache.org by George Aroush <ge...@aroush.net> on 2006/09/11 02:50:12 UTC
RE: Lucene.Net Indexing Large Databases
Hi Folks,
Since last weekend, I have been trying to narrow down the problem to this
memory leak without much of a luck.
Does anyone have a tool (or could recommend one, without costing me $$)
which hopefully show the source of the leak?
Unlike C++ code, the leak here, obviously, is due to not releasing
references to temporary or real objects. The trick is finding the object.
This leak can be created with this simple code:
public static void Main(System.String[] args)
{
IndexWriter diskIndex;
Directory directory;
Analyzer analyzer;
Document doc;
int count;
string indexDirectory;
System.IO.FileInfo fi;
indexDirectory = "C:\\Index.Bad";
fi = new System.IO.FileInfo(indexDirectory);
directory = Lucene.Net.Store.FSDirectory.GetDirectory(fi,
true);
analyzer = new SimpleAnalyzer();
diskIndex = new IndexWriter(directory, analyzer, true);
count = 0;
while (count < 10000)
{
doc = new Document();
diskIndex.AddDocument(doc);
count++;
}
diskIndex.Close();
}
This code will show a leak in 1.9, 1.9.1 and 2.0 but not 1.4.3. I also
verified and it doesn't leak under the Java version of Lucene (2.0 is where
I tested.)
Regards,
-- George
-----Original Message-----
From: George Aroush [mailto:george@aroush.net]
Sent: Friday, September 01, 2006 9:21 PM
To: lucene-net-user@incubator.apache.org
Subject: RE: Lucene.Net Indexing Large Databases
Hi Chris,
I am using 1.9.1 in production and I am not having this problem. Sorry, I
don't have enough cycles to try your code on 1.9.
This problem was reported on 1.4.x and was fixed. I am sure I carried it
over to 1.9.x and 2.0 -- or maybe this is a new issue. I will double check
when I get the cycles.
You can get 1.4.3's source code as ZIP from the download site of Lucene.Net
which is here:
https://svn.apache.org/repos/asf/incubator/lucene.net/site/download/ or you
can SVN the source code from here:
https://svn.apache.org/repos/asf/incubator/lucene.net/tags/
Regards,
-- George Aroush
-----Original Message-----
From: Chris David [mailto:chris@snapstream.com]
Sent: Friday, September 01, 2006 1:46 PM
To: lucene-net-user@incubator.apache.org
Subject: RE: Lucene.Net Indexing Large Databases
Thanks René, so its not just me with this problem. Now where can I get a
hold of this wonderful 1.4 Build of Lucene. Its not listed directly on
Apache's Lucene.NET Page. I am anxious to see if my code actually does work.
Thanks again for all your help, I really do appreciate it.
Chris
Snapstream Media
-----Original Message-----
From: René de Vries [mailto:rene.de.vries@howardshome.com]
Sent: Friday, September 01, 2006 7:32 AM
To: lucene-net-user@incubator.apache.org
Subject: RE: Lucene.Net Indexing Large Databases
Update: I didn't realize my earlier code example ran against 1.4.
If I run this with 1.9final-005 build, I am experiencing the exact same
problems as Chris mentions. Memory consumtion keeps growing, I had to kill
it at 1.5Gb. Exact same code, but with a 1.4 version of the lucene.net DLL,
and it runs along at 50Mb
René
Memory leak (was: RE: Lucene.Net Indexing Large Databases)
Posted by George Aroush <ge...@aroush.net>.
Thanks Neil.
Per my response to Eyal, the CLR Profile did show that I am looking at the
right suspected, but it's not showing me who is holding on the references.
Maybe it's there but I haven't figured out how to use it?
-- George
-----Original Message-----
From: Michael Mitiaguin [mailto:mitiaguin@gmail.com]
Sent: Sunday, September 10, 2006 9:54 PM
To: lucene-net-user@incubator.apache.org
Subject: Re: Lucene.Net Indexing Large Databases
George,
Microsoft provides a free CLR profiler. Can't say anything since never used.
http://www.microsoft.com/downloads/details.aspx?familyid=A362781C-3870-43BE-
8926-862B40AA0CD0&displaylang=en
Michael
On 9/11/06, Neil Carson <nc...@everdreamcorp.com> wrote:
>
> George
>
> An excellent tool is YourKit CPU and memory profiler. Trivial to use,
> 30 day trial, just gets the job done and nothing more/less. See
> www.yourkit.com (http://www.yourkit.com/dotnet/download/index.jsp)
>
> Neil
>
> ________________________________
>
> From: George Aroush [mailto:george@aroush.net]
> Sent: Sun 9/10/2006 5:50 PM
> To: lucene-net-user@incubator.apache.org;
> lucene-net-dev@incubator.apache.org
> Subject: RE: Lucene.Net Indexing Large Databases
>
>
>
> Hi Folks,
>
> Since last weekend, I have been trying to narrow down the problem to
> this memory leak without much of a luck.
>
> Does anyone have a tool (or could recommend one, without costing me
> $$) which hopefully show the source of the leak?
>
> Unlike C++ code, the leak here, obviously, is due to not releasing
> references to temporary or real objects. The trick is finding the object.
>
> This leak can be created with this simple code:
>
> public static void Main(System.String[] args)
> {
> IndexWriter diskIndex;
> Directory directory;
> Analyzer analyzer;
> Document doc;
> int count;
> string indexDirectory;
> System.IO.FileInfo fi;
>
> indexDirectory = "C:\\Index.Bad";
>
> fi = new System.IO.FileInfo(indexDirectory);
> directory =
> Lucene.Net.Store.FSDirectory.GetDirectory(fi,
> true);
>
> analyzer = new SimpleAnalyzer();
> diskIndex = new IndexWriter(directory, analyzer,
> true);
>
> count = 0;
> while (count < 10000)
> {
> doc = new Document();
> diskIndex.AddDocument(doc);
> count++;
> }
>
> diskIndex.Close();
> }
>
> This code will show a leak in 1.9, 1.9.1 and 2.0 but not 1.4.3. I
> also verified and it doesn't leak under the Java version of Lucene
> (2.0 is where I tested.)
>
> Regards,
>
> -- George
>
>
> -----Original Message-----
> From: George Aroush [mailto:george@aroush.net]
> Sent: Friday, September 01, 2006 9:21 PM
> To: lucene-net-user@incubator.apache.org
> Subject: RE: Lucene.Net Indexing Large Databases
>
> Hi Chris,
>
> I am using 1.9.1 in production and I am not having this problem.
> Sorry, I don't have enough cycles to try your code on 1.9.
>
> This problem was reported on 1.4.x and was fixed. I am sure I carried
> it over to 1.9.x and 2.0 -- or maybe this is a new issue. I will
> double check when I get the cycles.
>
> You can get 1.4.3's source code as ZIP from the download site of
> Lucene.Net which is here:
> https://svn.apache.org/repos/asf/incubator/lucene.net/site/download/
> or you can SVN the source code from here:
> https://svn.apache.org/repos/asf/incubator/lucene.net/tags/
>
> Regards,
>
> -- George Aroush
>
>
> -----Original Message-----
> From: Chris David [mailto:chris@snapstream.com]
> Sent: Friday, September 01, 2006 1:46 PM
> To: lucene-net-user@incubator.apache.org
> Subject: RE: Lucene.Net Indexing Large Databases
>
> Thanks René, so its not just me with this problem. Now where can I
> get a hold of this wonderful 1.4 Build of Lucene. Its not listed
> directly on Apache's Lucene.NET Page. I am anxious to see if my code
> actually does work.
> Thanks again for all your help, I really do appreciate it.
>
> Chris
> Snapstream Media
> -----Original Message-----
> From: René de Vries [mailto:rene.de.vries@howardshome.com]
> Sent: Friday, September 01, 2006 7:32 AM
> To: lucene-net-user@incubator.apache.org
> Subject: RE: Lucene.Net Indexing Large Databases
>
> Update: I didn't realize my earlier code example ran against 1.4.
>
> If I run this with 1.9final-005 build, I am experiencing the exact
> same problems as Chris mentions. Memory consumtion keeps growing, I
> had to kill it at 1.5Gb. Exact same code, but with a 1.4 version of
> the lucene.netDLL, and it runs along at 50Mb
>
> René
>
>
>
>
>
Re: Lucene.Net Indexing Large Databases
Posted by Michael Mitiaguin <mi...@gmail.com>.
George,
Microsoft provides a free CLR profiler. Can't say anything since never used.
http://www.microsoft.com/downloads/details.aspx?familyid=A362781C-3870-43BE-8926-862B40AA0CD0&displaylang=en
Michael
On 9/11/06, Neil Carson <nc...@everdreamcorp.com> wrote:
>
> George
>
> An excellent tool is YourKit CPU and memory profiler. Trivial to use, 30
> day trial, just gets the job done and nothing more/less. See
> www.yourkit.com (http://www.yourkit.com/dotnet/download/index.jsp)
>
> Neil
>
> ________________________________
>
> From: George Aroush [mailto:george@aroush.net]
> Sent: Sun 9/10/2006 5:50 PM
> To: lucene-net-user@incubator.apache.org;
> lucene-net-dev@incubator.apache.org
> Subject: RE: Lucene.Net Indexing Large Databases
>
>
>
> Hi Folks,
>
> Since last weekend, I have been trying to narrow down the problem to this
> memory leak without much of a luck.
>
> Does anyone have a tool (or could recommend one, without costing me $$)
> which hopefully show the source of the leak?
>
> Unlike C++ code, the leak here, obviously, is due to not releasing
> references to temporary or real objects. The trick is finding the object.
>
> This leak can be created with this simple code:
>
> public static void Main(System.String[] args)
> {
> IndexWriter diskIndex;
> Directory directory;
> Analyzer analyzer;
> Document doc;
> int count;
> string indexDirectory;
> System.IO.FileInfo fi;
>
> indexDirectory = "C:\\Index.Bad";
>
> fi = new System.IO.FileInfo(indexDirectory);
> directory = Lucene.Net.Store.FSDirectory.GetDirectory(fi,
> true);
>
> analyzer = new SimpleAnalyzer();
> diskIndex = new IndexWriter(directory, analyzer, true);
>
> count = 0;
> while (count < 10000)
> {
> doc = new Document();
> diskIndex.AddDocument(doc);
> count++;
> }
>
> diskIndex.Close();
> }
>
> This code will show a leak in 1.9, 1.9.1 and 2.0 but not 1.4.3. I also
> verified and it doesn't leak under the Java version of Lucene (2.0 is
> where
> I tested.)
>
> Regards,
>
> -- George
>
>
> -----Original Message-----
> From: George Aroush [mailto:george@aroush.net]
> Sent: Friday, September 01, 2006 9:21 PM
> To: lucene-net-user@incubator.apache.org
> Subject: RE: Lucene.Net Indexing Large Databases
>
> Hi Chris,
>
> I am using 1.9.1 in production and I am not having this problem. Sorry, I
> don't have enough cycles to try your code on 1.9.
>
> This problem was reported on 1.4.x and was fixed. I am sure I carried it
> over to 1.9.x and 2.0 -- or maybe this is a new issue. I will double
> check
> when I get the cycles.
>
> You can get 1.4.3's source code as ZIP from the download site of
> Lucene.Net
> which is here:
> https://svn.apache.org/repos/asf/incubator/lucene.net/site/download/ or
> you
> can SVN the source code from here:
> https://svn.apache.org/repos/asf/incubator/lucene.net/tags/
>
> Regards,
>
> -- George Aroush
>
>
> -----Original Message-----
> From: Chris David [mailto:chris@snapstream.com]
> Sent: Friday, September 01, 2006 1:46 PM
> To: lucene-net-user@incubator.apache.org
> Subject: RE: Lucene.Net Indexing Large Databases
>
> Thanks René, so its not just me with this problem. Now where can I get a
> hold of this wonderful 1.4 Build of Lucene. Its not listed directly on
> Apache's Lucene.NET Page. I am anxious to see if my code actually does
> work.
> Thanks again for all your help, I really do appreciate it.
>
> Chris
> Snapstream Media
> -----Original Message-----
> From: René de Vries [mailto:rene.de.vries@howardshome.com]
> Sent: Friday, September 01, 2006 7:32 AM
> To: lucene-net-user@incubator.apache.org
> Subject: RE: Lucene.Net Indexing Large Databases
>
> Update: I didn't realize my earlier code example ran against 1.4.
>
> If I run this with 1.9final-005 build, I am experiencing the exact same
> problems as Chris mentions. Memory consumtion keeps growing, I had to kill
> it at 1.5Gb. Exact same code, but with a 1.4 version of the lucene.netDLL,
> and it runs along at 50Mb
>
> René
>
>
>
>
>
RE: Lucene.Net Indexing Large Databases
Posted by Neil Carson <nc...@everdreamcorp.com>.
George
An excellent tool is YourKit CPU and memory profiler. Trivial to use, 30 day trial, just gets the job done and nothing more/less. See www.yourkit.com (http://www.yourkit.com/dotnet/download/index.jsp)
Neil
________________________________
From: George Aroush [mailto:george@aroush.net]
Sent: Sun 9/10/2006 5:50 PM
To: lucene-net-user@incubator.apache.org; lucene-net-dev@incubator.apache.org
Subject: RE: Lucene.Net Indexing Large Databases
Hi Folks,
Since last weekend, I have been trying to narrow down the problem to this
memory leak without much of a luck.
Does anyone have a tool (or could recommend one, without costing me $$)
which hopefully show the source of the leak?
Unlike C++ code, the leak here, obviously, is due to not releasing
references to temporary or real objects. The trick is finding the object.
This leak can be created with this simple code:
public static void Main(System.String[] args)
{
IndexWriter diskIndex;
Directory directory;
Analyzer analyzer;
Document doc;
int count;
string indexDirectory;
System.IO.FileInfo fi;
indexDirectory = "C:\\Index.Bad";
fi = new System.IO.FileInfo(indexDirectory);
directory = Lucene.Net.Store.FSDirectory.GetDirectory(fi,
true);
analyzer = new SimpleAnalyzer();
diskIndex = new IndexWriter(directory, analyzer, true);
count = 0;
while (count < 10000)
{
doc = new Document();
diskIndex.AddDocument(doc);
count++;
}
diskIndex.Close();
}
This code will show a leak in 1.9, 1.9.1 and 2.0 but not 1.4.3. I also
verified and it doesn't leak under the Java version of Lucene (2.0 is where
I tested.)
Regards,
-- George
-----Original Message-----
From: George Aroush [mailto:george@aroush.net]
Sent: Friday, September 01, 2006 9:21 PM
To: lucene-net-user@incubator.apache.org
Subject: RE: Lucene.Net Indexing Large Databases
Hi Chris,
I am using 1.9.1 in production and I am not having this problem. Sorry, I
don't have enough cycles to try your code on 1.9.
This problem was reported on 1.4.x and was fixed. I am sure I carried it
over to 1.9.x and 2.0 -- or maybe this is a new issue. I will double check
when I get the cycles.
You can get 1.4.3's source code as ZIP from the download site of Lucene.Net
which is here:
https://svn.apache.org/repos/asf/incubator/lucene.net/site/download/ or you
can SVN the source code from here:
https://svn.apache.org/repos/asf/incubator/lucene.net/tags/
Regards,
-- George Aroush
-----Original Message-----
From: Chris David [mailto:chris@snapstream.com]
Sent: Friday, September 01, 2006 1:46 PM
To: lucene-net-user@incubator.apache.org
Subject: RE: Lucene.Net Indexing Large Databases
Thanks René, so its not just me with this problem. Now where can I get a
hold of this wonderful 1.4 Build of Lucene. Its not listed directly on
Apache's Lucene.NET Page. I am anxious to see if my code actually does work.
Thanks again for all your help, I really do appreciate it.
Chris
Snapstream Media
-----Original Message-----
From: René de Vries [mailto:rene.de.vries@howardshome.com]
Sent: Friday, September 01, 2006 7:32 AM
To: lucene-net-user@incubator.apache.org
Subject: RE: Lucene.Net Indexing Large Databases
Update: I didn't realize my earlier code example ran against 1.4.
If I run this with 1.9final-005 build, I am experiencing the exact same
problems as Chris mentions. Memory consumtion keeps growing, I had to kill
it at 1.5Gb. Exact same code, but with a 1.4 version of the lucene.net DLL,
and it runs along at 50Mb
René
RE: Lucene.Net Indexing Large Databases
Posted by Neil Carson <nc...@everdreamcorp.com>.
George
An excellent tool is YourKit CPU and memory profiler. Trivial to use, 30 day trial, just gets the job done and nothing more/less. See www.yourkit.com (http://www.yourkit.com/dotnet/download/index.jsp)
Neil
________________________________
From: George Aroush [mailto:george@aroush.net]
Sent: Sun 9/10/2006 5:50 PM
To: lucene-net-user@incubator.apache.org; lucene-net-dev@incubator.apache.org
Subject: RE: Lucene.Net Indexing Large Databases
Hi Folks,
Since last weekend, I have been trying to narrow down the problem to this
memory leak without much of a luck.
Does anyone have a tool (or could recommend one, without costing me $$)
which hopefully show the source of the leak?
Unlike C++ code, the leak here, obviously, is due to not releasing
references to temporary or real objects. The trick is finding the object.
This leak can be created with this simple code:
public static void Main(System.String[] args)
{
IndexWriter diskIndex;
Directory directory;
Analyzer analyzer;
Document doc;
int count;
string indexDirectory;
System.IO.FileInfo fi;
indexDirectory = "C:\\Index.Bad";
fi = new System.IO.FileInfo(indexDirectory);
directory = Lucene.Net.Store.FSDirectory.GetDirectory(fi,
true);
analyzer = new SimpleAnalyzer();
diskIndex = new IndexWriter(directory, analyzer, true);
count = 0;
while (count < 10000)
{
doc = new Document();
diskIndex.AddDocument(doc);
count++;
}
diskIndex.Close();
}
This code will show a leak in 1.9, 1.9.1 and 2.0 but not 1.4.3. I also
verified and it doesn't leak under the Java version of Lucene (2.0 is where
I tested.)
Regards,
-- George
-----Original Message-----
From: George Aroush [mailto:george@aroush.net]
Sent: Friday, September 01, 2006 9:21 PM
To: lucene-net-user@incubator.apache.org
Subject: RE: Lucene.Net Indexing Large Databases
Hi Chris,
I am using 1.9.1 in production and I am not having this problem. Sorry, I
don't have enough cycles to try your code on 1.9.
This problem was reported on 1.4.x and was fixed. I am sure I carried it
over to 1.9.x and 2.0 -- or maybe this is a new issue. I will double check
when I get the cycles.
You can get 1.4.3's source code as ZIP from the download site of Lucene.Net
which is here:
https://svn.apache.org/repos/asf/incubator/lucene.net/site/download/ or you
can SVN the source code from here:
https://svn.apache.org/repos/asf/incubator/lucene.net/tags/
Regards,
-- George Aroush
-----Original Message-----
From: Chris David [mailto:chris@snapstream.com]
Sent: Friday, September 01, 2006 1:46 PM
To: lucene-net-user@incubator.apache.org
Subject: RE: Lucene.Net Indexing Large Databases
Thanks René, so its not just me with this problem. Now where can I get a
hold of this wonderful 1.4 Build of Lucene. Its not listed directly on
Apache's Lucene.NET Page. I am anxious to see if my code actually does work.
Thanks again for all your help, I really do appreciate it.
Chris
Snapstream Media
-----Original Message-----
From: René de Vries [mailto:rene.de.vries@howardshome.com]
Sent: Friday, September 01, 2006 7:32 AM
To: lucene-net-user@incubator.apache.org
Subject: RE: Lucene.Net Indexing Large Databases
Update: I didn't realize my earlier code example ran against 1.4.
If I run this with 1.9final-005 build, I am experiencing the exact same
problems as Chris mentions. Memory consumtion keeps growing, I had to kill
it at 1.5Gb. Exact same code, but with a 1.4 version of the lucene.net DLL,
and it runs along at 50Mb
René
Re: Lucene.Net Indexing Large Databases
Posted by Jeff Rodenburg <je...@gmail.com>.
Hi George -
About a year ago we had a memory leak around some issues with the
1.4.3code. A few of us wrote some sample programs that manifested the
error, but
I was able to do a fair amount of sleuthing with Memprofiler (
http://memprofiler.com/). It's a pretty good tool for $100.
-- j
On 9/10/06, George Aroush <ge...@aroush.net> wrote:
>
> Hi Folks,
>
> Since last weekend, I have been trying to narrow down the problem to this
> memory leak without much of a luck.
>
> Does anyone have a tool (or could recommend one, without costing me $$)
> which hopefully show the source of the leak?
>
> Unlike C++ code, the leak here, obviously, is due to not releasing
> references to temporary or real objects. The trick is finding the object.
>
> This leak can be created with this simple code:
>
> public static void Main(System.String[] args)
> {
> IndexWriter diskIndex;
> Directory directory;
> Analyzer analyzer;
> Document doc;
> int count;
> string indexDirectory;
> System.IO.FileInfo fi;
>
> indexDirectory = "C:\\Index.Bad";
>
> fi = new System.IO.FileInfo(indexDirectory);
> directory = Lucene.Net.Store.FSDirectory.GetDirectory(fi,
> true);
>
> analyzer = new SimpleAnalyzer();
> diskIndex = new IndexWriter(directory, analyzer, true);
>
> count = 0;
> while (count < 10000)
> {
> doc = new Document();
> diskIndex.AddDocument(doc);
> count++;
> }
>
> diskIndex.Close();
> }
>
> This code will show a leak in 1.9, 1.9.1 and 2.0 but not 1.4.3. I also
> verified and it doesn't leak under the Java version of Lucene (2.0 is
> where
> I tested.)
>
> Regards,
>
> -- George
>
>
> -----Original Message-----
> From: George Aroush [mailto:george@aroush.net]
> Sent: Friday, September 01, 2006 9:21 PM
> To: lucene-net-user@incubator.apache.org
> Subject: RE: Lucene.Net Indexing Large Databases
>
> Hi Chris,
>
> I am using 1.9.1 in production and I am not having this problem. Sorry, I
> don't have enough cycles to try your code on 1.9.
>
> This problem was reported on 1.4.x and was fixed. I am sure I carried it
> over to 1.9.x and 2.0 -- or maybe this is a new issue. I will double
> check
> when I get the cycles.
>
> You can get 1.4.3's source code as ZIP from the download site of
> Lucene.Net
> which is here:
> https://svn.apache.org/repos/asf/incubator/lucene.net/site/download/ or
> you
> can SVN the source code from here:
> https://svn.apache.org/repos/asf/incubator/lucene.net/tags/
>
> Regards,
>
> -- George Aroush
>
>
> -----Original Message-----
> From: Chris David [mailto:chris@snapstream.com]
> Sent: Friday, September 01, 2006 1:46 PM
> To: lucene-net-user@incubator.apache.org
> Subject: RE: Lucene.Net Indexing Large Databases
>
> Thanks René, so its not just me with this problem. Now where can I get a
> hold of this wonderful 1.4 Build of Lucene. Its not listed directly on
> Apache's Lucene.NET Page. I am anxious to see if my code actually does
> work.
> Thanks again for all your help, I really do appreciate it.
>
> Chris
> Snapstream Media
> -----Original Message-----
> From: René de Vries [mailto:rene.de.vries@howardshome.com]
> Sent: Friday, September 01, 2006 7:32 AM
> To: lucene-net-user@incubator.apache.org
> Subject: RE: Lucene.Net Indexing Large Databases
>
> Update: I didn't realize my earlier code example ran against 1.4.
>
> If I run this with 1.9final-005 build, I am experiencing the exact same
> problems as Chris mentions. Memory consumtion keeps growing, I had to kill
> it at 1.5Gb. Exact same code, but with a 1.4 version of the lucene.netDLL,
> and it runs along at 50Mb
>
> René
>
>
Re: Lucene.Net Indexing Large Databases
Posted by Jeff Rodenburg <je...@gmail.com>.
Hi George -
About a year ago we had a memory leak around some issues with the
1.4.3code. A few of us wrote some sample programs that manifested the
error, but
I was able to do a fair amount of sleuthing with Memprofiler (
http://memprofiler.com/). It's a pretty good tool for $100.
-- j
On 9/10/06, George Aroush <ge...@aroush.net> wrote:
>
> Hi Folks,
>
> Since last weekend, I have been trying to narrow down the problem to this
> memory leak without much of a luck.
>
> Does anyone have a tool (or could recommend one, without costing me $$)
> which hopefully show the source of the leak?
>
> Unlike C++ code, the leak here, obviously, is due to not releasing
> references to temporary or real objects. The trick is finding the object.
>
> This leak can be created with this simple code:
>
> public static void Main(System.String[] args)
> {
> IndexWriter diskIndex;
> Directory directory;
> Analyzer analyzer;
> Document doc;
> int count;
> string indexDirectory;
> System.IO.FileInfo fi;
>
> indexDirectory = "C:\\Index.Bad";
>
> fi = new System.IO.FileInfo(indexDirectory);
> directory = Lucene.Net.Store.FSDirectory.GetDirectory(fi,
> true);
>
> analyzer = new SimpleAnalyzer();
> diskIndex = new IndexWriter(directory, analyzer, true);
>
> count = 0;
> while (count < 10000)
> {
> doc = new Document();
> diskIndex.AddDocument(doc);
> count++;
> }
>
> diskIndex.Close();
> }
>
> This code will show a leak in 1.9, 1.9.1 and 2.0 but not 1.4.3. I also
> verified and it doesn't leak under the Java version of Lucene (2.0 is
> where
> I tested.)
>
> Regards,
>
> -- George
>
>
> -----Original Message-----
> From: George Aroush [mailto:george@aroush.net]
> Sent: Friday, September 01, 2006 9:21 PM
> To: lucene-net-user@incubator.apache.org
> Subject: RE: Lucene.Net Indexing Large Databases
>
> Hi Chris,
>
> I am using 1.9.1 in production and I am not having this problem. Sorry, I
> don't have enough cycles to try your code on 1.9.
>
> This problem was reported on 1.4.x and was fixed. I am sure I carried it
> over to 1.9.x and 2.0 -- or maybe this is a new issue. I will double
> check
> when I get the cycles.
>
> You can get 1.4.3's source code as ZIP from the download site of
> Lucene.Net
> which is here:
> https://svn.apache.org/repos/asf/incubator/lucene.net/site/download/ or
> you
> can SVN the source code from here:
> https://svn.apache.org/repos/asf/incubator/lucene.net/tags/
>
> Regards,
>
> -- George Aroush
>
>
> -----Original Message-----
> From: Chris David [mailto:chris@snapstream.com]
> Sent: Friday, September 01, 2006 1:46 PM
> To: lucene-net-user@incubator.apache.org
> Subject: RE: Lucene.Net Indexing Large Databases
>
> Thanks René, so its not just me with this problem. Now where can I get a
> hold of this wonderful 1.4 Build of Lucene. Its not listed directly on
> Apache's Lucene.NET Page. I am anxious to see if my code actually does
> work.
> Thanks again for all your help, I really do appreciate it.
>
> Chris
> Snapstream Media
> -----Original Message-----
> From: René de Vries [mailto:rene.de.vries@howardshome.com]
> Sent: Friday, September 01, 2006 7:32 AM
> To: lucene-net-user@incubator.apache.org
> Subject: RE: Lucene.Net Indexing Large Databases
>
> Update: I didn't realize my earlier code example ran against 1.4.
>
> If I run this with 1.9final-005 build, I am experiencing the exact same
> problems as Chris mentions. Memory consumtion keeps growing, I had to kill
> it at 1.5Gb. Exact same code, but with a 1.4 version of the lucene.netDLL,
> and it runs along at 50Mb
>
> René
>
>
RE: Memory leak (was: RE: Lucene.Net Indexing Large Databases)
Posted by Dean Harding <de...@dload.com.au>.
> The difference was that the implementation in Java used a WeakHashMap vs.
> the C# version used a straight Hashtable. Java's WeakHashMap uses weak
> references; the Hashtable has strong references
.NET has a WeakReference class, though I don't think there's a built-in
"WeakHashtable" class. Shouldn't be impossible to write one...
Dean.
> -----Original Message-----
> From: Jeff Rodenburg [mailto:jeff.rodenburg@gmail.com]
> Sent: Tuesday, 12 September 2006 11:59 am
> To: lucene-net-user@incubator.apache.org
> Subject: Re: Memory leak (was: RE: Lucene.Net Indexing Large Databases)
>
>
> -- j
>
>
> On 9/11/06, George Aroush <ge...@aroush.net> wrote:
> >
> > Hi Eyal,
> >
> > If you run this sample program (see email thread) using Lucene.Net
> 1.4.3you
> > will not see any memory growth, but if you try with 1.9, 1.9.1 and 2.0,
> > memory will consistently grow.
> >
> > Also, if you port this program to the Java version, again, using 1.4.3
> or
> > 2.0 (I didn't try 1.9 or 1.9.1) you won't see any memory growth.
> >
> > Clearly, we have a leak somewhere in Lucene.Net 1.9, 1.9.1 and 2.0.
> There
> > was a similar problem with 1.4.3 and the fix was adding a Close() --
> > sorry,
> > I don't remember in which method, but if you are interested you can look
> > it
> > up on dotLucne when it was at SourceForge. I should point out that the
> > Java
> > version doesn't need the Close() call.
> >
> > Regards,
> >
> > -- George
> >
> > -----Original Message-----
> > From: Eyal Post [mailto:eyalpost@epocalipse.com]
> > Sent: Monday, September 11, 2006 1:19 PM
> > To: lucene-net-user@incubator.apache.org
> > Subject: RE: Lucene.Net Indexing Large Databases
> >
> > Hi George,
> >
> > Can you explain how do you see that there's actually a leak?
> >
> > Eyal
> >
> > > -----Original Message-----
> > > From: George Aroush [mailto:george@aroush.net]
> > > Sent: Monday, September 11, 2006 3:50 AM
> > > To: lucene-net-user@incubator.apache.org;
> > > lucene-net-dev@incubator.apache.org
> > > Subject: RE: Lucene.Net Indexing Large Databases
> > >
> > > Hi Folks,
> > >
> > > Since last weekend, I have been trying to narrow down the problem to
> > > this memory leak without much of a luck.
> > >
> > > Does anyone have a tool (or could recommend one, without costing me
> > > $$) which hopefully show the source of the leak?
> > >
> > > Unlike C++ code, the leak here, obviously, is due to not releasing
> > > references to temporary or real objects. The trick is finding the
> > > object.
> > >
> > > This leak can be created with this simple code:
> > >
> > > public static void Main(System.String[] args)
> > > {
> > > IndexWriter diskIndex;
> > > Directory directory;
> > > Analyzer analyzer;
> > > Document doc;
> > > int count;
> > > string indexDirectory;
> > > System.IO.FileInfo fi;
> > >
> > > indexDirectory = "C:\\Index.Bad";
> > >
> > > fi = new System.IO.FileInfo(indexDirectory);
> > > directory =
> > > Lucene.Net.Store.FSDirectory.GetDirectory(fi,
> > > true);
> > >
> > > analyzer = new SimpleAnalyzer();
> > > diskIndex = new IndexWriter(directory, analyzer, true);
> > >
> > > count = 0;
> > > while (count < 10000)
> > > {
> > > doc = new Document();
> > > diskIndex.AddDocument(doc);
> > > count++;
> > > }
> > >
> > > diskIndex.Close();
> > > }
> > >
> > > This code will show a leak in 1.9, 1.9.1 and 2.0 but not 1.4.3. I
> > > also verified and it doesn't leak under the Java version of Lucene
> > > (2.0 is where I tested.)
> > >
> > > Regards,
> > >
> > > -- George
> > >
> > >
> > > -----Original Message-----
> > > From: George Aroush [mailto:george@aroush.net]
> > > Sent: Friday, September 01, 2006 9:21 PM
> > > To: lucene-net-user@incubator.apache.org
> > > Subject: RE: Lucene.Net Indexing Large Databases
> > >
> > > Hi Chris,
> > >
> > > I am using 1.9.1 in production and I am not having this problem.
> > > Sorry, I don't have enough cycles to try your code on 1.9.
> > >
> > > This problem was reported on 1.4.x and was fixed. I am sure I carried
> > > it over to 1.9.x and 2.0 -- or maybe this is a new issue. I will
> > > double check when I get the cycles.
> > >
> > > You can get 1.4.3's source code as ZIP from the download site of
> > > Lucene.Net which is here:
> > > https://svn.apache.org/repos/asf/incubator/lucene.net/site/dow
> > nload/ or you can SVN the source code from here:
> > https://svn.apache.org/repos/asf/incubator/lucene.net/tags/
> >
> > Regards,
> >
> > -- George Aroush
> >
> >
> > -----Original Message-----
> > From: Chris David [mailto:chris@snapstream.com]
> > Sent: Friday, September 01, 2006 1:46 PM
> > To: lucene-net-user@incubator.apache.org
> > Subject: RE: Lucene.Net Indexing Large Databases
> >
> > Thanks René, so its not just me with this problem. Now where can I get
> a
> > hold of this wonderful 1.4 Build of Lucene. Its not listed directly on
> > Apache's Lucene.NET Page. I am anxious to see if my code actually does
> > work.
> > Thanks again for all your help, I really do appreciate it.
> >
> > Chris
> > Snapstream Media
> > -----Original Message-----
> > From: René de Vries [mailto:rene.de.vries@howardshome.com]
> > Sent: Friday, September 01, 2006 7:32 AM
> > To: lucene-net-user@incubator.apache.org
> > Subject: RE: Lucene.Net Indexing Large Databases
> >
> > Update: I didn't realize my earlier code example ran against 1.4.
> >
> > If I run this with 1.9final-005 build, I am experiencing the exact same
> > problems as Chris mentions. Memory consumtion keeps growing, I had to
> kill
> > it at 1.5Gb. Exact same code, but with a 1.4 version of the
> lucene.netDLL,
> > and it runs along at 50Mb
> >
> > René
> >
> >
> >
> >
Re: Memory leak (was: RE: Lucene.Net Indexing Large Databases)
Posted by Jeff Rodenburg <je...@gmail.com>.
The Close() method addition in the 1.4.3 code was in the FieldCache class.
The difference was that the implementation in Java used a WeakHashMap vs.
the C# version used a straight Hashtable. Java's WeakHashMap uses weak
references; the Hashtable has strong references. GC wasn't able to clear
the FieldCache, so a virtual Close() method with calls to explicitly release
references from internal Hashtable was added.
-- j
On 9/11/06, George Aroush <ge...@aroush.net> wrote:
>
> Hi Eyal,
>
> If you run this sample program (see email thread) using Lucene.Net 1.4.3you
> will not see any memory growth, but if you try with 1.9, 1.9.1 and 2.0,
> memory will consistently grow.
>
> Also, if you port this program to the Java version, again, using 1.4.3 or
> 2.0 (I didn't try 1.9 or 1.9.1) you won't see any memory growth.
>
> Clearly, we have a leak somewhere in Lucene.Net 1.9, 1.9.1 and 2.0. There
> was a similar problem with 1.4.3 and the fix was adding a Close() --
> sorry,
> I don't remember in which method, but if you are interested you can look
> it
> up on dotLucne when it was at SourceForge. I should point out that the
> Java
> version doesn't need the Close() call.
>
> Regards,
>
> -- George
>
> -----Original Message-----
> From: Eyal Post [mailto:eyalpost@epocalipse.com]
> Sent: Monday, September 11, 2006 1:19 PM
> To: lucene-net-user@incubator.apache.org
> Subject: RE: Lucene.Net Indexing Large Databases
>
> Hi George,
>
> Can you explain how do you see that there's actually a leak?
>
> Eyal
>
> > -----Original Message-----
> > From: George Aroush [mailto:george@aroush.net]
> > Sent: Monday, September 11, 2006 3:50 AM
> > To: lucene-net-user@incubator.apache.org;
> > lucene-net-dev@incubator.apache.org
> > Subject: RE: Lucene.Net Indexing Large Databases
> >
> > Hi Folks,
> >
> > Since last weekend, I have been trying to narrow down the problem to
> > this memory leak without much of a luck.
> >
> > Does anyone have a tool (or could recommend one, without costing me
> > $$) which hopefully show the source of the leak?
> >
> > Unlike C++ code, the leak here, obviously, is due to not releasing
> > references to temporary or real objects. The trick is finding the
> > object.
> >
> > This leak can be created with this simple code:
> >
> > public static void Main(System.String[] args)
> > {
> > IndexWriter diskIndex;
> > Directory directory;
> > Analyzer analyzer;
> > Document doc;
> > int count;
> > string indexDirectory;
> > System.IO.FileInfo fi;
> >
> > indexDirectory = "C:\\Index.Bad";
> >
> > fi = new System.IO.FileInfo(indexDirectory);
> > directory =
> > Lucene.Net.Store.FSDirectory.GetDirectory(fi,
> > true);
> >
> > analyzer = new SimpleAnalyzer();
> > diskIndex = new IndexWriter(directory, analyzer, true);
> >
> > count = 0;
> > while (count < 10000)
> > {
> > doc = new Document();
> > diskIndex.AddDocument(doc);
> > count++;
> > }
> >
> > diskIndex.Close();
> > }
> >
> > This code will show a leak in 1.9, 1.9.1 and 2.0 but not 1.4.3. I
> > also verified and it doesn't leak under the Java version of Lucene
> > (2.0 is where I tested.)
> >
> > Regards,
> >
> > -- George
> >
> >
> > -----Original Message-----
> > From: George Aroush [mailto:george@aroush.net]
> > Sent: Friday, September 01, 2006 9:21 PM
> > To: lucene-net-user@incubator.apache.org
> > Subject: RE: Lucene.Net Indexing Large Databases
> >
> > Hi Chris,
> >
> > I am using 1.9.1 in production and I am not having this problem.
> > Sorry, I don't have enough cycles to try your code on 1.9.
> >
> > This problem was reported on 1.4.x and was fixed. I am sure I carried
> > it over to 1.9.x and 2.0 -- or maybe this is a new issue. I will
> > double check when I get the cycles.
> >
> > You can get 1.4.3's source code as ZIP from the download site of
> > Lucene.Net which is here:
> > https://svn.apache.org/repos/asf/incubator/lucene.net/site/dow
> nload/ or you can SVN the source code from here:
> https://svn.apache.org/repos/asf/incubator/lucene.net/tags/
>
> Regards,
>
> -- George Aroush
>
>
> -----Original Message-----
> From: Chris David [mailto:chris@snapstream.com]
> Sent: Friday, September 01, 2006 1:46 PM
> To: lucene-net-user@incubator.apache.org
> Subject: RE: Lucene.Net Indexing Large Databases
>
> Thanks René, so its not just me with this problem. Now where can I get a
> hold of this wonderful 1.4 Build of Lucene. Its not listed directly on
> Apache's Lucene.NET Page. I am anxious to see if my code actually does
> work.
> Thanks again for all your help, I really do appreciate it.
>
> Chris
> Snapstream Media
> -----Original Message-----
> From: René de Vries [mailto:rene.de.vries@howardshome.com]
> Sent: Friday, September 01, 2006 7:32 AM
> To: lucene-net-user@incubator.apache.org
> Subject: RE: Lucene.Net Indexing Large Databases
>
> Update: I didn't realize my earlier code example ran against 1.4.
>
> If I run this with 1.9final-005 build, I am experiencing the exact same
> problems as Chris mentions. Memory consumtion keeps growing, I had to kill
> it at 1.5Gb. Exact same code, but with a 1.4 version of the lucene.netDLL,
> and it runs along at 50Mb
>
> René
>
>
>
>
Memory leak (was: RE: Lucene.Net Indexing Large Databases)
Posted by George Aroush <ge...@aroush.net>.
Hi Eyal,
If you run this sample program (see email thread) using Lucene.Net 1.4.3 you
will not see any memory growth, but if you try with 1.9, 1.9.1 and 2.0,
memory will consistently grow.
Also, if you port this program to the Java version, again, using 1.4.3 or
2.0 (I didn't try 1.9 or 1.9.1) you won't see any memory growth.
Clearly, we have a leak somewhere in Lucene.Net 1.9, 1.9.1 and 2.0. There
was a similar problem with 1.4.3 and the fix was adding a Close() -- sorry,
I don't remember in which method, but if you are interested you can look it
up on dotLucne when it was at SourceForge. I should point out that the Java
version doesn't need the Close() call.
Regards,
-- George
-----Original Message-----
From: Eyal Post [mailto:eyalpost@epocalipse.com]
Sent: Monday, September 11, 2006 1:19 PM
To: lucene-net-user@incubator.apache.org
Subject: RE: Lucene.Net Indexing Large Databases
Hi George,
Can you explain how do you see that there's actually a leak?
Eyal
> -----Original Message-----
> From: George Aroush [mailto:george@aroush.net]
> Sent: Monday, September 11, 2006 3:50 AM
> To: lucene-net-user@incubator.apache.org;
> lucene-net-dev@incubator.apache.org
> Subject: RE: Lucene.Net Indexing Large Databases
>
> Hi Folks,
>
> Since last weekend, I have been trying to narrow down the problem to
> this memory leak without much of a luck.
>
> Does anyone have a tool (or could recommend one, without costing me
> $$) which hopefully show the source of the leak?
>
> Unlike C++ code, the leak here, obviously, is due to not releasing
> references to temporary or real objects. The trick is finding the
> object.
>
> This leak can be created with this simple code:
>
> public static void Main(System.String[] args)
> {
> IndexWriter diskIndex;
> Directory directory;
> Analyzer analyzer;
> Document doc;
> int count;
> string indexDirectory;
> System.IO.FileInfo fi;
>
> indexDirectory = "C:\\Index.Bad";
>
> fi = new System.IO.FileInfo(indexDirectory);
> directory =
> Lucene.Net.Store.FSDirectory.GetDirectory(fi,
> true);
>
> analyzer = new SimpleAnalyzer();
> diskIndex = new IndexWriter(directory, analyzer, true);
>
> count = 0;
> while (count < 10000)
> {
> doc = new Document();
> diskIndex.AddDocument(doc);
> count++;
> }
>
> diskIndex.Close();
> }
>
> This code will show a leak in 1.9, 1.9.1 and 2.0 but not 1.4.3. I
> also verified and it doesn't leak under the Java version of Lucene
> (2.0 is where I tested.)
>
> Regards,
>
> -- George
>
>
> -----Original Message-----
> From: George Aroush [mailto:george@aroush.net]
> Sent: Friday, September 01, 2006 9:21 PM
> To: lucene-net-user@incubator.apache.org
> Subject: RE: Lucene.Net Indexing Large Databases
>
> Hi Chris,
>
> I am using 1.9.1 in production and I am not having this problem.
> Sorry, I don't have enough cycles to try your code on 1.9.
>
> This problem was reported on 1.4.x and was fixed. I am sure I carried
> it over to 1.9.x and 2.0 -- or maybe this is a new issue. I will
> double check when I get the cycles.
>
> You can get 1.4.3's source code as ZIP from the download site of
> Lucene.Net which is here:
> https://svn.apache.org/repos/asf/incubator/lucene.net/site/dow
nload/ or you can SVN the source code from here:
https://svn.apache.org/repos/asf/incubator/lucene.net/tags/
Regards,
-- George Aroush
-----Original Message-----
From: Chris David [mailto:chris@snapstream.com]
Sent: Friday, September 01, 2006 1:46 PM
To: lucene-net-user@incubator.apache.org
Subject: RE: Lucene.Net Indexing Large Databases
Thanks René, so its not just me with this problem. Now where can I get a
hold of this wonderful 1.4 Build of Lucene. Its not listed directly on
Apache's Lucene.NET Page. I am anxious to see if my code actually does work.
Thanks again for all your help, I really do appreciate it.
Chris
Snapstream Media
-----Original Message-----
From: René de Vries [mailto:rene.de.vries@howardshome.com]
Sent: Friday, September 01, 2006 7:32 AM
To: lucene-net-user@incubator.apache.org
Subject: RE: Lucene.Net Indexing Large Databases
Update: I didn't realize my earlier code example ran against 1.4.
If I run this with 1.9final-005 build, I am experiencing the exact same
problems as Chris mentions. Memory consumtion keeps growing, I had to kill
it at 1.5Gb. Exact same code, but with a 1.4 version of the lucene.net DLL,
and it runs along at 50Mb
René
RE: Lucene.Net Indexing Large Databases
Posted by Eyal Post <ey...@epocalipse.com>.
Hi George,
Can you explain how do you see that there's actually a leak?
Eyal
> -----Original Message-----
> From: George Aroush [mailto:george@aroush.net]
> Sent: Monday, September 11, 2006 3:50 AM
> To: lucene-net-user@incubator.apache.org;
> lucene-net-dev@incubator.apache.org
> Subject: RE: Lucene.Net Indexing Large Databases
>
> Hi Folks,
>
> Since last weekend, I have been trying to narrow down the
> problem to this memory leak without much of a luck.
>
> Does anyone have a tool (or could recommend one, without
> costing me $$) which hopefully show the source of the leak?
>
> Unlike C++ code, the leak here, obviously, is due to not
> releasing references to temporary or real objects. The trick
> is finding the object.
>
> This leak can be created with this simple code:
>
> public static void Main(System.String[] args)
> {
> IndexWriter diskIndex;
> Directory directory;
> Analyzer analyzer;
> Document doc;
> int count;
> string indexDirectory;
> System.IO.FileInfo fi;
>
> indexDirectory = "C:\\Index.Bad";
>
> fi = new System.IO.FileInfo(indexDirectory);
> directory =
> Lucene.Net.Store.FSDirectory.GetDirectory(fi,
> true);
>
> analyzer = new SimpleAnalyzer();
> diskIndex = new IndexWriter(directory, analyzer, true);
>
> count = 0;
> while (count < 10000)
> {
> doc = new Document();
> diskIndex.AddDocument(doc);
> count++;
> }
>
> diskIndex.Close();
> }
>
> This code will show a leak in 1.9, 1.9.1 and 2.0 but not
> 1.4.3. I also verified and it doesn't leak under the Java
> version of Lucene (2.0 is where I tested.)
>
> Regards,
>
> -- George
>
>
> -----Original Message-----
> From: George Aroush [mailto:george@aroush.net]
> Sent: Friday, September 01, 2006 9:21 PM
> To: lucene-net-user@incubator.apache.org
> Subject: RE: Lucene.Net Indexing Large Databases
>
> Hi Chris,
>
> I am using 1.9.1 in production and I am not having this
> problem. Sorry, I don't have enough cycles to try your code on 1.9.
>
> This problem was reported on 1.4.x and was fixed. I am sure
> I carried it over to 1.9.x and 2.0 -- or maybe this is a new
> issue. I will double check when I get the cycles.
>
> You can get 1.4.3's source code as ZIP from the download site
> of Lucene.Net which is here:
> https://svn.apache.org/repos/asf/incubator/lucene.net/site/dow
nload/ or you can SVN the source code from here:
https://svn.apache.org/repos/asf/incubator/lucene.net/tags/
Regards,
-- George Aroush
-----Original Message-----
From: Chris David [mailto:chris@snapstream.com]
Sent: Friday, September 01, 2006 1:46 PM
To: lucene-net-user@incubator.apache.org
Subject: RE: Lucene.Net Indexing Large Databases
Thanks René, so its not just me with this problem. Now where can I get a
hold of this wonderful 1.4 Build of Lucene. Its not listed directly on
Apache's Lucene.NET Page. I am anxious to see if my code actually does work.
Thanks again for all your help, I really do appreciate it.
Chris
Snapstream Media
-----Original Message-----
From: René de Vries [mailto:rene.de.vries@howardshome.com]
Sent: Friday, September 01, 2006 7:32 AM
To: lucene-net-user@incubator.apache.org
Subject: RE: Lucene.Net Indexing Large Databases
Update: I didn't realize my earlier code example ran against 1.4.
If I run this with 1.9final-005 build, I am experiencing the exact same
problems as Chris mentions. Memory consumtion keeps growing, I had to kill
it at 1.5Gb. Exact same code, but with a 1.4 version of the lucene.net DLL,
and it runs along at 50Mb
René
RE: Memory leak (was: RE: Lucene.Net Indexing Large Databases)
Posted by Narendra Tumkur <nt...@drive.com.au>.
Hi George,
Any news on this memory leak ?
Narendra Tumkur
Web Developer
drive.com.au
-----Original Message-----
From: George Aroush [mailto:george@aroush.net]
Sent: Wednesday, 13 September 2006 8:01 AM
To: lucene-net-dev@incubator.apache.org; lucene-net-user@incubator.apache.org
Subject: RE: Memory leak (was: RE: Lucene.Net Indexing Large Databases)
Hi Folks,
Some more update on the memory leak.
If I compile Lucene.Net using VC8 (i.e.: .NET 2.0) the leak won't occur. In
addition, with .NET 2.0, the destructor/finalize of TermInfosReader is
getting called, but not when compiled with .NET 1.1.
I checked the Java version, and there too, TermInfosReader.finalize() is
getting called.
No, the answer is not to simply switch over to .NET 2.0, I have a need to
run Lucene.Net on .NET 1.1 and I am sure many others do too.
Regards,
-- George
-----Original Message-----
From: George Aroush [mailto:george@aroush.net]
Sent: Monday, September 11, 2006 9:34 PM
To: lucene-net-dev@incubator.apache.org
Subject: Memory leak (was: RE: Lucene.Net Indexing Large Databases)
No, adding directory.Close() will not do the trick. However, I do believe
we are missing a Close() somewhere, most likely to SegmentReader or
TermInfosReader.
All indication are that one or both of those classes is the source of the
leak. Those were my suspects and the MS CLR Profiler is pointing in this
direction.
The Java version lands you into the Finalize of those two classes, but with
the C# version, the Finalize of those two classes is never getting called!
I hope to find out more in the next few days, but if anyone got the cycles
and got a nice tool to help with this defect, please give this a try and let
me know what you find.
Regards,
-- George
-----Original Message-----
From: Eyal Post [mailto:eyalpost@epocalipse.com]
Sent: Monday, September 11, 2006 6:50 PM
To: lucene-net-dev@incubator.apache.org
Subject: RE: Lucene.Net Indexing Large Databases
Adding:
directory.Close();
At the end of the method should help. Do you see still see leaks after that?
Eyal
> -----Original Message-----
> From: George Aroush [mailto:george@aroush.net]
> Sent: Monday, September 11, 2006 3:50 AM
> To: lucene-net-user@incubator.apache.org;
> lucene-net-dev@incubator.apache.org
> Subject: RE: Lucene.Net Indexing Large Databases
>
> Hi Folks,
>
> Since last weekend, I have been trying to narrow down the problem to
> this memory leak without much of a luck.
>
> Does anyone have a tool (or could recommend one, without costing me
> $$) which hopefully show the source of the leak?
>
> Unlike C++ code, the leak here, obviously, is due to not releasing
> references to temporary or real objects. The trick is finding the
> object.
>
> This leak can be created with this simple code:
>
> public static void Main(System.String[] args)
> {
> IndexWriter diskIndex;
> Directory directory;
> Analyzer analyzer;
> Document doc;
> int count;
> string indexDirectory;
> System.IO.FileInfo fi;
>
> indexDirectory = "C:\\Index.Bad";
>
> fi = new System.IO.FileInfo(indexDirectory);
> directory =
> Lucene.Net.Store.FSDirectory.GetDirectory(fi,
> true);
>
> analyzer = new SimpleAnalyzer();
> diskIndex = new IndexWriter(directory, analyzer, true);
>
> count = 0;
> while (count < 10000)
> {
> doc = new Document();
> diskIndex.AddDocument(doc);
> count++;
> }
>
> diskIndex.Close();
> }
>
> This code will show a leak in 1.9, 1.9.1 and 2.0 but not 1.4.3. I
> also verified and it doesn't leak under the Java version of Lucene
> (2.0 is where I tested.)
>
> Regards,
>
> -- George
>
>
> -----Original Message-----
> From: George Aroush [mailto:george@aroush.net]
> Sent: Friday, September 01, 2006 9:21 PM
> To: lucene-net-user@incubator.apache.org
> Subject: RE: Lucene.Net Indexing Large Databases
>
> Hi Chris,
>
> I am using 1.9.1 in production and I am not having this problem.
> Sorry, I don't have enough cycles to try your code on 1.9.
>
> This problem was reported on 1.4.x and was fixed. I am sure I carried
> it over to 1.9.x and 2.0 -- or maybe this is a new issue. I will
> double check when I get the cycles.
>
> You can get 1.4.3's source code as ZIP from the download site of
> Lucene.Net which is here:
> https://svn.apache.org/repos/asf/incubator/lucene.net/site/dow
nload/ or you can SVN the source code from here:
https://svn.apache.org/repos/asf/incubator/lucene.net/tags/
Regards,
-- George Aroush
-----Original Message-----
From: Chris David [mailto:chris@snapstream.com]
Sent: Friday, September 01, 2006 1:46 PM
To: lucene-net-user@incubator.apache.org
Subject: RE: Lucene.Net Indexing Large Databases
Thanks René, so its not just me with this problem. Now where can I get a
hold of this wonderful 1.4 Build of Lucene. Its not listed directly on
Apache's Lucene.NET Page. I am anxious to see if my code actually does work.
Thanks again for all your help, I really do appreciate it.
Chris
Snapstream Media
-----Original Message-----
From: René de Vries [mailto:rene.de.vries@howardshome.com]
Sent: Friday, September 01, 2006 7:32 AM
To: lucene-net-user@incubator.apache.org
Subject: RE: Lucene.Net Indexing Large Databases
Update: I didn't realize my earlier code example ran against 1.4.
If I run this with 1.9final-005 build, I am experiencing the exact same
problems as Chris mentions. Memory consumtion keeps growing, I had to kill
it at 1.5Gb. Exact same code, but with a 1.4 version of the lucene.net DLL,
and it runs along at 50Mb
René
RE: Memory leak (was: RE: Lucene.Net Indexing Large Databases)
Posted by George Aroush <ge...@aroush.net>.
Hi Folks,
Some more update on the memory leak.
If I compile Lucene.Net using VC8 (i.e.: .NET 2.0) the leak won't occur. In
addition, with .NET 2.0, the destructor/finalize of TermInfosReader is
getting called, but not when compiled with .NET 1.1.
I checked the Java version, and there too, TermInfosReader.finalize() is
getting called.
No, the answer is not to simply switch over to .NET 2.0, I have a need to
run Lucene.Net on .NET 1.1 and I am sure many others do too.
Regards,
-- George
-----Original Message-----
From: George Aroush [mailto:george@aroush.net]
Sent: Monday, September 11, 2006 9:34 PM
To: lucene-net-dev@incubator.apache.org
Subject: Memory leak (was: RE: Lucene.Net Indexing Large Databases)
No, adding directory.Close() will not do the trick. However, I do believe
we are missing a Close() somewhere, most likely to SegmentReader or
TermInfosReader.
All indication are that one or both of those classes is the source of the
leak. Those were my suspects and the MS CLR Profiler is pointing in this
direction.
The Java version lands you into the Finalize of those two classes, but with
the C# version, the Finalize of those two classes is never getting called!
I hope to find out more in the next few days, but if anyone got the cycles
and got a nice tool to help with this defect, please give this a try and let
me know what you find.
Regards,
-- George
-----Original Message-----
From: Eyal Post [mailto:eyalpost@epocalipse.com]
Sent: Monday, September 11, 2006 6:50 PM
To: lucene-net-dev@incubator.apache.org
Subject: RE: Lucene.Net Indexing Large Databases
Adding:
directory.Close();
At the end of the method should help. Do you see still see leaks after that?
Eyal
> -----Original Message-----
> From: George Aroush [mailto:george@aroush.net]
> Sent: Monday, September 11, 2006 3:50 AM
> To: lucene-net-user@incubator.apache.org;
> lucene-net-dev@incubator.apache.org
> Subject: RE: Lucene.Net Indexing Large Databases
>
> Hi Folks,
>
> Since last weekend, I have been trying to narrow down the problem to
> this memory leak without much of a luck.
>
> Does anyone have a tool (or could recommend one, without costing me
> $$) which hopefully show the source of the leak?
>
> Unlike C++ code, the leak here, obviously, is due to not releasing
> references to temporary or real objects. The trick is finding the
> object.
>
> This leak can be created with this simple code:
>
> public static void Main(System.String[] args)
> {
> IndexWriter diskIndex;
> Directory directory;
> Analyzer analyzer;
> Document doc;
> int count;
> string indexDirectory;
> System.IO.FileInfo fi;
>
> indexDirectory = "C:\\Index.Bad";
>
> fi = new System.IO.FileInfo(indexDirectory);
> directory =
> Lucene.Net.Store.FSDirectory.GetDirectory(fi,
> true);
>
> analyzer = new SimpleAnalyzer();
> diskIndex = new IndexWriter(directory, analyzer, true);
>
> count = 0;
> while (count < 10000)
> {
> doc = new Document();
> diskIndex.AddDocument(doc);
> count++;
> }
>
> diskIndex.Close();
> }
>
> This code will show a leak in 1.9, 1.9.1 and 2.0 but not 1.4.3. I
> also verified and it doesn't leak under the Java version of Lucene
> (2.0 is where I tested.)
>
> Regards,
>
> -- George
>
>
> -----Original Message-----
> From: George Aroush [mailto:george@aroush.net]
> Sent: Friday, September 01, 2006 9:21 PM
> To: lucene-net-user@incubator.apache.org
> Subject: RE: Lucene.Net Indexing Large Databases
>
> Hi Chris,
>
> I am using 1.9.1 in production and I am not having this problem.
> Sorry, I don't have enough cycles to try your code on 1.9.
>
> This problem was reported on 1.4.x and was fixed. I am sure I carried
> it over to 1.9.x and 2.0 -- or maybe this is a new issue. I will
> double check when I get the cycles.
>
> You can get 1.4.3's source code as ZIP from the download site of
> Lucene.Net which is here:
> https://svn.apache.org/repos/asf/incubator/lucene.net/site/dow
nload/ or you can SVN the source code from here:
https://svn.apache.org/repos/asf/incubator/lucene.net/tags/
Regards,
-- George Aroush
-----Original Message-----
From: Chris David [mailto:chris@snapstream.com]
Sent: Friday, September 01, 2006 1:46 PM
To: lucene-net-user@incubator.apache.org
Subject: RE: Lucene.Net Indexing Large Databases
Thanks René, so its not just me with this problem. Now where can I get a
hold of this wonderful 1.4 Build of Lucene. Its not listed directly on
Apache's Lucene.NET Page. I am anxious to see if my code actually does work.
Thanks again for all your help, I really do appreciate it.
Chris
Snapstream Media
-----Original Message-----
From: René de Vries [mailto:rene.de.vries@howardshome.com]
Sent: Friday, September 01, 2006 7:32 AM
To: lucene-net-user@incubator.apache.org
Subject: RE: Lucene.Net Indexing Large Databases
Update: I didn't realize my earlier code example ran against 1.4.
If I run this with 1.9final-005 build, I am experiencing the exact same
problems as Chris mentions. Memory consumtion keeps growing, I had to kill
it at 1.5Gb. Exact same code, but with a 1.4 version of the lucene.net DLL,
and it runs along at 50Mb
René
RE: Memory leak (was: RE: Lucene.Net Indexing Large Databases)
Posted by George Aroush <ge...@aroush.net>.
Hi Folks,
Some more update on the memory leak.
If I compile Lucene.Net using VC8 (i.e.: .NET 2.0) the leak won't occur. In
addition, with .NET 2.0, the destructor/finalize of TermInfosReader is
getting called, but not when compiled with .NET 1.1.
I checked the Java version, and there too, TermInfosReader.finalize() is
getting called.
No, the answer is not to simply switch over to .NET 2.0, I have a need to
run Lucene.Net on .NET 1.1 and I am sure many others do too.
Regards,
-- George
-----Original Message-----
From: George Aroush [mailto:george@aroush.net]
Sent: Monday, September 11, 2006 9:34 PM
To: lucene-net-dev@incubator.apache.org
Subject: Memory leak (was: RE: Lucene.Net Indexing Large Databases)
No, adding directory.Close() will not do the trick. However, I do believe
we are missing a Close() somewhere, most likely to SegmentReader or
TermInfosReader.
All indication are that one or both of those classes is the source of the
leak. Those were my suspects and the MS CLR Profiler is pointing in this
direction.
The Java version lands you into the Finalize of those two classes, but with
the C# version, the Finalize of those two classes is never getting called!
I hope to find out more in the next few days, but if anyone got the cycles
and got a nice tool to help with this defect, please give this a try and let
me know what you find.
Regards,
-- George
-----Original Message-----
From: Eyal Post [mailto:eyalpost@epocalipse.com]
Sent: Monday, September 11, 2006 6:50 PM
To: lucene-net-dev@incubator.apache.org
Subject: RE: Lucene.Net Indexing Large Databases
Adding:
directory.Close();
At the end of the method should help. Do you see still see leaks after that?
Eyal
> -----Original Message-----
> From: George Aroush [mailto:george@aroush.net]
> Sent: Monday, September 11, 2006 3:50 AM
> To: lucene-net-user@incubator.apache.org;
> lucene-net-dev@incubator.apache.org
> Subject: RE: Lucene.Net Indexing Large Databases
>
> Hi Folks,
>
> Since last weekend, I have been trying to narrow down the problem to
> this memory leak without much of a luck.
>
> Does anyone have a tool (or could recommend one, without costing me
> $$) which hopefully show the source of the leak?
>
> Unlike C++ code, the leak here, obviously, is due to not releasing
> references to temporary or real objects. The trick is finding the
> object.
>
> This leak can be created with this simple code:
>
> public static void Main(System.String[] args)
> {
> IndexWriter diskIndex;
> Directory directory;
> Analyzer analyzer;
> Document doc;
> int count;
> string indexDirectory;
> System.IO.FileInfo fi;
>
> indexDirectory = "C:\\Index.Bad";
>
> fi = new System.IO.FileInfo(indexDirectory);
> directory =
> Lucene.Net.Store.FSDirectory.GetDirectory(fi,
> true);
>
> analyzer = new SimpleAnalyzer();
> diskIndex = new IndexWriter(directory, analyzer, true);
>
> count = 0;
> while (count < 10000)
> {
> doc = new Document();
> diskIndex.AddDocument(doc);
> count++;
> }
>
> diskIndex.Close();
> }
>
> This code will show a leak in 1.9, 1.9.1 and 2.0 but not 1.4.3. I
> also verified and it doesn't leak under the Java version of Lucene
> (2.0 is where I tested.)
>
> Regards,
>
> -- George
>
>
> -----Original Message-----
> From: George Aroush [mailto:george@aroush.net]
> Sent: Friday, September 01, 2006 9:21 PM
> To: lucene-net-user@incubator.apache.org
> Subject: RE: Lucene.Net Indexing Large Databases
>
> Hi Chris,
>
> I am using 1.9.1 in production and I am not having this problem.
> Sorry, I don't have enough cycles to try your code on 1.9.
>
> This problem was reported on 1.4.x and was fixed. I am sure I carried
> it over to 1.9.x and 2.0 -- or maybe this is a new issue. I will
> double check when I get the cycles.
>
> You can get 1.4.3's source code as ZIP from the download site of
> Lucene.Net which is here:
> https://svn.apache.org/repos/asf/incubator/lucene.net/site/dow
nload/ or you can SVN the source code from here:
https://svn.apache.org/repos/asf/incubator/lucene.net/tags/
Regards,
-- George Aroush
-----Original Message-----
From: Chris David [mailto:chris@snapstream.com]
Sent: Friday, September 01, 2006 1:46 PM
To: lucene-net-user@incubator.apache.org
Subject: RE: Lucene.Net Indexing Large Databases
Thanks René, so its not just me with this problem. Now where can I get a
hold of this wonderful 1.4 Build of Lucene. Its not listed directly on
Apache's Lucene.NET Page. I am anxious to see if my code actually does work.
Thanks again for all your help, I really do appreciate it.
Chris
Snapstream Media
-----Original Message-----
From: René de Vries [mailto:rene.de.vries@howardshome.com]
Sent: Friday, September 01, 2006 7:32 AM
To: lucene-net-user@incubator.apache.org
Subject: RE: Lucene.Net Indexing Large Databases
Update: I didn't realize my earlier code example ran against 1.4.
If I run this with 1.9final-005 build, I am experiencing the exact same
problems as Chris mentions. Memory consumtion keeps growing, I had to kill
it at 1.5Gb. Exact same code, but with a 1.4 version of the lucene.net DLL,
and it runs along at 50Mb
René
Memory leak (was: RE: Lucene.Net Indexing Large Databases)
Posted by George Aroush <ge...@aroush.net>.
No, adding directory.Close() will not do the trick. However, I do believe
we are missing a Close() somewhere, most likely to SegmentReader or
TermInfosReader.
All indication are that one or both of those classes is the source of the
leak. Those were my suspects and the MS CLR Profiler is pointing in this
direction.
The Java version lands you into the Finalize of those two classes, but with
the C# version, the Finalize of those two classes is never getting called!
I hope to find out more in the next few days, but if anyone got the cycles
and got a nice tool to help with this defect, please give this a try and let
me know what you find.
Regards,
-- George
-----Original Message-----
From: Eyal Post [mailto:eyalpost@epocalipse.com]
Sent: Monday, September 11, 2006 6:50 PM
To: lucene-net-dev@incubator.apache.org
Subject: RE: Lucene.Net Indexing Large Databases
Adding:
directory.Close();
At the end of the method should help. Do you see still see leaks after that?
Eyal
> -----Original Message-----
> From: George Aroush [mailto:george@aroush.net]
> Sent: Monday, September 11, 2006 3:50 AM
> To: lucene-net-user@incubator.apache.org;
> lucene-net-dev@incubator.apache.org
> Subject: RE: Lucene.Net Indexing Large Databases
>
> Hi Folks,
>
> Since last weekend, I have been trying to narrow down the problem to
> this memory leak without much of a luck.
>
> Does anyone have a tool (or could recommend one, without costing me
> $$) which hopefully show the source of the leak?
>
> Unlike C++ code, the leak here, obviously, is due to not releasing
> references to temporary or real objects. The trick is finding the
> object.
>
> This leak can be created with this simple code:
>
> public static void Main(System.String[] args)
> {
> IndexWriter diskIndex;
> Directory directory;
> Analyzer analyzer;
> Document doc;
> int count;
> string indexDirectory;
> System.IO.FileInfo fi;
>
> indexDirectory = "C:\\Index.Bad";
>
> fi = new System.IO.FileInfo(indexDirectory);
> directory =
> Lucene.Net.Store.FSDirectory.GetDirectory(fi,
> true);
>
> analyzer = new SimpleAnalyzer();
> diskIndex = new IndexWriter(directory, analyzer, true);
>
> count = 0;
> while (count < 10000)
> {
> doc = new Document();
> diskIndex.AddDocument(doc);
> count++;
> }
>
> diskIndex.Close();
> }
>
> This code will show a leak in 1.9, 1.9.1 and 2.0 but not 1.4.3. I
> also verified and it doesn't leak under the Java version of Lucene
> (2.0 is where I tested.)
>
> Regards,
>
> -- George
>
>
> -----Original Message-----
> From: George Aroush [mailto:george@aroush.net]
> Sent: Friday, September 01, 2006 9:21 PM
> To: lucene-net-user@incubator.apache.org
> Subject: RE: Lucene.Net Indexing Large Databases
>
> Hi Chris,
>
> I am using 1.9.1 in production and I am not having this problem.
> Sorry, I don't have enough cycles to try your code on 1.9.
>
> This problem was reported on 1.4.x and was fixed. I am sure I carried
> it over to 1.9.x and 2.0 -- or maybe this is a new issue. I will
> double check when I get the cycles.
>
> You can get 1.4.3's source code as ZIP from the download site of
> Lucene.Net which is here:
> https://svn.apache.org/repos/asf/incubator/lucene.net/site/dow
nload/ or you can SVN the source code from here:
https://svn.apache.org/repos/asf/incubator/lucene.net/tags/
Regards,
-- George Aroush
-----Original Message-----
From: Chris David [mailto:chris@snapstream.com]
Sent: Friday, September 01, 2006 1:46 PM
To: lucene-net-user@incubator.apache.org
Subject: RE: Lucene.Net Indexing Large Databases
Thanks René, so its not just me with this problem. Now where can I get a
hold of this wonderful 1.4 Build of Lucene. Its not listed directly on
Apache's Lucene.NET Page. I am anxious to see if my code actually does work.
Thanks again for all your help, I really do appreciate it.
Chris
Snapstream Media
-----Original Message-----
From: René de Vries [mailto:rene.de.vries@howardshome.com]
Sent: Friday, September 01, 2006 7:32 AM
To: lucene-net-user@incubator.apache.org
Subject: RE: Lucene.Net Indexing Large Databases
Update: I didn't realize my earlier code example ran against 1.4.
If I run this with 1.9final-005 build, I am experiencing the exact same
problems as Chris mentions. Memory consumtion keeps growing, I had to kill
it at 1.5Gb. Exact same code, but with a 1.4 version of the lucene.net DLL,
and it runs along at 50Mb
René
RE: Lucene.Net Indexing Large Databases
Posted by Eyal Post <ey...@epocalipse.com>.
Adding:
directory.Close();
At the end of the method should help. Do you see still see leaks after that?
Eyal
> -----Original Message-----
> From: George Aroush [mailto:george@aroush.net]
> Sent: Monday, September 11, 2006 3:50 AM
> To: lucene-net-user@incubator.apache.org;
> lucene-net-dev@incubator.apache.org
> Subject: RE: Lucene.Net Indexing Large Databases
>
> Hi Folks,
>
> Since last weekend, I have been trying to narrow down the
> problem to this memory leak without much of a luck.
>
> Does anyone have a tool (or could recommend one, without
> costing me $$) which hopefully show the source of the leak?
>
> Unlike C++ code, the leak here, obviously, is due to not
> releasing references to temporary or real objects. The trick
> is finding the object.
>
> This leak can be created with this simple code:
>
> public static void Main(System.String[] args)
> {
> IndexWriter diskIndex;
> Directory directory;
> Analyzer analyzer;
> Document doc;
> int count;
> string indexDirectory;
> System.IO.FileInfo fi;
>
> indexDirectory = "C:\\Index.Bad";
>
> fi = new System.IO.FileInfo(indexDirectory);
> directory =
> Lucene.Net.Store.FSDirectory.GetDirectory(fi,
> true);
>
> analyzer = new SimpleAnalyzer();
> diskIndex = new IndexWriter(directory, analyzer, true);
>
> count = 0;
> while (count < 10000)
> {
> doc = new Document();
> diskIndex.AddDocument(doc);
> count++;
> }
>
> diskIndex.Close();
> }
>
> This code will show a leak in 1.9, 1.9.1 and 2.0 but not
> 1.4.3. I also verified and it doesn't leak under the Java
> version of Lucene (2.0 is where I tested.)
>
> Regards,
>
> -- George
>
>
> -----Original Message-----
> From: George Aroush [mailto:george@aroush.net]
> Sent: Friday, September 01, 2006 9:21 PM
> To: lucene-net-user@incubator.apache.org
> Subject: RE: Lucene.Net Indexing Large Databases
>
> Hi Chris,
>
> I am using 1.9.1 in production and I am not having this
> problem. Sorry, I don't have enough cycles to try your code on 1.9.
>
> This problem was reported on 1.4.x and was fixed. I am sure
> I carried it over to 1.9.x and 2.0 -- or maybe this is a new
> issue. I will double check when I get the cycles.
>
> You can get 1.4.3's source code as ZIP from the download site
> of Lucene.Net which is here:
> https://svn.apache.org/repos/asf/incubator/lucene.net/site/dow
nload/ or you can SVN the source code from here:
https://svn.apache.org/repos/asf/incubator/lucene.net/tags/
Regards,
-- George Aroush
-----Original Message-----
From: Chris David [mailto:chris@snapstream.com]
Sent: Friday, September 01, 2006 1:46 PM
To: lucene-net-user@incubator.apache.org
Subject: RE: Lucene.Net Indexing Large Databases
Thanks René, so its not just me with this problem. Now where can I get a
hold of this wonderful 1.4 Build of Lucene. Its not listed directly on
Apache's Lucene.NET Page. I am anxious to see if my code actually does work.
Thanks again for all your help, I really do appreciate it.
Chris
Snapstream Media
-----Original Message-----
From: René de Vries [mailto:rene.de.vries@howardshome.com]
Sent: Friday, September 01, 2006 7:32 AM
To: lucene-net-user@incubator.apache.org
Subject: RE: Lucene.Net Indexing Large Databases
Update: I didn't realize my earlier code example ran against 1.4.
If I run this with 1.9final-005 build, I am experiencing the exact same
problems as Chris mentions. Memory consumtion keeps growing, I had to kill
it at 1.5Gb. Exact same code, but with a 1.4 version of the lucene.net DLL,
and it runs along at 50Mb
René