You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucenenet.apache.org by "Anton K. (JIRA)" <ji...@apache.org> on 2007/12/28 18:22:43 UTC
[jira] Created: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Lucene.NET (Revision: 603121) is leaking memory
-----------------------------------------------
Key: LUCENENET-106
URL: https://issues.apache.org/jira/browse/LUCENENET-106
Project: Lucene.Net
Issue Type: Bug
Environment: .NET 2.0
Reporter: Anton K.
Priority: Critical
readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
This bug can be reproduced only when Sort functionality used during search.
See following link for additional information.
http://www.gossamer-threads.com/lists/lucene/java-user/55681
Steps to reproduce:
1)Create index
2) Modify index by IndexWiter; Close IndexWriter
3) Use IndexSearcher for searching with Sort; Close InexSearcher
4) Go to step 2
You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is leaking memory
Posted by TJ Kolev <tj...@gmail.com>.
Hi Digy,
Ha, ha... I totally forgot about a Remove. I agree, that's pushing it.
:) I'll add one.
A fully implemented IDictionary contract would be really nice, and
I'll support such effort. But at this point, I really, really want to
get an official, final 2.3.2 tag and use it. A full implementation
will require additional testing, review, etc - i.e. time. So, I'll
suggest we enter a Jira enhancement "Complete WeakHashTable
implementation" for 2.4.0. and have the nunit tests to go with it.
tjk :)
On Thu, Jan 15, 2009 at 3:08 PM, Digy (JIRA) <ji...@apache.org> wrote:
>
> [ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12664264#action_12664264 ]
>
> Digy commented on LUCENENET-106:
> --------------------------------
>
>>>terating and counting elements in such storage brings additional problems - like handling cases when the count does not match the number of iterated elements, objects not being there when at some point they were, etc
>
> That problems can be overcome with storing elements in a temp-table(making a strong reference to them) and returning it, as I tried in V2. But I see the discussion of having "full fledged" or minimal WeakHashTable as a endless story. Therefore I am OK with the TJ's solution. I tested it and everything seems to be alright.
>
> PS: Not having a "Remove" method is a very minimalist approach.
>
>
> DIGY
>
>> Lucene.NET (Revision: 603121) is leaking memory
>> -----------------------------------------------
>>
>> Key: LUCENENET-106
>> URL: https://issues.apache.org/jira/browse/LUCENENET-106
>> Project: Lucene.Net
>> Issue Type: Bug
>> Environment: .NET 2.0
>> Reporter: Anton K.
>> Assignee: Digy
>> Priority: Critical
>> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable v2.patch, WeakHashTable v2.patch, WeakHashTable+FieldCacheImpl.rar, WeakHashTable_tj.zip, WeakReferences.rar
>>
>>
>> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
>> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
>> This bug can be reproduced only when Sort functionality used during search.
>> See following link for additional information.
>> http://www.gossamer-threads.com/lists/lucene/java-user/55681
>> Steps to reproduce:
>> 1)Create index
>> 2) Modify index by IndexWiter; Close IndexWriter
>> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
>> 4) Go to step 2
>> You'll get OutOfMemoryException after some time of running this algorithm.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
RE: [jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is leaking memory
Posted by Digy <di...@gmail.com>.
It does, in fact, use a HashTable table internally. ArrayLists are used to
store WeakEntries whose keys' HashCodes are the same. Since the probability
of a collision is very low, ArrayLists contain most of the time only one
WeakEntry.
* HashTable: +------> WeakEntry
* +----------|--------------------------+ |
* | HashCode | WeakEntries (ArrayList) |----+------> WeakEntry
* +----------+--------------------------+ |
* | HashCode | WeakEntries (ArrayList) | +------> WeakEntry
* +----------+--------------------------+
DIGY
-----Original Message-----
From: Eyal Post [mailto:eyalpost@epocalipse.com]
Sent: Sunday, January 11, 2009 1:08 PM
To: lucene-net-dev@incubator.apache.org
Subject: RE: [jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121)
is leaking memory
As Luc noted in the first comment to this patch, I think that if this class
is to be called WeakHashTable (and therefor imply it is a general purpose
weak hash table) then it should be implemented as a real hash table.
Therefor, you should not use an array list and make linear comparisons.
Instead WeakHashTable should internally use a hashtable. The first
implementation was in the right direction, the only fix should have been to
use WeakEntry as a key to the table and not the hash code.
Eyal
> -----Original Message-----
> From: Digy [mailto:digydigy@gmail.com]
> Sent: Friday, January 09, 2009 20:03 PM
> To: lucene-net-dev@incubator.apache.org
> Subject: RE: [jira] Commented: (LUCENENET-106) Lucene.NET
> (Revision: 603121) is leaking memory
>
> Hi Tim,
>
>
>
> Thanks for the comments. I will correct the 3,4,5.
>
>
>
> 1) Since ArrayList contains WeakEntries & sync.ed for
> multiple threads, they can not be null and can not be any
> other type. I can not find a bug here. Maybe you can send a
> more detailed description.
>
> 2), WeakHashTable is compatible with "HashTable" in .Net
> world. "Ht.Add(key,value)" throws an exception for duplicate keys but
>
> "Ht[key]=value" just adds or replaces.
>
>
>
>
>
> DIGY
>
>
>
>
>
>
>
> -----Original Message-----
> From: Tim Januario (JIRA) [mailto:jira@apache.org]
> Sent: Friday, January 09, 2009 7:16 PM
> To: lucene-net-dev@incubator.apache.org
> Subject: [jira] Commented: (LUCENENET-106) Lucene.NET
> (Revision: 603121) is leaking memory
>
>
>
>
>
> [
> https://issues.apache.org/jira/browse/LUCENENET-106?page=com.a
tlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId
=12662433#action_12662433 ]
Tim Januario commented on LUCENENET-106:
----------------------------------------
Digy,
1) The WeakHashTable.Exists method creates a possible NullReferenceException
because you used WeakEntries[i] As WeakEntry instead of
(WeakEntry)WeakEntries[i]. This is only a warning but...
2) The Java WeakHashMap does not throw an execption when put() is called
with a key that already exists in the table. It simply replaces the old
value with the new value. .NET Hashtable does the same. The Add method of
WeakHashTable should remove the WeakEntry if the key is found and replace it
with the new value in order to stay in line instead of throwing the
Exception.
3) WeakHashTableEnumarator should be WeakHashTableEnumerator
4) WeakEntry.HashCode is not used and should be removed.
5) WeakHashTable.InitialMem is not used and should be removed.
Other than that, I think this looks good.
-tim
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL:
> https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Assignee: Digy
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar,
> luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable
> v2.patch, WeakHashTable+FieldCacheImpl.rar, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some
hash items that have closed IndexReader object as a key. So a lot of Term
instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this
problem.
> This bug can be reproduced only when Sort functionality used during
search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is leaking memory
Posted by TJ Kolev <tj...@gmail.com>.
Hi guys,
I just came across this thread looking for something else. The version
I added in the Jira today works on exactly the idea being discussed
here.
tjk :)
On Mon, Jan 12, 2009 at 7:21 AM, digy digy <di...@gmail.com> wrote:
> OK. I see. It can be used in WeakHashTable v3.
>
> DIGY
>
>
> On Mon, Jan 12, 2009 at 11:28 AM, Eyal Post <ey...@epocalipse.com> wrote:
>
>> void Add(object keyX,object valueX)
>> {
>> InternalTable.Add( new WeakEntry(keyX), value);
>> }
>>
>> Object Get(object keyX)
>> {
>> return InternalTable[new WeakEntry(keyX)];
>> }
>>
>>
>> WeakEntry.GetHashCode = key.getHashCode;
>> WeakEntry.Equals = keyA == keyB
>>
>> Not sure this is clear enough - I can post full implementation if you'd
>> like
>> but that'll take me some more time.
>>
>> Eyal
>>
>>
>> > -----Original Message-----
>> > From: Digy [mailto:digydigy@gmail.com]
>> > Sent: Sunday, January 11, 2009 23:16 PM
>> > To: lucene-net-dev@incubator.apache.org
>> > Subject: RE: [jira] Commented: (LUCENENET-106) Lucene.NET
>> > (Revision: 603121) is leaking memory
>> > Importance: High
>> >
>> > Hi Eyal,
>> >
>> > I couldn't find a simple way to use the WeakEntry as a key
>> > and then access its value without enumerating all the keys in
>> > InternalTable.
>> > I think you suggest
>> >
>> > void Add(object keyX,object valueX)
>> > {
>> > InternalTable.Add( new WeakEntry(keyX,valueX), value);
>> > } // or something like that.
>> >
>> > Then how do you think that we get the value of "Get(keyX)" ?
>> > Enumerating all the keys of InternalTable and comparing the
>> > "weakEntry.Key" with "keyX"?
>> > When thinking of successive "Add"s, this has the complexity of O(n*n).
>> >
>> > Can you share your solution?
>> >
>> > DIGY
>> >
>> >
>> >
>> > -----Original Message-----
>> > From: Eyal Post [mailto:eyalpost@epocalipse.com]
>> > Sent: Sunday, January 11, 2009 1:08 PM
>> > To: lucene-net-dev@incubator.apache.org
>> > Subject: RE: [jira] Commented: (LUCENENET-106) Lucene.NET
>> > (Revision: 603121) is leaking memory
>> >
>> > As Luc noted in the first comment to this patch, I think that
>> > if this class is to be called WeakHashTable (and therefor
>> > imply it is a general purpose weak hash table) then it should
>> > be implemented as a real hash table.
>> > Therefor, you should not use an array list and make linear
>> > comparisons.
>> > Instead WeakHashTable should internally use a hashtable. The
>> > first implementation was in the right direction, the only fix
>> > should have been to use WeakEntry as a key to the table and
>> > not the hash code.
>> >
>> > Eyal
>> >
>> >
>> > > -----Original Message-----
>> > > From: Digy [mailto:digydigy@gmail.com]
>> > > Sent: Friday, January 09, 2009 20:03 PM
>> > > To: lucene-net-dev@incubator.apache.org
>> > > Subject: RE: [jira] Commented: (LUCENENET-106) Lucene.NET
>> > > (Revision: 603121) is leaking memory
>> > >
>> > > Hi Tim,
>> > >
>> > >
>> > >
>> > > Thanks for the comments. I will correct the 3,4,5.
>> > >
>> > >
>> > >
>> > > 1) Since ArrayList contains WeakEntries & sync.ed for multiple
>> > > threads, they can not be null and can not be any other
>> > type. I can not
>> > > find a bug here. Maybe you can send a more detailed description.
>> > >
>> > > 2), WeakHashTable is compatible with "HashTable" in .Net world.
>> > > "Ht.Add(key,value)" throws an exception for duplicate keys but
>> > >
>> > > "Ht[key]=value" just adds or replaces.
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > DIGY
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > -----Original Message-----
>> > > From: Tim Januario (JIRA) [mailto:jira@apache.org]
>> > > Sent: Friday, January 09, 2009 7:16 PM
>> > > To: lucene-net-dev@incubator.apache.org
>> > > Subject: [jira] Commented: (LUCENENET-106) Lucene.NET
>> > > (Revision: 603121) is leaking memory
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > [
>> > > https://issues.apache.org/jira/browse/LUCENENET-106?page=com.a
>> > tlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&fo
>> > cusedCommentId
>> > =12662433#action_12662433 ]
>> >
>> >
>> >
>> > Tim Januario commented on LUCENENET-106:
>> >
>> > ----------------------------------------
>> >
>> >
>> >
>> > Digy,
>> >
>> > 1) The WeakHashTable.Exists method creates a possible
>> > NullReferenceException because you used WeakEntries[i] As
>> > WeakEntry instead of (WeakEntry)WeakEntries[i]. This is only
>> > a warning but...
>> >
>> > 2) The Java WeakHashMap does not throw an execption when
>> > put() is called with a key that already exists in the table.
>> > It simply replaces the old value with the new value. .NET
>> > Hashtable does the same. The Add method of WeakHashTable
>> > should remove the WeakEntry if the key is found and replace
>> > it with the new value in order to stay in line instead of
>> > throwing the Exception.
>> >
>> > 3) WeakHashTableEnumarator should be WeakHashTableEnumerator
>> >
>> > 4) WeakEntry.HashCode is not used and should be removed.
>> >
>> > 5) WeakHashTable.InitialMem is not used and should be removed.
>> >
>> >
>> >
>> > Other than that, I think this looks good.
>> >
>> >
>> >
>> > -tim
>> >
>> >
>> >
>> > > Lucene.NET (Revision: 603121) is leaking memory
>> >
>> > > -----------------------------------------------
>> >
>> > >
>> >
>> > > Key: LUCENENET-106
>> >
>> > > URL:
>> > > https://issues.apache.org/jira/browse/LUCENENET-106
>> >
>> > > Project: Lucene.Net
>> >
>> > > Issue Type: Bug
>> >
>> > > Environment: .NET 2.0
>> >
>> > > Reporter: Anton K.
>> >
>> > > Assignee: Digy
>> >
>> > > Priority: Critical
>> >
>> > > Attachments: DIGY-FieldCacheImpl.patch, Digy.rar,
>> > > luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable
>> > > v2.patch, WeakHashTable+FieldCacheImpl.rar, WeakReferences.rar
>> >
>> > >
>> >
>> > >
>> >
>> > > readerCache Hashtable field (see FieldCacheImpl.cs) never releases
>> > > some
>> > hash items that have closed IndexReader object as a key. So a
>> > lot of Term instances are never released.
>> >
>> > > Java version of Lucene uses WeakHashMap and therefore doesn't have
>> > > this
>> > problem.
>> >
>> > > This bug can be reproduced only when Sort functionality used during
>> > search.
>> >
>> > > See following link for additional information.
>> >
>> > > http://www.gossamer-threads.com/lists/lucene/java-user/55681
>> >
>> > > Steps to reproduce:
>> >
>> > > 1)Create index
>> >
>> > > 2) Modify index by IndexWiter; Close IndexWriter
>> >
>> > > 3) Use IndexSearcher for searching with Sort; Close InexSearcher
>> >
>> > > 4) Go to step 2
>> >
>> > > You'll get OutOfMemoryException after some time of running
>> > this algorithm.
>> >
>> >
>> >
>> > --
>> >
>> > This message is automatically generated by JIRA.
>> >
>> > -
>> >
>> > You can reply to this email to add a comment to the issue online.
>> >
>> >
>>
>>
>
Re: [jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is leaking memory
Posted by digy digy <di...@gmail.com>.
OK. I see. It can be used in WeakHashTable v3.
DIGY
On Mon, Jan 12, 2009 at 11:28 AM, Eyal Post <ey...@epocalipse.com> wrote:
> void Add(object keyX,object valueX)
> {
> InternalTable.Add( new WeakEntry(keyX), value);
> }
>
> Object Get(object keyX)
> {
> return InternalTable[new WeakEntry(keyX)];
> }
>
>
> WeakEntry.GetHashCode = key.getHashCode;
> WeakEntry.Equals = keyA == keyB
>
> Not sure this is clear enough - I can post full implementation if you'd
> like
> but that'll take me some more time.
>
> Eyal
>
>
> > -----Original Message-----
> > From: Digy [mailto:digydigy@gmail.com]
> > Sent: Sunday, January 11, 2009 23:16 PM
> > To: lucene-net-dev@incubator.apache.org
> > Subject: RE: [jira] Commented: (LUCENENET-106) Lucene.NET
> > (Revision: 603121) is leaking memory
> > Importance: High
> >
> > Hi Eyal,
> >
> > I couldn't find a simple way to use the WeakEntry as a key
> > and then access its value without enumerating all the keys in
> > InternalTable.
> > I think you suggest
> >
> > void Add(object keyX,object valueX)
> > {
> > InternalTable.Add( new WeakEntry(keyX,valueX), value);
> > } // or something like that.
> >
> > Then how do you think that we get the value of "Get(keyX)" ?
> > Enumerating all the keys of InternalTable and comparing the
> > "weakEntry.Key" with "keyX"?
> > When thinking of successive "Add"s, this has the complexity of O(n*n).
> >
> > Can you share your solution?
> >
> > DIGY
> >
> >
> >
> > -----Original Message-----
> > From: Eyal Post [mailto:eyalpost@epocalipse.com]
> > Sent: Sunday, January 11, 2009 1:08 PM
> > To: lucene-net-dev@incubator.apache.org
> > Subject: RE: [jira] Commented: (LUCENENET-106) Lucene.NET
> > (Revision: 603121) is leaking memory
> >
> > As Luc noted in the first comment to this patch, I think that
> > if this class is to be called WeakHashTable (and therefor
> > imply it is a general purpose weak hash table) then it should
> > be implemented as a real hash table.
> > Therefor, you should not use an array list and make linear
> > comparisons.
> > Instead WeakHashTable should internally use a hashtable. The
> > first implementation was in the right direction, the only fix
> > should have been to use WeakEntry as a key to the table and
> > not the hash code.
> >
> > Eyal
> >
> >
> > > -----Original Message-----
> > > From: Digy [mailto:digydigy@gmail.com]
> > > Sent: Friday, January 09, 2009 20:03 PM
> > > To: lucene-net-dev@incubator.apache.org
> > > Subject: RE: [jira] Commented: (LUCENENET-106) Lucene.NET
> > > (Revision: 603121) is leaking memory
> > >
> > > Hi Tim,
> > >
> > >
> > >
> > > Thanks for the comments. I will correct the 3,4,5.
> > >
> > >
> > >
> > > 1) Since ArrayList contains WeakEntries & sync.ed for multiple
> > > threads, they can not be null and can not be any other
> > type. I can not
> > > find a bug here. Maybe you can send a more detailed description.
> > >
> > > 2), WeakHashTable is compatible with "HashTable" in .Net world.
> > > "Ht.Add(key,value)" throws an exception for duplicate keys but
> > >
> > > "Ht[key]=value" just adds or replaces.
> > >
> > >
> > >
> > >
> > >
> > > DIGY
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Tim Januario (JIRA) [mailto:jira@apache.org]
> > > Sent: Friday, January 09, 2009 7:16 PM
> > > To: lucene-net-dev@incubator.apache.org
> > > Subject: [jira] Commented: (LUCENENET-106) Lucene.NET
> > > (Revision: 603121) is leaking memory
> > >
> > >
> > >
> > >
> > >
> > > [
> > > https://issues.apache.org/jira/browse/LUCENENET-106?page=com.a
> > tlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&fo
> > cusedCommentId
> > =12662433#action_12662433 ]
> >
> >
> >
> > Tim Januario commented on LUCENENET-106:
> >
> > ----------------------------------------
> >
> >
> >
> > Digy,
> >
> > 1) The WeakHashTable.Exists method creates a possible
> > NullReferenceException because you used WeakEntries[i] As
> > WeakEntry instead of (WeakEntry)WeakEntries[i]. This is only
> > a warning but...
> >
> > 2) The Java WeakHashMap does not throw an execption when
> > put() is called with a key that already exists in the table.
> > It simply replaces the old value with the new value. .NET
> > Hashtable does the same. The Add method of WeakHashTable
> > should remove the WeakEntry if the key is found and replace
> > it with the new value in order to stay in line instead of
> > throwing the Exception.
> >
> > 3) WeakHashTableEnumarator should be WeakHashTableEnumerator
> >
> > 4) WeakEntry.HashCode is not used and should be removed.
> >
> > 5) WeakHashTable.InitialMem is not used and should be removed.
> >
> >
> >
> > Other than that, I think this looks good.
> >
> >
> >
> > -tim
> >
> >
> >
> > > Lucene.NET (Revision: 603121) is leaking memory
> >
> > > -----------------------------------------------
> >
> > >
> >
> > > Key: LUCENENET-106
> >
> > > URL:
> > > https://issues.apache.org/jira/browse/LUCENENET-106
> >
> > > Project: Lucene.Net
> >
> > > Issue Type: Bug
> >
> > > Environment: .NET 2.0
> >
> > > Reporter: Anton K.
> >
> > > Assignee: Digy
> >
> > > Priority: Critical
> >
> > > Attachments: DIGY-FieldCacheImpl.patch, Digy.rar,
> > > luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable
> > > v2.patch, WeakHashTable+FieldCacheImpl.rar, WeakReferences.rar
> >
> > >
> >
> > >
> >
> > > readerCache Hashtable field (see FieldCacheImpl.cs) never releases
> > > some
> > hash items that have closed IndexReader object as a key. So a
> > lot of Term instances are never released.
> >
> > > Java version of Lucene uses WeakHashMap and therefore doesn't have
> > > this
> > problem.
> >
> > > This bug can be reproduced only when Sort functionality used during
> > search.
> >
> > > See following link for additional information.
> >
> > > http://www.gossamer-threads.com/lists/lucene/java-user/55681
> >
> > > Steps to reproduce:
> >
> > > 1)Create index
> >
> > > 2) Modify index by IndexWiter; Close IndexWriter
> >
> > > 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> >
> > > 4) Go to step 2
> >
> > > You'll get OutOfMemoryException after some time of running
> > this algorithm.
> >
> >
> >
> > --
> >
> > This message is automatically generated by JIRA.
> >
> > -
> >
> > You can reply to this email to add a comment to the issue online.
> >
> >
>
>
RE: [jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is leaking memory
Posted by Eyal Post <ey...@epocalipse.com>.
void Add(object keyX,object valueX)
{
InternalTable.Add( new WeakEntry(keyX), value);
}
Object Get(object keyX)
{
return InternalTable[new WeakEntry(keyX)];
}
WeakEntry.GetHashCode = key.getHashCode;
WeakEntry.Equals = keyA == keyB
Not sure this is clear enough - I can post full implementation if you'd like
but that'll take me some more time.
Eyal
> -----Original Message-----
> From: Digy [mailto:digydigy@gmail.com]
> Sent: Sunday, January 11, 2009 23:16 PM
> To: lucene-net-dev@incubator.apache.org
> Subject: RE: [jira] Commented: (LUCENENET-106) Lucene.NET
> (Revision: 603121) is leaking memory
> Importance: High
>
> Hi Eyal,
>
> I couldn't find a simple way to use the WeakEntry as a key
> and then access its value without enumerating all the keys in
> InternalTable.
> I think you suggest
>
> void Add(object keyX,object valueX)
> {
> InternalTable.Add( new WeakEntry(keyX,valueX), value);
> } // or something like that.
>
> Then how do you think that we get the value of "Get(keyX)" ?
> Enumerating all the keys of InternalTable and comparing the
> "weakEntry.Key" with "keyX"?
> When thinking of successive "Add"s, this has the complexity of O(n*n).
>
> Can you share your solution?
>
> DIGY
>
>
>
> -----Original Message-----
> From: Eyal Post [mailto:eyalpost@epocalipse.com]
> Sent: Sunday, January 11, 2009 1:08 PM
> To: lucene-net-dev@incubator.apache.org
> Subject: RE: [jira] Commented: (LUCENENET-106) Lucene.NET
> (Revision: 603121) is leaking memory
>
> As Luc noted in the first comment to this patch, I think that
> if this class is to be called WeakHashTable (and therefor
> imply it is a general purpose weak hash table) then it should
> be implemented as a real hash table.
> Therefor, you should not use an array list and make linear
> comparisons.
> Instead WeakHashTable should internally use a hashtable. The
> first implementation was in the right direction, the only fix
> should have been to use WeakEntry as a key to the table and
> not the hash code.
>
> Eyal
>
>
> > -----Original Message-----
> > From: Digy [mailto:digydigy@gmail.com]
> > Sent: Friday, January 09, 2009 20:03 PM
> > To: lucene-net-dev@incubator.apache.org
> > Subject: RE: [jira] Commented: (LUCENENET-106) Lucene.NET
> > (Revision: 603121) is leaking memory
> >
> > Hi Tim,
> >
> >
> >
> > Thanks for the comments. I will correct the 3,4,5.
> >
> >
> >
> > 1) Since ArrayList contains WeakEntries & sync.ed for multiple
> > threads, they can not be null and can not be any other
> type. I can not
> > find a bug here. Maybe you can send a more detailed description.
> >
> > 2), WeakHashTable is compatible with "HashTable" in .Net world.
> > "Ht.Add(key,value)" throws an exception for duplicate keys but
> >
> > "Ht[key]=value" just adds or replaces.
> >
> >
> >
> >
> >
> > DIGY
> >
> >
> >
> >
> >
> >
> >
> > -----Original Message-----
> > From: Tim Januario (JIRA) [mailto:jira@apache.org]
> > Sent: Friday, January 09, 2009 7:16 PM
> > To: lucene-net-dev@incubator.apache.org
> > Subject: [jira] Commented: (LUCENENET-106) Lucene.NET
> > (Revision: 603121) is leaking memory
> >
> >
> >
> >
> >
> > [
> > https://issues.apache.org/jira/browse/LUCENENET-106?page=com.a
> tlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&fo
> cusedCommentId
> =12662433#action_12662433 ]
>
>
>
> Tim Januario commented on LUCENENET-106:
>
> ----------------------------------------
>
>
>
> Digy,
>
> 1) The WeakHashTable.Exists method creates a possible
> NullReferenceException because you used WeakEntries[i] As
> WeakEntry instead of (WeakEntry)WeakEntries[i]. This is only
> a warning but...
>
> 2) The Java WeakHashMap does not throw an execption when
> put() is called with a key that already exists in the table.
> It simply replaces the old value with the new value. .NET
> Hashtable does the same. The Add method of WeakHashTable
> should remove the WeakEntry if the key is found and replace
> it with the new value in order to stay in line instead of
> throwing the Exception.
>
> 3) WeakHashTableEnumarator should be WeakHashTableEnumerator
>
> 4) WeakEntry.HashCode is not used and should be removed.
>
> 5) WeakHashTable.InitialMem is not used and should be removed.
>
>
>
> Other than that, I think this looks good.
>
>
>
> -tim
>
>
>
> > Lucene.NET (Revision: 603121) is leaking memory
>
> > -----------------------------------------------
>
> >
>
> > Key: LUCENENET-106
>
> > URL:
> > https://issues.apache.org/jira/browse/LUCENENET-106
>
> > Project: Lucene.Net
>
> > Issue Type: Bug
>
> > Environment: .NET 2.0
>
> > Reporter: Anton K.
>
> > Assignee: Digy
>
> > Priority: Critical
>
> > Attachments: DIGY-FieldCacheImpl.patch, Digy.rar,
> > luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable
> > v2.patch, WeakHashTable+FieldCacheImpl.rar, WeakReferences.rar
>
> >
>
> >
>
> > readerCache Hashtable field (see FieldCacheImpl.cs) never releases
> > some
> hash items that have closed IndexReader object as a key. So a
> lot of Term instances are never released.
>
> > Java version of Lucene uses WeakHashMap and therefore doesn't have
> > this
> problem.
>
> > This bug can be reproduced only when Sort functionality used during
> search.
>
> > See following link for additional information.
>
> > http://www.gossamer-threads.com/lists/lucene/java-user/55681
>
> > Steps to reproduce:
>
> > 1)Create index
>
> > 2) Modify index by IndexWiter; Close IndexWriter
>
> > 3) Use IndexSearcher for searching with Sort; Close InexSearcher
>
> > 4) Go to step 2
>
> > You'll get OutOfMemoryException after some time of running
> this algorithm.
>
>
>
> --
>
> This message is automatically generated by JIRA.
>
> -
>
> You can reply to this email to add a comment to the issue online.
>
>
RE: [jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is leaking memory
Posted by Digy <di...@gmail.com>.
Hi Eyal,
I couldn't find a simple way to use the WeakEntry as a key and then access
its value without enumerating all the keys in InternalTable.
I think you suggest
void Add(object keyX,object valueX)
{
InternalTable.Add( new WeakEntry(keyX,valueX), value);
} // or something like that.
Then how do you think that we get the value of "Get(keyX)" ? Enumerating all
the keys of InternalTable and comparing the "weakEntry.Key" with "keyX"?
When thinking of successive "Add"s, this has the complexity of O(n*n).
Can you share your solution?
DIGY
-----Original Message-----
From: Eyal Post [mailto:eyalpost@epocalipse.com]
Sent: Sunday, January 11, 2009 1:08 PM
To: lucene-net-dev@incubator.apache.org
Subject: RE: [jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121)
is leaking memory
As Luc noted in the first comment to this patch, I think that if this class
is to be called WeakHashTable (and therefor imply it is a general purpose
weak hash table) then it should be implemented as a real hash table.
Therefor, you should not use an array list and make linear comparisons.
Instead WeakHashTable should internally use a hashtable. The first
implementation was in the right direction, the only fix should have been to
use WeakEntry as a key to the table and not the hash code.
Eyal
> -----Original Message-----
> From: Digy [mailto:digydigy@gmail.com]
> Sent: Friday, January 09, 2009 20:03 PM
> To: lucene-net-dev@incubator.apache.org
> Subject: RE: [jira] Commented: (LUCENENET-106) Lucene.NET
> (Revision: 603121) is leaking memory
>
> Hi Tim,
>
>
>
> Thanks for the comments. I will correct the 3,4,5.
>
>
>
> 1) Since ArrayList contains WeakEntries & sync.ed for
> multiple threads, they can not be null and can not be any
> other type. I can not find a bug here. Maybe you can send a
> more detailed description.
>
> 2), WeakHashTable is compatible with "HashTable" in .Net
> world. "Ht.Add(key,value)" throws an exception for duplicate keys but
>
> "Ht[key]=value" just adds or replaces.
>
>
>
>
>
> DIGY
>
>
>
>
>
>
>
> -----Original Message-----
> From: Tim Januario (JIRA) [mailto:jira@apache.org]
> Sent: Friday, January 09, 2009 7:16 PM
> To: lucene-net-dev@incubator.apache.org
> Subject: [jira] Commented: (LUCENENET-106) Lucene.NET
> (Revision: 603121) is leaking memory
>
>
>
>
>
> [
> https://issues.apache.org/jira/browse/LUCENENET-106?page=com.a
tlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId
=12662433#action_12662433 ]
Tim Januario commented on LUCENENET-106:
----------------------------------------
Digy,
1) The WeakHashTable.Exists method creates a possible NullReferenceException
because you used WeakEntries[i] As WeakEntry instead of
(WeakEntry)WeakEntries[i]. This is only a warning but...
2) The Java WeakHashMap does not throw an execption when put() is called
with a key that already exists in the table. It simply replaces the old
value with the new value. .NET Hashtable does the same. The Add method of
WeakHashTable should remove the WeakEntry if the key is found and replace it
with the new value in order to stay in line instead of throwing the
Exception.
3) WeakHashTableEnumarator should be WeakHashTableEnumerator
4) WeakEntry.HashCode is not used and should be removed.
5) WeakHashTable.InitialMem is not used and should be removed.
Other than that, I think this looks good.
-tim
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL:
> https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Assignee: Digy
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar,
> luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable
> v2.patch, WeakHashTable+FieldCacheImpl.rar, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some
hash items that have closed IndexReader object as a key. So a lot of Term
instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this
problem.
> This bug can be reproduced only when Sort functionality used during
search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
RE: [jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is leaking memory
Posted by Eyal Post <ey...@epocalipse.com>.
As Luc noted in the first comment to this patch, I think that if this class
is to be called WeakHashTable (and therefor imply it is a general purpose
weak hash table) then it should be implemented as a real hash table.
Therefor, you should not use an array list and make linear comparisons.
Instead WeakHashTable should internally use a hashtable. The first
implementation was in the right direction, the only fix should have been to
use WeakEntry as a key to the table and not the hash code.
Eyal
> -----Original Message-----
> From: Digy [mailto:digydigy@gmail.com]
> Sent: Friday, January 09, 2009 20:03 PM
> To: lucene-net-dev@incubator.apache.org
> Subject: RE: [jira] Commented: (LUCENENET-106) Lucene.NET
> (Revision: 603121) is leaking memory
>
> Hi Tim,
>
>
>
> Thanks for the comments. I will correct the 3,4,5.
>
>
>
> 1) Since ArrayList contains WeakEntries & sync.ed for
> multiple threads, they can not be null and can not be any
> other type. I can not find a bug here. Maybe you can send a
> more detailed description.
>
> 2), WeakHashTable is compatible with "HashTable" in .Net
> world. "Ht.Add(key,value)" throws an exception for duplicate keys but
>
> "Ht[key]=value" just adds or replaces.
>
>
>
>
>
> DIGY
>
>
>
>
>
>
>
> -----Original Message-----
> From: Tim Januario (JIRA) [mailto:jira@apache.org]
> Sent: Friday, January 09, 2009 7:16 PM
> To: lucene-net-dev@incubator.apache.org
> Subject: [jira] Commented: (LUCENENET-106) Lucene.NET
> (Revision: 603121) is leaking memory
>
>
>
>
>
> [
> https://issues.apache.org/jira/browse/LUCENENET-106?page=com.a
tlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId
=12662433#action_12662433 ]
Tim Januario commented on LUCENENET-106:
----------------------------------------
Digy,
1) The WeakHashTable.Exists method creates a possible NullReferenceException
because you used WeakEntries[i] As WeakEntry instead of
(WeakEntry)WeakEntries[i]. This is only a warning but...
2) The Java WeakHashMap does not throw an execption when put() is called
with a key that already exists in the table. It simply replaces the old
value with the new value. .NET Hashtable does the same. The Add method of
WeakHashTable should remove the WeakEntry if the key is found and replace it
with the new value in order to stay in line instead of throwing the
Exception.
3) WeakHashTableEnumarator should be WeakHashTableEnumerator
4) WeakEntry.HashCode is not used and should be removed.
5) WeakHashTable.InitialMem is not used and should be removed.
Other than that, I think this looks good.
-tim
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL:
> https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Assignee: Digy
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar,
> luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable
> v2.patch, WeakHashTable+FieldCacheImpl.rar, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some
hash items that have closed IndexReader object as a key. So a lot of Term
instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this
problem.
> This bug can be reproduced only when Sort functionality used during
search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
RE: [jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is leaking memory
Posted by Digy <di...@gmail.com>.
Hi Tim,
Thanks for the comments. I will correct the 3,4,5.
1) Since ArrayList contains WeakEntries & sync.ed for multiple threads, they can not be null and can not be any other type. I can not find a bug here. Maybe you can send a more detailed description.
2), WeakHashTable is compatible with "HashTable" in .Net world. "Ht.Add(key,value)" throws an exception for duplicate keys but
"Ht[key]=value" just adds or replaces.
DIGY
-----Original Message-----
From: Tim Januario (JIRA) [mailto:jira@apache.org]
Sent: Friday, January 09, 2009 7:16 PM
To: lucene-net-dev@incubator.apache.org
Subject: [jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is leaking memory
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662433#action_12662433 ]
Tim Januario commented on LUCENENET-106:
----------------------------------------
Digy,
1) The WeakHashTable.Exists method creates a possible NullReferenceException because you used WeakEntries[i] As WeakEntry instead of (WeakEntry)WeakEntries[i]. This is only a warning but...
2) The Java WeakHashMap does not throw an execption when put() is called with a key that already exists in the table. It simply replaces the old value with the new value. .NET Hashtable does the same. The Add method of WeakHashTable should remove the WeakEntry if the key is found and replace it with the new value in order to stay in line instead of throwing the Exception.
3) WeakHashTableEnumarator should be WeakHashTableEnumerator
4) WeakEntry.HashCode is not used and should be removed.
5) WeakHashTable.InitialMem is not used and should be removed.
Other than that, I think this looks good.
-tim
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Assignee: Digy
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable v2.patch, WeakHashTable+FieldCacheImpl.rar, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "TJ Kolev (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
TJ Kolev updated LUCENENET-106:
-------------------------------
Attachment: WeakHashTable_tj.zip
Here's another WeakHashTable implementation for consideration. The whole nunit test solution runs green. The zip also has a console application with the code that Digy proposed here on 26/Jun/08 (with tiny mods), and the nunit test code digy propsed on the dev list (with formatting changes only). The code is a drop in replacement, not a patch (yet).
Now my general comments on the issue. I agree with Eyal Post's comments on the mailing list. Hash codes must be the same for two objects only if Equals() returns true for them. But you can have the same hash code for objects that are !Equals(). The table just won't be as efficient. In any case, has collisions are handled, and that's a core hash table functionality.
In Java's WeakHashMap the keys are the weak references, and this is what I did in my code. The trick that made it work was keeping the hash code of the caller's key object. That way the internal hash table (and its cleanup) is able to work correctly, even when the Target of the weak reference has been GC collected.
I have only provided the minimum functionality needed by Lucene. I don't believe it is reasonable to assume someone will pull out the code, and try to use it as a general purpose weak hash table. I can certainly make the implementation complete. But from Lucene's usage, the weak hash table is just a transient storage for looking up resources, which may or may not be there. If they are still there - it is a nice benefit, if not - the caller creates them again. (Iterating and counting elements in such storage brings additional problems - like handling cases when the count does not match the number of iterated elements, objects not being there when at some point they were, etc.)
Now, I may have certainly missed something, and the tests I ran did not catch it. I would much appreciate any feedback and comments, especially test code to go along with them.
tjk :)
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Assignee: Digy
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable v2.patch, WeakHashTable v2.patch, WeakHashTable+FieldCacheImpl.rar, WeakHashTable_tj.zip, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "Anton K. (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Anton K. updated LUCENENET-106:
-------------------------------
Attachment: luceneSrc_memUsage.patch
the patch
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Priority: Critical
> Attachments: luceneSrc_memUsage.patch
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "TJ Kolev (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12665168#action_12665168 ]
TJ Kolev commented on LUCENENET-106:
------------------------------------
Eyal's implementation looks solid. Tests pass for me. Using a comparer is a good idea - WeakReference objects are finalizable and need more management from the CRL - the fewer created the better.
I'd only suggest adding comments for IEqualityComparer.Equals and GCMonitor. A few months after we are done tuning the code, it will be difficult to recall the logic of "why was this done and why it works?".
tjk :)
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Assignee: Digy
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable v2.patch, WeakHashTable v2.patch, WeakHashTable+FieldCacheImpl.rar, WeakHashTable_ep.zip, WeakHashTable_tj.zip, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "eyal post (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
eyal post updated LUCENENET-106:
--------------------------------
Attachment: WeakHashTable_ep.zip
Here's my version for review. I decided to go for the other extreme - that is implementing a fully working WeakHashTable implementing all IDictionary methods. I also decided to implement it a bit different than Tj's for performance reasons.
All tests are passing with this including Tj's tests and some of my own tests.
How is this implementation different:
1. Fully implements IDictionary according to IDictionary rules (i.e. Add() throws an exception if the same key is passed twice)
2. Internally uses a regular hashtable with a custom comparer. This means that to search for a key there is no need to create a WeakKey instance - instead you can search by using the original key and the comparer does the searching. This increases performance since for regular access you don't need to create a WeakKey object just to compare it to exisiting keys.
3. Cleanup is called only if a GC occoured. This greatly improves performance since there's no need to iterate over the entire table trying to clean entries if no GC occoured.
Some of the tests I included are performance tests. Running them on my machines show the following:
TjWeakHashTable WeakHashTable
Get 2.5 0.7
Add 2.2 1.9
Contains 2.2 0.65
Remove 4.5 2.5
Replace 2.8 2.8
Please review and let me know if you have any comments
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Assignee: Digy
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable v2.patch, WeakHashTable v2.patch, WeakHashTable+FieldCacheImpl.rar, WeakHashTable_ep.zip, WeakHashTable_tj.zip, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "Erich Eichinger (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Erich Eichinger updated LUCENENET-106:
--------------------------------------
Attachment: WeakReferences.rar
It seems this bug has been reintroduced from r1.4.3 - see my mails to the lucene-net list here:
http://mail-archives.apache.org/mod_mbox/incubator-lucene-net-dev/200704.mbox/%3C5819CEE5BB511B4285A1F4BC147C537719AA11@maildog.office.dd.loc%3E
http://markmail.org/message/tilrugo4kux2yysq
http://markmail.org/message/nv4l346tecrkpwgx
http://www.mail-archive.com/lucene-net-user@incubator.apache.org/msg00392.html
I attached an implementation of WeakHashtable that I've done exactly to fix those memory leaks. It is based on NHibernate's WeakHashtable implementation so they might want to get some credits.
hope this helps
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "eyal post (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12670256#action_12670256 ]
eyal post commented on LUCENENET-106:
-------------------------------------
Seems like a compile optimization indeed. The idea is to hold a references to what you don't want collected so I guess the following should suffice:
Test_Weak_2:
Assert.IsNotNull(key2);
Test_Work_ForEach:
Assert.IsNotNull(keys1);
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Assignee: Digy
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable v2.patch, WeakHashTable v2.patch, WeakHashTable+FieldCacheImpl.rar, WeakHashTable_ep.zip, WeakHashTable_ep_v2.zip, WeakHashTable_ep_v3.zip, WeakHashTable_tj.zip, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (LUCENENET-106) Lucene.NET (Revision:
603121) is leaking memory
Posted by "Marco Dissel (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12557244#action_12557244 ]
mdissel edited comment on LUCENENET-106 at 1/9/08 4:31 AM:
----------------------------------------------------------------
I got another serious memory leak. This is the result from DebugDiag tool. The patch in this issue doesn't solve this one.. Anyone solved this already?
(doing a seach with a custom sort)
Is this line the problem in the FieldCacheImpl?
class Cache.Get
//
innerCache = new System.Collections.Hashtable(); // shouldn't this be WeakHashtable()???
//
[DEFAULT] [hasThis] Object Lucene.Net.Search.FieldCacheImpl/AnonymousClassCache3.CreateValue(Class Lucene.Net.Index.IndexReader,Object) 0xC920F4
[DEFAULT] [hasThis] Object Lucene.Net.Search.FieldCacheImpl/Cache.Get(Class Lucene.Net.Index.IndexReader,Object)
[DEFAULT] [hasThis] Class Lucene.Net.Search.StringIndex Lucene.Net.Search.FieldCacheImpl.GetStringIndex(Class Lucene.Net.Index.IndexReader,String)
[DEFAULT] [hasThis] Object Lucene.Net.Search.FieldCacheImpl/AnonymousClassCache4.CreateValue(Class Lucene.Net.Index.IndexReader,Object)
[DEFAULT] [hasThis] Object Lucene.Net.Search.FieldCacheImpl/Cache.Get(Class Lucene.Net.Index.IndexReader,Object)
[DEFAULT] [hasThis] Object Lucene.Net.Search.FieldCacheImpl.GetAuto(Class Lucene.Net.Index.IndexReader,String)
[DEFAULT] Class Lucene.Net.Search.ScoreDocComparator Lucene.Net.Search.FieldSortedHitQueue.ComparatorAuto(Class Lucene.Net.Index.IndexReader,String)
[DEFAULT] [hasThis] Object Lucene.Net.Search.FieldSortedHitQueue/AnonymousClassCache.CreateValue(Class Lucene.Net.Index.IndexReader,Object) [DEFAULT] Class Lucene.Net.Search.ScoreDocComparator Lucene.Net.Search.FieldSortedHitQueue.ComparatorAuto(Class Lucene.Net.Index.IndexReader,String)
[DEFAULT] [hasThis] Object Lucene.Net.Search.FieldCacheImpl/Cache.Get(Class Lucene.Net.Index.IndexReader,Object)
[DEFAULT] Class Lucene.Net.Search.ScoreDocComparator Lucene.Net.Search.FieldSortedHitQueue.GetCachedComparator(Class Lucene.Net.Index.IndexReader,String,I4,Class System.Globalization.CultureInfo,Class Lucene.Net.Search.SortComparatorSource)
[DEFAULT] [hasThis] Void Lucene.Net.Search.FieldSortedHitQueue..ctor(Class Lucene.Net.Index.IndexReader,SZArray Class Lucene.Net.Search.SortField,I4) [DEFAULT] Class Lucene.Net.Search.ScoreDocComparator Lucene.Net.Search.FieldSortedHitQueue.GetCachedComparator(Class Lucene.Net.Index.IndexReader,String,I4,Class System.Globalization.CultureInfo,Class Lucene.Net.Search.SortComparatorSource)
[DEFAULT] [hasThis] Class Lucene.Net.Search.TopFieldDocs Lucene.Net.Search.IndexSearcher.Search(Class Lucene.Net.Search.Weight,Class Lucene.Net.Search.Filter,I4,Class Lucene.Net.Search.Sort) [DEFAULT] [hasThis] Void Lucene.Net.Search.FieldSortedHitQueue..ctor(Class Lucene.Net.Index.IndexReader,SZArray Class Lucene.Net.Search.SortField,I4)
[DEFAULT] [hasThis] Class Lucene.Net.Search.TopFieldDocs Lucene.Net.Search.Searcher.Search(Class Lucene.Net.Search.Query,Class Lucene.Net.Search.Filter,I4,Class Lucene.Net.Search.Sort)
was (Author: mdissel):
I got another serious memory leak. This is the result from DebugDiag tool. The attached patched doesn't solve this one.. Anyone solved this already?
Is this line the problem in the FieldCacheImpl?
class Cache.Get
//
innerCache = new System.Collections.Hashtable(); // shouldn't this be WeakHashtable()???
//
[DEFAULT] [hasThis] Object Lucene.Net.Search.FieldCacheImpl/AnonymousClassCache3.CreateValue(Class Lucene.Net.Index.IndexReader,Object) 0xC920F4
[DEFAULT] [hasThis] Object Lucene.Net.Search.FieldCacheImpl/Cache.Get(Class Lucene.Net.Index.IndexReader,Object)
[DEFAULT] [hasThis] Class Lucene.Net.Search.StringIndex Lucene.Net.Search.FieldCacheImpl.GetStringIndex(Class Lucene.Net.Index.IndexReader,String)
[DEFAULT] [hasThis] Object Lucene.Net.Search.FieldCacheImpl/AnonymousClassCache4.CreateValue(Class Lucene.Net.Index.IndexReader,Object)
[DEFAULT] [hasThis] Object Lucene.Net.Search.FieldCacheImpl/Cache.Get(Class Lucene.Net.Index.IndexReader,Object)
[DEFAULT] [hasThis] Object Lucene.Net.Search.FieldCacheImpl.GetAuto(Class Lucene.Net.Index.IndexReader,String)
[DEFAULT] Class Lucene.Net.Search.ScoreDocComparator Lucene.Net.Search.FieldSortedHitQueue.ComparatorAuto(Class Lucene.Net.Index.IndexReader,String)
[DEFAULT] [hasThis] Object Lucene.Net.Search.FieldSortedHitQueue/AnonymousClassCache.CreateValue(Class Lucene.Net.Index.IndexReader,Object) [DEFAULT] Class Lucene.Net.Search.ScoreDocComparator Lucene.Net.Search.FieldSortedHitQueue.ComparatorAuto(Class Lucene.Net.Index.IndexReader,String)
[DEFAULT] [hasThis] Object Lucene.Net.Search.FieldCacheImpl/Cache.Get(Class Lucene.Net.Index.IndexReader,Object)
[DEFAULT] Class Lucene.Net.Search.ScoreDocComparator Lucene.Net.Search.FieldSortedHitQueue.GetCachedComparator(Class Lucene.Net.Index.IndexReader,String,I4,Class System.Globalization.CultureInfo,Class Lucene.Net.Search.SortComparatorSource)
[DEFAULT] [hasThis] Void Lucene.Net.Search.FieldSortedHitQueue..ctor(Class Lucene.Net.Index.IndexReader,SZArray Class Lucene.Net.Search.SortField,I4) [DEFAULT] Class Lucene.Net.Search.ScoreDocComparator Lucene.Net.Search.FieldSortedHitQueue.GetCachedComparator(Class Lucene.Net.Index.IndexReader,String,I4,Class System.Globalization.CultureInfo,Class Lucene.Net.Search.SortComparatorSource)
[DEFAULT] [hasThis] Class Lucene.Net.Search.TopFieldDocs Lucene.Net.Search.IndexSearcher.Search(Class Lucene.Net.Search.Weight,Class Lucene.Net.Search.Filter,I4,Class Lucene.Net.Search.Sort) [DEFAULT] [hasThis] Void Lucene.Net.Search.FieldSortedHitQueue..ctor(Class Lucene.Net.Index.IndexReader,SZArray Class Lucene.Net.Search.SortField,I4)
[DEFAULT] [hasThis] Class Lucene.Net.Search.TopFieldDocs Lucene.Net.Search.Searcher.Search(Class Lucene.Net.Search.Query,Class Lucene.Net.Search.Filter,I4,Class Lucene.Net.Search.Sort)
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, WeakHashTable+FieldCacheImpl.rar, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
RE: Problems with patch for LUCENENET-106
Posted by George Aroush <ge...@aroush.net>.
The fix should be in the implementation / port of WeakHashMap, and not in
Lucene.Net base code.
I believe Luc is working on providing an alternative patch / fix.
-- George
> -----Original Message-----
> From: Timothy Januario [mailto:Timothy.Januario@BDMetrics.com]
> Sent: Monday, January 05, 2009 10:19 AM
> To: lucene-net-dev@incubator.apache.org
> Subject: RE: Problems with patch for LUCENENET-106
>
> I stand corrected. I do see the possibility of the collision
> as a potentially serious bug. The java version still uses
> the object as the key which means that if the hashcode is
> identical across all instances, it just becomes less
> efficient. In the current .NET implementation, it stays very
> efficient, it would just only ever have a single instance in
> the cache which is a very different and incorrect result.
>
> -----Original Message-----
> From: Eyal Post [mailto:eyalpost@epocalipse.com]
> Sent: Saturday, January 03, 2009 1:56 PM
> To: lucene-net-dev@incubator.apache.org
> Subject: RE: Problems with patch for LUCENENET-106
>
> I tend to agree with Luc.
> Hashtables don't rely on small chance for collisions to
> operate correctly.
> They only rely on it to operate efficiently.
> This means you can store 2 elements with the same hashcode in
> a hashtable and they won't override each other, but the
> hashtable lookups will be less efficient with collisions.
> The problem with code provided in the patch is that it uses
> the hashcode as the key in the hashtable and altough the
> chance of collisions is small once they happen the results
> are undefined.
> I think the best way to solve that is to use WeakEntry as the
> key and override both GetHashCode and Equals. GetHashCode
> should return the HashCode field and Equals should compare the Keys
>
>
> Eyal Post
>
>
> > -----Original Message-----
> > From: George Aroush [mailto:george@aroush.net]
> > Sent: Friday, January 02, 2009 0:52 AM
> > To: lucene-net-dev@incubator.apache.org
> > Subject: RE: Problems with patch for LUCENENET-106
> >
> > Hi Luc,
> >
> > I have to agree with Tim.
> >
> > The .NET's implementation / port of WeakHashMap in Lucene.Net is
> > consistent with Java Lucene. If there is a "very" small
> chance of a
> > hash key collusion, it is also possible in the Java version
> of Lucene.
> > Here is why.
> >
> > In Java, when WeakHashMap.put(Object key, Object value) is called,
> > this function calls key.hashCode() to get a hash value (I
> had a look
> > at Java's source code for WeakHashMap.) The same logic is
> happening in
> > .NET's implementation / port of WeakHashMap. In addition, at least
> > for now, the "key" is always of type IndexReader. Looking in
> > IndexReader code, there is no override of Java::hashCode() /
> > C#::GetHashCode().
> > So, unless if Java's
> > hashCode() guaranties uniqueness (which it doesn't), then the
> > possibility of a hash value collusion also exists in the
> Java version
> > of Lucene. If this is the case, then we have a valid
> defect in Lucene
> > and it should be addressed.
> >
> > Erik, do you agree, can you comment?
> >
> > Regarding #2, & #3, I agree that the .NET implementation of
> > WeakHashMap could have been confined to just get() / put().
> > Since you already have a patch for Lucene.Net 2.1, please
> submit a new
> > JIRA issue and attach your patch to it (when you are
> ready.) I look
> > forward to your patch.
> >
> > Regards, and Happy New Year everyone!
> >
> > -- George
> >
> >
> > > -----Original Message-----
> > > From: Timothy Januario [mailto:Timothy.Januario@BDMetrics.com]
> > > Sent: Wednesday, December 31, 2008 9:44 AM
> > > To: lucene-net-dev@incubator.apache.org
> > > Subject: RE: Problems with patch for LUCENENET-106
> > >
> > > Luc,
> > > I'm not sure that I completely agree with your assessment of the
> > > WeakHashMap.
> > >
> > > It does assume that the hash value is unique which is a
> > correct usage.
> > > It is the responsibility of
> > > object.GetHashCode() to return a unique value. This is a
> > problem in
> > > any Hashtable implementation and although I haven't seen
> > the internal
> > > java code, I assume that it has the same limitation. The
> > object would
> > > be the key only superficially as the hashcode would be used
> > internally
> > > as well (based on the name of the object, WeakHashMap, anyway).
> > > If this is incorrect, I apologize for the noise. I did, however,
> > > notice that GetHashCode() is not overridden in the
> IndexReader and
> > > since the .NET framework does not guarantee a unique value
> > (according
> > > to the documentation for oject.GetHashCode()), this could
> > present the
> > > problem that you have identified. Is there a need to guarantee
> > > uniqueness by overriding the method or does anyone know if
> > the value
> > > is always unique already? When the Cache object was
> > implemented with
> > > Hashtable prior to this patch, the same problem would have been
> > > inherent with that implementation (of course with the
> > additional bonus
> > > of the memory leak ;)) and I don't think I ever saw collisions
> > > although quite honestly I was never really looking for them.
> > >
> > > Also, you said that the call to WeakHashMap.Keys could
> return NULL
> > > values. This may be the reason why the Java implementation
> > supports
> > > null keys. It does seem to be a possibility unless the
> > call to Keys
> > > checks that WeakReference.Key.Target is not null at the time of
> > > iteration (this would eliminate this possibility since we
> would now
> > > have a strong reference to the object ie: object key =
> > > entries[i].Key.Target; if (object != null)
> keys.Add(object);). The
> > > same problem would exist with the Values property.
> > >
> > > Also, optimizing for few IndexReaders in the WeakHashMap is an
> > > assumption that specializes the class for IndexReaders.
> > > There is no guarantee that this will be the only use case in the
> > > future. The Java documentation says that the same initial
> > parameters
> > > (loadfactor and capacity) as HashMap are used which
> > suggests that it
> > > should be left in the .NET implementation with the same
> > parameters as
> > > the default Hashtable which is its underlying container.
> > >
> > > Comments?
> > > -tim
> > >
> > > -----Original Message-----
> > > From: Vanlerberghe, Luc [mailto:Luc.Vanlerberghe@bvdep.com]
> > > Sent: Wednesday, December 31, 2008 4:48 AM
> > > To: lucene-net-dev@incubator.apache.org
> > > Subject: Problems with patch for LUCENENET-106
> > >
> > > Hi,
> > >
> > > I reviewed the fix for LUCENENET-106 and I see a couple of issues:
> > >
> > > 1. The implementation of WeakHashMap in SupportClass.cs has
> > a bug in
> > > that it assumes all keys will have a unique HashCode.
> > > Although the chances of this occurring are *very* small, it
> > cannot be
> > > guaranteed. A collision would mean that the cached value for one
> > > indexReader could be returned as cached value for another
> > indexReader
> > > (using the way this class is used in Lucene as example)
> > >
> > > 2. This class attempts to be a complete port of the
> > > java.util.WeakHashMap, but:
> > > - A correct implementation is difficult to implement, and
> .Net does
> > > not have an equivalent of java.lang.ref.ReferenceQueue to
> > accelerate
> > > the cleanup process.
> > > - Since the behaviour of the WeakHashMap is closely tied to the
> > > behaviour of the garbage collector, it is very difficult to test
> > > properly, if at all.
> > > - Lucene only uses the get(Object key) and put(K key, V
> > value) methods
> > > - In Lucene, the keys are IndexReader instances. In normal
> > use cases
> > > there will be only 1 IndexReader live, and perhaps another one if
> > > 'warming up' is used before switching IndexSearchers
> after an index
> > > update.
> > > - If this class is presented as a complete port of
> > > java.util.WeakHashMap, users might assume this is
> > production quality
> > > code and copy/paste its code in their own projects.
> > > Any further bugs found might harm the credibility of the Lucene
> > > project, even if those sections are never used in Lucene.
> > (e.g.: The
> > > collection returned by Keys might contain null values.
> There's no
> > > guarantee the GC won't intervene between the call to
> > Cleanup and the
> > > calls to keys.Add(w.Key.Target).
> > >
> > > 3. Most support classes that are needed for the conversion
> > of java to
> > > .Net are in the anonymous namespace, but this one is in
> > > Lucene.Net.Util. I would propose to keep the namespaces
> > corresponding
> > > to the original java packages clean and either put all
> > support classes
> > > in the anonymous namespace or in a Lucene.Net specific one
> > > (Lucene.Net.DotNetPort ?) (I see that George moved it already to
> > > SupportClass, thanks George!)
> > >
> > > I have been using the java version of Lucene in a project
> > for a long
> > > time.
> > > For a new project that is currently under development, we will be
> > > using the .Net version.
> > > For now we are using the 2.1 version using a custom patch
> > to avoid the
> > > memory leak problem.
> > >
> > > I'll post an up to date version of this patch as a
> > replacement for the
> > > current implementation of Lucene.Net.Util.WeakHashMap
> sometime next
> > > week...
> > > - It doesn't try to be a complete implementation of
> > > WeakHashMap: only the methods strictly necessary are
> > implemented, all
> > > other methods throw a NotImplementedException. (It should
> > probably be
> > > renamed to SimplifiedWeakHashMap or something)
> > > - It's optimized for the 'low number of keys' case.
> > > - It's simple code, but I want to test it thoroughly first.
> > > The original patch was put directly in the FieldCacheImpl
> > code, I want
> > > to make sure I didn't make some stupid mistake while
> converting it.
> > >
> > > I have an account on Jira, but I don't have rights to reopen the
> > > issue...
> > >
> > > Regards,
> > >
> > > Luc
> > >
> > >
> > > -----Original Message-----
> > > From: Digy (JIRA) [mailto:jira@apache.org]
> > > Sent: zaterdag 27 december 2008 19:47
> > > To: lucene-net-dev@incubator.apache.org
> > > Subject: [jira] Closed: () Lucene.NET (Revision: 603121)
> is leaking
> > > memory
> > >
> > >
> > > [
> > > https://issues.apache.org/jira/browse/LUCENENET-106?page=com.a
> > tlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
> > >
> > > Digy closed LUCENENET-106.
> > > --------------------------
> > >
> > > Resolution: Fixed
> > > Assignee: Digy
> > >
> > > Patches are applied.
> > >
> > > DIGY.
> > >
> > > > Lucene.NET (Revision: 603121) is leaking memory
> > > > -----------------------------------------------
> > > >
> > > > Key: LUCENENET-106
> > > > URL:
> > > https://issues.apache.org/jira/browse/LUCENENET-106
> > > > Project: Lucene.Net
> > > > Issue Type: Bug
> > > > Environment: .NET 2.0
> > > > Reporter: Anton K.
> > > > Assignee: Digy
> > > > Priority: Critical
> > > > Attachments: DIGY-FieldCacheImpl.patch, Digy.rar,
> > > luceneSrc_memUsage.patch, Paches for v2.3.1.rar,
> > > WeakHashTable+FieldCacheImpl.rar, WeakReferences.rar
> > > >
> > > >
> > > > readerCache Hashtable field (see FieldCacheImpl.cs) never
> > > releases some hash items that have closed IndexReader
> > object as a key.
> > > So a lot of Term instances are never released.
> > > > Java version of Lucene uses WeakHashMap and therefore
> > > doesn't have this problem.
> > > > This bug can be reproduced only when Sort functionality
> > > used during search.
> > > > See following link for additional information.
> > > > http://www.gossamer-threads.com/lists/lucene/java-user/55681
> > > > Steps to reproduce:
> > > > 1)Create index
> > > > 2) Modify index by IndexWiter; Close IndexWriter
> > > > 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> > > > 4) Go to step 2
> > > > You'll get OutOfMemoryException after some time of running
> > > this algorithm.
> > >
> > > --
> > > This message is automatically generated by JIRA.
> > > -
> > > You can reply to this email to add a comment to the issue online.
> > >
> > >
> > >
> >
> >
>
RE: Problems with patch for LUCENENET-106
Posted by Timothy Januario <Ti...@BDMetrics.com>.
I stand corrected. I do see the possibility of the collision as a
potentially serious bug. The java version still uses the object as the
key which means that if the hashcode is identical across all instances,
it just becomes less efficient. In the current .NET implementation, it
stays very efficient, it would just only ever have a single instance in
the cache which is a very different and incorrect result.
-----Original Message-----
From: Eyal Post [mailto:eyalpost@epocalipse.com]
Sent: Saturday, January 03, 2009 1:56 PM
To: lucene-net-dev@incubator.apache.org
Subject: RE: Problems with patch for LUCENENET-106
I tend to agree with Luc.
Hashtables don't rely on small chance for collisions to operate
correctly.
They only rely on it to operate efficiently.
This means you can store 2 elements with the same hashcode in a
hashtable
and they won't override each other, but the hashtable lookups will be
less
efficient with collisions.
The problem with code provided in the patch is that it uses the hashcode
as
the key in the hashtable and altough the chance of collisions is small
once
they happen the results are undefined.
I think the best way to solve that is to use WeakEntry as the key and
override both GetHashCode and Equals. GetHashCode should return the
HashCode
field and Equals should compare the Keys
Eyal Post
> -----Original Message-----
> From: George Aroush [mailto:george@aroush.net]
> Sent: Friday, January 02, 2009 0:52 AM
> To: lucene-net-dev@incubator.apache.org
> Subject: RE: Problems with patch for LUCENENET-106
>
> Hi Luc,
>
> I have to agree with Tim.
>
> The .NET's implementation / port of WeakHashMap in Lucene.Net
> is consistent with Java Lucene. If there is a "very" small
> chance of a hash key collusion, it is also possible in the
> Java version of Lucene. Here is why.
>
> In Java, when WeakHashMap.put(Object key, Object value) is
> called, this function calls key.hashCode() to get a hash
> value (I had a look at Java's source code for WeakHashMap.)
> The same logic is happening in .NET's implementation / port
> of WeakHashMap. In addition, at least for now, the "key" is
> always of type IndexReader. Looking in IndexReader code,
> there is no override of Java::hashCode() / C#::GetHashCode().
> So, unless if Java's
> hashCode() guaranties uniqueness (which it doesn't), then the
> possibility of a hash value collusion also exists in the Java
> version of Lucene. If this is the case, then we have a valid
> defect in Lucene and it should be addressed.
>
> Erik, do you agree, can you comment?
>
> Regarding #2, & #3, I agree that the .NET implementation of
> WeakHashMap could have been confined to just get() / put().
> Since you already have a patch for Lucene.Net 2.1, please
> submit a new JIRA issue and attach your patch to it (when you
> are ready.) I look forward to your patch.
>
> Regards, and Happy New Year everyone!
>
> -- George
>
>
> > -----Original Message-----
> > From: Timothy Januario [mailto:Timothy.Januario@BDMetrics.com]
> > Sent: Wednesday, December 31, 2008 9:44 AM
> > To: lucene-net-dev@incubator.apache.org
> > Subject: RE: Problems with patch for LUCENENET-106
> >
> > Luc,
> > I'm not sure that I completely agree with your assessment of the
> > WeakHashMap.
> >
> > It does assume that the hash value is unique which is a
> correct usage.
> > It is the responsibility of
> > object.GetHashCode() to return a unique value. This is a
> problem in
> > any Hashtable implementation and although I haven't seen
> the internal
> > java code, I assume that it has the same limitation. The
> object would
> > be the key only superficially as the hashcode would be used
> internally
> > as well (based on the name of the object, WeakHashMap, anyway).
> > If this is incorrect, I apologize for the noise. I did, however,
> > notice that GetHashCode() is not overridden in the IndexReader and
> > since the .NET framework does not guarantee a unique value
> (according
> > to the documentation for oject.GetHashCode()), this could
> present the
> > problem that you have identified. Is there a need to guarantee
> > uniqueness by overriding the method or does anyone know if
> the value
> > is always unique already? When the Cache object was
> implemented with
> > Hashtable prior to this patch, the same problem would have been
> > inherent with that implementation (of course with the
> additional bonus
> > of the memory leak ;)) and I don't think I ever saw collisions
> > although quite honestly I was never really looking for them.
> >
> > Also, you said that the call to WeakHashMap.Keys could return NULL
> > values. This may be the reason why the Java implementation
> supports
> > null keys. It does seem to be a possibility unless the
> call to Keys
> > checks that WeakReference.Key.Target is not null at the time of
> > iteration (this would eliminate this possibility since we would now
> > have a strong reference to the object ie: object key =
> > entries[i].Key.Target; if (object != null) keys.Add(object);). The
> > same problem would exist with the Values property.
> >
> > Also, optimizing for few IndexReaders in the WeakHashMap is an
> > assumption that specializes the class for IndexReaders.
> > There is no guarantee that this will be the only use case in the
> > future. The Java documentation says that the same initial
> parameters
> > (loadfactor and capacity) as HashMap are used which
> suggests that it
> > should be left in the .NET implementation with the same
> parameters as
> > the default Hashtable which is its underlying container.
> >
> > Comments?
> > -tim
> >
> > -----Original Message-----
> > From: Vanlerberghe, Luc [mailto:Luc.Vanlerberghe@bvdep.com]
> > Sent: Wednesday, December 31, 2008 4:48 AM
> > To: lucene-net-dev@incubator.apache.org
> > Subject: Problems with patch for LUCENENET-106
> >
> > Hi,
> >
> > I reviewed the fix for LUCENENET-106 and I see a couple of issues:
> >
> > 1. The implementation of WeakHashMap in SupportClass.cs has
> a bug in
> > that it assumes all keys will have a unique HashCode.
> > Although the chances of this occurring are *very* small, it
> cannot be
> > guaranteed. A collision would mean that the cached value for one
> > indexReader could be returned as cached value for another
> indexReader
> > (using the way this class is used in Lucene as example)
> >
> > 2. This class attempts to be a complete port of the
> > java.util.WeakHashMap, but:
> > - A correct implementation is difficult to implement, and .Net does
> > not have an equivalent of java.lang.ref.ReferenceQueue to
> accelerate
> > the cleanup process.
> > - Since the behaviour of the WeakHashMap is closely tied to the
> > behaviour of the garbage collector, it is very difficult to test
> > properly, if at all.
> > - Lucene only uses the get(Object key) and put(K key, V
> value) methods
> > - In Lucene, the keys are IndexReader instances. In normal
> use cases
> > there will be only 1 IndexReader live, and perhaps another one if
> > 'warming up' is used before switching IndexSearchers after an index
> > update.
> > - If this class is presented as a complete port of
> > java.util.WeakHashMap, users might assume this is
> production quality
> > code and copy/paste its code in their own projects.
> > Any further bugs found might harm the credibility of the Lucene
> > project, even if those sections are never used in Lucene.
> (e.g.: The
> > collection returned by Keys might contain null values. There's no
> > guarantee the GC won't intervene between the call to
> Cleanup and the
> > calls to keys.Add(w.Key.Target).
> >
> > 3. Most support classes that are needed for the conversion
> of java to
> > .Net are in the anonymous namespace, but this one is in
> > Lucene.Net.Util. I would propose to keep the namespaces
> corresponding
> > to the original java packages clean and either put all
> support classes
> > in the anonymous namespace or in a Lucene.Net specific one
> > (Lucene.Net.DotNetPort ?) (I see that George moved it already to
> > SupportClass, thanks George!)
> >
> > I have been using the java version of Lucene in a project
> for a long
> > time.
> > For a new project that is currently under development, we will be
> > using the .Net version.
> > For now we are using the 2.1 version using a custom patch
> to avoid the
> > memory leak problem.
> >
> > I'll post an up to date version of this patch as a
> replacement for the
> > current implementation of Lucene.Net.Util.WeakHashMap sometime next
> > week...
> > - It doesn't try to be a complete implementation of
> > WeakHashMap: only the methods strictly necessary are
> implemented, all
> > other methods throw a NotImplementedException. (It should
> probably be
> > renamed to SimplifiedWeakHashMap or something)
> > - It's optimized for the 'low number of keys' case.
> > - It's simple code, but I want to test it thoroughly first.
> > The original patch was put directly in the FieldCacheImpl
> code, I want
> > to make sure I didn't make some stupid mistake while converting it.
> >
> > I have an account on Jira, but I don't have rights to reopen the
> > issue...
> >
> > Regards,
> >
> > Luc
> >
> >
> > -----Original Message-----
> > From: Digy (JIRA) [mailto:jira@apache.org]
> > Sent: zaterdag 27 december 2008 19:47
> > To: lucene-net-dev@incubator.apache.org
> > Subject: [jira] Closed: () Lucene.NET (Revision: 603121) is leaking
> > memory
> >
> >
> > [
> > https://issues.apache.org/jira/browse/LUCENENET-106?page=com.a
> tlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
> >
> > Digy closed LUCENENET-106.
> > --------------------------
> >
> > Resolution: Fixed
> > Assignee: Digy
> >
> > Patches are applied.
> >
> > DIGY.
> >
> > > Lucene.NET (Revision: 603121) is leaking memory
> > > -----------------------------------------------
> > >
> > > Key: LUCENENET-106
> > > URL:
> > https://issues.apache.org/jira/browse/LUCENENET-106
> > > Project: Lucene.Net
> > > Issue Type: Bug
> > > Environment: .NET 2.0
> > > Reporter: Anton K.
> > > Assignee: Digy
> > > Priority: Critical
> > > Attachments: DIGY-FieldCacheImpl.patch, Digy.rar,
> > luceneSrc_memUsage.patch, Paches for v2.3.1.rar,
> > WeakHashTable+FieldCacheImpl.rar, WeakReferences.rar
> > >
> > >
> > > readerCache Hashtable field (see FieldCacheImpl.cs) never
> > releases some hash items that have closed IndexReader
> object as a key.
> > So a lot of Term instances are never released.
> > > Java version of Lucene uses WeakHashMap and therefore
> > doesn't have this problem.
> > > This bug can be reproduced only when Sort functionality
> > used during search.
> > > See following link for additional information.
> > > http://www.gossamer-threads.com/lists/lucene/java-user/55681
> > > Steps to reproduce:
> > > 1)Create index
> > > 2) Modify index by IndexWiter; Close IndexWriter
> > > 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> > > 4) Go to step 2
> > > You'll get OutOfMemoryException after some time of running
> > this algorithm.
> >
> > --
> > This message is automatically generated by JIRA.
> > -
> > You can reply to this email to add a comment to the issue online.
> >
> >
> >
>
>
RE: Problems with patch for LUCENENET-106
Posted by Eyal Post <ey...@epocalipse.com>.
I tend to agree with Luc.
Hashtables don't rely on small chance for collisions to operate correctly.
They only rely on it to operate efficiently.
This means you can store 2 elements with the same hashcode in a hashtable
and they won't override each other, but the hashtable lookups will be less
efficient with collisions.
The problem with code provided in the patch is that it uses the hashcode as
the key in the hashtable and altough the chance of collisions is small once
they happen the results are undefined.
I think the best way to solve that is to use WeakEntry as the key and
override both GetHashCode and Equals. GetHashCode should return the HashCode
field and Equals should compare the Keys
Eyal Post
> -----Original Message-----
> From: George Aroush [mailto:george@aroush.net]
> Sent: Friday, January 02, 2009 0:52 AM
> To: lucene-net-dev@incubator.apache.org
> Subject: RE: Problems with patch for LUCENENET-106
>
> Hi Luc,
>
> I have to agree with Tim.
>
> The .NET's implementation / port of WeakHashMap in Lucene.Net
> is consistent with Java Lucene. If there is a "very" small
> chance of a hash key collusion, it is also possible in the
> Java version of Lucene. Here is why.
>
> In Java, when WeakHashMap.put(Object key, Object value) is
> called, this function calls key.hashCode() to get a hash
> value (I had a look at Java's source code for WeakHashMap.)
> The same logic is happening in .NET's implementation / port
> of WeakHashMap. In addition, at least for now, the "key" is
> always of type IndexReader. Looking in IndexReader code,
> there is no override of Java::hashCode() / C#::GetHashCode().
> So, unless if Java's
> hashCode() guaranties uniqueness (which it doesn't), then the
> possibility of a hash value collusion also exists in the Java
> version of Lucene. If this is the case, then we have a valid
> defect in Lucene and it should be addressed.
>
> Erik, do you agree, can you comment?
>
> Regarding #2, & #3, I agree that the .NET implementation of
> WeakHashMap could have been confined to just get() / put().
> Since you already have a patch for Lucene.Net 2.1, please
> submit a new JIRA issue and attach your patch to it (when you
> are ready.) I look forward to your patch.
>
> Regards, and Happy New Year everyone!
>
> -- George
>
>
> > -----Original Message-----
> > From: Timothy Januario [mailto:Timothy.Januario@BDMetrics.com]
> > Sent: Wednesday, December 31, 2008 9:44 AM
> > To: lucene-net-dev@incubator.apache.org
> > Subject: RE: Problems with patch for LUCENENET-106
> >
> > Luc,
> > I'm not sure that I completely agree with your assessment of the
> > WeakHashMap.
> >
> > It does assume that the hash value is unique which is a
> correct usage.
> > It is the responsibility of
> > object.GetHashCode() to return a unique value. This is a
> problem in
> > any Hashtable implementation and although I haven't seen
> the internal
> > java code, I assume that it has the same limitation. The
> object would
> > be the key only superficially as the hashcode would be used
> internally
> > as well (based on the name of the object, WeakHashMap, anyway).
> > If this is incorrect, I apologize for the noise. I did, however,
> > notice that GetHashCode() is not overridden in the IndexReader and
> > since the .NET framework does not guarantee a unique value
> (according
> > to the documentation for oject.GetHashCode()), this could
> present the
> > problem that you have identified. Is there a need to guarantee
> > uniqueness by overriding the method or does anyone know if
> the value
> > is always unique already? When the Cache object was
> implemented with
> > Hashtable prior to this patch, the same problem would have been
> > inherent with that implementation (of course with the
> additional bonus
> > of the memory leak ;)) and I don't think I ever saw collisions
> > although quite honestly I was never really looking for them.
> >
> > Also, you said that the call to WeakHashMap.Keys could return NULL
> > values. This may be the reason why the Java implementation
> supports
> > null keys. It does seem to be a possibility unless the
> call to Keys
> > checks that WeakReference.Key.Target is not null at the time of
> > iteration (this would eliminate this possibility since we would now
> > have a strong reference to the object ie: object key =
> > entries[i].Key.Target; if (object != null) keys.Add(object);). The
> > same problem would exist with the Values property.
> >
> > Also, optimizing for few IndexReaders in the WeakHashMap is an
> > assumption that specializes the class for IndexReaders.
> > There is no guarantee that this will be the only use case in the
> > future. The Java documentation says that the same initial
> parameters
> > (loadfactor and capacity) as HashMap are used which
> suggests that it
> > should be left in the .NET implementation with the same
> parameters as
> > the default Hashtable which is its underlying container.
> >
> > Comments?
> > -tim
> >
> > -----Original Message-----
> > From: Vanlerberghe, Luc [mailto:Luc.Vanlerberghe@bvdep.com]
> > Sent: Wednesday, December 31, 2008 4:48 AM
> > To: lucene-net-dev@incubator.apache.org
> > Subject: Problems with patch for LUCENENET-106
> >
> > Hi,
> >
> > I reviewed the fix for LUCENENET-106 and I see a couple of issues:
> >
> > 1. The implementation of WeakHashMap in SupportClass.cs has
> a bug in
> > that it assumes all keys will have a unique HashCode.
> > Although the chances of this occurring are *very* small, it
> cannot be
> > guaranteed. A collision would mean that the cached value for one
> > indexReader could be returned as cached value for another
> indexReader
> > (using the way this class is used in Lucene as example)
> >
> > 2. This class attempts to be a complete port of the
> > java.util.WeakHashMap, but:
> > - A correct implementation is difficult to implement, and .Net does
> > not have an equivalent of java.lang.ref.ReferenceQueue to
> accelerate
> > the cleanup process.
> > - Since the behaviour of the WeakHashMap is closely tied to the
> > behaviour of the garbage collector, it is very difficult to test
> > properly, if at all.
> > - Lucene only uses the get(Object key) and put(K key, V
> value) methods
> > - In Lucene, the keys are IndexReader instances. In normal
> use cases
> > there will be only 1 IndexReader live, and perhaps another one if
> > 'warming up' is used before switching IndexSearchers after an index
> > update.
> > - If this class is presented as a complete port of
> > java.util.WeakHashMap, users might assume this is
> production quality
> > code and copy/paste its code in their own projects.
> > Any further bugs found might harm the credibility of the Lucene
> > project, even if those sections are never used in Lucene.
> (e.g.: The
> > collection returned by Keys might contain null values. There's no
> > guarantee the GC won't intervene between the call to
> Cleanup and the
> > calls to keys.Add(w.Key.Target).
> >
> > 3. Most support classes that are needed for the conversion
> of java to
> > .Net are in the anonymous namespace, but this one is in
> > Lucene.Net.Util. I would propose to keep the namespaces
> corresponding
> > to the original java packages clean and either put all
> support classes
> > in the anonymous namespace or in a Lucene.Net specific one
> > (Lucene.Net.DotNetPort ?) (I see that George moved it already to
> > SupportClass, thanks George!)
> >
> > I have been using the java version of Lucene in a project
> for a long
> > time.
> > For a new project that is currently under development, we will be
> > using the .Net version.
> > For now we are using the 2.1 version using a custom patch
> to avoid the
> > memory leak problem.
> >
> > I'll post an up to date version of this patch as a
> replacement for the
> > current implementation of Lucene.Net.Util.WeakHashMap sometime next
> > week...
> > - It doesn't try to be a complete implementation of
> > WeakHashMap: only the methods strictly necessary are
> implemented, all
> > other methods throw a NotImplementedException. (It should
> probably be
> > renamed to SimplifiedWeakHashMap or something)
> > - It's optimized for the 'low number of keys' case.
> > - It's simple code, but I want to test it thoroughly first.
> > The original patch was put directly in the FieldCacheImpl
> code, I want
> > to make sure I didn't make some stupid mistake while converting it.
> >
> > I have an account on Jira, but I don't have rights to reopen the
> > issue...
> >
> > Regards,
> >
> > Luc
> >
> >
> > -----Original Message-----
> > From: Digy (JIRA) [mailto:jira@apache.org]
> > Sent: zaterdag 27 december 2008 19:47
> > To: lucene-net-dev@incubator.apache.org
> > Subject: [jira] Closed: () Lucene.NET (Revision: 603121) is leaking
> > memory
> >
> >
> > [
> > https://issues.apache.org/jira/browse/LUCENENET-106?page=com.a
> tlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
> >
> > Digy closed LUCENENET-106.
> > --------------------------
> >
> > Resolution: Fixed
> > Assignee: Digy
> >
> > Patches are applied.
> >
> > DIGY.
> >
> > > Lucene.NET (Revision: 603121) is leaking memory
> > > -----------------------------------------------
> > >
> > > Key: LUCENENET-106
> > > URL:
> > https://issues.apache.org/jira/browse/LUCENENET-106
> > > Project: Lucene.Net
> > > Issue Type: Bug
> > > Environment: .NET 2.0
> > > Reporter: Anton K.
> > > Assignee: Digy
> > > Priority: Critical
> > > Attachments: DIGY-FieldCacheImpl.patch, Digy.rar,
> > luceneSrc_memUsage.patch, Paches for v2.3.1.rar,
> > WeakHashTable+FieldCacheImpl.rar, WeakReferences.rar
> > >
> > >
> > > readerCache Hashtable field (see FieldCacheImpl.cs) never
> > releases some hash items that have closed IndexReader
> object as a key.
> > So a lot of Term instances are never released.
> > > Java version of Lucene uses WeakHashMap and therefore
> > doesn't have this problem.
> > > This bug can be reproduced only when Sort functionality
> > used during search.
> > > See following link for additional information.
> > > http://www.gossamer-threads.com/lists/lucene/java-user/55681
> > > Steps to reproduce:
> > > 1)Create index
> > > 2) Modify index by IndexWiter; Close IndexWriter
> > > 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> > > 4) Go to step 2
> > > You'll get OutOfMemoryException after some time of running
> > this algorithm.
> >
> > --
> > This message is automatically generated by JIRA.
> > -
> > You can reply to this email to add a comment to the issue online.
> >
> >
> >
>
>
RE: Problems with patch for LUCENENET-106
Posted by George Aroush <ge...@aroush.net>.
Hi Luc,
I have to agree with Tim.
The .NET's implementation / port of WeakHashMap in Lucene.Net is consistent
with Java Lucene. If there is a "very" small chance of a hash key
collusion, it is also possible in the Java version of Lucene. Here is why.
In Java, when WeakHashMap.put(Object key, Object value) is called, this
function calls key.hashCode() to get a hash value (I had a look at Java's
source code for WeakHashMap.) The same logic is happening in .NET's
implementation / port of WeakHashMap. In addition, at least for now, the
"key" is always of type IndexReader. Looking in IndexReader code, there is
no override of Java::hashCode() / C#::GetHashCode(). So, unless if Java's
hashCode() guaranties uniqueness (which it doesn't), then the possibility of
a hash value collusion also exists in the Java version of Lucene. If this
is the case, then we have a valid defect in Lucene and it should be
addressed.
Erik, do you agree, can you comment?
Regarding #2, & #3, I agree that the .NET implementation of WeakHashMap
could have been confined to just get() / put(). Since you already have a
patch for Lucene.Net 2.1, please submit a new JIRA issue and attach your
patch to it (when you are ready.) I look forward to your patch.
Regards, and Happy New Year everyone!
-- George
> -----Original Message-----
> From: Timothy Januario [mailto:Timothy.Januario@BDMetrics.com]
> Sent: Wednesday, December 31, 2008 9:44 AM
> To: lucene-net-dev@incubator.apache.org
> Subject: RE: Problems with patch for LUCENENET-106
>
> Luc,
> I'm not sure that I completely agree with your assessment of
> the WeakHashMap.
>
> It does assume that the hash value is unique which is a
> correct usage. It is the responsibility of
> object.GetHashCode() to return a unique value. This is a
> problem in any Hashtable implementation and although I
> haven't seen the internal java code, I assume that it has the
> same limitation. The object would be the key only
> superficially as the hashcode would be used internally as
> well (based on the name of the object, WeakHashMap, anyway).
> If this is incorrect, I apologize for the noise. I did,
> however, notice that GetHashCode() is not overridden in the
> IndexReader and since the .NET framework does not guarantee a
> unique value (according to the documentation for
> oject.GetHashCode()), this could present the problem that you
> have identified. Is there a need to guarantee uniqueness by
> overriding the method or does anyone know if the value is
> always unique already? When the Cache object was implemented
> with Hashtable prior to this patch, the same problem would
> have been inherent with that implementation (of course with
> the additional bonus of the memory leak ;)) and I don't think
> I ever saw collisions although quite honestly I was never
> really looking for them.
>
> Also, you said that the call to WeakHashMap.Keys could return
> NULL values. This may be the reason why the Java
> implementation supports null keys. It does seem to be a
> possibility unless the call to Keys checks that
> WeakReference.Key.Target is not null at the time of iteration
> (this would eliminate this possibility since we would now
> have a strong reference to the object ie: object key =
> entries[i].Key.Target; if (object != null)
> keys.Add(object);). The same problem would exist with the
> Values property.
>
> Also, optimizing for few IndexReaders in the WeakHashMap is
> an assumption that specializes the class for IndexReaders.
> There is no guarantee that this will be the only use case in
> the future. The Java documentation says that the same
> initial parameters (loadfactor and capacity) as HashMap are
> used which suggests that it should be left in the .NET
> implementation with the same parameters as the default
> Hashtable which is its underlying container.
>
> Comments?
> -tim
>
> -----Original Message-----
> From: Vanlerberghe, Luc [mailto:Luc.Vanlerberghe@bvdep.com]
> Sent: Wednesday, December 31, 2008 4:48 AM
> To: lucene-net-dev@incubator.apache.org
> Subject: Problems with patch for LUCENENET-106
>
> Hi,
>
> I reviewed the fix for LUCENENET-106 and I see a couple of issues:
>
> 1. The implementation of WeakHashMap in SupportClass.cs has a
> bug in that it assumes all keys will have a unique HashCode.
> Although the chances of this occurring are *very* small, it
> cannot be guaranteed. A collision would mean that the cached
> value for one indexReader could be returned as cached value
> for another indexReader (using the way this class is used in
> Lucene as example)
>
> 2. This class attempts to be a complete port of the
> java.util.WeakHashMap, but:
> - A correct implementation is difficult to implement, and
> .Net does not have an equivalent of
> java.lang.ref.ReferenceQueue to accelerate the cleanup process.
> - Since the behaviour of the WeakHashMap is closely tied to
> the behaviour of the garbage collector, it is very difficult
> to test properly, if at all.
> - Lucene only uses the get(Object key) and put(K key, V value) methods
> - In Lucene, the keys are IndexReader instances. In normal
> use cases there will be only 1 IndexReader live, and perhaps
> another one if 'warming up' is used before switching
> IndexSearchers after an index update.
> - If this class is presented as a complete port of
> java.util.WeakHashMap, users might assume this is production
> quality code and copy/paste its code in their own projects.
> Any further bugs found might harm the credibility of the
> Lucene project, even if those sections are never used in
> Lucene. (e.g.: The collection returned by Keys might contain
> null values. There's no guarantee the GC won't intervene
> between the call to Cleanup and the calls to keys.Add(w.Key.Target).
>
> 3. Most support classes that are needed for the conversion of
> java to .Net are in the anonymous namespace, but this one is
> in Lucene.Net.Util. I would propose to keep the namespaces
> corresponding to the original java packages clean and either
> put all support classes in the anonymous namespace or in a
> Lucene.Net specific one (Lucene.Net.DotNetPort ?) (I see that
> George moved it already to SupportClass, thanks George!)
>
> I have been using the java version of Lucene in a project for
> a long time.
> For a new project that is currently under development, we
> will be using the .Net version.
> For now we are using the 2.1 version using a custom patch to
> avoid the memory leak problem.
>
> I'll post an up to date version of this patch as a
> replacement for the current implementation of
> Lucene.Net.Util.WeakHashMap sometime next week...
> - It doesn't try to be a complete implementation of
> WeakHashMap: only the methods strictly necessary are
> implemented, all other methods throw a
> NotImplementedException. (It should probably be renamed to
> SimplifiedWeakHashMap or something)
> - It's optimized for the 'low number of keys' case.
> - It's simple code, but I want to test it thoroughly first.
> The original patch was put directly in the FieldCacheImpl
> code, I want to make sure I didn't make some stupid mistake
> while converting it.
>
> I have an account on Jira, but I don't have rights to reopen
> the issue...
>
> Regards,
>
> Luc
>
>
> -----Original Message-----
> From: Digy (JIRA) [mailto:jira@apache.org]
> Sent: zaterdag 27 december 2008 19:47
> To: lucene-net-dev@incubator.apache.org
> Subject: [jira] Closed: () Lucene.NET (Revision: 603121) is
> leaking memory
>
>
> [
> https://issues.apache.org/jira/browse/LUCENENET-106?page=com.a
tlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
>
> Digy closed LUCENENET-106.
> --------------------------
>
> Resolution: Fixed
> Assignee: Digy
>
> Patches are applied.
>
> DIGY.
>
> > Lucene.NET (Revision: 603121) is leaking memory
> > -----------------------------------------------
> >
> > Key: LUCENENET-106
> > URL:
> https://issues.apache.org/jira/browse/LUCENENET-106
> > Project: Lucene.Net
> > Issue Type: Bug
> > Environment: .NET 2.0
> > Reporter: Anton K.
> > Assignee: Digy
> > Priority: Critical
> > Attachments: DIGY-FieldCacheImpl.patch, Digy.rar,
> luceneSrc_memUsage.patch, Paches for v2.3.1.rar,
> WeakHashTable+FieldCacheImpl.rar, WeakReferences.rar
> >
> >
> > readerCache Hashtable field (see FieldCacheImpl.cs) never
> releases some hash items that have closed IndexReader object
> as a key. So a lot of Term instances are never released.
> > Java version of Lucene uses WeakHashMap and therefore
> doesn't have this problem.
> > This bug can be reproduced only when Sort functionality
> used during search.
> > See following link for additional information.
> > http://www.gossamer-threads.com/lists/lucene/java-user/55681
> > Steps to reproduce:
> > 1)Create index
> > 2) Modify index by IndexWiter; Close IndexWriter
> > 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> > 4) Go to step 2
> > You'll get OutOfMemoryException after some time of running
> this algorithm.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
>
RE: Problems with patch for LUCENENET-106
Posted by Timothy Januario <Ti...@BDMetrics.com>.
Luc,
I'm not sure that I completely agree with your assessment of the WeakHashMap.
It does assume that the hash value is unique which is a correct usage. It is the responsibility of object.GetHashCode() to return a unique value. This is a problem in any Hashtable implementation and although I haven't seen the internal java code, I assume that it has the same limitation. The object would be the key only superficially as the hashcode would be used internally as well (based on the name of the object, WeakHashMap, anyway). If this is incorrect, I apologize for the noise. I did, however, notice that GetHashCode() is not overridden in the IndexReader and since the .NET framework does not guarantee a unique value (according to the documentation for oject.GetHashCode()), this could present the problem that you have identified. Is there a need to guarantee uniqueness by overriding the method or does anyone know if the value is always unique already? When the Cache object was implemented with Hashtable prior to this patch, the same problem would have been inherent with that implementation (of course with the additional bonus of the memory leak ;)) and I don't think I ever saw collisions although quite honestly I was never really looking for them.
Also, you said that the call to WeakHashMap.Keys could return NULL values. This may be the reason why the Java implementation supports null keys. It does seem to be a possibility unless the call to Keys checks that WeakReference.Key.Target is not null at the time of iteration (this would eliminate this possibility since we would now have a strong reference to the object ie: object key = entries[i].Key.Target; if (object != null) keys.Add(object);). The same problem would exist with the Values property.
Also, optimizing for few IndexReaders in the WeakHashMap is an assumption that specializes the class for IndexReaders. There is no guarantee that this will be the only use case in the future. The Java documentation says that the same initial parameters (loadfactor and capacity) as HashMap are used which suggests that it should be left in the .NET implementation with the same parameters as the default Hashtable which is its underlying container.
Comments?
-tim
-----Original Message-----
From: Vanlerberghe, Luc [mailto:Luc.Vanlerberghe@bvdep.com]
Sent: Wednesday, December 31, 2008 4:48 AM
To: lucene-net-dev@incubator.apache.org
Subject: Problems with patch for LUCENENET-106
Hi,
I reviewed the fix for LUCENENET-106 and I see a couple of issues:
1. The implementation of WeakHashMap in SupportClass.cs has a bug in that it assumes all keys will have a unique HashCode. Although the chances of this occurring are *very* small, it cannot be guaranteed. A collision would mean that the cached value for one indexReader could be returned as cached value for another indexReader (using the way this class is used in Lucene as example)
2. This class attempts to be a complete port of the java.util.WeakHashMap, but:
- A correct implementation is difficult to implement, and .Net does not have an equivalent of java.lang.ref.ReferenceQueue to accelerate the cleanup process.
- Since the behaviour of the WeakHashMap is closely tied to the behaviour of the garbage collector, it is very difficult to test properly, if at all.
- Lucene only uses the get(Object key) and put(K key, V value) methods
- In Lucene, the keys are IndexReader instances. In normal use cases there will be only 1 IndexReader live, and perhaps another one if 'warming up' is used before switching IndexSearchers after an index update.
- If this class is presented as a complete port of java.util.WeakHashMap, users might assume this is production quality code and copy/paste its code in their own projects. Any further bugs found might harm the credibility of the Lucene project, even if those sections are never used in Lucene. (e.g.: The collection returned by Keys might contain null values. There's no guarantee the GC won't intervene between the call to Cleanup and the calls to keys.Add(w.Key.Target).
3. Most support classes that are needed for the conversion of java to .Net are in the anonymous namespace, but this one is in Lucene.Net.Util. I would propose to keep the namespaces corresponding to the original java packages clean and either put all support classes in the anonymous namespace or in a Lucene.Net specific one (Lucene.Net.DotNetPort ?) (I see that George moved it already to SupportClass, thanks George!)
I have been using the java version of Lucene in a project for a long time.
For a new project that is currently under development, we will be using the .Net version.
For now we are using the 2.1 version using a custom patch to avoid the memory leak problem.
I'll post an up to date version of this patch as a replacement for the current implementation of Lucene.Net.Util.WeakHashMap sometime next week...
- It doesn't try to be a complete implementation of WeakHashMap: only the methods strictly necessary are implemented, all other methods throw a NotImplementedException. (It should probably be renamed to SimplifiedWeakHashMap or something)
- It's optimized for the 'low number of keys' case.
- It's simple code, but I want to test it thoroughly first. The original patch was put directly in the FieldCacheImpl code, I want to make sure I didn't make some stupid mistake while converting it.
I have an account on Jira, but I don't have rights to reopen the issue...
Regards,
Luc
-----Original Message-----
From: Digy (JIRA) [mailto:jira@apache.org]
Sent: zaterdag 27 december 2008 19:47
To: lucene-net-dev@incubator.apache.org
Subject: [jira] Closed: () Lucene.NET (Revision: 603121) is leaking memory
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Digy closed LUCENENET-106.
--------------------------
Resolution: Fixed
Assignee: Digy
Patches are applied.
DIGY.
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Assignee: Digy
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable+FieldCacheImpl.rar, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Problems with patch for LUCENENET-106
Posted by "Vanlerberghe, Luc" <Lu...@bvdep.com>.
Hi,
I reviewed the fix for LUCENENET-106 and I see a couple of issues:
1. The implementation of WeakHashMap in SupportClass.cs has a bug in that it assumes all keys will have a unique HashCode. Although the chances of this occurring are *very* small, it cannot be guaranteed. A collision would mean that the cached value for one indexReader could be returned as cached value for another indexReader (using the way this class is used in Lucene as example)
2. This class attempts to be a complete port of the java.util.WeakHashMap, but:
- A correct implementation is difficult to implement, and .Net does not have an equivalent of java.lang.ref.ReferenceQueue to accelerate the cleanup process.
- Since the behaviour of the WeakHashMap is closely tied to the behaviour of the garbage collector, it is very difficult to test properly, if at all.
- Lucene only uses the get(Object key) and put(K key, V value) methods
- In Lucene, the keys are IndexReader instances. In normal use cases there will be only 1 IndexReader live, and perhaps another one if 'warming up' is used before switching IndexSearchers after an index update.
- If this class is presented as a complete port of java.util.WeakHashMap, users might assume this is production quality code and copy/paste its code in their own projects. Any further bugs found might harm the credibility of the Lucene project, even if those sections are never used in Lucene. (e.g.: The collection returned by Keys might contain null values. There's no guarantee the GC won't intervene between the call to Cleanup and the calls to keys.Add(w.Key.Target).
3. Most support classes that are needed for the conversion of java to .Net are in the anonymous namespace, but this one is in Lucene.Net.Util. I would propose to keep the namespaces corresponding to the original java packages clean and either put all support classes in the anonymous namespace or in a Lucene.Net specific one (Lucene.Net.DotNetPort ?) (I see that George moved it already to SupportClass, thanks George!)
I have been using the java version of Lucene in a project for a long time.
For a new project that is currently under development, we will be using the .Net version.
For now we are using the 2.1 version using a custom patch to avoid the memory leak problem.
I'll post an up to date version of this patch as a replacement for the current implementation of Lucene.Net.Util.WeakHashMap sometime next week...
- It doesn't try to be a complete implementation of WeakHashMap: only the methods strictly necessary are implemented, all other methods throw a NotImplementedException. (It should probably be renamed to SimplifiedWeakHashMap or something)
- It's optimized for the 'low number of keys' case.
- It's simple code, but I want to test it thoroughly first. The original patch was put directly in the FieldCacheImpl code, I want to make sure I didn't make some stupid mistake while converting it.
I have an account on Jira, but I don't have rights to reopen the issue...
Regards,
Luc
-----Original Message-----
From: Digy (JIRA) [mailto:jira@apache.org]
Sent: zaterdag 27 december 2008 19:47
To: lucene-net-dev@incubator.apache.org
Subject: [jira] Closed: () Lucene.NET (Revision: 603121) is leaking memory
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Digy closed LUCENENET-106.
--------------------------
Resolution: Fixed
Assignee: Digy
Patches are applied.
DIGY.
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Assignee: Digy
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable+FieldCacheImpl.rar, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Closed: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Digy closed LUCENENET-106.
--------------------------
Resolution: Fixed
Assignee: Digy
Patches are applied.
DIGY.
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Assignee: Digy
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable+FieldCacheImpl.rar, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (LUCENENET-106) Lucene.NET (Revision:
603121) is leaking memory
Posted by "Marco Dissel (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12557244#action_12557244 ]
mdissel edited comment on LUCENENET-106 at 1/9/08 4:21 AM:
----------------------------------------------------------------
I got another serious memory leak. This is the result from DebugDiag tool. The attached patched doesn't solve this one.. Anyone solved this already?
Is this line the problem in the FieldCacheImpl?
class Cache.Get
//
innerCache = new System.Collections.Hashtable(); // shouldn't this be WeakHashtable()???
//
[DEFAULT] [hasThis] Object Lucene.Net.Search.FieldCacheImpl/AnonymousClassCache3.CreateValue(Class Lucene.Net.Index.IndexReader,Object) 0xC920F4
[DEFAULT] [hasThis] Object Lucene.Net.Search.FieldCacheImpl/Cache.Get(Class Lucene.Net.Index.IndexReader,Object)
[DEFAULT] [hasThis] Class Lucene.Net.Search.StringIndex Lucene.Net.Search.FieldCacheImpl.GetStringIndex(Class Lucene.Net.Index.IndexReader,String)
[DEFAULT] [hasThis] Object Lucene.Net.Search.FieldCacheImpl/AnonymousClassCache4.CreateValue(Class Lucene.Net.Index.IndexReader,Object)
[DEFAULT] [hasThis] Object Lucene.Net.Search.FieldCacheImpl/Cache.Get(Class Lucene.Net.Index.IndexReader,Object)
[DEFAULT] [hasThis] Object Lucene.Net.Search.FieldCacheImpl.GetAuto(Class Lucene.Net.Index.IndexReader,String)
[DEFAULT] Class Lucene.Net.Search.ScoreDocComparator Lucene.Net.Search.FieldSortedHitQueue.ComparatorAuto(Class Lucene.Net.Index.IndexReader,String)
[DEFAULT] [hasThis] Object Lucene.Net.Search.FieldSortedHitQueue/AnonymousClassCache.CreateValue(Class Lucene.Net.Index.IndexReader,Object) [DEFAULT] Class Lucene.Net.Search.ScoreDocComparator Lucene.Net.Search.FieldSortedHitQueue.ComparatorAuto(Class Lucene.Net.Index.IndexReader,String)
[DEFAULT] [hasThis] Object Lucene.Net.Search.FieldCacheImpl/Cache.Get(Class Lucene.Net.Index.IndexReader,Object)
[DEFAULT] Class Lucene.Net.Search.ScoreDocComparator Lucene.Net.Search.FieldSortedHitQueue.GetCachedComparator(Class Lucene.Net.Index.IndexReader,String,I4,Class System.Globalization.CultureInfo,Class Lucene.Net.Search.SortComparatorSource)
[DEFAULT] [hasThis] Void Lucene.Net.Search.FieldSortedHitQueue..ctor(Class Lucene.Net.Index.IndexReader,SZArray Class Lucene.Net.Search.SortField,I4) [DEFAULT] Class Lucene.Net.Search.ScoreDocComparator Lucene.Net.Search.FieldSortedHitQueue.GetCachedComparator(Class Lucene.Net.Index.IndexReader,String,I4,Class System.Globalization.CultureInfo,Class Lucene.Net.Search.SortComparatorSource)
[DEFAULT] [hasThis] Class Lucene.Net.Search.TopFieldDocs Lucene.Net.Search.IndexSearcher.Search(Class Lucene.Net.Search.Weight,Class Lucene.Net.Search.Filter,I4,Class Lucene.Net.Search.Sort) [DEFAULT] [hasThis] Void Lucene.Net.Search.FieldSortedHitQueue..ctor(Class Lucene.Net.Index.IndexReader,SZArray Class Lucene.Net.Search.SortField,I4)
[DEFAULT] [hasThis] Class Lucene.Net.Search.TopFieldDocs Lucene.Net.Search.Searcher.Search(Class Lucene.Net.Search.Query,Class Lucene.Net.Search.Filter,I4,Class Lucene.Net.Search.Sort)
was (Author: mdissel):
I got another serious memory leak. This is the result from DebugDiag tool. The attached patched doesn't solve this one.. Anyone solved this already?
[DEFAULT] [hasThis] Object Lucene.Net.Search.FieldCacheImpl/AnonymousClassCache3.CreateValue(Class Lucene.Net.Index.IndexReader,Object) 0xC920F4
[DEFAULT] [hasThis] Object Lucene.Net.Search.FieldCacheImpl/Cache.Get(Class Lucene.Net.Index.IndexReader,Object)
[DEFAULT] [hasThis] Class Lucene.Net.Search.StringIndex Lucene.Net.Search.FieldCacheImpl.GetStringIndex(Class Lucene.Net.Index.IndexReader,String)
[DEFAULT] [hasThis] Object Lucene.Net.Search.FieldCacheImpl/AnonymousClassCache4.CreateValue(Class Lucene.Net.Index.IndexReader,Object)
[DEFAULT] [hasThis] Object Lucene.Net.Search.FieldCacheImpl/Cache.Get(Class Lucene.Net.Index.IndexReader,Object)
[DEFAULT] [hasThis] Object Lucene.Net.Search.FieldCacheImpl.GetAuto(Class Lucene.Net.Index.IndexReader,String)
[DEFAULT] Class Lucene.Net.Search.ScoreDocComparator Lucene.Net.Search.FieldSortedHitQueue.ComparatorAuto(Class Lucene.Net.Index.IndexReader,String)
[DEFAULT] [hasThis] Object Lucene.Net.Search.FieldSortedHitQueue/AnonymousClassCache.CreateValue(Class Lucene.Net.Index.IndexReader,Object) [DEFAULT] Class Lucene.Net.Search.ScoreDocComparator Lucene.Net.Search.FieldSortedHitQueue.ComparatorAuto(Class Lucene.Net.Index.IndexReader,String)
[DEFAULT] [hasThis] Object Lucene.Net.Search.FieldCacheImpl/Cache.Get(Class Lucene.Net.Index.IndexReader,Object)
[DEFAULT] Class Lucene.Net.Search.ScoreDocComparator Lucene.Net.Search.FieldSortedHitQueue.GetCachedComparator(Class Lucene.Net.Index.IndexReader,String,I4,Class System.Globalization.CultureInfo,Class Lucene.Net.Search.SortComparatorSource)
[DEFAULT] [hasThis] Void Lucene.Net.Search.FieldSortedHitQueue..ctor(Class Lucene.Net.Index.IndexReader,SZArray Class Lucene.Net.Search.SortField,I4) [DEFAULT] Class Lucene.Net.Search.ScoreDocComparator Lucene.Net.Search.FieldSortedHitQueue.GetCachedComparator(Class Lucene.Net.Index.IndexReader,String,I4,Class System.Globalization.CultureInfo,Class Lucene.Net.Search.SortComparatorSource)
[DEFAULT] [hasThis] Class Lucene.Net.Search.TopFieldDocs Lucene.Net.Search.IndexSearcher.Search(Class Lucene.Net.Search.Weight,Class Lucene.Net.Search.Filter,I4,Class Lucene.Net.Search.Sort) [DEFAULT] [hasThis] Void Lucene.Net.Search.FieldSortedHitQueue..ctor(Class Lucene.Net.Index.IndexReader,SZArray Class Lucene.Net.Search.SortField,I4)
[DEFAULT] [hasThis] Class Lucene.Net.Search.TopFieldDocs Lucene.Net.Search.Searcher.Search(Class Lucene.Net.Search.Query,Class Lucene.Net.Search.Filter,I4,Class Lucene.Net.Search.Sort)
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, WeakHashTable+FieldCacheImpl.rar, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Digy updated LUCENENET-106:
---------------------------
Attachment: WeakHashTable v2.patch
Some corrections for "WeakHashTable v2.patch" proposed by Tim Januario.
DIGY
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Assignee: Digy
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable v2.patch, WeakHashTable v2.patch, WeakHashTable+FieldCacheImpl.rar, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "Tim Januario (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12658665#action_12658665 ]
Tim Januario commented on LUCENENET-106:
----------------------------------------
I'm not sure what the protocols are here with regards to when things get committed so I apologize for my possible ignorance.
This bug can be pretty serious depending on how Lucene is being used and how often the sort caches are loaded. Is there a reason that these patches haven't been committed to the HEAD or branch yet? The .NET Hashtable and the Java WeakHashMap are very different structures. By using a Hashtable, the code is out-of-line with the Java release. It seems strange that users have to track down the patches that should be applied to fix this bug. Digy, Erich's and then your code were both pretty solid implementations of the Java port. They should really be included in the build.
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable+FieldCacheImpl.rar, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Digy updated LUCENENET-106:
---------------------------
Attachment: DIGY-FieldCacheImpl.patch
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "Anton K. (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12554835 ]
Anton K. commented on LUCENENET-106:
------------------------------------
Ross,
Terms also can be stored inside static Lucene.Net.Search.FieldCache_Fields.DEFAULT field.
Make shure that you clean it when closing IndexReader.
I implemented bool IsClosed property in IndexReader and remove all closed IndexReaders inside following method.
public virtual System.Object Get(IndexReader reader, System.Object key)
This solves all memory problems that I had.
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Priority: Critical
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (LUCENENET-106) Lucene.NET (Revision:
603121) is leaking memory
Posted by "Anton K. (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12555416#action_12555416 ]
anton! edited comment on LUCENENET-106 at 1/2/08 12:53 PM:
-------------------------------------------------------------
the patch was added
was (Author: anton!):
the patch
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Priority: Critical
> Attachments: luceneSrc_memUsage.patch
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "Marco Dissel (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558556#action_12558556 ]
Marco Dissel commented on LUCENENET-106:
----------------------------------------
When can we expect these fixes in the trunk?
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, WeakHashTable+FieldCacheImpl.rar, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "Anton K. (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Anton K. updated LUCENENET-106:
-------------------------------
Comment: was deleted
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Priority: Critical
> Attachments: Digy.rar, luceneSrc_memUsage.patch
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "Anton K. (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12555684#action_12555684 ]
Anton K. commented on LUCENENET-106:
------------------------------------
I like DIGY's solution. It's flexible.
But I'm not sure that there are no calls to closed IndexReader(s) that may create hashtable item(s) which will never be deleted (since IndexReaderClosed has been invoked already). Probably, there should be protection from this.
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Priority: Critical
> Attachments: Digy.rar, luceneSrc_memUsage.patch
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "TJ Kolev (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12665560#action_12665560 ]
TJ Kolev commented on LUCENENET-106:
------------------------------------
Wow guys, lots of action, good work.
On Digy's last comment, how about the following enumerator. Tests I have pass. But I am not certain it totally lives up to the expected IDictionaryEnumerator semantics. The enumerator relies on the Keys from the WeakHashTable to get the right elements.
tjk :)
{noformat}
class WeakDictionaryEnumerator : IDictionaryEnumerator
{
private ArrayList _entries = null;
private int _currentNdx = -1;
public WeakDictionaryEnumerator(WeakHashTable tbl)
{
_entries = new ArrayList(tbl.Keys.Count);
foreach(object key in tbl.Keys)
{
_entries.Add(new DictionaryEntry(key, tbl[key]));
}
}
public object Key
{
get { return Entry.Key; }
}
public object Value
{
get { return Entry.Value; }
}
public DictionaryEntry Entry
{
get { return (DictionaryEntry) _entries[_currentNdx]; }
}
public bool MoveNext()
{
if (_currentNdx + 1 >= _entries.Count)
return false;
_currentNdx++;
return true;
}
public void Reset()
{
_currentNdx = -1;
}
public object Current
{
get { return Entry; }
}
}
{noformat}
In WeakHashTable:
{noformat}
public override IDictionaryEnumerator GetEnumerator()
{
return new WeakDictionaryEnumerator(this);
}
{noformat}
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Assignee: Digy
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable v2.patch, WeakHashTable v2.patch, WeakHashTable+FieldCacheImpl.rar, WeakHashTable_ep.zip, WeakHashTable_ep_v2.zip, WeakHashTable_tj.zip, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Digy updated LUCENENET-106:
---------------------------
Attachment: WeakHashTable v2.patch
While I was waiting the patch from Luc, I prepared another version of WeakHashTable. I suppose It is free from the bug "HashCode collision"
(See Discussions on dev mailing list "Problems with patch for LUCENENET-106" )
DIGY
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Assignee: Digy
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable v2.patch, WeakHashTable+FieldCacheImpl.rar, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (LUCENENET-106) Lucene.NET (Revision:
603121) is leaking memory
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12556259#action_12556259 ]
digydigy edited comment on LUCENENET-106 at 1/5/08 2:48 PM:
--------------------------------------------------------
Since "readerCache" in "FieldCacheImpl.Cache.Get()" stores references to IndexReaders, IndexReaders are never garbage-collected.
Instead of using the IndexReader itself,using the HashCode of it as key allows it to get garbage-collected. This change eliminates most of the memory leakage.
But still the entry of the garbage-collected IndexReader remains in readerCache. This is the second reason of the memory leakage.
I will send another patch that utilizes WeakReference to determine whether an IndexReader is garbage-collected or not. If Yes,then the entry in readerCache can safely be removed.
This patch(DIGY-FieldCacheImpl.patch) does not require a change in IndexReader class.
was (Author: digydigy):
Since "readerCache" in "FieldCacheImpl.Cache.Get()" stores a reference to IndexReaders, IndexReaders are never garbage-collected.
Instead of using the IndexReader itself,using the HashCode of it as key allows it to get garbage-collected. This change eliminates most of the memory leakage.
But still the entry of the garbage-collected IndexReader remains in readerCache. This is the second reason of the memory leakage.
I will send another patch that utilizes WeakReference to determine whether an IndexReader is garbage-collected or not. If Yes,then the entry in readerCache can safely be removed.
This patch(DIGY-FieldCacheImpl.patch) does not require a change in IndexReader class.
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "eyal post (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
eyal post updated LUCENENET-106:
--------------------------------
Attachment: WeakHashTable_ep_v3.zip
Last version with comments and License
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Assignee: Digy
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable v2.patch, WeakHashTable v2.patch, WeakHashTable+FieldCacheImpl.rar, WeakHashTable_ep.zip, WeakHashTable_ep_v2.zip, WeakHashTable_ep_v3.zip, WeakHashTable_tj.zip, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "George Aroush (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
George Aroush reopened LUCENENET-106:
-------------------------------------
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Assignee: Digy
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable v2.patch, WeakHashTable+FieldCacheImpl.rar, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12556259#action_12556259 ]
Digy commented on LUCENENET-106:
--------------------------------
Since "readerCache" in "FieldCacheImpl.Cache.Get()" stores a reference to IndexReaders, IndexReaders are never garbage-collected.
Instead of using the IndexReader itself,using the HashCode of it as key allows it to get garbage-collected. This change eliminates most of the memory leakage.
But still the entry of the garbage-collected IndexReader remains in readerCache. This is the second reason of the memory leakage.
I will send another patch that utilizes WeakReference to determine whether an IndexReader is garbage-collected or not. If Yes,then the entry in readerCache can safely be removed.
This patch(DIGY-FieldCacheImpl.patch) does not require a change in IndexReader class.
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12664264#action_12664264 ]
Digy commented on LUCENENET-106:
--------------------------------
>>terating and counting elements in such storage brings additional problems - like handling cases when the count does not match the number of iterated elements, objects not being there when at some point they were, etc
That problems can be overcome with storing elements in a temp-table(making a strong reference to them) and returning it, as I tried in V2. But I see the discussion of having "full fledged" or minimal WeakHashTable as a endless story. Therefore I am OK with the TJ's solution. I tested it and everything seems to be alright.
PS: Not having a "Remove" method is a very minimalist approach.
DIGY
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Assignee: Digy
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable v2.patch, WeakHashTable v2.patch, WeakHashTable+FieldCacheImpl.rar, WeakHashTable_tj.zip, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Digy updated LUCENENET-106:
---------------------------
Attachment: (was: WeakHashTable+FieldCacheImpl.rar)
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, WeakHashTable+FieldCacheImpl.rar, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "eyal post (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12665719#action_12665719 ]
eyal post commented on LUCENENET-106:
-------------------------------------
The fix for Digy's comment is very simple using a while loop:
public bool MoveNext()
{
while (baseEnumerator.MoveNext())
{
object key = ((WeakKey)baseEnumerator.Key).Target;
if (key != null)
{
this.currentKey = key;
this.currentValue = baseEnumerator.Value;
return true;
}
}
return false;
}
The problem with relying on Keys for enumeration is that calling Keys creates a temporary copy of all keys (which is slow).
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Assignee: Digy
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable v2.patch, WeakHashTable v2.patch, WeakHashTable+FieldCacheImpl.rar, WeakHashTable_ep.zip, WeakHashTable_ep_v2.zip, WeakHashTable_tj.zip, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12665210#action_12665210 ]
Digy commented on LUCENENET-106:
--------------------------------
A very nice code from Eyal.
I'd suggest a tiny improvement to avoid the use of Finalizer and to simplify the code a little bit( No need for fields "cleanupNeeded", "monitor" and the class "GCMonitor"):
{code}
WeakReference collectableObject = new WeakReference(new Object());
private void CleanIfNeeded()
{
if (collectableObject.Target==null)
{
Clean();
collectableObject = new WeakReference(new Object());
}
}
{code}
DIGY
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Assignee: Digy
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable v2.patch, WeakHashTable v2.patch, WeakHashTable+FieldCacheImpl.rar, WeakHashTable_ep.zip, WeakHashTable_tj.zip, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "Ross Faneuf (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12554833 ]
Ross Faneuf commented on LUCENENET-106:
---------------------------------------
I also ran into this bug. The translation of the Java WeakHashMap to .Net results in one of the few ways I've encountered to completely defeat the CRL garbage collector. My solution was to replace the use of the thread data cache with an instance variable, since the effect of the .Net code was to re-cache the information for every instance.
Our code closes a cached IndexSearcher and reacquires a new instance whenever the index version changes. This bug resulted in a steady memory leak which could bring down a service process after running for a relatively few days.
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Priority: Critical
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "Marco Dissel (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558805#action_12558805 ]
Marco Dissel commented on LUCENENET-106:
----------------------------------------
Indeed with form1.cs it seems to be fixed (added innerCache = new WeakHashtable() as well ), but in my asp.net 1.1 webapplication i still get a big memory leak (> 500mb) when the site is searched and a backend thread is updating the index. After some hours or ~ day the site is dead with a OutOfMemoryException.. If i remove the extra sorting in the search there's no memory leak..
I traced the site with debugdiag, see the stracktrace in the above comment.
ps. no code that i can copy/paste here...
Thanks
Marco
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, WeakHashTable+FieldCacheImpl.rar, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "Andy Macourek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12668166#action_12668166 ]
Andy Macourek commented on LUCENENET-106:
-----------------------------------------
Any idea when I can will be able to get a full build with this fix? I am using Community Server and it utilizies Lucene. I am getting Out of Memory exceptions. I have tried to implement the patches one by one, but I still get the errors after hammering the server.
-AM
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Assignee: Digy
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable v2.patch, WeakHashTable v2.patch, WeakHashTable+FieldCacheImpl.rar, WeakHashTable_ep.zip, WeakHashTable_ep_v2.zip, WeakHashTable_ep_v3.zip, WeakHashTable_tj.zip, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12665892#action_12665892 ]
Digy commented on LUCENENET-106:
--------------------------------
I think the WeakHashTable in "WeakHashTable_ep_v3.zip" is mature enough.
I plan to commit it. (and it's test case).
Any comments on it?
DIGY
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Assignee: Digy
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable v2.patch, WeakHashTable v2.patch, WeakHashTable+FieldCacheImpl.rar, WeakHashTable_ep.zip, WeakHashTable_ep_v2.zip, WeakHashTable_ep_v3.zip, WeakHashTable_tj.zip, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "Tim Januario (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662433#action_12662433 ]
Tim Januario commented on LUCENENET-106:
----------------------------------------
Digy,
1) The WeakHashTable.Exists method creates a possible NullReferenceException because you used WeakEntries[i] As WeakEntry instead of (WeakEntry)WeakEntries[i]. This is only a warning but...
2) The Java WeakHashMap does not throw an execption when put() is called with a key that already exists in the table. It simply replaces the old value with the new value. .NET Hashtable does the same. The Add method of WeakHashTable should remove the WeakEntry if the key is found and replace it with the new value in order to stay in line instead of throwing the Exception.
3) WeakHashTableEnumarator should be WeakHashTableEnumerator
4) WeakEntry.HashCode is not used and should be removed.
5) WeakHashTable.InitialMem is not used and should be removed.
Other than that, I think this looks good.
-tim
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Assignee: Digy
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable v2.patch, WeakHashTable+FieldCacheImpl.rar, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "eyal post (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
eyal post updated LUCENENET-106:
--------------------------------
Attachment: WeakHashTable_ep_v2.zip
Digy, I took your suggestion another step and removed the need for the EqualityComparer since hashtable provides virtual methods just for that. This also slightly improved performance of Get
I also fixed performance issues with ForEach and CopyTo methods. New implementation attached
perf results:
WeakHashTableV1 WeakHashTableV2
Get 0.7 0.55
Add 1.9 2.0
Contains 0.65 0.55
Remove 2.5 2.5
Replace 2.8 2.7
ForEach 1.6 0.6
Keys 3.5 3.5
CopyTo 1.7 0.8
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Assignee: Digy
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable v2.patch, WeakHashTable v2.patch, WeakHashTable+FieldCacheImpl.rar, WeakHashTable_ep.zip, WeakHashTable_ep_v2.zip, WeakHashTable_tj.zip, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (LUCENENET-106) Lucene.NET (Revision:
603121) is leaking memory
Posted by "Anton K. (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12554835 ]
anton! edited comment on LUCENENET-106 at 12/28/07 12:19 PM:
---------------------------------------------------------------
Ross,
Terms also can be stored inside static Lucene.Net.Search.FieldCache_Fields.DEFAULT field.
Make sure that you clean it after closing IndexReader.
I implemented bool IsClosed property in IndexReader and remove all closed IndexReaders inside following method.
public virtual System.Object Get(IndexReader reader, System.Object key)
This solved all memory problems that I had.
was (Author: anton!):
Ross,
Terms also can be stored inside static Lucene.Net.Search.FieldCache_Fields.DEFAULT field.
Make shure that you clean it when closing IndexReader.
I implemented bool IsClosed property in IndexReader and remove all closed IndexReaders inside following method.
public virtual System.Object Get(IndexReader reader, System.Object key)
This solves all memory problems that I had.
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Priority: Critical
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12608489#action_12608489 ]
Digy commented on LUCENENET-106:
--------------------------------
Lucene.Net 2.3.1 also suffers from the same bug.
Below are the results of memory usage of the sample program
Original Lucene.Net 2.3.1
01) MemUsage: 34828 KB
02) MemUsage: 36352 KB
03) MemUsage: 39452 KB
04) MemUsage: 41768 KB
05) MemUsage: 47420 KB
06) MemUsage: 49720 KB
07) MemUsage: 52904 KB
08) MemUsage: 56712 KB
09) MemUsage: 61520 KB
10) MemUsage: 64988 KB
Patch applied to FieldCacheImpl.cs
01) MemUsage: 34644 KB
02) MemUsage: 35564 KB
03) MemUsage: 38532 KB
04) MemUsage: 41960 KB
05) MemUsage: 47144 KB
06) MemUsage: 49996 KB
07) MemUsage: 52232 KB
08) MemUsage: 54736 KB
09) MemUsage: 55276 KB
10) MemUsage: 61544 KB
Patch applied to FieldCacheImpl.cs , CachingWrapperFilter.cs , CachingSpanFilter.cs
01) MemUsage: 32764 KB
02) MemUsage: 32760 KB
03) MemUsage: 31404 KB
04) MemUsage: 31404 KB
05) MemUsage: 31404 KB
06) MemUsage: 31408 KB
07) MemUsage: 31408 KB
08) MemUsage: 31408 KB
09) MemUsage: 31408 KB
10) MemUsage: 31408 KB
{code}
//SAMPLE CODE
namespace WindowsApplication1
{
public partial class Form1 : Form
{
public Form1()
{
InitializeComponent();
}
private void Form1_Load(object sender, EventArgs e)
{
CreateTestIndex();
}
Lucene.Net.Documents.Document GetDoc()
{
Lucene.Net.Documents.Document doc = new Lucene.Net.Documents.Document();
Lucene.Net.Documents.Field f1 = new Lucene.Net.Documents.Field("field1", "value1 value11", Lucene.Net.Documents.Field.Store.YES, Lucene.Net.Documents.Field.Index.TOKENIZED);
Lucene.Net.Documents.Field f2 = new Lucene.Net.Documents.Field("field2", "value2 value22", Lucene.Net.Documents.Field.Store.YES, Lucene.Net.Documents.Field.Index.UN_TOKENIZED);
doc.Add(f1);
doc.Add(f2);
return doc;
}
void CreateTestIndex()
{
Lucene.Net.Index.IndexWriter wr = new Lucene.Net.Index.IndexWriter("blabla", new Lucene.Net.Analysis.Standard.StandardAnalyzer(), true);
for (int i = 0; i < 500; i++)
{
wr.AddDocument(GetDoc());
}
wr.Close();
}
int Count = 0;
public void TestMemoryLeakage()
{
for (int i = 0; i < 250; i++)
{
Lucene.Net.Search.IndexSearcher s = new Lucene.Net.Search.IndexSearcher("blabla");
Lucene.Net.QueryParsers.QueryParser qp = new Lucene.Net.QueryParsers.QueryParser("field1", new Lucene.Net.Analysis.Standard.StandardAnalyzer());
Lucene.Net.Search.Query q = qp.Parse("value1");
Lucene.Net.Search.Sort sort = new Lucene.Net.Search.Sort("field2");
Lucene.Net.Search.Hits hits = s.Search(q, null, sort);
s.Close();
}
GC.Collect();
Count++;
richTextBox1.AppendText(Count.ToString("00") + ") MemUsage: " + (System.Diagnostics.Process.GetCurrentProcess().WorkingSet / 1024).ToString() + " KB\n");
}
private void button2_Click(object sender, EventArgs e)
{
for (int i = 0; i < 10; i++)
{
TestMemoryLeakage();
Application.DoEvents();
}
}
}
}
{code}
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, WeakHashTable+FieldCacheImpl.rar, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12669733#action_12669733 ]
Digy commented on LUCENENET-106:
--------------------------------
Hi Eyal,
I don't remember what I have changed in my test configuration, but "Test_Weak_2" and "Test_Weak_ForEach" have started to fail.
I think It is related with an optimization of the compiler.
For ex., at line 377 (TestWeakHashTable) , GC collects key2 and BitObject(2) since there are no references to them.
What do you think? Another test case?
DIGY
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Assignee: Digy
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable v2.patch, WeakHashTable v2.patch, WeakHashTable+FieldCacheImpl.rar, WeakHashTable_ep.zip, WeakHashTable_ep_v2.zip, WeakHashTable_ep_v3.zip, WeakHashTable_tj.zip, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "George Aroush (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12659017#action_12659017 ]
George Aroush commented on LUCENENET-106:
-----------------------------------------
Hi DIGY,
I reviewed "Paches for v2.3.1.rar", and I have the following suggestion to make:
1) CachingSpanFilter.cs, and CachingWrapperFilter.cs -- remove the line: "//cache = new System.Collections.Hashtable();"
2) FieldCacheImpl.cs -- remove the line: "//private System.Collections.IDictionary readerCache = new System.Collections.Hashtable();"
3) FieldCacheImpl.cs -- remove the method "Close(IndexReader reader)", it's no longer need (doesn't exists in the Java ver.); it was added to the C# version in 2.0 to eliminate this memory leak issue regarding hash-n-sort; look in the HISTORY.txt file under 27Nov06 for the details.
4) WeakHashTable.cs -- move this code to SupportClass.cs and then you don't have to commit a new file -- this file -- to SVN; just add a new class, WeakHashTable in SupportClass.cs. The reason why I'm making this comment is to keep the port 1-to-1 with the Java version.
5) Form1.cs -- do not commit this file to SVN.
If you agree with my review and comments, please go ahead and commit with my suggested changes.
Regards,
-- George
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable+FieldCacheImpl.rar, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Digy updated LUCENENET-106:
---------------------------
Attachment: Digy.rar
Here is a test case and another patch (main idea is from Anton).
The difference is that I used a delegate instead of polling the IndexReader.
DIGY
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Priority: Critical
> Attachments: Digy.rar, luceneSrc_memUsage.patch
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Digy updated LUCENENET-106:
---------------------------
Attachment: WeakHashTable+FieldCacheImpl.rar
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, WeakHashTable+FieldCacheImpl.rar, WeakHashTable+FieldCacheImpl.rar, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12669763#action_12669763 ]
Digy commented on LUCENENET-106:
--------------------------------
Hi Eyal,
For ex,
Adding
Console.WriteLine(key!=null ? key.ToString() : "NULL");
Console.WriteLine(key2 != null ? key2.ToString() : "NULL");
Console.WriteLine(key3!= null ? key3.ToString() : "NULL");
to Test_Weak_2
and
Adding
Console.WriteLine(keys1 != null ? keys1.Length.ToString() : "NULL");
Console.WriteLine(keys2 != null ? keys2.Length.ToString() : "NULL");
to Test_Weak_ForEach
solves the problem.
DIGY
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Assignee: Digy
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable v2.patch, WeakHashTable v2.patch, WeakHashTable+FieldCacheImpl.rar, WeakHashTable_ep.zip, WeakHashTable_ep_v2.zip, WeakHashTable_ep_v3.zip, WeakHashTable_tj.zip, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Digy updated LUCENENET-106:
---------------------------
Attachment: WeakHashTable+FieldCacheImpl.rar
Erich's solution is fine.
To be able to use the WeakHashTable codes freely, I wrote another one and attached the apache licence.
(FieldCacheImpl patch is also included in "WeakHashTable+FieldCacheImpl.rar")
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, WeakHashTable+FieldCacheImpl.rar, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (LUCENENET-106) Lucene.NET (Revision:
603121) is leaking memory
Posted by "Marco Dissel (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558556#action_12558556 ]
mdissel edited comment on LUCENENET-106 at 1/14/08 1:13 PM:
-----------------------------------------------------------------
anyone hunting this bug ? after applying both fixes i still get a big memory leak when doing custom sorting, but i can't find the cause of this leak...
was (Author: mdissel):
When can we expect these fixes in the trunk?
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, WeakHashTable+FieldCacheImpl.rar, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12665231#action_12665231 ]
Digy commented on LUCENENET-106:
--------------------------------
Hi Eyal,
To simplify the code, I'd suggest another variant of your code: Instead of implementing whole IDictionary, inheriting from Hashtable and overriding necessary methods only.
A sample code is below. The only method which is different is "GetEnumerator"
{code}
public class WeakHashTable : Hashtable
{
class WeakComparer : IEqualityComparer
{
static WeakComparer _instance;
static WeakComparer()
{
_instance = new WeakComparer();
}
bool IEqualityComparer.Equals(object x, object y)
{
if (x == y)
return true;
if (x is WeakKey)
{
x = ((WeakKey)x).Target;
if (x == null)
return false;
}
if (y is WeakKey)
{
y = ((WeakKey)y).Target;
if (y == null)
return false;
}
return x.Equals(y);
}
int IEqualityComparer.GetHashCode(object obj)
{
return obj.GetHashCode();
}
public static WeakComparer Instance
{
get
{
return _instance;
}
}
}
class WeakKey : WeakReference
{
int hashCode;
public WeakKey(object key)
: base(key)
{
if (key == null)
throw new ArgumentNullException("key");
hashCode = key.GetHashCode();
}
public override int GetHashCode()
{
return hashCode;
}
}
WeakReference collectableObject = new WeakReference(new Object());
public WeakHashTable() : base(WeakComparer.Instance)
{
}
private void CleanIfNeeded()
{
if (collectableObject.Target == null)
{
Clean();
collectableObject = new WeakReference(new Object());
}
}
private void Clean()
{
ArrayList keysToDelete = new ArrayList();
foreach (WeakKey wtk in base.Keys)
{
if (!wtk.IsAlive)
{
keysToDelete.Add(wtk);
}
}
foreach (WeakKey wtk in keysToDelete)
base.Remove(wtk);
}
public override void Add(object key, object value)
{
CleanIfNeeded();
base.Add(new WeakKey(key), value);
}
public override IDictionaryEnumerator GetEnumerator()
{
System.Collections.Hashtable dictEntires = new Hashtable();
foreach (WeakKey key in base.Keys)
{
object realKey = key.Target;
if (realKey != null)
dictEntires.Add( realKey,base[realKey] );
}
return dictEntires.GetEnumerator();
}
public override ICollection Keys
{
get
{
ArrayList keys = new ArrayList(Count);
foreach (WeakKey key in base.Keys)
{
object realKey = key.Target;
if (realKey != null)
keys.Add(realKey);
}
return keys;
}
}
public override object this[object key]
{
get
{
return base[key];
}
set
{
CleanIfNeeded();
base[new WeakKey(key)] = value;
}
}
public override void CopyTo(Array array, int index)
{
int arrayIndex = index;
foreach (object key in Keys)
{
DictionaryEntry de = new DictionaryEntry(key, this[key]);
array.SetValue(de, arrayIndex++);
}
}
public override int Count
{
get
{
CleanIfNeeded();
return base.Count;
}
}
}
{code}
DIGY
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Assignee: Digy
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable v2.patch, WeakHashTable v2.patch, WeakHashTable+FieldCacheImpl.rar, WeakHashTable_ep.zip, WeakHashTable_tj.zip, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "George Aroush (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12608734#action_12608734 ]
George Aroush commented on LUCENENET-106:
-----------------------------------------
What about Lucene.Net 2.0, 1.9.1, and prior? Do they too have this memory leak?
There was a memory leak which was fixed with this LUCENENET-29.
Can you run your test on 2.0 or 1.9.1?
Thanks.
-- George
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable+FieldCacheImpl.rar, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "eyal post (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12665233#action_12665233 ]
eyal post commented on LUCENENET-106:
-------------------------------------
A very elegant solution Digy! Let's go with that.
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Assignee: Digy
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable v2.patch, WeakHashTable v2.patch, WeakHashTable+FieldCacheImpl.rar, WeakHashTable_ep.zip, WeakHashTable_tj.zip, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (LUCENENET-106) Lucene.NET (Revision:
603121) is leaking memory
Posted by "eyal post (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12665719#action_12665719 ]
eyalp edited comment on LUCENENET-106 at 1/20/09 10:24 PM:
---------------------------------------------------------------
The fix for Digy's comment is very simple using a while loop:
public bool MoveNext()
{
while (baseEnumerator.MoveNext())
{
object key = ((WeakKey)baseEnumerator.Key).Target;
if (key != null)
{
this.currentKey = key;
this.currentValue = baseEnumerator.Value;
return true;
}
}
return false;
}
The problem with relying on Keys for enumeration is that calling Keys creates a temporary
copy of all keys (which is slow).
I'm adding comments to my last version and then i'll post it for inclusion.
was (Author: eyalp):
The fix for Digy's comment is very simple using a while loop:
public bool MoveNext()
{
while (baseEnumerator.MoveNext())
{
object key = ((WeakKey)baseEnumerator.Key).Target;
if (key != null)
{
this.currentKey = key;
this.currentValue = baseEnumerator.Value;
return true;
}
}
return false;
}
The problem with relying on Keys for enumeration is that calling Keys creates a temporary copy of all keys (which is slow).
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Assignee: Digy
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable v2.patch, WeakHashTable v2.patch, WeakHashTable+FieldCacheImpl.rar, WeakHashTable_ep.zip, WeakHashTable_ep_v2.zip, WeakHashTable_tj.zip, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Digy updated LUCENENET-106:
---------------------------
Attachment: Paches for v2.3.1.rar
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable+FieldCacheImpl.rar, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12557325#action_12557325 ]
Digy commented on LUCENENET-106:
--------------------------------
I think, we all talk about the same bug and replacing the Hashtable with a WeakHashTable in line
" innerCache = new System.Collections.Hashtable() "
solves it( at least for the test case in Digy.rar)
DIGY
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, WeakHashTable+FieldCacheImpl.rar, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12665278#action_12665278 ]
Digy commented on LUCENENET-106:
--------------------------------
Recursive "WeakDictionaryEnumerator.MoveNext" can be costly in some rare cases, I would prefer a loop there.
Anyway, This code passes my all tests and has also the best performance.
Thanks Eyal.
PS: I don't know much about licensing issues, but I think, it would be good to add a apache license to your code.
DIGY.
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Assignee: Digy
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable v2.patch, WeakHashTable v2.patch, WeakHashTable+FieldCacheImpl.rar, WeakHashTable_ep.zip, WeakHashTable_ep_v2.zip, WeakHashTable_tj.zip, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "Marco Dissel (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12557244#action_12557244 ]
Marco Dissel commented on LUCENENET-106:
----------------------------------------
I got another serious memory leak. This is the result from DebugDiag tool. The attached patched doesn't solve this one.. Anyone solved this already?
[DEFAULT] [hasThis] Object Lucene.Net.Search.FieldCacheImpl/AnonymousClassCache3.CreateValue(Class Lucene.Net.Index.IndexReader,Object) 0xC920F4
[DEFAULT] [hasThis] Object Lucene.Net.Search.FieldCacheImpl/Cache.Get(Class Lucene.Net.Index.IndexReader,Object)
[DEFAULT] [hasThis] Class Lucene.Net.Search.StringIndex Lucene.Net.Search.FieldCacheImpl.GetStringIndex(Class Lucene.Net.Index.IndexReader,String)
[DEFAULT] [hasThis] Object Lucene.Net.Search.FieldCacheImpl/AnonymousClassCache4.CreateValue(Class Lucene.Net.Index.IndexReader,Object)
[DEFAULT] [hasThis] Object Lucene.Net.Search.FieldCacheImpl/Cache.Get(Class Lucene.Net.Index.IndexReader,Object)
[DEFAULT] [hasThis] Object Lucene.Net.Search.FieldCacheImpl.GetAuto(Class Lucene.Net.Index.IndexReader,String)
[DEFAULT] Class Lucene.Net.Search.ScoreDocComparator Lucene.Net.Search.FieldSortedHitQueue.ComparatorAuto(Class Lucene.Net.Index.IndexReader,String)
[DEFAULT] [hasThis] Object Lucene.Net.Search.FieldSortedHitQueue/AnonymousClassCache.CreateValue(Class Lucene.Net.Index.IndexReader,Object) [DEFAULT] Class Lucene.Net.Search.ScoreDocComparator Lucene.Net.Search.FieldSortedHitQueue.ComparatorAuto(Class Lucene.Net.Index.IndexReader,String)
[DEFAULT] [hasThis] Object Lucene.Net.Search.FieldCacheImpl/Cache.Get(Class Lucene.Net.Index.IndexReader,Object)
[DEFAULT] Class Lucene.Net.Search.ScoreDocComparator Lucene.Net.Search.FieldSortedHitQueue.GetCachedComparator(Class Lucene.Net.Index.IndexReader,String,I4,Class System.Globalization.CultureInfo,Class Lucene.Net.Search.SortComparatorSource)
[DEFAULT] [hasThis] Void Lucene.Net.Search.FieldSortedHitQueue..ctor(Class Lucene.Net.Index.IndexReader,SZArray Class Lucene.Net.Search.SortField,I4) [DEFAULT] Class Lucene.Net.Search.ScoreDocComparator Lucene.Net.Search.FieldSortedHitQueue.GetCachedComparator(Class Lucene.Net.Index.IndexReader,String,I4,Class System.Globalization.CultureInfo,Class Lucene.Net.Search.SortComparatorSource)
[DEFAULT] [hasThis] Class Lucene.Net.Search.TopFieldDocs Lucene.Net.Search.IndexSearcher.Search(Class Lucene.Net.Search.Weight,Class Lucene.Net.Search.Filter,I4,Class Lucene.Net.Search.Sort) [DEFAULT] [hasThis] Void Lucene.Net.Search.FieldSortedHitQueue..ctor(Class Lucene.Net.Index.IndexReader,SZArray Class Lucene.Net.Search.SortField,I4)
[DEFAULT] [hasThis] Class Lucene.Net.Search.TopFieldDocs Lucene.Net.Search.Searcher.Search(Class Lucene.Net.Search.Query,Class Lucene.Net.Search.Filter,I4,Class Lucene.Net.Search.Sort)
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, WeakHashTable+FieldCacheImpl.rar, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558801#action_12558801 ]
Digy commented on LUCENENET-106:
--------------------------------
Hi Marco,
With the patch "WeakHashTable+FieldCacheImpl.rar", the test case in Digy.rar(Form1.cs) must have been fixed.
If not, Can you send a sample code leaking?
DIGY
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, WeakHashTable+FieldCacheImpl.rar, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Closed: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Digy closed LUCENENET-106.
--------------------------
Resolution: Fixed
Patch for WeakHashTable and its test case in WeakHashTable_ep_v3.zip committed.
Including: Test_Weak_2 and Test_Work_ForEach corrections.
DIGY
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Assignee: Digy
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable v2.patch, WeakHashTable v2.patch, WeakHashTable+FieldCacheImpl.rar, WeakHashTable_ep.zip, WeakHashTable_ep_v2.zip, WeakHashTable_ep_v3.zip, WeakHashTable_tj.zip, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12555407#action_12555407 ]
Digy commented on LUCENENET-106:
--------------------------------
Hi Ross & Anton
Can you send a patch for other Lucene.Net users?
DIGY
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Priority: Critical
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12608768#action_12608768 ]
Digy commented on LUCENENET-106:
--------------------------------
01) MemUsage: 36460 KB
02) MemUsage: 36768 KB
03) MemUsage: 36840 KB
04) MemUsage: 36736 KB
05) MemUsage: 36740 KB
06) MemUsage: 36740 KB
07) MemUsage: 36744 KB
08) MemUsage: 36744 KB
09) MemUsage: 36744 KB
10) MemUsage: 36744 KB
There is no memory leakage in Lucene.Net 2.0.
But there is a change in logic bettween 2.0 and 2.1. For ex, there is no class "internal abstract class Cache" in 2.0 , instead it uses a HashTable named "cache". Also AnonymousClassCache1 ... AnonymousClassCache7 do no exist in Lucene.Net 2.0
or
{code}
//2.1
public virtual void Close(IndexReader reader)
{
}
{code}
{code}
//2.0
public virtual void Close(IndexReader reader)
{
lock (this)
{
System.Collections.Hashtable readerCache = (System.Collections.Hashtable) cache[reader];
if (readerCache != null)
{
readerCache.Clear();
readerCache = null;
}
cache.Remove(reader);
}
}
{code}
So, lot of changes are done with 2.1
DIGY
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable+FieldCacheImpl.rar, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.