You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucenenet.apache.org by "Tim Januario (JIRA)" <ji...@apache.org> on 2009/01/09 18:15:59 UTC
[jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is
leaking memory
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662433#action_12662433 ]
Tim Januario commented on LUCENENET-106:
----------------------------------------
Digy,
1) The WeakHashTable.Exists method creates a possible NullReferenceException because you used WeakEntries[i] As WeakEntry instead of (WeakEntry)WeakEntries[i]. This is only a warning but...
2) The Java WeakHashMap does not throw an execption when put() is called with a key that already exists in the table. It simply replaces the old value with the new value. .NET Hashtable does the same. The Add method of WeakHashTable should remove the WeakEntry if the key is found and replace it with the new value in order to stay in line instead of throwing the Exception.
3) WeakHashTableEnumarator should be WeakHashTableEnumerator
4) WeakEntry.HashCode is not used and should be removed.
5) WeakHashTable.InitialMem is not used and should be removed.
Other than that, I think this looks good.
-tim
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Assignee: Digy
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable v2.patch, WeakHashTable+FieldCacheImpl.rar, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
RE: [jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is leaking memory
Posted by Digy <di...@gmail.com>.
It does, in fact, use a HashTable table internally. ArrayLists are used to
store WeakEntries whose keys' HashCodes are the same. Since the probability
of a collision is very low, ArrayLists contain most of the time only one
WeakEntry.
* HashTable: +------> WeakEntry
* +----------|--------------------------+ |
* | HashCode | WeakEntries (ArrayList) |----+------> WeakEntry
* +----------+--------------------------+ |
* | HashCode | WeakEntries (ArrayList) | +------> WeakEntry
* +----------+--------------------------+
DIGY
-----Original Message-----
From: Eyal Post [mailto:eyalpost@epocalipse.com]
Sent: Sunday, January 11, 2009 1:08 PM
To: lucene-net-dev@incubator.apache.org
Subject: RE: [jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121)
is leaking memory
As Luc noted in the first comment to this patch, I think that if this class
is to be called WeakHashTable (and therefor imply it is a general purpose
weak hash table) then it should be implemented as a real hash table.
Therefor, you should not use an array list and make linear comparisons.
Instead WeakHashTable should internally use a hashtable. The first
implementation was in the right direction, the only fix should have been to
use WeakEntry as a key to the table and not the hash code.
Eyal
> -----Original Message-----
> From: Digy [mailto:digydigy@gmail.com]
> Sent: Friday, January 09, 2009 20:03 PM
> To: lucene-net-dev@incubator.apache.org
> Subject: RE: [jira] Commented: (LUCENENET-106) Lucene.NET
> (Revision: 603121) is leaking memory
>
> Hi Tim,
>
>
>
> Thanks for the comments. I will correct the 3,4,5.
>
>
>
> 1) Since ArrayList contains WeakEntries & sync.ed for
> multiple threads, they can not be null and can not be any
> other type. I can not find a bug here. Maybe you can send a
> more detailed description.
>
> 2), WeakHashTable is compatible with "HashTable" in .Net
> world. "Ht.Add(key,value)" throws an exception for duplicate keys but
>
> "Ht[key]=value" just adds or replaces.
>
>
>
>
>
> DIGY
>
>
>
>
>
>
>
> -----Original Message-----
> From: Tim Januario (JIRA) [mailto:jira@apache.org]
> Sent: Friday, January 09, 2009 7:16 PM
> To: lucene-net-dev@incubator.apache.org
> Subject: [jira] Commented: (LUCENENET-106) Lucene.NET
> (Revision: 603121) is leaking memory
>
>
>
>
>
> [
> https://issues.apache.org/jira/browse/LUCENENET-106?page=com.a
tlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId
=12662433#action_12662433 ]
Tim Januario commented on LUCENENET-106:
----------------------------------------
Digy,
1) The WeakHashTable.Exists method creates a possible NullReferenceException
because you used WeakEntries[i] As WeakEntry instead of
(WeakEntry)WeakEntries[i]. This is only a warning but...
2) The Java WeakHashMap does not throw an execption when put() is called
with a key that already exists in the table. It simply replaces the old
value with the new value. .NET Hashtable does the same. The Add method of
WeakHashTable should remove the WeakEntry if the key is found and replace it
with the new value in order to stay in line instead of throwing the
Exception.
3) WeakHashTableEnumarator should be WeakHashTableEnumerator
4) WeakEntry.HashCode is not used and should be removed.
5) WeakHashTable.InitialMem is not used and should be removed.
Other than that, I think this looks good.
-tim
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL:
> https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Assignee: Digy
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar,
> luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable
> v2.patch, WeakHashTable+FieldCacheImpl.rar, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some
hash items that have closed IndexReader object as a key. So a lot of Term
instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this
problem.
> This bug can be reproduced only when Sort functionality used during
search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is leaking memory
Posted by TJ Kolev <tj...@gmail.com>.
Hi guys,
I just came across this thread looking for something else. The version
I added in the Jira today works on exactly the idea being discussed
here.
tjk :)
On Mon, Jan 12, 2009 at 7:21 AM, digy digy <di...@gmail.com> wrote:
> OK. I see. It can be used in WeakHashTable v3.
>
> DIGY
>
>
> On Mon, Jan 12, 2009 at 11:28 AM, Eyal Post <ey...@epocalipse.com> wrote:
>
>> void Add(object keyX,object valueX)
>> {
>> InternalTable.Add( new WeakEntry(keyX), value);
>> }
>>
>> Object Get(object keyX)
>> {
>> return InternalTable[new WeakEntry(keyX)];
>> }
>>
>>
>> WeakEntry.GetHashCode = key.getHashCode;
>> WeakEntry.Equals = keyA == keyB
>>
>> Not sure this is clear enough - I can post full implementation if you'd
>> like
>> but that'll take me some more time.
>>
>> Eyal
>>
>>
>> > -----Original Message-----
>> > From: Digy [mailto:digydigy@gmail.com]
>> > Sent: Sunday, January 11, 2009 23:16 PM
>> > To: lucene-net-dev@incubator.apache.org
>> > Subject: RE: [jira] Commented: (LUCENENET-106) Lucene.NET
>> > (Revision: 603121) is leaking memory
>> > Importance: High
>> >
>> > Hi Eyal,
>> >
>> > I couldn't find a simple way to use the WeakEntry as a key
>> > and then access its value without enumerating all the keys in
>> > InternalTable.
>> > I think you suggest
>> >
>> > void Add(object keyX,object valueX)
>> > {
>> > InternalTable.Add( new WeakEntry(keyX,valueX), value);
>> > } // or something like that.
>> >
>> > Then how do you think that we get the value of "Get(keyX)" ?
>> > Enumerating all the keys of InternalTable and comparing the
>> > "weakEntry.Key" with "keyX"?
>> > When thinking of successive "Add"s, this has the complexity of O(n*n).
>> >
>> > Can you share your solution?
>> >
>> > DIGY
>> >
>> >
>> >
>> > -----Original Message-----
>> > From: Eyal Post [mailto:eyalpost@epocalipse.com]
>> > Sent: Sunday, January 11, 2009 1:08 PM
>> > To: lucene-net-dev@incubator.apache.org
>> > Subject: RE: [jira] Commented: (LUCENENET-106) Lucene.NET
>> > (Revision: 603121) is leaking memory
>> >
>> > As Luc noted in the first comment to this patch, I think that
>> > if this class is to be called WeakHashTable (and therefor
>> > imply it is a general purpose weak hash table) then it should
>> > be implemented as a real hash table.
>> > Therefor, you should not use an array list and make linear
>> > comparisons.
>> > Instead WeakHashTable should internally use a hashtable. The
>> > first implementation was in the right direction, the only fix
>> > should have been to use WeakEntry as a key to the table and
>> > not the hash code.
>> >
>> > Eyal
>> >
>> >
>> > > -----Original Message-----
>> > > From: Digy [mailto:digydigy@gmail.com]
>> > > Sent: Friday, January 09, 2009 20:03 PM
>> > > To: lucene-net-dev@incubator.apache.org
>> > > Subject: RE: [jira] Commented: (LUCENENET-106) Lucene.NET
>> > > (Revision: 603121) is leaking memory
>> > >
>> > > Hi Tim,
>> > >
>> > >
>> > >
>> > > Thanks for the comments. I will correct the 3,4,5.
>> > >
>> > >
>> > >
>> > > 1) Since ArrayList contains WeakEntries & sync.ed for multiple
>> > > threads, they can not be null and can not be any other
>> > type. I can not
>> > > find a bug here. Maybe you can send a more detailed description.
>> > >
>> > > 2), WeakHashTable is compatible with "HashTable" in .Net world.
>> > > "Ht.Add(key,value)" throws an exception for duplicate keys but
>> > >
>> > > "Ht[key]=value" just adds or replaces.
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > DIGY
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > -----Original Message-----
>> > > From: Tim Januario (JIRA) [mailto:jira@apache.org]
>> > > Sent: Friday, January 09, 2009 7:16 PM
>> > > To: lucene-net-dev@incubator.apache.org
>> > > Subject: [jira] Commented: (LUCENENET-106) Lucene.NET
>> > > (Revision: 603121) is leaking memory
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > [
>> > > https://issues.apache.org/jira/browse/LUCENENET-106?page=com.a
>> > tlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&fo
>> > cusedCommentId
>> > =12662433#action_12662433 ]
>> >
>> >
>> >
>> > Tim Januario commented on LUCENENET-106:
>> >
>> > ----------------------------------------
>> >
>> >
>> >
>> > Digy,
>> >
>> > 1) The WeakHashTable.Exists method creates a possible
>> > NullReferenceException because you used WeakEntries[i] As
>> > WeakEntry instead of (WeakEntry)WeakEntries[i]. This is only
>> > a warning but...
>> >
>> > 2) The Java WeakHashMap does not throw an execption when
>> > put() is called with a key that already exists in the table.
>> > It simply replaces the old value with the new value. .NET
>> > Hashtable does the same. The Add method of WeakHashTable
>> > should remove the WeakEntry if the key is found and replace
>> > it with the new value in order to stay in line instead of
>> > throwing the Exception.
>> >
>> > 3) WeakHashTableEnumarator should be WeakHashTableEnumerator
>> >
>> > 4) WeakEntry.HashCode is not used and should be removed.
>> >
>> > 5) WeakHashTable.InitialMem is not used and should be removed.
>> >
>> >
>> >
>> > Other than that, I think this looks good.
>> >
>> >
>> >
>> > -tim
>> >
>> >
>> >
>> > > Lucene.NET (Revision: 603121) is leaking memory
>> >
>> > > -----------------------------------------------
>> >
>> > >
>> >
>> > > Key: LUCENENET-106
>> >
>> > > URL:
>> > > https://issues.apache.org/jira/browse/LUCENENET-106
>> >
>> > > Project: Lucene.Net
>> >
>> > > Issue Type: Bug
>> >
>> > > Environment: .NET 2.0
>> >
>> > > Reporter: Anton K.
>> >
>> > > Assignee: Digy
>> >
>> > > Priority: Critical
>> >
>> > > Attachments: DIGY-FieldCacheImpl.patch, Digy.rar,
>> > > luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable
>> > > v2.patch, WeakHashTable+FieldCacheImpl.rar, WeakReferences.rar
>> >
>> > >
>> >
>> > >
>> >
>> > > readerCache Hashtable field (see FieldCacheImpl.cs) never releases
>> > > some
>> > hash items that have closed IndexReader object as a key. So a
>> > lot of Term instances are never released.
>> >
>> > > Java version of Lucene uses WeakHashMap and therefore doesn't have
>> > > this
>> > problem.
>> >
>> > > This bug can be reproduced only when Sort functionality used during
>> > search.
>> >
>> > > See following link for additional information.
>> >
>> > > http://www.gossamer-threads.com/lists/lucene/java-user/55681
>> >
>> > > Steps to reproduce:
>> >
>> > > 1)Create index
>> >
>> > > 2) Modify index by IndexWiter; Close IndexWriter
>> >
>> > > 3) Use IndexSearcher for searching with Sort; Close InexSearcher
>> >
>> > > 4) Go to step 2
>> >
>> > > You'll get OutOfMemoryException after some time of running
>> > this algorithm.
>> >
>> >
>> >
>> > --
>> >
>> > This message is automatically generated by JIRA.
>> >
>> > -
>> >
>> > You can reply to this email to add a comment to the issue online.
>> >
>> >
>>
>>
>
Re: [jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is leaking memory
Posted by digy digy <di...@gmail.com>.
OK. I see. It can be used in WeakHashTable v3.
DIGY
On Mon, Jan 12, 2009 at 11:28 AM, Eyal Post <ey...@epocalipse.com> wrote:
> void Add(object keyX,object valueX)
> {
> InternalTable.Add( new WeakEntry(keyX), value);
> }
>
> Object Get(object keyX)
> {
> return InternalTable[new WeakEntry(keyX)];
> }
>
>
> WeakEntry.GetHashCode = key.getHashCode;
> WeakEntry.Equals = keyA == keyB
>
> Not sure this is clear enough - I can post full implementation if you'd
> like
> but that'll take me some more time.
>
> Eyal
>
>
> > -----Original Message-----
> > From: Digy [mailto:digydigy@gmail.com]
> > Sent: Sunday, January 11, 2009 23:16 PM
> > To: lucene-net-dev@incubator.apache.org
> > Subject: RE: [jira] Commented: (LUCENENET-106) Lucene.NET
> > (Revision: 603121) is leaking memory
> > Importance: High
> >
> > Hi Eyal,
> >
> > I couldn't find a simple way to use the WeakEntry as a key
> > and then access its value without enumerating all the keys in
> > InternalTable.
> > I think you suggest
> >
> > void Add(object keyX,object valueX)
> > {
> > InternalTable.Add( new WeakEntry(keyX,valueX), value);
> > } // or something like that.
> >
> > Then how do you think that we get the value of "Get(keyX)" ?
> > Enumerating all the keys of InternalTable and comparing the
> > "weakEntry.Key" with "keyX"?
> > When thinking of successive "Add"s, this has the complexity of O(n*n).
> >
> > Can you share your solution?
> >
> > DIGY
> >
> >
> >
> > -----Original Message-----
> > From: Eyal Post [mailto:eyalpost@epocalipse.com]
> > Sent: Sunday, January 11, 2009 1:08 PM
> > To: lucene-net-dev@incubator.apache.org
> > Subject: RE: [jira] Commented: (LUCENENET-106) Lucene.NET
> > (Revision: 603121) is leaking memory
> >
> > As Luc noted in the first comment to this patch, I think that
> > if this class is to be called WeakHashTable (and therefor
> > imply it is a general purpose weak hash table) then it should
> > be implemented as a real hash table.
> > Therefor, you should not use an array list and make linear
> > comparisons.
> > Instead WeakHashTable should internally use a hashtable. The
> > first implementation was in the right direction, the only fix
> > should have been to use WeakEntry as a key to the table and
> > not the hash code.
> >
> > Eyal
> >
> >
> > > -----Original Message-----
> > > From: Digy [mailto:digydigy@gmail.com]
> > > Sent: Friday, January 09, 2009 20:03 PM
> > > To: lucene-net-dev@incubator.apache.org
> > > Subject: RE: [jira] Commented: (LUCENENET-106) Lucene.NET
> > > (Revision: 603121) is leaking memory
> > >
> > > Hi Tim,
> > >
> > >
> > >
> > > Thanks for the comments. I will correct the 3,4,5.
> > >
> > >
> > >
> > > 1) Since ArrayList contains WeakEntries & sync.ed for multiple
> > > threads, they can not be null and can not be any other
> > type. I can not
> > > find a bug here. Maybe you can send a more detailed description.
> > >
> > > 2), WeakHashTable is compatible with "HashTable" in .Net world.
> > > "Ht.Add(key,value)" throws an exception for duplicate keys but
> > >
> > > "Ht[key]=value" just adds or replaces.
> > >
> > >
> > >
> > >
> > >
> > > DIGY
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Tim Januario (JIRA) [mailto:jira@apache.org]
> > > Sent: Friday, January 09, 2009 7:16 PM
> > > To: lucene-net-dev@incubator.apache.org
> > > Subject: [jira] Commented: (LUCENENET-106) Lucene.NET
> > > (Revision: 603121) is leaking memory
> > >
> > >
> > >
> > >
> > >
> > > [
> > > https://issues.apache.org/jira/browse/LUCENENET-106?page=com.a
> > tlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&fo
> > cusedCommentId
> > =12662433#action_12662433 ]
> >
> >
> >
> > Tim Januario commented on LUCENENET-106:
> >
> > ----------------------------------------
> >
> >
> >
> > Digy,
> >
> > 1) The WeakHashTable.Exists method creates a possible
> > NullReferenceException because you used WeakEntries[i] As
> > WeakEntry instead of (WeakEntry)WeakEntries[i]. This is only
> > a warning but...
> >
> > 2) The Java WeakHashMap does not throw an execption when
> > put() is called with a key that already exists in the table.
> > It simply replaces the old value with the new value. .NET
> > Hashtable does the same. The Add method of WeakHashTable
> > should remove the WeakEntry if the key is found and replace
> > it with the new value in order to stay in line instead of
> > throwing the Exception.
> >
> > 3) WeakHashTableEnumarator should be WeakHashTableEnumerator
> >
> > 4) WeakEntry.HashCode is not used and should be removed.
> >
> > 5) WeakHashTable.InitialMem is not used and should be removed.
> >
> >
> >
> > Other than that, I think this looks good.
> >
> >
> >
> > -tim
> >
> >
> >
> > > Lucene.NET (Revision: 603121) is leaking memory
> >
> > > -----------------------------------------------
> >
> > >
> >
> > > Key: LUCENENET-106
> >
> > > URL:
> > > https://issues.apache.org/jira/browse/LUCENENET-106
> >
> > > Project: Lucene.Net
> >
> > > Issue Type: Bug
> >
> > > Environment: .NET 2.0
> >
> > > Reporter: Anton K.
> >
> > > Assignee: Digy
> >
> > > Priority: Critical
> >
> > > Attachments: DIGY-FieldCacheImpl.patch, Digy.rar,
> > > luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable
> > > v2.patch, WeakHashTable+FieldCacheImpl.rar, WeakReferences.rar
> >
> > >
> >
> > >
> >
> > > readerCache Hashtable field (see FieldCacheImpl.cs) never releases
> > > some
> > hash items that have closed IndexReader object as a key. So a
> > lot of Term instances are never released.
> >
> > > Java version of Lucene uses WeakHashMap and therefore doesn't have
> > > this
> > problem.
> >
> > > This bug can be reproduced only when Sort functionality used during
> > search.
> >
> > > See following link for additional information.
> >
> > > http://www.gossamer-threads.com/lists/lucene/java-user/55681
> >
> > > Steps to reproduce:
> >
> > > 1)Create index
> >
> > > 2) Modify index by IndexWiter; Close IndexWriter
> >
> > > 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> >
> > > 4) Go to step 2
> >
> > > You'll get OutOfMemoryException after some time of running
> > this algorithm.
> >
> >
> >
> > --
> >
> > This message is automatically generated by JIRA.
> >
> > -
> >
> > You can reply to this email to add a comment to the issue online.
> >
> >
>
>
RE: [jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is leaking memory
Posted by Eyal Post <ey...@epocalipse.com>.
void Add(object keyX,object valueX)
{
InternalTable.Add( new WeakEntry(keyX), value);
}
Object Get(object keyX)
{
return InternalTable[new WeakEntry(keyX)];
}
WeakEntry.GetHashCode = key.getHashCode;
WeakEntry.Equals = keyA == keyB
Not sure this is clear enough - I can post full implementation if you'd like
but that'll take me some more time.
Eyal
> -----Original Message-----
> From: Digy [mailto:digydigy@gmail.com]
> Sent: Sunday, January 11, 2009 23:16 PM
> To: lucene-net-dev@incubator.apache.org
> Subject: RE: [jira] Commented: (LUCENENET-106) Lucene.NET
> (Revision: 603121) is leaking memory
> Importance: High
>
> Hi Eyal,
>
> I couldn't find a simple way to use the WeakEntry as a key
> and then access its value without enumerating all the keys in
> InternalTable.
> I think you suggest
>
> void Add(object keyX,object valueX)
> {
> InternalTable.Add( new WeakEntry(keyX,valueX), value);
> } // or something like that.
>
> Then how do you think that we get the value of "Get(keyX)" ?
> Enumerating all the keys of InternalTable and comparing the
> "weakEntry.Key" with "keyX"?
> When thinking of successive "Add"s, this has the complexity of O(n*n).
>
> Can you share your solution?
>
> DIGY
>
>
>
> -----Original Message-----
> From: Eyal Post [mailto:eyalpost@epocalipse.com]
> Sent: Sunday, January 11, 2009 1:08 PM
> To: lucene-net-dev@incubator.apache.org
> Subject: RE: [jira] Commented: (LUCENENET-106) Lucene.NET
> (Revision: 603121) is leaking memory
>
> As Luc noted in the first comment to this patch, I think that
> if this class is to be called WeakHashTable (and therefor
> imply it is a general purpose weak hash table) then it should
> be implemented as a real hash table.
> Therefor, you should not use an array list and make linear
> comparisons.
> Instead WeakHashTable should internally use a hashtable. The
> first implementation was in the right direction, the only fix
> should have been to use WeakEntry as a key to the table and
> not the hash code.
>
> Eyal
>
>
> > -----Original Message-----
> > From: Digy [mailto:digydigy@gmail.com]
> > Sent: Friday, January 09, 2009 20:03 PM
> > To: lucene-net-dev@incubator.apache.org
> > Subject: RE: [jira] Commented: (LUCENENET-106) Lucene.NET
> > (Revision: 603121) is leaking memory
> >
> > Hi Tim,
> >
> >
> >
> > Thanks for the comments. I will correct the 3,4,5.
> >
> >
> >
> > 1) Since ArrayList contains WeakEntries & sync.ed for multiple
> > threads, they can not be null and can not be any other
> type. I can not
> > find a bug here. Maybe you can send a more detailed description.
> >
> > 2), WeakHashTable is compatible with "HashTable" in .Net world.
> > "Ht.Add(key,value)" throws an exception for duplicate keys but
> >
> > "Ht[key]=value" just adds or replaces.
> >
> >
> >
> >
> >
> > DIGY
> >
> >
> >
> >
> >
> >
> >
> > -----Original Message-----
> > From: Tim Januario (JIRA) [mailto:jira@apache.org]
> > Sent: Friday, January 09, 2009 7:16 PM
> > To: lucene-net-dev@incubator.apache.org
> > Subject: [jira] Commented: (LUCENENET-106) Lucene.NET
> > (Revision: 603121) is leaking memory
> >
> >
> >
> >
> >
> > [
> > https://issues.apache.org/jira/browse/LUCENENET-106?page=com.a
> tlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&fo
> cusedCommentId
> =12662433#action_12662433 ]
>
>
>
> Tim Januario commented on LUCENENET-106:
>
> ----------------------------------------
>
>
>
> Digy,
>
> 1) The WeakHashTable.Exists method creates a possible
> NullReferenceException because you used WeakEntries[i] As
> WeakEntry instead of (WeakEntry)WeakEntries[i]. This is only
> a warning but...
>
> 2) The Java WeakHashMap does not throw an execption when
> put() is called with a key that already exists in the table.
> It simply replaces the old value with the new value. .NET
> Hashtable does the same. The Add method of WeakHashTable
> should remove the WeakEntry if the key is found and replace
> it with the new value in order to stay in line instead of
> throwing the Exception.
>
> 3) WeakHashTableEnumarator should be WeakHashTableEnumerator
>
> 4) WeakEntry.HashCode is not used and should be removed.
>
> 5) WeakHashTable.InitialMem is not used and should be removed.
>
>
>
> Other than that, I think this looks good.
>
>
>
> -tim
>
>
>
> > Lucene.NET (Revision: 603121) is leaking memory
>
> > -----------------------------------------------
>
> >
>
> > Key: LUCENENET-106
>
> > URL:
> > https://issues.apache.org/jira/browse/LUCENENET-106
>
> > Project: Lucene.Net
>
> > Issue Type: Bug
>
> > Environment: .NET 2.0
>
> > Reporter: Anton K.
>
> > Assignee: Digy
>
> > Priority: Critical
>
> > Attachments: DIGY-FieldCacheImpl.patch, Digy.rar,
> > luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable
> > v2.patch, WeakHashTable+FieldCacheImpl.rar, WeakReferences.rar
>
> >
>
> >
>
> > readerCache Hashtable field (see FieldCacheImpl.cs) never releases
> > some
> hash items that have closed IndexReader object as a key. So a
> lot of Term instances are never released.
>
> > Java version of Lucene uses WeakHashMap and therefore doesn't have
> > this
> problem.
>
> > This bug can be reproduced only when Sort functionality used during
> search.
>
> > See following link for additional information.
>
> > http://www.gossamer-threads.com/lists/lucene/java-user/55681
>
> > Steps to reproduce:
>
> > 1)Create index
>
> > 2) Modify index by IndexWiter; Close IndexWriter
>
> > 3) Use IndexSearcher for searching with Sort; Close InexSearcher
>
> > 4) Go to step 2
>
> > You'll get OutOfMemoryException after some time of running
> this algorithm.
>
>
>
> --
>
> This message is automatically generated by JIRA.
>
> -
>
> You can reply to this email to add a comment to the issue online.
>
>
RE: [jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is leaking memory
Posted by Digy <di...@gmail.com>.
Hi Eyal,
I couldn't find a simple way to use the WeakEntry as a key and then access
its value without enumerating all the keys in InternalTable.
I think you suggest
void Add(object keyX,object valueX)
{
InternalTable.Add( new WeakEntry(keyX,valueX), value);
} // or something like that.
Then how do you think that we get the value of "Get(keyX)" ? Enumerating all
the keys of InternalTable and comparing the "weakEntry.Key" with "keyX"?
When thinking of successive "Add"s, this has the complexity of O(n*n).
Can you share your solution?
DIGY
-----Original Message-----
From: Eyal Post [mailto:eyalpost@epocalipse.com]
Sent: Sunday, January 11, 2009 1:08 PM
To: lucene-net-dev@incubator.apache.org
Subject: RE: [jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121)
is leaking memory
As Luc noted in the first comment to this patch, I think that if this class
is to be called WeakHashTable (and therefor imply it is a general purpose
weak hash table) then it should be implemented as a real hash table.
Therefor, you should not use an array list and make linear comparisons.
Instead WeakHashTable should internally use a hashtable. The first
implementation was in the right direction, the only fix should have been to
use WeakEntry as a key to the table and not the hash code.
Eyal
> -----Original Message-----
> From: Digy [mailto:digydigy@gmail.com]
> Sent: Friday, January 09, 2009 20:03 PM
> To: lucene-net-dev@incubator.apache.org
> Subject: RE: [jira] Commented: (LUCENENET-106) Lucene.NET
> (Revision: 603121) is leaking memory
>
> Hi Tim,
>
>
>
> Thanks for the comments. I will correct the 3,4,5.
>
>
>
> 1) Since ArrayList contains WeakEntries & sync.ed for
> multiple threads, they can not be null and can not be any
> other type. I can not find a bug here. Maybe you can send a
> more detailed description.
>
> 2), WeakHashTable is compatible with "HashTable" in .Net
> world. "Ht.Add(key,value)" throws an exception for duplicate keys but
>
> "Ht[key]=value" just adds or replaces.
>
>
>
>
>
> DIGY
>
>
>
>
>
>
>
> -----Original Message-----
> From: Tim Januario (JIRA) [mailto:jira@apache.org]
> Sent: Friday, January 09, 2009 7:16 PM
> To: lucene-net-dev@incubator.apache.org
> Subject: [jira] Commented: (LUCENENET-106) Lucene.NET
> (Revision: 603121) is leaking memory
>
>
>
>
>
> [
> https://issues.apache.org/jira/browse/LUCENENET-106?page=com.a
tlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId
=12662433#action_12662433 ]
Tim Januario commented on LUCENENET-106:
----------------------------------------
Digy,
1) The WeakHashTable.Exists method creates a possible NullReferenceException
because you used WeakEntries[i] As WeakEntry instead of
(WeakEntry)WeakEntries[i]. This is only a warning but...
2) The Java WeakHashMap does not throw an execption when put() is called
with a key that already exists in the table. It simply replaces the old
value with the new value. .NET Hashtable does the same. The Add method of
WeakHashTable should remove the WeakEntry if the key is found and replace it
with the new value in order to stay in line instead of throwing the
Exception.
3) WeakHashTableEnumarator should be WeakHashTableEnumerator
4) WeakEntry.HashCode is not used and should be removed.
5) WeakHashTable.InitialMem is not used and should be removed.
Other than that, I think this looks good.
-tim
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL:
> https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Assignee: Digy
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar,
> luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable
> v2.patch, WeakHashTable+FieldCacheImpl.rar, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some
hash items that have closed IndexReader object as a key. So a lot of Term
instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this
problem.
> This bug can be reproduced only when Sort functionality used during
search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
RE: [jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is leaking memory
Posted by Eyal Post <ey...@epocalipse.com>.
As Luc noted in the first comment to this patch, I think that if this class
is to be called WeakHashTable (and therefor imply it is a general purpose
weak hash table) then it should be implemented as a real hash table.
Therefor, you should not use an array list and make linear comparisons.
Instead WeakHashTable should internally use a hashtable. The first
implementation was in the right direction, the only fix should have been to
use WeakEntry as a key to the table and not the hash code.
Eyal
> -----Original Message-----
> From: Digy [mailto:digydigy@gmail.com]
> Sent: Friday, January 09, 2009 20:03 PM
> To: lucene-net-dev@incubator.apache.org
> Subject: RE: [jira] Commented: (LUCENENET-106) Lucene.NET
> (Revision: 603121) is leaking memory
>
> Hi Tim,
>
>
>
> Thanks for the comments. I will correct the 3,4,5.
>
>
>
> 1) Since ArrayList contains WeakEntries & sync.ed for
> multiple threads, they can not be null and can not be any
> other type. I can not find a bug here. Maybe you can send a
> more detailed description.
>
> 2), WeakHashTable is compatible with "HashTable" in .Net
> world. "Ht.Add(key,value)" throws an exception for duplicate keys but
>
> "Ht[key]=value" just adds or replaces.
>
>
>
>
>
> DIGY
>
>
>
>
>
>
>
> -----Original Message-----
> From: Tim Januario (JIRA) [mailto:jira@apache.org]
> Sent: Friday, January 09, 2009 7:16 PM
> To: lucene-net-dev@incubator.apache.org
> Subject: [jira] Commented: (LUCENENET-106) Lucene.NET
> (Revision: 603121) is leaking memory
>
>
>
>
>
> [
> https://issues.apache.org/jira/browse/LUCENENET-106?page=com.a
tlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId
=12662433#action_12662433 ]
Tim Januario commented on LUCENENET-106:
----------------------------------------
Digy,
1) The WeakHashTable.Exists method creates a possible NullReferenceException
because you used WeakEntries[i] As WeakEntry instead of
(WeakEntry)WeakEntries[i]. This is only a warning but...
2) The Java WeakHashMap does not throw an execption when put() is called
with a key that already exists in the table. It simply replaces the old
value with the new value. .NET Hashtable does the same. The Add method of
WeakHashTable should remove the WeakEntry if the key is found and replace it
with the new value in order to stay in line instead of throwing the
Exception.
3) WeakHashTableEnumarator should be WeakHashTableEnumerator
4) WeakEntry.HashCode is not used and should be removed.
5) WeakHashTable.InitialMem is not used and should be removed.
Other than that, I think this looks good.
-tim
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL:
> https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Assignee: Digy
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar,
> luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable
> v2.patch, WeakHashTable+FieldCacheImpl.rar, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some
hash items that have closed IndexReader object as a key. So a lot of Term
instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this
problem.
> This bug can be reproduced only when Sort functionality used during
search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
RE: [jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is leaking memory
Posted by Digy <di...@gmail.com>.
Hi Tim,
Thanks for the comments. I will correct the 3,4,5.
1) Since ArrayList contains WeakEntries & sync.ed for multiple threads, they can not be null and can not be any other type. I can not find a bug here. Maybe you can send a more detailed description.
2), WeakHashTable is compatible with "HashTable" in .Net world. "Ht.Add(key,value)" throws an exception for duplicate keys but
"Ht[key]=value" just adds or replaces.
DIGY
-----Original Message-----
From: Tim Januario (JIRA) [mailto:jira@apache.org]
Sent: Friday, January 09, 2009 7:16 PM
To: lucene-net-dev@incubator.apache.org
Subject: [jira] Commented: (LUCENENET-106) Lucene.NET (Revision: 603121) is leaking memory
[ https://issues.apache.org/jira/browse/LUCENENET-106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662433#action_12662433 ]
Tim Januario commented on LUCENENET-106:
----------------------------------------
Digy,
1) The WeakHashTable.Exists method creates a possible NullReferenceException because you used WeakEntries[i] As WeakEntry instead of (WeakEntry)WeakEntries[i]. This is only a warning but...
2) The Java WeakHashMap does not throw an execption when put() is called with a key that already exists in the table. It simply replaces the old value with the new value. .NET Hashtable does the same. The Add method of WeakHashTable should remove the WeakEntry if the key is found and replace it with the new value in order to stay in line instead of throwing the Exception.
3) WeakHashTableEnumarator should be WeakHashTableEnumerator
4) WeakEntry.HashCode is not used and should be removed.
5) WeakHashTable.InitialMem is not used and should be removed.
Other than that, I think this looks good.
-tim
> Lucene.NET (Revision: 603121) is leaking memory
> -----------------------------------------------
>
> Key: LUCENENET-106
> URL: https://issues.apache.org/jira/browse/LUCENENET-106
> Project: Lucene.Net
> Issue Type: Bug
> Environment: .NET 2.0
> Reporter: Anton K.
> Assignee: Digy
> Priority: Critical
> Attachments: DIGY-FieldCacheImpl.patch, Digy.rar, luceneSrc_memUsage.patch, Paches for v2.3.1.rar, WeakHashTable v2.patch, WeakHashTable+FieldCacheImpl.rar, WeakReferences.rar
>
>
> readerCache Hashtable field (see FieldCacheImpl.cs) never releases some hash items that have closed IndexReader object as a key. So a lot of Term instances are never released.
> Java version of Lucene uses WeakHashMap and therefore doesn't have this problem.
> This bug can be reproduced only when Sort functionality used during search.
> See following link for additional information.
> http://www.gossamer-threads.com/lists/lucene/java-user/55681
> Steps to reproduce:
> 1)Create index
> 2) Modify index by IndexWiter; Close IndexWriter
> 3) Use IndexSearcher for searching with Sort; Close InexSearcher
> 4) Go to step 2
> You'll get OutOfMemoryException after some time of running this algorithm.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.