You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucenenet.apache.org by "Rezgar Cadro (JIRA)" <ji...@apache.org> on 2010/04/27 14:17:32 UTC
[jira] Created: (LUCENENET-358) CloseableThreadLocal memory leak
in LocalDataStoreSlot (with workaround)
CloseableThreadLocal memory leak in LocalDataStoreSlot (with workaround)
-------------------------------------------------------------------------
Key: LUCENENET-358
URL: https://issues.apache.org/jira/browse/LUCENENET-358
Project: Lucene.Net
Issue Type: Bug
Environment: Microsoft WIndows Server 2008 Enterprise x64. SP2.
.NET Framework 4.0
Reporter: Rezgar Cadro
Priority: Critical
Attachments: CloseableThreadLocal MemoryLeak.patch
Recently we have been suffering from a severe memory leak when executing intense open/close operations on IndexSearcher and IndexModifier.
Memory profiling showed that memory is being held by LocalDataStore[] objects.
After some digging, the root of the problem has been found in CloseableThreadLocal class:
private System.LocalDataStoreSlot t = System.Threading.Thread.AllocateDataSlot();
What we see is that every instantiated object of CloseableThreadLocal causes new data slot allocation performed for every thread.
Thread.AllocateDataSlot() does not simply allocate a new slot, replacing an old one, but enlarging an existing buffer in-thread, appending data to the end of internal LocalDataStore[] collection, which causes a severe memory leak .
As long as "t" variable is instantiated on every object creation, and (in current class implementation) every object is used by a single thread, replacing "private System.LocalDataStoreSlot t = System.Threading.Thread.AllocateDataSlot();" with simple "private object dataSlot;" and removing "hardRefs" Dictionary solves the problem and prevents memory leak.
We have tried to implement the expected behavior by using [ThreadStatic] attribute instead of LocalDataStoreSlot, but the attempt failed because of unexpected exceptions being thrown.
Patch can be found at Lucene.Net repository under
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (LUCENENET-358) CloseableThreadLocal memory leak
in LocalDataStoreSlot (with workaround)
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Digy resolved LUCENENET-358.
----------------------------
Assignee: Digy
Resolution: Fixed
Patch committed.
DIGY
> CloseableThreadLocal memory leak in LocalDataStoreSlot (with workaround)
> -------------------------------------------------------------------------
>
> Key: LUCENENET-358
> URL: https://issues.apache.org/jira/browse/LUCENENET-358
> Project: Lucene.Net
> Issue Type: Bug
> Environment: Microsoft WIndows Server 2008 Enterprise x64. SP2.
> .NET Framework 4.0
> Reporter: Rezgar Cadro
> Assignee: Digy
> Priority: Critical
> Attachments: CloseableThreadLocal MemoryLeak.patch, CloseableThreadLocal.diff, CloseableThreadLocal.patch
>
>
> Recently we have been suffering from a severe memory leak when executing intense open/close operations on IndexSearcher and IndexModifier.
> Memory profiling showed that memory is being held by LocalDataStore[] objects.
> After some digging, the root of the problem has been found in CloseableThreadLocal class:
> private System.LocalDataStoreSlot t = System.Threading.Thread.AllocateDataSlot();
> What we see is that every instantiated object of CloseableThreadLocal causes new data slot allocation performed for every thread.
> Thread.AllocateDataSlot() does not simply allocate a new slot, replacing an old one, but enlarging an existing buffer in-thread, appending data to the end of internal LocalDataStore[] collection, which causes a severe memory leak .
> As long as "t" variable is instantiated on every object creation, and (in current class implementation) every object is used by a single thread, replacing "private System.LocalDataStoreSlot t = System.Threading.Thread.AllocateDataSlot();" with simple "private object dataSlot;" and removing "hardRefs" Dictionary solves the problem and prevents memory leak.
> We have tried to implement the expected behavior by using [ThreadStatic] attribute instead of LocalDataStoreSlot, but the attempt failed because of unexpected exceptions being thrown.
> Patch can be found at Lucene.Net repository under
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (LUCENENET-358) CloseableThreadLocal memory leak
in LocalDataStoreSlot (with workaround)
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Digy updated LUCENENET-358:
---------------------------
Attachment: CloseableThreadLocal.patch
Hi Robert,
Very nice patch, thanks.
But I have problems with your .diff files (maybe since I use TortoiseSVN). So, I will attach a ".patch" file for those who use TortoiseSVN.
DIGY
> CloseableThreadLocal memory leak in LocalDataStoreSlot (with workaround)
> -------------------------------------------------------------------------
>
> Key: LUCENENET-358
> URL: https://issues.apache.org/jira/browse/LUCENENET-358
> Project: Lucene.Net
> Issue Type: Bug
> Environment: Microsoft WIndows Server 2008 Enterprise x64. SP2.
> .NET Framework 4.0
> Reporter: Rezgar Cadro
> Priority: Critical
> Attachments: CloseableThreadLocal MemoryLeak.patch, CloseableThreadLocal.diff, CloseableThreadLocal.patch
>
>
> Recently we have been suffering from a severe memory leak when executing intense open/close operations on IndexSearcher and IndexModifier.
> Memory profiling showed that memory is being held by LocalDataStore[] objects.
> After some digging, the root of the problem has been found in CloseableThreadLocal class:
> private System.LocalDataStoreSlot t = System.Threading.Thread.AllocateDataSlot();
> What we see is that every instantiated object of CloseableThreadLocal causes new data slot allocation performed for every thread.
> Thread.AllocateDataSlot() does not simply allocate a new slot, replacing an old one, but enlarging an existing buffer in-thread, appending data to the end of internal LocalDataStore[] collection, which causes a severe memory leak .
> As long as "t" variable is instantiated on every object creation, and (in current class implementation) every object is used by a single thread, replacing "private System.LocalDataStoreSlot t = System.Threading.Thread.AllocateDataSlot();" with simple "private object dataSlot;" and removing "hardRefs" Dictionary solves the problem and prevents memory leak.
> We have tried to implement the expected behavior by using [ThreadStatic] attribute instead of LocalDataStoreSlot, but the attempt failed because of unexpected exceptions being thrown.
> Patch can be found at Lucene.Net repository under
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-358) CloseableThreadLocal memory leak
in LocalDataStoreSlot (with workaround)
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12865616#action_12865616 ]
Digy commented on LUCENENET-358:
--------------------------------
Ben,
Can you test the patch? Although it is 6-8 times faster( in successive sets & gets), I don't see a noticable performance gain in searches.
DIGY
> CloseableThreadLocal memory leak in LocalDataStoreSlot (with workaround)
> -------------------------------------------------------------------------
>
> Key: LUCENENET-358
> URL: https://issues.apache.org/jira/browse/LUCENENET-358
> Project: Lucene.Net
> Issue Type: Bug
> Environment: Microsoft WIndows Server 2008 Enterprise x64. SP2.
> .NET Framework 4.0
> Reporter: Rezgar Cadro
> Priority: Critical
> Attachments: CloseableThreadLocal MemoryLeak.patch, CloseableThreadLocal.diff, CloseableThreadLocal.patch
>
>
> Recently we have been suffering from a severe memory leak when executing intense open/close operations on IndexSearcher and IndexModifier.
> Memory profiling showed that memory is being held by LocalDataStore[] objects.
> After some digging, the root of the problem has been found in CloseableThreadLocal class:
> private System.LocalDataStoreSlot t = System.Threading.Thread.AllocateDataSlot();
> What we see is that every instantiated object of CloseableThreadLocal causes new data slot allocation performed for every thread.
> Thread.AllocateDataSlot() does not simply allocate a new slot, replacing an old one, but enlarging an existing buffer in-thread, appending data to the end of internal LocalDataStore[] collection, which causes a severe memory leak .
> As long as "t" variable is instantiated on every object creation, and (in current class implementation) every object is used by a single thread, replacing "private System.LocalDataStoreSlot t = System.Threading.Thread.AllocateDataSlot();" with simple "private object dataSlot;" and removing "hardRefs" Dictionary solves the problem and prevents memory leak.
> We have tried to implement the expected behavior by using [ThreadStatic] attribute instead of LocalDataStoreSlot, but the attempt failed because of unexpected exceptions being thrown.
> Patch can be found at Lucene.Net repository under
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (LUCENENET-358) CloseableThreadLocal memory leak
in LocalDataStoreSlot (with workaround)
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Digy reopened LUCENENET-358:
----------------------------
Hi Ben,
Ignore my previous comment, I will reopen this issue since the topic evolves here.
DIGY
> CloseableThreadLocal memory leak in LocalDataStoreSlot (with workaround)
> -------------------------------------------------------------------------
>
> Key: LUCENENET-358
> URL: https://issues.apache.org/jira/browse/LUCENENET-358
> Project: Lucene.Net
> Issue Type: Bug
> Environment: Microsoft WIndows Server 2008 Enterprise x64. SP2.
> .NET Framework 4.0
> Reporter: Rezgar Cadro
> Priority: Critical
> Attachments: CloseableThreadLocal MemoryLeak.patch, CloseableThreadLocal.diff, CloseableThreadLocal.patch
>
>
> Recently we have been suffering from a severe memory leak when executing intense open/close operations on IndexSearcher and IndexModifier.
> Memory profiling showed that memory is being held by LocalDataStore[] objects.
> After some digging, the root of the problem has been found in CloseableThreadLocal class:
> private System.LocalDataStoreSlot t = System.Threading.Thread.AllocateDataSlot();
> What we see is that every instantiated object of CloseableThreadLocal causes new data slot allocation performed for every thread.
> Thread.AllocateDataSlot() does not simply allocate a new slot, replacing an old one, but enlarging an existing buffer in-thread, appending data to the end of internal LocalDataStore[] collection, which causes a severe memory leak .
> As long as "t" variable is instantiated on every object creation, and (in current class implementation) every object is used by a single thread, replacing "private System.LocalDataStoreSlot t = System.Threading.Thread.AllocateDataSlot();" with simple "private object dataSlot;" and removing "hardRefs" Dictionary solves the problem and prevents memory leak.
> We have tried to implement the expected behavior by using [ThreadStatic] attribute instead of LocalDataStoreSlot, but the attempt failed because of unexpected exceptions being thrown.
> Patch can be found at Lucene.Net repository under
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-358) CloseableThreadLocal memory leak
in LocalDataStoreSlot (with workaround)
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12870780#action_12870780 ]
Digy commented on LUCENENET-358:
--------------------------------
Thanks Robert,
I commited your new patch. (both to trunk and to 2.9.2 tag).
DIGY
> CloseableThreadLocal memory leak in LocalDataStoreSlot (with workaround)
> -------------------------------------------------------------------------
>
> Key: LUCENENET-358
> URL: https://issues.apache.org/jira/browse/LUCENENET-358
> Project: Lucene.Net
> Issue Type: Bug
> Environment: Microsoft WIndows Server 2008 Enterprise x64. SP2.
> .NET Framework 4.0
> Reporter: Rezgar Cadro
> Assignee: Digy
> Priority: Critical
> Attachments: CloseableThreadLocal MemoryLeak.patch, CloseableThreadLocal.diff, CloseableThreadLocal.diff, CloseableThreadLocal.patch
>
>
> Recently we have been suffering from a severe memory leak when executing intense open/close operations on IndexSearcher and IndexModifier.
> Memory profiling showed that memory is being held by LocalDataStore[] objects.
> After some digging, the root of the problem has been found in CloseableThreadLocal class:
> private System.LocalDataStoreSlot t = System.Threading.Thread.AllocateDataSlot();
> What we see is that every instantiated object of CloseableThreadLocal causes new data slot allocation performed for every thread.
> Thread.AllocateDataSlot() does not simply allocate a new slot, replacing an old one, but enlarging an existing buffer in-thread, appending data to the end of internal LocalDataStore[] collection, which causes a severe memory leak .
> As long as "t" variable is instantiated on every object creation, and (in current class implementation) every object is used by a single thread, replacing "private System.LocalDataStoreSlot t = System.Threading.Thread.AllocateDataSlot();" with simple "private object dataSlot;" and removing "hardRefs" Dictionary solves the problem and prevents memory leak.
> We have tried to implement the expected behavior by using [ThreadStatic] attribute instead of LocalDataStoreSlot, but the attempt failed because of unexpected exceptions being thrown.
> Patch can be found at Lucene.Net repository under
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (LUCENENET-358) CloseableThreadLocal memory leak
in LocalDataStoreSlot (with workaround)
Posted by "Robert Jordan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Jordan updated LUCENENET-358:
------------------------------------
Attachment: CloseableThreadLocal.diff
replace Dictionary with WeakHashTable
> CloseableThreadLocal memory leak in LocalDataStoreSlot (with workaround)
> -------------------------------------------------------------------------
>
> Key: LUCENENET-358
> URL: https://issues.apache.org/jira/browse/LUCENENET-358
> Project: Lucene.Net
> Issue Type: Bug
> Environment: Microsoft WIndows Server 2008 Enterprise x64. SP2.
> .NET Framework 4.0
> Reporter: Rezgar Cadro
> Assignee: Digy
> Priority: Critical
> Attachments: CloseableThreadLocal MemoryLeak.patch, CloseableThreadLocal.diff, CloseableThreadLocal.diff, CloseableThreadLocal.patch
>
>
> Recently we have been suffering from a severe memory leak when executing intense open/close operations on IndexSearcher and IndexModifier.
> Memory profiling showed that memory is being held by LocalDataStore[] objects.
> After some digging, the root of the problem has been found in CloseableThreadLocal class:
> private System.LocalDataStoreSlot t = System.Threading.Thread.AllocateDataSlot();
> What we see is that every instantiated object of CloseableThreadLocal causes new data slot allocation performed for every thread.
> Thread.AllocateDataSlot() does not simply allocate a new slot, replacing an old one, but enlarging an existing buffer in-thread, appending data to the end of internal LocalDataStore[] collection, which causes a severe memory leak .
> As long as "t" variable is instantiated on every object creation, and (in current class implementation) every object is used by a single thread, replacing "private System.LocalDataStoreSlot t = System.Threading.Thread.AllocateDataSlot();" with simple "private object dataSlot;" and removing "hardRefs" Dictionary solves the problem and prevents memory leak.
> We have tried to implement the expected behavior by using [ThreadStatic] attribute instead of LocalDataStoreSlot, but the attempt failed because of unexpected exceptions being thrown.
> Patch can be found at Lucene.Net repository under
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-358) CloseableThreadLocal memory leak
in LocalDataStoreSlot (with workaround)
Posted by "Ben West (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12865785#action_12865785 ]
Ben West commented on LUCENENET-358:
------------------------------------
wow. Searching fell from low hundred milliseconds to teens, so order of magnitude improvement.
Thanks guys!
> CloseableThreadLocal memory leak in LocalDataStoreSlot (with workaround)
> -------------------------------------------------------------------------
>
> Key: LUCENENET-358
> URL: https://issues.apache.org/jira/browse/LUCENENET-358
> Project: Lucene.Net
> Issue Type: Bug
> Environment: Microsoft WIndows Server 2008 Enterprise x64. SP2.
> .NET Framework 4.0
> Reporter: Rezgar Cadro
> Priority: Critical
> Attachments: CloseableThreadLocal MemoryLeak.patch, CloseableThreadLocal.diff, CloseableThreadLocal.patch
>
>
> Recently we have been suffering from a severe memory leak when executing intense open/close operations on IndexSearcher and IndexModifier.
> Memory profiling showed that memory is being held by LocalDataStore[] objects.
> After some digging, the root of the problem has been found in CloseableThreadLocal class:
> private System.LocalDataStoreSlot t = System.Threading.Thread.AllocateDataSlot();
> What we see is that every instantiated object of CloseableThreadLocal causes new data slot allocation performed for every thread.
> Thread.AllocateDataSlot() does not simply allocate a new slot, replacing an old one, but enlarging an existing buffer in-thread, appending data to the end of internal LocalDataStore[] collection, which causes a severe memory leak .
> As long as "t" variable is instantiated on every object creation, and (in current class implementation) every object is used by a single thread, replacing "private System.LocalDataStoreSlot t = System.Threading.Thread.AllocateDataSlot();" with simple "private object dataSlot;" and removing "hardRefs" Dictionary solves the problem and prevents memory leak.
> We have tried to implement the expected behavior by using [ThreadStatic] attribute instead of LocalDataStoreSlot, but the attempt failed because of unexpected exceptions being thrown.
> Patch can be found at Lucene.Net repository under
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Closed: (LUCENENET-358) CloseableThreadLocal memory leak in
LocalDataStoreSlot (with workaround)
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Digy closed LUCENENET-358.
--------------------------
Resolution: Invalid
No response so far.
Closing this issue.
DIGY
> CloseableThreadLocal memory leak in LocalDataStoreSlot (with workaround)
> -------------------------------------------------------------------------
>
> Key: LUCENENET-358
> URL: https://issues.apache.org/jira/browse/LUCENENET-358
> Project: Lucene.Net
> Issue Type: Bug
> Environment: Microsoft WIndows Server 2008 Enterprise x64. SP2.
> .NET Framework 4.0
> Reporter: Rezgar Cadro
> Priority: Critical
> Attachments: CloseableThreadLocal MemoryLeak.patch
>
>
> Recently we have been suffering from a severe memory leak when executing intense open/close operations on IndexSearcher and IndexModifier.
> Memory profiling showed that memory is being held by LocalDataStore[] objects.
> After some digging, the root of the problem has been found in CloseableThreadLocal class:
> private System.LocalDataStoreSlot t = System.Threading.Thread.AllocateDataSlot();
> What we see is that every instantiated object of CloseableThreadLocal causes new data slot allocation performed for every thread.
> Thread.AllocateDataSlot() does not simply allocate a new slot, replacing an old one, but enlarging an existing buffer in-thread, appending data to the end of internal LocalDataStore[] collection, which causes a severe memory leak .
> As long as "t" variable is instantiated on every object creation, and (in current class implementation) every object is used by a single thread, replacing "private System.LocalDataStoreSlot t = System.Threading.Thread.AllocateDataSlot();" with simple "private object dataSlot;" and removing "hardRefs" Dictionary solves the problem and prevents memory leak.
> We have tried to implement the expected behavior by using [ThreadStatic] attribute instead of LocalDataStoreSlot, but the attempt failed because of unexpected exceptions being thrown.
> Patch can be found at Lucene.Net repository under
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (LUCENENET-358) CloseableThreadLocal memory leak
in LocalDataStoreSlot (with workaround)
Posted by "Rezgar Cadro (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rezgar Cadro updated LUCENENET-358:
-----------------------------------
Attachment: CloseableThreadLocal MemoryLeak.patch
Patch (workaround)
> CloseableThreadLocal memory leak in LocalDataStoreSlot (with workaround)
> -------------------------------------------------------------------------
>
> Key: LUCENENET-358
> URL: https://issues.apache.org/jira/browse/LUCENENET-358
> Project: Lucene.Net
> Issue Type: Bug
> Environment: Microsoft WIndows Server 2008 Enterprise x64. SP2.
> .NET Framework 4.0
> Reporter: Rezgar Cadro
> Priority: Critical
> Attachments: CloseableThreadLocal MemoryLeak.patch
>
>
> Recently we have been suffering from a severe memory leak when executing intense open/close operations on IndexSearcher and IndexModifier.
> Memory profiling showed that memory is being held by LocalDataStore[] objects.
> After some digging, the root of the problem has been found in CloseableThreadLocal class:
> private System.LocalDataStoreSlot t = System.Threading.Thread.AllocateDataSlot();
> What we see is that every instantiated object of CloseableThreadLocal causes new data slot allocation performed for every thread.
> Thread.AllocateDataSlot() does not simply allocate a new slot, replacing an old one, but enlarging an existing buffer in-thread, appending data to the end of internal LocalDataStore[] collection, which causes a severe memory leak .
> As long as "t" variable is instantiated on every object creation, and (in current class implementation) every object is used by a single thread, replacing "private System.LocalDataStoreSlot t = System.Threading.Thread.AllocateDataSlot();" with simple "private object dataSlot;" and removing "hardRefs" Dictionary solves the problem and prevents memory leak.
> We have tried to implement the expected behavior by using [ThreadStatic] attribute instead of LocalDataStoreSlot, but the attempt failed because of unexpected exceptions being thrown.
> Patch can be found at Lucene.Net repository under
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-358) CloseableThreadLocal memory leak
in LocalDataStoreSlot (with workaround)
Posted by "Robert Jordan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12869549#action_12869549 ]
Robert Jordan commented on LUCENENET-358:
-----------------------------------------
It seems the the new code may leak if Close() isn't called. This is prevalent on long running threads (like application's main thread).
The new patch is using a SupportClass.WeakHashTable in place of the Dictionary.
> CloseableThreadLocal memory leak in LocalDataStoreSlot (with workaround)
> -------------------------------------------------------------------------
>
> Key: LUCENENET-358
> URL: https://issues.apache.org/jira/browse/LUCENENET-358
> Project: Lucene.Net
> Issue Type: Bug
> Environment: Microsoft WIndows Server 2008 Enterprise x64. SP2.
> .NET Framework 4.0
> Reporter: Rezgar Cadro
> Assignee: Digy
> Priority: Critical
> Attachments: CloseableThreadLocal MemoryLeak.patch, CloseableThreadLocal.diff, CloseableThreadLocal.patch
>
>
> Recently we have been suffering from a severe memory leak when executing intense open/close operations on IndexSearcher and IndexModifier.
> Memory profiling showed that memory is being held by LocalDataStore[] objects.
> After some digging, the root of the problem has been found in CloseableThreadLocal class:
> private System.LocalDataStoreSlot t = System.Threading.Thread.AllocateDataSlot();
> What we see is that every instantiated object of CloseableThreadLocal causes new data slot allocation performed for every thread.
> Thread.AllocateDataSlot() does not simply allocate a new slot, replacing an old one, but enlarging an existing buffer in-thread, appending data to the end of internal LocalDataStore[] collection, which causes a severe memory leak .
> As long as "t" variable is instantiated on every object creation, and (in current class implementation) every object is used by a single thread, replacing "private System.LocalDataStoreSlot t = System.Threading.Thread.AllocateDataSlot();" with simple "private object dataSlot;" and removing "hardRefs" Dictionary solves the problem and prevents memory leak.
> We have tried to implement the expected behavior by using [ThreadStatic] attribute instead of LocalDataStoreSlot, but the attempt failed because of unexpected exceptions being thrown.
> Patch can be found at Lucene.Net repository under
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-358) CloseableThreadLocal memory leak
in LocalDataStoreSlot (with workaround)
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861473#action_12861473 ]
Digy commented on LUCENENET-358:
--------------------------------
Additionally:
It seems that your are making concurrent updates/searches on your index.
If so, then just use a single instance of IndexWriter throughout your application (without closing it) and use IndexWriter.GetReader method whenever an IndexReader is required(ie. to create an IndexSearcher and don't forget to close it after your search is finished).
With this approach, I am pretty sure that your mem-leakage will drop/disappear, but -of course- this doesn't mean that we don't have to fix the bug, if exists.
PS: IndexModifier is deprecated and has a terrible performance.
DIGY
> CloseableThreadLocal memory leak in LocalDataStoreSlot (with workaround)
> -------------------------------------------------------------------------
>
> Key: LUCENENET-358
> URL: https://issues.apache.org/jira/browse/LUCENENET-358
> Project: Lucene.Net
> Issue Type: Bug
> Environment: Microsoft WIndows Server 2008 Enterprise x64. SP2.
> .NET Framework 4.0
> Reporter: Rezgar Cadro
> Priority: Critical
> Attachments: CloseableThreadLocal MemoryLeak.patch
>
>
> Recently we have been suffering from a severe memory leak when executing intense open/close operations on IndexSearcher and IndexModifier.
> Memory profiling showed that memory is being held by LocalDataStore[] objects.
> After some digging, the root of the problem has been found in CloseableThreadLocal class:
> private System.LocalDataStoreSlot t = System.Threading.Thread.AllocateDataSlot();
> What we see is that every instantiated object of CloseableThreadLocal causes new data slot allocation performed for every thread.
> Thread.AllocateDataSlot() does not simply allocate a new slot, replacing an old one, but enlarging an existing buffer in-thread, appending data to the end of internal LocalDataStore[] collection, which causes a severe memory leak .
> As long as "t" variable is instantiated on every object creation, and (in current class implementation) every object is used by a single thread, replacing "private System.LocalDataStoreSlot t = System.Threading.Thread.AllocateDataSlot();" with simple "private object dataSlot;" and removing "hardRefs" Dictionary solves the problem and prevents memory leak.
> We have tried to implement the expected behavior by using [ThreadStatic] attribute instead of LocalDataStoreSlot, but the attempt failed because of unexpected exceptions being thrown.
> Patch can be found at Lucene.Net repository under
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-358) CloseableThreadLocal memory leak
in LocalDataStoreSlot (with workaround)
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12865303#action_12865303 ]
Digy commented on LUCENENET-358:
--------------------------------
Hi Ben,
This issue is about a possible memory leakage.
Instead of reopening this, it would be better if you create a new issue that focuses on the performance of "AllocateDataSlot"
DIGY
> CloseableThreadLocal memory leak in LocalDataStoreSlot (with workaround)
> -------------------------------------------------------------------------
>
> Key: LUCENENET-358
> URL: https://issues.apache.org/jira/browse/LUCENENET-358
> Project: Lucene.Net
> Issue Type: Bug
> Environment: Microsoft WIndows Server 2008 Enterprise x64. SP2.
> .NET Framework 4.0
> Reporter: Rezgar Cadro
> Priority: Critical
> Attachments: CloseableThreadLocal MemoryLeak.patch
>
>
> Recently we have been suffering from a severe memory leak when executing intense open/close operations on IndexSearcher and IndexModifier.
> Memory profiling showed that memory is being held by LocalDataStore[] objects.
> After some digging, the root of the problem has been found in CloseableThreadLocal class:
> private System.LocalDataStoreSlot t = System.Threading.Thread.AllocateDataSlot();
> What we see is that every instantiated object of CloseableThreadLocal causes new data slot allocation performed for every thread.
> Thread.AllocateDataSlot() does not simply allocate a new slot, replacing an old one, but enlarging an existing buffer in-thread, appending data to the end of internal LocalDataStore[] collection, which causes a severe memory leak .
> As long as "t" variable is instantiated on every object creation, and (in current class implementation) every object is used by a single thread, replacing "private System.LocalDataStoreSlot t = System.Threading.Thread.AllocateDataSlot();" with simple "private object dataSlot;" and removing "hardRefs" Dictionary solves the problem and prevents memory leak.
> We have tried to implement the expected behavior by using [ThreadStatic] attribute instead of LocalDataStoreSlot, but the attempt failed because of unexpected exceptions being thrown.
> Patch can be found at Lucene.Net repository under
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (LUCENENET-358) CloseableThreadLocal memory leak
in LocalDataStoreSlot (with workaround)
Posted by "Robert Jordan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Jordan updated LUCENENET-358:
------------------------------------
Attachment: CloseableThreadLocal.diff
Here a patch that uses [ThreadStatic] instead of LocalDataStoreSlot.
I don't know if it's faster, so please test it. It does pass the unit tests.
Since MS.NET does not have Java's retention issues, I've removed WeakReferences.
Don't be worried about the missing locks. They are not needed because [ThreadStatic] ensures that "slots" is null in any new thread.
> CloseableThreadLocal memory leak in LocalDataStoreSlot (with workaround)
> -------------------------------------------------------------------------
>
> Key: LUCENENET-358
> URL: https://issues.apache.org/jira/browse/LUCENENET-358
> Project: Lucene.Net
> Issue Type: Bug
> Environment: Microsoft WIndows Server 2008 Enterprise x64. SP2.
> .NET Framework 4.0
> Reporter: Rezgar Cadro
> Priority: Critical
> Attachments: CloseableThreadLocal MemoryLeak.patch, CloseableThreadLocal.diff
>
>
> Recently we have been suffering from a severe memory leak when executing intense open/close operations on IndexSearcher and IndexModifier.
> Memory profiling showed that memory is being held by LocalDataStore[] objects.
> After some digging, the root of the problem has been found in CloseableThreadLocal class:
> private System.LocalDataStoreSlot t = System.Threading.Thread.AllocateDataSlot();
> What we see is that every instantiated object of CloseableThreadLocal causes new data slot allocation performed for every thread.
> Thread.AllocateDataSlot() does not simply allocate a new slot, replacing an old one, but enlarging an existing buffer in-thread, appending data to the end of internal LocalDataStore[] collection, which causes a severe memory leak .
> As long as "t" variable is instantiated on every object creation, and (in current class implementation) every object is used by a single thread, replacing "private System.LocalDataStoreSlot t = System.Threading.Thread.AllocateDataSlot();" with simple "private object dataSlot;" and removing "hardRefs" Dictionary solves the problem and prevents memory leak.
> We have tried to implement the expected behavior by using [ThreadStatic] attribute instead of LocalDataStoreSlot, but the attempt failed because of unexpected exceptions being thrown.
> Patch can be found at Lucene.Net repository under
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-358) CloseableThreadLocal memory leak
in LocalDataStoreSlot (with workaround)
Posted by "Digy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861446#action_12861446 ]
Digy commented on LUCENENET-358:
--------------------------------
Hi Rezgar,
After applying your patch many NUnit test failed such as TestThreadSafety, TestImmediateDiskFullWithThreads, TestIOExceptionDuringAbortWithThreads, TestRandomExceptionsThreads and more.
Can you post a small code showing the leakage, so that I can work on it too?
DIGY
> CloseableThreadLocal memory leak in LocalDataStoreSlot (with workaround)
> -------------------------------------------------------------------------
>
> Key: LUCENENET-358
> URL: https://issues.apache.org/jira/browse/LUCENENET-358
> Project: Lucene.Net
> Issue Type: Bug
> Environment: Microsoft WIndows Server 2008 Enterprise x64. SP2.
> .NET Framework 4.0
> Reporter: Rezgar Cadro
> Priority: Critical
> Attachments: CloseableThreadLocal MemoryLeak.patch
>
>
> Recently we have been suffering from a severe memory leak when executing intense open/close operations on IndexSearcher and IndexModifier.
> Memory profiling showed that memory is being held by LocalDataStore[] objects.
> After some digging, the root of the problem has been found in CloseableThreadLocal class:
> private System.LocalDataStoreSlot t = System.Threading.Thread.AllocateDataSlot();
> What we see is that every instantiated object of CloseableThreadLocal causes new data slot allocation performed for every thread.
> Thread.AllocateDataSlot() does not simply allocate a new slot, replacing an old one, but enlarging an existing buffer in-thread, appending data to the end of internal LocalDataStore[] collection, which causes a severe memory leak .
> As long as "t" variable is instantiated on every object creation, and (in current class implementation) every object is used by a single thread, replacing "private System.LocalDataStoreSlot t = System.Threading.Thread.AllocateDataSlot();" with simple "private object dataSlot;" and removing "hardRefs" Dictionary solves the problem and prevents memory leak.
> We have tried to implement the expected behavior by using [ThreadStatic] attribute instead of LocalDataStoreSlot, but the attempt failed because of unexpected exceptions being thrown.
> Patch can be found at Lucene.Net repository under
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (LUCENENET-358) CloseableThreadLocal memory leak
in LocalDataStoreSlot (with workaround)
Posted by "Ben West (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENENET-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12865244#action_12865244 ]
Ben West commented on LUCENENET-358:
------------------------------------
Could we re-open this? I ran the profiler on a test app which just searches for random strings over and over again, and I found that half (!!) of the time was spent on the line AllocateDataSlot() and another 25% on Thread.SetData(). I'm just using a vanilla IndexSearcher. (By way of comparison, BinaryReader.Read() took only .5% of the total time.)
The documentation (http://msdn.microsoft.com/en-us/library/system.threading.thread.allocatedataslot.aspx) says "For better performance, use fields that are marked with the ThreadStaticAttribute attribute instead".
I can't get a patch to pass those tests either, but perhaps someone who knows more about the code could improve the performance here while still maintaining functionality?
> CloseableThreadLocal memory leak in LocalDataStoreSlot (with workaround)
> -------------------------------------------------------------------------
>
> Key: LUCENENET-358
> URL: https://issues.apache.org/jira/browse/LUCENENET-358
> Project: Lucene.Net
> Issue Type: Bug
> Environment: Microsoft WIndows Server 2008 Enterprise x64. SP2.
> .NET Framework 4.0
> Reporter: Rezgar Cadro
> Priority: Critical
> Attachments: CloseableThreadLocal MemoryLeak.patch
>
>
> Recently we have been suffering from a severe memory leak when executing intense open/close operations on IndexSearcher and IndexModifier.
> Memory profiling showed that memory is being held by LocalDataStore[] objects.
> After some digging, the root of the problem has been found in CloseableThreadLocal class:
> private System.LocalDataStoreSlot t = System.Threading.Thread.AllocateDataSlot();
> What we see is that every instantiated object of CloseableThreadLocal causes new data slot allocation performed for every thread.
> Thread.AllocateDataSlot() does not simply allocate a new slot, replacing an old one, but enlarging an existing buffer in-thread, appending data to the end of internal LocalDataStore[] collection, which causes a severe memory leak .
> As long as "t" variable is instantiated on every object creation, and (in current class implementation) every object is used by a single thread, replacing "private System.LocalDataStoreSlot t = System.Threading.Thread.AllocateDataSlot();" with simple "private object dataSlot;" and removing "hardRefs" Dictionary solves the problem and prevents memory leak.
> We have tried to implement the expected behavior by using [ThreadStatic] attribute instead of LocalDataStoreSlot, but the attempt failed because of unexpected exceptions being thrown.
> Patch can be found at Lucene.Net repository under
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.