You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by GitBox <gi...@apache.org> on 2022/12/07 16:51:28 UTC
[GitHub] [lucene] gvaysman-at-github opened a new issue, #12000: Lucene-facet leaves ThreadLocal that creates a memory leak
gvaysman-at-github opened a new issue, #12000:
URL: https://github.com/apache/lucene/issues/12000
### Description
When the web application that uses Lucene Facets is shut down (say, from Tomcat Manager app), ThreadLocal<BytesRefBuilder> from UTF8TaxonomyWriterCache (the field bytes) is never cleaned up.
As a result, not only Tomcat complains of possible thread leak, the heap dump confirms this (below).
The code examination (including the latest lucene-facet 9.4.2) shows no bytes.remove() anywhere. Just clearing the BytesRefBuilder internal array is not sufficient.
Tomcat outputs to the console the following:
SEVERE [Thread-213] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [mh] created a ThreadLocal with key of type [org.apache.lucene.facet.taxonomy.writercache.UTF8TaxonomyWriterCache$1] (value [org.apache.lucene.facet.taxonomy.writercache.UTF8TaxonomyWriterCache$1@4c6f99ab]) and a value of type [org.apache.lucene.util.BytesRefBuilder] (value [Unknown]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Heap dump shows that the web app class loader, with all statics, etc. (unrelated to Lucene), cannot be released because of this leak
### Version and environment details
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] rmuir commented on issue #12000: Lucene-facet leaves ThreadLocal that creates a memory leak
Posted by GitBox <gi...@apache.org>.
rmuir commented on issue #12000:
URL: https://github.com/apache/lucene/issues/12000#issuecomment-1341280462
The UTF8TaxonomyWriterCache does have a `public void close() {}` but I'm not sure when/if it gets invoked as I'm not familiar with this cache and will need to look more into it. But one possibility to fix this cache is to replace the ThreadLocal with CloseableThreadLocal, and close it out in `close()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] vigyasharma commented on issue #12000: Lucene-facet leaves ThreadLocal that creates a memory leak
Posted by GitBox <gi...@apache.org>.
vigyasharma commented on issue #12000:
URL: https://github.com/apache/lucene/issues/12000#issuecomment-1396551522
Removed UTF8TaxonomyWriterCache from main, and deprecated it in 9.x. We now default to LruTaxonomyWriterCache. PRs have been merged in. Closing this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] rmuir commented on issue #12000: Lucene-facet leaves ThreadLocal that creates a memory leak
Posted by GitBox <gi...@apache.org>.
rmuir commented on issue #12000:
URL: https://github.com/apache/lucene/issues/12000#issuecomment-1341291174
A possible workaround for now might be to configure `LruTaxonomyWriterCache` instead of the default `UTF8TaxonomyWriterCache`. The LRU-based implementation doesn't use any threadlocals.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] rmuir commented on issue #12000: Lucene-facet leaves ThreadLocal that creates a memory leak
Posted by GitBox <gi...@apache.org>.
rmuir commented on issue #12000:
URL: https://github.com/apache/lucene/issues/12000#issuecomment-1341271839
Thanks @gvaysman-at-github for reporting this. I was unaware of this threadlocal, indeed it looks problematic just from the description in this cache:
```
/** A "cache" that never frees memory, and stores labels in a BytesRefHash (utf-8 encoding). */
public final class UTF8TaxonomyWriterCache implements TaxonomyWriterCache, Accountable {
private final ThreadLocal<BytesRefBuilder> bytes =
new ThreadLocal<BytesRefBuilder>() {
@Override
protected BytesRefBuilder initialValue() {
return new BytesRefBuilder();
}
};
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] vigyasharma commented on issue #12000: Lucene-facet leaves ThreadLocal that creates a memory leak
Posted by GitBox <gi...@apache.org>.
vigyasharma commented on issue #12000:
URL: https://github.com/apache/lucene/issues/12000#issuecomment-1347406654
> But one possibility to fix this cache is to replace the ThreadLocal with CloseableThreadLocal, and close it out in close()
`DirectoryTaxonomyWriter` does call `cache.close()` when it gets closed (via `closeResources()`), so I think this is a good idea.
I updated the cache to use CloseableThreadLocal. Don't have a good way to test for a memory leak, but I added a test to atleast verify that threadlocal memory is removed on close.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] vigyasharma closed issue #12000: Lucene-facet leaves ThreadLocal that creates a memory leak
Posted by GitBox <gi...@apache.org>.
vigyasharma closed issue #12000: Lucene-facet leaves ThreadLocal that creates a memory leak
URL: https://github.com/apache/lucene/issues/12000
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org