You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2021/01/28 12:38:15 UTC
[GitHub] [hive] zmatyus opened a new pull request #1923: HIVE-24636: Memory leak due LogFactory retaining ClassLoaders
zmatyus opened a new pull request #1923:
URL: https://github.com/apache/hive/pull/1923
### What changes were proposed in this pull request?
When a class loader is being closed, it should also be released from the `org.apache.commons.logging.LogFactory#factories`, where it is being used as a key.
### Why are the changes needed?
Current implementation has a slow but steady memory leak.
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] zmatyus commented on pull request #1923: HIVE-24636: Memory leak due LogFactory retaining ClassLoaders
Posted by GitBox <gi...@apache.org>.
zmatyus commented on pull request #1923:
URL: https://github.com/apache/hive/pull/1923#issuecomment-786571251
@pdbenitez91 : not sure what queries can make this the most visible. All I've seen was a report (Grafana screenshot) about increasing memory usage and a heap dump where the `factories` hashmap of `org.apache.commons.logging.LogFactory` holds on to thousands of class loaders, each of which retains some memory. I'm not sure either monitoring or head dump is an option with AWS.
Seeing how this is about class loaders, you might experiment with queries that use Custom UDFs with lots of classes defined in their jar files... but I'm just guessing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pdbenitez91 commented on pull request #1923: HIVE-24636: Memory leak due LogFactory retaining ClassLoaders
Posted by GitBox <gi...@apache.org>.
pdbenitez91 commented on pull request #1923:
URL: https://github.com/apache/hive/pull/1923#issuecomment-786157864
@zmatyus what is the best way to reproduce this issue in Hive with queries?
I am using Hive3.1.2(from EMR/AWS) and I think this is causing us a memory leak problem in hive-server.
Thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] lcspinter commented on pull request #1923: HIVE-24636: Memory leak due LogFactory retaining ClassLoaders
Posted by GitBox <gi...@apache.org>.
lcspinter commented on pull request #1923:
URL: https://github.com/apache/hive/pull/1923#issuecomment-770666974
Thanks for the patch @zmatyus and for the review @klcopp.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] lcspinter merged pull request #1923: HIVE-24636: Memory leak due LogFactory retaining ClassLoaders
Posted by GitBox <gi...@apache.org>.
lcspinter merged pull request #1923:
URL: https://github.com/apache/hive/pull/1923
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] zmatyus commented on pull request #1923: HIVE-24636: Memory leak due LogFactory retaining ClassLoaders
Posted by GitBox <gi...@apache.org>.
zmatyus commented on pull request #1923:
URL: https://github.com/apache/hive/pull/1923#issuecomment-770661455
No tests on `branch-3`, however this change has been merged to `branch-3.1` in #1924.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org