You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by "rzuo (via GitHub)" <gi...@apache.org> on 2023/08/03 10:37:23 UTC
[GitHub] [bigtop] rzuo opened a new pull request, #1158: BIGTOP-3963: Merge hive memory leak PRs (for hive-3.1.3)
rzuo opened a new pull request, #1158:
URL: https://github.com/apache/bigtop/pull/1158
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/BIGTOP/How+to+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g., 'BIGTOP-3638: Your PR title ...'.
-->
### Description of PR
Currently, Hive3 has some issues, especially following memory leak related issues, which have been blocked the usage of hive in production environment due to frequently OOM. For this reason, I'd like to merge them in patch format to bigtop project.
They are:
[HIVE-16455](https://issues.apache.org/jira/browse/HIVE-16455): ADD JAR command leaks JAR Files
https://issues.apache.org/jira/browse/HIVE-16455
[HIVE-24236](https://issues.apache.org/jira/browse/HIVE-24236)/[HIVE-25483](https://issues.apache.org/jira/browse/HIVE-25483): Connection leak in TxnHandler
https://issues.apache.org/jira/browse/HIVE-24236
https://issues.apache.org/jira/browse/HIVE-25483
[HIVE-24552](https://issues.apache.org/jira/browse/HIVE-24552): Possible HMS connections leak or accumulation in loadDynamicPartitions
https://issues.apache.org/jira/browse/HIVE-24552
[HIVE-24858](https://issues.apache.org/jira/browse/HIVE-24858): UDFClassLoader leak in Configuration.CACHE_CLASSES
https://issues.apache.org/jira/browse/HIVE-24858
[HIVE-26404](https://issues.apache.org/jira/browse/HIVE-26404): HMS memory leak when compaction cleaner fails to remove obsolete files
https://issues.apache.org/jira/browse/HIVE-26404
[HIVE-26530](https://issues.apache.org/jira/browse/HIVE-26530): HS2 OOM-OperationManager.queryIdOperation does not properly clean up multiple queryIds (This PR has been fixed in hive-3.1.3)
https://issues.apache.org/jira/browse/HIVE-26530
### How was this patch tested?
Local smoke test:
1. create test database and test table (TPC-DS: web_sales)
2. load test data (1GB TPC-DS test data) to tables;
3. run select count(1) from web_sales;
<img width="849" alt="image" src="https://github.com/apache/bigtop/assets/12960246/4f9c8076-1d7d-41b3-b031-724b4a5da6b4">
### For code changes:
- [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'BIGTOP-3638. Your PR title ...')?
- [ ] Make sure that newly added files do not have any licensing issues. When in doubt refer to https://www.apache.org/licenses/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] JiaLiangC commented on pull request #1158: BIGTOP-3963: Merge hive memory leak PRs (for hive-3.1.3)
Posted by "JiaLiangC (via GitHub)" <gi...@apache.org>.
JiaLiangC commented on PR #1158:
URL: https://github.com/apache/bigtop/pull/1158#issuecomment-1665264820
+
> which have been blocked the usage of hive in production environment due to frequently OOM
+1
We and our fellow tech community members from other groups have all encountered the frequent OOM (Out of Memory) issues with Hive 3, which resulted in the unavailability of the hive sever2. This problem has been tormenting everyone. Thank you for contributing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] liqinwyyx commented on pull request #1158: BIGTOP-3963: Merge hive memory leak PRs (for hive-3.1.3)
Posted by "liqinwyyx (via GitHub)" <gi...@apache.org>.
liqinwyyx commented on PR #1158:
URL: https://github.com/apache/bigtop/pull/1158#issuecomment-1665433214
Thanks for your contributing. I'm gonna merge this PR into my pre-release branch for new POC environment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] guyuqi commented on pull request #1158: BIGTOP-3963: Merge hive memory leak PRs (for hive-3.1.3)
Posted by "guyuqi (via GitHub)" <gi...@apache.org>.
guyuqi commented on PR #1158:
URL: https://github.com/apache/bigtop/pull/1158#issuecomment-1664894480
Thanks for working on it.
As mentioned in comments in BIGTOP-3963, could you please rebase this PR on branch-3.2 not on master?
Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #1158: BIGTOP-3963: Merge hive memory leak PRs (for hive-3.1.3)
Posted by "iwasakims (via GitHub)" <gi...@apache.org>.
iwasakims commented on PR #1158:
URL: https://github.com/apache/bigtop/pull/1158#issuecomment-1708165706
> In one of our production environment, with about thousands of GB data, about 2~3 days, the HiveServer2s were running OOM.
I spent some days to analysis the JVM dumps, compare the information from dumps and logs with the hive PRs, and applied those patches finally. After the patches applied, no OOM found any more.
You mean all 5 patches are tested in your production? If patch20-memleak-HIVE-26530-branch-3.1.diff is not related to your OOM in the field, it should be excluded. @rzuo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] BIGTOP-3963: Merge hive memory leak PRs (for hive-3.1.3) [bigtop]
Posted by "kevinw66 (via GitHub)" <gi...@apache.org>.
kevinw66 commented on PR #1158:
URL: https://github.com/apache/bigtop/pull/1158#issuecomment-1751556132
Thanks @rzuo , I also met some OOM problems on production environment, will test this pr
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] rzuo commented on pull request #1158: BIGTOP-3963: Merge hive memory leak PRs (for hive-3.1.3)
Posted by "rzuo (via GitHub)" <gi...@apache.org>.
rzuo commented on PR #1158:
URL: https://github.com/apache/bigtop/pull/1158#issuecomment-1666427541
I added HIVE-26530 patch to this pr, since it has not been merged into hive 3.1.3 release.
*************** important notice ***************
But this patch has been merge to hive git branch-3.1.
*************** important notice ***************
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #1158: BIGTOP-3963: Merge hive memory leak PRs (for hive-3.1.3)
Posted by "iwasakims (via GitHub)" <gi...@apache.org>.
iwasakims commented on PR #1158:
URL: https://github.com/apache/bigtop/pull/1158#issuecomment-1664916929
@guyuqi I think merging the patch to master is ok now. We can just drop the patches when we bump Hive to 4.
@rzuo Could you run unit tests of Hive to see if there is regression? You should run smoke-tests of Bigtop too after that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] BIGTOP-3963: Merge hive memory leak PRs (for hive-3.1.3) [bigtop]
Posted by "kevinw66 (via GitHub)" <gi...@apache.org>.
kevinw66 commented on PR #1158:
URL: https://github.com/apache/bigtop/pull/1158#issuecomment-1776894835
https://issues.apache.org/jira/browse/HIVE-24590
After testing on my prod env, I need one more patch to fix my OOM problem, could you add this too? @rzuo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] rzuo commented on pull request #1158: BIGTOP-3963: Merge hive memory leak PRs (for hive-3.1.3)
Posted by "rzuo (via GitHub)" <gi...@apache.org>.
rzuo commented on PR #1158:
URL: https://github.com/apache/bigtop/pull/1158#issuecomment-1665293801
> > Local smoke test:
> > create test database and test table (TPC-DS: web_sales)
> > load test data (1GB TPC-DS test data) to tables;
> > run select count(1) from web_sales;
>
> @rzuo Could you elaborate the purpose and coverage of this test? Does this cover the all added patches? How much memory consumption is changed after the patch is applied?
Hi iwasakims,
By run one single smoke test SQL, just to verify hive works normally.
Basiclly, it's difficult to cover all added patches by running one single sql query. The patches cover different business scenarioes, also need much more data, time, different kinds of queries to verify them completely.
In one of our production environment, with about thousands of GB data, about 2~3 days, the HiveServer2s were running OOM.
I spent some days to analysis the JVM dumps, compare the information from dumps and logs with the hive PRs, and applied those patches finally. After the patches applied, no OOM found any more.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] BIGTOP-3963: Merge hive memory leak PRs (for hive-3.1.3) [bigtop]
Posted by "kevinw66 (via GitHub)" <gi...@apache.org>.
kevinw66 closed pull request #1158: BIGTOP-3963: Merge hive memory leak PRs (for hive-3.1.3)
URL: https://github.com/apache/bigtop/pull/1158
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #1158: BIGTOP-3963: Merge hive memory leak PRs (for hive-3.1.3)
Posted by "iwasakims (via GitHub)" <gi...@apache.org>.
iwasakims commented on PR #1158:
URL: https://github.com/apache/bigtop/pull/1158#issuecomment-1664920195
> Local smoke test:
>
> create test database and test table (TPC-DS: web_sales)
> load test data (1GB TPC-DS test data) to tables;
> run select count(1) from web_sales;
@rzuo Could you elaborate the purpose and coverage of this test? Does this cover the all added patches? How much memory consumption is changed after the patch is applied?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] BIGTOP-3963: Merge hive memory leak PRs (for hive-3.1.3) [bigtop]
Posted by "kevinw66 (via GitHub)" <gi...@apache.org>.
kevinw66 commented on PR #1158:
URL: https://github.com/apache/bigtop/pull/1158#issuecomment-2041390382
Thanks @rzuo , the work is now moving to #1248 , I'll close the PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] BIGTOP-3963: Merge hive memory leak PRs (for hive-3.1.3) [bigtop]
Posted by "rzuo (via GitHub)" <gi...@apache.org>.
rzuo commented on PR #1158:
URL: https://github.com/apache/bigtop/pull/1158#issuecomment-2031358423
> Hi @rzuo , what's the progress here? Can I take this over if you don't have time for this?
That's fine, thanks @kevinw66
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] BIGTOP-3963: Merge hive memory leak PRs (for hive-3.1.3) [bigtop]
Posted by "kevinw66 (via GitHub)" <gi...@apache.org>.
kevinw66 commented on PR #1158:
URL: https://github.com/apache/bigtop/pull/1158#issuecomment-2031348556
Hi @rzuo , what's the progress here? Can I take this over if you don't have time for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org