You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "chenruotao (Jira)" <ji...@apache.org> on 2023/11/06 11:49:00 UTC
[jira] [Work started] (HIVE-27854) Driver compile may causing memory leak
[ https://issues.apache.org/jira/browse/HIVE-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on HIVE-27854 started by chenruotao.
-----------------------------------------
> Driver compile may causing memory leak
> --------------------------------------
>
> Key: HIVE-27854
> URL: https://issues.apache.org/jira/browse/HIVE-27854
> Project: Hive
> Issue Type: Bug
> Components: Hive
> Affects Versions: 3.1.2, 4.0.0-beta-1
> Reporter: chenruotao
> Assignee: chenruotao
> Priority: Major
> Labels: pull-request-available
> Attachments: image-2023-11-06-18-11-47-832.png, image-2023-11-06-18-13-00-842.png, image-2023-11-06-18-13-43-182.png, image-2023-11-06-18-21-19-056.png, image-2023-11-06-18-22-45-812.png, image-2023-11-06-18-24-41-982.png, image-2023-11-06-19-30-14-120.png, image-2023-11-06-19-40-49-488.png, image-2023-11-06-19-45-03-388.png
>
>
> when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute the addShutdownHook method, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak
> !image-2023-11-06-18-11-47-832.png|width=943,height=241!
> !image-2023-11-06-19-45-03-388.png!
> the cancel from SQLOperation will set it's driver = null, so it can not call the close method of driver after SQLOperation exexute cancel
> !image-2023-11-06-18-13-43-182.png!
> !image-2023-11-06-18-13-00-842.png!
> it will not trigger everytime, but we could do this to trigger it
> 1、sleep 5s before compile
> !image-2023-11-06-18-21-19-056.png!
> 2、set hive.query.timeout.seconds to 4
> !image-2023-11-06-18-22-45-812.png!
> 3、now everytime I submit sql , the shutdownHookManager$HookEntry will add one
> !image-2023-11-06-18-24-41-982.png|width=842,height=134!
> i solve it in hive 3.1.2 by adding this code :
> !image-2023-11-06-19-30-14-120.png!
> Obviously we see no more memory leak, and shutdownHookManager$HookEntry num will not increase every time i submit sql
> !image-2023-11-06-19-40-49-488.png|width=956,height=140!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)