[jira] [Commented] (HIVE-17586) Make HS2 BackgroundOperationPool not fixed

["Hive QA (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-17586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16179927#comment-16179927 ] 

Hive QA commented on HIVE-17586:
--------------------------------



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12888949/HIVE-17586.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 11055 tests executed
*Failed tests:*
{noformat}
TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=231)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=231)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[empty_join] (batchId=76)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats] (batchId=156)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1] (batchId=170)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] (batchId=235)
org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver.org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver (batchId=242)
org.apache.hive.service.cli.session.TestSessionManagerMetrics.testThreadPoolMetrics (batchId=197)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6979/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6979/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6979/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12888949 - PreCommit-HIVE-Build

> Make HS2 BackgroundOperationPool not fixed
> ------------------------------------------
>
>                 Key: HIVE-17586
>                 URL: https://issues.apache.org/jira/browse/HIVE-17586
>             Project: Hive
>          Issue Type: Bug
>          Components: HiveServer2
>    Affects Versions: 1.1.0
>            Reporter: Xuefu Zhang
>            Assignee: Xuefu Zhang
>         Attachments: HIVE-17586.patch
>
>
> Currently the threadpool for background asynchronous operatons has a fixed size controled by {{hive.server2.async.exec.threads}}. However, the thread factory supplied for this threadpool is {{ThreadFactoryWithGarbageCleanup}} which creates ThreadWithGarbageCleanup. Since this is a fixed threadpool, the thread is actually never killed, defecting the purpose of garbage cleanup as noted in the thread class name. On the other hand, since these threads never go away, significant resources such as threadlocal variables (classloaders, hiveconfs, etc) are holding up even if there is no operation running. This can lead to escalated HS2 memory usage.
> Ideally, the threadpool should not be fixed, allowing thread to die out so resources can be reclaimed. The existing config {{hive.server2.async.exec.threads}} is treated as the max, and we can add a min for the threadpool {{hive.server2.async.exec.min.threads}}. Default value for this configure is -1, which keeps the existing behavior.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)