[jira] [Comment Edited] (HIVE-23164) Server is not properly terminated because of non-daemon threads

["Zoltan Haindrich (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-23164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432360#comment-17432360 ] 

Zoltan Haindrich edited comment on HIVE-23164 at 10/21/21, 10:54 AM:
---------------------------------------------------------------------

[~lpinter] have noticed that this patch has some sideffect:

this patch have added a {{ThreadPool#shutdown}} to {{HMSHandler#shutdown}} - which could cause trouble in case a {{HMSHandler}} is shutdown and a new one is created

I was looking for cases in which a HMSHandler is created inside the metastore (beyond the one HiveMetaStore is using) - and I think tasks like Msck use it to access the metastore - and they close the client - which closes the hmshandler client ; which will shut down the threadpool

opened: HIVE-25633


was (Author: kgyrtkirk):
this patch have added a {{ThreadPool#shutdown}} to {{HMSHandler#shutdown}} - which could cause trouble in case a {{HMSHandler}} is shutdown and a new one is created

> Server is not properly terminated because of non-daemon threads
> ---------------------------------------------------------------
>
>                 Key: HIVE-23164
>                 URL: https://issues.apache.org/jira/browse/HIVE-23164
>             Project: Hive
>          Issue Type: Bug
>          Components: HiveServer2
>            Reporter: Eugene Chung
>            Assignee: Eugene Chung
>            Priority: Major
>             Fix For: 4.0.0
>
>         Attachments: HIVE-23164.01.patch, HIVE-23164.02.patch, HIVE-23164.03.patch, HIVE-23164.04.patch, thread_dump_hiveserver2_is_not_terminated.txt
>
>
> HiveServer2 which receives the deregister command is at first preparing shutdown. If there's no remaining session, HiveServer2.stop() is called to shut down. But I found the case that the HiveServer2 JVM is not terminated even if HiveServer2.stop() has been called and processed. The case is always occurred when the local(embedded) metastore is used.
> I've attached the full thread dump describing the situation.
> [^thread_dump_hiveserver2_is_not_terminated.txt]
> In this thread dump, you can see some bunch of 'daemon' threads, NO main thread, and some 'non-daemon' thread(or user thread)s. As specified by [https://www.baeldung.com/java-daemon-thread], if there is at least one user thread exists, JVM does not terminate. (Note that DestroyJavaVM thread is non-daemon but it's special.)
>  
> {code:java}
> "pool-8-thread-1" #24 prio=5 os_prio=0 tid=0x00007f52ad1fc000 nid=0x821c waiting on condition [0x00007f525c500000]
>  java.lang.Thread.State: TIMED_WAITING (parking)
>  at sun.misc.Unsafe.park(Native Method)
>  - parking to wait for <0x00000003cfa057c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>  at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>  at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>  at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>  at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
>  - None
> {code}
> The thread above is created by the ScheduledThreadPoolExecutor(int coreSize) constructor with default ThreadFactory which always makes a thread non-daemon. If such thread pool is not destroyed by calling ScheduledThreadPoolExecutor.shutdown() method, JVM cannot terminate! The only way to kill is TERM signal. If JVM receives TERM signal, it ignores non-daemon threads and terminates.
> So I have been digging modules which create ScheduledThreadPoolExecutor with non-daemon threads and I got it. As you may guess, it's the local(embedded) metastore. ScheduledThreadPoolExecutor is created by org.apache.hadoop.hive.metastore.HiveMetaStore.HMSHandler#startAlwaysTaskThreads() and ScheduledThreadPoolExecutor.shutdown() is never called.
> Plus, I found another usage of creating such ScheduledThreadPoolExecutor and not calling its shutdown. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)