[jira] [Commented] (SENTRY-1526) Sentry processed stays alive after being killed

["Misha Dmitriev (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SENTRY-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15753666#comment-15753666 ] 

Misha Dmitriev commented on SENTRY-1526:
----------------------------------------

In short, the problem can be fixed by removing 'synchronized' from the SentryService.waitOnFuture() method. This method doesn't need synchronization, because nobody calls it concurrently, nor we need calls to it be mutually exclusive with other synchronized calls in SentryService class. And the fact that it's synchronized prevents the shutdown mechanism from working properly, as explained below.

To diagnose the problem, I ran 'jstack' after calling Ctrl+C. It turns out that many threads still exist. Here are the three relevant threads:

{code}
"Thread-1" #274 prio=5 os_prio=31 tid=0x00007fc2e115b000 nid=0xa703 waiting for monitor entry [0x0000700003f50000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at org.apache.sentry.service.thrift.SentryService.stop(SentryService.java:276)
    - waiting to lock <0x00000006c1850620> (a org.apache.sentry.service.thrift.SentryService)
    at org.apache.sentry.service.thrift.SentryService$CommandImpl$1.run(SentryService.java:390)

"SIGINT handler" #311 daemon prio=9 os_prio=31 tid=0x00007fc2e1159000 nid=0xa183 in Object.wait() [0x0000700003b44000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.Thread.join(Thread.java:1245)
    - locked <0x00000006c160f320> (a org.apache.sentry.service.thrift.SentryService$CommandImpl$1)
    at java.lang.Thread.join(Thread.java:1319)
    at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:106)
    at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46)
    at java.lang.Shutdown.runHooks(Shutdown.java:123)
    at java.lang.Shutdown.sequence(Shutdown.java:167)
    at java.lang.Shutdown.exit(Shutdown.java:212)
    - locked <0x00000006c04baf88> (a java.lang.Class for java.lang.Shutdown)
    at java.lang.Terminator$1.handle(Terminator.java:52)
    at sun.misc.Signal$1.run(Signal.java:212)
    at java.lang.Thread.run(Thread.java:745)

"org.apache.sentry.SentryMain.main()" #11 prio=5 os_prio=31 tid=0x00007f8f5cb4c000 nid=0x5603 waiting on condition [0x00007000015d4000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000006c1b2a740> (a java.util.concurrent.FutureTask)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
    at java.util.concurrent.FutureTask.get(FutureTask.java:191)
    at org.apache.sentry.service.thrift.SentryService.waitOnFuture(SentryService.java:313)
    - locked <0x00000006c1b2a778> (a org.apache.sentry.service.thrift.SentryService)
    at org.apache.sentry.service.thrift.SentryService$CommandImpl.run(SentryService.java:399)
    at org.apache.sentry.SentryMain.main(SentryMain.java:114)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:291)
    at java.lang.Thread.run(Thread.java:745)
{code}

The thread that's responsible for starting shutdown hook(s) (#311) waits for the shutdown hook (that in turn runs in a separate thread), to finish. The shutdown hook code runs in thread #274. That thread is in turn stuck. It cannot enter synchronized method SentryService.stop(), waiting on lock 0x00000006c160f320, which is associated with the instance of SentryService class. This lock, it turns out, is being held by the third thread (#11, the main one), because this thread is inside a chain of calls with the 'synchronized waitOnFuture()' method in the middle. So, until the main thread exits waitOnFuture(), stop() cannot proceed. In other words, thread A that wants to stop thread B needs to grab lock X that B will never free until B is stopped.

If B (the main thread) needed to hold this lock for some good reason, it could be a non-trivial problem to solve. But fortunately, here it looks like the main thread doesn't need it for any good reason. Perhaps someone just added that 'synchronized' to waitOnFuture() at some point in the past for perceived consistency or some similar consideration, and accidentally broke shutdown. When 'synchronized' is removed from waitOnFuture(), shutdown upon Ctrl+C immediately works as expected.

> Sentry processed stays alive after being killed
> -----------------------------------------------
>
>                 Key: SENTRY-1526
>                 URL: https://issues.apache.org/jira/browse/SENTRY-1526
>             Project: Sentry
>          Issue Type: Bug
>          Components: Sentry
>    Affects Versions: 1.8.0, sentry-ha-redesign
>            Reporter: Alexander Kolbasov
>
> When a running Sentry daemon receives SIGINT it doesn't terminate.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)