[jira] [Commented] (IMPALA-7482) Deadlock with unknown lock holder in JVM in java.security.Provider.getService()

["Tim Armstrong (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-7482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590981#comment-16590981 ] 

Tim Armstrong commented on IMPALA-7482:
---------------------------------------

A few of us spent a while trying to debug this on a cluster without making too much headway. I think the key things we figured out were:
* We've only ever seen this in an Impalad's embedded JVM
* We've only seen it for sun.security.provider.Sun
* We can't reliably reproduce it
* It could be misuse of the JNI APIs somehow or memory corruption
* It's probably not just random memory corruption because that wouldn't explain why it's always the same object
* The memory address of the mystery owner (0x0000000014120800 in the example) is a valid pointer

> Deadlock with unknown lock holder in JVM in java.security.Provider.getService()
> -------------------------------------------------------------------------------
>
>                 Key: IMPALA-7482
>                 URL: https://issues.apache.org/jira/browse/IMPALA-7482
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Backend
>    Affects Versions: Impala 3.0
>            Reporter: Tim Armstrong
>            Assignee: Tim Armstrong
>            Priority: Critical
>              Labels: hang
>         Attachments: vb1220-jstack2.out
>
>
> We've seen several instances of these mystery deadlocks in impalad's embedded JVM. The signature is a deadlock stemming from sun.security.provider.Sun being locked by an unknown owner.
> {noformat}
> Found one Java-level deadlock:
> =============================
> "Thread-24":
>   waiting to lock monitor 0x0000000012364688 (object 0x000000008027ef30, a sun.security.provider.Sun),
>   which is held by UNKNOWN_owner_addr=0x0000000014120800
> {noformat}
> If this happens in HDFS, it causes HDFS I/O to hang and queries to get stuck. If it happens in the Kudu client it also causes hangs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org