[jira] [Created] (YARN-2874) Dead lock in "DelegationTokenRenewer" which blocks RM to execute any further apps

["Naganarasimha G R (JIRA)" <ji...@apache.org>]

Naganarasimha G R created YARN-2874:
---------------------------------------

             Summary: Dead lock in "DelegationTokenRenewer" which blocks RM to execute any further apps
                 Key: YARN-2874
                 URL: https://issues.apache.org/jira/browse/YARN-2874
             Project: Hadoop YARN
          Issue Type: Bug
          Components: resourcemanager
    Affects Versions: 2.5.1, 2.4.1, 2.5.0
            Reporter: Naganarasimha G R
            Assignee: Naganarasimha G R
            Priority: Critical


When token renewal fails and the application finishes this dead lock can occur
Jstack dump :
{quote}
Found one Java-level deadlock:
=============================
"DelegationTokenRenewer #181865":
  waiting to lock monitor 0x0000000000900918 (object 0x00000000c18a9998, a java.util.Collections$SynchronizedSet),
  which is held by "DelayedTokenCanceller"
"DelayedTokenCanceller":
  waiting to lock monitor 0x0000000004141718 (object 0x00000000c7eae720, a org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask),
  which is held by "Timer-4"
"Timer-4":
  waiting to lock monitor 0x0000000000900918 (object 0x00000000c18a9998, a java.util.Collections$SynchronizedSet),
  which is held by "DelayedTokenCanceller"
 
Java stack information for the threads listed above:
===================================================
"DelegationTokenRenewer #181865":
at java.util.Collections$SynchronizedCollection.add(Collections.java:1636)
- waiting to lock <0x00000000c18a9998> (a java.util.Collections$SynchronizedSet)
at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.addTokenToList(DelegationTokenRenewer.java:322)
at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:398)
at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$500(DelegationTokenRenewer.java:70)
at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:657)
at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:638)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"DelayedTokenCanceller":
at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask.cancel(DelegationTokenRenewer.java:443)
- waiting to lock <0x00000000c7eae720> (a org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask)
at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.removeApplicationFromRenewal(DelegationTokenRenewer.java:558)
- locked <0x00000000c18a9998> (a java.util.Collections$SynchronizedSet)
at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$300(DelegationTokenRenewer.java:70)
at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelayedTokenRemovalRunnable.run(DelegationTokenRenewer.java:599)
at java.lang.Thread.run(Thread.java:745)
"Timer-4":
at java.util.Collections$SynchronizedCollection.remove(Collections.java:1639)
- waiting to lock <0x00000000c18a9998> (a java.util.Collections$SynchronizedSet)
at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.removeFailedDelegationToken(DelegationTokenRenewer.java:503)
at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$100(DelegationTokenRenewer.java:70)
at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask.run(DelegationTokenRenewer.java:437)
- locked <0x00000000c7eae720> (a org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
 
Found 1 deadlock.
{quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)