[jira] [Created] (FLINK-13421) Unexpected ConcurrentModificationException when RM notify JM about allocation failure

["Zhu Zhu (JIRA)" <ji...@apache.org>]

Zhu Zhu created FLINK-13421:
-------------------------------

             Summary: Unexpected ConcurrentModificationException when RM notify JM about allocation failure
                 Key: FLINK-13421
                 URL: https://issues.apache.org/jira/browse/FLINK-13421
             Project: Flink
          Issue Type: Bug
            Reporter: Zhu Zhu


When a TM lost and RM identified it first, it will notify JM about it through JobMaster#notifyAllocationFailure. 

We observed unexpected ConcurrentModificationException in this process, stack as below:

 

Caused by: java.util.ConcurrentModificationException

        at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)

        at java.util.HashMap$ValueIterator.next(HashMap.java:1466)

        at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:477)

        at org.apache.flink.runtime.jobmaster.slotpool.AllocatedSlot.releasePayload(AllocatedSlot.java:149)

        at org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.tryFailingAllocatedSlot(SlotPoolImpl.java:712)

        at org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.failAllocation(SlotPoolImpl.java:692)

        at org.apache.flink.runtime.jobmaster.JobMaster.internalFailAllocation(JobMaster.java:538)

        at org.apache.flink.runtime.jobmaster.JobMaster.notifyAllocationFailure(JobMaster.java:664)

        ... 26 more

 

This can cause a allocated slot removed from SlotPool#allocatedSlots but not all of its payload tasks get failed. Tasks may hang in scheduled forever, as in the attached log.

It is not figured out yet that how a concurrent modification can happen. We do not have a debug log for it and is not able to re-pro it with debug log enabled yet.

However, we can let SlotSharingManager$MultiTaskSlot do not iterate on its children directly to avoid ConcurrentModificationException to occur in any case.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)