You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Zhu Zhu (JIRA)" <ji...@apache.org> on 2019/07/25 08:49:00 UTC
[jira] [Updated] (FLINK-13421) Unexpected
ConcurrentModificationException when RM notifies JM about allocation
failure
[ https://issues.apache.org/jira/browse/FLINK-13421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhu Zhu updated FLINK-13421:
----------------------------
Summary: Unexpected ConcurrentModificationException when RM notifies JM about allocation failure (was: Unexpected ConcurrentModificationException when RM notify JM about allocation failure)
> Unexpected ConcurrentModificationException when RM notifies JM about allocation failure
> ---------------------------------------------------------------------------------------
>
> Key: FLINK-13421
> URL: https://issues.apache.org/jira/browse/FLINK-13421
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
> Affects Versions: 1.9.0
> Reporter: Zhu Zhu
> Priority: Blocker
> Attachments: jm_concurrentmodification.log
>
>
> When a TM lost and RM identified it first, it will notify JM about it through JobMaster#notifyAllocationFailure.
> We observed unexpected ConcurrentModificationException in this process, stack as below:
>
> Caused by: java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
> at java.util.HashMap$ValueIterator.next(HashMap.java:1466)
> at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:477)
> at org.apache.flink.runtime.jobmaster.slotpool.AllocatedSlot.releasePayload(AllocatedSlot.java:149)
> at org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.tryFailingAllocatedSlot(SlotPoolImpl.java:712)
> at org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.failAllocation(SlotPoolImpl.java:692)
> at org.apache.flink.runtime.jobmaster.JobMaster.internalFailAllocation(JobMaster.java:538)
> at org.apache.flink.runtime.jobmaster.JobMaster.notifyAllocationFailure(JobMaster.java:664)
> ... 26 more
>
> This can cause an allocated slot to be removed from SlotPool#allocatedSlots but not all of its payload tasks get failed. Tasks may hang in scheduled forever, not able to fail or timeout, as in the attached log.
> It is not figured out yet that how a concurrent modification can happen. We do not have a debug log for it and is not able to re-pro it with debug log enabled yet.
> However, we can let SlotSharingManager$MultiTaskSlot do not iterate on its children directly to avoid ConcurrentModificationException to occur in any case.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)