You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Attila Doroszlai (Jira)" <ji...@apache.org> on 2026/06/11 08:32:00 UTC
[jira] [Commented] (HDDS-12669) Race condition between entries of ContainerSet#recoveringContainerMap
[ https://issues.apache.org/jira/browse/HDDS-12669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18088205#comment-18088205 ]
Attila Doroszlai commented on HDDS-12669:
-----------------------------------------
This turned out to be a real bug:
bq. Currently in ContainerSet.java, recoveringContainerMap records recovering containers and identifies them by their timeout values. However, this introduces a issue: if two or more containers start recovering at the exact same time, they will have identical timeout values. Because it's a map, the newer entry overwrites the older one. As a result, the overwritten container is silently dropped from the tracking map. If the actual recovery action for this untracked container stucks, the StaleRecoveringContainerScrubbingService will be unaware of it and cannot trigger the timeout cleanup. Consequently, the container becomes permanently orphaned and stuck in the 'recovering' state.
> Race condition between entries of ContainerSet#recoveringContainerMap
> ---------------------------------------------------------------------
>
> Key: HDDS-12669
> URL: https://issues.apache.org/jira/browse/HDDS-12669
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Attila Doroszlai
> Assignee: Chung-En Lee
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.3.0
>
>
> {code}
> at org.apache.ozone.test.GenericTestUtils.waitFor(GenericTestUtils.java:152)
> at org.apache.hadoop.ozone.container.TestECContainerRecovery.testECContainerRecoveryWithTimedOutRecovery(TestECContainerRecovery.java:350)
> {code}
> {code:title=https://github.com/apache/ozone/blob/ebf5cc662ba6055fb8a18ad9036cbeabd3c49c29/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/TestECContainerRecovery.java#L350-L351}
> GenericTestUtils.waitFor(() -> reconstructedDN.get() != null, 10000,
> 100000);
> {code}
> * https://github.com/adoroszlai/ozone-build-results/blob/master/2025/01/11/35926/it-container/hadoop-ozone/integration-test/org.apache.hadoop.ozone.container.TestECContainerRecovery.txt
> * https://github.com/adoroszlai/ozone-build-results/blob/master/2025/02/19/36861/it-container/hadoop-ozone/integration-test/org.apache.hadoop.ozone.container.TestECContainerRecovery.txt
> * https://github.com/adoroszlai/ozone-build-results/blob/master/2025/03/17/37636/integration-container/hadoop-ozone/integration-test/org.apache.hadoop.ozone.container.TestECContainerRecovery.txt
> * https://github.com/adoroszlai/ozone-build-results/blob/master/2025/03/22/37865/integration-container/hadoop-ozone/integration-test/org.apache.hadoop.ozone.container.TestECContainerRecovery.txt
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org