You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Attila Doroszlai (Jira)" <ji...@apache.org> on 2022/10/14 12:21:00 UTC
[jira] [Resolved] (HDDS-7300) Race condition between full data scan and block deletion
[ https://issues.apache.org/jira/browse/HDDS-7300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Attila Doroszlai resolved HDDS-7300.
------------------------------------
Fix Version/s: 1.3.0
Resolution: Fixed
> Race condition between full data scan and block deletion
> --------------------------------------------------------
>
> Key: HDDS-7300
> URL: https://issues.apache.org/jira/browse/HDDS-7300
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Xu Shao Hong
> Assignee: Xu Shao Hong
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.3.0
>
>
> We have enabled the full data scan and found that one container is marked as unhealthy due to the conflict between full data scan and block deletion.
> The block deleting service first deletes the block and then updates the DB, while the data scan first scans the DB and then checks the existence of the blocks.
> Once getting the DB record and finds the block not existing in the FS, the `Missing chunk file exception` will be thrown and the container will be marked as unhealthy.
>
> *The block deleting service has a write lock during the process but the data scan has no read lock to avoid the conflict.*
> Even by double checking the block if the block is still in the block-data table when the block is not found on the FS for the first time, the problem still happens. The flush time of DB batch operation is not predictable, so the direct second retrieval may not be a good solution as we cannot determine a fixed delay that could guarantee every batch could be flushed after this delay.
>
> *The log trace:*
> * 2022-09-30 16:07:38,535 [BlockDeletingService#5] INFO org.apache.hadoop.ozone.container.keyvalue.impl.FilePerBlockStrategy: Deleted block file: /data11/ozone-ec/data/storage/hdds/CID-9090d68f-eb34-44f0-b54f-10df5e42a347/current/containerDir12/6595/chunks/109611004723333878.block
>
> * 2022-09-30 16:07:39,244 [ContainerDataScanner(/data11/ozone-ec/data/storage/hdds)] ERROR org.apache.hadoop.ozone.container.keyvalue.KeyValueContainerCheck: Corruption detected in container: [6595] Exception: [Missing chunk file /data11/ozone-ec/data/storage/hdds/CID-9090d68f-eb34-44f0-b54f-10df5e42a347/current/containerDir12/6595/chunks/109611004723333878.block]
>
> * 2022-09-30 16:07:39,545 [ContainerDataScanner(/data11/ozone-ec/data/storage/hdds)] WARN org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer: Moving container /data11/ozone-ec/data/storage/hdds/CID-9090d68f-eb34-44f0-b54f-10df5e42a347/current/containerDir12/6595 to state UNHEALTHY from state:UNHEALTHY Trace:java.lang.Thread.getStackTrace(Thread.java:1559) org.apache.hadoop.util.StringUtils.getStackTrace(StringUtils.java:1060) org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.markContainerUnhealthy(KeyValueContainer.java:340) org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.markContainerUnhealthy(KeyValueHandler.java:1017) org.apache.hadoop.ozone.container.ozoneimpl.ContainerController.markContainerUnhealthy(ContainerController.java:116) org.apache.hadoop.ozone.container.ozoneimpl.ContainerDataScanner.scanContainer(ContainerDataScanner.java:72) org.apache.hadoop.ozone.container.ozoneimpl.AbstractContainerScanner.scanContainers(AbstractContainerScanner.java:99) org.apache.hadoop.ozone.container.ozoneimpl.AbstractContainerScanner.runIteration(AbstractContainerScanner.java:74)
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org