You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Devesh Kumar Singh (Jira)" <ji...@apache.org> on 2023/02/07 04:00:00 UTC
[jira] [Commented] (HDDS-7465) Recon - {containerId}/keys API is not giving details of keys for missing containers when recon was down
[ https://issues.apache.org/jira/browse/HDDS-7465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17685044#comment-17685044 ]
Devesh Kumar Singh commented on HDDS-7465:
------------------------------------------
#HDDS-7468 has resolved this issue.
> Recon - {containerId}/keys API is not giving details of keys for missing containers when recon was down
> -------------------------------------------------------------------------------------------------------
>
> Key: HDDS-7465
> URL: https://issues.apache.org/jira/browse/HDDS-7465
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Devesh Kumar Singh
> Assignee: Devesh Kumar Singh
> Priority: Major
> Attachments: image-2022-11-09-09-34-16-930.png
>
>
> Recon - \{containerId}/keys API is not giving details of keys for missing containers when recon was down.
>
> Steps to reproduce:
> # Create a 3 data node cluster and non-HA single SCM cluster
> # Bring Recon down.
> # Create few volumes and buckets and keys to have few (say 5-6) containers created.
> # SCM rocks DB got update of these containers. Make sure that
> "ozone.scm.ratis.enable" property is set as false to immediate flush of containers and their state in SCM rocks DB.
> 5. Bring all the 3 the data nodes down.
> 6. Containers will move from OPEN to CLOSING. Bring all the 3 DNs up and make sure that containers state moved from QUASI CLOSED to CLOSED.
> 7. Bring all the 3 data nodes down again.
> 8. Bring Recon up and all containers should be reported as missing.
> 9. For all missing containers, expand container id to list the key and volume info.
> Expected: Key and volume info should be listed for missing container id.
> Actual: Key and volume is not listed for missing container id.
>
> Root cause: When Recon was down, recon doesn't get any updates (PUT, UPDATE, DELETE) for keys created from OM, so no delta updates of OM gets applied for Recon OM DB and containerKey mapping data is empty.
>
> Solution: Get full periodic snapshot of OM rocks db and and then process the data to get containerKeyMapping info because if Recon was down when containers gets created, full snapshot of OM DB will never gets taken due to below logic as this logic relies on currentSequenceNumber which will never be less than equal to zero as OM DB sequence number has already moved beyond zero when keys were written.
>
> !image-2022-11-09-09-34-16-930.png!
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org