[jira] [Created] (HDDS-6721) Ozone key deletion often dose not delete the real data blocks in datanode

["Shawn (Jira)" <ji...@apache.org>]

Shawn created HDDS-6721:
---------------------------

             Summary: Ozone key deletion often dose not delete the real data blocks in datanode
                 Key: HDDS-6721
                 URL: https://issues.apache.org/jira/browse/HDDS-6721
             Project: Apache Ozone
          Issue Type: Bug
            Reporter: Shawn


When I delete a key of ozone (by using either ozone cli or aws cli), it seems ozone does not clean up the real data blocks in datanodes. The key is gone at the meta data layer (OM), but not at datanode layer.

I did the following investigation. 

OM layer: it seems OM functions correctly. I do see the log saying it deletes the block. Also The rocksDB table deletedTable is empty.

SCM layer: it does receive the deletion command from OM. And it tries to delete the block. But it seems it only add the transaction in memory, and dose not record the transaction in the rocksDB, and thus it does not call datanode to do the deletion. 

* No log likes "Totally added x blocks to be deleted for y datanodes" in SCM leader node. 
* The deletedBlocks rocksDB table does not have the info of the containerID being deleted. 
* And in audit log, it has the log "2022-05-09 18:18:46,553 | INFO  | SCMAudit | user=om/om@TEST.OZONE-DEV.SVC.K8S.ABC.COM | ip=100.116.76.9 | op=DELETE_KEY_BLOCK {KeyBlockToDelete=BlockGroup[groupID='/s3v/shawn-test2/tmp/pvc.yaml', blockIDs=[conID: 9001 locID: 109611004723209001 bcsId: 0]]} | ret=SUCCESS |"

Datanode layer: not receive any command from SCM to delete the blocks. The blocks of the deleting container still exists on disk. The rocksDB table delete_txns of that container is empty.

I also tried to restart the SCM to flush the ratis logs. But it ends up the same as above.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org