[GitHub] [hadoop-ozone] lokeshj1703 commented on pull request #1459: HDDS-4297. Allow multiple transactions per container to be sent for deletion by SCM.

[GitBox <gi...@apache.org>]

lokeshj1703 commented on pull request #1459:
URL: https://github.com/apache/hadoop-ozone/pull/1459#issuecomment-711778460


   @linyiqun currently we allow block deletion for only closed containers. For open containers deletion needs to be done via ratis pipeline which is not currently implemented. So there is no block allocation advantage right now. I do not think there is a huge window where containers are open and are simultaneously receiving block deletion requests in bulk.
   Further it is better to accumulate transactions of a container if possible. There is an overhead to handling a block deletion request for datanode. It needs to load rockdb instance and then add those transactions to the db. It is better for datanode to process multiple such transactions at once.
   I tested the PR and found that it gives better deletion speed for deleting a milllion keys. 
   Without PR - SCM sends total 180 blocks for deletion in a minute. This number does not increase even after increasing configs. Datanode deletion speed was very slow. I monitored a datanode disk space and it deleted less than a GB even after 5 minutes.
   With PR - SCM sends 6000 blocks for deletion. Datanode was deleting 1 GB of data every minute.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org