You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2022/08/10 03:13:12 UTC
[GitHub] [ozone] Xushaohong opened a new pull request, #3670: HDDS-7111. SCMHADBTransactionBuffer not flushed in time
Xushaohong opened a new pull request, #3670:
URL: https://github.com/apache/ozone/pull/3670
## What changes were proposed in this pull request?
We recently found that after we deleted all the keys, the cluster will remain with some data. One of the reasons is due to this:
SCM will add deleted blocks into transactions when receiving the request from OM. When HA is enabled, DBTransactionbuffer is implemented as the SCMHADBTransactionbufferImpl. Inside this, the buffer will not be flushed immediately. Normally, it will be flushed when SCM takes a snapshot. The snapshot gap threshold is default 1000. If the user has little load pressure on the cluster (no writing more ratis logs), the buffer will be always pending in the memory. Real deletion happened in SCMBlockDeletingService, which will scan the DB and get the transactions, for those txns in-memory buffer, it won't find them. This is why DN has not yet received these deleted block info.
This PR adds a flush monitor checking regularly to trigger the non-empty flush, overall, this could be a common mechanism if some other cases like the deleted blocks appear in the future.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-7111
## How was this patch tested?
UT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] Xushaohong commented on pull request #3670: HDDS-7111. SCMHADBTransactionBuffer not flushed in time
Posted by GitBox <gi...@apache.org>.
Xushaohong commented on PR #3670:
URL: https://github.com/apache/ozone/pull/3670#issuecomment-1216199295
> Hi @Xushaohong. The issue you've brought up is definitely valid. We have discussed this as well in [HDDS-6721](https://issues.apache.org/jira/browse/HDDS-6721). After some discussion the preferred solution in that Jira was to add a time based Ratis snapshot feature in [RATIS-1583](https://issues.apache.org/jira/browse/RATIS-1583), which has not been implemented yet. IMO time based Ratis snapshot is the easiest way forward because it will provide a general purpose Ratis feature that we can use with a single config. For example, have Ratis take a snapshot (flush to DB) every 1000 transactions or every 10 minutes (arbitrary example value), whichever comes first.
Make sense, but the refactor in ratis need a long time comes into ozone. If i have time, i will take a look. Close this PR now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] errose28 closed pull request #3670: HDDS-7111. SCMHADBTransactionBuffer not flushed in time
Posted by GitBox <gi...@apache.org>.
errose28 closed pull request #3670: HDDS-7111. SCMHADBTransactionBuffer not flushed in time
URL: https://github.com/apache/ozone/pull/3670
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] errose28 commented on pull request #3670: HDDS-7111. SCMHADBTransactionBuffer not flushed in time
Posted by GitBox <gi...@apache.org>.
errose28 commented on PR #3670:
URL: https://github.com/apache/ozone/pull/3670#issuecomment-1211366982
Hi @Xushaohong. The issue you've brought up is definitely valid. We have discussed this as well in HDDS-6721. After some discussion the preferred solution in that Jira was to add a time based Ratis snapshot feature in [RATIS-1583](https://issues.apache.org/jira/browse/RATIS-1583), which has not been implemented yet. IMO time based Ratis snapshot is the easiest way forward because it will provide a general purpose Ratis feature that we can use with a single config. For example, have Ratis take a snapshot (flush to DB) every 1000 transactions or every 10 minutes (arbitrary example value), whichever comes first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] Xushaohong commented on pull request #3670: HDDS-7111. SCMHADBTransactionBuffer not flushed in time
Posted by GitBox <gi...@apache.org>.
Xushaohong commented on PR #3670:
URL: https://github.com/apache/ozone/pull/3670#issuecomment-1210100980
@ChenSammi PTAL~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] errose28 commented on pull request #3670: HDDS-7111. SCMHADBTransactionBuffer not flushed in time
Posted by GitBox <gi...@apache.org>.
errose28 commented on PR #3670:
URL: https://github.com/apache/ozone/pull/3670#issuecomment-1216799827
Thanks @Xushaohong, I will close this for now. If you need a temporary workaround to get data deleted you can restart the SCMs to force the logs to flush.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org