You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2022/08/22 08:02:15 UTC
[GitHub] [ozone] symious opened a new pull request, #3703: HDDS-7156. Clean container's inconsistent pendingDeletingBlockCount
symious opened a new pull request, #3703:
URL: https://github.com/apache/ozone/pull/3703
## What changes were proposed in this pull request?
The metadata of "pendingDeleteBlockCount" can be inconsistent in some cases, this ticket is to clean this inconsistency.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-7156
## How was this patch tested?
Not a normal case, works fine in the prod cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on pull request #3703: HDDS-7156. Clean container's inconsistent pendingDeletingBlockCount
Posted by "DaveTeng0 (via GitHub)" <gi...@apache.org>.
DaveTeng0 commented on PR #3703:
URL: https://github.com/apache/ozone/pull/3703#issuecomment-1408121431
hey @symious ~ would you mind providing some cases when you encountered this inconsistent number? That'd be great! Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] symious commented on pull request #3703: HDDS-7156. Clean container's inconsistent pendingDeletingBlockCount
Posted by "symious (via GitHub)" <gi...@apache.org>.
symious commented on PR #3703:
URL: https://github.com/apache/ozone/pull/3703#issuecomment-1408161815
Sorry for so late reply.
It happened quite a long time ago in our cluster, I remembered there are some related error logs indicating the metadata is inconsistent.
Since we applied this patch internally and it seems hard to reproduce the error log again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on pull request #3703: HDDS-7156. Clean container's inconsistent pendingDeletingBlockCount
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on PR #3703:
URL: https://github.com/apache/ozone/pull/3703#issuecomment-1451320670
@symious please take a look at https://github.com/apache/ozone/pull/4324
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] errose28 commented on pull request #3703: HDDS-7156. Clean container's inconsistent pendingDeletingBlockCount
Posted by GitBox <gi...@apache.org>.
errose28 commented on PR #3703:
URL: https://github.com/apache/ozone/pull/3703#issuecomment-1229046923
Hi @symious, could you please explain how the pending delete block count diverges from the correct value? It seems like we should identify the cause of the divergence before deciding the best way to fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] errose28 commented on pull request #3703: HDDS-7156. Clean container's inconsistent pendingDeletingBlockCount
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on PR #3703:
URL: https://github.com/apache/ozone/pull/3703#issuecomment-1444588465
It turns out the pending delete block count can remain high enough on empty containers where it causes starvation in the top N container choosing policy and freezes all pending deletes, making this bug much more serious than I initially thought. I am +1 to merge this type of reset fix with unit tests added, as long as we open a follow-up jira to actually figure out why the pending delete block count number is too high.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] errose28 commented on pull request #3703: HDDS-7156. Clean container's inconsistent pendingDeletingBlockCount
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on PR #3703:
URL: https://github.com/apache/ozone/pull/3703#issuecomment-1444609736
We should make sure this fix works well with [HDDS-7259](https://issues.apache.org/jira/browse/HDDS-7259) and [HDDS-7302](https://issues.apache.org/jira/browse/HDDS-7302) which are newer than this patch, but still don't seem to completely address the problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] arp7 commented on pull request #3703: HDDS-7156. Clean container's inconsistent pendingDeletingBlockCount
Posted by "arp7 (via GitHub)" <gi...@apache.org>.
arp7 commented on PR #3703:
URL: https://github.com/apache/ozone/pull/3703#issuecomment-1444580127
This does need unit tests. We hit this issue locally and @errose28 has a version of the patch with tests - we forgot this review was pending 😞
We should update this patch with Ethan's addendum changes before we commit it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] symious commented on pull request #3703: HDDS-7156. Clean container's inconsistent pendingDeletingBlockCount
Posted by GitBox <gi...@apache.org>.
symious commented on PR #3703:
URL: https://github.com/apache/ozone/pull/3703#issuecomment-1221994304
@adoroszlai Could you help to review?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on pull request #3703: HDDS-7156. Clean container's inconsistent pendingDeletingBlockCount
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #3703:
URL: https://github.com/apache/ozone/pull/3703#issuecomment-1511696817
/ready
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on pull request #3703: HDDS-7156. Clean container's inconsistent pendingDeletingBlockCount
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on PR #3703:
URL: https://github.com/apache/ozone/pull/3703#issuecomment-1418561478
+1 We need to root cause the divergence. This is a good hack to reset and fix the inconsistency but there might be a larger problem at hand.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on pull request #3703: HDDS-7156. Clean container's inconsistent pendingDeletingBlockCount
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on PR #3703:
URL: https://github.com/apache/ozone/pull/3703#issuecomment-1398488557
/pending please explain how the pending delete block count diverges from the correct value
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] errose28 commented on pull request #3703: HDDS-7156. Clean container's inconsistent pendingDeletingBlockCount
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on PR #3703:
URL: https://github.com/apache/ozone/pull/3703#issuecomment-1511714013
This issue was fixed in #HDDS-7156 and the jira is resolved. I think we can close this PR. Please reopen if this is not correct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] errose28 closed pull request #3703: HDDS-7156. Clean container's inconsistent pendingDeletingBlockCount
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 closed pull request #3703: HDDS-7156. Clean container's inconsistent pendingDeletingBlockCount
URL: https://github.com/apache/ozone/pull/3703
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org