You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "sodonnel (via GitHub)" <gi...@apache.org> on 2024/04/11 16:31:58 UTC
[PR] HDDS-10681. EC Reconstruction does not issue put block to data index if it is unused [ozone]
sodonnel opened a new pull request, #6514:
URL: https://github.com/apache/ozone/pull/6514
## What changes were proposed in this pull request?
Given a small EC block:
* <= 2MB for EC-3-2
* <= 5MB for EC-6-3
* <= 9MB for EC-10-4
So that it is less than a full stripe and does not use all the datanodes.
When reconstruction happens to replace a replica which is not used in the stripe, the unused containers are not issued with the put block to store the details of the empty block within the container. Note that the container replica will likely have other blocks, so it will still get reconstructed, but it will not be given a reference to this empty block.
All containers are checked for the presence of all blocks during reconstruction. If any of the containers do not have a reference to the block, it is considered an orphan block / abandoned stripe and will not be reconstructed.
Therefore if one replica has no entry for the block, then it is used it another reconstruction for another replica later, that block will not get reconstructed into a second replica. Over time this can result in the reference getting removed from all copies.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-10681
## How was this patch tested?
Reproduced the issue using a new unit test, and then fixed the code to make the test pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10681. EC Reconstruction does not issue put block to data index if it is unused [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #6514:
URL: https://github.com/apache/ozone/pull/6514#discussion_r1562167344
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/storage/TestContainerCommandsEC.java:
##########
@@ -614,8 +614,15 @@ void testECReconstructionCoordinatorWith(List<Integer> missingIndexes)
testECReconstructionCoordinator(missingIndexes, 3);
}
+ @ParameterizedTest
+ @MethodSource("recoverableMissingIndexes")
+ void testECReconstructionCoordinatorWithPartialStripe(List<Integer> missingIndexes)
+ throws Exception {
+ testECReconstructionCoordinator(missingIndexes, 1);
+ }
+
@Test
- void testECReconstructionWithPartialStripe()
+ void testECReconstructParityWithPartialStripe()
throws Exception {
testECReconstructionCoordinator(ImmutableList.of(4, 5), 1);
}
Review Comment:
nit: this can be removed, since `recoverableMissingIndexes` includes `(4, 5)`, so `testECReconstructionCoordinatorWithPartialStripe` covers the same case.
It's OK to do it in #6515, so this can be merged as is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10681. EC Reconstruction does not issue put block to data index if it is unused [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel merged PR #6514:
URL: https://github.com/apache/ozone/pull/6514
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10681. EC Reconstruction does not issue put block to data index if it is unused [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on PR #6514:
URL: https://github.com/apache/ozone/pull/6514#issuecomment-2051307447
Thanks for the review @adoroszlai - merging this to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10681. EC Reconstruction does not issue put block to data index if it is unused [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on code in PR #6514:
URL: https://github.com/apache/ozone/pull/6514#discussion_r1562236029
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/storage/TestContainerCommandsEC.java:
##########
@@ -614,8 +614,15 @@ void testECReconstructionCoordinatorWith(List<Integer> missingIndexes)
testECReconstructionCoordinator(missingIndexes, 3);
}
+ @ParameterizedTest
+ @MethodSource("recoverableMissingIndexes")
+ void testECReconstructionCoordinatorWithPartialStripe(List<Integer> missingIndexes)
+ throws Exception {
+ testECReconstructionCoordinator(missingIndexes, 1);
+ }
+
@Test
- void testECReconstructionWithPartialStripe()
+ void testECReconstructParityWithPartialStripe()
throws Exception {
testECReconstructionCoordinator(ImmutableList.of(4, 5), 1);
}
Review Comment:
Good point. I will fix this when I rebase #6515
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10681. EC Reconstruction does not issue put block to data index if it is unused [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #6514:
URL: https://github.com/apache/ozone/pull/6514#discussion_r1562167344
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/storage/TestContainerCommandsEC.java:
##########
@@ -614,8 +614,15 @@ void testECReconstructionCoordinatorWith(List<Integer> missingIndexes)
testECReconstructionCoordinator(missingIndexes, 3);
}
+ @ParameterizedTest
+ @MethodSource("recoverableMissingIndexes")
+ void testECReconstructionCoordinatorWithPartialStripe(List<Integer> missingIndexes)
+ throws Exception {
+ testECReconstructionCoordinator(missingIndexes, 1);
+ }
+
@Test
- void testECReconstructionWithPartialStripe()
+ void testECReconstructParityWithPartialStripe()
throws Exception {
testECReconstructionCoordinator(ImmutableList.of(4, 5), 1);
}
Review Comment:
nit: this can be removed, since `recoverableMissingIndexes` includes `(4, 5)`, so `testECReconstructionCoordinatorWithPartialStripe` covers the same case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org