You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2022/03/15 11:26:48 UTC
[GitHub] [ozone] sodonnel opened a new pull request #3197: HDDS-6448. EC: Reconstructed Input Streams should free resources after reading to end of block
sodonnel opened a new pull request #3197:
URL: https://github.com/apache/ozone/pull/3197
## What changes were proposed in this pull request?
When a read to ECBlockReconstructedInputStream and ECBlockReconstructedStripeInputStream returns and there are zero bytes remaining in the block, we should free the buffers held due to the issue described in [HDDS-6424](https://issues.apache.org/jira/browse/HDDS-6424), where KeyInputStream opens all its inputStreams upfront and does not close any of them until the entire key is closed.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-6448
## How was this patch tested?
Some modifications to existing tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] umamaheswararao commented on a change in pull request #3197: HDDS-6448. EC: Reconstructed Input Streams should free resources after reading to end of block
Posted by GitBox <gi...@apache.org>.
umamaheswararao commented on a change in pull request #3197:
URL: https://github.com/apache/ozone/pull/3197#discussion_r833477035
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/read/TestECBlockReconstructedStripeInputStream.java
##########
@@ -487,15 +495,8 @@ public void testErrorReadingBlockContinuesReading() throws IOException {
.randomFill(dataBufs, repConfig.getEcChunkSize(), dataGen, blockLength);
ByteBuffer[] parity = generateParity(dataBufs, repConfig);
- List<List<Integer>> failLists = new ArrayList<>();
- failLists.add(indexesToList(0, 1));
- // These will be the first parity read and then the next parity read as a
- // replacement
- failLists.add(indexesToList(2, 3));
- // First parity and then the data block
- failLists.add(indexesToList(2, 0));
-
- for (List<Integer> failList : failLists) {
+ for (int k = 0; k < 5; k++) {
Review comment:
Got it. I have not got full clarity on that test initially....Now I get it. Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] umamaheswararao commented on a change in pull request #3197: HDDS-6448. EC: Reconstructed Input Streams should free resources after reading to end of block
Posted by GitBox <gi...@apache.org>.
umamaheswararao commented on a change in pull request #3197:
URL: https://github.com/apache/ozone/pull/3197#discussion_r832918397
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/read/TestECBlockReconstructedStripeInputStream.java
##########
@@ -487,15 +495,8 @@ public void testErrorReadingBlockContinuesReading() throws IOException {
.randomFill(dataBufs, repConfig.getEcChunkSize(), dataGen, blockLength);
ByteBuffer[] parity = generateParity(dataBufs, repConfig);
- List<List<Integer>> failLists = new ArrayList<>();
- failLists.add(indexesToList(0, 1));
- // These will be the first parity read and then the next parity read as a
- // replacement
- failLists.add(indexesToList(2, 3));
- // First parity and then the data block
- failLists.add(indexesToList(2, 0));
-
- for (List<Integer> failList : failLists) {
+ for (int k = 0; k < 5; k++) {
Review comment:
is this 5 nothing but data+parity ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] umamaheswararao merged pull request #3197: HDDS-6448. EC: Reconstructed Input Streams should free resources after reading to end of block
Posted by GitBox <gi...@apache.org>.
umamaheswararao merged pull request #3197:
URL: https://github.com/apache/ozone/pull/3197
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sodonnel commented on a change in pull request #3197: HDDS-6448. EC: Reconstructed Input Streams should free resources after reading to end of block
Posted by GitBox <gi...@apache.org>.
sodonnel commented on a change in pull request #3197:
URL: https://github.com/apache/ozone/pull/3197#discussion_r832946189
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/read/TestECBlockReconstructedStripeInputStream.java
##########
@@ -487,15 +495,8 @@ public void testErrorReadingBlockContinuesReading() throws IOException {
.randomFill(dataBufs, repConfig.getEcChunkSize(), dataGen, blockLength);
ByteBuffer[] parity = generateParity(dataBufs, repConfig);
- List<List<Integer>> failLists = new ArrayList<>();
- failLists.add(indexesToList(0, 1));
- // These will be the first parity read and then the next parity read as a
- // replacement
- failLists.add(indexesToList(2, 3));
- // First parity and then the data block
- failLists.add(indexesToList(2, 0));
-
- for (List<Integer> failList : failLists) {
+ for (int k = 0; k < 5; k++) {
Review comment:
Not, its just repeating the main loop of the test 5 files to allow a random node to be failed each time and have some randomness in the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org