You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2022/01/29 00:10:08 UTC
[GitHub] [ozone] umamaheswararao opened a new pull request #3026: HDDS-6237: EC: putBlock should pass close flag true on end of block group/close file.
umamaheswararao opened a new pull request #3026:
URL: https://github.com/apache/ozone/pull/3026
## What changes were proposed in this pull request?
Passing the close flag correctly on end of blockgroup or file close.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-6237
## How was this patch tested?
Added a test check the behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] umamaheswararao merged pull request #3026: HDDS-6237: EC: putBlock should pass close flag true on end of block group/close file.
Posted by GitBox <gi...@apache.org>.
umamaheswararao merged pull request #3026:
URL: https://github.com/apache/ozone/pull/3026
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] umamaheswararao commented on a change in pull request #3026: HDDS-6237: EC: putBlock should pass close flag true on end of block group/close file.
Posted by GitBox <gi...@apache.org>.
umamaheswararao commented on a change in pull request #3026:
URL: https://github.com/apache/ozone/pull/3026#discussion_r796844628
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestECKeyOutputStream.java
##########
@@ -229,6 +237,40 @@ public void testMultipleChunksInSingleWriteOp(int numChunks)
validateContent(inputData, bucket, bucket.getKey(keyName));
}
+ @Test
Review comment:
Yes, you are right.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] umamaheswararao commented on a change in pull request #3026: HDDS-6237: EC: putBlock should pass close flag true on end of block group/close file.
Posted by GitBox <gi...@apache.org>.
umamaheswararao commented on a change in pull request #3026:
URL: https://github.com/apache/ozone/pull/3026#discussion_r796845281
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestECKeyOutputStream.java
##########
@@ -229,6 +237,40 @@ public void testMultipleChunksInSingleWriteOp(int numChunks)
validateContent(inputData, bucket, bucket.getKey(keyName));
}
+ @Test
+ public void testECContainerKeysCount()
+ throws IOException, InterruptedException, TimeoutException,
+ AuthenticationException {
+ byte[] inputData = getInputBytes(1);
+ final OzoneBucket bucket = getOzoneBucket();
+ ContainerOperationClient containerOperationClient =
+ new ContainerOperationClient(conf);
+ List<ContainerInfo> containerInfos =
+ containerOperationClient.listContainer(1, 100);
+ Map<ContainerID, Long> containerKeys = new HashMap<>();
+ for (ContainerInfo info : containerInfos) {
+ containerKeys.put(info.containerID(),
+ containerKeys.getOrDefault(info.containerID(), 0L) + 1);
+ }
+
+ String keyName = UUID.randomUUID().toString();
+ try (OzoneOutputStream out = bucket.createKey(keyName, 4096,
+ new ECReplicationConfig(3, 2, ECReplicationConfig.EcCodec.RS,
+ chunkSize), new HashMap<>())) {
+ out.write(inputData);
+ }
+ OzoneKeyDetails key = bucket.getKey(keyName);
+ cluster.restartStorageContainerManager(true);
Review comment:
This make sense. I have updated the test with your suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sodonnel commented on a change in pull request #3026: HDDS-6237: EC: putBlock should pass close flag true on end of block group/close file.
Posted by GitBox <gi...@apache.org>.
sodonnel commented on a change in pull request #3026:
URL: https://github.com/apache/ozone/pull/3026#discussion_r795595348
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestECKeyOutputStream.java
##########
@@ -229,6 +237,40 @@ public void testMultipleChunksInSingleWriteOp(int numChunks)
validateContent(inputData, bucket, bucket.getKey(keyName));
}
+ @Test
Review comment:
I have a question on this - is the container key count only incremented when the block is closed via sending the close flag? Is that why this test is checking the key count increases after writing a key?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sodonnel commented on a change in pull request #3026: HDDS-6237: EC: putBlock should pass close flag true on end of block group/close file.
Posted by GitBox <gi...@apache.org>.
sodonnel commented on a change in pull request #3026:
URL: https://github.com/apache/ozone/pull/3026#discussion_r795627816
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestECKeyOutputStream.java
##########
@@ -229,6 +237,40 @@ public void testMultipleChunksInSingleWriteOp(int numChunks)
validateContent(inputData, bucket, bucket.getKey(keyName));
}
+ @Test
+ public void testECContainerKeysCount()
+ throws IOException, InterruptedException, TimeoutException,
+ AuthenticationException {
+ byte[] inputData = getInputBytes(1);
+ final OzoneBucket bucket = getOzoneBucket();
+ ContainerOperationClient containerOperationClient =
+ new ContainerOperationClient(conf);
+ List<ContainerInfo> containerInfos =
+ containerOperationClient.listContainer(1, 100);
+ Map<ContainerID, Long> containerKeys = new HashMap<>();
+ for (ContainerInfo info : containerInfos) {
+ containerKeys.put(info.containerID(),
+ containerKeys.getOrDefault(info.containerID(), 0L) + 1);
+ }
+
+ String keyName = UUID.randomUUID().toString();
+ try (OzoneOutputStream out = bucket.createKey(keyName, 4096,
+ new ECReplicationConfig(3, 2, ECReplicationConfig.EcCodec.RS,
+ chunkSize), new HashMap<>())) {
+ out.write(inputData);
+ }
+ OzoneKeyDetails key = bucket.getKey(keyName);
+ cluster.restartStorageContainerManager(true);
Review comment:
I guess this restart is here so SCM gets a full container report to get the updated key counts in the containers? Rather than restarting SCM, what about setting the FCR interval to a very low value, eg, in the init() method
```
conf.setTimeDuration(HDDS_HEARTBEAT_INTERVAL, 500, TimeUnit.MILLISECONDS);
conf.setTimeDuration(HDDS_CONTAINER_REPORT_INTERVAL, 1, TimeUnit.SECONDS);
```
Then change the test to something like:
```
...
OzoneKeyDetails key = bucket.getKey(keyName);
long currentKeyContainerID =
key.getOzoneKeyLocations().get(0).getContainerID();
Long priorKeys = containerKeys.get(new ContainerID(currentKeyContainerID));
long expectedKeys = (priorKeys != null ? priorKeys : 0) + 1L;
GenericTestUtils.waitFor(() -> {
try {
return containerOperationClient
.listContainer(currentKeyContainerID, 100)
.get(0).getNumberOfKeys() == expectedKeys;
} catch (IOException exception) {
fail("Unexpected exception " + exception);
return false;
}
}, 100, 10000);
validateContent(inputData, bucket, bucket.getKey(keyName));
```
On my laptop the test completes in 2 - 3 seconds now, rather than 7. Its a small win, probably worth it. I think its a better test, as it relies on the contaier report to be received by an already running SCM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org