You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2022/05/06 03:42:36 UTC
[GitHub] [ozone] guihecheng opened a new pull request, #3385: HDDS-6543. BlockDeletingService adaptation for schema v3 containers.
guihecheng opened a new pull request, #3385:
URL: https://github.com/apache/ozone/pull/3385
## What changes were proposed in this pull request?
BlockDeletingService adaptation for schema v3 containers.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-6543
## How was this patch tested?
turned on previously ignored UT for schema v3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ChenSammi commented on a diff in pull request #3385: HDDS-6543. [Merge rocksdb in datanode] BlockDeletingService adaptation for schema v3 containers.
Posted by GitBox <gi...@apache.org>.
ChenSammi commented on code in PR #3385:
URL: https://github.com/apache/ozone/pull/3385#discussion_r867658239
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/statemachine/background/BlockDeletingService.java:
##########
@@ -516,4 +547,10 @@ public int getPriority() {
return priority;
}
}
+
+ private interface Deleter {
+ void apply(Table<?, DeletedBlocksTransaction> deleteTxnsTable,
+ BatchOperation batch, long txnID)
+ throws IOException;
Review Comment:
L553 and L553 can be merged into one line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] guihecheng commented on a diff in pull request #3385: HDDS-6543. [Merge rocksdb in datanode] BlockDeletingService adaptation for schema v3 containers.
Posted by GitBox <gi...@apache.org>.
guihecheng commented on code in PR #3385:
URL: https://github.com/apache/ozone/pull/3385#discussion_r867668634
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/metadata/DatanodeStoreSchemaThreeImpl.java:
##########
@@ -40,7 +40,8 @@
* - All keys have containerID as prefix.
* - The table 3 has String as key instead of Long since we want to use prefix.
*/
-public class DatanodeStoreSchemaThreeImpl extends AbstractDatanodeStore {
+public class DatanodeStoreSchemaThreeImpl extends AbstractDatanodeStore
+ implements DeleteTransactionStore<String> {
Review Comment:
Ah, I think I just renamed the helper function in `BlockUtils` but not the one in `DatanodeStoreSchemaThreeImpl`.
I think we could use a slightly different name like `removeKVContainerData`, because I found a existing name for those KV pairs for container: `KeyValueContainerUtil.parseKVContainerData()`.
And I'll add `dumpKVContainerDataXXX` and `loadKVContainerDataXXX` in the next patch for container export and import.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] guihecheng commented on pull request #3385: HDDS-6543. BlockDeletingService adaptation for schema v3 containers.
Posted by GitBox <gi...@apache.org>.
guihecheng commented on PR #3385:
URL: https://github.com/apache/ozone/pull/3385#issuecomment-1119234242
cc @ChenSammi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] guihecheng commented on a diff in pull request #3385: HDDS-6543. [Merge rocksdb in datanode] BlockDeletingService adaptation for schema v3 containers.
Posted by GitBox <gi...@apache.org>.
guihecheng commented on code in PR #3385:
URL: https://github.com/apache/ozone/pull/3385#discussion_r867665944
##########
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/TestBlockDeletingService.java:
##########
@@ -236,6 +241,26 @@ private void createPendingDeleteBlocksSchema2(int numOfBlocksPerContainer,
int txnID, long containerID, int numOfChunksPerBlock, ChunkBuffer buffer,
ChunkManager chunkManager, KeyValueContainer container,
KeyValueContainerData data) {
+ createPendingDeleteBlocksViaTxn(numOfBlocksPerContainer, txnID,
+ containerID, numOfChunksPerBlock, buffer, chunkManager,
+ container, data);
+ }
+
+ @SuppressWarnings("checkstyle:parameternumber")
+ private void createPendingDeleteBlocksSchema3(int numOfBlocksPerContainer,
Review Comment:
Yes, it is, here I just intend to keep the schema check style of function in `createToDeleteBlocks` aligned, but I could remove one of the identical functions and use the lower level function directly when we have schema v2 || v3.
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/statemachine/background/BlockDeletingService.java:
##########
@@ -516,4 +547,10 @@ public int getPriority() {
return priority;
}
}
+
+ private interface Deleter {
+ void apply(Table<?, DeletedBlocksTransaction> deleteTxnsTable,
+ BatchOperation batch, long txnID)
+ throws IOException;
Review Comment:
Sure, I'll merge it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ChenSammi commented on a diff in pull request #3385: HDDS-6543. [Merge rocksdb in datanode] BlockDeletingService adaptation for schema v3 containers.
Posted by GitBox <gi...@apache.org>.
ChenSammi commented on code in PR #3385:
URL: https://github.com/apache/ozone/pull/3385#discussion_r867660073
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/metadata/DatanodeStoreSchemaThreeImpl.java:
##########
@@ -40,7 +40,8 @@
* - All keys have containerID as prefix.
* - The table 3 has String as key instead of Long since we want to use prefix.
*/
-public class DatanodeStoreSchemaThreeImpl extends AbstractDatanodeStore {
+public class DatanodeStoreSchemaThreeImpl extends AbstractDatanodeStore
+ implements DeleteTransactionStore<String> {
Review Comment:
"Rename dropAllWithPrefix to removeContainerFromDB". Change is missed from last patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ChenSammi merged pull request #3385: HDDS-6543. [Merge rocksdb in datanode] BlockDeletingService adaptation for schema v3 containers.
Posted by GitBox <gi...@apache.org>.
ChenSammi merged PR #3385:
URL: https://github.com/apache/ozone/pull/3385
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ChenSammi commented on a diff in pull request #3385: HDDS-6543. [Merge rocksdb in datanode] BlockDeletingService adaptation for schema v3 containers.
Posted by GitBox <gi...@apache.org>.
ChenSammi commented on code in PR #3385:
URL: https://github.com/apache/ozone/pull/3385#discussion_r867657657
##########
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/TestBlockDeletingService.java:
##########
@@ -236,6 +241,26 @@ private void createPendingDeleteBlocksSchema2(int numOfBlocksPerContainer,
int txnID, long containerID, int numOfChunksPerBlock, ChunkBuffer buffer,
ChunkManager chunkManager, KeyValueContainer container,
KeyValueContainerData data) {
+ createPendingDeleteBlocksViaTxn(numOfBlocksPerContainer, txnID,
+ containerID, numOfChunksPerBlock, buffer, chunkManager,
+ container, data);
+ }
+
+ @SuppressWarnings("checkstyle:parameternumber")
+ private void createPendingDeleteBlocksSchema3(int numOfBlocksPerContainer,
Review Comment:
Looks like createPendingDeleteBlocksSchema3 and createPendingDeleteBlocksSchema2 are identical.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ChenSammi commented on pull request #3385: HDDS-6543. [Merge rocksdb in datanode] BlockDeletingService adaptation for schema v3 containers.
Posted by GitBox <gi...@apache.org>.
ChenSammi commented on PR #3385:
URL: https://github.com/apache/ozone/pull/3385#issuecomment-1120685378
Thanks @guihecheng, the patch overall looks good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ChenSammi commented on pull request #3385: HDDS-6543. [Merge rocksdb in datanode] BlockDeletingService adaptation for schema v3 containers.
Posted by GitBox <gi...@apache.org>.
ChenSammi commented on PR #3385:
URL: https://github.com/apache/ozone/pull/3385#issuecomment-1120865149
+1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] guihecheng commented on a diff in pull request #3385: HDDS-6543. [Merge rocksdb in datanode] BlockDeletingService adaptation for schema v3 containers.
Posted by GitBox <gi...@apache.org>.
guihecheng commented on code in PR #3385:
URL: https://github.com/apache/ozone/pull/3385#discussion_r867668634
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/metadata/DatanodeStoreSchemaThreeImpl.java:
##########
@@ -40,7 +40,8 @@
* - All keys have containerID as prefix.
* - The table 3 has String as key instead of Long since we want to use prefix.
*/
-public class DatanodeStoreSchemaThreeImpl extends AbstractDatanodeStore {
+public class DatanodeStoreSchemaThreeImpl extends AbstractDatanodeStore
+ implements DeleteTransactionStore<String> {
Review Comment:
Ah, I think I just renamed the helper function in `BlockUtils` but not the one in `DatanodeStoreSchemaThreeImpl`, let's rename it to removeContainerFromDB.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org