You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2022/10/26 23:39:43 UTC
[GitHub] [ozone] aryangupta1998 opened a new pull request, #3893: HDDS-7423. DB Scanner should allow iteration from specific key
aryangupta1998 opened a new pull request, #3893:
URL: https://github.com/apache/ozone/pull/3893
## What changes were proposed in this pull request?
The current RocksDB scanner command allows only a simple length parameter to limit the number of records to scan in a specific col family. In the case of a huge table like keyTable or containers, while we are looking for a specific entry, the search is needlessly large and explorative (since we do not know the 'index' of the key)
A startKey param would be useful.
ozone debug ldb --db=<> scan --column_family=<> --length=<> --startKey=<>
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-7423
## How was this patch tested?
Tested Manually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sadanand48 merged pull request #3893: HDDS-7423. DB Scanner should allow iteration from specific key
Posted by GitBox <gi...@apache.org>.
sadanand48 merged PR #3893:
URL: https://github.com/apache/ozone/pull/3893
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sadanand48 commented on a diff in pull request #3893: HDDS-7423. DB Scanner should allow iteration from specific key
Posted by GitBox <gi...@apache.org>.
sadanand48 commented on code in PR #3893:
URL: https://github.com/apache/ozone/pull/3893#discussion_r1007021770
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/debug/DBScanner.java:
##########
@@ -97,10 +104,32 @@ public class DBScanner implements Callable<Void>, SubcommandWithParent {
private List<Object> scannedObjects;
+ public static byte[] getValueObject(
+ DBColumnFamilyDefinition dbColumnFamilyDefinition) throws IOException {
+ String str = dbColumnFamilyDefinition.getKeyType().getName();
Review Comment:
Its better to use Class instead of String to handle cases in which certain classes moved to different package.
```suggestion
Class<?> keyType = dbColumnFamilyDefinition.getKeyType();
if (keyType.equals(String.class)) {
return startKey.getBytes();
} else if (keyType.equals(ContainerID.class)) {
return new ContainerID(Long.parseLong(startKey)).getBytes();
} else if (keyType.equals(Long.class)) {
return Longs.toByteArray(Long.parseLong(startKey));
} else if (keyType.equals(PipelineID.class)) {
return PipelineID.valueOf(UUID.fromString(startKey)).getProtobuf()
.toByteArray();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sadanand48 commented on a diff in pull request #3893: HDDS-7423. DB Scanner should allow iteration from specific key
Posted by GitBox <gi...@apache.org>.
sadanand48 commented on code in PR #3893:
URL: https://github.com/apache/ozone/pull/3893#discussion_r1006987653
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestLDBCli.java:
##########
@@ -113,6 +113,7 @@ public void testOMDB() throws Exception {
}
rdbParser.setDbPath(dbStore.getDbLocation().getAbsolutePath());
dbScanner.setParent(rdbParser);
+ dbScanner.setStartKey("/sampleVol/sampleBuck/key3");
Review Comment:
If we're setting startKey as `/sampleVol/sampleBuck/key3` ,it has to output only 3 keys whereas next line we are asserting for 5. The format here should be `dbScanner.setKey("key3") `here and lets add this case after all other cases in this test.
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/debug/DBScanner.java:
##########
@@ -97,10 +104,32 @@ public class DBScanner implements Callable<Void>, SubcommandWithParent {
private List<Object> scannedObjects;
+ public static byte[] getValueObject(
+ DBColumnFamilyDefinition dbColumnFamilyDefinition) throws IOException {
+ String str = dbColumnFamilyDefinition.getKeyType().getName();
Review Comment:
Its better to use Class instead of String to handle cases in which certain classes moved to different package.
```suggestion
Class<?> keyType = dbColumnFamilyDefinition.getKeyType();
if (keyType.equals(String.class)) {
return startKey.getBytes();
} else if (keyType.equals(ContainerID.class)) {
return new ContainerID(Long.parseLong(startKey)).getBytes();
} else if (keyType.equals(Long.class)) {
return Longs.toByteArray(Long.parseLong(startKey));
} else if (keyType.equals(PipelineID.class)) {
return PipelineID.valueOf(UUID.fromString(startKey)).getProtobuf()
.toByteArray();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] aryangupta1998 commented on a diff in pull request #3893: HDDS-7423. DB Scanner should allow iteration from specific key
Posted by GitBox <gi...@apache.org>.
aryangupta1998 commented on code in PR #3893:
URL: https://github.com/apache/ozone/pull/3893#discussion_r1010963818
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/debug/DBScanner.java:
##########
@@ -97,10 +104,32 @@ public class DBScanner implements Callable<Void>, SubcommandWithParent {
private List<Object> scannedObjects;
+ public static byte[] getValueObject(
+ DBColumnFamilyDefinition dbColumnFamilyDefinition) throws IOException {
+ String str = dbColumnFamilyDefinition.getKeyType().getName();
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] aryangupta1998 commented on a diff in pull request #3893: HDDS-7423. DB Scanner should allow iteration from specific key
Posted by GitBox <gi...@apache.org>.
aryangupta1998 commented on code in PR #3893:
URL: https://github.com/apache/ozone/pull/3893#discussion_r1012227559
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestLDBCli.java:
##########
@@ -113,6 +113,7 @@ public void testOMDB() throws Exception {
}
rdbParser.setDbPath(dbStore.getDbLocation().getAbsolutePath());
dbScanner.setParent(rdbParser);
+ dbScanner.setStartKey("/sampleVol/sampleBuck/key3");
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sadanand48 commented on a diff in pull request #3893: HDDS-7423. DB Scanner should allow iteration from specific key
Posted by GitBox <gi...@apache.org>.
sadanand48 commented on code in PR #3893:
URL: https://github.com/apache/ozone/pull/3893#discussion_r1006987653
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestLDBCli.java:
##########
@@ -113,6 +113,7 @@ public void testOMDB() throws Exception {
}
rdbParser.setDbPath(dbStore.getDbLocation().getAbsolutePath());
dbScanner.setParent(rdbParser);
+ dbScanner.setStartKey("/sampleVol/sampleBuck/key3");
Review Comment:
If we're setting startKey as `/sampleVol/sampleBuck/key3` ,it has to output only 3 keys whereas next line we are asserting for 5. The format here should be `dbScanner.setStartKey("key3") `here and lets add this case after all other cases in this test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org