You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Tejaskriya (via GitHub)" <gi...@apache.org> on 2024/10/09 08:04:51 UTC
[PR] HDDS-11546. Add regex operation to filter option of ldb scan command [ozone]
Tejaskriya opened a new pull request, #7289:
URL: https://github.com/apache/ozone/pull/7289
## What changes were proposed in this pull request?
Currently, `ozone debug ldb scan` command has a `--filter` option supporting the following operations for the records stored in the rocksDB: `EQUALS, LESSER, GREATER`.
Adding support for matching with regex will be useful while debugging as any pattern of keys can be found with it.
For example, if a value has many fields like [name, location->[address, DN, IP], version, lastUpdateTime], using the option `--filter={name:regex:^a.*}` will display records with name that start with the letter 'a'
A fields along with the regex that should be used to is given to the command, and only those records passing the condition it will be shown.
eg.) `ozone debug ldb --db=/data/metadata/om.db scan --cf=volumeTable --filter="usedNamespace:greater:2,adminName:regex:^a.*a$" `
--> Will show only the records having usedNamespace>2 and adminName starting and ending with the letter 'a'.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-11546
## How was this patch tested?
Added unit tests in TestLDBCli.java and added robot test in ozone-debug-ldb.robot.
Also tested locally in a db:
```
$ ozone debug ldb --db=ZDump/bigDB/omMBdb/om.db scan --cf=keyTable -l=3 --fields="objectID,volumeName,bucketName,keyName,dataSize,creationTime" --filter="acls.name:equals:systest,creationTime:regex:^172[0-9]*237$"
{ "/55406660-10tb/ozone-legacy-ec-1st-bucket-55406660/10T-1-terasort-input/_SUCCESS": {
"bucketName" : "ozone-legacy-ec-1st-bucket-55406660",
"creationTime" : 1722744719237,
"volumeName" : "55406660-10tb",
"keyName" : "10T-1-terasort-input/_SUCCESS",
"dataSize" : 0,
"objectID" : -9223372036249583104
}
}
$ ozone debug ldb --db=ZDump/bigDB/om.db scan --cf=volumeTable --fields=volume,usedNamespace,adminName --filter="usedNamespace:equals:2,adminName:regex:^i.*"
{ "/cai": {
"volume" : "cai",
"adminName" : "impala",
"usedNamespace" : 2
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11546. Add regex operation to filter option of ldb scan command [ozone]
Posted by "Tejaskriya (via GitHub)" <gi...@apache.org>.
Tejaskriya commented on code in PR #7289:
URL: https://github.com/apache/ozone/pull/7289#discussion_r1795263253
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/debug/TestLDBCli.java:
##########
@@ -206,6 +207,18 @@ private static Stream<Arguments> scanTestCases() {
Named.of("Filter dataSize>2000", Arrays.asList("--filter", "dataSize:greater:2000")),
Named.of("Expect empty result", null)
),
+ Arguments.of(
+ Named.of(KEY_TABLE, Pair.of(KEY_TABLE, false)),
+ Named.of("Default", Pair.of(0, "")),
+ Named.of("Filter key3 regex", Arrays.asList("--filter", "keyName:regex:^.*3$")),
+ Named.of("Expect key3", Pair.of("key3", "key4"))
+ ),
+ Arguments.of(
+ Named.of(KEY_TABLE, Pair.of(KEY_TABLE, false)),
+ Named.of("Default", Pair.of(0, "")),
+ Named.of("Filter dataSize>500 using regex", Arrays.asList("--filter", "dataSize:regex:^5.*$")),
Review Comment:
Thanks for the review, I have updated the test now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11546. Add regex operation to filter option of ldb scan command [ozone]
Posted by "sadanand48 (via GitHub)" <gi...@apache.org>.
sadanand48 commented on code in PR #7289:
URL: https://github.com/apache/ozone/pull/7289#discussion_r1794637094
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/debug/TestLDBCli.java:
##########
@@ -206,6 +207,18 @@ private static Stream<Arguments> scanTestCases() {
Named.of("Filter dataSize>2000", Arrays.asList("--filter", "dataSize:greater:2000")),
Named.of("Expect empty result", null)
),
+ Arguments.of(
+ Named.of(KEY_TABLE, Pair.of(KEY_TABLE, false)),
+ Named.of("Default", Pair.of(0, "")),
+ Named.of("Filter key3 regex", Arrays.asList("--filter", "keyName:regex:^.*3$")),
+ Named.of("Expect key3", Pair.of("key3", "key4"))
+ ),
+ Arguments.of(
+ Named.of(KEY_TABLE, Pair.of(KEY_TABLE, false)),
+ Named.of("Default", Pair.of(0, "")),
+ Named.of("Filter dataSize>500 using regex", Arrays.asList("--filter", "dataSize:regex:^5.*$")),
Review Comment:
nit : ^5.*$ means any data size starting with 5
```suggestion
Named.of("Filter keys whose dataSize digits start with 5 using regex", Arrays.asList("--filter", "dataSize:regex:^5.*$")),
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11546. Add regex operation to filter option of ldb scan command [ozone]
Posted by "sadanand48 (via GitHub)" <gi...@apache.org>.
sadanand48 commented on PR #7289:
URL: https://github.com/apache/ozone/pull/7289#issuecomment-2405221442
Thanks @Tejaskriya for the patch & @sumitagrawl for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11546. Add regex operation to filter option of ldb scan command [ozone]
Posted by "sadanand48 (via GitHub)" <gi...@apache.org>.
sadanand48 merged PR #7289:
URL: https://github.com/apache/ozone/pull/7289
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org