You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "SaketaChalamchala (via GitHub)" <gi...@apache.org> on 2023/02/01 06:03:07 UTC
[GitHub] [ozone] SaketaChalamchala opened a new pull request, #4157: HDDS-7747. Ozone cli to support listing of encrypted buckets.
SaketaChalamchala opened a new pull request, #4157:
URL: https://github.com/apache/ozone/pull/4157
## What changes were proposed in this pull request?
Extended the usage of `ozone sh bucket list`. Added an additional option in Bucket List Handler to filter out encrypted buckets similar to `hdfs crypto -listZones`
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-7747
## How was this patch tested?
Manual test
```
sh-4.2$ ozone sh bucket list --help
Usage: ozone sh bucket list [-ehV] [-l=<limit>] [-p=<prefix>] [-s=<startItem>]
<value>
lists the buckets in a volume.
<value> URI of the volume.
Ozone URI could either be a full URI or short URI.
Full URI should start with o3://, in case of non-HA
clusters it should be followed by the host name and
optionally the port number. In case of HA clusters
the service id should be used. Service id provides a
logical name for multiple hosts and it is defined
in the property ozone.om.service.ids.
Example of a full URI with host name and port number
for a key:
o3://omhostname:9862/vol1/bucket1/key1
With a service id for a volume:
o3://omserviceid/vol1/
Short URI should start from the volume.
Example of a short URI for a bucket:
vol1/bucket1
Any unspecified information will be identified from
the config files.
-e, --encrypted List all buckets with encryption enabled.
-h, --help Show this help message and exit.
-l, --length=<limit> Maximum number of items to list
Default: 100
-p, --prefix=<prefix> Prefix to filter the items
-s, --start=<startItem> The item to start the listing from.
This will be excluded from the result.
-V, --version Print version information and exit.
sh-4.2$ ozone sh bucket list /vol1
[ {
"metadata" : { },
"volumeName" : "vol1",
"name" : "bucket1",
"storageType" : "DISK",
"versioning" : false,
"usedBytes" : 30720,
"usedNamespace" : 1,
"creationTime" : "2023-01-07T22:30:16.295Z",
"modificationTime" : "2023-01-07T22:30:16.295Z",
"quotaInBytes" : -1,
"quotaInNamespace" : -1,
"bucketLayout" : "LEGACY",
"owner" : "testuser",
"link" : false
}, {
"metadata" : { },
"volumeName" : "vol1",
"name" : "encbucket1",
"storageType" : "DISK",
"versioning" : false,
"usedBytes" : 0,
"usedNamespace" : 0,
"creationTime" : "2023-01-07T22:40:42.437Z",
"modificationTime" : "2023-01-07T22:40:42.437Z",
"encryptionKeyName" : "key1",
"quotaInBytes" : -1,
"quotaInNamespace" : -1,
"bucketLayout" : "LEGACY",
"owner" : "testuser",
"link" : false
} ]
sh-4.2$ ozone sh bucket list /vol1 -e
[ {
"metadata" : { },
"volumeName" : "vol1",
"name" : "encbucket1",
"storageType" : "DISK",
"versioning" : false,
"usedBytes" : 0,
"usedNamespace" : 0,
"creationTime" : "2023-01-07T22:40:42.437Z",
"modificationTime" : "2023-01-07T22:40:42.437Z",
"encryptionKeyName" : "key1",
"quotaInBytes" : -1,
"quotaInNamespace" : -1,
"bucketLayout" : "LEGACY",
"owner" : "testuser",
"link" : false
} ]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] SaketaChalamchala commented on pull request #4157: HDDS-7747. Ozone cli to support listing of encrypted buckets.
Posted by "SaketaChalamchala (via GitHub)" <gi...@apache.org>.
SaketaChalamchala commented on PR #4157:
URL: https://github.com/apache/ozone/pull/4157#issuecomment-1409000780
@kerneltime & @errose28 Agree. Users can get more out of `jq` filtering than any filters we build into the client.
> or @SaketaChalamchala may want to do it as well. A link to jq docs and some example filters added as a section to the bottom of the [CLI docs](https://ozone.apache.org/docs/current/interface/cli.html) should do it.
I will add some examples to the documentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a diff in pull request #4157: HDDS-7747. Ozone cli to support listing of encrypted buckets.
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on code in PR #4157:
URL: https://github.com/apache/ozone/pull/4157#discussion_r1065069803
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/bucket/ListBucketHandler.java:
##########
@@ -44,6 +45,10 @@ public class ListBucketHandler extends VolumeHandler {
@CommandLine.Mixin
private ListOptions listOptions;
+ @Option(names = {"--encrypted", "-e"},
+ description = "List all buckets with encryption enabled.")
Review Comment:
```suggestion
description = "Only list buckets with encryption enabled.")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai merged pull request #4157: HDDS-7747. Document jq filtering examples for Cli responses.
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai merged PR #4157:
URL: https://github.com/apache/ozone/pull/4157
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] errose28 commented on pull request #4157: HDDS-7747. Ozone cli to support listing of encrypted buckets.
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on PR #4157:
URL: https://github.com/apache/ozone/pull/4157#issuecomment-1401415898
> For now I think adding extra flags that are client side only should be ok.
The problem is that our current compatibility guarantees mean we cannot just add CLI flags "for now". Once they are released they must be there forever, so we should put more thought in to how we handle these cases.
> We should add more documentation and samples for how to use jq if we want that to be the primary way forward.
Definitely. I can add the documentation, or @SaketaChalamchala may want to do it as well. A link to jq docs and some example filters added as a section to the bottom of the [CLI docs](https://ozone.apache.org/docs/current/interface/cli.html) should do it.
With some doc examples simple queries should be fairly straightforward. For example, here is how you get the info for all buckets that are not links: `ozone sh bucket list /vol1 | jq '.[] | select(.link==false)`. Generalizing and expanding is straightforward once given a basic example since our json structure is pretty basic: `ozone sh bucket list /vol1 | jq '.[] | select(.link==false and .bucketLayout=="LEGACY")`
I think that would be a more simple and worthwhile change than this one. If we continue to follow this PR's precedent for client side filtering our CLI will be even more messy, and users may not even be aware that there is a more powerful way to filter results beyond what is built in to Ozone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] errose28 commented on a diff in pull request #4157: HDDS-7747. Ozone cli to support listing of encrypted buckets.
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #4157:
URL: https://github.com/apache/ozone/pull/4157#discussion_r1093769144
##########
hadoop-hdds/docs/content/interface/Cli.md:
##########
@@ -207,3 +207,59 @@ $ ozone sh key info /vol1/bucket1/README.md
```shell
$ ozone sh key get /vol1/bucket1/README.md /tmp/
```
+
+## Querying Cli Results
+
+Ozone cli returns JSON responses. [jq](https://stedolan.github.io/jq/manual/) is a command line JSON processor that can be used to filter cli output for desired information.
+
+For example:
+```shell
+# List encrypted bucket names
+$ ozone sh bucket list /vol1 | jq -r '.[] | select(.encryptionKeyName != null) | [.name, .encryptionKeyName] | @tsv'
+ec5 key1
+encrypted-bucket key1
+
+# List EC buckets with replication config
+$ $ ozone sh bucket list /vol1 | jq -r '.[] | select(.replicationConfig.replicationType == "EC") | [.name, .replicationConfig]'
+[
+ "ec5",
+ {
+ "data": 3,
+ "parity": 2,
+ "ecChunkSize": 1048576,
+ "codec": "RS",
+ "replicationType": "EC",
+ "requiredNodes": 5
+ }
+]
+[
+ "ec9",
+ {
+ "data": 6,
+ "parity": 3,
+ "ecChunkSize": 1048576,
+ "codec": "RS",
+ "replicationType": "EC",
+ "requiredNodes": 9
+ }
+]
+
+# List FSO buckets that are not links
+$ ozone sh bucket list /s3v | jq '.[] | select(.link==false and .bucketLayout=="FILE_SYSTEM_OPTIMIZED")'
+{
+ "metadata": {},
+ "volumeName": "s3v",
+ "name": "fso-bucket",
+ "storageType": "DISK",
+ "versioning": false,
+ "usedBytes": 0,
+ "usedNamespace": 0,
+ "creationTime": "2023-02-01T05:18:46.974Z",
+ "modificationTime": "2023-02-01T05:18:46.974Z",
+ "quotaInBytes": -1,
+ "quotaInNamespace": -1,
+ "bucketLayout": "FILE_SYSTEM_OPTIMIZED",
+ "owner": "om",
+ "link": false
+}
+```
Review Comment:
The three chosen examples are good. I just have some minor suggestions for readability. Mostly reordering them from least complex to most complex and splitting them into bullets since most people skim docs. I typed the markdown here directly so it would render in the comment.
- List FSO buckets that are not links
```shell
$ ozone sh bucket list /s3v | jq '.[] | select(.link == false and .bucketLayout == " FILE_SYSTEM_OPTIMIZED")'
{
"metadata": {},
"volumeName": "s3v",
"name": "fso-bucket",
"storageType": "DISK",
"versioning": false,
"usedBytes": 0,
"usedNamespace": 0,
"creationTime": "2023-02-01T05:18:46.974Z",
"modificationTime": "2023-02-01T05:18:46.974Z",
"quotaInBytes": -1,
"quotaInNamespace": -1,
"bucketLayout": "FILE_SYSTEM_OPTIMIZED",
"owner": "om",
"link": false
}
```
- List EC bucket names with their replication type.
```shell
ozone sh bucket list /vol1 | jq -r '.[] | select(.replicationConfig.replicationType == "EC") | {"name": .name, "replicationType": .replicationConfig.replicationType}'
{
"name": "ec5",
"replicationType": "EC"
}
{
"name": "ec9",
"replicationType": "EC"
}
```
- List names of encrypted buckets and their encryption key names in tab-separated-value format.
```shell
$ ozone sh bucket list /vol1 | jq -r '.[] | select(.encryptionKeyName != null) | [.name, .encryptionKeyName] | @tsv'
ec5 key1
encrypted-bucket key1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] errose28 commented on pull request #4157: HDDS-7747. Ozone cli to support listing of encrypted buckets.
Posted by GitBox <gi...@apache.org>.
errose28 commented on PR #4157:
URL: https://github.com/apache/ozone/pull/4157#issuecomment-1377757136
Should we try to make this more generic? We may want to list encrypted buckets now, but in the future users may want to see all FSO buckets, all buckets with quotas set, or all buckets with EC as the default rep type. With the current design this will lead to an explosion of config flags.
One option is to use `jq` to filter on the client side. This would require more information being pushed over the wire, but is much more flexible. If we want some filtering handled on the server side and some to be left to the user, we should have some criteria for what falls in each category.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] SaketaChalamchala closed pull request #4157: HDDS-7747. Ozone cli to support listing of encrypted buckets.
Posted by "SaketaChalamchala (via GitHub)" <gi...@apache.org>.
SaketaChalamchala closed pull request #4157: HDDS-7747. Ozone cli to support listing of encrypted buckets.
URL: https://github.com/apache/ozone/pull/4157
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] SaketaChalamchala commented on pull request #4157: HDDS-7747. Ozone cli to support listing of encrypted buckets.
Posted by GitBox <gi...@apache.org>.
SaketaChalamchala commented on PR #4157:
URL: https://github.com/apache/ozone/pull/4157#issuecomment-1376693234
Thanks @kerneltime and @adoroszlai for the review. I added a robot test and updated the option description as suggested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on pull request #4157: HDDS-7747. Document jq filtering examples for Cli responses.
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #4157:
URL: https://github.com/apache/ozone/pull/4157#issuecomment-1416157124
Thanks @SaketaChalamchala for the patch and your flexibility, @errose28 for the great suggestion about converting it to a doc enhancement and reviewing it, @kerneltime for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] SaketaChalamchala commented on a diff in pull request #4157: HDDS-7747. Ozone cli to support listing of encrypted buckets.
Posted by GitBox <gi...@apache.org>.
SaketaChalamchala commented on code in PR #4157:
URL: https://github.com/apache/ozone/pull/4157#discussion_r1065310579
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/bucket/ListBucketHandler.java:
##########
@@ -44,6 +45,10 @@ public class ListBucketHandler extends VolumeHandler {
@CommandLine.Mixin
private ListOptions listOptions;
+ @Option(names = {"--encrypted", "-e"},
+ description = "List all buckets with encryption enabled.")
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a diff in pull request #4157: HDDS-7747. Ozone cli to support listing of encrypted buckets.
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on code in PR #4157:
URL: https://github.com/apache/ozone/pull/4157#discussion_r1065608806
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/bucket/ListBucketHandler.java:
##########
@@ -56,12 +61,15 @@ protected void execute(OzoneClient client, OzoneAddress address)
int counter = 0;
while (bucketIterator.hasNext() && counter < listOptions.getLimit()) {
OzoneBucket bucket = bucketIterator.next();
- if (bucket.isLink()) {
- bucketList.add(new InfoBucketHandler.LinkBucket(bucket));
- } else {
- bucketList.add(bucket);
+ if (!encryptedBuckets ||
+ (encryptedBuckets && (bucket.getEncryptionKeyName() != null))) {
Review Comment:
Nit: condition can be simplified. (Sorry for missing this earlier.)
```suggestion
if (!encryptedBuckets || bucket.getEncryptionKeyName() != null) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on pull request #4157: HDDS-7747. Ozone cli to support listing of encrypted buckets.
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on PR #4157:
URL: https://github.com/apache/ozone/pull/4157#issuecomment-1399812715
@errose28 this change is a client side filter and does what `jq` would achieve without needing the user to be aware of `jq`. We should add more documentation and samples for how to use `jq` if we want that to be the primary way forward. For now I think adding extra flags that are client side only should be ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] errose28 commented on a diff in pull request #4157: HDDS-7747. Document jq filtering examples for Cli responses.
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #4157:
URL: https://github.com/apache/ozone/pull/4157#discussion_r1095112749
##########
hadoop-hdds/docs/content/interface/Cli.md:
##########
@@ -207,3 +207,65 @@ $ ozone sh key info /vol1/bucket1/README.md
```shell
$ ozone sh key get /vol1/bucket1/README.md /tmp/
```
+
+## Querying Cli Results
+
+Ozone cli returns JSON responses. [jq](https://stedolan.github.io/jq/manual/) is a command line JSON processor that can be used to filter cli output for desired information.
Review Comment:
super minor nit but I think capitalizing CLI looks better.
##########
hadoop-hdds/docs/content/interface/Cli.md:
##########
@@ -207,3 +207,65 @@ $ ozone sh key info /vol1/bucket1/README.md
```shell
$ ozone sh key get /vol1/bucket1/README.md /tmp/
```
+
+## Querying Cli Results
+
+Ozone cli returns JSON responses. [jq](https://stedolan.github.io/jq/manual/) is a command line JSON processor that can be used to filter cli output for desired information.
+
+For example:
+
+* List FSO buckets that are not links.
+```shell
+$ ozone sh bucket list /s3v | jq '.[] | select(.link==false and .bucketLayout=="FILE_SYSTEM_OPTIMIZED")'
+{
+ "metadata": {},
+ "volumeName": "s3v",
+ "name": "fso-bucket",
+ "storageType": "DISK",
+ "versioning": false,
+ "usedBytes": 0,
+ "usedNamespace": 0,
+ "creationTime": "2023-02-01T05:18:46.974Z",
+ "modificationTime": "2023-02-01T05:18:46.974Z",
+ "quotaInBytes": -1,
+ "quotaInNamespace": -1,
+ "bucketLayout": "FILE_SYSTEM_OPTIMIZED",
+ "owner": "om",
+ "link": false
+}
+```
+
+* List EC buckets with their replication config.
+```shell
+$ $ ozone sh bucket list /vol1 | jq -r '.[] | select(.replicationConfig.replicationType == "EC") | [.name, .replicationConfig]'
+[
+ "ec5",
+ {
+ "data": 3,
+ "parity": 2,
+ "ecChunkSize": 1048576,
+ "codec": "RS",
+ "replicationType": "EC",
+ "requiredNodes": 5
+ }
+]
+[
+ "ec9",
+ {
+ "data": 6,
+ "parity": 3,
+ "ecChunkSize": 1048576,
+ "codec": "RS",
+ "replicationType": "EC",
+ "requiredNodes": 9
+ }
+]
Review Comment:
I think printing each result as an object instead of a list makes more sense.
```
ozone sh bucket list /vol1 | jq -r '.[] | select(.replicationConfig.replicationType == "EC") | {"name": .name, "replicationType": .replicationConfig.replicationType}'
{
"name": "ec5",
"replicationType": "EC"
}
{
"name": "ec9",
"replicationType": "EC"
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org