You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2022/12/23 18:43:20 UTC
[GitHub] [ozone] xBis7 opened a new pull request, #4125: HDDS-7277. [Snapshot] ozone sh key list /vol1/bucket1/.snapshot/snap1 fails
xBis7 opened a new pull request, #4125:
URL: https://github.com/apache/ozone/pull/4125
## What changes were proposed in this pull request?
This error was coming from `OzoneAddress.ensureBucketAddress()` but if we were to modify that method to accept the snapshot name as part of the keyName while checking a bucket path, then every bucket command would accept a snapshot path appended to it. Therefore, `ozone sh key list` couldn't work with a snapshot without having other unwanted commands doing it as well.
In order to fix this, a new command was added, `ozone sh snapshot key-list` which extends a new class `SnapshotHandler` that calls `SnapshotUri.convert()` and from `OzoneAddress.ensureSnapshotAddress()` checks for a snapshot prefix and name. This method could be modified and extended in the future to accept keys under the snapshot address.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-7277
## How was this patch tested?
A new unit test was added for checking the snapshot address. Also, this patch was tested manually in a docker cluster like so
in `/hadoop-ozone/dist/target/ozone-1.3.0-SNAPSHOT/compose/ozone`
```
❯ docker-compose up --scale datanode=3 -d
❯ docker exec -it ozone_om_1 bash
bash-4.2$ ozone sh volume create /vol1
bash-4.2$ ozone sh bucket create /vol1/bucket1
bash-4.2$ ozone sh key put /vol1/bucket1/key1 /etc/hosts
bash-4.2$ ozone sh key put /vol1/bucket1/dir1/key2 /etc/os-release
bash-4.2$ ozone sh snapshot create /vol1/bucket1 snap1
bash-4.2$ ozone sh snapshot key-list /vol1/bucket1/.snapshot/snap1
[ {
"volumeName" : "vol1",
"bucketName" : "bucket1",
"name" : ".snapshot/snap1/dir1/key2",
"dataSize" : 393,
"creationTime" : "2022-12-23T18:23:36.063Z",
"modificationTime" : "2022-12-23T18:23:36.827Z",
"replicationConfig" : {
"replicationFactor" : "THREE",
"requiredNodes" : 3,
"replicationType" : "RATIS"
}
}, {
"volumeName" : "vol1",
"bucketName" : "bucket1",
"name" : ".snapshot/snap1/key1",
"dataSize" : 174,
"creationTime" : "2022-12-23T18:23:22.899Z",
"modificationTime" : "2022-12-23T18:23:23.920Z",
"replicationConfig" : {
"replicationFactor" : "THREE",
"requiredNodes" : 3,
"replicationType" : "RATIS"
}
} ]
bash-4.2$ ozone sh snapshot key-list --prefix=dir1 /vol1/bucket1/.snapshot/snap1
[ {
"volumeName" : "vol1",
"bucketName" : "bucket1",
"name" : ".snapshot/snap1/dir1/key2",
"dataSize" : 393,
"creationTime" : "2022-12-23T18:23:36.063Z",
"modificationTime" : "2022-12-23T18:23:36.827Z",
"replicationConfig" : {
"replicationFactor" : "THREE",
"requiredNodes" : 3,
"replicationType" : "RATIS"
}
} ]
bash-4.2$ ozone sh snapshot key-list --prefix=.snapshot/snap1/dir1 /vol1/bucket1
Invalid value for positional parameter at index 0 (<value>): cannot convert '/vol1/bucket1' to OzoneAddress (org.apache.hadoop.ozone.client.OzoneClientException: Snapshot name is missing.)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on pull request #4125: HDDS-7277. [Snapshot] ozone sh key list /vol1/bucket1/.snapshot/snap1 fails
Posted by GitBox <gi...@apache.org>.
xBis7 commented on PR #4125:
URL: https://github.com/apache/ozone/pull/4125#issuecomment-1364221793
@smengcl @sadanand48 Can you please take a look at this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on a diff in pull request #4125: HDDS-7277. [Snapshot] Implement snapshot key listing in Ozone Shell
Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on code in PR #4125:
URL: https://github.com/apache/ozone/pull/4125#discussion_r1094938260
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/IncrementalContainerReportHandler.java:
##########
@@ -88,8 +88,10 @@ public void onMessage(final IncrementalContainerReportFromDatanode report,
// Ensure we reuse the same ContainerID instance in containerInfo
id = container.containerID();
} finally {
- if (!replicaProto.getState().equals(
+ if (replicaProto.getState().equals(
ContainerReplicaProto.State.DELETED)) {
+ nodeManager.removeContainer(dd, id);
+ } else {
Review Comment:
Another merge, fixed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on a diff in pull request #4125: HDDS-7277. [Snapshot] ozone sh key list /vol1/bucket1/.snapshot/snap1 fails
Posted by GitBox <gi...@apache.org>.
xBis7 commented on code in PR #4125:
URL: https://github.com/apache/ozone/pull/4125#discussion_r1072056247
##########
hadoop-ozone/tools/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneAddress.java:
##########
@@ -96,14 +108,38 @@ public void checkUrlTypes() throws OzoneClientException, IOException {
Assert.assertEquals("key1/key3/key", address.getKeyName());
Assert.assertFalse("this should not be a prefix",
address.isPrefix());
+ }
+ @Test
+ public void checkPrefixUrlType() throws OzoneClientException {
address = new OzoneAddress(prefix + "vol1/bucket/prefix");
address.ensurePrefixAddress();
Assert.assertEquals("vol1", address.getVolumeName());
Assert.assertEquals("bucket", address.getBucketName());
Assert.assertEquals("prefix", address.getKeyName());
Assert.assertTrue("this should be a prefix",
address.isPrefix());
+ }
+
+ @Test
+ public void checkSnapshotUrlType() throws OzoneClientException {
+ address = new OzoneAddress(prefix + "vol1/bucket/.snapshot/snap1");
+ address.ensureSnapshotAddress();
+ Assert.assertEquals("vol1", address.getVolumeName());
+ Assert.assertEquals("bucket", address.getBucketName());
+ Assert.assertEquals(".snapshot/snap1", address.getSnapshotName());
+ Assert.assertEquals(".snapshot/snap1", address.getKeyName());
+
+
+ String message = "Delimiters (/) not allowed following " +
Review Comment:
@GeorgeJahad It's snapshot specific and it won't be reused in another method, so I reduced it and kept only the snapshot part.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on a diff in pull request #4125: HDDS-7277. [Snapshot] Implement snapshot key listing in Ozone Shell
Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on code in PR #4125:
URL: https://github.com/apache/ozone/pull/4125#discussion_r1087633396
##########
hadoop-ozone/tools/pom.xml:
##########
@@ -124,6 +124,10 @@ https://maven.apache.org/xsd/maven-4.0.0.xsd">
<artifactId>mockito-core</artifactId>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>org.junit.jupiter</groupId>
+ <artifactId>junit-jupiter-params</artifactId>
+ </dependency>
Review Comment:
yes, IDE auto imported it and I forgot to check. I'll fix it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on a diff in pull request #4125: HDDS-7277. [Snapshot] Implement snapshot key listing in Ozone Shell
Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on code in PR #4125:
URL: https://github.com/apache/ozone/pull/4125#discussion_r1085581600
##########
hadoop-ozone/tools/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneAddress.java:
##########
@@ -96,14 +108,37 @@ public void checkUrlTypes() throws OzoneClientException, IOException {
Assert.assertEquals("key1/key3/key", address.getKeyName());
Assert.assertFalse("this should not be a prefix",
address.isPrefix());
+ }
+ @Test
+ public void checkPrefixUrlType() throws OzoneClientException {
address = new OzoneAddress(prefix + "vol1/bucket/prefix");
address.ensurePrefixAddress();
Assert.assertEquals("vol1", address.getVolumeName());
Assert.assertEquals("bucket", address.getBucketName());
Assert.assertEquals("prefix", address.getKeyName());
Assert.assertTrue("this should be a prefix",
address.isPrefix());
+ }
+
+ @Test
+ public void checkSnapshotUrlType() throws OzoneClientException {
+ address = new OzoneAddress(prefix + "vol1/bucket/.snapshot/snap1");
+ address.ensureSnapshotAddress();
+ Assert.assertEquals("vol1", address.getVolumeName());
+ Assert.assertEquals("bucket", address.getBucketName());
+ Assert.assertEquals(".snapshot/snap1", address.getSnapshotName());
+ Assert.assertEquals(".snapshot/snap1", address.getKeyName());
+
+
+ String message = "Only a snapshot name with " +
+ "a snapshot indicator is accepted";
+
+ address = new OzoneAddress(prefix + "vol1/bucket/.snapshot");
+
+ exception.expect(OzoneClientException.class);
+ exception.expectMessage(message);
+ address.ensureSnapshotAddress();
}
}
Review Comment:
I added it but is there a reason for this? It's not there in the first place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #4125: HDDS-7277. [Snapshot] ozone sh key list /vol1/bucket1/.snapshot/snap1 fails
Posted by GitBox <gi...@apache.org>.
GeorgeJahad commented on code in PR #4125:
URL: https://github.com/apache/ozone/pull/4125#discussion_r1071541408
##########
hadoop-ozone/tools/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneAddress.java:
##########
@@ -96,14 +108,38 @@ public void checkUrlTypes() throws OzoneClientException, IOException {
Assert.assertEquals("key1/key3/key", address.getKeyName());
Assert.assertFalse("this should not be a prefix",
address.isPrefix());
+ }
+ @Test
+ public void checkPrefixUrlType() throws OzoneClientException {
address = new OzoneAddress(prefix + "vol1/bucket/prefix");
address.ensurePrefixAddress();
Assert.assertEquals("vol1", address.getVolumeName());
Assert.assertEquals("bucket", address.getBucketName());
Assert.assertEquals("prefix", address.getKeyName());
Assert.assertTrue("this should be a prefix",
address.isPrefix());
+ }
+
+ @Test
+ public void checkSnapshotUrlType() throws OzoneClientException {
+ address = new OzoneAddress(prefix + "vol1/bucket/.snapshot/snap1");
+ address.ensureSnapshotAddress();
+ Assert.assertEquals("vol1", address.getVolumeName());
+ Assert.assertEquals("bucket", address.getBucketName());
+ Assert.assertEquals(".snapshot/snap1", address.getSnapshotName());
+ Assert.assertEquals(".snapshot/snap1", address.getKeyName());
+
+
+ String message = "Delimiters (/) not allowed following " +
Review Comment:
NIT: if you are going to check for a precise error message, I would define it in a string constant in the class being tested, and then use that string constant in the test. DRY!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] hemantk-12 commented on a diff in pull request #4125: HDDS-7277. [Snapshot] Implement snapshot key listing in Ozone Shell
Posted by "hemantk-12 (via GitHub)" <gi...@apache.org>.
hemantk-12 commented on code in PR #4125:
URL: https://github.com/apache/ozone/pull/4125#discussion_r1087174799
##########
hadoop-ozone/tools/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneAddress.java:
##########
@@ -47,26 +48,34 @@ public static Collection<Object[]> data() {
});
}
- private String prefix;
+ @Rule
+ public ExpectedException exception = ExpectedException.none();
+
+ private final String prefix;
+ private OzoneAddress address;
public TestOzoneAddress(String prefix) {
this.prefix = prefix;
}
@Test
- public void checkUrlTypes() throws OzoneClientException, IOException {
- OzoneAddress address;
-
+ public void checkRootUrlType() throws OzoneClientException {
address = new OzoneAddress("");
Review Comment:
I still don't get to have empty string in test cases. It is unnecessary to me. But I'll leave it because it was already there.
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/keys/ListKeyHandler.java:
##########
@@ -21,25 +21,26 @@
import java.io.IOException;
import java.util.Iterator;
+import com.google.common.base.Strings;
import org.apache.hadoop.ozone.client.OzoneBucket;
import org.apache.hadoop.ozone.client.OzoneClient;
import org.apache.hadoop.ozone.client.OzoneClientException;
import org.apache.hadoop.ozone.client.OzoneKey;
import org.apache.hadoop.ozone.client.OzoneVolume;
import org.apache.hadoop.ozone.shell.ListOptions;
import org.apache.hadoop.ozone.shell.OzoneAddress;
-import org.apache.hadoop.ozone.shell.bucket.BucketHandler;
+import org.apache.hadoop.ozone.shell.snapshot.SnapshotHandler;
import picocli.CommandLine;
import picocli.CommandLine.Command;
/**
- * Executes List Keys.
+ * Executes List Keys for a bucket or snapshot.
*/
@Command(name = "list",
aliases = "ls",
- description = "list all keys in a given bucket")
-public class ListKeyHandler extends BucketHandler {
+ description = "list all keys in a given bucket or snapshot")
+public class ListKeyHandler extends SnapshotHandler {
Review Comment:
Even tho we need to verify address, I still don't believe that it is a good idea to have it this way if you think in term of the code structure and hierarchy.
##########
hadoop-ozone/tools/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneAddress.java:
##########
@@ -47,26 +48,34 @@ public static Collection<Object[]> data() {
});
}
- private String prefix;
+ @Rule
+ public ExpectedException exception = ExpectedException.none();
+
+ private final String prefix;
+ private OzoneAddress address;
public TestOzoneAddress(String prefix) {
this.prefix = prefix;
}
@Test
- public void checkUrlTypes() throws OzoneClientException, IOException {
- OzoneAddress address;
-
+ public void checkRootUrlType() throws OzoneClientException {
address = new OzoneAddress("");
address.ensureRootAddress();
address = new OzoneAddress(prefix + "");
address.ensureRootAddress();
+ }
+ @Test
Review Comment:
Thanks for upgrading to Junit-5.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on a diff in pull request #4125: HDDS-7277. [Snapshot] Implement snapshot key listing in Ozone Shell
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #4125:
URL: https://github.com/apache/ozone/pull/4125#discussion_r1094909961
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/IncrementalContainerReportHandler.java:
##########
@@ -88,8 +88,10 @@ public void onMessage(final IncrementalContainerReportFromDatanode report,
// Ensure we reuse the same ContainerID instance in containerInfo
id = container.containerID();
} finally {
- if (!replicaProto.getState().equals(
+ if (replicaProto.getState().equals(
ContainerReplicaProto.State.DELETED)) {
+ nodeManager.removeContainer(dd, id);
+ } else {
Review Comment:
Could be a GitHub PR diff display bug. See https://github.com/apache/ozone/pull/4134/files#r1094909600
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on pull request #4125: HDDS-7277. [Snapshot] Implement snapshot key listing in Ozone Shell
Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on PR #4125:
URL: https://github.com/apache/ozone/pull/4125#issuecomment-1415355009
@smengcl @hemantk-12 @GeorgeJahad Thanks for the reviews!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on a diff in pull request #4125: HDDS-7277. [Snapshot] Implement snapshot key listing in Ozone Shell
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #4125:
URL: https://github.com/apache/ozone/pull/4125#discussion_r1094894719
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/IncrementalContainerReportHandler.java:
##########
@@ -88,8 +88,10 @@ public void onMessage(final IncrementalContainerReportFromDatanode report,
// Ensure we reuse the same ContainerID instance in containerInfo
id = container.containerID();
} finally {
- if (!replicaProto.getState().equals(
+ if (replicaProto.getState().equals(
ContainerReplicaProto.State.DELETED)) {
+ nodeManager.removeContainer(dd, id);
+ } else {
Review Comment:
Is this change brought in accidentally by the merge conflict resolution?
https://github.com/apache/ozone/pull/4125/commits/b2bac5584d575dbf02eb1d52628a19a6034f2a81
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on a diff in pull request #4125: HDDS-7277. [Snapshot] Implement snapshot key listing in Ozone Shell
Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on code in PR #4125:
URL: https://github.com/apache/ozone/pull/4125#discussion_r1094914141
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/IncrementalContainerReportHandler.java:
##########
@@ -88,8 +88,10 @@ public void onMessage(final IncrementalContainerReportFromDatanode report,
// Ensure we reuse the same ContainerID instance in containerInfo
id = container.containerID();
} finally {
- if (!replicaProto.getState().equals(
+ if (replicaProto.getState().equals(
ContainerReplicaProto.State.DELETED)) {
+ nodeManager.removeContainer(dd, id);
+ } else {
Review Comment:
@smengcl What if I try another rebase?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on pull request #4125: HDDS-7277. [Snapshot] Implement snapshot key listing in Ozone Shell
Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on PR #4125:
URL: https://github.com/apache/ozone/pull/4125#issuecomment-1414483103
@hemantk-12 @smengcl Thanks for the reviews, I've updated the patch. I get your point about the `SnapshotHandler` class.
> You can have different internal implementations of ListKeyHandler based on path type is bucket or snapshot if needed but high level it should be independent.
This would be the best, but we would have to reorganize all Handler classes and that would require changes across all shell commands. If we are to go that way, it should be in another PR.
> we could rename SnapshotHandler into something like BucketAndSnapshotHandler in order not to give the wrong impression?
This seemed the easiest approach, so I renamed the class to `BucketSnapshotHandler`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on a diff in pull request #4125: HDDS-7277. [Snapshot] Implement snapshot key listing in Ozone Shell
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #4125:
URL: https://github.com/apache/ozone/pull/4125#discussion_r1094990777
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/snapshot/SnapshotHandler.java:
##########
@@ -0,0 +1,36 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.shell.snapshot;
+
+import org.apache.hadoop.ozone.shell.Handler;
+import org.apache.hadoop.ozone.shell.OzoneAddress;
+import picocli.CommandLine;
+
+/**
+ * Base class for snapshot command handlers.
+ */
+public abstract class SnapshotHandler extends Handler {
Review Comment:
Should this extend `BucketHandler` instead of plain `Handler`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] hemantk-12 commented on a diff in pull request #4125: HDDS-7277. [Snapshot] Implement snapshot key listing in Ozone Shell
Posted by "hemantk-12 (via GitHub)" <gi...@apache.org>.
hemantk-12 commented on code in PR #4125:
URL: https://github.com/apache/ozone/pull/4125#discussion_r1087167605
##########
hadoop-ozone/tools/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneAddress.java:
##########
@@ -96,14 +108,37 @@ public void checkUrlTypes() throws OzoneClientException, IOException {
Assert.assertEquals("key1/key3/key", address.getKeyName());
Assert.assertFalse("this should not be a prefix",
address.isPrefix());
+ }
+ @Test
+ public void checkPrefixUrlType() throws OzoneClientException {
address = new OzoneAddress(prefix + "vol1/bucket/prefix");
address.ensurePrefixAddress();
Assert.assertEquals("vol1", address.getVolumeName());
Assert.assertEquals("bucket", address.getBucketName());
Assert.assertEquals("prefix", address.getKeyName());
Assert.assertTrue("this should be a prefix",
address.isPrefix());
+ }
+
+ @Test
+ public void checkSnapshotUrlType() throws OzoneClientException {
+ address = new OzoneAddress(prefix + "vol1/bucket/.snapshot/snap1");
+ address.ensureSnapshotAddress();
+ Assert.assertEquals("vol1", address.getVolumeName());
+ Assert.assertEquals("bucket", address.getBucketName());
+ Assert.assertEquals(".snapshot/snap1", address.getSnapshotName());
+ Assert.assertEquals(".snapshot/snap1", address.getKeyName());
+
+
+ String message = "Only a snapshot name with " +
+ "a snapshot indicator is accepted";
+
+ address = new OzoneAddress(prefix + "vol1/bucket/.snapshot");
+
+ exception.expect(OzoneClientException.class);
+ exception.expectMessage(message);
+ address.ensureSnapshotAddress();
}
}
Review Comment:
It is general practice to add new line at the end of the file. AFAIK, the idea comes from the Unix file system.
https://unix.stackexchange.com/questions/18743/whats-the-point-in-adding-a-new-line-to-the-end-of-a-file, https://gist.github.com/OleksiyRudenko/d51388345ea55767b7672307fe35adf3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on a diff in pull request #4125: HDDS-7277. [Snapshot] Implement snapshot key listing in Ozone Shell
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #4125:
URL: https://github.com/apache/ozone/pull/4125#discussion_r1094968495
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/OzoneAddress.java:
##########
@@ -343,6 +350,35 @@ public void ensureKeyAddress() throws OzoneClientException {
}
}
+ /**
+ * Checking for a volume and a bucket
+ * but also accepting a snapshot
+ * indicator and a snapshot name.
+ * If the keyName can't be considered
+ * a valid snapshot, an exception is thrown.
+ *
+ * @throws OzoneClientException
+ */
+ public void ensureSnapshotAddress()
+ throws OzoneClientException {
+ if (keyName.length() > 0) {
+ if (OmUtils.isBucketSnapshotIndicator(keyName)) {
+ snapshotNameWithIndicator += keyName;
Review Comment:
Is there a reason to use `+=` instead of `=` here?
What if `ensureSnapshotAddress()` is called twice?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on pull request #4125: HDDS-7277. [Snapshot] Implement snapshot key listing in Ozone Shell
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on PR #4125:
URL: https://github.com/apache/ozone/pull/4125#issuecomment-1414915044
Thanks @xBis7 for the patch. Thanks @GeorgeJahad @hemantk-12 for the reviews.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on a diff in pull request #4125: HDDS-7277. [Snapshot] Implement snapshot key listing in Ozone Shell
Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on code in PR #4125:
URL: https://github.com/apache/ozone/pull/4125#discussion_r1085515720
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/OzoneAddress.java:
##########
@@ -343,6 +349,35 @@ public void ensureKeyAddress() throws OzoneClientException {
}
}
+ /**
+ * Similar to #ensureBucketAddress()
+ * but also accepting a snapshot
+ * indicator and a snapshot name.
+ * If the keyName can't be considered
+ * a valid snapshot, an exception is thrown.
+ *
+ * @throws OzoneClientException
+ */
+ public void ensureSnapshotAddress()
+ throws OzoneClientException {
+ if (keyName.length() > 0) {
+ if (OmUtils.isBucketSnapshotIndicator(keyName)) {
+ snapshotName += keyName;
Review Comment:
You are right, it's misleading. I will give it a better name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on a diff in pull request #4125: HDDS-7277. [Snapshot] Implement snapshot key listing in Ozone Shell
Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on code in PR #4125:
URL: https://github.com/apache/ozone/pull/4125#discussion_r1085531016
##########
hadoop-ozone/tools/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneAddress.java:
##########
@@ -47,26 +48,34 @@ public static Collection<Object[]> data() {
});
}
- private String prefix;
+ @Rule
+ public ExpectedException exception = ExpectedException.none();
+
+ private final String prefix;
+ private OzoneAddress address;
public TestOzoneAddress(String prefix) {
this.prefix = prefix;
}
@Test
- public void checkUrlTypes() throws OzoneClientException, IOException {
- OzoneAddress address;
-
+ public void checkRootUrlType() throws OzoneClientException {
address = new OzoneAddress("");
Review Comment:
There is no difference, check [here](https://github.com/apache/ozone/blob/a20e84ef03517d8970d8094cd4ec85d3d53caee2/hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/OzoneAddress.java#L70), but we need this call to initialize `OzoneAddress`, even if we are setting an empty address.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on a diff in pull request #4125: HDDS-7277. [Snapshot] Implement snapshot key listing in Ozone Shell
Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on code in PR #4125:
URL: https://github.com/apache/ozone/pull/4125#discussion_r1085531016
##########
hadoop-ozone/tools/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneAddress.java:
##########
@@ -47,26 +48,34 @@ public static Collection<Object[]> data() {
});
}
- private String prefix;
+ @Rule
+ public ExpectedException exception = ExpectedException.none();
+
+ private final String prefix;
+ private OzoneAddress address;
public TestOzoneAddress(String prefix) {
this.prefix = prefix;
}
@Test
- public void checkUrlTypes() throws OzoneClientException, IOException {
- OzoneAddress address;
-
+ public void checkRootUrlType() throws OzoneClientException {
address = new OzoneAddress("");
Review Comment:
One is referring to the ozone scheme prefix and the other one to the entity level(root, volume, bucket, etc). In this case there is no difference, check [here](https://github.com/apache/ozone/blob/a20e84ef03517d8970d8094cd4ec85d3d53caee2/hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/OzoneAddress.java#L70), but we need this call to initialize `OzoneAddress`, even if we are setting an empty address.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] hemantk-12 commented on a diff in pull request #4125: HDDS-7277. [Snapshot] Implement snapshot key listing in Ozone Shell
Posted by "hemantk-12 (via GitHub)" <gi...@apache.org>.
hemantk-12 commented on code in PR #4125:
URL: https://github.com/apache/ozone/pull/4125#discussion_r1084560530
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/OzoneAddress.java:
##########
@@ -343,6 +349,35 @@ public void ensureKeyAddress() throws OzoneClientException {
}
}
+ /**
+ * Similar to #ensureBucketAddress()
+ * but also accepting a snapshot
+ * indicator and a snapshot name.
+ * If the keyName can't be considered
+ * a valid snapshot, an exception is thrown.
+ *
+ * @throws OzoneClientException
+ */
+ public void ensureSnapshotAddress()
+ throws OzoneClientException {
+ if (keyName.length() > 0) {
+ if (OmUtils.isBucketSnapshotIndicator(keyName)) {
+ snapshotName += keyName;
Review Comment:
I think this is wrong to just append `keyName` to `snapshotName` because key name contains `SNAPSHOT_INDICATOR`= `.snapshot`. May be use better name or extract out the snapshotName.
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/OzoneAddress.java:
##########
@@ -343,6 +349,35 @@ public void ensureKeyAddress() throws OzoneClientException {
}
}
+ /**
+ * Similar to #ensureBucketAddress()
Review Comment:
nit:
1. I don't think it is a good idea to mention `Similar to XYZ` as starting of Java doc comment.
2. Method name says that ensuring snapshot address but it does more than just ensuring snapshot address.
##########
hadoop-ozone/tools/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneAddress.java:
##########
@@ -47,26 +48,34 @@ public static Collection<Object[]> data() {
});
}
- private String prefix;
+ @Rule
+ public ExpectedException exception = ExpectedException.none();
+
+ private final String prefix;
+ private OzoneAddress address;
public TestOzoneAddress(String prefix) {
this.prefix = prefix;
}
@Test
- public void checkUrlTypes() throws OzoneClientException, IOException {
- OzoneAddress address;
-
+ public void checkRootUrlType() throws OzoneClientException {
address = new OzoneAddress("");
Review Comment:
What's the difference between this and last parameter in line 47?
##########
hadoop-ozone/tools/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneAddress.java:
##########
@@ -96,14 +108,37 @@ public void checkUrlTypes() throws OzoneClientException, IOException {
Assert.assertEquals("key1/key3/key", address.getKeyName());
Assert.assertFalse("this should not be a prefix",
address.isPrefix());
+ }
+ @Test
+ public void checkPrefixUrlType() throws OzoneClientException {
address = new OzoneAddress(prefix + "vol1/bucket/prefix");
address.ensurePrefixAddress();
Assert.assertEquals("vol1", address.getVolumeName());
Assert.assertEquals("bucket", address.getBucketName());
Assert.assertEquals("prefix", address.getKeyName());
Assert.assertTrue("this should be a prefix",
address.isPrefix());
+ }
+
+ @Test
+ public void checkSnapshotUrlType() throws OzoneClientException {
+ address = new OzoneAddress(prefix + "vol1/bucket/.snapshot/snap1");
+ address.ensureSnapshotAddress();
+ Assert.assertEquals("vol1", address.getVolumeName());
+ Assert.assertEquals("bucket", address.getBucketName());
+ Assert.assertEquals(".snapshot/snap1", address.getSnapshotName());
+ Assert.assertEquals(".snapshot/snap1", address.getKeyName());
+
+
+ String message = "Only a snapshot name with " +
+ "a snapshot indicator is accepted";
+
+ address = new OzoneAddress(prefix + "vol1/bucket/.snapshot");
+
+ exception.expect(OzoneClientException.class);
+ exception.expectMessage(message);
+ address.ensureSnapshotAddress();
Review Comment:
```suggestion
OzoneClientException exception = assertThrows(OzoneClientException.class, () -> {
address.ensureSnapshotAddress();
});
assertTrue(exception.getMessage().contains(message));
```
It would be nicer and simpler if you use Junit5. https://www.baeldung.com/junit-assert-exception
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/keys/ListKeyHandler.java:
##########
@@ -21,25 +21,26 @@
import java.io.IOException;
import java.util.Iterator;
+import com.google.common.base.Strings;
import org.apache.hadoop.ozone.client.OzoneBucket;
import org.apache.hadoop.ozone.client.OzoneClient;
import org.apache.hadoop.ozone.client.OzoneClientException;
import org.apache.hadoop.ozone.client.OzoneKey;
import org.apache.hadoop.ozone.client.OzoneVolume;
import org.apache.hadoop.ozone.shell.ListOptions;
import org.apache.hadoop.ozone.shell.OzoneAddress;
-import org.apache.hadoop.ozone.shell.bucket.BucketHandler;
+import org.apache.hadoop.ozone.shell.snapshot.SnapshotHandler;
import picocli.CommandLine;
import picocli.CommandLine.Command;
/**
- * Executes List Keys.
+ * Executes List Keys for a bucket or snapshot.
*/
@Command(name = "list",
aliases = "ls",
- description = "list all keys in a given bucket")
-public class ListKeyHandler extends BucketHandler {
+ description = "list all keys in a given bucket or snapshot")
+public class ListKeyHandler extends SnapshotHandler {
Review Comment:
I don't think we need to create `SnapshotHandler` because it is still listing keys from snapshot bucket.
Also structurally `SnapshotHandler` gives the wrong impression. Either it is to handle snapshot creation request or snapshot listing request not list keys from snapshotted bucket.
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/keys/ListKeyHandler.java:
##########
@@ -50,11 +51,27 @@ protected void execute(OzoneClient client, OzoneAddress address)
String volumeName = address.getVolumeName();
String bucketName = address.getBucketName();
+ String snapshotName = "";
Review Comment:
```suggestion
String snapshotName = address.getSnapshotName();
String keyPrefix = "";
if (!Strings.isNullOrEmpty(snapshotName)) {
keyPrefix += snapshotName;
if (!Strings.isNullOrEmpty(listOptions.getPrefix())) {
keyPrefix += "/";
}
}
if (!Strings.isNullOrEmpty(listOptions.getPrefix())) {
keyPrefix += listOptions.getPrefix();
}
##########
hadoop-ozone/tools/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneAddress.java:
##########
@@ -47,26 +48,34 @@ public static Collection<Object[]> data() {
});
}
- private String prefix;
+ @Rule
+ public ExpectedException exception = ExpectedException.none();
+
+ private final String prefix;
+ private OzoneAddress address;
public TestOzoneAddress(String prefix) {
this.prefix = prefix;
}
@Test
- public void checkUrlTypes() throws OzoneClientException, IOException {
- OzoneAddress address;
-
+ public void checkRootUrlType() throws OzoneClientException {
address = new OzoneAddress("");
address.ensureRootAddress();
address = new OzoneAddress(prefix + "");
address.ensureRootAddress();
+ }
+ @Test
Review Comment:
You can also upgrade these tests to Junit5.
##########
hadoop-ozone/tools/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneAddress.java:
##########
@@ -96,14 +108,37 @@ public void checkUrlTypes() throws OzoneClientException, IOException {
Assert.assertEquals("key1/key3/key", address.getKeyName());
Assert.assertFalse("this should not be a prefix",
address.isPrefix());
+ }
+ @Test
+ public void checkPrefixUrlType() throws OzoneClientException {
address = new OzoneAddress(prefix + "vol1/bucket/prefix");
address.ensurePrefixAddress();
Assert.assertEquals("vol1", address.getVolumeName());
Assert.assertEquals("bucket", address.getBucketName());
Assert.assertEquals("prefix", address.getKeyName());
Assert.assertTrue("this should be a prefix",
address.isPrefix());
+ }
+
+ @Test
+ public void checkSnapshotUrlType() throws OzoneClientException {
+ address = new OzoneAddress(prefix + "vol1/bucket/.snapshot/snap1");
+ address.ensureSnapshotAddress();
+ Assert.assertEquals("vol1", address.getVolumeName());
+ Assert.assertEquals("bucket", address.getBucketName());
+ Assert.assertEquals(".snapshot/snap1", address.getSnapshotName());
+ Assert.assertEquals(".snapshot/snap1", address.getKeyName());
+
+
+ String message = "Only a snapshot name with " +
+ "a snapshot indicator is accepted";
+
+ address = new OzoneAddress(prefix + "vol1/bucket/.snapshot");
+
+ exception.expect(OzoneClientException.class);
+ exception.expectMessage(message);
+ address.ensureSnapshotAddress();
}
}
Review Comment:
Please add a new line at the end.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] hemantk-12 commented on pull request #4125: HDDS-7277. [Snapshot] Implement snapshot key listing in Ozone Shell
Posted by "hemantk-12 (via GitHub)" <gi...@apache.org>.
hemantk-12 commented on PR #4125:
URL: https://github.com/apache/ozone/pull/4125#issuecomment-1414509405
> @hemantk-12 @smengcl Thanks for the reviews, I've updated the patch. I get your point about the `SnapshotHandler` class.
>
> > You can have different internal implementations of ListKeyHandler based on path type is bucket or snapshot if needed but high level it should be independent.
>
> This would be the best, but we would have to reorganize all Handler classes and that would require changes across all shell commands. If we are to go that way, it should be in another PR.
>
> > we could rename SnapshotHandler into something like BucketAndSnapshotHandler in order not to give the wrong impression?
>
> This seemed the easiest approach, so I renamed the class to `BucketSnapshotHandler`.
Doing it as part of another PR is completely fine by me.
Thanks for change the name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl merged pull request #4125: HDDS-7277. [Snapshot] Implement snapshot key listing in Ozone Shell
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl merged PR #4125:
URL: https://github.com/apache/ozone/pull/4125
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on a diff in pull request #4125: HDDS-7277. [Snapshot] Implement snapshot key listing in Ozone Shell
Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on code in PR #4125:
URL: https://github.com/apache/ozone/pull/4125#discussion_r1085483506
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/keys/ListKeyHandler.java:
##########
@@ -21,25 +21,26 @@
import java.io.IOException;
import java.util.Iterator;
+import com.google.common.base.Strings;
import org.apache.hadoop.ozone.client.OzoneBucket;
import org.apache.hadoop.ozone.client.OzoneClient;
import org.apache.hadoop.ozone.client.OzoneClientException;
import org.apache.hadoop.ozone.client.OzoneKey;
import org.apache.hadoop.ozone.client.OzoneVolume;
import org.apache.hadoop.ozone.shell.ListOptions;
import org.apache.hadoop.ozone.shell.OzoneAddress;
-import org.apache.hadoop.ozone.shell.bucket.BucketHandler;
+import org.apache.hadoop.ozone.shell.snapshot.SnapshotHandler;
import picocli.CommandLine;
import picocli.CommandLine.Command;
/**
- * Executes List Keys.
+ * Executes List Keys for a bucket or snapshot.
*/
@Command(name = "list",
aliases = "ls",
- description = "list all keys in a given bucket")
-public class ListKeyHandler extends BucketHandler {
+ description = "list all keys in a given bucket or snapshot")
+public class ListKeyHandler extends SnapshotHandler {
Review Comment:
> Also structurally SnapshotHandler gives the wrong impression.
You are right, but despite the class name, it's related to what the path accepts and not the commands. We need a way to verify an address by calling `OzoneAddress.ensureSnapshotAddress()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on pull request #4125: HDDS-7277. [Snapshot] Implement snapshot key listing in Ozone Shell
Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on PR #4125:
URL: https://github.com/apache/ozone/pull/4125#issuecomment-1414213826
@hemantk-12 Would [this approach](https://github.com/xBis7/ozone/blob/5509ccd365232933fc51c0a688a61a4e2d5d4699/hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/snapshot/ListSnapshotKeyHandler.java) seem cleaner? That's from commit 5509ccd365232933fc51c0a688a61a4e2d5d4699
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on a diff in pull request #4125: HDDS-7277. [Snapshot] Implement snapshot key listing in Ozone Shell
Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on code in PR #4125:
URL: https://github.com/apache/ozone/pull/4125#discussion_r1095172484
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/snapshot/SnapshotHandler.java:
##########
@@ -0,0 +1,36 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.shell.snapshot;
+
+import org.apache.hadoop.ozone.shell.Handler;
+import org.apache.hadoop.ozone.shell.OzoneAddress;
+import picocli.CommandLine;
+
+/**
+ * Base class for snapshot command handlers.
+ */
+public abstract class SnapshotHandler extends Handler {
Review Comment:
Extending `BucketHandler` won't work because we will end up calling `ensureBucketAddress()` instead of `ensureSnapshotAddress()`.
```
bash-4.2$ ozone sh key list /vol1/bucket1/.snapshot/snap1
Invalid value for positional parameter at index 0 (<value>): cannot convert '/vol1/bucket1/.snapshot/snap1' to OzoneAddress (org.apache.hadoop.ozone.client.OzoneClientException: Invalid bucket name. Delimiters (/) not allowed in bucket name)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on a diff in pull request #4125: HDDS-7277. [Snapshot] Implement snapshot key listing in Ozone Shell
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #4125:
URL: https://github.com/apache/ozone/pull/4125#discussion_r1095003258
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/keys/ListKeyHandler.java:
##########
@@ -64,8 +79,18 @@ protected void execute(OzoneClient client, OzoneAddress address)
out().println("Listing first " + maxKeyLimit + " entries of the " +
"result. Use --length (-l) to override max returned keys.");
} else if (isVerbose()) {
- out().printf("Found : %d keys for bucket %s in volume : %s ",
- counter, bucketName, volumeName);
+ if (!Strings.isNullOrEmpty(snapshotNameWithIndicator)) {
+ // snapshot[0] = .snapshot
+ // snapshot[1] = snapshot name
+ String[] snapshotValues = snapshotNameWithIndicator.split("/");
Review Comment:
nit
```suggestion
// snapshotValues[0] = ".snapshot"
// snapshotValues[1] = snapshot name
String[] snapshotValues = snapshotNameWithIndicator.split("/");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on a diff in pull request #4125: HDDS-7277. [Snapshot] Implement snapshot key listing in Ozone Shell
Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on code in PR #4125:
URL: https://github.com/apache/ozone/pull/4125#discussion_r1094906102
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/IncrementalContainerReportHandler.java:
##########
@@ -88,8 +88,10 @@ public void onMessage(final IncrementalContainerReportFromDatanode report,
// Ensure we reuse the same ContainerID instance in containerInfo
id = container.containerID();
} finally {
- if (!replicaProto.getState().equals(
+ if (replicaProto.getState().equals(
ContainerReplicaProto.State.DELETED)) {
+ nodeManager.removeContainer(dd, id);
+ } else {
Review Comment:
@smengcl Yes, and the changes in `TestIncrementalContainerReportHandler` as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on a diff in pull request #4125: HDDS-7277. [Snapshot] Implement snapshot key listing in Ozone Shell
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #4125:
URL: https://github.com/apache/ozone/pull/4125#discussion_r1095006261
##########
hadoop-ozone/tools/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneAddress.java:
##########
@@ -18,25 +18,21 @@
package org.apache.hadoop.ozone.shell;
-import java.io.IOException;
import java.util.Arrays;
import java.util.Collection;
import org.apache.hadoop.ozone.client.OzoneClientException;
-import org.junit.Assert;
-import org.junit.Test;
-import org.junit.runner.RunWith;
-import org.junit.runners.Parameterized;
-import org.junit.runners.Parameterized.Parameters;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.MethodSource;
+
Review Comment:
Thanks for bumping this to JUnit 5!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on pull request #4125: HDDS-7277. [Snapshot] ozone sh key list /vol1/bucket1/.snapshot/snap1 fails
Posted by GitBox <gi...@apache.org>.
xBis7 commented on PR #4125:
URL: https://github.com/apache/ozone/pull/4125#issuecomment-1380558315
@GeorgeJahad I made some changes and now it works with the `ozone sh key list` command. Could you take another look at this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on a diff in pull request #4125: HDDS-7277. [Snapshot] ozone sh key list /vol1/bucket1/.snapshot/snap1 fails
Posted by GitBox <gi...@apache.org>.
xBis7 commented on code in PR #4125:
URL: https://github.com/apache/ozone/pull/4125#discussion_r1072056247
##########
hadoop-ozone/tools/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneAddress.java:
##########
@@ -96,14 +108,38 @@ public void checkUrlTypes() throws OzoneClientException, IOException {
Assert.assertEquals("key1/key3/key", address.getKeyName());
Assert.assertFalse("this should not be a prefix",
address.isPrefix());
+ }
+ @Test
+ public void checkPrefixUrlType() throws OzoneClientException {
address = new OzoneAddress(prefix + "vol1/bucket/prefix");
address.ensurePrefixAddress();
Assert.assertEquals("vol1", address.getVolumeName());
Assert.assertEquals("bucket", address.getBucketName());
Assert.assertEquals("prefix", address.getKeyName());
Assert.assertTrue("this should be a prefix",
address.isPrefix());
+ }
+
+ @Test
+ public void checkSnapshotUrlType() throws OzoneClientException {
+ address = new OzoneAddress(prefix + "vol1/bucket/.snapshot/snap1");
+ address.ensureSnapshotAddress();
+ Assert.assertEquals("vol1", address.getVolumeName());
+ Assert.assertEquals("bucket", address.getBucketName());
+ Assert.assertEquals(".snapshot/snap1", address.getSnapshotName());
+ Assert.assertEquals(".snapshot/snap1", address.getKeyName());
+
+
+ String message = "Delimiters (/) not allowed following " +
Review Comment:
@GeorgeJahad It's snapshot specific and it won't be reused in another method, so I simplified it and kept only the snapshot part.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on a diff in pull request #4125: HDDS-7277. [Snapshot] Implement snapshot key listing in Ozone Shell
Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on code in PR #4125:
URL: https://github.com/apache/ozone/pull/4125#discussion_r1085581600
##########
hadoop-ozone/tools/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneAddress.java:
##########
@@ -96,14 +108,37 @@ public void checkUrlTypes() throws OzoneClientException, IOException {
Assert.assertEquals("key1/key3/key", address.getKeyName());
Assert.assertFalse("this should not be a prefix",
address.isPrefix());
+ }
+ @Test
+ public void checkPrefixUrlType() throws OzoneClientException {
address = new OzoneAddress(prefix + "vol1/bucket/prefix");
address.ensurePrefixAddress();
Assert.assertEquals("vol1", address.getVolumeName());
Assert.assertEquals("bucket", address.getBucketName());
Assert.assertEquals("prefix", address.getKeyName());
Assert.assertTrue("this should be a prefix",
address.isPrefix());
+ }
+
+ @Test
+ public void checkSnapshotUrlType() throws OzoneClientException {
+ address = new OzoneAddress(prefix + "vol1/bucket/.snapshot/snap1");
+ address.ensureSnapshotAddress();
+ Assert.assertEquals("vol1", address.getVolumeName());
+ Assert.assertEquals("bucket", address.getBucketName());
+ Assert.assertEquals(".snapshot/snap1", address.getSnapshotName());
+ Assert.assertEquals(".snapshot/snap1", address.getKeyName());
+
+
+ String message = "Only a snapshot name with " +
+ "a snapshot indicator is accepted";
+
+ address = new OzoneAddress(prefix + "vol1/bucket/.snapshot");
+
+ exception.expect(OzoneClientException.class);
+ exception.expectMessage(message);
+ address.ensureSnapshotAddress();
}
}
Review Comment:
I added it but is there a particular reason for this? It's not there in the first place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #4125: HDDS-7277. [Snapshot] ozone sh key list /vol1/bucket1/.snapshot/snap1 fails
Posted by GitBox <gi...@apache.org>.
GeorgeJahad commented on code in PR #4125:
URL: https://github.com/apache/ozone/pull/4125#discussion_r1062924843
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/snapshot/ListSnapshotKeyHandler.java:
##########
@@ -0,0 +1,80 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.shell.snapshot;
+
+import com.google.common.base.Strings;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.OzoneClientException;
+import org.apache.hadoop.ozone.client.OzoneVolume;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneKey;
+import org.apache.hadoop.ozone.shell.ListOptions;
+import org.apache.hadoop.ozone.shell.OzoneAddress;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.Iterator;
+
+/**
+ * Executes List Keys for a Snapshot.
+ */
+@CommandLine.Command(name = "key-list",
+ aliases = "key-ls",
+ description = "list all keys in a given " +
+ "snapshot under a bucket")
+public class ListSnapshotKeyHandler extends SnapshotHandler {
Review Comment:
The original bug report was to fix the "ozone sh key list" command. This adds a new but similar snapshot "key-list" command, which is fine. But couldn't you also fix the "ozone sh key list" command by changing this line to be extended from "SnapshotHandler" instead of BucketHandler?: https://github.com/apache/ozone/blob/51a0311425e88b56844f05101c18a9b2075fda3a/hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/keys/ListKeyHandler.java#L42
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] hemantk-12 commented on pull request #4125: HDDS-7277. [Snapshot] Implement snapshot key listing in Ozone Shell
Posted by "hemantk-12 (via GitHub)" <gi...@apache.org>.
hemantk-12 commented on PR #4125:
URL: https://github.com/apache/ozone/pull/4125#issuecomment-1414272443
> @hemantk-12 Would [this approach](https://github.com/xBis7/ozone/blob/5509ccd365232933fc51c0a688a61a4e2d5d4699/hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/snapshot/ListSnapshotKeyHandler.java) seem cleaner? That's from commit [5509ccd](https://github.com/apache/ozone/commit/5509ccd365232933fc51c0a688a61a4e2d5d4699).
TBH, I'm not sure if that it clearen either. To me, `ListKeyHandler` should be independent of `bucketPath` or `snapshotPath`. You can have different internal implementations of `ListKeyHandler` based on path type is bucket or snapshot if needed but high level it should be independent.
As I said earlier, `SnapshotHandler` says that it is to handle snapshot reqeust like snapshot create, delete or list snapshot not listing the keys from snapshot.
Again this is what I think based on my experience with Ozone so far. May be @smengcl or @prashantpogde can look and help us here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on pull request #4125: HDDS-7277. [Snapshot] Implement snapshot key listing in Ozone Shell
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on PR #4125:
URL: https://github.com/apache/ozone/pull/4125#issuecomment-1414419236
> @hemantk-12 Would [this approach](https://github.com/xBis7/ozone/blob/5509ccd365232933fc51c0a688a61a4e2d5d4699/hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/snapshot/ListSnapshotKeyHandler.java) seem cleaner? That's from commit [5509ccd](https://github.com/apache/ozone/commit/5509ccd365232933fc51c0a688a61a4e2d5d4699).
Hi @xBis7 , looks like this approach is add a brand new subcommand to Ozone Shell? If this is the case it is not desired, as we do want a unified interface for listing under regular buckets and snapshotted ones.
On the other hand, I agree with @hemantk-12 that we should try to keep using the previous `BucketHandler` [here](https://github.com/apache/ozone/pull/4125#discussion_r1084605774). Or maybe we could rename `SnapshotHandler` into something like `BucketAndSnapshotHandler` in order not to give the wrong impression?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on a diff in pull request #4125: HDDS-7277. [Snapshot] Implement snapshot key listing in Ozone Shell
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #4125:
URL: https://github.com/apache/ozone/pull/4125#discussion_r1095270901
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/snapshot/SnapshotHandler.java:
##########
@@ -0,0 +1,36 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.shell.snapshot;
+
+import org.apache.hadoop.ozone.shell.Handler;
+import org.apache.hadoop.ozone.shell.OzoneAddress;
+import picocli.CommandLine;
+
+/**
+ * Base class for snapshot command handlers.
+ */
+public abstract class SnapshotHandler extends Handler {
Review Comment:
Got it. Thanks @xBis7
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on a diff in pull request #4125: HDDS-7277. [Snapshot] Implement snapshot key listing in Ozone Shell
Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on code in PR #4125:
URL: https://github.com/apache/ozone/pull/4125#discussion_r1095067361
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/OzoneAddress.java:
##########
@@ -343,6 +350,35 @@ public void ensureKeyAddress() throws OzoneClientException {
}
}
+ /**
+ * Checking for a volume and a bucket
+ * but also accepting a snapshot
+ * indicator and a snapshot name.
+ * If the keyName can't be considered
+ * a valid snapshot, an exception is thrown.
+ *
+ * @throws OzoneClientException
+ */
+ public void ensureSnapshotAddress()
+ throws OzoneClientException {
+ if (keyName.length() > 0) {
+ if (OmUtils.isBucketSnapshotIndicator(keyName)) {
+ snapshotNameWithIndicator += keyName;
Review Comment:
You are right, I'll fix it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on a diff in pull request #4125: HDDS-7277. [Snapshot] Implement snapshot key listing in Ozone Shell
Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on code in PR #4125:
URL: https://github.com/apache/ozone/pull/4125#discussion_r1087632290
##########
hadoop-ozone/tools/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneAddress.java:
##########
@@ -96,14 +108,37 @@ public void checkUrlTypes() throws OzoneClientException, IOException {
Assert.assertEquals("key1/key3/key", address.getKeyName());
Assert.assertFalse("this should not be a prefix",
address.isPrefix());
+ }
+ @Test
+ public void checkPrefixUrlType() throws OzoneClientException {
address = new OzoneAddress(prefix + "vol1/bucket/prefix");
address.ensurePrefixAddress();
Assert.assertEquals("vol1", address.getVolumeName());
Assert.assertEquals("bucket", address.getBucketName());
Assert.assertEquals("prefix", address.getKeyName());
Assert.assertTrue("this should be a prefix",
address.isPrefix());
+ }
+
+ @Test
+ public void checkSnapshotUrlType() throws OzoneClientException {
+ address = new OzoneAddress(prefix + "vol1/bucket/.snapshot/snap1");
+ address.ensureSnapshotAddress();
+ Assert.assertEquals("vol1", address.getVolumeName());
+ Assert.assertEquals("bucket", address.getBucketName());
+ Assert.assertEquals(".snapshot/snap1", address.getSnapshotName());
+ Assert.assertEquals(".snapshot/snap1", address.getKeyName());
+
+
+ String message = "Only a snapshot name with " +
+ "a snapshot indicator is accepted";
+
+ address = new OzoneAddress(prefix + "vol1/bucket/.snapshot");
+
+ exception.expect(OzoneClientException.class);
+ exception.expectMessage(message);
+ address.ensureSnapshotAddress();
}
}
Review Comment:
@hemantk-12 That's interesting! Thanks for letting me know, I'll keep it in mind.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on a diff in pull request #4125: HDDS-7277. [Snapshot] ozone sh key list /vol1/bucket1/.snapshot/snap1 fails
Posted by GitBox <gi...@apache.org>.
xBis7 commented on code in PR #4125:
URL: https://github.com/apache/ozone/pull/4125#discussion_r1064866481
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/snapshot/ListSnapshotKeyHandler.java:
##########
@@ -0,0 +1,80 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.shell.snapshot;
+
+import com.google.common.base.Strings;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.OzoneClientException;
+import org.apache.hadoop.ozone.client.OzoneVolume;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneKey;
+import org.apache.hadoop.ozone.shell.ListOptions;
+import org.apache.hadoop.ozone.shell.OzoneAddress;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.Iterator;
+
+/**
+ * Executes List Keys for a Snapshot.
+ */
+@CommandLine.Command(name = "key-list",
+ aliases = "key-ls",
+ description = "list all keys in a given " +
+ "snapshot under a bucket")
+public class ListSnapshotKeyHandler extends SnapshotHandler {
Review Comment:
> But couldn't you also fix the "ozone sh key list" command by changing this line to be extended from "SnapshotHandler" instead of BucketHandler?:
@GeorgeJahad If we make that change then `ozone sh key list` will only work with snapshots and it will give us an error if we try to get the keys under a bucket.
```
bash-4.2$ ozone sh key list /vol1/bucket1
Invalid value for positional parameter at index 0 (<value>): cannot convert '/vol1/bucket1' to OzoneAddress (org.apache.hadoop.ozone.client.OzoneClientException: Snapshot name is missing.)
Usage: ozone sh key list [-hV] [-l=<limit>] [-p=<prefix>] [-s=<startItem>]
<value>
list all keys in a given bucket
...
bash-4.2$ ozone sh key list /vol1/bucket1/.snapshot/snap1
[ {
"volumeName" : "vol1",
"bucketName" : "bucket1",
"name" : "key1",
"dataSize" : 3811,
"creationTime" : "2023-01-09T16:56:04.248Z",
"modificationTime" : "2023-01-09T16:56:05.308Z",
"replicationConfig" : {
"replicationFactor" : "THREE",
"requiredNodes" : 3,
"replicationType" : "RATIS"
}
} ]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] hemantk-12 commented on a diff in pull request #4125: HDDS-7277. [Snapshot] Implement snapshot key listing in Ozone Shell
Posted by "hemantk-12 (via GitHub)" <gi...@apache.org>.
hemantk-12 commented on code in PR #4125:
URL: https://github.com/apache/ozone/pull/4125#discussion_r1087167605
##########
hadoop-ozone/tools/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneAddress.java:
##########
@@ -96,14 +108,37 @@ public void checkUrlTypes() throws OzoneClientException, IOException {
Assert.assertEquals("key1/key3/key", address.getKeyName());
Assert.assertFalse("this should not be a prefix",
address.isPrefix());
+ }
+ @Test
+ public void checkPrefixUrlType() throws OzoneClientException {
address = new OzoneAddress(prefix + "vol1/bucket/prefix");
address.ensurePrefixAddress();
Assert.assertEquals("vol1", address.getVolumeName());
Assert.assertEquals("bucket", address.getBucketName());
Assert.assertEquals("prefix", address.getKeyName());
Assert.assertTrue("this should be a prefix",
address.isPrefix());
+ }
+
+ @Test
+ public void checkSnapshotUrlType() throws OzoneClientException {
+ address = new OzoneAddress(prefix + "vol1/bucket/.snapshot/snap1");
+ address.ensureSnapshotAddress();
+ Assert.assertEquals("vol1", address.getVolumeName());
+ Assert.assertEquals("bucket", address.getBucketName());
+ Assert.assertEquals(".snapshot/snap1", address.getSnapshotName());
+ Assert.assertEquals(".snapshot/snap1", address.getKeyName());
+
+
+ String message = "Only a snapshot name with " +
+ "a snapshot indicator is accepted";
+
+ address = new OzoneAddress(prefix + "vol1/bucket/.snapshot");
+
+ exception.expect(OzoneClientException.class);
+ exception.expectMessage(message);
+ address.ensureSnapshotAddress();
}
}
Review Comment:
It is general practice to add new line at the end of the file. AFAIK, the idea comes from the Unix file system. There is no problem as such if it is not there. It is more of a coding guideline.
https://unix.stackexchange.com/questions/18743/whats-the-point-in-adding-a-new-line-to-the-end-of-a-file, https://gist.github.com/OleksiyRudenko/d51388345ea55767b7672307fe35adf3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on a diff in pull request #4125: HDDS-7277. [Snapshot] Implement snapshot key listing in Ozone Shell
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #4125:
URL: https://github.com/apache/ozone/pull/4125#discussion_r1087454364
##########
hadoop-ozone/tools/pom.xml:
##########
@@ -124,6 +124,10 @@ https://maven.apache.org/xsd/maven-4.0.0.xsd">
<artifactId>mockito-core</artifactId>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>org.junit.jupiter</groupId>
+ <artifactId>junit-jupiter-params</artifactId>
+ </dependency>
Review Comment:
Should this have a scope of `test`?
```suggestion
<artifactId>junit-jupiter-params</artifactId>
<scope>test</scope>
</dependency>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] hemantk-12 commented on pull request #4125: HDDS-7277. [Snapshot] Implement snapshot key listing in Ozone Shell
Posted by "hemantk-12 (via GitHub)" <gi...@apache.org>.
hemantk-12 commented on PR #4125:
URL: https://github.com/apache/ozone/pull/4125#issuecomment-1414509883
LGTM.
Feel free to merge once Siyao approves it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on a diff in pull request #4125: HDDS-7277. [Snapshot] ozone sh key list /vol1/bucket1/.snapshot/snap1 fails
Posted by GitBox <gi...@apache.org>.
xBis7 commented on code in PR #4125:
URL: https://github.com/apache/ozone/pull/4125#discussion_r1064866481
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/snapshot/ListSnapshotKeyHandler.java:
##########
@@ -0,0 +1,80 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.shell.snapshot;
+
+import com.google.common.base.Strings;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.OzoneClientException;
+import org.apache.hadoop.ozone.client.OzoneVolume;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneKey;
+import org.apache.hadoop.ozone.shell.ListOptions;
+import org.apache.hadoop.ozone.shell.OzoneAddress;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.Iterator;
+
+/**
+ * Executes List Keys for a Snapshot.
+ */
+@CommandLine.Command(name = "key-list",
+ aliases = "key-ls",
+ description = "list all keys in a given " +
+ "snapshot under a bucket")
+public class ListSnapshotKeyHandler extends SnapshotHandler {
Review Comment:
> But couldn't you also fix the "ozone sh key list" command by changing this line to be extended from "SnapshotHandler" instead of BucketHandler?:
@GeorgeJahad If we make that change then `ozone sh key list` will only work with snapshots and it will give us an error if we try to get the keys under a bucket.
```
bash-4.2$ ozone sh key list /vol1/bucket1
Invalid value for positional parameter at index 0 (<value>): cannot convert '/vol1/bucket1' to OzoneAddress (org.apache.hadoop.ozone.client.OzoneClientException: Snapshot name is missing.)
Usage: ozone sh key list [-hV] [-l=<limit>] [-p=<prefix>] [-s=<startItem>]
<value>
list all keys in a given bucket
...
```
but it works for a snapshot
```
bash-4.2$ ozone sh key list /vol1/bucket1/.snapshot/snap1
[ {
"volumeName" : "vol1",
"bucketName" : "bucket1",
"name" : "key1",
"dataSize" : 3811,
"creationTime" : "2023-01-09T16:56:04.248Z",
"modificationTime" : "2023-01-09T16:56:05.308Z",
"replicationConfig" : {
"replicationFactor" : "THREE",
"requiredNodes" : 3,
"replicationType" : "RATIS"
}
} ]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on a diff in pull request #4125: HDDS-7277. [Snapshot] ozone sh key list /vol1/bucket1/.snapshot/snap1 fails
Posted by GitBox <gi...@apache.org>.
xBis7 commented on code in PR #4125:
URL: https://github.com/apache/ozone/pull/4125#discussion_r1064866481
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/snapshot/ListSnapshotKeyHandler.java:
##########
@@ -0,0 +1,80 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.shell.snapshot;
+
+import com.google.common.base.Strings;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.OzoneClientException;
+import org.apache.hadoop.ozone.client.OzoneVolume;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneKey;
+import org.apache.hadoop.ozone.shell.ListOptions;
+import org.apache.hadoop.ozone.shell.OzoneAddress;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.Iterator;
+
+/**
+ * Executes List Keys for a Snapshot.
+ */
+@CommandLine.Command(name = "key-list",
+ aliases = "key-ls",
+ description = "list all keys in a given " +
+ "snapshot under a bucket")
+public class ListSnapshotKeyHandler extends SnapshotHandler {
Review Comment:
> But couldn't you also fix the "ozone sh key list" command by changing this line to be extended from "SnapshotHandler" instead of BucketHandler?:
@GeorgeJahad If we make that change then `ozone sh key list` will only work with snapshots and it will give us an error if we try to get the keys under a bucket.
```
bash-4.2$ ozone sh key list /vol1/bucket1
Invalid value for positional parameter at index 0 (<value>): cannot convert '/vol1/bucket1' to OzoneAddress (org.apache.hadoop.ozone.client.OzoneClientException: Snapshot name is missing.)
Usage: ozone sh key list [-hV] [-l=<limit>] [-p=<prefix>] [-s=<startItem>]
<value>
...
...
bash-4.2$ ozone sh key list /vol1/bucket1/.snapshot/snap1
[ {
"volumeName" : "vol1",
"bucketName" : "bucket1",
"name" : "key1",
"dataSize" : 3811,
"creationTime" : "2023-01-09T16:56:04.248Z",
"modificationTime" : "2023-01-09T16:56:05.308Z",
"replicationConfig" : {
"replicationFactor" : "THREE",
"requiredNodes" : 3,
"replicationType" : "RATIS"
}
} ]
```
It requires the snapshot to be provided and the key we are getting using the snapshot path, is the bucket key and not the snapshot key.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] GeorgeJahad commented on pull request #4125: HDDS-7277. [Snapshot] ozone sh key list /vol1/bucket1/.snapshot/snap1 fails
Posted by GitBox <gi...@apache.org>.
GeorgeJahad commented on PR #4125:
URL: https://github.com/apache/ozone/pull/4125#issuecomment-1384481468
+1 LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org