You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Montura (via GitHub)" <gi...@apache.org> on 2024/04/23 13:03:23 UTC

[PR] HDDS-10742. Close all pipelines subcommand [ozone]

Montura opened a new pull request, #6577:
URL: https://github.com/apache/ozone/pull/6577

   ## What changes were proposed in this pull request?
   Suggested a new subcommand to close all pipelines (or some filtered pipeline subset)
   
   Please describe your PR in detail:
   For now ozone CLI allows to [get pipelines list](https://github.com/apache/ozone/blob/master/hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/pipeline/ListPipelinesSubcommand.java#L49) or [close pipeline](https://github.com/apache/ozone/blob/master/hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/pipeline/ClosePipelineSubcommand.java) by PipelineId.
   
   Suggested a new command to close all pipelines (or some filtered pipeline subset):
     1.  get pipeline list from `ScmClient`
     2. filter `CLOSED` pipelines (to close only `OPEN` and `ALLOCATED`)
     3. filter pipelines by `REPLICATION` or `REPLICATION_TYPE` or `REPLICATION_FACTOR` (as in [ListPipelinesSubcommand](https://github.com/apache/ozone/blob/master/hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/pipeline/ListPipelinesSubcommand.java#L103) done)
       close the remain pipelines set
   
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-10742
   
   ## How was this patch tested?
   
   This subcommand triggers [ClosePipelineSubcommand](https://github.com/apache/ozone/blob/master/hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/pipeline/ClosePipelineSubcommand.java)  that is already tested.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10742. Add option to close all pipelines [ozone]

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #6577:
URL: https://github.com/apache/ozone/pull/6577#issuecomment-2104183147

   Thanks @Montura for the patch, @errose28 for the review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10742. Close all pipelines subcommand [ozone]

Posted by "Montura (via GitHub)" <gi...@apache.org>.
Montura commented on code in PR #6577:
URL: https://github.com/apache/ozone/pull/6577#discussion_r1577352164


##########
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/pipeline/CloseAllPipelinesSubcommand.java:
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.cli.pipeline;
+
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.client.ReplicationFactor;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.scm.cli.ScmSubcommand;
+import org.apache.hadoop.hdds.scm.client.ScmClient;
+import org.apache.hadoop.hdds.scm.pipeline.Pipeline;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.Optional;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+
+/**
+ * Handler of close pipeline command.
+ */
+@CommandLine.Command(
+    name = "closeAll",

Review Comment:
   @adoroszlai, thanks a lot!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10742. Add option to close all pipelines [ozone]

Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #6577:
URL: https://github.com/apache/ozone/pull/6577#discussion_r1583761758


##########
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/pipeline/ClosePipelineSubcommand.java:
##########
@@ -51,7 +51,7 @@ public class ClosePipelineSubcommand extends ScmSubcommand {
   public void execute(ScmClient scmClient) throws IOException {
     if (!Strings.isNullOrEmpty(closeOption.pipelineId)) {
       if (filterOptions.getReplicationFilter().isPresent()) {
-        System.out.println("When close pipeline by ID filters are ignored.");
+        System.err.println("Replication filters can only be used with --all");

Review Comment:
   Sorry if the last comment was not clear. We should also fail the command here instead of closing the pipeline anyways since the user did something invalid. We can probably throw `IllegalArgumentException` like `FilterPipelineOptions#getReplicationFilter` does. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10742. Add option to close all pipelines [ozone]

Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #6577:
URL: https://github.com/apache/ozone/pull/6577#discussion_r1580237914


##########
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/pipeline/ClosePipelineSubcommand.java:
##########
@@ -35,13 +41,49 @@
     mixinStandardHelpOptions = true,
     versionProvider = HddsVersionProvider.class)
 public class ClosePipelineSubcommand extends ScmSubcommand {
+  @CommandLine.ArgGroup(multiplicity = "1")
+  private CloseOptionGroup closeOption;
 
-  @CommandLine.Parameters(description = "ID of the pipeline to close")
-  private String pipelineId;
+  @CommandLine.Mixin
+  private final FilterPipelineOptions filterOptions = new FilterPipelineOptions();
 
   @Override
   public void execute(ScmClient scmClient) throws IOException {
-    scmClient.closePipeline(
-        HddsProtos.PipelineID.newBuilder().setId(pipelineId).build());
+    if (!Strings.isNullOrEmpty(closeOption.pipelineId)) {
+      if (filterOptions.getReplicationFilter().isPresent()) {
+        System.out.println("When close pipeline by ID filters are ignored.");

Review Comment:
   ```suggestion
           System.err.println("Replication filters can only be used with --all");
   ```
   
   This check looks like a good way to do it. I think we should fail the command with a message like this to make sure the command is not doing something the user did not expect given the combination of flags they tried to use.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10742. Add option to close all pipelines [ozone]

Posted by "Montura (via GitHub)" <gi...@apache.org>.
Montura commented on code in PR #6577:
URL: https://github.com/apache/ozone/pull/6577#discussion_r1590580908


##########
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/pipeline/ClosePipelineSubcommand.java:
##########
@@ -51,7 +51,7 @@ public class ClosePipelineSubcommand extends ScmSubcommand {
   public void execute(ScmClient scmClient) throws IOException {
     if (!Strings.isNullOrEmpty(closeOption.pipelineId)) {
       if (filterOptions.getReplicationFilter().isPresent()) {
-        System.out.println("When close pipeline by ID filters are ignored.");
+        System.err.println("Replication filters can only be used with --all");

Review Comment:
   @errose28, could you comment please the [previous](https://github.com/apache/ozone/pull/6577#discussion_r1587316937) message  



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10742. Close all pipelines subcommand [ozone]

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #6577:
URL: https://github.com/apache/ozone/pull/6577#discussion_r1577322427


##########
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/pipeline/CloseAllPipelinesSubcommand.java:
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.cli.pipeline;
+
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.client.ReplicationFactor;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.scm.cli.ScmSubcommand;
+import org.apache.hadoop.hdds.scm.client.ScmClient;
+import org.apache.hadoop.hdds.scm.pipeline.Pipeline;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.Optional;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+
+/**
+ * Handler of close pipeline command.
+ */
+@CommandLine.Command(
+    name = "closeAll",

Review Comment:
   Arity means the number of times the parameter can/should be given to the command.
   
   You want to use the new options (`--all` or the ones in the original patch) without any pipeline ID.  The command will complain about that:
   
   ```
   Missing required parameter: '<pipelineId>'
   ```
   
   `arity = "0..1"` makes the parameter optional.
   
   https://picocli.info/#_optional_values



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10742. Close all pipelines subcommand [ozone]

Posted by "Montura (via GitHub)" <gi...@apache.org>.
Montura commented on code in PR #6577:
URL: https://github.com/apache/ozone/pull/6577#discussion_r1576819846


##########
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/pipeline/CloseAllPipelinesSubcommand.java:
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.cli.pipeline;
+
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.client.ReplicationFactor;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.scm.cli.ScmSubcommand;
+import org.apache.hadoop.hdds.scm.client.ScmClient;
+import org.apache.hadoop.hdds.scm.pipeline.Pipeline;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.Optional;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+
+/**
+ * Handler of close pipeline command.
+ */
+@CommandLine.Command(
+    name = "closeAll",

Review Comment:
   Sorry, but I don't understand waht do you mean about `pipeline ID can be changed to arity = 0..1.`. 
   
   I supose I can just add one more cmd option for `ClosePipelineSubcommand`
   ```
    @CommandLine.Parameters(names = {"--all"},
                    description = "ID of the pipeline to close",
                    defaultValue = "")
     private String closeAll;
   ``` 
   
   So, there are two scenarious:
   1. `ozone admin pipeline close PIPELINE_ID`
   2. `ozone admin pipeline close --all`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10742. Close all pipelines with ClosePipelineSubcommand [ozone]

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #6577:
URL: https://github.com/apache/ozone/pull/6577#discussion_r1577784337


##########
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/pipeline/FilterPipelineOptions.java:
##########
@@ -0,0 +1,65 @@
+package org.apache.hadoop.hdds.scm.cli.pipeline;
+
+import com.google.common.base.Strings;
+import org.apache.hadoop.hdds.client.RatisReplicationConfig;
+import org.apache.hadoop.hdds.client.ReplicationConfig;
+import org.apache.hadoop.hdds.client.ReplicationFactor;
+import org.apache.hadoop.hdds.client.ReplicationType;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.scm.pipeline.Pipeline;
+import picocli.CommandLine;
+
+import java.util.Optional;
+import java.util.function.Predicate;
+
+public class FilterPipelineOptions {

Review Comment:
   https://github.com/unisonteam/ozone/actions/runs/8813149705/job/24190292814#step:7:12
   
   Javadoc comment is required for public classes.



##########
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/pipeline/ClosePipelineSubcommand.java:
##########
@@ -35,13 +40,52 @@
     mixinStandardHelpOptions = true,
     versionProvider = HddsVersionProvider.class)
 public class ClosePipelineSubcommand extends ScmSubcommand {
+  @CommandLine.ArgGroup(multiplicity = "1")
+  private CloseOptionGroup closeOption;
 
-  @CommandLine.Parameters(description = "ID of the pipeline to close")
-  private String pipelineId;
+  @CommandLine.Mixin
+  private final FilterPipelineOptions filterOptions = new FilterPipelineOptions();
 
   @Override
   public void execute(ScmClient scmClient) throws IOException {
-    scmClient.closePipeline(
-        HddsProtos.PipelineID.newBuilder().setId(pipelineId).build());
+    if (!Strings.isNullOrEmpty(closeOption.pipelineId)) {
+      scmClient.closePipeline(HddsProtos.PipelineID.newBuilder().setId(closeOption.pipelineId).build());
+    }
+
+    if (closeOption.closeAll) {
+      Optional<Predicate<? super Pipeline>> replicationFilter = filterOptions.getReplicationFilter();
+
+      Stream<Pipeline> stream = scmClient.listPipelines()
+          .stream()
+          .filter(p -> p.getPipelineState() != Pipeline.PipelineState.CLOSED);
+      if (replicationFilter.isPresent()) {
+        stream = stream.filter(replicationFilter.get());
+      }
+
+      stream.forEach(pipeline -> {
+        try {
+          scmClient.closePipeline(
+              HddsProtos.PipelineID.newBuilder().setId(pipeline.getId().getId().toString()).build());
+        } catch (IOException e) {
+          System.err.println("Error closing pipeline: " + pipeline.getId() + ", cause: " + e.getMessage());
+        }
+      });
+    }
+  }
+
+  private static class CloseOptionGroup {
+    @CommandLine.Option(
+        names = {"pipelineId"},
+        arity = "1",
+        defaultValue = "",
+        description = "ID of the pipeline to close")
+    private String pipelineId;
+
+    @CommandLine.Option(
+        names = {"--all"},
+        arity = "0",

Review Comment:
   The command:
   
   ```
   $ ozone admin pipeline close --all
   ```
   
   does nothing, maybe due to `arity=0`.



##########
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/pipeline/ClosePipelineSubcommand.java:
##########
@@ -35,13 +40,52 @@
     mixinStandardHelpOptions = true,
     versionProvider = HddsVersionProvider.class)
 public class ClosePipelineSubcommand extends ScmSubcommand {
+  @CommandLine.ArgGroup(multiplicity = "1")
+  private CloseOptionGroup closeOption;
 
-  @CommandLine.Parameters(description = "ID of the pipeline to close")
-  private String pipelineId;
+  @CommandLine.Mixin
+  private final FilterPipelineOptions filterOptions = new FilterPipelineOptions();
 
   @Override
   public void execute(ScmClient scmClient) throws IOException {
-    scmClient.closePipeline(
-        HddsProtos.PipelineID.newBuilder().setId(pipelineId).build());
+    if (!Strings.isNullOrEmpty(closeOption.pipelineId)) {
+      scmClient.closePipeline(HddsProtos.PipelineID.newBuilder().setId(closeOption.pipelineId).build());
+    }
+
+    if (closeOption.closeAll) {
+      Optional<Predicate<? super Pipeline>> replicationFilter = filterOptions.getReplicationFilter();
+
+      Stream<Pipeline> stream = scmClient.listPipelines()
+          .stream()
+          .filter(p -> p.getPipelineState() != Pipeline.PipelineState.CLOSED);
+      if (replicationFilter.isPresent()) {
+        stream = stream.filter(replicationFilter.get());
+      }
+
+      stream.forEach(pipeline -> {
+        try {
+          scmClient.closePipeline(
+              HddsProtos.PipelineID.newBuilder().setId(pipeline.getId().getId().toString()).build());
+        } catch (IOException e) {
+          System.err.println("Error closing pipeline: " + pipeline.getId() + ", cause: " + e.getMessage());
+        }
+      });
+    }
+  }
+
+  private static class CloseOptionGroup {
+    @CommandLine.Option(
+        names = {"pipelineId"},

Review Comment:
   This should still be a `Parameter` (which is a value without a name).  Currently the command gives error:
   
   ```
   $ ozone admin pipeline close 123
   Unmatched argument at index 2: '123'
   ```



##########
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/pipeline/FilterPipelineOptions.java:
##########
@@ -0,0 +1,65 @@
+package org.apache.hadoop.hdds.scm.cli.pipeline;

Review Comment:
   https://github.com/unisonteam/ozone/actions/runs/8813149705/job/24190293734#step:7:11
   
   License header is needed in all files.  Example:
   
   https://github.com/apache/ozone/blob/131eec0587adefa9131eadef046dd20dc7a0dda9/hadoop-hdds/config/src/main/java/org/apache/hadoop/hdds/conf/ConfigFileAppender.java#L1-L17



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10742. Add option to close all pipelines [ozone]

Posted by "Montura (via GitHub)" <gi...@apache.org>.
Montura commented on code in PR #6577:
URL: https://github.com/apache/ozone/pull/6577#discussion_r1587316937


##########
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/pipeline/ClosePipelineSubcommand.java:
##########
@@ -51,7 +51,7 @@ public class ClosePipelineSubcommand extends ScmSubcommand {
   public void execute(ScmClient scmClient) throws IOException {
     if (!Strings.isNullOrEmpty(closeOption.pipelineId)) {
       if (filterOptions.getReplicationFilter().isPresent()) {
-        System.out.println("When close pipeline by ID filters are ignored.");
+        System.err.println("Replication filters can only be used with --all");

Review Comment:
   @errose28, let's sync :)
   
   1. We can log an error and don't close pipeline (`if/else` block): I have to fix current behaviour with `else` block.
   ```
   if (filterOptions.getReplicationFilter().isPresent()) {
     System.err.println("Replication filters can only be used with --all");
   } else {
     scmClient.closePipeline(...);
   }
   ```
   2. We can log an error and throw and exception (`if block`), like `org.apache.hadoop.hdds.scm.cli.cert.InfoSubcommand` [does](https://github.com/apache/ozone/blob/master/hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/cert/InfoSubcommand.java#L59): I have to fix current behaviour with `throw` an exception.
   ```
   if (filterOptions.getReplicationFilter().isPresent()) {
     System.err.println("Replication filters can only be used with --all");
     throw new IllegalArgumentException("Replication filters can only be used with --all");
   }
   scmClient.closePipeline(...);
   ```
   3. We can just throw and exception (`if block`), like `org.apache.hadoop.hdds.scm.cli.pipeline.ListPipelineSubcommand` [does](https://github.com/apache/ozone/blob/master/hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/pipeline/ListPipelinesSubcommand.java#L110) : I have to fix current behaviour with `throw` an execption instead of log to `System.err`.
   ```
   if (filterOptions.getReplicationFilter().isPresent()) {
     throw new IllegalArgumentException("Replication filters can only be used with --all");
   }
   scmClient.closePipeline(...);
   ```
   
   What is the best one (or what do you expect) ? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10742. Add option to close all pipelines [ozone]

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai merged PR #6577:
URL: https://github.com/apache/ozone/pull/6577


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10742. Add option to close all pipelines [ozone]

Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on PR #6577:
URL: https://github.com/apache/ozone/pull/6577#issuecomment-2101735021

   Just tried out the suggestions. Once [this](https://github.com/apache/ozone/pull/6577#discussion_r1594845672) is done I think we are good to merge 👍 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10742. Add option to close all pipelines [ozone]

Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on PR #6577:
URL: https://github.com/apache/ozone/pull/6577#issuecomment-2099799922

   Hi @Montura apologies for the delay. It's late night here currently but I will check this out tomorrow. Thanks for outlining the options.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10742. Add option to close all pipelines [ozone]

Posted by "Montura (via GitHub)" <gi...@apache.org>.
Montura commented on PR #6577:
URL: https://github.com/apache/ozone/pull/6577#issuecomment-2102156811

   Done!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10742. Close all pipelines subcommand [ozone]

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #6577:
URL: https://github.com/apache/ozone/pull/6577#discussion_r1576493535


##########
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/pipeline/CloseAllPipelinesSubcommand.java:
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.cli.pipeline;
+
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.client.ReplicationFactor;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.scm.cli.ScmSubcommand;
+import org.apache.hadoop.hdds.scm.client.ScmClient;
+import org.apache.hadoop.hdds.scm.pipeline.Pipeline;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.Optional;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+
+/**
+ * Handler of close pipeline command.
+ */
+@CommandLine.Command(
+    name = "closeAll",

Review Comment:
   I suggest adding the new options to the existing `ozone admin pipeline close` command, instead of introducing a new command.  Existing parameter for pipeline ID can be changed to arity = 0..1.



##########
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/pipeline/ListPipelinesSubcommand.java:
##########
@@ -100,7 +100,9 @@ public void execute(ScmClient scmClient) throws IOException {
     }
   }
 
-  private Optional<Predicate<? super Pipeline>> getReplicationFilter() {
+  static Optional<Predicate<? super Pipeline>> getReplicationFilter(
+      String replication, ReplicationFactor factor, String replicationType
+  ) {

Review Comment:
   This method can then be moved to the mixin class.



##########
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/pipeline/CloseAllPipelinesSubcommand.java:
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.cli.pipeline;
+
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.client.ReplicationFactor;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.scm.cli.ScmSubcommand;
+import org.apache.hadoop.hdds.scm.client.ScmClient;
+import org.apache.hadoop.hdds.scm.pipeline.Pipeline;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.Optional;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+
+/**
+ * Handler of close pipeline command.
+ */
+@CommandLine.Command(
+    name = "closeAll",
+    description = "Close all pipelines",
+    mixinStandardHelpOptions = true,
+    versionProvider = HddsVersionProvider.class)
+public class CloseAllPipelinesSubcommand extends ScmSubcommand {
+
+  @CommandLine.Option(names = {"-t", "--type"},
+      description = "Filter listed pipelines by replication type, RATIS or EC",
+      defaultValue = "")
+  private String replicationType;
+
+  @CommandLine.Option(
+      names = {"-r", "--replication"},
+      description = "Filter listed pipelines by replication, eg ONE, THREE or "

Review Comment:
   Wording related to listing (e.g. `listed` here) should be omitted.



##########
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/pipeline/CloseAllPipelinesSubcommand.java:
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.cli.pipeline;
+
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.client.ReplicationFactor;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.scm.cli.ScmSubcommand;
+import org.apache.hadoop.hdds.scm.client.ScmClient;
+import org.apache.hadoop.hdds.scm.pipeline.Pipeline;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.Optional;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+
+/**
+ * Handler of close pipeline command.
+ */
+@CommandLine.Command(
+    name = "closeAll",
+    description = "Close all pipelines",
+    mixinStandardHelpOptions = true,
+    versionProvider = HddsVersionProvider.class)
+public class CloseAllPipelinesSubcommand extends ScmSubcommand {
+
+  @CommandLine.Option(names = {"-t", "--type"},
+      description = "Filter listed pipelines by replication type, RATIS or EC",
+      defaultValue = "")
+  private String replicationType;
+
+  @CommandLine.Option(
+      names = {"-r", "--replication"},
+      description = "Filter listed pipelines by replication, eg ONE, THREE or "
+          + "for EC rs-3-2-1024k",
+      defaultValue = "")
+  private String replication;
+
+  @CommandLine.Option(
+      names = {"-ffc", "--filterByFactor", "--filter-by-factor"},
+      description = "[deprecated] Filter pipelines by factor (e.g. ONE, THREE) "
+          + " (implies RATIS replication type)")
+  private ReplicationFactor factor;
+
+  @Override
+  public void execute(ScmClient scmClient) throws IOException {
+    Optional<Predicate<? super Pipeline>> replicationFilter =
+        ListPipelinesSubcommand.getReplicationFilter(replication, factor, replicationType);
+
+    Stream<Pipeline> stream =
+        scmClient.listPipelines()
+            .stream()
+            .filter(p -> p.getPipelineState() != Pipeline.PipelineState.CLOSED);
+    if (replicationFilter.isPresent()) {
+      stream = stream.filter(replicationFilter.get());
+    }
+
+    stream.forEach(pipeline -> {
+      try {
+        scmClient.closePipeline(HddsProtos.PipelineID.newBuilder().setId(pipeline.getId().getId().toString()).build());
+      } catch (IOException e) {
+        System.out.println("Can't close pipeline: " + pipeline.getId() + ", cause: " + e.getMessage());

Review Comment:
   Let's print problems to `System.err` instead.
   
   Also, I think `Error closing pipeline` sounds better than `Can't close pipeline`.



##########
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/pipeline/CloseAllPipelinesSubcommand.java:
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.cli.pipeline;
+
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.client.ReplicationFactor;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.scm.cli.ScmSubcommand;
+import org.apache.hadoop.hdds.scm.client.ScmClient;
+import org.apache.hadoop.hdds.scm.pipeline.Pipeline;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.Optional;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+
+/**
+ * Handler of close pipeline command.
+ */
+@CommandLine.Command(
+    name = "closeAll",
+    description = "Close all pipelines",
+    mixinStandardHelpOptions = true,
+    versionProvider = HddsVersionProvider.class)
+public class CloseAllPipelinesSubcommand extends ScmSubcommand {
+
+  @CommandLine.Option(names = {"-t", "--type"},
+      description = "Filter listed pipelines by replication type, RATIS or EC",
+      defaultValue = "")
+  private String replicationType;
+
+  @CommandLine.Option(
+      names = {"-r", "--replication"},
+      description = "Filter listed pipelines by replication, eg ONE, THREE or "
+          + "for EC rs-3-2-1024k",
+      defaultValue = "")
+  private String replication;
+
+  @CommandLine.Option(
+      names = {"-ffc", "--filterByFactor", "--filter-by-factor"},
+      description = "[deprecated] Filter pipelines by factor (e.g. ONE, THREE) "
+          + " (implies RATIS replication type)")
+  private ReplicationFactor factor;

Review Comment:
   Instead of duplicating these options from `ListPipelinesSubcommand`, please extract them into a separate class that can be used as a `Mixin` in both.  See `ListOptions` as an example.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10742. Add option to close all pipelines [ozone]

Posted by "Montura (via GitHub)" <gi...@apache.org>.
Montura commented on PR #6577:
URL: https://github.com/apache/ozone/pull/6577#issuecomment-2076469283

   >
   > * A command like `ozone admin pipeline close --replication=EC 1` will close pipeline 1 even if it is a Ratis pipeline.
   >   
   >   * It would be good to disallow the pipeline filter options when ID is specified, but I'm not sure how easy that is to do with pico CLI.
   
   * Befor my changes `ozone admin pipeline close PIPELINE_ID` was working only for specified `PIPELINE_ID`, so when user provides `PIPELINE_ID` he knows it's _replicataion factor_ and _replication type_. There were no filters for this command.
   * Filters for replication type and factor appiled only for `ozone admin pipeline list --filters` command. 
   * I thought that it would be nice to add the same filter option when run `ozone admin pipeline close --all` to be able to close only `EC` or only `RATIS` pipelines for example.
   
   So, I'm confused a bit about adding any filters for closing pipeline by ID.  
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10742. Add option to close all pipelines [ozone]

Posted by "Montura (via GitHub)" <gi...@apache.org>.
Montura commented on PR #6577:
URL: https://github.com/apache/ozone/pull/6577#issuecomment-2076472579

   > * Before my changes `ozone admin pipeline close PIPELINE_ID` was working only for specified `PIPELINE_ID`, so when user provides `PIPELINE_ID` he knows it's _replicataion factor_ and _replication type_. There were no filters for this command.
   
   I realized, @errose28 means that you call `ozone admin close pipeline 1 --replication=EC` and filter is ignored... 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10742. Close all pipelines subcommand [ozone]

Posted by "Montura (via GitHub)" <gi...@apache.org>.
Montura commented on code in PR #6577:
URL: https://github.com/apache/ozone/pull/6577#discussion_r1576819846


##########
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/pipeline/CloseAllPipelinesSubcommand.java:
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.cli.pipeline;
+
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.client.ReplicationFactor;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.scm.cli.ScmSubcommand;
+import org.apache.hadoop.hdds.scm.client.ScmClient;
+import org.apache.hadoop.hdds.scm.pipeline.Pipeline;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.Optional;
+import java.util.function.Predicate;
+import java.util.stream.Stream;
+
+/**
+ * Handler of close pipeline command.
+ */
+@CommandLine.Command(
+    name = "closeAll",

Review Comment:
   Sorry, but I don't understand waht do you mean about `pipeline ID can be changed to arity = 0..1.`. 
   
   I supose I can just add one more cmd option
   ```
    @CommandLine.Parameters(names = {"--all"},
                    description = "ID of the pipeline to close",
                    defaultValue = "")
     private String closeAll;
   ``` 
   
   So, there are two scenarious:
   1. `ozone admin pipeline close PIPELINE_ID`
   2. `ozone admin pipeline close --all`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10742. Add option to close all pipelines [ozone]

Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #6577:
URL: https://github.com/apache/ozone/pull/6577#discussion_r1594845672


##########
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/pipeline/ClosePipelineSubcommand.java:
##########
@@ -51,7 +51,7 @@ public class ClosePipelineSubcommand extends ScmSubcommand {
   public void execute(ScmClient scmClient) throws IOException {
     if (!Strings.isNullOrEmpty(closeOption.pipelineId)) {
       if (filterOptions.getReplicationFilter().isPresent()) {
-        System.out.println("When close pipeline by ID filters are ignored.");
+        System.err.println("Replication filters can only be used with --all");

Review Comment:
   I just tried all three of these to double check and option 3 is what we want. There is logic farther up to print just the message to stderr and not the stack trace. Option 2 will print the message twice. Option 1 will exit 0 when it should exit nonzero.
   
   Just looking at the example you provided I think cert.InfoSubcommand might print the message twice as well, but I did not try it out.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org