You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2021/04/29 13:43:52 UTC
[GitHub] [ozone] symious opened a new pull request #2198: HDDS-5167. Add clean option for Freon randomkeys to clean generated objects
symious opened a new pull request #2198:
URL: https://github.com/apache/ozone/pull/2198
## What changes were proposed in this pull request?
Freon randomkeys would generate a lot of random volumes, buckets and keys, it would be quite difficult for users to delete these objects. This patch added a new option named "--clean-objects" to clean all the generated objects.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-5167
## How was this patch tested?
Unit test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] symious commented on pull request #2198: HDDS-5167. Add clean option for Freon randomkeys to clean generated objects
Posted by GitBox <gi...@apache.org>.
symious commented on pull request #2198:
URL: https://github.com/apache/ozone/pull/2198#issuecomment-836832201
@adoroszlai Thanks for the comment, I will update the patch later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] symious commented on a change in pull request #2198: HDDS-5167. Add clean option for Freon randomkeys to clean generated objects
Posted by GitBox <gi...@apache.org>.
symious commented on a change in pull request #2198:
URL: https://github.com/apache/ozone/pull/2198#discussion_r632515188
##########
File path: hadoop-ozone/dist/src/main/smoketest/freon/remove.robot
##########
@@ -0,0 +1,43 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+*** Settings ***
+Documentation Test freon data remove commands
+Resource ../lib/os.robot
+Test Timeout 5 minutes
+
+*** Variables ***
+${PREFIX} ${EMPTY}
+
+*** Test Cases ***
+Ozone Client Remover (Error)
+ ${result} = Execute and checkrc ozone freon ocr ${OM_HA_PARAM} -t=1 -n=1 -p ocr${PREFIX} --remove-key --remove-bucket 255
+ Should contain ${result} Invalid Option
+
+Ozone Client Key Generator For Remover
+ ${result} = Execute ozone freon ockg ${OM_HA_PARAM} -t=1 -n=1 -p ocr${PREFIX}
+ Should contain ${result} Successful executions: 1
+
+Ozone Client Remover (KEY)
+ ${result} = Execute ozone freon ocr ${OM_HA_PARAM} -t=1 -n=1 -p ocr${PREFIX} --remove-key
+ Should contain ${result} Successful executions: 1
+
+OM Bucket Generator For Remover
+ ${result} = Execute ozone freon ombg ${OM_HA_PARAM} -t=1 -n=1 -p ocr${PREFIX}
+ Should contain ${result} Successful executions: 1
Review comment:
Changed to `Setup` in robot script.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a change in pull request #2198: HDDS-5167. Add clean option for Freon randomkeys to clean generated objects
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #2198:
URL: https://github.com/apache/ozone/pull/2198#discussion_r629335566
##########
File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/BaseFreonGenerator.java
##########
@@ -406,6 +432,24 @@ public void ensureVolumeExists(
} catch (OMException ex) {
if (ex.getResult() == ResultCodes.VOLUME_NOT_FOUND) {
rpcClient.getObjectStore().createVolume(volumeName);
+ volumeCreated = true;
+ } else {
+ throw ex;
+ }
+ }
+ }
+
+ /**
+ * Create missing target bucket for S3KeyGenerator.
+ */
+ public void ensureBucketExist(AmazonS3 s3, String bucketName) {
+ HeadBucketRequest headBucketRequest = new HeadBucketRequest(bucketName);
+ try {
+ s3.headBucket(headBucketRequest);
+ } catch (AmazonServiceException ex) {
+ if (ex.getErrorCode().equals("NoSuchBucket")) {
+ s3.createBucket(bucketName);
+ setBucketCreated();
Review comment:
I don't think S3-specific code belongs to `BaseFreonGenerator`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] symious commented on pull request #2198: HDDS-5167. Add clean option for Freon randomkeys to clean generated objects
Posted by GitBox <gi...@apache.org>.
symious commented on pull request #2198:
URL: https://github.com/apache/ozone/pull/2198#issuecomment-831094782
@adoroszlai Thanks for the review and advice. I will try to generalize the clean feature for the framework.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai merged pull request #2198: HDDS-5167. Add clean option for Freon randomkeys to clean generated objects
Posted by GitBox <gi...@apache.org>.
adoroszlai merged pull request #2198:
URL: https://github.com/apache/ozone/pull/2198
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] symious commented on pull request #2198: HDDS-5167. Add clean option for Freon randomkeys to clean generated objects
Posted by GitBox <gi...@apache.org>.
symious commented on pull request #2198:
URL: https://github.com/apache/ozone/pull/2198#issuecomment-842774190
@adoroszlai Thanks for the detailed review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a change in pull request #2198: HDDS-5167. Add clean option for Freon randomkeys to clean generated objects
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #2198:
URL: https://github.com/apache/ozone/pull/2198#discussion_r631985263
##########
File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientRemover.java
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+import com.codahale.metrics.Timer;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.OzoneVolume;
+import picocli.CommandLine.Command;
+import picocli.CommandLine.Option;
+
+import java.util.concurrent.Callable;
+
+/**
+ * Data remover tool test om performance.
+ */
+@Command(name = "ocr",
+ aliases = "ozone-client-remover",
+ description = "Remove keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientRemover extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @Option(names = {"-v", "--volume"},
+ description = "Name of the bucket which contains the test data. Will be"
+ + " created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data. Will be"
+ + " ignored when \"--remove-bucket\" is set.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @Option(
+ names = "--om-service-id",
+ description = "OM Service ID"
+ )
+ private String omServiceID = null;
+
+ @Option(names = {"--remove-bucket"},
+ description = "If turned on, Remover will be used to remove buckets. "
+ + "Can only set one option between \"--remove-key\" "
+ + "and \"--remove-bucket\"")
+ private boolean isRemoveBucket;
+
+ @Option(names = {"--remove-key"},
+ description = "If turned on, Remover will be used to remove keys. "
+ + "Can only set one option between \"--remove-key\" "
+ + "and \"--remove-bucket\"")
+ private boolean isRemoveKey;
Review comment:
By default both of these flags are `false`. If invoked without either, the program will spin forever, not removing anything:
```
$ docker-compose exec -T scm ozone freon ocr
2021-05-13 17:38:51,277 [main] INFO impl.MetricsConfig: Loaded properties from hadoop-metrics2.properties
2021-05-13 17:38:51,371 [main] INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
2021-05-13 17:38:51,372 [main] INFO impl.MetricsSystemImpl: ozone-freon metrics system started
2021-05-13 17:38:51,475 [main] INFO freon.BaseFreonGenerator: Executing test with prefix e90lalfdcs
2021-05-13 17:38:51,484 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0 out of 1000)
2021-05-13 17:38:52,488 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0 out of 1000)
2021-05-13 17:38:53,490 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0 out of 1000)
2021-05-13 17:38:54,492 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0 out of 1000)
2021-05-13 17:38:55,493 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0 out of 1000)
^C
```
`checkOption()` should also verify that one of these options is selected.
I think this, and the fact that `--bucket` is `ignored when "--remove-bucket" is set`, indicates that key remover and bucket remover should be separate, just like `OzoneClientKeyGenerator` and `OmBucketGenerator`.
##########
File path: hadoop-ozone/dist/src/main/smoketest/freon/remove.robot
##########
@@ -0,0 +1,43 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+*** Settings ***
+Documentation Test freon data remove commands
+Resource ../lib/os.robot
+Test Timeout 5 minutes
+
+*** Variables ***
+${PREFIX} ${EMPTY}
+
+*** Test Cases ***
+Ozone Client Remover (Error)
+ ${result} = Execute and checkrc ozone freon ocr ${OM_HA_PARAM} -t=1 -n=1 -p ocr${PREFIX} --remove-key --remove-bucket 255
+ Should contain ${result} Invalid Option
+
+Ozone Client Key Generator For Remover
+ ${result} = Execute ozone freon ockg ${OM_HA_PARAM} -t=1 -n=1 -p ocr${PREFIX}
+ Should contain ${result} Successful executions: 1
Review comment:
Ideally this should be a `[setup]` step, not a separate test case.
http://robotframework.org/robotframework/3.2.1/RobotFrameworkUserGuide.html#test-setup-and-teardown
##########
File path: hadoop-ozone/dist/src/main/smoketest/freon/remove.robot
##########
@@ -0,0 +1,43 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+*** Settings ***
+Documentation Test freon data remove commands
+Resource ../lib/os.robot
+Test Timeout 5 minutes
+
+*** Variables ***
+${PREFIX} ${EMPTY}
+
+*** Test Cases ***
+Ozone Client Remover (Error)
+ ${result} = Execute and checkrc ozone freon ocr ${OM_HA_PARAM} -t=1 -n=1 -p ocr${PREFIX} --remove-key --remove-bucket 255
+ Should contain ${result} Invalid Option
+
+Ozone Client Key Generator For Remover
+ ${result} = Execute ozone freon ockg ${OM_HA_PARAM} -t=1 -n=1 -p ocr${PREFIX}
+ Should contain ${result} Successful executions: 1
+
+Ozone Client Remover (KEY)
+ ${result} = Execute ozone freon ocr ${OM_HA_PARAM} -t=1 -n=1 -p ocr${PREFIX} --remove-key
+ Should contain ${result} Successful executions: 1
+
+OM Bucket Generator For Remover
+ ${result} = Execute ozone freon ombg ${OM_HA_PARAM} -t=1 -n=1 -p ocr${PREFIX}
+ Should contain ${result} Successful executions: 1
Review comment:
Same here regarding `[setup]`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] symious commented on pull request #2198: HDDS-5167. Add clean option for Freon randomkeys to clean generated objects
Posted by GitBox <gi...@apache.org>.
symious commented on pull request #2198:
URL: https://github.com/apache/ozone/pull/2198#issuecomment-838654259
@adoroszlai Updated a new version of remover, could you help to check?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] symious commented on a change in pull request #2198: HDDS-5167. Add clean option for Freon randomkeys to clean generated objects
Posted by GitBox <gi...@apache.org>.
symious commented on a change in pull request #2198:
URL: https://github.com/apache/ozone/pull/2198#discussion_r630267113
##########
File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/BaseFreonGenerator.java
##########
@@ -406,6 +432,24 @@ public void ensureVolumeExists(
} catch (OMException ex) {
if (ex.getResult() == ResultCodes.VOLUME_NOT_FOUND) {
rpcClient.getObjectStore().createVolume(volumeName);
+ volumeCreated = true;
+ } else {
+ throw ex;
+ }
+ }
+ }
+
+ /**
+ * Create missing target bucket for S3KeyGenerator.
+ */
+ public void ensureBucketExist(AmazonS3 s3, String bucketName) {
+ HeadBucketRequest headBucketRequest = new HeadBucketRequest(bucketName);
+ try {
+ s3.headBucket(headBucketRequest);
+ } catch (AmazonServiceException ex) {
+ if (ex.getErrorCode().equals("NoSuchBucket")) {
+ s3.createBucket(bucketName);
+ setBucketCreated();
Review comment:
Removed S3 code from `baseFreonGenerator`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] symious commented on a change in pull request #2198: HDDS-5167. Add clean option for Freon randomkeys to clean generated objects
Posted by GitBox <gi...@apache.org>.
symious commented on a change in pull request #2198:
URL: https://github.com/apache/ozone/pull/2198#discussion_r632515341
##########
File path: hadoop-ozone/dist/src/main/smoketest/freon/remove.robot
##########
@@ -0,0 +1,43 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+*** Settings ***
+Documentation Test freon data remove commands
+Resource ../lib/os.robot
+Test Timeout 5 minutes
+
+*** Variables ***
+${PREFIX} ${EMPTY}
+
+*** Test Cases ***
+Ozone Client Remover (Error)
+ ${result} = Execute and checkrc ozone freon ocr ${OM_HA_PARAM} -t=1 -n=1 -p ocr${PREFIX} --remove-key --remove-bucket 255
+ Should contain ${result} Invalid Option
+
+Ozone Client Key Generator For Remover
+ ${result} = Execute ozone freon ockg ${OM_HA_PARAM} -t=1 -n=1 -p ocr${PREFIX}
+ Should contain ${result} Successful executions: 1
Review comment:
Changed to `Setup` in robot script.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] symious commented on a change in pull request #2198: HDDS-5167. Add clean option for Freon randomkeys to clean generated objects
Posted by GitBox <gi...@apache.org>.
symious commented on a change in pull request #2198:
URL: https://github.com/apache/ozone/pull/2198#discussion_r632515774
##########
File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientRemover.java
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+import com.codahale.metrics.Timer;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.OzoneVolume;
+import picocli.CommandLine.Command;
+import picocli.CommandLine.Option;
+
+import java.util.concurrent.Callable;
+
+/**
+ * Data remover tool test om performance.
+ */
+@Command(name = "ocr",
+ aliases = "ozone-client-remover",
+ description = "Remove keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientRemover extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @Option(names = {"-v", "--volume"},
+ description = "Name of the bucket which contains the test data. Will be"
+ + " created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data. Will be"
+ + " ignored when \"--remove-bucket\" is set.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @Option(
+ names = "--om-service-id",
+ description = "OM Service ID"
+ )
+ private String omServiceID = null;
+
+ @Option(names = {"--remove-bucket"},
+ description = "If turned on, Remover will be used to remove buckets. "
+ + "Can only set one option between \"--remove-key\" "
+ + "and \"--remove-bucket\"")
+ private boolean isRemoveBucket;
+
+ @Option(names = {"--remove-key"},
+ description = "If turned on, Remover will be used to remove keys. "
+ + "Can only set one option between \"--remove-key\" "
+ + "and \"--remove-bucket\"")
+ private boolean isRemoveKey;
Review comment:
Created separate `OzoneClientKeyRemover` and `OmBucketRemover`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] symious commented on pull request #2198: HDDS-5167. Add clean option for Freon randomkeys to clean generated objects
Posted by GitBox <gi...@apache.org>.
symious commented on pull request #2198:
URL: https://github.com/apache/ozone/pull/2198#issuecomment-841233953
@adoroszlai Thanks for the detailed review, have updated the patch to resolve the conversation. Could you help to check?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] symious commented on a change in pull request #2198: HDDS-5167. Add clean option for Freon randomkeys to clean generated objects
Posted by GitBox <gi...@apache.org>.
symious commented on a change in pull request #2198:
URL: https://github.com/apache/ozone/pull/2198#discussion_r630267549
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/freon/TestOzoneClientKeyGenerator.java
##########
@@ -91,4 +92,41 @@ public void testOzoneClientKeyGenerator() throws Exception {
shutdown(cluster);
}
+ @Test
+ public void testCleanUpOzoneClientKeyGenerator() throws Exception {
Review comment:
Added Robot test for remover.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] symious commented on pull request #2198: HDDS-5167. Add clean option for Freon randomkeys to clean generated objects
Posted by GitBox <gi...@apache.org>.
symious commented on pull request #2198:
URL: https://github.com/apache/ozone/pull/2198#issuecomment-836699926
@adoroszlai Thanks for the detailed clarification.
If we separate the class for the delete function, I think one class should work fine for all the key and bucket generators? I didn't do that because I found that class quite dangerous, removing all the keys in a bucket or all buckets in a volume. It's like planting a bomb that can be misused by malicious people or users not familiar with the tool. So I build the clean option inside each generator making sure only the generated objects get cleaned.
It's my personal view, if this is not a problem, then I'll build a separate class for the deleting function.
For the performance monitoring, I think the data are deleted by datanodes asynchronously, seems we can't monitor the deletion precisely?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] symious commented on pull request #2198: HDDS-5167. Add clean option for Freon randomkeys to clean generated objects
Posted by GitBox <gi...@apache.org>.
symious commented on pull request #2198:
URL: https://github.com/apache/ozone/pull/2198#issuecomment-831855274
@adoroszlai Updated the version to generalize the clean operation, could you help to check?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a change in pull request #2198: HDDS-5167. Add clean option for Freon randomkeys to clean generated objects
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #2198:
URL: https://github.com/apache/ozone/pull/2198#discussion_r629490135
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/freon/TestOzoneClientKeyGenerator.java
##########
@@ -91,4 +92,41 @@ public void testOzoneClientKeyGenerator() throws Exception {
shutdown(cluster);
}
+ @Test
+ public void testCleanUpOzoneClientKeyGenerator() throws Exception {
Review comment:
Also, please note that this test class is being removed in #2194. I think similar test case can be added to acceptance tests (`hadoop-ozone/dist/src/main/smoketest/freon`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on pull request #2198: HDDS-5167. Add clean option for Freon randomkeys to clean generated objects
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on pull request #2198:
URL: https://github.com/apache/ozone/pull/2198#issuecomment-836737466
> removing all the keys in a bucket or all buckets in a volume
Freon creates objects with a specific pattern, composed of a prefix and a counter. The prefix is either random or user-specified. The validator can only be used with a user-specified prefix, as it has to read previously created keys. The counter starts at 0 and counts up to N-1, where N is user-specified.
Similarly, the new cleaner would only remove items `$prefix/0`, `$prefix/1`, etc. with the given prefix. I don't think that would be a dangerous tool.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org