You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2022/07/27 10:59:04 UTC
[GitHub] [ozone] Zeddling opened a new pull request, #3629: HDDS-5779. Create s3g Grpc performance test
Zeddling opened a new pull request, #3629:
URL: https://github.com/apache/ozone/pull/3629
## What changes were proposed in this pull request?
Add test class for freon. It generates random buckets for the s3 gateway.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-5779
## How was this patch tested?
This patch was tested manually using a docker based secure cluster and using the ozone external terminal under ```hadoop-ozone/dist/target/ozone-X.X.X/bin/``` .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai merged pull request #3629: HDDS-5779. Create s3g bucket creation performance test
Posted by GitBox <gi...@apache.org>.
adoroszlai merged PR #3629:
URL: https://github.com/apache/ozone/pull/3629
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] neils-dev commented on pull request #3629: HDDS-5779. Create s3g bucket creation performance test
Posted by GitBox <gi...@apache.org>.
neils-dev commented on PR #3629:
URL: https://github.com/apache/ozone/pull/3629#issuecomment-1198556253
> Nit: Is the goal only to measure performance of a single bucket?
The test measures the performance of s3 bucket creation. Similar to the freon `s3kg` test, it tests the time to create the object from the _number-of-tests_ (objects created) specified by the `--number-of-tests` (`-n`) parameter. In other words measuring from creating multiple buckets specified by -n or, 1000 buckets by default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on pull request #3629: HDDS-5779. Create s3g bucket creation performance test
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on PR #3629:
URL: https://github.com/apache/ozone/pull/3629#issuecomment-1206201734
Thanks @Zeddling for enabling the workflow. There are some failures:
```
S3g bucket creation performance | FAIL |
Keyword 'Execute' failed after retrying for 2 minutes. The last error was: 1 != 0
```
https://github.com/Zeddling/ozone/runs/7681050738?check_suite_focus=true#step:5:349
https://github.com/Zeddling/ozone/runs/7681050967?check_suite_focus=true#step:5:175
```
S3g bucket creation performance | FAIL |
255 != 0
```
https://github.com/Zeddling/ozone/runs/7681050967?check_suite_focus=true#step:5:585
Robot and Docker logs are available in artifacts you can download from https://github.com/Zeddling/ozone/actions/runs/2799557796#artifacts Check for first failure in each `log.html`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on a diff in pull request #3629: HDDS-5779. Create s3g Grpc performance test
Posted by GitBox <gi...@apache.org>.
kerneltime commented on code in PR #3629:
URL: https://github.com/apache/ozone/pull/3629#discussion_r931496188
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/S3BucketGenerator.java:
##########
@@ -0,0 +1,99 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+import com.amazonaws.auth.EnvironmentVariableCredentialsProvider;
+import com.amazonaws.client.builder.AwsClientBuilder.EndpointConfiguration;
+import com.amazonaws.regions.Regions;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.AmazonS3ClientBuilder;
+import com.codahale.metrics.Timer;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine.Command;
+import picocli.CommandLine.Option;
+import java.util.concurrent.Callable;
+
+import static com.amazonaws.services.s3.internal.SkipMd5CheckStrategy.DISABLE_PUT_OBJECT_MD5_VALIDATION_PROPERTY;
+
+/**
+ * Generate buckets via the s3 interface.
+ */
+@Command(name = "s3bg",
+ aliases = "s3-bucket-generator",
+ description = "Create buckets via the s3 interface.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class S3BucketGenerator extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(S3BucketGenerator.class);
+
+ @Option(names = {"-b", "--bucket"},
+ description =
+ "Prefix name to use for bucket creation.",
+ defaultValue = "bucket")
+ private String bucketName;
+
+ @Option(names = {"-e", "--endpoint"},
Review Comment:
please add documentation for how to specify credentials for S3?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on pull request #3629: HDDS-5779. Create s3g bucket creation performance test
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on PR #3629:
URL: https://github.com/apache/ozone/pull/3629#issuecomment-1221247505
Thanks @Zeddling for the patch, @kerneltime, @neils-dev for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] neils-dev commented on pull request #3629: HDDS-5779. Create s3g bucket creation performance test
Posted by GitBox <gi...@apache.org>.
neils-dev commented on PR #3629:
URL: https://github.com/apache/ozone/pull/3629#issuecomment-1198549362
> please add documentation for how to specify credentials for S3?
Thanks @kerneltime - on this in addition to the update to the PR documenting the secure cluster setup that was used to reproduce the test,
it can _also_ be run in an unsecured ozone environment with the user .aws/credentials file set or shell environment with setting the following environment variables before running:
`$ export AWS_ACCESS_KEY_ID=ANYID`
`$ export AWS_SECRET_ACCESS_KEY=ANYKEY`
` $ bin/ozone freon s3bg`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] neils-dev commented on a diff in pull request #3629: HDDS-5779. Create s3g bucket creation performance test
Posted by GitBox <gi...@apache.org>.
neils-dev commented on code in PR #3629:
URL: https://github.com/apache/ozone/pull/3629#discussion_r934053009
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/S3BucketGenerator.java:
##########
@@ -0,0 +1,73 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+import com.codahale.metrics.Timer;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine.Command;
+
+import java.util.concurrent.Callable;
+
+import static com.amazonaws.services.s3.internal.SkipMd5CheckStrategy.DISABLE_PUT_OBJECT_MD5_VALIDATION_PROPERTY;
+
+/**
+ * Generate buckets via the s3 interface.
+ *
+ * For a secure cluster,
+ * $> init user keytab
+ * $> kinit -kt /etc/security/keytabs/testuser.keytab testuser/scm@EXAMPLE.COM
+ * $> eval $(ozone s3 getsecret -e)
+ * for getting and exporting access_key_id and secret_access_key
+ * to freon shell test environment
+ * secret access key.
+ */
+@Command(name = "s3bg",
+ aliases = "s3-bucket-generator",
+ description = "Create buckets via the s3 interface.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class S3BucketGenerator extends S3EntityGenerator
+ implements Callable<Void> {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(S3BucketGenerator.class);
+
+ private Timer timer;
+
+ @Override
+ public Void call() throws Exception {
+ s3ClientInit();
+
+ timer = getMetrics().timer("bucket-create");
+
+ System.setProperty(DISABLE_PUT_OBJECT_MD5_VALIDATION_PROPERTY, "true");
Review Comment:
The `DISABLE_PUT_OBJECT_MD5_VALIDATION_PROPERTY` shouldn't be needed to be set for the bucket creation test. I believe this is an artifact left from creating this test based upon the `s3kg` test.
You can check if the test runs as expected without setting this property and remove. If the case, then the property can remain set only in the `s3kg` test, https://github.com/apache/ozone/blob/0d5e65656304b0f50e2b47629eba58d3ff7345e5/hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/S3KeyGenerator.java#L101.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a diff in pull request #3629: HDDS-5779. Create s3g bucket creation performance test
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on code in PR #3629:
URL: https://github.com/apache/ozone/pull/3629#discussion_r947742026
##########
hadoop-ozone/dist/src/main/compose/ozone/docker-config:
##########
@@ -48,3 +48,5 @@ OZONE_CONF_DIR=/etc/hadoop
OZONE_LOG_DIR=/var/log/hadoop
no_proxy=om,scm,s3g,recon,kdc,localhost,127.0.0.1
+
+export SECURITY_ENABLED=false
Review Comment:
I'm not sure why that happens, but it should not be read from environment variable, see my [comment](https://github.com/apache/ozone/pull/3629#discussion_r947005888).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on pull request #3629: HDDS-5779. Create s3g Grpc performance test
Posted by GitBox <gi...@apache.org>.
kerneltime commented on PR #3629:
URL: https://github.com/apache/ozone/pull/3629#issuecomment-1197256613
cc @DaveTeng0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on pull request #3629: HDDS-5779. Create s3g bucket creation performance test
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on PR #3629:
URL: https://github.com/apache/ozone/pull/3629#issuecomment-1205742804
Thanks @Zeddling for adding the test. Can you please enable the [build-branch](https://github.com/Zeddling/ozone/actions/workflows/post-commit.yml) workflow in your fork? I think you'll also need to push another commit (can be empty) to your [branch](https://github.com/Zeddling/ozone/commits/HDDS-5779) to trigger the checks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a diff in pull request #3629: HDDS-5779. Create s3g bucket creation performance test
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on code in PR #3629:
URL: https://github.com/apache/ozone/pull/3629#discussion_r949235475
##########
hadoop-ozone/dist/src/main/smoketest/ozone-lib/freon.robot:
##########
@@ -67,3 +67,9 @@ Freon OMBR
[arguments] ${prefix}=ombg ${n}=1 ${threads}=1 ${args}=${EMPTY}
${result} = Execute ozone freon ombr ${OM_HA_PARAM} -t ${threads} -n${n} -p ${prefix} ${args}
Should contain ${result} Successful executions: ${n}
+
+
+Freon S3BG
+ [arguments] ${prefix}=s3bg ${n}=100 ${threads}=5000 ${args}=${EMPTY}
+ ${result} = Execute ozone freon s3bg ${OM_HA_PARAM} -t ${threads} -n ${n} -p ${prefix} ${args}
+ Should contain ${result} Successful executions: ${n}
Review Comment:
@Zeddling Sorry, I just noticed this. I think this almost duplicates the one in `s3/freon.robot` and is unused. Let's keep only the other one.
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a diff in pull request #3629: HDDS-5779. Create s3g Grpc performance test
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on code in PR #3629:
URL: https://github.com/apache/ozone/pull/3629#discussion_r931394093
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/S3BucketGenerator.java:
##########
@@ -0,0 +1,99 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+import com.amazonaws.auth.EnvironmentVariableCredentialsProvider;
+import com.amazonaws.client.builder.AwsClientBuilder.EndpointConfiguration;
+import com.amazonaws.regions.Regions;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.AmazonS3ClientBuilder;
+import com.codahale.metrics.Timer;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine.Command;
+import picocli.CommandLine.Option;
+import java.util.concurrent.Callable;
+
+import static com.amazonaws.services.s3.internal.SkipMd5CheckStrategy.DISABLE_PUT_OBJECT_MD5_VALIDATION_PROPERTY;
+
+/**
+ * Generate buckets via the s3 interface.
+ */
+@Command(name = "s3bg",
+ aliases = "s3-bucket-generator",
+ description = "Create buckets via the s3 interface.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class S3BucketGenerator extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(S3BucketGenerator.class);
+
+ @Option(names = {"-b", "--bucket"},
+ description =
+ "Prefix name to use for bucket creation.",
+ defaultValue = "bucket")
+ private String bucketName;
Review Comment:
Do we really need this parameter? `--prefix` seems to do the job.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] Zeddling commented on pull request #3629: HDDS-5779. Create s3g bucket creation performance test
Posted by GitBox <gi...@apache.org>.
Zeddling commented on PR #3629:
URL: https://github.com/apache/ozone/pull/3629#issuecomment-1212054776
@adoroszlai just added the robot test. The changes passed all CI workflows
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] Zeddling commented on pull request #3629: HDDS-5779. Create s3g bucket creation performance test
Posted by GitBox <gi...@apache.org>.
Zeddling commented on PR #3629:
URL: https://github.com/apache/ozone/pull/3629#issuecomment-1213143196
Added an extra change. The tests weren't picking up the `${SECURITY_ENABLED}` value from the environment and thus each test would have `${SECURITY_ENABLED}=false` (set in `commonlib.robot`) on all environments. The change gets the variable from the environment updates `${SECURITY_ENABLED}` before running the other keywords in Setup v4 headers. You would have to `export SECURITY_ENABLED=<true or false>` in the environment before setting the credentials
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] Zeddling commented on a diff in pull request #3629: HDDS-5779. Create s3g bucket creation performance test
Posted by GitBox <gi...@apache.org>.
Zeddling commented on code in PR #3629:
URL: https://github.com/apache/ozone/pull/3629#discussion_r947736389
##########
hadoop-ozone/dist/src/main/compose/ozone/docker-config:
##########
@@ -48,3 +48,5 @@ OZONE_CONF_DIR=/etc/hadoop
OZONE_LOG_DIR=/var/log/hadoop
no_proxy=om,scm,s3g,recon,kdc,localhost,127.0.0.1
+
+export SECURITY_ENABLED=false
Review Comment:
Thank you for replying.
For some reason, when running the tests without setting `export SECURITY_ENABLED` in the docker-config, the S3 tests would fail with the error `Environment variable 'SECURITY_ENABLED' does not exist.`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] Zeddling commented on a diff in pull request #3629: HDDS-5779. Create s3g bucket creation performance test
Posted by GitBox <gi...@apache.org>.
Zeddling commented on code in PR #3629:
URL: https://github.com/apache/ozone/pull/3629#discussion_r949237850
##########
hadoop-ozone/dist/src/main/smoketest/ozone-lib/freon.robot:
##########
@@ -67,3 +67,9 @@ Freon OMBR
[arguments] ${prefix}=ombg ${n}=1 ${threads}=1 ${args}=${EMPTY}
${result} = Execute ozone freon ombr ${OM_HA_PARAM} -t ${threads} -n${n} -p ${prefix} ${args}
Should contain ${result} Successful executions: ${n}
+
+
+Freon S3BG
+ [arguments] ${prefix}=s3bg ${n}=100 ${threads}=5000 ${args}=${EMPTY}
+ ${result} = Execute ozone freon s3bg ${OM_HA_PARAM} -t ${threads} -n ${n} -p ${prefix} ${args}
+ Should contain ${result} Successful executions: ${n}
Review Comment:
Oh sorry. I missed that. Thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on pull request #3629: HDDS-5779. Create s3g Grpc performance test
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on PR #3629:
URL: https://github.com/apache/ozone/pull/3629#issuecomment-1197324893
> Maybe add the ability to create multiple buckets ?
It will create multiple buckets (`-n` parameter).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] Zeddling commented on a diff in pull request #3629: HDDS-5779. Create s3g bucket creation performance test
Posted by GitBox <gi...@apache.org>.
Zeddling commented on code in PR #3629:
URL: https://github.com/apache/ozone/pull/3629#discussion_r934778212
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/S3BucketGenerator.java:
##########
@@ -0,0 +1,73 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+import com.codahale.metrics.Timer;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine.Command;
+
+import java.util.concurrent.Callable;
+
+import static com.amazonaws.services.s3.internal.SkipMd5CheckStrategy.DISABLE_PUT_OBJECT_MD5_VALIDATION_PROPERTY;
+
+/**
+ * Generate buckets via the s3 interface.
+ *
+ * For a secure cluster,
+ * $> init user keytab
+ * $> kinit -kt /etc/security/keytabs/testuser.keytab testuser/scm@EXAMPLE.COM
+ * $> eval $(ozone s3 getsecret -e)
+ * for getting and exporting access_key_id and secret_access_key
+ * to freon shell test environment
+ * secret access key.
+ */
+@Command(name = "s3bg",
+ aliases = "s3-bucket-generator",
+ description = "Create buckets via the s3 interface.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class S3BucketGenerator extends S3EntityGenerator
+ implements Callable<Void> {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(S3BucketGenerator.class);
+
+ private Timer timer;
+
+ @Override
+ public Void call() throws Exception {
+ s3ClientInit();
+
+ timer = getMetrics().timer("bucket-create");
+
+ System.setProperty(DISABLE_PUT_OBJECT_MD5_VALIDATION_PROPERTY, "true");
Review Comment:
Thank you for that. It runs as expected without it. I've made the update
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on pull request #3629: HDDS-5779. Create s3g bucket creation performance test
Posted by GitBox <gi...@apache.org>.
kerneltime commented on PR #3629:
URL: https://github.com/apache/ozone/pull/3629#issuecomment-1198689720
> > Maybe add the ability to create multiple buckets ?
>
> It will create multiple buckets (`-n` parameter).
Ah! Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on pull request #3629: HDDS-5779. Create s3g bucket creation performance test
Posted by GitBox <gi...@apache.org>.
kerneltime commented on PR #3629:
URL: https://github.com/apache/ozone/pull/3629#issuecomment-1201716469
Please add basic sanity tests under the robot framework
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a diff in pull request #3629: HDDS-5779. Create s3g bucket creation performance test
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on code in PR #3629:
URL: https://github.com/apache/ozone/pull/3629#discussion_r947006838
##########
hadoop-ozone/dist/src/main/compose/ozone/docker-config:
##########
@@ -48,3 +48,5 @@ OZONE_CONF_DIR=/etc/hadoop
OZONE_LOG_DIR=/var/log/hadoop
no_proxy=om,scm,s3g,recon,kdc,localhost,127.0.0.1
+
+export SECURITY_ENABLED=false
Review Comment:
`SECURITY_ENABLED` is already defined in `test.sh`.
```suggestion
```
##########
hadoop-ozone/dist/src/main/compose/ozonesecure/docker-config:
##########
@@ -124,6 +124,7 @@ KMS-SITE.XML_hadoop.kms.proxyuser.s3g.hosts=*
#BYTEMAN_SCRIPT_URL=https://raw.githubusercontent.com/apache/hadoop/trunk/dev-support/byteman/hadooprpc.btm
OZONE_DATANODE_SECURE_USER=root
+export SECURITY_ENABLED=true
Review Comment:
`SECURITY_ENABLED` is already defined in `test.sh`.
```suggestion
```
##########
hadoop-ozone/dist/src/main/compose/ozonesecure-ha/docker-config:
##########
@@ -130,6 +130,7 @@ HDFS-SITE.XML_rpc.metrics.percentiles.intervals=60,300
#BYTEMAN_SCRIPT_URL=https://raw.githubusercontent.com/apache/hadoop/trunk/dev-support/byteman/hadooprpc.btm
OZONE_DATANODE_SECURE_USER=root
+export SECURITY_ENABLED=true
Review Comment:
```suggestion
```
##########
hadoop-ozone/dist/src/main/smoketest/s3/freon.robot:
##########
@@ -18,26 +18,35 @@ Documentation S3 gateway test with aws cli
Library OperatingSystem
Library String
Library BuiltIn
-Resource ../commonlib.robot
-Resource ../ozone-lib/freon.robot
-Test Timeout 5 minutes
+Resource ./commonawslib.robot
+Suite Setup Setup v4 headers
+
*** Variables ***
-${testuser} testuser
+${ENDPOINT_URL} http://s3g:9878
*** Keywords ***
-Set Access Key and Id
- ${result} = Execute ozone s3 getsecret
- ${accessKey} = Get Regexp Matches ${result} (?<=awsAccessKey=).*
- ${secret} = Get Regexp Matches ${result} (?<=awsSecret=).*
- ${accessKey} = Set Variable ${accessKey[0]}
- ${secret} = Set Variable ${secret[0]}
- Set Environment Variable AWS_SECRET_ACCESS_KEY ${secret}
- Set Environment Variable AWS_ACCESS_KEY_ID ${accessKey}
+# Export access key and secret to the environment
+Setup aws credentials
+ ${accessKey} = Execute aws configure get aws_access_key_id
+ ${secret} = Execute aws configure get aws_secret_access_key
+ Set Environment Variable AWS_SECRET_ACCESS_KEY ${secret}
+ Set Environment Variable AWS_ACCESS_KEY_ID ${accessKey}
+
+Access key id
+ ${env_var}= Get Environment Variable AWS_ACCESS_KEY_ID
+
+Freon S3BG
+ [arguments] ${prefix}=s3bg ${n}=100 ${threads}=10 ${args}=${EMPTY}
+ ${result} = Execute ozone freon s3bg -e ${ENDPOINT_URL} -t ${threads} -n ${n} -p ${prefix} ${args}
+ Should contain ${result} Successful executions: ${n}
*** Test Cases ***
-S3g bucket creation performance
- Kinit test user ${testuser} ${testuser}.keytab
- Set Access Key and Id
- Freon S3BG
+Export AWS credentials
+ Setup aws credentials
+
+Check access key id
+ Access key id
Review Comment:
This seems to be unnecessary.
##########
hadoop-ozone/dist/src/main/compose/ozone-ha/docker-config:
##########
@@ -48,3 +48,5 @@ OZONE_CONF_DIR=/etc/hadoop
OZONE_LOG_DIR=/var/log/hadoop
no_proxy=om1,om2,om3,scm,s3g,recon,kdc,localhost,127.0.0.1
+
+export SECURITY_ENABLED=false
Review Comment:
```suggestion
```
##########
hadoop-ozone/dist/src/main/smoketest/s3/freon.robot:
##########
@@ -18,26 +18,35 @@ Documentation S3 gateway test with aws cli
Library OperatingSystem
Library String
Library BuiltIn
-Resource ../commonlib.robot
-Resource ../ozone-lib/freon.robot
-Test Timeout 5 minutes
+Resource ./commonawslib.robot
+Suite Setup Setup v4 headers
Review Comment:
```suggestion
Suite Setup Setup s3 tests
```
##########
hadoop-ozone/dist/src/main/smoketest/s3/freon.robot:
##########
@@ -18,26 +18,35 @@ Documentation S3 gateway test with aws cli
Library OperatingSystem
Library String
Library BuiltIn
-Resource ../commonlib.robot
-Resource ../ozone-lib/freon.robot
-Test Timeout 5 minutes
+Resource ./commonawslib.robot
+Suite Setup Setup v4 headers
+
*** Variables ***
-${testuser} testuser
+${ENDPOINT_URL} http://s3g:9878
*** Keywords ***
-Set Access Key and Id
- ${result} = Execute ozone s3 getsecret
- ${accessKey} = Get Regexp Matches ${result} (?<=awsAccessKey=).*
- ${secret} = Get Regexp Matches ${result} (?<=awsSecret=).*
- ${accessKey} = Set Variable ${accessKey[0]}
- ${secret} = Set Variable ${secret[0]}
- Set Environment Variable AWS_SECRET_ACCESS_KEY ${secret}
- Set Environment Variable AWS_ACCESS_KEY_ID ${accessKey}
+# Export access key and secret to the environment
+Setup aws credentials
+ ${accessKey} = Execute aws configure get aws_access_key_id
+ ${secret} = Execute aws configure get aws_secret_access_key
+ Set Environment Variable AWS_SECRET_ACCESS_KEY ${secret}
+ Set Environment Variable AWS_ACCESS_KEY_ID ${accessKey}
+
+Access key id
+ ${env_var}= Get Environment Variable AWS_ACCESS_KEY_ID
+
+Freon S3BG
+ [arguments] ${prefix}=s3bg ${n}=100 ${threads}=10 ${args}=${EMPTY}
+ ${result} = Execute ozone freon s3bg -e ${ENDPOINT_URL} -t ${threads} -n ${n} -p ${prefix} ${args}
+ Should contain ${result} Successful executions: ${n}
*** Test Cases ***
-S3g bucket creation performance
- Kinit test user ${testuser} ${testuser}.keytab
- Set Access Key and Id
- Freon S3BG
+Export AWS credentials
+ Setup aws credentials
Review Comment:
This should be a `[setup]` step for `Run Freon S3BG`, not a separate test case.
##########
hadoop-ozone/dist/src/main/smoketest/s3/commonawslib.robot:
##########
@@ -58,6 +58,7 @@ Setup v2 headers
Set Environment Variable AWS_SECRET_ACCESS_KEY ANYKEY
Setup v4 headers
+ ${SECURITY_ENABLED} = Get Environment Variable SECURITY_ENABLED
Review Comment:
`SECURITY_ENABLED` is defined as a Robot variable, propagated from the environment variable by:
https://github.com/apache/ozone/blob/ec3d55f6b5101476ef0d889a26b8fb9cf6e4d243/hadoop-ozone/dist/src/main/compose/testlib.sh#L193
It does not need to be looked up from the environment again.
```suggestion
```
##########
hadoop-ozone/dist/src/main/smoketest/s3/freon.robot:
##########
@@ -18,26 +18,35 @@ Documentation S3 gateway test with aws cli
Library OperatingSystem
Library String
Library BuiltIn
-Resource ../commonlib.robot
-Resource ../ozone-lib/freon.robot
-Test Timeout 5 minutes
+Resource ./commonawslib.robot
+Suite Setup Setup v4 headers
+
*** Variables ***
-${testuser} testuser
+${ENDPOINT_URL} http://s3g:9878
*** Keywords ***
-Set Access Key and Id
- ${result} = Execute ozone s3 getsecret
- ${accessKey} = Get Regexp Matches ${result} (?<=awsAccessKey=).*
- ${secret} = Get Regexp Matches ${result} (?<=awsSecret=).*
- ${accessKey} = Set Variable ${accessKey[0]}
- ${secret} = Set Variable ${secret[0]}
- Set Environment Variable AWS_SECRET_ACCESS_KEY ${secret}
- Set Environment Variable AWS_ACCESS_KEY_ID ${accessKey}
+# Export access key and secret to the environment
+Setup aws credentials
+ ${accessKey} = Execute aws configure get aws_access_key_id
+ ${secret} = Execute aws configure get aws_secret_access_key
+ Set Environment Variable AWS_SECRET_ACCESS_KEY ${secret}
+ Set Environment Variable AWS_ACCESS_KEY_ID ${accessKey}
+
+Access key id
+ ${env_var}= Get Environment Variable AWS_ACCESS_KEY_ID
Review Comment:
This seems to be unnecessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on pull request #3629: HDDS-5779. Create s3g bucket creation performance test
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on PR #3629:
URL: https://github.com/apache/ozone/pull/3629#issuecomment-1197340318
> This patch was tested manually using a docker based secure cluster and using the ozone external terminal under `hadoop-ozone/dist/target/ozone-X.X.X/bin/` .
Providing exact commands helps reviewers reproduce the test.
```bash
cd hadoop-ozone/dist/target/ozone-1.3.0-SNAPSHOT/compose/ozone
OZONE_REPLICATION_FACTOR=3 ./run.sh -d
sleep 60 # wait for cluster start
docker-compose exec -T scm bash -c \
"export AWS_ACCESS_KEY_ID=any AWS_SECRET_KEY=any
ozone freon s3bg --endpoint http://s3g:9878/ -n10"
```
Output:
```
...
2022-07-27 20:33:11,634 [main] INFO freon.BaseFreonGenerator: Executing test with prefix l2nphehnly
2022-07-27 20:33:11,678 [Thread-5] INFO freon.ProgressBar: Progress: 0.00 % (0 out of 10)
2022-07-27 20:33:12,662 [Thread-5] INFO freon.ProgressBar: Progress: 100.00 % (10 out of 10)
2022-07-27 20:33:13,334 [shutdown-hook-0] INFO metrics: type=TIMER, name=bucket-create, count=10, min=181.285259, max=214.404077, mean=202.73142240000004, stddev=10.869563898219672, median=209.286641, p75=210.5612, p95=214.404077, p98=214.404077, p99=214.404077, p999=214.404077, mean_rate=9.689749933590571, m1=0.0, m5=0.0, m15=0.0, rate_unit=events/second, duration_unit=milliseconds
2022-07-27 20:33:13,335 [shutdown-hook-0] INFO freon.BaseFreonGenerator: Total execution time (sec): 2
2022-07-27 20:33:13,335 [shutdown-hook-0] INFO freon.BaseFreonGenerator: Failures: 0
2022-07-27 20:33:13,335 [shutdown-hook-0] INFO freon.BaseFreonGenerator: Successful executions: 10
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] Zeddling commented on pull request #3629: HDDS-5779. Create s3g bucket creation performance test
Posted by GitBox <gi...@apache.org>.
Zeddling commented on PR #3629:
URL: https://github.com/apache/ozone/pull/3629#issuecomment-1205727325
> Please add basic sanity tests under the robot framework
I've added the test @kerneltime
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] Zeddling commented on a diff in pull request #3629: HDDS-5779. Create s3g bucket creation performance test
Posted by GitBox <gi...@apache.org>.
Zeddling commented on code in PR #3629:
URL: https://github.com/apache/ozone/pull/3629#discussion_r934844318
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/S3EntityGenerator.java:
##########
@@ -0,0 +1,65 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.hadoop.ozone.freon;
+
+import com.amazonaws.auth.EnvironmentVariableCredentialsProvider;
+import com.amazonaws.client.builder.AwsClientBuilder.EndpointConfiguration;
+import com.amazonaws.regions.Regions;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.AmazonS3ClientBuilder;
+import picocli.CommandLine;
+
+
+/**
+ * Initiliazing common aspects of S3BucketGenerator and S3KeyGenerator.
+ */
+public class S3EntityGenerator extends BaseFreonGenerator {
+ @CommandLine.Option(names = {"-e", "--endpoint"},
+ description = "S3 HTTP endpoint",
+ defaultValue = "http://localhost:9878")
+ private String endpoint;
+
+ private AmazonS3 getS3;
Review Comment:
Thank you for that. I have made some updates. I refactored the `s3` and also removed the system.property setting from `S3BucketGenerator`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] Zeddling commented on a diff in pull request #3629: HDDS-5779. Create s3g bucket creation performance test
Posted by GitBox <gi...@apache.org>.
Zeddling commented on code in PR #3629:
URL: https://github.com/apache/ozone/pull/3629#discussion_r933455345
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/S3BucketGenerator.java:
##########
@@ -0,0 +1,99 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+import com.amazonaws.auth.EnvironmentVariableCredentialsProvider;
+import com.amazonaws.client.builder.AwsClientBuilder.EndpointConfiguration;
+import com.amazonaws.regions.Regions;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.AmazonS3ClientBuilder;
+import com.codahale.metrics.Timer;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine.Command;
+import picocli.CommandLine.Option;
+import java.util.concurrent.Callable;
+
+import static com.amazonaws.services.s3.internal.SkipMd5CheckStrategy.DISABLE_PUT_OBJECT_MD5_VALIDATION_PROPERTY;
+
+/**
+ * Generate buckets via the s3 interface.
+ */
+@Command(name = "s3bg",
+ aliases = "s3-bucket-generator",
+ description = "Create buckets via the s3 interface.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class S3BucketGenerator extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(S3BucketGenerator.class);
+
+ @Option(names = {"-b", "--bucket"},
+ description =
+ "Prefix name to use for bucket creation.",
+ defaultValue = "bucket")
+ private String bucketName;
Review Comment:
Thank you the review. I've added added a class to initialize common aspects of the `S3BucketGenerator` and `S3KeyGenerator`, removed the bucketName variable from `S3BucketGenerator` and added some documentation on getting the s3 secrets
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on pull request #3629: HDDS-5779. Create s3g bucket creation performance test
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on PR #3629:
URL: https://github.com/apache/ozone/pull/3629#issuecomment-1197739599
Thanks @xBis7 for the steps in `ozonesecure` env. Note that passing the `-e` parameter to `ozone s3 getsecret` makes it print credentials in a format suitable for shell:
```bash
$ ozone s3 getsecret -h
Usage: ozone s3 getsecret [-ehV] [--om-service-id=<omServiceID>] [-u=<username>]
Returns S3 secret for a user
-e Print out variables together with 'export' prefix, to use it
from 'eval $(ozone s3 getsecret)'
...
$ ozone s3 getsecret -e
export AWS_ACCESS_KEY_ID='testuser/scm@EXAMPLE.COM'
export AWS_SECRET_ACCESS_KEY='...'
```
This way we can void copy-paste:
```bash
kinit -kt /etc/security/keytabs/testuser.keytab testuser/scm@EXAMPLE.COM
eval $(ozone s3 getsecret -e)
ozone freon s3bg ...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on pull request #3629: HDDS-5779. Create s3g bucket creation performance test
Posted by GitBox <gi...@apache.org>.
xBis7 commented on PR #3629:
URL: https://github.com/apache/ozone/pull/3629#issuecomment-1197893837
Thanks @adoroszlai for the clarification. @Zeddling should add that as well in the description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on pull request #3629: HDDS-5779. Create s3g bucket creation performance test
Posted by GitBox <gi...@apache.org>.
xBis7 commented on PR #3629:
URL: https://github.com/apache/ozone/pull/3629#issuecomment-1197363150
@adoroszlai The exact commands to reproduce the test would be
<br/>
Running docker-compose and getting the aws key
```
cd hadoop-ozone/dist/target/ozone-1.3.0-SNAPSHOT/compose/ozonesecure
docker-compose up -d --scale datanode=3
docker-compose exec scm bash
bash-4.2$ kinit -kt /etc/security/keytabs/testuser.keytab testuser/scm@EXAMPLE.COM
bash-4.2$ ozone s3 getsecret
awsAccessKey=testuser/scm@EXAMPLE.COM
awsSecret=<key>
bash-4.2$ exit
```
<br/>
Copying the awsSecret key, exporting it to freon test environment and then running the Freon bucket generator test
```
docker exec -it ozonesecure_s3g_1 bash
export AWS_ACCESS_KEY=testuser/scm@EXAMPLE.COM AWS_SECRET_KEY=<key>
ozone freon s3bg -t 100 -n 500000
```
<br/>
And for running the test from the external shell
```
cd hadoop-ozone/dist/target/ozone-1.3.0-SNAPSHOT/bin/
./ozone freon s3bg -t 100 -n 500000
```
Maybe @Zeddling can add these steps to the PR description
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a diff in pull request #3629: HDDS-5779. Create s3g bucket creation performance test
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on code in PR #3629:
URL: https://github.com/apache/ozone/pull/3629#discussion_r938600304
##########
hadoop-ozone/dist/src/main/smoketest/s3/bucketcreateperformance.robot:
##########
@@ -0,0 +1,43 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+*** Settings ***
+Documentation S3 gateway test with aws cli
+Library OperatingSystem
+Library String
+Library BuiltIn
+Resource ../commonlib.robot
+Resource ../ozone-lib/freon.robot
+Test Timeout 5 minutes
+
+*** Variables ***
+${testuser} testuser
+
+*** Keywords ***
+Set Access Key and Id
+ ${result} = Execute ozone s3 getsecret
+ ${accessKey} = Get Regexp Matches ${result} (?<=awsAccessKey=).*
+ ${secret} = Get Regexp Matches ${result} (?<=awsSecret=).*
+ ${accessKey} = Set Variable ${accessKey[0]}
+ ${secret} = Set Variable ${secret[0]}
+ Set Environment Variable AWS_SECRET_ACCESS_KEY ${secret}
+ Set Environment Variable AWS_ACCESS_KEY_ID ${accessKey}
+
+*** Test Cases ***
+S3g bucket creation performance
+ Kinit test user ${testuser} ${testuser}.keytab
+ Set Access Key and Id
Review Comment:
Test is executed in both secure and unsecure environments, it needs to work in both. This is handled by:
https://github.com/apache/ozone/blob/51897d4aa104d028e5c504ffa8795a5c20191b5b/hadoop-ozone/dist/src/main/smoketest/s3/commonawslib.robot#L60-L82
All that code shouldn't be duplicated in each test. Common test setup can be performed by adding this in the `Settings` section:
https://github.com/apache/ozone/blob/51897d4aa104d028e5c504ffa8795a5c20191b5b/hadoop-ozone/dist/src/main/smoketest/s3/bucketcreate.robot#L23
I also suggest renaming `bucketcreateperformance.robot` to a more generic `freon.robot`, so we can add other Freon smoketest here later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on pull request #3629: HDDS-5779. Create s3g Grpc performance test
Posted by GitBox <gi...@apache.org>.
xBis7 commented on PR #3629:
URL: https://github.com/apache/ozone/pull/3629#issuecomment-1196592106
@neils-dev
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on pull request #3629: HDDS-5779. Create s3g Grpc performance test
Posted by GitBox <gi...@apache.org>.
kerneltime commented on PR #3629:
URL: https://github.com/apache/ozone/pull/3629#issuecomment-1197262035
Nit: Is the goal only to measure performance of a single bucket? Maybe add the ability to create multiple buckets ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on pull request #3629: HDDS-5779. Create s3g bucket creation performance test
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on PR #3629:
URL: https://github.com/apache/ozone/pull/3629#issuecomment-1197577774
> cc @DaveTeng0
Thank you Ritesh! checking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on pull request #3629: HDDS-5779. Create s3g bucket creation performance test
Posted by GitBox <gi...@apache.org>.
kerneltime commented on PR #3629:
URL: https://github.com/apache/ozone/pull/3629#issuecomment-1198690875
> > Maybe add the ability to create multiple buckets ?
>
> It will create multiple buckets (`-n` parameter).
Thanks! I missed the `runTests` framework
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a diff in pull request #3629: HDDS-5779. Create s3g bucket creation performance test
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on code in PR #3629:
URL: https://github.com/apache/ozone/pull/3629#discussion_r933775807
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/S3BucketGenerator.java:
##########
@@ -0,0 +1,73 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+import com.codahale.metrics.Timer;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine.Command;
+
+import java.util.concurrent.Callable;
+
+import static com.amazonaws.services.s3.internal.SkipMd5CheckStrategy.DISABLE_PUT_OBJECT_MD5_VALIDATION_PROPERTY;
+
+/**
+ * Generate buckets via the s3 interface.
+ *
+ * For a secure cluster,
+ * $> init user keytab
+ * $> kinit -kt /etc/security/keytabs/testuser.keytab testuser/scm@EXAMPLE.COM
+ * $> eval $(ozone s3 getsecret -e)
+ * for getting and exporting access_key_id and secret_access_key
+ * to freon shell test environment
+ * secret access key.
+ */
+@Command(name = "s3bg",
+ aliases = "s3-bucket-generator",
+ description = "Create buckets via the s3 interface.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class S3BucketGenerator extends S3EntityGenerator
+ implements Callable<Void> {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(S3BucketGenerator.class);
+
+ private Timer timer;
+
+ @Override
+ public Void call() throws Exception {
+ s3ClientInit();
+
+ timer = getMetrics().timer("bucket-create");
+
+ System.setProperty(DISABLE_PUT_OBJECT_MD5_VALIDATION_PROPERTY, "true");
Review Comment:
I'm not sure this is needed for bucket generator. Anyway, it could be set in `s3ClientInit` to.
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/S3EntityGenerator.java:
##########
@@ -0,0 +1,65 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.hadoop.ozone.freon;
+
+import com.amazonaws.auth.EnvironmentVariableCredentialsProvider;
+import com.amazonaws.client.builder.AwsClientBuilder.EndpointConfiguration;
+import com.amazonaws.regions.Regions;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.AmazonS3ClientBuilder;
+import picocli.CommandLine;
+
+
+/**
+ * Initiliazing common aspects of S3BucketGenerator and S3KeyGenerator.
+ */
+public class S3EntityGenerator extends BaseFreonGenerator {
+ @CommandLine.Option(names = {"-e", "--endpoint"},
+ description = "S3 HTTP endpoint",
+ defaultValue = "http://localhost:9878")
+ private String endpoint;
+
+ private AmazonS3 getS3;
Review Comment:
Nit: simply `s3` would do, I guess this is an artifact of refactoring.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org