You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2022/09/14 18:41:44 UTC
[GitHub] [ozone] DaveTeng0 opened a new pull request, #3754: HDDS 7199: Implement new mix workload Read/Write Freon command which meets specific test requirements
DaveTeng0 opened a new pull request, #3754:
URL: https://github.com/apache/ozone/pull/3754
## What changes were proposed in this pull request?
On Cisco/Intel cluster, populate OM with a large object count and measure performance when working set is larger than cache available.
- Pure read
- Pure write
- Mixed workload, Read + Write
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-7199
## How was this patch tested?
Robot tests, manual tests in cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on a diff in pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on code in PR #3754:
URL: https://github.com/apache/ozone/pull/3754#discussion_r991688668
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-m", "--read-metadata-only"},
+ description = "If only read key's metadata. " +
+ "Supported values are Y, F.",
Review Comment:
updated!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on a diff in pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
kerneltime commented on code in PR #3754:
URL: https://github.com/apache/ozone/pull/3754#discussion_r981685042
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-m", "--read-metadata-only"},
+ description = "If only read key's metadata. " +
+ "Supported values are Y, F.",
+ defaultValue = "false")
+ private boolean readMetadataOnly;
+ @CommandLine.Option(names = {"-s", "--start-index"},
+ description = "Start index of keys of read/write operation.",
Review Comment:
```suggestion
description = "Start index of keys of read/write operation. This can allow adding keys incrementally. Exampel: Create keys 0-1000000 followed by keys 1000001-2000000.",
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on a diff in pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on code in PR #3754:
URL: https://github.com/apache/ozone/pull/3754#discussion_r992832728
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-m", "--read-metadata-only"},
+ description = "If only read key's metadata. " +
+ "Supported values are Y, F.",
+ defaultValue = "false")
+ private boolean readMetadataOnly;
+ @CommandLine.Option(names = {"-s", "--start-index"},
+ description = "Start index of keys of read/write operation.",
+ defaultValue = "0")
+ private int startIndex;
+
+ @CommandLine.Option(names = {"-r", "--range"},
+ description = "index range of read/write operations.",
+ defaultValue = "0")
+ private int range;
+ @CommandLine.Option(names = {"--size"},
+ description = "Object size (in bytes) " +
+ "to be generated.",
+ defaultValue = "1")
+ private int objectSizeInBytes;
+
+ @CommandLine.Option(names = {"--keySorted"},
+ description = "Generated sorted key or not. The key name " +
+ "will be generated via md5 hash if choose " +
+ "to use unsorted key.",
+ defaultValue = "false")
+ private boolean keySorted;
+
+ @CommandLine.Option(names = {"--percentage-read"},
+ description = "Percentage of read tasks in mix workload.",
+ defaultValue = "100")
+ private int percentageRead;
+
+ @CommandLine.Option(names = {"--clients"},
+ description =
+ "Number of clients, defaults 1.",
+ defaultValue = "1")
+ private int clientsCount = 1;
+
+ @CommandLine.Option(
+ names = "--om-service-id",
+ description = "OM Service ID"
+ )
+ private String omServiceID = null;
+
+ private Timer timer;
+
+ private OzoneBucket[] ozoneBuckets;
+
+ private byte[] keyContent;
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneClientKeyReadWriteOps.class);
+
+ /**
+ * Task type of read task, or write task.
+ */
+ public enum TaskType {
+ READ_TASK,
+ WRITE_TASK
+ }
+ private KeyGeneratorUtil kg;
+
+
+ @Override
+ public Void call() throws Exception {
+ init();
+ OzoneConfiguration ozoneConfiguration = createOzoneConfiguration();
+ OzoneClient[] ozoneClients = new OzoneClient[clientsCount];
+ for (int i = 0; i < clientsCount; i++) {
+ ozoneClients[i] = createOzoneClient(omServiceID, ozoneConfiguration);
+ }
+
+ ensureVolumeAndBucketExist(ozoneClients[0], volumeName, bucketName);
+ ozoneBuckets = new OzoneBucket[clientsCount];
+ for (int i = 0; i < clientsCount; i++) {
+ ozoneBuckets[i] = ozoneClients[i].getObjectStore().getVolume(volumeName)
+ .getBucket(bucketName);
+ }
+
+ timer = getMetrics().timer("key-read-write");
+ if (objectSizeInBytes >= 0) {
+ keyContent = RandomUtils.nextBytes(objectSizeInBytes);
+ }
+ if (kg == null) {
+ kg = new KeyGeneratorUtil();
+ }
+ runTests(this::readWriteKeys);
+
+ for (int i = 0; i < clientsCount; i++) {
+ if (ozoneClients[i] != null) {
+ ozoneClients[i].close();
+ }
+ }
+ return null;
+ }
+
+ public void readWriteKeys(long counter) throws Exception {
+ int clientIndex = (int)((counter) % clientsCount);
+ OzoneBucket ozoneBucket = ozoneBuckets[clientIndex];
+ TaskType taskType = decideReadOrWriteTask();
+ String keyName = getKeyName(clientIndex);
+
+ timer.time(() -> {
+ try {
+ switch (taskType) {
+ case READ_TASK:
+ processReadTasks(keyName, ozoneBucket);
+ break;
+ case WRITE_TASK:
+ processWriteTasks(keyName, ozoneBucket);
+ break;
+ default:
+ break;
+ }
+ } catch (RuntimeException ex) {
+ LOG.error(ex.getMessage());
+ throw ex;
+ } catch (IOException ex) {
+ LOG.error(ex.getMessage());
+ throw new RuntimeException(ex.getMessage());
+ }
+
+ });
+ }
+
+ public void processReadTasks(String keyName, OzoneBucket ozoneBucket)
+ throws RuntimeException, IOException {
+ if (readMetadataOnly) {
+ ozoneBucket.getKey(keyName);
+ } else {
+ byte[] data = new byte[objectSizeInBytes];
+ try (OzoneInputStream inputStream = ozoneBucket.readKey(keyName)) {
+ inputStream.read(data);
+ }
+ }
+ }
+ public void processWriteTasks(String keyName, OzoneBucket ozoneBucket)
+ throws RuntimeException, IOException {
+ try (OzoneOutputStream out =
+ ozoneBucket.createKey(keyName, objectSizeInBytes)) {
+ out.write(keyContent);
+ } catch (Exception ex) {
+ throw ex;
+ }
+ }
+ public TaskType decideReadOrWriteTask() {
+ if (!isMixWorkload()) {
+ if (percentageRead == 100) {
+ return TaskType.READ_TASK;
+ } else {
+ return TaskType.WRITE_TASK;
+ }
+ }
+ //mix workload
+ int tmp = ThreadLocalRandom.current().nextInt(100) + 1; // 1 ~ 100
+ if (tmp < percentageRead) {
+ return TaskType.READ_TASK;
+ } else {
+ return TaskType.WRITE_TASK;
+ }
+ }
+
+ public String getKeyName(int clientIndex) {
+ int start, end;
+ // separate tasks evenly to each client
+ if (range < clientsCount) {
Review Comment:
Then we utilize multiple nodes to separate the total range of keys the user want to test on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on PR #3754:
URL: https://github.com/apache/ozone/pull/3754#issuecomment-1284826327
Sorry, I had to revert this, because checkstyle and findbugs failures affect all other PRs. Please fix the failures and open new PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on a diff in pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on code in PR #3754:
URL: https://github.com/apache/ozone/pull/3754#discussion_r991687396
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
Review Comment:
updated!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on PR #3754:
URL: https://github.com/apache/ozone/pull/3754#issuecomment-1284609289
> > Yes! The command will potentially read some key which doesn't exist in the cluster.
> > Currently the freon command would stop running & report failure, but let me think more about this how to make it better!!
>
> Code can continue testing and add a metric at the end for successful reads vs. not found read. It is a valid test to look at the performance of OM to report a key does not exist. Object Stores can be bombarded with nonexistent keys, and the performance of the underlying storage to report keys that don't exist is important. LSM tree-based storage has to scan all levels before reporting a key as not found and in some ways represents the worst case performance at scale.
Thanks Ritesh for the context!! I'll create a separate jira regarding to this! This definitely make sense!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime merged pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
kerneltime merged PR #3754:
URL: https://github.com/apache/ozone/pull/3754
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on a diff in pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on code in PR #3754:
URL: https://github.com/apache/ozone/pull/3754#discussion_r981857166
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/RangeKeysGenerator.java:
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hadoop.ozone.freon;
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.util.concurrent.Callable;
+import java.util.function.Function;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.PURE_INDEX;
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.MD5;
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone range keys generator for performance test.
+ */
+@CommandLine.Command(name = "ork",
+ description = "write range keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class RangeKeysGenerator extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(RangeKeysGenerator.class);
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-r", "--range-each-client-write"},
+ description = "Write range for each client.",
+ defaultValue = "0")
+ private int range;
+
+ @CommandLine.Option(names = {"-s", "--key-start-index"},
+ description = "Start index of key.",
+ defaultValue = "0")
+ private int startIndex;
+
+
+ @CommandLine.Option(names = {"-k", "--key-encode"},
+ description = "The algorithm to generate key names. " +
+ "Options are pureIndex, md5",
+ defaultValue = "md5")
+ private String encodeFormat;
+
+ @CommandLine.Option(names = {"-g", "--size"},
+ description = "Generated object size (in bytes) " +
+ "to be written.",
+ defaultValue = "1")
+ private int writeSizeInBytes;
+
+ @CommandLine.Option(names = {"--clients"},
+ description =
+ "Number of clients, defaults 1.",
+ defaultValue = "1")
+ private int clientsCount = 1;
+
+ @CommandLine.Option(
+ names = "--debug",
+ description = "Enable debugging message.",
+ defaultValue = "false"
+ )
+ private boolean debug;
+
+
+ @CommandLine.Option(
+ names = "--om-service-id",
+ description = "OM Service ID"
+ )
+ private String omServiceID = null;
+ private KeyGeneratorUtil kg;
+ private OzoneClient[] rpcClients;
+ private byte[] keyContent;
+ private Timer timer;
+
+
+ @Override
+ public Void call() throws Exception {
+ init();
+ OzoneConfiguration ozoneConfiguration = createOzoneConfiguration();
+ rpcClients = new OzoneClient[clientsCount];
+ for (int i = 0; i < clientsCount; i++) {
+ rpcClients[i] = createOzoneClient(omServiceID, ozoneConfiguration);
+ }
+
+ ensureVolumeAndBucketExist(rpcClients[0], volumeName, bucketName);
+ if (writeSizeInBytes >= 0) {
+ keyContent = RandomUtils.nextBytes(writeSizeInBytes);
+ }
+ timer = getMetrics().timer("key-read-write");
+
+ kg = new KeyGeneratorUtil();
+ runTests(this::generateRangeKeys);
+ for (int i = 0; i < clientsCount; i++) {
+ if (rpcClients[i] != null) {
+ rpcClients[i].close();
+ }
+ }
+
+ return null;
+ }
+
+ public void generateRangeKeys(long count) throws Exception {
+ int clientIndex = (int)(count % clientsCount);
+ OzoneClient client = rpcClients[clientIndex];
+ int start = startIndex + (int)count * range;
+ int end = start + range;
+
+ timer.time(() -> {
+ switch (encodeFormat) {
+ case PURE_INDEX:
+ loopRunner(kg.pureIndexKeyNameFunc(), client, start, end);
+ break;
+ case MD5:
+ loopRunner(kg.md5KeyNameFunc(), client, start, end);
+ break;
+ default:
+ loopRunner(kg.md5KeyNameFunc(), client, start, end);
+ break;
+ }
+ return null;
+ });
+ }
+
+
+ public void loopRunner(Function<Integer, String> f, OzoneClient client,
+ int start, int end) throws Exception {
+ OzoneBucket ozbk = client.getObjectStore().getVolume(volumeName)
+ .getBucket(bucketName);
+
+ String keyName;
Review Comment:
ha! you're absolutely right! (I originally wanted to reduce the amount of new String created, but java String is immutable! -_-
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] duongkame commented on a diff in pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
duongkame commented on code in PR #3754:
URL: https://github.com/apache/ozone/pull/3754#discussion_r981516075
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/RangeKeysGenerator.java:
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hadoop.ozone.freon;
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.util.concurrent.Callable;
+import java.util.function.Function;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.PURE_INDEX;
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.MD5;
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone range keys generator for performance test.
+ */
+@CommandLine.Command(name = "ork",
+ description = "write range keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class RangeKeysGenerator extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(RangeKeysGenerator.class);
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-r", "--range-each-client-write"},
+ description = "Write range for each client.",
+ defaultValue = "0")
+ private int range;
+
+ @CommandLine.Option(names = {"-s", "--key-start-index"},
+ description = "Start index of key.",
+ defaultValue = "0")
+ private int startIndex;
+
+
+ @CommandLine.Option(names = {"-k", "--key-encode"},
+ description = "The algorithm to generate key names. " +
+ "Options are pureIndex, md5",
+ defaultValue = "md5")
+ private String encodeFormat;
+
+ @CommandLine.Option(names = {"-g", "--size"},
+ description = "Generated object size (in bytes) " +
+ "to be written.",
+ defaultValue = "1")
+ private int writeSizeInBytes;
+
+ @CommandLine.Option(names = {"--clients"},
+ description =
+ "Number of clients, defaults 1.",
+ defaultValue = "1")
+ private int clientsCount = 1;
+
+ @CommandLine.Option(
+ names = "--debug",
+ description = "Enable debugging message.",
+ defaultValue = "false"
+ )
+ private boolean debug;
+
+
+ @CommandLine.Option(
+ names = "--om-service-id",
+ description = "OM Service ID"
+ )
+ private String omServiceID = null;
+ private KeyGeneratorUtil kg;
+ private OzoneClient[] rpcClients;
+ private byte[] keyContent;
+ private Timer timer;
+
+
+ @Override
+ public Void call() throws Exception {
+ init();
+ OzoneConfiguration ozoneConfiguration = createOzoneConfiguration();
+ rpcClients = new OzoneClient[clientsCount];
+ for (int i = 0; i < clientsCount; i++) {
+ rpcClients[i] = createOzoneClient(omServiceID, ozoneConfiguration);
+ }
+
+ ensureVolumeAndBucketExist(rpcClients[0], volumeName, bucketName);
+ if (writeSizeInBytes >= 0) {
+ keyContent = RandomUtils.nextBytes(writeSizeInBytes);
+ }
+ timer = getMetrics().timer("key-read-write");
+
+ kg = new KeyGeneratorUtil();
+ runTests(this::generateRangeKeys);
+ for (int i = 0; i < clientsCount; i++) {
+ if (rpcClients[i] != null) {
+ rpcClients[i].close();
+ }
+ }
+
+ return null;
+ }
+
+ public void generateRangeKeys(long count) throws Exception {
+ int clientIndex = (int)(count % clientsCount);
+ OzoneClient client = rpcClients[clientIndex];
+ int start = startIndex + (int)count * range;
+ int end = start + range;
+
+ timer.time(() -> {
+ switch (encodeFormat) {
+ case PURE_INDEX:
+ loopRunner(kg.pureIndexKeyNameFunc(), client, start, end);
+ break;
+ case MD5:
+ loopRunner(kg.md5KeyNameFunc(), client, start, end);
+ break;
+ default:
+ loopRunner(kg.md5KeyNameFunc(), client, start, end);
+ break;
+ }
+ return null;
+ });
+ }
+
+
+ public void loopRunner(Function<Integer, String> f, OzoneClient client,
+ int start, int end) throws Exception {
+ OzoneBucket ozbk = client.getObjectStore().getVolume(volumeName)
+ .getBucket(bucketName);
+
+ String keyName;
Review Comment:
nit: `keyName` can be declared inside the loop.
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,247 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.*;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-m", "--read-metadata-only"},
+ description = "If only read key's metadata. " +
+ "Supported values are Y, F.",
+ defaultValue = "false")
+ private boolean readMetadataOnly;
+ @CommandLine.Option(names = {"-s", "--start-index"},
+ description = "Start index of keys of read/write operation.",
+ defaultValue = "0")
+ private int startIndex;
+
+ @CommandLine.Option(names = {"-r", "--range"},
+ description = "index range of read/write operations.",
+ defaultValue = "0")
+ private int range;
+ @CommandLine.Option(names = {"--size"},
+ description = "Object size (in bytes) " +
+ "to be generated.",
+ defaultValue = "1")
+ private int objectSizeInBytes;
+
+ @CommandLine.Option(names = {"--keySorted"},
+ description = "Generated sorted key or not. The key name " +
+ "will be generated via md5 hash if choose " +
+ "to use unsorted key.",
+ defaultValue = "false")
+ private boolean keySorted;
+
+ @CommandLine.Option(names = {"--percentage-read"},
+ description = "Percentage of read tasks in mix workload.",
+ defaultValue = "100")
+ private int percentageRead;
+
+ @CommandLine.Option(names = {"--clients"},
+ description =
+ "Number of clients, defaults 1.",
+ defaultValue = "1")
+ private int clientsCount = 1;
+
+ @CommandLine.Option(
+ names = "--om-service-id",
+ description = "OM Service ID"
+ )
+ private String omServiceID = null;
+
+ private Timer timer;
+
+ private OzoneBucket[] ozoneBuckets;
+
+ private byte[] keyContent;
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneClientKeyReadWriteOps.class);
+ public enum TaskType {
+ READ_TASK,
+ WRITE_TASK
+ }
+ private KeyGeneratorUtil kg;
+
+
+ @Override
+ public Void call() throws Exception {
+ init();
+ OzoneConfiguration ozoneConfiguration = createOzoneConfiguration();
+ OzoneClient[] ozoneClients = new OzoneClient[clientsCount];
+ for (int i = 0; i < clientsCount; i++) {
+ ozoneClients[i] = createOzoneClient(omServiceID, ozoneConfiguration);
+ }
+
+ ensureVolumeAndBucketExist(ozoneClients[0], volumeName, bucketName);
+ ozoneBuckets = new OzoneBucket[clientsCount];
+ for (int i = 0; i < clientsCount; i++) {
+ ozoneBuckets[i] = ozoneClients[i].getObjectStore().getVolume(volumeName)
+ .getBucket(bucketName);
+ }
+
+ timer = getMetrics().timer("key-read-write");
+ if (objectSizeInBytes >= 0) {
+ keyContent = RandomUtils.nextBytes(objectSizeInBytes);
+ }
+ if (kg == null) {
+ kg = new KeyGeneratorUtil();
+ }
+ runTests(this::readWriteKeys);
+
+ for (int i = 0; i < clientsCount; i++) {
+ if (ozoneClients[i] != null) {
+ ozoneClients[i].close();
+ }
+ }
+ return null;
+ }
+
+ public void readWriteKeys(long counter) throws Exception {
+ int clientIndex = (int)((counter) % clientsCount);
+ OzoneBucket ozoneBucket = ozoneBuckets[clientIndex];
+ TaskType taskType = decideReadOrWriteTask();
+ String keyName = getKeyName(taskType, clientIndex);
+
+ timer.time(() -> {
+ try {
+ switch (taskType) {
+ case READ_TASK:
+ processReadTasks(keyName, ozoneBucket);
+ break;
+ case WRITE_TASK:
+ processWriteTasks(keyName, ozoneBucket);
+ break;
+ default:
+ break;
+ }
+ } catch (RuntimeException ex) {
+ LOG.error(ex.getMessage());
+ throw ex;
+ } catch (IOException ex) {
+ LOG.error(ex.getMessage());
+ throw new RuntimeException(ex.getMessage());
+ }
+
+ });
+ }
+
+ public void processReadTasks(String keyName, OzoneBucket ozoneBucket)
+ throws RuntimeException, IOException {
+ if (readMetadataOnly) {
+ ozoneBucket.getKey(keyName);
+ } else {
+ byte[] data = new byte[objectSizeInBytes];
+ try (OzoneInputStream inputStream = ozoneBucket.readKey(keyName)) {
+ inputStream.read(data);
Review Comment:
You can add a `@SuppressWarnings({"unused"})` to tell findbugs make an exception here.
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-m", "--read-metadata-only"},
+ description = "If only read key's metadata. " +
+ "Supported values are Y, F.",
+ defaultValue = "false")
+ private boolean readMetadataOnly;
+ @CommandLine.Option(names = {"-s", "--start-index"},
+ description = "Start index of keys of read/write operation.",
+ defaultValue = "0")
+ private int startIndex;
+
+ @CommandLine.Option(names = {"-r", "--range"},
+ description = "index range of read/write operations.",
+ defaultValue = "0")
+ private int range;
+ @CommandLine.Option(names = {"--size"},
+ description = "Object size (in bytes) " +
+ "to be generated.",
+ defaultValue = "1")
+ private int objectSizeInBytes;
+
+ @CommandLine.Option(names = {"--keySorted"},
+ description = "Generated sorted key or not. The key name " +
+ "will be generated via md5 hash if choose " +
+ "to use unsorted key.",
+ defaultValue = "false")
+ private boolean keySorted;
+
+ @CommandLine.Option(names = {"--percentage-read"},
+ description = "Percentage of read tasks in mix workload.",
+ defaultValue = "100")
+ private int percentageRead;
+
+ @CommandLine.Option(names = {"--clients"},
+ description =
+ "Number of clients, defaults 1.",
+ defaultValue = "1")
+ private int clientsCount = 1;
+
+ @CommandLine.Option(
+ names = "--om-service-id",
+ description = "OM Service ID"
+ )
+ private String omServiceID = null;
+
+ private Timer timer;
+
+ private OzoneBucket[] ozoneBuckets;
+
+ private byte[] keyContent;
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneClientKeyReadWriteOps.class);
+
+ /**
+ * Task type of read task, or write task.
+ */
+ public enum TaskType {
+ READ_TASK,
+ WRITE_TASK
+ }
+ private KeyGeneratorUtil kg;
+
+
+ @Override
+ public Void call() throws Exception {
+ init();
+ OzoneConfiguration ozoneConfiguration = createOzoneConfiguration();
+ OzoneClient[] ozoneClients = new OzoneClient[clientsCount];
+ for (int i = 0; i < clientsCount; i++) {
+ ozoneClients[i] = createOzoneClient(omServiceID, ozoneConfiguration);
+ }
+
+ ensureVolumeAndBucketExist(ozoneClients[0], volumeName, bucketName);
+ ozoneBuckets = new OzoneBucket[clientsCount];
+ for (int i = 0; i < clientsCount; i++) {
+ ozoneBuckets[i] = ozoneClients[i].getObjectStore().getVolume(volumeName)
+ .getBucket(bucketName);
+ }
+
+ timer = getMetrics().timer("key-read-write");
+ if (objectSizeInBytes >= 0) {
+ keyContent = RandomUtils.nextBytes(objectSizeInBytes);
+ }
+ if (kg == null) {
+ kg = new KeyGeneratorUtil();
+ }
+ runTests(this::readWriteKeys);
+
+ for (int i = 0; i < clientsCount; i++) {
+ if (ozoneClients[i] != null) {
+ ozoneClients[i].close();
+ }
+ }
+ return null;
+ }
+
+ public void readWriteKeys(long counter) throws Exception {
+ int clientIndex = (int)((counter) % clientsCount);
+ OzoneBucket ozoneBucket = ozoneBuckets[clientIndex];
+ TaskType taskType = decideReadOrWriteTask();
+ String keyName = getKeyName(clientIndex);
+
+ timer.time(() -> {
+ try {
+ switch (taskType) {
+ case READ_TASK:
+ processReadTasks(keyName, ozoneBucket);
+ break;
+ case WRITE_TASK:
+ processWriteTasks(keyName, ozoneBucket);
+ break;
+ default:
+ break;
+ }
+ } catch (RuntimeException ex) {
+ LOG.error(ex.getMessage());
+ throw ex;
+ } catch (IOException ex) {
+ LOG.error(ex.getMessage());
+ throw new RuntimeException(ex.getMessage());
+ }
+
+ });
+ }
+
+ public void processReadTasks(String keyName, OzoneBucket ozoneBucket)
+ throws RuntimeException, IOException {
+ if (readMetadataOnly) {
+ ozoneBucket.getKey(keyName);
+ } else {
+ byte[] data = new byte[objectSizeInBytes];
+ try (OzoneInputStream inputStream = ozoneBucket.readKey(keyName)) {
+ inputStream.read(data);
+ }
+ }
+ }
+ public void processWriteTasks(String keyName, OzoneBucket ozoneBucket)
+ throws RuntimeException, IOException {
+ try (OzoneOutputStream out =
+ ozoneBucket.createKey(keyName, objectSizeInBytes)) {
+ out.write(keyContent);
+ } catch (Exception ex) {
+ throw ex;
+ }
+ }
+ public TaskType decideReadOrWriteTask() {
+ if (!isMixWorkload()) {
+ if (percentageRead == 100) {
+ return TaskType.READ_TASK;
+ } else {
+ return TaskType.WRITE_TASK;
+ }
+ }
Review Comment:
Maybe it's simpler this way:
```suggestion
if (percentageRead == 100)) {
return TaskType.READ_TASK;
} else if (percentageRead == 0) {
return TaskType.WRITE_TASK;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on a diff in pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on code in PR #3754:
URL: https://github.com/apache/ozone/pull/3754#discussion_r981855973
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-m", "--read-metadata-only"},
+ description = "If only read key's metadata. " +
+ "Supported values are Y, F.",
+ defaultValue = "false")
+ private boolean readMetadataOnly;
+ @CommandLine.Option(names = {"-s", "--start-index"},
+ description = "Start index of keys of read/write operation.",
+ defaultValue = "0")
+ private int startIndex;
+
+ @CommandLine.Option(names = {"-r", "--range"},
+ description = "index range of read/write operations.",
+ defaultValue = "0")
+ private int range;
+ @CommandLine.Option(names = {"--size"},
+ description = "Object size (in bytes) " +
+ "to be generated.",
+ defaultValue = "1")
+ private int objectSizeInBytes;
+
+ @CommandLine.Option(names = {"--keySorted"},
+ description = "Generated sorted key or not. The key name " +
+ "will be generated via md5 hash if choose " +
+ "to use unsorted key.",
+ defaultValue = "false")
+ private boolean keySorted;
+
+ @CommandLine.Option(names = {"--percentage-read"},
+ description = "Percentage of read tasks in mix workload.",
+ defaultValue = "100")
+ private int percentageRead;
+
+ @CommandLine.Option(names = {"--clients"},
+ description =
+ "Number of clients, defaults 1.",
+ defaultValue = "1")
+ private int clientsCount = 1;
+
+ @CommandLine.Option(
+ names = "--om-service-id",
+ description = "OM Service ID"
+ )
+ private String omServiceID = null;
+
+ private Timer timer;
+
+ private OzoneBucket[] ozoneBuckets;
+
+ private byte[] keyContent;
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneClientKeyReadWriteOps.class);
+
+ /**
+ * Task type of read task, or write task.
+ */
+ public enum TaskType {
+ READ_TASK,
+ WRITE_TASK
+ }
+ private KeyGeneratorUtil kg;
+
+
+ @Override
+ public Void call() throws Exception {
+ init();
+ OzoneConfiguration ozoneConfiguration = createOzoneConfiguration();
+ OzoneClient[] ozoneClients = new OzoneClient[clientsCount];
+ for (int i = 0; i < clientsCount; i++) {
+ ozoneClients[i] = createOzoneClient(omServiceID, ozoneConfiguration);
+ }
+
+ ensureVolumeAndBucketExist(ozoneClients[0], volumeName, bucketName);
+ ozoneBuckets = new OzoneBucket[clientsCount];
+ for (int i = 0; i < clientsCount; i++) {
+ ozoneBuckets[i] = ozoneClients[i].getObjectStore().getVolume(volumeName)
+ .getBucket(bucketName);
+ }
+
+ timer = getMetrics().timer("key-read-write");
+ if (objectSizeInBytes >= 0) {
+ keyContent = RandomUtils.nextBytes(objectSizeInBytes);
+ }
+ if (kg == null) {
+ kg = new KeyGeneratorUtil();
+ }
+ runTests(this::readWriteKeys);
+
+ for (int i = 0; i < clientsCount; i++) {
+ if (ozoneClients[i] != null) {
+ ozoneClients[i].close();
+ }
+ }
+ return null;
+ }
+
+ public void readWriteKeys(long counter) throws Exception {
+ int clientIndex = (int)((counter) % clientsCount);
+ OzoneBucket ozoneBucket = ozoneBuckets[clientIndex];
+ TaskType taskType = decideReadOrWriteTask();
+ String keyName = getKeyName(clientIndex);
+
+ timer.time(() -> {
+ try {
+ switch (taskType) {
+ case READ_TASK:
+ processReadTasks(keyName, ozoneBucket);
+ break;
+ case WRITE_TASK:
+ processWriteTasks(keyName, ozoneBucket);
+ break;
+ default:
+ break;
+ }
+ } catch (RuntimeException ex) {
+ LOG.error(ex.getMessage());
+ throw ex;
+ } catch (IOException ex) {
+ LOG.error(ex.getMessage());
+ throw new RuntimeException(ex.getMessage());
+ }
+
+ });
+ }
+
+ public void processReadTasks(String keyName, OzoneBucket ozoneBucket)
+ throws RuntimeException, IOException {
+ if (readMetadataOnly) {
+ ozoneBucket.getKey(keyName);
+ } else {
+ byte[] data = new byte[objectSizeInBytes];
+ try (OzoneInputStream inputStream = ozoneBucket.readKey(keyName)) {
+ inputStream.read(data);
+ }
+ }
+ }
+ public void processWriteTasks(String keyName, OzoneBucket ozoneBucket)
+ throws RuntimeException, IOException {
+ try (OzoneOutputStream out =
+ ozoneBucket.createKey(keyName, objectSizeInBytes)) {
+ out.write(keyContent);
+ } catch (Exception ex) {
+ throw ex;
+ }
+ }
+ public TaskType decideReadOrWriteTask() {
+ if (!isMixWorkload()) {
+ if (percentageRead == 100) {
+ return TaskType.READ_TASK;
+ } else {
+ return TaskType.WRITE_TASK;
+ }
+ }
Review Comment:
oh!! this definitely make sense!! cool
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on a diff in pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on code in PR #3754:
URL: https://github.com/apache/ozone/pull/3754#discussion_r992831636
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-m", "--read-metadata-only"},
+ description = "If only read key's metadata. " +
+ "Supported values are Y, F.",
+ defaultValue = "false")
+ private boolean readMetadataOnly;
+ @CommandLine.Option(names = {"-s", "--start-index"},
+ description = "Start index of keys of read/write operation.",
+ defaultValue = "0")
+ private int startIndex;
+
+ @CommandLine.Option(names = {"-r", "--range"},
+ description = "index range of read/write operations.",
+ defaultValue = "0")
+ private int range;
+ @CommandLine.Option(names = {"--size"},
+ description = "Object size (in bytes) " +
+ "to be generated.",
+ defaultValue = "1")
+ private int objectSizeInBytes;
+
+ @CommandLine.Option(names = {"--keySorted"},
+ description = "Generated sorted key or not. The key name " +
+ "will be generated via md5 hash if choose " +
+ "to use unsorted key.",
+ defaultValue = "false")
+ private boolean keySorted;
+
+ @CommandLine.Option(names = {"--percentage-read"},
+ description = "Percentage of read tasks in mix workload.",
+ defaultValue = "100")
+ private int percentageRead;
+
+ @CommandLine.Option(names = {"--clients"},
+ description =
+ "Number of clients, defaults 1.",
+ defaultValue = "1")
+ private int clientsCount = 1;
+
+ @CommandLine.Option(
+ names = "--om-service-id",
+ description = "OM Service ID"
+ )
+ private String omServiceID = null;
+
+ private Timer timer;
+
+ private OzoneBucket[] ozoneBuckets;
+
+ private byte[] keyContent;
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneClientKeyReadWriteOps.class);
+
+ /**
+ * Task type of read task, or write task.
+ */
+ public enum TaskType {
+ READ_TASK,
+ WRITE_TASK
+ }
+ private KeyGeneratorUtil kg;
+
+
+ @Override
+ public Void call() throws Exception {
+ init();
+ OzoneConfiguration ozoneConfiguration = createOzoneConfiguration();
+ OzoneClient[] ozoneClients = new OzoneClient[clientsCount];
+ for (int i = 0; i < clientsCount; i++) {
+ ozoneClients[i] = createOzoneClient(omServiceID, ozoneConfiguration);
+ }
+
+ ensureVolumeAndBucketExist(ozoneClients[0], volumeName, bucketName);
+ ozoneBuckets = new OzoneBucket[clientsCount];
+ for (int i = 0; i < clientsCount; i++) {
+ ozoneBuckets[i] = ozoneClients[i].getObjectStore().getVolume(volumeName)
+ .getBucket(bucketName);
+ }
+
+ timer = getMetrics().timer("key-read-write");
+ if (objectSizeInBytes >= 0) {
+ keyContent = RandomUtils.nextBytes(objectSizeInBytes);
+ }
+ if (kg == null) {
+ kg = new KeyGeneratorUtil();
+ }
+ runTests(this::readWriteKeys);
+
+ for (int i = 0; i < clientsCount; i++) {
+ if (ozoneClients[i] != null) {
+ ozoneClients[i].close();
+ }
+ }
+ return null;
+ }
+
+ public void readWriteKeys(long counter) throws Exception {
+ int clientIndex = (int)((counter) % clientsCount);
+ OzoneBucket ozoneBucket = ozoneBuckets[clientIndex];
+ TaskType taskType = decideReadOrWriteTask();
+ String keyName = getKeyName(clientIndex);
+
+ timer.time(() -> {
+ try {
+ switch (taskType) {
+ case READ_TASK:
+ processReadTasks(keyName, ozoneBucket);
+ break;
+ case WRITE_TASK:
+ processWriteTasks(keyName, ozoneBucket);
+ break;
+ default:
+ break;
+ }
+ } catch (RuntimeException ex) {
+ LOG.error(ex.getMessage());
+ throw ex;
+ } catch (IOException ex) {
+ LOG.error(ex.getMessage());
+ throw new RuntimeException(ex.getMessage());
+ }
+
+ });
+ }
+
+ public void processReadTasks(String keyName, OzoneBucket ozoneBucket)
+ throws RuntimeException, IOException {
+ if (readMetadataOnly) {
+ ozoneBucket.getKey(keyName);
+ } else {
+ byte[] data = new byte[objectSizeInBytes];
+ try (OzoneInputStream inputStream = ozoneBucket.readKey(keyName)) {
+ inputStream.read(data);
+ }
+ }
+ }
+ public void processWriteTasks(String keyName, OzoneBucket ozoneBucket)
+ throws RuntimeException, IOException {
+ try (OzoneOutputStream out =
+ ozoneBucket.createKey(keyName, objectSizeInBytes)) {
+ out.write(keyContent);
+ } catch (Exception ex) {
+ throw ex;
+ }
+ }
+ public TaskType decideReadOrWriteTask() {
+ if (!isMixWorkload()) {
+ if (percentageRead == 100) {
+ return TaskType.READ_TASK;
+ } else {
+ return TaskType.WRITE_TASK;
+ }
+ }
+ //mix workload
+ int tmp = ThreadLocalRandom.current().nextInt(100) + 1; // 1 ~ 100
+ if (tmp < percentageRead) {
+ return TaskType.READ_TASK;
+ } else {
+ return TaskType.WRITE_TASK;
+ }
+ }
+
+ public String getKeyName(int clientIndex) {
+ int start, end;
+ // separate tasks evenly to each client
+ if (range < clientsCount) {
Review Comment:
ok I update the logic a little bit!
Each thread has its own client which holds tcp connection to OM.
We set the range here which would become the range of each thread/client could read/write on.
For example, If we set range = 10, start_index (index of key) = 20, threadNo. = 30, and we run the Freon test 5 times.
Then only 5 clients would process the read/write operation from key index 20 to 29 (range=10).
If we set range = 10, start_index (index of key) = 20, threadNo. = 30 (clients = 30), and we run the Freon test 40 times.
Then first 30 tests would be assign to 30 client to read/write from key index 20 to 29 (range=10).
The rest of 10 tests would be picked up by any 10 clients whoever finish their tasks first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] duongkame commented on pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
duongkame commented on PR #3754:
URL: https://github.com/apache/ozone/pull/3754#issuecomment-1284634984
> Code can continue testing and add a metric at the end for successful reads vs. not found read. It is a valid test to look at the performance of OM to report a key does not exist. Object Stores can be bombarded with nonexistent keys, and the performance of the underlying storage to report keys that don't exist is important. LSM tree-based storage has to scan all levels before reporting a key as not found and in some ways represents the worst case performance at scale.
agree, reading nonexistent keys is a valid test case and the tool should support it deterministically. To do that, it has to know (on its own) which key exists and which doesn't. Warp does that by having a pre-test phase in which warp creates a set of keys (10K or so) and keeps the created keys in memory for the real read test.
We can also do the same for this tool, by maintaining a set of known keys that can be initialized by a pretest phase and grows with the write test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on a diff in pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
kerneltime commented on code in PR #3754:
URL: https://github.com/apache/ozone/pull/3754#discussion_r981683834
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
Review Comment:
```suggestion
defaultValue = "ockrwbucket")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on a diff in pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on code in PR #3754:
URL: https://github.com/apache/ozone/pull/3754#discussion_r981858212
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-m", "--read-metadata-only"},
+ description = "If only read key's metadata. " +
+ "Supported values are Y, F.",
Review Comment:
Sorry! That value is boolean type. So I should remove that part of sentence "Supported ... T, F." from description!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on a diff in pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on code in PR #3754:
URL: https://github.com/apache/ozone/pull/3754#discussion_r975897036
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-m", "--read-metadata-only"},
+ description = "If only read key's metadata. " +
+ "Supported values are Y, F.",
+ defaultValue = "false")
+ private boolean readMetadataOnly;
+
+ @CommandLine.Option(names = {"-r", "--range-client-read"},
+ description = "range of read operation of each client.",
+ defaultValue = "0")
+ private int readRange;
+
+
+ @CommandLine.Option(names = {"-w", "--range-client-write"},
+ description = "range of write operation of each client.",
+ defaultValue = "0")
+ private int writeRange;
+
+// @CommandLine.Option(names = {"-e", "--end-index-each-client-read"},
+// description = "end-index of each client's read operation.",
+// defaultValue = "0")
+// private int endIndexForRead;
+
+// @CommandLine.Option(names = {"-j", "--end-index-each-client-write"},
+// description = "end-index of each client's write operation.",
+// defaultValue = "0")
+// private int endIndexForWrite;
+
+ @CommandLine.Option(names = {"-g", "--size"},
+ description = "Generated data size (in bytes) of " +
+ "each key/file to be " +
+ "written.",
+ defaultValue = "256")
+ private int writeSizeInBytes;
+
+ @CommandLine.Option(names = {"-k", "--keySorted"},
+ description = "Generated sorted key or not. The key name " +
+ "will be generated via md5 hash if choose " +
+ "to use unsorted key.",
+ defaultValue = "false")
+ private boolean keySorted;
+
+ @CommandLine.Option(names = {"-x", "--mix-workload"},
+ description = "Set to True if you would like to " +
+ "generate mix workload (Read and Write).",
+ defaultValue = "false")
+ private boolean isMixWorkload;
+
+ @CommandLine.Option(names = {"--percentage-read"},
+ description = "Percentage of read tasks in mix workload.",
+ defaultValue = "0")
+ private int percentageRead;
+
+ @CommandLine.Option(names = {"--clients"},
+ description =
+ "Number of clients, defaults 1.",
+ defaultValue = "1")
+ private int clientsCount = 1;
+
+ @CommandLine.Option(
+ names = "--om-service-id",
+ description = "OM Service ID"
+ )
+ private String omServiceID = null;
+
+ private Timer timer;
+
+ private OzoneClient[] rpcClients;
+
+ private byte[] keyContent;
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneClientKeyReadWriteOps.class);
+
+ private final String readTask = "READ_TASK";
+ private final String writeTask = "WRITE_TASK";
+ private KeyGeneratorUtil kg;
+
+ @Override
+ public Void call() throws Exception {
+ init();
+ OzoneConfiguration ozoneConfiguration = createOzoneConfiguration();
+ rpcClients = new OzoneClient[clientsCount];
+ for (int i = 0; i < clientsCount; i++) {
+ rpcClients[i] = createOzoneClient(omServiceID, ozoneConfiguration);
+ }
+
+ ensureVolumeAndBucketExist(rpcClients[0], volumeName, bucketName);
+ timer = getMetrics().timer("key-read-write");
+ if (writeSizeInBytes >= 0) {
+ keyContent = RandomUtils.nextBytes(writeSizeInBytes);
+ }
+ if (kg == null) {
+ kg = new KeyGeneratorUtil();
+ }
+ runTests(this::readWriteKeys);
+
+ for (int i = 0; i < clientsCount; i++) {
+ if (rpcClients[i] != null) {
+ rpcClients[i].close();
+ }
+ }
+ return null;
+ }
+
+ public void readWriteKeys(long counter) throws Exception {
+ int clientIndex = (int)(counter % clientsCount);
+ OzoneClient client = rpcClients[clientIndex];
+ String operationType = decideReadOrWriteTask();
+ String keyName = getKeyName(operationType, clientIndex);
+ timer.time(() -> {
+ try {
+ switch (operationType) {
+ case readTask:
+ processReadTasks(keyName, client);
Review Comment:
oh!! checking!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on a diff in pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on code in PR #3754:
URL: https://github.com/apache/ozone/pull/3754#discussion_r979132298
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/SecurityConfig.java:
##########
@@ -141,12 +141,15 @@ public SecurityConfig(ConfigurationSource configuration) {
this.certificateFileName = this.configuration.get(HDDS_X509_FILE_NAME,
HDDS_X509_FILE_NAME_DEFAULT);
- this.blockTokenEnabled = this.configuration.getBoolean(
- HDDS_BLOCK_TOKEN_ENABLED,
- HDDS_BLOCK_TOKEN_ENABLED_DEFAULT);
- this.containerTokenEnabled = this.configuration.getBoolean(
- HDDS_CONTAINER_TOKEN_ENABLED,
- HDDS_CONTAINER_TOKEN_ENABLED_DEFAULT);
+// this.blockTokenEnabled = this.configuration.getBoolean(
+// HDDS_BLOCK_TOKEN_ENABLED,
+// HDDS_BLOCK_TOKEN_ENABLED_DEFAULT);
+ this.blockTokenEnabled = false;
Review Comment:
yes!! let me change it back!
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java:
##########
@@ -493,8 +493,9 @@ private OzoneManager(OzoneConfiguration conf, StartupOption startupOption)
this.preallocateBlocksMax = conf.getInt(
OZONE_KEY_PREALLOCATION_BLOCKS_MAX,
OZONE_KEY_PREALLOCATION_BLOCKS_MAX_DEFAULT);
- this.grpcBlockTokenEnabled = conf.getBoolean(HDDS_BLOCK_TOKEN_ENABLED,
- HDDS_BLOCK_TOKEN_ENABLED_DEFAULT);
+// this.grpcBlockTokenEnabled = conf.getBoolean(HDDS_BLOCK_TOKEN_ENABLED,
+// HDDS_BLOCK_TOKEN_ENABLED_DEFAULT);
+ this.grpcBlockTokenEnabled = false;
Review Comment:
yes!! let me change it back!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on pull request #3754: HDDS 7199: Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on PR #3754:
URL: https://github.com/apache/ozone/pull/3754#issuecomment-1247164525
cc. @kerneltime @jojochuang @umamaheswararao
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on a diff in pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
kerneltime commented on code in PR #3754:
URL: https://github.com/apache/ozone/pull/3754#discussion_r981721268
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-m", "--read-metadata-only"},
+ description = "If only read key's metadata. " +
+ "Supported values are Y, F.",
+ defaultValue = "false")
+ private boolean readMetadataOnly;
+ @CommandLine.Option(names = {"-s", "--start-index"},
+ description = "Start index of keys of read/write operation.",
+ defaultValue = "0")
+ private int startIndex;
+
+ @CommandLine.Option(names = {"-r", "--range"},
+ description = "index range of read/write operations.",
+ defaultValue = "0")
+ private int range;
+ @CommandLine.Option(names = {"--size"},
+ description = "Object size (in bytes) " +
+ "to be generated.",
+ defaultValue = "1")
+ private int objectSizeInBytes;
+
+ @CommandLine.Option(names = {"--keySorted"},
+ description = "Generated sorted key or not. The key name " +
+ "will be generated via md5 hash if choose " +
+ "to use unsorted key.",
+ defaultValue = "false")
+ private boolean keySorted;
+
+ @CommandLine.Option(names = {"--percentage-read"},
+ description = "Percentage of read tasks in mix workload.",
+ defaultValue = "100")
+ private int percentageRead;
+
+ @CommandLine.Option(names = {"--clients"},
+ description =
+ "Number of clients, defaults 1.",
Review Comment:
This is the number of clients instantiated which can equate the number of concurrent TCP sessions to OM or Datanode. The range of keys is partitioned across the clients, and each client generates the read and or write load as per the parameters specified.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on PR #3754:
URL: https://github.com/apache/ozone/pull/3754#issuecomment-1260281950
>
Yes! The command will potentially read some key which doesn't exist in the cluster.
Currently the freon command would stop running & report failure, but let me think more about this how to make it better!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on a diff in pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
kerneltime commented on code in PR #3754:
URL: https://github.com/apache/ozone/pull/3754#discussion_r981719458
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-m", "--read-metadata-only"},
+ description = "If only read key's metadata. " +
+ "Supported values are Y, F.",
+ defaultValue = "false")
+ private boolean readMetadataOnly;
+ @CommandLine.Option(names = {"-s", "--start-index"},
+ description = "Start index of keys of read/write operation.",
+ defaultValue = "0")
+ private int startIndex;
+
+ @CommandLine.Option(names = {"-r", "--range"},
+ description = "index range of read/write operations.",
+ defaultValue = "0")
+ private int range;
+ @CommandLine.Option(names = {"--size"},
+ description = "Object size (in bytes) " +
+ "to be generated.",
+ defaultValue = "1")
+ private int objectSizeInBytes;
+
+ @CommandLine.Option(names = {"--keySorted"},
+ description = "Generated sorted key or not. The key name " +
+ "will be generated via md5 hash if choose " +
+ "to use unsorted key.",
+ defaultValue = "false")
Review Comment:
```suggestion
@CommandLine.Option(names = {"--contiguous"},
description = "By default, the keys are randomized lexically by calculating the md5 of the key name. If this option is set, the keys are written lexically contiguously.",
defaultValue = "false")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on a diff in pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
kerneltime commented on code in PR #3754:
URL: https://github.com/apache/ozone/pull/3754#discussion_r981731884
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-m", "--read-metadata-only"},
+ description = "If only read key's metadata. " +
+ "Supported values are Y, F.",
+ defaultValue = "false")
+ private boolean readMetadataOnly;
+ @CommandLine.Option(names = {"-s", "--start-index"},
+ description = "Start index of keys of read/write operation.",
+ defaultValue = "0")
+ private int startIndex;
+
+ @CommandLine.Option(names = {"-r", "--range"},
+ description = "index range of read/write operations.",
+ defaultValue = "0")
+ private int range;
+ @CommandLine.Option(names = {"--size"},
+ description = "Object size (in bytes) " +
+ "to be generated.",
+ defaultValue = "1")
+ private int objectSizeInBytes;
+
+ @CommandLine.Option(names = {"--keySorted"},
+ description = "Generated sorted key or not. The key name " +
+ "will be generated via md5 hash if choose " +
+ "to use unsorted key.",
+ defaultValue = "false")
+ private boolean keySorted;
+
+ @CommandLine.Option(names = {"--percentage-read"},
+ description = "Percentage of read tasks in mix workload.",
+ defaultValue = "100")
+ private int percentageRead;
+
+ @CommandLine.Option(names = {"--clients"},
+ description =
+ "Number of clients, defaults 1.",
+ defaultValue = "1")
+ private int clientsCount = 1;
+
+ @CommandLine.Option(
+ names = "--om-service-id",
+ description = "OM Service ID"
+ )
+ private String omServiceID = null;
+
+ private Timer timer;
+
+ private OzoneBucket[] ozoneBuckets;
+
+ private byte[] keyContent;
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneClientKeyReadWriteOps.class);
+
+ /**
+ * Task type of read task, or write task.
+ */
+ public enum TaskType {
+ READ_TASK,
+ WRITE_TASK
+ }
+ private KeyGeneratorUtil kg;
+
+
+ @Override
+ public Void call() throws Exception {
+ init();
+ OzoneConfiguration ozoneConfiguration = createOzoneConfiguration();
+ OzoneClient[] ozoneClients = new OzoneClient[clientsCount];
+ for (int i = 0; i < clientsCount; i++) {
+ ozoneClients[i] = createOzoneClient(omServiceID, ozoneConfiguration);
+ }
+
+ ensureVolumeAndBucketExist(ozoneClients[0], volumeName, bucketName);
+ ozoneBuckets = new OzoneBucket[clientsCount];
+ for (int i = 0; i < clientsCount; i++) {
+ ozoneBuckets[i] = ozoneClients[i].getObjectStore().getVolume(volumeName)
+ .getBucket(bucketName);
+ }
+
+ timer = getMetrics().timer("key-read-write");
+ if (objectSizeInBytes >= 0) {
+ keyContent = RandomUtils.nextBytes(objectSizeInBytes);
+ }
+ if (kg == null) {
+ kg = new KeyGeneratorUtil();
+ }
+ runTests(this::readWriteKeys);
+
+ for (int i = 0; i < clientsCount; i++) {
+ if (ozoneClients[i] != null) {
+ ozoneClients[i].close();
+ }
+ }
+ return null;
+ }
+
+ public void readWriteKeys(long counter) throws Exception {
+ int clientIndex = (int)((counter) % clientsCount);
+ OzoneBucket ozoneBucket = ozoneBuckets[clientIndex];
+ TaskType taskType = decideReadOrWriteTask();
+ String keyName = getKeyName(clientIndex);
+
+ timer.time(() -> {
+ try {
+ switch (taskType) {
+ case READ_TASK:
+ processReadTasks(keyName, ozoneBucket);
+ break;
+ case WRITE_TASK:
+ processWriteTasks(keyName, ozoneBucket);
+ break;
+ default:
+ break;
+ }
+ } catch (RuntimeException ex) {
+ LOG.error(ex.getMessage());
+ throw ex;
+ } catch (IOException ex) {
+ LOG.error(ex.getMessage());
+ throw new RuntimeException(ex.getMessage());
+ }
+
+ });
+ }
+
+ public void processReadTasks(String keyName, OzoneBucket ozoneBucket)
+ throws RuntimeException, IOException {
+ if (readMetadataOnly) {
+ ozoneBucket.getKey(keyName);
+ } else {
+ byte[] data = new byte[objectSizeInBytes];
+ try (OzoneInputStream inputStream = ozoneBucket.readKey(keyName)) {
+ inputStream.read(data);
+ }
+ }
+ }
+ public void processWriteTasks(String keyName, OzoneBucket ozoneBucket)
+ throws RuntimeException, IOException {
+ try (OzoneOutputStream out =
+ ozoneBucket.createKey(keyName, objectSizeInBytes)) {
+ out.write(keyContent);
+ } catch (Exception ex) {
+ throw ex;
+ }
+ }
+ public TaskType decideReadOrWriteTask() {
+ if (!isMixWorkload()) {
+ if (percentageRead == 100) {
+ return TaskType.READ_TASK;
+ } else {
+ return TaskType.WRITE_TASK;
+ }
+ }
+ //mix workload
+ int tmp = ThreadLocalRandom.current().nextInt(100) + 1; // 1 ~ 100
+ if (tmp < percentageRead) {
+ return TaskType.READ_TASK;
+ } else {
+ return TaskType.WRITE_TASK;
+ }
+ }
+
+ public String getKeyName(int clientIndex) {
+ int start, end;
+ // separate tasks evenly to each client
+ if (range < clientsCount) {
Review Comment:
Please validate the expected behavior when client >> range. It might make sense to cap the client to be <= range, so in the case client configured is less than range, we only create enough clients such that each client reads only one object. Or to hammer the same client from multiple clients, if range < clients, each client gets the entire range.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on a diff in pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on code in PR #3754:
URL: https://github.com/apache/ozone/pull/3754#discussion_r975899262
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-m", "--read-metadata-only"},
+ description = "If only read key's metadata. " +
+ "Supported values are Y, F.",
+ defaultValue = "false")
+ private boolean readMetadataOnly;
+
+ @CommandLine.Option(names = {"-r", "--range-client-read"},
+ description = "range of read operation of each client.",
+ defaultValue = "0")
+ private int readRange;
+
+
+ @CommandLine.Option(names = {"-w", "--range-client-write"},
+ description = "range of write operation of each client.",
+ defaultValue = "0")
+ private int writeRange;
+
+// @CommandLine.Option(names = {"-e", "--end-index-each-client-read"},
+// description = "end-index of each client's read operation.",
+// defaultValue = "0")
+// private int endIndexForRead;
+
+// @CommandLine.Option(names = {"-j", "--end-index-each-client-write"},
+// description = "end-index of each client's write operation.",
+// defaultValue = "0")
+// private int endIndexForWrite;
+
+ @CommandLine.Option(names = {"-g", "--size"},
+ description = "Generated data size (in bytes) of " +
+ "each key/file to be " +
+ "written.",
+ defaultValue = "256")
+ private int writeSizeInBytes;
+
+ @CommandLine.Option(names = {"-k", "--keySorted"},
+ description = "Generated sorted key or not. The key name " +
+ "will be generated via md5 hash if choose " +
+ "to use unsorted key.",
+ defaultValue = "false")
+ private boolean keySorted;
+
+ @CommandLine.Option(names = {"-x", "--mix-workload"},
+ description = "Set to True if you would like to " +
+ "generate mix workload (Read and Write).",
+ defaultValue = "false")
+ private boolean isMixWorkload;
+
+ @CommandLine.Option(names = {"--percentage-read"},
+ description = "Percentage of read tasks in mix workload.",
+ defaultValue = "0")
+ private int percentageRead;
+
+ @CommandLine.Option(names = {"--clients"},
+ description =
+ "Number of clients, defaults 1.",
+ defaultValue = "1")
+ private int clientsCount = 1;
+
+ @CommandLine.Option(
+ names = "--om-service-id",
+ description = "OM Service ID"
+ )
+ private String omServiceID = null;
+
+ private Timer timer;
+
+ private OzoneClient[] rpcClients;
+
+ private byte[] keyContent;
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneClientKeyReadWriteOps.class);
+
+ private final String readTask = "READ_TASK";
+ private final String writeTask = "WRITE_TASK";
+ private KeyGeneratorUtil kg;
+
+ @Override
+ public Void call() throws Exception {
+ init();
+ OzoneConfiguration ozoneConfiguration = createOzoneConfiguration();
+ rpcClients = new OzoneClient[clientsCount];
+ for (int i = 0; i < clientsCount; i++) {
+ rpcClients[i] = createOzoneClient(omServiceID, ozoneConfiguration);
+ }
+
+ ensureVolumeAndBucketExist(rpcClients[0], volumeName, bucketName);
+ timer = getMetrics().timer("key-read-write");
+ if (writeSizeInBytes >= 0) {
+ keyContent = RandomUtils.nextBytes(writeSizeInBytes);
+ }
+ if (kg == null) {
+ kg = new KeyGeneratorUtil();
+ }
+ runTests(this::readWriteKeys);
+
+ for (int i = 0; i < clientsCount; i++) {
+ if (rpcClients[i] != null) {
+ rpcClients[i].close();
+ }
+ }
+ return null;
+ }
+
+ public void readWriteKeys(long counter) throws Exception {
+ int clientIndex = (int)(counter % clientsCount);
+ OzoneClient client = rpcClients[clientIndex];
+ String operationType = decideReadOrWriteTask();
+ String keyName = getKeyName(operationType, clientIndex);
+ timer.time(() -> {
+ try {
+ switch (operationType) {
+ case readTask:
+ processReadTasks(keyName, client);
+ break;
+ case writeTask:
+ processWriteTasks(keyName, client);
+ break;
+ default:
+ break;
+ }
+ } catch (Exception ex) {
+ LOG.error(ex.getMessage());
+ }
+ });
+ }
+
+ public void processReadTasks(String keyName, OzoneClient client)
+ throws Exception {
+ OzoneBucket ozbk = client.getObjectStore().getVolume(volumeName)
Review Comment:
oh!! cool!! thanks!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on a diff in pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on code in PR #3754:
URL: https://github.com/apache/ozone/pull/3754#discussion_r975896335
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-m", "--read-metadata-only"},
+ description = "If only read key's metadata. " +
+ "Supported values are Y, F.",
+ defaultValue = "false")
+ private boolean readMetadataOnly;
+
+ @CommandLine.Option(names = {"-r", "--range-client-read"},
+ description = "range of read operation of each client.",
+ defaultValue = "0")
+ private int readRange;
+
+
+ @CommandLine.Option(names = {"-w", "--range-client-write"},
+ description = "range of write operation of each client.",
+ defaultValue = "0")
+ private int writeRange;
+
+// @CommandLine.Option(names = {"-e", "--end-index-each-client-read"},
Review Comment:
sure!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on a diff in pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
kerneltime commented on code in PR #3754:
URL: https://github.com/apache/ozone/pull/3754#discussion_r981717081
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-m", "--read-metadata-only"},
+ description = "If only read key's metadata. " +
+ "Supported values are Y, F.",
+ defaultValue = "false")
+ private boolean readMetadataOnly;
+ @CommandLine.Option(names = {"-s", "--start-index"},
+ description = "Start index of keys of read/write operation.",
+ defaultValue = "0")
+ private int startIndex;
+
+ @CommandLine.Option(names = {"-r", "--range"},
+ description = "index range of read/write operations.",
+ defaultValue = "0")
Review Comment:
We need to establish if `--range` is optional? It could read contiguously till it hit not found for a key and limit the range to that point. Or it could always be a required option.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on a diff in pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on code in PR #3754:
URL: https://github.com/apache/ozone/pull/3754#discussion_r992774629
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-m", "--read-metadata-only"},
+ description = "If only read key's metadata. " +
+ "Supported values are Y, F.",
+ defaultValue = "false")
+ private boolean readMetadataOnly;
+ @CommandLine.Option(names = {"-s", "--start-index"},
+ description = "Start index of keys of read/write operation.",
+ defaultValue = "0")
+ private int startIndex;
+
+ @CommandLine.Option(names = {"-r", "--range"},
+ description = "index range of read/write operations.",
+ defaultValue = "0")
+ private int range;
+ @CommandLine.Option(names = {"--size"},
+ description = "Object size (in bytes) " +
+ "to be generated.",
Review Comment:
Yes! just tested, if a user sets a read size which is larger than the key size, it only reads bytes up to key size. I've updated the description accordingly!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on a diff in pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on code in PR #3754:
URL: https://github.com/apache/ozone/pull/3754#discussion_r992785367
##########
hadoop-ozone/dist/src/main/smoketest/freon/read-write-key.robot:
##########
@@ -0,0 +1,39 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+*** Settings ***
+Documentation Test freon read/write key commands
+Resource ../ozone-lib/freon.robot
+Test Timeout 5 minutes
+
+*** Variables ***
+${PREFIX} ${EMPTY}
+
+*** Test Cases ***
+Pre-generate 100 keys of size 1 byte each to Ozone
+ ${result} = Execute ozone freon ork -n 1 -t 10 -v voltest -b buckettest -p performanceTest
+
+Read 10 keys from pre-generated keys
+ ${result} = Execute ozone freon ockrw -n 10 -t 10 -r 100 -v voltest -b buckettest -p performanceTest
+
+Read 10 keys' metadata from pre-generated keys
+ ${result} = Execute ozone freon ockrw -n 10 -t 10 -m -r 100 -v voltest -b buckettest -p performanceTest
+
+Write 10 keys of size 1 byte each from key index 0 to 99
+ ${result} = Execute ozone freon ockrw -n 10 -t 10 --percentage-read -r 100 -v voltest -b buckettest -p performanceTest
Review Comment:
Sure! added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
kerneltime commented on PR #3754:
URL: https://github.com/apache/ozone/pull/3754#issuecomment-1285940953
> Sorry, I had to revert this, because checkstyle and findbugs failures affect all other PRs. Please fix the failures and open new PR.
Thank you @adoroszlai!
I should have checked
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on PR #3754:
URL: https://github.com/apache/ozone/pull/3754#issuecomment-1284850094
> There are failures in CI, please make sure CI pass before merge @kerneltime.
Sorry!! I'll take a look!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kaijchen commented on pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
kaijchen commented on PR #3754:
URL: https://github.com/apache/ozone/pull/3754#issuecomment-1284817478
There are failures in CI, please make sure CI pass before merge @kerneltime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on a diff in pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
kerneltime commented on code in PR #3754:
URL: https://github.com/apache/ozone/pull/3754#discussion_r981679239
##########
hadoop-ozone/dist/src/main/smoketest/freon/read-write-key.robot:
##########
@@ -0,0 +1,39 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+*** Settings ***
+Documentation Test freon read/write key commands
+Resource ../ozone-lib/freon.robot
+Test Timeout 5 minutes
+
+*** Variables ***
+${PREFIX} ${EMPTY}
+
+*** Test Cases ***
+Pre-generate 100 keys of size 1 byte each to Ozone
+ ${result} = Execute ozone freon ork -n 1 -t 10 -v voltest -b buckettest -p performanceTest
+
+Read 10 keys from pre-generated keys
+ ${result} = Execute ozone freon ockrw -n 10 -t 10 -r 100 -v voltest -b buckettest -p performanceTest
+
+Read 10 keys' metadata from pre-generated keys
+ ${result} = Execute ozone freon ockrw -n 10 -t 10 -m -r 100 -v voltest -b buckettest -p performanceTest
+
+Write 10 keys of size 1 byte each from key index 0 to 99
+ ${result} = Execute ozone freon ockrw -n 10 -t 10 --percentage-read -r 100 -v voltest -b buckettest -p performanceTest
Review Comment:
Add a test that lists keys under the prefix and checks if the result matches the expected count and if the object size is the expected size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on a diff in pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on code in PR #3754:
URL: https://github.com/apache/ozone/pull/3754#discussion_r992720524
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-m", "--read-metadata-only"},
+ description = "If only read key's metadata. " +
+ "Supported values are Y, F.",
+ defaultValue = "false")
+ private boolean readMetadataOnly;
+ @CommandLine.Option(names = {"-s", "--start-index"},
+ description = "Start index of keys of read/write operation.",
+ defaultValue = "0")
+ private int startIndex;
+
+ @CommandLine.Option(names = {"-r", "--range"},
+ description = "index range of read/write operations.",
+ defaultValue = "0")
Review Comment:
Let's put it as required for now! We could think about to make it become optional as an improvement later!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on a diff in pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on code in PR #3754:
URL: https://github.com/apache/ozone/pull/3754#discussion_r975896165
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-m", "--read-metadata-only"},
+ description = "If only read key's metadata. " +
+ "Supported values are Y, F.",
+ defaultValue = "false")
+ private boolean readMetadataOnly;
+
+ @CommandLine.Option(names = {"-r", "--range-client-read"},
+ description = "range of read operation of each client.",
+ defaultValue = "0")
+ private int readRange;
+
+
+ @CommandLine.Option(names = {"-w", "--range-client-write"},
+ description = "range of write operation of each client.",
+ defaultValue = "0")
+ private int writeRange;
+
+// @CommandLine.Option(names = {"-e", "--end-index-each-client-read"},
+// description = "end-index of each client's read operation.",
+// defaultValue = "0")
+// private int endIndexForRead;
+
+// @CommandLine.Option(names = {"-j", "--end-index-each-client-write"},
+// description = "end-index of each client's write operation.",
+// defaultValue = "0")
+// private int endIndexForWrite;
+
+ @CommandLine.Option(names = {"-g", "--size"},
Review Comment:
yeah, make sense!
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-m", "--read-metadata-only"},
+ description = "If only read key's metadata. " +
+ "Supported values are Y, F.",
+ defaultValue = "false")
+ private boolean readMetadataOnly;
+
+ @CommandLine.Option(names = {"-r", "--range-client-read"},
+ description = "range of read operation of each client.",
+ defaultValue = "0")
+ private int readRange;
+
+
+ @CommandLine.Option(names = {"-w", "--range-client-write"},
+ description = "range of write operation of each client.",
+ defaultValue = "0")
+ private int writeRange;
+
+// @CommandLine.Option(names = {"-e", "--end-index-each-client-read"},
+// description = "end-index of each client's read operation.",
+// defaultValue = "0")
+// private int endIndexForRead;
+
+// @CommandLine.Option(names = {"-j", "--end-index-each-client-write"},
+// description = "end-index of each client's write operation.",
+// defaultValue = "0")
+// private int endIndexForWrite;
+
+ @CommandLine.Option(names = {"-g", "--size"},
+ description = "Generated data size (in bytes) of " +
+ "each key/file to be " +
+ "written.",
+ defaultValue = "256")
+ private int writeSizeInBytes;
+
+ @CommandLine.Option(names = {"-k", "--keySorted"},
Review Comment:
yeah, make sense!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on a diff in pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on code in PR #3754:
URL: https://github.com/apache/ozone/pull/3754#discussion_r975899472
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-m", "--read-metadata-only"},
+ description = "If only read key's metadata. " +
+ "Supported values are Y, F.",
+ defaultValue = "false")
+ private boolean readMetadataOnly;
+
+ @CommandLine.Option(names = {"-r", "--range-client-read"},
+ description = "range of read operation of each client.",
+ defaultValue = "0")
+ private int readRange;
+
+
+ @CommandLine.Option(names = {"-w", "--range-client-write"},
+ description = "range of write operation of each client.",
+ defaultValue = "0")
+ private int writeRange;
+
+// @CommandLine.Option(names = {"-e", "--end-index-each-client-read"},
+// description = "end-index of each client's read operation.",
+// defaultValue = "0")
+// private int endIndexForRead;
+
+// @CommandLine.Option(names = {"-j", "--end-index-each-client-write"},
+// description = "end-index of each client's write operation.",
+// defaultValue = "0")
+// private int endIndexForWrite;
+
+ @CommandLine.Option(names = {"-g", "--size"},
+ description = "Generated data size (in bytes) of " +
+ "each key/file to be " +
+ "written.",
+ defaultValue = "256")
+ private int writeSizeInBytes;
+
+ @CommandLine.Option(names = {"-k", "--keySorted"},
+ description = "Generated sorted key or not. The key name " +
+ "will be generated via md5 hash if choose " +
+ "to use unsorted key.",
+ defaultValue = "false")
+ private boolean keySorted;
+
+ @CommandLine.Option(names = {"-x", "--mix-workload"},
+ description = "Set to True if you would like to " +
+ "generate mix workload (Read and Write).",
+ defaultValue = "false")
+ private boolean isMixWorkload;
+
+ @CommandLine.Option(names = {"--percentage-read"},
+ description = "Percentage of read tasks in mix workload.",
+ defaultValue = "0")
+ private int percentageRead;
+
+ @CommandLine.Option(names = {"--clients"},
+ description =
+ "Number of clients, defaults 1.",
+ defaultValue = "1")
+ private int clientsCount = 1;
+
+ @CommandLine.Option(
+ names = "--om-service-id",
+ description = "OM Service ID"
+ )
+ private String omServiceID = null;
+
+ private Timer timer;
+
+ private OzoneClient[] rpcClients;
+
+ private byte[] keyContent;
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneClientKeyReadWriteOps.class);
+
+ private final String readTask = "READ_TASK";
+ private final String writeTask = "WRITE_TASK";
+ private KeyGeneratorUtil kg;
+
+ @Override
+ public Void call() throws Exception {
+ init();
+ OzoneConfiguration ozoneConfiguration = createOzoneConfiguration();
+ rpcClients = new OzoneClient[clientsCount];
+ for (int i = 0; i < clientsCount; i++) {
+ rpcClients[i] = createOzoneClient(omServiceID, ozoneConfiguration);
+ }
+
+ ensureVolumeAndBucketExist(rpcClients[0], volumeName, bucketName);
+ timer = getMetrics().timer("key-read-write");
+ if (writeSizeInBytes >= 0) {
+ keyContent = RandomUtils.nextBytes(writeSizeInBytes);
+ }
+ if (kg == null) {
+ kg = new KeyGeneratorUtil();
+ }
+ runTests(this::readWriteKeys);
+
+ for (int i = 0; i < clientsCount; i++) {
+ if (rpcClients[i] != null) {
+ rpcClients[i].close();
+ }
+ }
+ return null;
+ }
+
+ public void readWriteKeys(long counter) throws Exception {
+ int clientIndex = (int)(counter % clientsCount);
+ OzoneClient client = rpcClients[clientIndex];
+ String operationType = decideReadOrWriteTask();
+ String keyName = getKeyName(operationType, clientIndex);
+ timer.time(() -> {
+ try {
+ switch (operationType) {
+ case readTask:
+ processReadTasks(keyName, client);
+ break;
+ case writeTask:
+ processWriteTasks(keyName, client);
+ break;
+ default:
+ break;
+ }
+ } catch (Exception ex) {
+ LOG.error(ex.getMessage());
+ }
+ });
+ }
+
+ public void processReadTasks(String keyName, OzoneClient client)
+ throws Exception {
+ OzoneBucket ozbk = client.getObjectStore().getVolume(volumeName)
+ .getBucket(bucketName);
+
+ if (readMetadataOnly) {
+ ozbk.getKey(keyName);
+ } else {
+ byte[] data = new byte[writeSizeInBytes];
+ try (OzoneInputStream introStream = ozbk.readKey(keyName)) {
+ introStream.read(data);
+ }
+ }
+ }
+ public void processWriteTasks(String keyName, OzoneClient client)
+ throws Exception {
+ OzoneBucket ozbk = client.getObjectStore().getVolume(volumeName)
Review Comment:
definitely!! cool! thanks!!
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-m", "--read-metadata-only"},
+ description = "If only read key's metadata. " +
+ "Supported values are Y, F.",
+ defaultValue = "false")
+ private boolean readMetadataOnly;
+
+ @CommandLine.Option(names = {"-r", "--range-client-read"},
+ description = "range of read operation of each client.",
+ defaultValue = "0")
+ private int readRange;
+
+
+ @CommandLine.Option(names = {"-w", "--range-client-write"},
+ description = "range of write operation of each client.",
+ defaultValue = "0")
+ private int writeRange;
+
+// @CommandLine.Option(names = {"-e", "--end-index-each-client-read"},
+// description = "end-index of each client's read operation.",
+// defaultValue = "0")
+// private int endIndexForRead;
+
+// @CommandLine.Option(names = {"-j", "--end-index-each-client-write"},
+// description = "end-index of each client's write operation.",
+// defaultValue = "0")
+// private int endIndexForWrite;
+
+ @CommandLine.Option(names = {"-g", "--size"},
+ description = "Generated data size (in bytes) of " +
+ "each key/file to be " +
+ "written.",
+ defaultValue = "256")
+ private int writeSizeInBytes;
+
+ @CommandLine.Option(names = {"-k", "--keySorted"},
+ description = "Generated sorted key or not. The key name " +
+ "will be generated via md5 hash if choose " +
+ "to use unsorted key.",
+ defaultValue = "false")
+ private boolean keySorted;
+
+ @CommandLine.Option(names = {"-x", "--mix-workload"},
+ description = "Set to True if you would like to " +
+ "generate mix workload (Read and Write).",
+ defaultValue = "false")
+ private boolean isMixWorkload;
+
+ @CommandLine.Option(names = {"--percentage-read"},
+ description = "Percentage of read tasks in mix workload.",
+ defaultValue = "0")
+ private int percentageRead;
+
+ @CommandLine.Option(names = {"--clients"},
+ description =
+ "Number of clients, defaults 1.",
+ defaultValue = "1")
+ private int clientsCount = 1;
+
+ @CommandLine.Option(
+ names = "--om-service-id",
+ description = "OM Service ID"
+ )
+ private String omServiceID = null;
+
+ private Timer timer;
+
+ private OzoneClient[] rpcClients;
+
+ private byte[] keyContent;
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneClientKeyReadWriteOps.class);
+
+ private final String readTask = "READ_TASK";
+ private final String writeTask = "WRITE_TASK";
+ private KeyGeneratorUtil kg;
+
+ @Override
+ public Void call() throws Exception {
+ init();
+ OzoneConfiguration ozoneConfiguration = createOzoneConfiguration();
+ rpcClients = new OzoneClient[clientsCount];
+ for (int i = 0; i < clientsCount; i++) {
+ rpcClients[i] = createOzoneClient(omServiceID, ozoneConfiguration);
+ }
+
+ ensureVolumeAndBucketExist(rpcClients[0], volumeName, bucketName);
+ timer = getMetrics().timer("key-read-write");
+ if (writeSizeInBytes >= 0) {
+ keyContent = RandomUtils.nextBytes(writeSizeInBytes);
+ }
+ if (kg == null) {
+ kg = new KeyGeneratorUtil();
+ }
+ runTests(this::readWriteKeys);
+
+ for (int i = 0; i < clientsCount; i++) {
+ if (rpcClients[i] != null) {
+ rpcClients[i].close();
+ }
+ }
+ return null;
+ }
+
+ public void readWriteKeys(long counter) throws Exception {
+ int clientIndex = (int)(counter % clientsCount);
+ OzoneClient client = rpcClients[clientIndex];
+ String operationType = decideReadOrWriteTask();
+ String keyName = getKeyName(operationType, clientIndex);
+ timer.time(() -> {
+ try {
+ switch (operationType) {
+ case readTask:
Review Comment:
Sure!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on a diff in pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
kerneltime commented on code in PR #3754:
URL: https://github.com/apache/ozone/pull/3754#discussion_r981718251
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-m", "--read-metadata-only"},
+ description = "If only read key's metadata. " +
+ "Supported values are Y, F.",
+ defaultValue = "false")
+ private boolean readMetadataOnly;
+ @CommandLine.Option(names = {"-s", "--start-index"},
+ description = "Start index of keys of read/write operation.",
+ defaultValue = "0")
+ private int startIndex;
+
+ @CommandLine.Option(names = {"-r", "--range"},
+ description = "index range of read/write operations.",
+ defaultValue = "0")
+ private int range;
+ @CommandLine.Option(names = {"--size"},
+ description = "Object size (in bytes) " +
+ "to be generated.",
Review Comment:
If this is applicable for reads, specify the expected behavior, and if the size is larger than the key, what is expected then?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
kerneltime commented on PR #3754:
URL: https://github.com/apache/ozone/pull/3754#issuecomment-1275613258
> Yes! The command will potentially read some key which doesn't exist in the cluster.
> Currently the freon command would stop running & report failure, but let me think more about this how to make it better!!
Code can continue testing and add a metric at the end for successful reads vs. not found read. It is a valid test to look at the performance of OM to report a key does not exist. Object Stores can be bombarded with nonexistent keys, and the performance of the underlying storage to report keys that don't exist is important. LSM tree-based storage has to scan all levels before reporting a key as not found and in some ways represents the worst case performance at scale.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on a diff in pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on code in PR #3754:
URL: https://github.com/apache/ozone/pull/3754#discussion_r991691575
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-m", "--read-metadata-only"},
+ description = "If only read key's metadata. " +
+ "Supported values are Y, F.",
+ defaultValue = "false")
+ private boolean readMetadataOnly;
+ @CommandLine.Option(names = {"-s", "--start-index"},
+ description = "Start index of keys of read/write operation.",
+ defaultValue = "0")
+ private int startIndex;
+
+ @CommandLine.Option(names = {"-r", "--range"},
+ description = "index range of read/write operations.",
+ defaultValue = "0")
+ private int range;
+ @CommandLine.Option(names = {"--size"},
+ description = "Object size (in bytes) " +
+ "to be generated.",
+ defaultValue = "1")
+ private int objectSizeInBytes;
+
+ @CommandLine.Option(names = {"--keySorted"},
+ description = "Generated sorted key or not. The key name " +
+ "will be generated via md5 hash if choose " +
+ "to use unsorted key.",
+ defaultValue = "false")
+ private boolean keySorted;
+
+ @CommandLine.Option(names = {"--percentage-read"},
+ description = "Percentage of read tasks in mix workload.",
Review Comment:
updated!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on a diff in pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on code in PR #3754:
URL: https://github.com/apache/ozone/pull/3754#discussion_r991689395
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-m", "--read-metadata-only"},
+ description = "If only read key's metadata. " +
+ "Supported values are Y, F.",
+ defaultValue = "false")
+ private boolean readMetadataOnly;
+ @CommandLine.Option(names = {"-s", "--start-index"},
+ description = "Start index of keys of read/write operation.",
Review Comment:
updated!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on a diff in pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
kerneltime commented on code in PR #3754:
URL: https://github.com/apache/ozone/pull/3754#discussion_r981684120
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-m", "--read-metadata-only"},
+ description = "If only read key's metadata. " +
+ "Supported values are Y, F.",
Review Comment:
Only Y and F?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on a diff in pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on code in PR #3754:
URL: https://github.com/apache/ozone/pull/3754#discussion_r980536916
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,247 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.*;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-m", "--read-metadata-only"},
+ description = "If only read key's metadata. " +
+ "Supported values are Y, F.",
+ defaultValue = "false")
+ private boolean readMetadataOnly;
+ @CommandLine.Option(names = {"-s", "--start-index"},
+ description = "Start index of keys of read/write operation.",
+ defaultValue = "0")
+ private int startIndex;
+
+ @CommandLine.Option(names = {"-r", "--range"},
+ description = "index range of read/write operations.",
+ defaultValue = "0")
+ private int range;
+ @CommandLine.Option(names = {"--size"},
+ description = "Object size (in bytes) " +
+ "to be generated.",
+ defaultValue = "1")
+ private int objectSizeInBytes;
+
+ @CommandLine.Option(names = {"--keySorted"},
+ description = "Generated sorted key or not. The key name " +
+ "will be generated via md5 hash if choose " +
+ "to use unsorted key.",
+ defaultValue = "false")
+ private boolean keySorted;
+
+ @CommandLine.Option(names = {"--percentage-read"},
+ description = "Percentage of read tasks in mix workload.",
+ defaultValue = "100")
+ private int percentageRead;
+
+ @CommandLine.Option(names = {"--clients"},
+ description =
+ "Number of clients, defaults 1.",
+ defaultValue = "1")
+ private int clientsCount = 1;
+
+ @CommandLine.Option(
+ names = "--om-service-id",
+ description = "OM Service ID"
+ )
+ private String omServiceID = null;
+
+ private Timer timer;
+
+ private OzoneBucket[] ozoneBuckets;
+
+ private byte[] keyContent;
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneClientKeyReadWriteOps.class);
+ public enum TaskType {
+ READ_TASK,
+ WRITE_TASK
+ }
+ private KeyGeneratorUtil kg;
+
+
+ @Override
+ public Void call() throws Exception {
+ init();
+ OzoneConfiguration ozoneConfiguration = createOzoneConfiguration();
+ OzoneClient[] ozoneClients = new OzoneClient[clientsCount];
+ for (int i = 0; i < clientsCount; i++) {
+ ozoneClients[i] = createOzoneClient(omServiceID, ozoneConfiguration);
+ }
+
+ ensureVolumeAndBucketExist(ozoneClients[0], volumeName, bucketName);
+ ozoneBuckets = new OzoneBucket[clientsCount];
+ for (int i = 0; i < clientsCount; i++) {
+ ozoneBuckets[i] = ozoneClients[i].getObjectStore().getVolume(volumeName)
+ .getBucket(bucketName);
+ }
+
+ timer = getMetrics().timer("key-read-write");
+ if (objectSizeInBytes >= 0) {
+ keyContent = RandomUtils.nextBytes(objectSizeInBytes);
+ }
+ if (kg == null) {
+ kg = new KeyGeneratorUtil();
+ }
+ runTests(this::readWriteKeys);
+
+ for (int i = 0; i < clientsCount; i++) {
+ if (ozoneClients[i] != null) {
+ ozoneClients[i].close();
+ }
+ }
+ return null;
+ }
+
+ public void readWriteKeys(long counter) throws Exception {
+ int clientIndex = (int)((counter) % clientsCount);
+ OzoneBucket ozoneBucket = ozoneBuckets[clientIndex];
+ TaskType taskType = decideReadOrWriteTask();
+ String keyName = getKeyName(taskType, clientIndex);
+
+ timer.time(() -> {
+ try {
+ switch (taskType) {
+ case READ_TASK:
+ processReadTasks(keyName, ozoneBucket);
+ break;
+ case WRITE_TASK:
+ processWriteTasks(keyName, ozoneBucket);
+ break;
+ default:
+ break;
+ }
+ } catch (RuntimeException ex) {
+ LOG.error(ex.getMessage());
+ throw ex;
+ } catch (IOException ex) {
+ LOG.error(ex.getMessage());
+ throw new RuntimeException(ex.getMessage());
+ }
+
+ });
+ }
+
+ public void processReadTasks(String keyName, OzoneBucket ozoneBucket)
+ throws RuntimeException, IOException {
+ if (readMetadataOnly) {
+ ozoneBucket.getKey(keyName);
+ } else {
+ byte[] data = new byte[objectSizeInBytes];
+ try (OzoneInputStream inputStream = ozoneBucket.readKey(keyName)) {
+ inputStream.read(data);
Review Comment:
@kerneltime hey Ritesh, if I don't need to access the result from read method, is there a way we could suppress the build error thrown from git workflow by this line (because the build throws an error that we ignore the result of read method)? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on a diff in pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
kerneltime commented on code in PR #3754:
URL: https://github.com/apache/ozone/pull/3754#discussion_r981683578
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
Review Comment:
```suggestion
defaultValue = "ockrwvolume")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on a diff in pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
kerneltime commented on code in PR #3754:
URL: https://github.com/apache/ozone/pull/3754#discussion_r981682737
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
Review Comment:
```suggestion
description = "Generate keys with a fixed name and ranges that can be written and read as sub-ranges from multiple clients.",
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on PR #3754:
URL: https://github.com/apache/ozone/pull/3754#issuecomment-1284849717
> > Code can continue testing and add a metric at the end for successful reads vs. not found read. It is a valid test to look at the performance of OM to report a key does not exist. Object Stores can be bombarded with nonexistent keys, and the performance of the underlying storage to report keys that don't exist is important. LSM tree-based storage has to scan all levels before reporting a key as not found and in some ways represents the worst case performance at scale.
>
> agree, reading nonexistent keys is a valid test case and the tool should support it **deterministically**. To do that, it has to know (on its own) which key exists and which doesn't. Warp does that by having a pre-test phase in which warp creates a set of keys (10K or so) and keeps the created keys in memory for the real read test.
>
> We can also do the same for this tool, by maintaining a set of known keys that can be initialized by a pretest phase and grows with the write test.
Sure!! I'll take a look how Warp do it and create a jira for it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on PR #3754:
URL: https://github.com/apache/ozone/pull/3754#issuecomment-1284850834
> Sorry, I had to revert this, because checkstyle and findbugs failures affect all other PRs. Please fix the failures and open new PR.
Sorry!! I'll take a look!! thanks Attila!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] duongkame commented on a diff in pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
duongkame commented on code in PR #3754:
URL: https://github.com/apache/ozone/pull/3754#discussion_r975864222
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-m", "--read-metadata-only"},
+ description = "If only read key's metadata. " +
+ "Supported values are Y, F.",
+ defaultValue = "false")
+ private boolean readMetadataOnly;
+
+ @CommandLine.Option(names = {"-r", "--range-client-read"},
+ description = "range of read operation of each client.",
+ defaultValue = "0")
+ private int readRange;
+
+
+ @CommandLine.Option(names = {"-w", "--range-client-write"},
+ description = "range of write operation of each client.",
+ defaultValue = "0")
+ private int writeRange;
+
+// @CommandLine.Option(names = {"-e", "--end-index-each-client-read"},
+// description = "end-index of each client's read operation.",
+// defaultValue = "0")
+// private int endIndexForRead;
+
+// @CommandLine.Option(names = {"-j", "--end-index-each-client-write"},
+// description = "end-index of each client's write operation.",
+// defaultValue = "0")
+// private int endIndexForWrite;
+
+ @CommandLine.Option(names = {"-g", "--size"},
+ description = "Generated data size (in bytes) of " +
+ "each key/file to be " +
+ "written.",
+ defaultValue = "256")
+ private int writeSizeInBytes;
+
+ @CommandLine.Option(names = {"-k", "--keySorted"},
Review Comment:
nit: `-k` is not an intuitiative short-hand for `--keySorted `. maybe it's better not to have a shortcut.
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-m", "--read-metadata-only"},
+ description = "If only read key's metadata. " +
+ "Supported values are Y, F.",
+ defaultValue = "false")
+ private boolean readMetadataOnly;
+
+ @CommandLine.Option(names = {"-r", "--range-client-read"},
+ description = "range of read operation of each client.",
+ defaultValue = "0")
+ private int readRange;
+
+
+ @CommandLine.Option(names = {"-w", "--range-client-write"},
+ description = "range of write operation of each client.",
+ defaultValue = "0")
+ private int writeRange;
+
+// @CommandLine.Option(names = {"-e", "--end-index-each-client-read"},
+// description = "end-index of each client's read operation.",
+// defaultValue = "0")
+// private int endIndexForRead;
+
+// @CommandLine.Option(names = {"-j", "--end-index-each-client-write"},
+// description = "end-index of each client's write operation.",
+// defaultValue = "0")
+// private int endIndexForWrite;
+
+ @CommandLine.Option(names = {"-g", "--size"},
+ description = "Generated data size (in bytes) of " +
+ "each key/file to be " +
+ "written.",
+ defaultValue = "256")
+ private int writeSizeInBytes;
+
+ @CommandLine.Option(names = {"-k", "--keySorted"},
+ description = "Generated sorted key or not. The key name " +
+ "will be generated via md5 hash if choose " +
+ "to use unsorted key.",
+ defaultValue = "false")
+ private boolean keySorted;
+
+ @CommandLine.Option(names = {"-x", "--mix-workload"},
+ description = "Set to True if you would like to " +
+ "generate mix workload (Read and Write).",
+ defaultValue = "false")
+ private boolean isMixWorkload;
Review Comment:
Can this be determined inherently from `-percentage-read`? `e.g.` if `--percentage-read` is or `100` or `0`, is not a mixed workload.
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-m", "--read-metadata-only"},
+ description = "If only read key's metadata. " +
+ "Supported values are Y, F.",
+ defaultValue = "false")
+ private boolean readMetadataOnly;
+
+ @CommandLine.Option(names = {"-r", "--range-client-read"},
+ description = "range of read operation of each client.",
+ defaultValue = "0")
+ private int readRange;
+
+
+ @CommandLine.Option(names = {"-w", "--range-client-write"},
+ description = "range of write operation of each client.",
+ defaultValue = "0")
+ private int writeRange;
+
+// @CommandLine.Option(names = {"-e", "--end-index-each-client-read"},
+// description = "end-index of each client's read operation.",
+// defaultValue = "0")
+// private int endIndexForRead;
+
+// @CommandLine.Option(names = {"-j", "--end-index-each-client-write"},
+// description = "end-index of each client's write operation.",
+// defaultValue = "0")
+// private int endIndexForWrite;
+
+ @CommandLine.Option(names = {"-g", "--size"},
+ description = "Generated data size (in bytes) of " +
+ "each key/file to be " +
+ "written.",
+ defaultValue = "256")
+ private int writeSizeInBytes;
+
+ @CommandLine.Option(names = {"-k", "--keySorted"},
+ description = "Generated sorted key or not. The key name " +
+ "will be generated via md5 hash if choose " +
+ "to use unsorted key.",
+ defaultValue = "false")
+ private boolean keySorted;
+
+ @CommandLine.Option(names = {"-x", "--mix-workload"},
+ description = "Set to True if you would like to " +
+ "generate mix workload (Read and Write).",
+ defaultValue = "false")
+ private boolean isMixWorkload;
+
+ @CommandLine.Option(names = {"--percentage-read"},
+ description = "Percentage of read tasks in mix workload.",
+ defaultValue = "0")
+ private int percentageRead;
+
+ @CommandLine.Option(names = {"--clients"},
+ description =
+ "Number of clients, defaults 1.",
+ defaultValue = "1")
+ private int clientsCount = 1;
+
+ @CommandLine.Option(
+ names = "--om-service-id",
+ description = "OM Service ID"
+ )
+ private String omServiceID = null;
+
+ private Timer timer;
+
+ private OzoneClient[] rpcClients;
+
+ private byte[] keyContent;
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneClientKeyReadWriteOps.class);
+
+ private final String readTask = "READ_TASK";
+ private final String writeTask = "WRITE_TASK";
+ private KeyGeneratorUtil kg;
+
+ @Override
+ public Void call() throws Exception {
+ init();
+ OzoneConfiguration ozoneConfiguration = createOzoneConfiguration();
+ rpcClients = new OzoneClient[clientsCount];
+ for (int i = 0; i < clientsCount; i++) {
+ rpcClients[i] = createOzoneClient(omServiceID, ozoneConfiguration);
+ }
+
+ ensureVolumeAndBucketExist(rpcClients[0], volumeName, bucketName);
+ timer = getMetrics().timer("key-read-write");
+ if (writeSizeInBytes >= 0) {
+ keyContent = RandomUtils.nextBytes(writeSizeInBytes);
+ }
+ if (kg == null) {
+ kg = new KeyGeneratorUtil();
+ }
+ runTests(this::readWriteKeys);
+
+ for (int i = 0; i < clientsCount; i++) {
+ if (rpcClients[i] != null) {
+ rpcClients[i].close();
+ }
+ }
+ return null;
+ }
+
+ public void readWriteKeys(long counter) throws Exception {
+ int clientIndex = (int)(counter % clientsCount);
+ OzoneClient client = rpcClients[clientIndex];
+ String operationType = decideReadOrWriteTask();
+ String keyName = getKeyName(operationType, clientIndex);
+ timer.time(() -> {
+ try {
+ switch (operationType) {
+ case readTask:
+ processReadTasks(keyName, client);
Review Comment:
Looks like we have no control that the keyName to read is an existing one. I guess we need a better mechanism to ensure hitting existing keys, e.g. [warp](https://github.com/minio/warp) always pre-generates a set of 10K keys and uses them for the read workloads.
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-m", "--read-metadata-only"},
+ description = "If only read key's metadata. " +
+ "Supported values are Y, F.",
+ defaultValue = "false")
+ private boolean readMetadataOnly;
+
+ @CommandLine.Option(names = {"-r", "--range-client-read"},
+ description = "range of read operation of each client.",
+ defaultValue = "0")
+ private int readRange;
+
+
+ @CommandLine.Option(names = {"-w", "--range-client-write"},
+ description = "range of write operation of each client.",
+ defaultValue = "0")
+ private int writeRange;
+
+// @CommandLine.Option(names = {"-e", "--end-index-each-client-read"},
+// description = "end-index of each client's read operation.",
+// defaultValue = "0")
+// private int endIndexForRead;
+
+// @CommandLine.Option(names = {"-j", "--end-index-each-client-write"},
+// description = "end-index of each client's write operation.",
+// defaultValue = "0")
+// private int endIndexForWrite;
+
+ @CommandLine.Option(names = {"-g", "--size"},
+ description = "Generated data size (in bytes) of " +
+ "each key/file to be " +
+ "written.",
+ defaultValue = "256")
+ private int writeSizeInBytes;
+
+ @CommandLine.Option(names = {"-k", "--keySorted"},
+ description = "Generated sorted key or not. The key name " +
+ "will be generated via md5 hash if choose " +
+ "to use unsorted key.",
+ defaultValue = "false")
+ private boolean keySorted;
+
+ @CommandLine.Option(names = {"-x", "--mix-workload"},
+ description = "Set to True if you would like to " +
+ "generate mix workload (Read and Write).",
+ defaultValue = "false")
+ private boolean isMixWorkload;
+
+ @CommandLine.Option(names = {"--percentage-read"},
+ description = "Percentage of read tasks in mix workload.",
+ defaultValue = "0")
+ private int percentageRead;
+
+ @CommandLine.Option(names = {"--clients"},
+ description =
+ "Number of clients, defaults 1.",
+ defaultValue = "1")
+ private int clientsCount = 1;
+
+ @CommandLine.Option(
+ names = "--om-service-id",
+ description = "OM Service ID"
+ )
+ private String omServiceID = null;
+
+ private Timer timer;
+
+ private OzoneClient[] rpcClients;
+
+ private byte[] keyContent;
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneClientKeyReadWriteOps.class);
+
+ private final String readTask = "READ_TASK";
+ private final String writeTask = "WRITE_TASK";
+ private KeyGeneratorUtil kg;
+
+ @Override
+ public Void call() throws Exception {
+ init();
+ OzoneConfiguration ozoneConfiguration = createOzoneConfiguration();
+ rpcClients = new OzoneClient[clientsCount];
+ for (int i = 0; i < clientsCount; i++) {
+ rpcClients[i] = createOzoneClient(omServiceID, ozoneConfiguration);
+ }
+
+ ensureVolumeAndBucketExist(rpcClients[0], volumeName, bucketName);
+ timer = getMetrics().timer("key-read-write");
+ if (writeSizeInBytes >= 0) {
+ keyContent = RandomUtils.nextBytes(writeSizeInBytes);
+ }
+ if (kg == null) {
+ kg = new KeyGeneratorUtil();
+ }
+ runTests(this::readWriteKeys);
+
+ for (int i = 0; i < clientsCount; i++) {
+ if (rpcClients[i] != null) {
+ rpcClients[i].close();
+ }
+ }
+ return null;
+ }
+
+ public void readWriteKeys(long counter) throws Exception {
+ int clientIndex = (int)(counter % clientsCount);
+ OzoneClient client = rpcClients[clientIndex];
+ String operationType = decideReadOrWriteTask();
+ String keyName = getKeyName(operationType, clientIndex);
+ timer.time(() -> {
+ try {
+ switch (operationType) {
+ case readTask:
+ processReadTasks(keyName, client);
+ break;
+ case writeTask:
+ processWriteTasks(keyName, client);
+ break;
+ default:
+ break;
+ }
+ } catch (Exception ex) {
+ LOG.error(ex.getMessage());
+ }
+ });
+ }
+
+ public void processReadTasks(String keyName, OzoneClient client)
+ throws Exception {
+ OzoneBucket ozbk = client.getObjectStore().getVolume(volumeName)
+ .getBucket(bucketName);
+
+ if (readMetadataOnly) {
+ ozbk.getKey(keyName);
+ } else {
+ byte[] data = new byte[writeSizeInBytes];
+ try (OzoneInputStream introStream = ozbk.readKey(keyName)) {
+ introStream.read(data);
+ }
+ }
+ }
+ public void processWriteTasks(String keyName, OzoneClient client)
+ throws Exception {
+ OzoneBucket ozbk = client.getObjectStore().getVolume(volumeName)
+ .getBucket(bucketName);
+
+ try (OzoneOutputStream out = ozbk.createKey(keyName, writeSizeInBytes)) {
+ out.write(keyContent);
+ out.flush();
+ }
+ }
+ public String decideReadOrWriteTask() {
+ if (!isMixWorkload) {
+ if (readRange > 0) {
+ return readTask;
+ } else if (writeRange > 0) {
+ return writeTask;
+ }
+ }
+ //mix workload
+ int tmp = ThreadLocalRandom.current().nextInt(100) + 1; // 1 ~ 100
+ if (tmp < percentageRead) {
+ return readTask;
+ } else {
+ return writeTask;
+ }
+ }
+
+ public String getKeyName(String operationType, int counter) {
+ int startIdx, endIdx;
+ switch (operationType) {
+ case readTask:
+ startIdx = counter * readRange;
+ endIdx = startIdx + readRange;
+ break;
+ case writeTask:
+ startIdx = counter * writeRange;
+ endIdx = startIdx + writeRange;
+ break;
+ default:
+ startIdx = 0;
+ endIdx = 0;
+ break;
+ }
+ StringBuilder keyNameSb = new StringBuilder();
+ int randomIdxWithinRange = ThreadLocalRandom.current().
+ nextInt(endIdx + 1 - startIdx) + startIdx;
+ if (keySorted) {
+ keyNameSb.append(getPrefix()).append(FILE_DIR_SEPARATOR).
+ append(randomIdxWithinRange);
+ } else {
+ keyNameSb.append(getPrefix()).append(FILE_DIR_SEPARATOR).
Review Comment:
I can't quite get the intention of this hash. Does it basically map the generated key name to another name?
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-m", "--read-metadata-only"},
+ description = "If only read key's metadata. " +
+ "Supported values are Y, F.",
+ defaultValue = "false")
+ private boolean readMetadataOnly;
+
+ @CommandLine.Option(names = {"-r", "--range-client-read"},
+ description = "range of read operation of each client.",
+ defaultValue = "0")
+ private int readRange;
+
+
+ @CommandLine.Option(names = {"-w", "--range-client-write"},
+ description = "range of write operation of each client.",
+ defaultValue = "0")
+ private int writeRange;
+
+// @CommandLine.Option(names = {"-e", "--end-index-each-client-read"},
+// description = "end-index of each client's read operation.",
+// defaultValue = "0")
+// private int endIndexForRead;
+
+// @CommandLine.Option(names = {"-j", "--end-index-each-client-write"},
+// description = "end-index of each client's write operation.",
+// defaultValue = "0")
+// private int endIndexForWrite;
+
+ @CommandLine.Option(names = {"-g", "--size"},
+ description = "Generated data size (in bytes) of " +
+ "each key/file to be " +
+ "written.",
+ defaultValue = "256")
+ private int writeSizeInBytes;
+
+ @CommandLine.Option(names = {"-k", "--keySorted"},
+ description = "Generated sorted key or not. The key name " +
+ "will be generated via md5 hash if choose " +
+ "to use unsorted key.",
+ defaultValue = "false")
+ private boolean keySorted;
+
+ @CommandLine.Option(names = {"-x", "--mix-workload"},
+ description = "Set to True if you would like to " +
+ "generate mix workload (Read and Write).",
+ defaultValue = "false")
+ private boolean isMixWorkload;
+
+ @CommandLine.Option(names = {"--percentage-read"},
+ description = "Percentage of read tasks in mix workload.",
+ defaultValue = "0")
+ private int percentageRead;
+
+ @CommandLine.Option(names = {"--clients"},
+ description =
+ "Number of clients, defaults 1.",
+ defaultValue = "1")
+ private int clientsCount = 1;
+
+ @CommandLine.Option(
+ names = "--om-service-id",
+ description = "OM Service ID"
+ )
+ private String omServiceID = null;
+
+ private Timer timer;
+
+ private OzoneClient[] rpcClients;
+
+ private byte[] keyContent;
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneClientKeyReadWriteOps.class);
+
+ private final String readTask = "READ_TASK";
+ private final String writeTask = "WRITE_TASK";
+ private KeyGeneratorUtil kg;
+
+ @Override
+ public Void call() throws Exception {
+ init();
+ OzoneConfiguration ozoneConfiguration = createOzoneConfiguration();
+ rpcClients = new OzoneClient[clientsCount];
+ for (int i = 0; i < clientsCount; i++) {
+ rpcClients[i] = createOzoneClient(omServiceID, ozoneConfiguration);
+ }
+
+ ensureVolumeAndBucketExist(rpcClients[0], volumeName, bucketName);
+ timer = getMetrics().timer("key-read-write");
+ if (writeSizeInBytes >= 0) {
+ keyContent = RandomUtils.nextBytes(writeSizeInBytes);
+ }
+ if (kg == null) {
+ kg = new KeyGeneratorUtil();
+ }
+ runTests(this::readWriteKeys);
+
+ for (int i = 0; i < clientsCount; i++) {
+ if (rpcClients[i] != null) {
+ rpcClients[i].close();
+ }
+ }
+ return null;
+ }
+
+ public void readWriteKeys(long counter) throws Exception {
+ int clientIndex = (int)(counter % clientsCount);
+ OzoneClient client = rpcClients[clientIndex];
+ String operationType = decideReadOrWriteTask();
+ String keyName = getKeyName(operationType, clientIndex);
+ timer.time(() -> {
+ try {
+ switch (operationType) {
+ case readTask:
+ processReadTasks(keyName, client);
+ break;
+ case writeTask:
+ processWriteTasks(keyName, client);
+ break;
+ default:
+ break;
+ }
+ } catch (Exception ex) {
+ LOG.error(ex.getMessage());
+ }
+ });
+ }
+
+ public void processReadTasks(String keyName, OzoneClient client)
+ throws Exception {
+ OzoneBucket ozbk = client.getObjectStore().getVolume(volumeName)
Review Comment:
Should this be reused instead of re-read every request? it creates 2 more additional requests (getVolume and getBucket) to OM per lookupKey.
In fact, you don't need to read the bucket, use:
```
OzoneKeyDetails keyDetails = client.getProxy().getKeyDetails(volumeName, bucketName, keyName);
if (readMetadataOnly) {
byte[] data = new byte[writeSizeInBytes];
try (OzoneInputStream introStream = keyDetails.getContent()) {
introStream.read(data);
}
}
```
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-m", "--read-metadata-only"},
+ description = "If only read key's metadata. " +
+ "Supported values are Y, F.",
+ defaultValue = "false")
+ private boolean readMetadataOnly;
+
+ @CommandLine.Option(names = {"-r", "--range-client-read"},
+ description = "range of read operation of each client.",
+ defaultValue = "0")
+ private int readRange;
+
+
+ @CommandLine.Option(names = {"-w", "--range-client-write"},
+ description = "range of write operation of each client.",
+ defaultValue = "0")
+ private int writeRange;
+
+// @CommandLine.Option(names = {"-e", "--end-index-each-client-read"},
+// description = "end-index of each client's read operation.",
+// defaultValue = "0")
+// private int endIndexForRead;
+
+// @CommandLine.Option(names = {"-j", "--end-index-each-client-write"},
+// description = "end-index of each client's write operation.",
+// defaultValue = "0")
+// private int endIndexForWrite;
+
+ @CommandLine.Option(names = {"-g", "--size"},
+ description = "Generated data size (in bytes) of " +
+ "each key/file to be " +
+ "written.",
+ defaultValue = "256")
+ private int writeSizeInBytes;
+
+ @CommandLine.Option(names = {"-k", "--keySorted"},
+ description = "Generated sorted key or not. The key name " +
+ "will be generated via md5 hash if choose " +
+ "to use unsorted key.",
+ defaultValue = "false")
+ private boolean keySorted;
+
+ @CommandLine.Option(names = {"-x", "--mix-workload"},
+ description = "Set to True if you would like to " +
+ "generate mix workload (Read and Write).",
+ defaultValue = "false")
+ private boolean isMixWorkload;
+
+ @CommandLine.Option(names = {"--percentage-read"},
+ description = "Percentage of read tasks in mix workload.",
+ defaultValue = "0")
+ private int percentageRead;
+
+ @CommandLine.Option(names = {"--clients"},
+ description =
+ "Number of clients, defaults 1.",
+ defaultValue = "1")
+ private int clientsCount = 1;
+
+ @CommandLine.Option(
+ names = "--om-service-id",
+ description = "OM Service ID"
+ )
+ private String omServiceID = null;
+
+ private Timer timer;
+
+ private OzoneClient[] rpcClients;
+
+ private byte[] keyContent;
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneClientKeyReadWriteOps.class);
+
+ private final String readTask = "READ_TASK";
+ private final String writeTask = "WRITE_TASK";
+ private KeyGeneratorUtil kg;
+
+ @Override
+ public Void call() throws Exception {
+ init();
+ OzoneConfiguration ozoneConfiguration = createOzoneConfiguration();
+ rpcClients = new OzoneClient[clientsCount];
+ for (int i = 0; i < clientsCount; i++) {
+ rpcClients[i] = createOzoneClient(omServiceID, ozoneConfiguration);
+ }
+
+ ensureVolumeAndBucketExist(rpcClients[0], volumeName, bucketName);
+ timer = getMetrics().timer("key-read-write");
+ if (writeSizeInBytes >= 0) {
+ keyContent = RandomUtils.nextBytes(writeSizeInBytes);
+ }
+ if (kg == null) {
+ kg = new KeyGeneratorUtil();
+ }
+ runTests(this::readWriteKeys);
+
+ for (int i = 0; i < clientsCount; i++) {
+ if (rpcClients[i] != null) {
+ rpcClients[i].close();
+ }
+ }
+ return null;
+ }
+
+ public void readWriteKeys(long counter) throws Exception {
+ int clientIndex = (int)(counter % clientsCount);
+ OzoneClient client = rpcClients[clientIndex];
+ String operationType = decideReadOrWriteTask();
+ String keyName = getKeyName(operationType, clientIndex);
+ timer.time(() -> {
+ try {
+ switch (operationType) {
+ case readTask:
Review Comment:
Can we use Enum here?
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-m", "--read-metadata-only"},
+ description = "If only read key's metadata. " +
+ "Supported values are Y, F.",
+ defaultValue = "false")
+ private boolean readMetadataOnly;
+
+ @CommandLine.Option(names = {"-r", "--range-client-read"},
+ description = "range of read operation of each client.",
+ defaultValue = "0")
+ private int readRange;
+
+
+ @CommandLine.Option(names = {"-w", "--range-client-write"},
+ description = "range of write operation of each client.",
+ defaultValue = "0")
+ private int writeRange;
+
+// @CommandLine.Option(names = {"-e", "--end-index-each-client-read"},
Review Comment:
Please clean up the commented-out code.
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-m", "--read-metadata-only"},
+ description = "If only read key's metadata. " +
+ "Supported values are Y, F.",
+ defaultValue = "false")
+ private boolean readMetadataOnly;
+
+ @CommandLine.Option(names = {"-r", "--range-client-read"},
+ description = "range of read operation of each client.",
+ defaultValue = "0")
+ private int readRange;
+
+
+ @CommandLine.Option(names = {"-w", "--range-client-write"},
+ description = "range of write operation of each client.",
+ defaultValue = "0")
+ private int writeRange;
+
+// @CommandLine.Option(names = {"-e", "--end-index-each-client-read"},
+// description = "end-index of each client's read operation.",
+// defaultValue = "0")
+// private int endIndexForRead;
+
+// @CommandLine.Option(names = {"-j", "--end-index-each-client-write"},
+// description = "end-index of each client's write operation.",
+// defaultValue = "0")
+// private int endIndexForWrite;
+
+ @CommandLine.Option(names = {"-g", "--size"},
Review Comment:
nit: `-g` is not an intuitiative short-hand for `--size`. maybe it's better not to have a shortcut.
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-m", "--read-metadata-only"},
+ description = "If only read key's metadata. " +
+ "Supported values are Y, F.",
+ defaultValue = "false")
+ private boolean readMetadataOnly;
+
+ @CommandLine.Option(names = {"-r", "--range-client-read"},
+ description = "range of read operation of each client.",
+ defaultValue = "0")
+ private int readRange;
+
+
+ @CommandLine.Option(names = {"-w", "--range-client-write"},
+ description = "range of write operation of each client.",
+ defaultValue = "0")
+ private int writeRange;
+
+// @CommandLine.Option(names = {"-e", "--end-index-each-client-read"},
+// description = "end-index of each client's read operation.",
+// defaultValue = "0")
+// private int endIndexForRead;
+
+// @CommandLine.Option(names = {"-j", "--end-index-each-client-write"},
+// description = "end-index of each client's write operation.",
+// defaultValue = "0")
+// private int endIndexForWrite;
+
+ @CommandLine.Option(names = {"-g", "--size"},
+ description = "Generated data size (in bytes) of " +
+ "each key/file to be " +
+ "written.",
+ defaultValue = "256")
+ private int writeSizeInBytes;
+
+ @CommandLine.Option(names = {"-k", "--keySorted"},
+ description = "Generated sorted key or not. The key name " +
+ "will be generated via md5 hash if choose " +
+ "to use unsorted key.",
+ defaultValue = "false")
+ private boolean keySorted;
+
+ @CommandLine.Option(names = {"-x", "--mix-workload"},
+ description = "Set to True if you would like to " +
+ "generate mix workload (Read and Write).",
+ defaultValue = "false")
+ private boolean isMixWorkload;
+
+ @CommandLine.Option(names = {"--percentage-read"},
+ description = "Percentage of read tasks in mix workload.",
+ defaultValue = "0")
+ private int percentageRead;
+
+ @CommandLine.Option(names = {"--clients"},
+ description =
+ "Number of clients, defaults 1.",
+ defaultValue = "1")
+ private int clientsCount = 1;
+
+ @CommandLine.Option(
+ names = "--om-service-id",
+ description = "OM Service ID"
+ )
+ private String omServiceID = null;
+
+ private Timer timer;
+
+ private OzoneClient[] rpcClients;
+
+ private byte[] keyContent;
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneClientKeyReadWriteOps.class);
+
+ private final String readTask = "READ_TASK";
+ private final String writeTask = "WRITE_TASK";
+ private KeyGeneratorUtil kg;
+
+ @Override
+ public Void call() throws Exception {
+ init();
+ OzoneConfiguration ozoneConfiguration = createOzoneConfiguration();
+ rpcClients = new OzoneClient[clientsCount];
+ for (int i = 0; i < clientsCount; i++) {
+ rpcClients[i] = createOzoneClient(omServiceID, ozoneConfiguration);
+ }
+
+ ensureVolumeAndBucketExist(rpcClients[0], volumeName, bucketName);
+ timer = getMetrics().timer("key-read-write");
+ if (writeSizeInBytes >= 0) {
+ keyContent = RandomUtils.nextBytes(writeSizeInBytes);
+ }
+ if (kg == null) {
+ kg = new KeyGeneratorUtil();
+ }
+ runTests(this::readWriteKeys);
+
+ for (int i = 0; i < clientsCount; i++) {
+ if (rpcClients[i] != null) {
+ rpcClients[i].close();
+ }
+ }
+ return null;
+ }
+
+ public void readWriteKeys(long counter) throws Exception {
+ int clientIndex = (int)(counter % clientsCount);
+ OzoneClient client = rpcClients[clientIndex];
+ String operationType = decideReadOrWriteTask();
+ String keyName = getKeyName(operationType, clientIndex);
+ timer.time(() -> {
+ try {
+ switch (operationType) {
+ case readTask:
+ processReadTasks(keyName, client);
+ break;
+ case writeTask:
+ processWriteTasks(keyName, client);
+ break;
+ default:
+ break;
+ }
+ } catch (Exception ex) {
+ LOG.error(ex.getMessage());
+ }
+ });
+ }
+
+ public void processReadTasks(String keyName, OzoneClient client)
+ throws Exception {
+ OzoneBucket ozbk = client.getObjectStore().getVolume(volumeName)
+ .getBucket(bucketName);
+
+ if (readMetadataOnly) {
+ ozbk.getKey(keyName);
+ } else {
+ byte[] data = new byte[writeSizeInBytes];
+ try (OzoneInputStream introStream = ozbk.readKey(keyName)) {
+ introStream.read(data);
+ }
+ }
+ }
+ public void processWriteTasks(String keyName, OzoneClient client)
+ throws Exception {
+ OzoneBucket ozbk = client.getObjectStore().getVolume(volumeName)
Review Comment:
Similarly, you can use `client.getProxy().createKey()` directly instead of getting volume and bucket.
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/RangeKeysGenerator.java:
##########
@@ -0,0 +1,144 @@
+package org.apache.hadoop.ozone.freon;
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.util.concurrent.Callable;
+import java.util.function.Function;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.PURE_INDEX;
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.MD5;
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone range keys generator for performance test.
+ */
+@CommandLine.Command(name = "ork",
+ description = "write range keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class RangeKeysGenerator extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(RangeKeysGenerator.class);
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-r", "--range-each-client-write"},
+ description = "Write range for each client.",
+ defaultValue = "0")
+ private int range;
+
+ @CommandLine.Option(names = {"-k", "--key-encode"},
+ description = "The algorithm to generate key names. " +
+ "Options are pureIndex, md5, simpleHash",
+ defaultValue = "simpleHash")
+ private String encodeFormat;
Review Comment:
Maybe use Enum here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on a diff in pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on code in PR #3754:
URL: https://github.com/apache/ozone/pull/3754#discussion_r975898843
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-m", "--read-metadata-only"},
+ description = "If only read key's metadata. " +
+ "Supported values are Y, F.",
+ defaultValue = "false")
+ private boolean readMetadataOnly;
+
+ @CommandLine.Option(names = {"-r", "--range-client-read"},
+ description = "range of read operation of each client.",
+ defaultValue = "0")
+ private int readRange;
+
+
+ @CommandLine.Option(names = {"-w", "--range-client-write"},
+ description = "range of write operation of each client.",
+ defaultValue = "0")
+ private int writeRange;
+
+// @CommandLine.Option(names = {"-e", "--end-index-each-client-read"},
+// description = "end-index of each client's read operation.",
+// defaultValue = "0")
+// private int endIndexForRead;
+
+// @CommandLine.Option(names = {"-j", "--end-index-each-client-write"},
+// description = "end-index of each client's write operation.",
+// defaultValue = "0")
+// private int endIndexForWrite;
+
+ @CommandLine.Option(names = {"-g", "--size"},
+ description = "Generated data size (in bytes) of " +
+ "each key/file to be " +
+ "written.",
+ defaultValue = "256")
+ private int writeSizeInBytes;
+
+ @CommandLine.Option(names = {"-k", "--keySorted"},
+ description = "Generated sorted key or not. The key name " +
+ "will be generated via md5 hash if choose " +
+ "to use unsorted key.",
+ defaultValue = "false")
+ private boolean keySorted;
+
+ @CommandLine.Option(names = {"-x", "--mix-workload"},
+ description = "Set to True if you would like to " +
+ "generate mix workload (Read and Write).",
+ defaultValue = "false")
+ private boolean isMixWorkload;
+
+ @CommandLine.Option(names = {"--percentage-read"},
+ description = "Percentage of read tasks in mix workload.",
+ defaultValue = "0")
+ private int percentageRead;
+
+ @CommandLine.Option(names = {"--clients"},
+ description =
+ "Number of clients, defaults 1.",
+ defaultValue = "1")
+ private int clientsCount = 1;
+
+ @CommandLine.Option(
+ names = "--om-service-id",
+ description = "OM Service ID"
+ )
+ private String omServiceID = null;
+
+ private Timer timer;
+
+ private OzoneClient[] rpcClients;
+
+ private byte[] keyContent;
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneClientKeyReadWriteOps.class);
+
+ private final String readTask = "READ_TASK";
+ private final String writeTask = "WRITE_TASK";
+ private KeyGeneratorUtil kg;
+
+ @Override
+ public Void call() throws Exception {
+ init();
+ OzoneConfiguration ozoneConfiguration = createOzoneConfiguration();
+ rpcClients = new OzoneClient[clientsCount];
+ for (int i = 0; i < clientsCount; i++) {
+ rpcClients[i] = createOzoneClient(omServiceID, ozoneConfiguration);
+ }
+
+ ensureVolumeAndBucketExist(rpcClients[0], volumeName, bucketName);
+ timer = getMetrics().timer("key-read-write");
+ if (writeSizeInBytes >= 0) {
+ keyContent = RandomUtils.nextBytes(writeSizeInBytes);
+ }
+ if (kg == null) {
+ kg = new KeyGeneratorUtil();
+ }
+ runTests(this::readWriteKeys);
+
+ for (int i = 0; i < clientsCount; i++) {
+ if (rpcClients[i] != null) {
+ rpcClients[i].close();
+ }
+ }
+ return null;
+ }
+
+ public void readWriteKeys(long counter) throws Exception {
+ int clientIndex = (int)(counter % clientsCount);
+ OzoneClient client = rpcClients[clientIndex];
+ String operationType = decideReadOrWriteTask();
+ String keyName = getKeyName(operationType, clientIndex);
+ timer.time(() -> {
+ try {
+ switch (operationType) {
+ case readTask:
+ processReadTasks(keyName, client);
+ break;
+ case writeTask:
+ processWriteTasks(keyName, client);
+ break;
+ default:
+ break;
+ }
+ } catch (Exception ex) {
+ LOG.error(ex.getMessage());
+ }
+ });
+ }
+
+ public void processReadTasks(String keyName, OzoneClient client)
+ throws Exception {
+ OzoneBucket ozbk = client.getObjectStore().getVolume(volumeName)
+ .getBucket(bucketName);
+
+ if (readMetadataOnly) {
+ ozbk.getKey(keyName);
+ } else {
+ byte[] data = new byte[writeSizeInBytes];
+ try (OzoneInputStream introStream = ozbk.readKey(keyName)) {
+ introStream.read(data);
+ }
+ }
+ }
+ public void processWriteTasks(String keyName, OzoneClient client)
+ throws Exception {
+ OzoneBucket ozbk = client.getObjectStore().getVolume(volumeName)
+ .getBucket(bucketName);
+
+ try (OzoneOutputStream out = ozbk.createKey(keyName, writeSizeInBytes)) {
+ out.write(keyContent);
+ out.flush();
+ }
+ }
+ public String decideReadOrWriteTask() {
+ if (!isMixWorkload) {
+ if (readRange > 0) {
+ return readTask;
+ } else if (writeRange > 0) {
+ return writeTask;
+ }
+ }
+ //mix workload
+ int tmp = ThreadLocalRandom.current().nextInt(100) + 1; // 1 ~ 100
+ if (tmp < percentageRead) {
+ return readTask;
+ } else {
+ return writeTask;
+ }
+ }
+
+ public String getKeyName(String operationType, int counter) {
+ int startIdx, endIdx;
+ switch (operationType) {
+ case readTask:
+ startIdx = counter * readRange;
+ endIdx = startIdx + readRange;
+ break;
+ case writeTask:
+ startIdx = counter * writeRange;
+ endIdx = startIdx + writeRange;
+ break;
+ default:
+ startIdx = 0;
+ endIdx = 0;
+ break;
+ }
+ StringBuilder keyNameSb = new StringBuilder();
+ int randomIdxWithinRange = ThreadLocalRandom.current().
+ nextInt(endIdx + 1 - startIdx) + startIdx;
+ if (keySorted) {
+ keyNameSb.append(getPrefix()).append(FILE_DIR_SEPARATOR).
+ append(randomIdxWithinRange);
+ } else {
+ keyNameSb.append(getPrefix()).append(FILE_DIR_SEPARATOR).
Review Comment:
Yes! So default behavior is to map the index of key (from 0 to 1M, for example.) to their md5 string, so that the key is not in sorted order in rocksDB. But I left an option here to generate the key name as the index value, so that we could generate sorted key. I'm not sure if we'll test sorted key, I could remove that part if not required! Let me know!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on a diff in pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on code in PR #3754:
URL: https://github.com/apache/ozone/pull/3754#discussion_r975896800
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-m", "--read-metadata-only"},
+ description = "If only read key's metadata. " +
+ "Supported values are Y, F.",
+ defaultValue = "false")
+ private boolean readMetadataOnly;
+
+ @CommandLine.Option(names = {"-r", "--range-client-read"},
+ description = "range of read operation of each client.",
+ defaultValue = "0")
+ private int readRange;
+
+
+ @CommandLine.Option(names = {"-w", "--range-client-write"},
+ description = "range of write operation of each client.",
+ defaultValue = "0")
+ private int writeRange;
+
+// @CommandLine.Option(names = {"-e", "--end-index-each-client-read"},
+// description = "end-index of each client's read operation.",
+// defaultValue = "0")
+// private int endIndexForRead;
+
+// @CommandLine.Option(names = {"-j", "--end-index-each-client-write"},
+// description = "end-index of each client's write operation.",
+// defaultValue = "0")
+// private int endIndexForWrite;
+
+ @CommandLine.Option(names = {"-g", "--size"},
+ description = "Generated data size (in bytes) of " +
+ "each key/file to be " +
+ "written.",
+ defaultValue = "256")
+ private int writeSizeInBytes;
+
+ @CommandLine.Option(names = {"-k", "--keySorted"},
+ description = "Generated sorted key or not. The key name " +
+ "will be generated via md5 hash if choose " +
+ "to use unsorted key.",
+ defaultValue = "false")
+ private boolean keySorted;
+
+ @CommandLine.Option(names = {"-x", "--mix-workload"},
+ description = "Set to True if you would like to " +
+ "generate mix workload (Read and Write).",
+ defaultValue = "false")
+ private boolean isMixWorkload;
Review Comment:
yeah, make sense! Let me change this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on a diff in pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on code in PR #3754:
URL: https://github.com/apache/ozone/pull/3754#discussion_r975899571
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/RangeKeysGenerator.java:
##########
@@ -0,0 +1,144 @@
+package org.apache.hadoop.ozone.freon;
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.util.concurrent.Callable;
+import java.util.function.Function;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.PURE_INDEX;
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.MD5;
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone range keys generator for performance test.
+ */
+@CommandLine.Command(name = "ork",
+ description = "write range keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class RangeKeysGenerator extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(RangeKeysGenerator.class);
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-r", "--range-each-client-write"},
+ description = "Write range for each client.",
+ defaultValue = "0")
+ private int range;
+
+ @CommandLine.Option(names = {"-k", "--key-encode"},
+ description = "The algorithm to generate key names. " +
+ "Options are pureIndex, md5, simpleHash",
+ defaultValue = "simpleHash")
+ private String encodeFormat;
Review Comment:
Sure!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] duongkame commented on a diff in pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
duongkame commented on code in PR #3754:
URL: https://github.com/apache/ozone/pull/3754#discussion_r978058370
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/SecurityConfig.java:
##########
@@ -141,12 +141,15 @@ public SecurityConfig(ConfigurationSource configuration) {
this.certificateFileName = this.configuration.get(HDDS_X509_FILE_NAME,
HDDS_X509_FILE_NAME_DEFAULT);
- this.blockTokenEnabled = this.configuration.getBoolean(
- HDDS_BLOCK_TOKEN_ENABLED,
- HDDS_BLOCK_TOKEN_ENABLED_DEFAULT);
- this.containerTokenEnabled = this.configuration.getBoolean(
- HDDS_CONTAINER_TOKEN_ENABLED,
- HDDS_CONTAINER_TOKEN_ENABLED_DEFAULT);
+// this.blockTokenEnabled = this.configuration.getBoolean(
+// HDDS_BLOCK_TOKEN_ENABLED,
+// HDDS_BLOCK_TOKEN_ENABLED_DEFAULT);
+ this.blockTokenEnabled = false;
Review Comment:
I don't think those changes are supposed to get in.
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java:
##########
@@ -493,8 +493,9 @@ private OzoneManager(OzoneConfiguration conf, StartupOption startupOption)
this.preallocateBlocksMax = conf.getInt(
OZONE_KEY_PREALLOCATION_BLOCKS_MAX,
OZONE_KEY_PREALLOCATION_BLOCKS_MAX_DEFAULT);
- this.grpcBlockTokenEnabled = conf.getBoolean(HDDS_BLOCK_TOKEN_ENABLED,
- HDDS_BLOCK_TOKEN_ENABLED_DEFAULT);
+// this.grpcBlockTokenEnabled = conf.getBoolean(HDDS_BLOCK_TOKEN_ENABLED,
+// HDDS_BLOCK_TOKEN_ENABLED_DEFAULT);
+ this.grpcBlockTokenEnabled = false;
Review Comment:
Please ensure this doesn't get in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on PR #3754:
URL: https://github.com/apache/ozone/pull/3754#issuecomment-1260281499
>
Yes! The command will potentially read some key which doesn't exist in the cluster.
Currently the freon command would stop running & report failure, but let me think more about this how to make it better!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on a diff in pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on code in PR #3754:
URL: https://github.com/apache/ozone/pull/3754#discussion_r991692037
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-m", "--read-metadata-only"},
+ description = "If only read key's metadata. " +
+ "Supported values are Y, F.",
+ defaultValue = "false")
+ private boolean readMetadataOnly;
+ @CommandLine.Option(names = {"-s", "--start-index"},
+ description = "Start index of keys of read/write operation.",
+ defaultValue = "0")
+ private int startIndex;
+
+ @CommandLine.Option(names = {"-r", "--range"},
+ description = "index range of read/write operations.",
+ defaultValue = "0")
+ private int range;
+ @CommandLine.Option(names = {"--size"},
+ description = "Object size (in bytes) " +
+ "to be generated.",
+ defaultValue = "1")
+ private int objectSizeInBytes;
+
+ @CommandLine.Option(names = {"--keySorted"},
+ description = "Generated sorted key or not. The key name " +
+ "will be generated via md5 hash if choose " +
+ "to use unsorted key.",
+ defaultValue = "false")
+ private boolean keySorted;
+
+ @CommandLine.Option(names = {"--percentage-read"},
+ description = "Percentage of read tasks in mix workload.",
+ defaultValue = "100")
+ private int percentageRead;
+
+ @CommandLine.Option(names = {"--clients"},
+ description =
+ "Number of clients, defaults 1.",
Review Comment:
updated!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on a diff in pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on code in PR #3754:
URL: https://github.com/apache/ozone/pull/3754#discussion_r991688407
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
Review Comment:
updated!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on a diff in pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on code in PR #3754:
URL: https://github.com/apache/ozone/pull/3754#discussion_r991687553
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
Review Comment:
Updated!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
kerneltime commented on PR #3754:
URL: https://github.com/apache/ozone/pull/3754#issuecomment-1252973967
cc @duongkame
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] duongkame commented on a diff in pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
duongkame commented on code in PR #3754:
URL: https://github.com/apache/ozone/pull/3754#discussion_r975875110
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-m", "--read-metadata-only"},
+ description = "If only read key's metadata. " +
+ "Supported values are Y, F.",
+ defaultValue = "false")
+ private boolean readMetadataOnly;
+
+ @CommandLine.Option(names = {"-r", "--range-client-read"},
+ description = "range of read operation of each client.",
+ defaultValue = "0")
+ private int readRange;
+
+
+ @CommandLine.Option(names = {"-w", "--range-client-write"},
+ description = "range of write operation of each client.",
+ defaultValue = "0")
+ private int writeRange;
+
+// @CommandLine.Option(names = {"-e", "--end-index-each-client-read"},
+// description = "end-index of each client's read operation.",
+// defaultValue = "0")
+// private int endIndexForRead;
+
+// @CommandLine.Option(names = {"-j", "--end-index-each-client-write"},
+// description = "end-index of each client's write operation.",
+// defaultValue = "0")
+// private int endIndexForWrite;
+
+ @CommandLine.Option(names = {"-g", "--size"},
+ description = "Generated data size (in bytes) of " +
+ "each key/file to be " +
+ "written.",
+ defaultValue = "256")
+ private int writeSizeInBytes;
+
+ @CommandLine.Option(names = {"-k", "--keySorted"},
+ description = "Generated sorted key or not. The key name " +
+ "will be generated via md5 hash if choose " +
+ "to use unsorted key.",
+ defaultValue = "false")
+ private boolean keySorted;
+
+ @CommandLine.Option(names = {"-x", "--mix-workload"},
+ description = "Set to True if you would like to " +
+ "generate mix workload (Read and Write).",
+ defaultValue = "false")
+ private boolean isMixWorkload;
+
+ @CommandLine.Option(names = {"--percentage-read"},
+ description = "Percentage of read tasks in mix workload.",
+ defaultValue = "0")
+ private int percentageRead;
+
+ @CommandLine.Option(names = {"--clients"},
+ description =
+ "Number of clients, defaults 1.",
+ defaultValue = "1")
+ private int clientsCount = 1;
+
+ @CommandLine.Option(
+ names = "--om-service-id",
+ description = "OM Service ID"
+ )
+ private String omServiceID = null;
+
+ private Timer timer;
+
+ private OzoneClient[] rpcClients;
+
+ private byte[] keyContent;
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneClientKeyReadWriteOps.class);
+
+ private final String readTask = "READ_TASK";
+ private final String writeTask = "WRITE_TASK";
+ private KeyGeneratorUtil kg;
+
+ @Override
+ public Void call() throws Exception {
+ init();
+ OzoneConfiguration ozoneConfiguration = createOzoneConfiguration();
+ rpcClients = new OzoneClient[clientsCount];
+ for (int i = 0; i < clientsCount; i++) {
+ rpcClients[i] = createOzoneClient(omServiceID, ozoneConfiguration);
+ }
+
+ ensureVolumeAndBucketExist(rpcClients[0], volumeName, bucketName);
+ timer = getMetrics().timer("key-read-write");
+ if (writeSizeInBytes >= 0) {
+ keyContent = RandomUtils.nextBytes(writeSizeInBytes);
+ }
+ if (kg == null) {
+ kg = new KeyGeneratorUtil();
+ }
+ runTests(this::readWriteKeys);
+
+ for (int i = 0; i < clientsCount; i++) {
+ if (rpcClients[i] != null) {
+ rpcClients[i].close();
+ }
+ }
+ return null;
+ }
+
+ public void readWriteKeys(long counter) throws Exception {
+ int clientIndex = (int)(counter % clientsCount);
+ OzoneClient client = rpcClients[clientIndex];
+ String operationType = decideReadOrWriteTask();
+ String keyName = getKeyName(operationType, clientIndex);
+ timer.time(() -> {
+ try {
+ switch (operationType) {
+ case readTask:
+ processReadTasks(keyName, client);
+ break;
+ case writeTask:
+ processWriteTasks(keyName, client);
+ break;
+ default:
+ break;
+ }
+ } catch (Exception ex) {
+ LOG.error(ex.getMessage());
+ }
+ });
+ }
+
+ public void processReadTasks(String keyName, OzoneClient client)
+ throws Exception {
+ OzoneBucket ozbk = client.getObjectStore().getVolume(volumeName)
Review Comment:
Should this be reused instead of re-read every request? it creates 2 more additional requests (getVolume and getBucket) to OM per lookupKey.
In fact, you don't need to read the bucket, use:
```
OzoneKeyDetails keyDetails = client.getProxy().getKeyDetails(volumeName, bucketName, keyName);
if (!readMetadataOnly) {
byte[] data = new byte[writeSizeInBytes];
try (OzoneInputStream introStream = keyDetails.getContent()) {
introStream.read(data);
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on a diff in pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
kerneltime commented on code in PR #3754:
URL: https://github.com/apache/ozone/pull/3754#discussion_r981685042
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-m", "--read-metadata-only"},
+ description = "If only read key's metadata. " +
+ "Supported values are Y, F.",
+ defaultValue = "false")
+ private boolean readMetadataOnly;
+ @CommandLine.Option(names = {"-s", "--start-index"},
+ description = "Start index of keys of read/write operation.",
Review Comment:
```suggestion
description = "Start index of keys of read/write operation. This can allow adding keys incrementally or parallel from multiple clients. Example: Write keys 0-1000000 followed by keys 1000001-2000000.",
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on a diff in pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on code in PR #3754:
URL: https://github.com/apache/ozone/pull/3754#discussion_r981855726
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,247 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.*;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-m", "--read-metadata-only"},
+ description = "If only read key's metadata. " +
+ "Supported values are Y, F.",
+ defaultValue = "false")
+ private boolean readMetadataOnly;
+ @CommandLine.Option(names = {"-s", "--start-index"},
+ description = "Start index of keys of read/write operation.",
+ defaultValue = "0")
+ private int startIndex;
+
+ @CommandLine.Option(names = {"-r", "--range"},
+ description = "index range of read/write operations.",
+ defaultValue = "0")
+ private int range;
+ @CommandLine.Option(names = {"--size"},
+ description = "Object size (in bytes) " +
+ "to be generated.",
+ defaultValue = "1")
+ private int objectSizeInBytes;
+
+ @CommandLine.Option(names = {"--keySorted"},
+ description = "Generated sorted key or not. The key name " +
+ "will be generated via md5 hash if choose " +
+ "to use unsorted key.",
+ defaultValue = "false")
+ private boolean keySorted;
+
+ @CommandLine.Option(names = {"--percentage-read"},
+ description = "Percentage of read tasks in mix workload.",
+ defaultValue = "100")
+ private int percentageRead;
+
+ @CommandLine.Option(names = {"--clients"},
+ description =
+ "Number of clients, defaults 1.",
+ defaultValue = "1")
+ private int clientsCount = 1;
+
+ @CommandLine.Option(
+ names = "--om-service-id",
+ description = "OM Service ID"
+ )
+ private String omServiceID = null;
+
+ private Timer timer;
+
+ private OzoneBucket[] ozoneBuckets;
+
+ private byte[] keyContent;
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OzoneClientKeyReadWriteOps.class);
+ public enum TaskType {
+ READ_TASK,
+ WRITE_TASK
+ }
+ private KeyGeneratorUtil kg;
+
+
+ @Override
+ public Void call() throws Exception {
+ init();
+ OzoneConfiguration ozoneConfiguration = createOzoneConfiguration();
+ OzoneClient[] ozoneClients = new OzoneClient[clientsCount];
+ for (int i = 0; i < clientsCount; i++) {
+ ozoneClients[i] = createOzoneClient(omServiceID, ozoneConfiguration);
+ }
+
+ ensureVolumeAndBucketExist(ozoneClients[0], volumeName, bucketName);
+ ozoneBuckets = new OzoneBucket[clientsCount];
+ for (int i = 0; i < clientsCount; i++) {
+ ozoneBuckets[i] = ozoneClients[i].getObjectStore().getVolume(volumeName)
+ .getBucket(bucketName);
+ }
+
+ timer = getMetrics().timer("key-read-write");
+ if (objectSizeInBytes >= 0) {
+ keyContent = RandomUtils.nextBytes(objectSizeInBytes);
+ }
+ if (kg == null) {
+ kg = new KeyGeneratorUtil();
+ }
+ runTests(this::readWriteKeys);
+
+ for (int i = 0; i < clientsCount; i++) {
+ if (ozoneClients[i] != null) {
+ ozoneClients[i].close();
+ }
+ }
+ return null;
+ }
+
+ public void readWriteKeys(long counter) throws Exception {
+ int clientIndex = (int)((counter) % clientsCount);
+ OzoneBucket ozoneBucket = ozoneBuckets[clientIndex];
+ TaskType taskType = decideReadOrWriteTask();
+ String keyName = getKeyName(taskType, clientIndex);
+
+ timer.time(() -> {
+ try {
+ switch (taskType) {
+ case READ_TASK:
+ processReadTasks(keyName, ozoneBucket);
+ break;
+ case WRITE_TASK:
+ processWriteTasks(keyName, ozoneBucket);
+ break;
+ default:
+ break;
+ }
+ } catch (RuntimeException ex) {
+ LOG.error(ex.getMessage());
+ throw ex;
+ } catch (IOException ex) {
+ LOG.error(ex.getMessage());
+ throw new RuntimeException(ex.getMessage());
+ }
+
+ });
+ }
+
+ public void processReadTasks(String keyName, OzoneBucket ozoneBucket)
+ throws RuntimeException, IOException {
+ if (readMetadataOnly) {
+ ozoneBucket.getKey(keyName);
+ } else {
+ byte[] data = new byte[objectSizeInBytes];
+ try (OzoneInputStream inputStream = ozoneBucket.readKey(keyName)) {
+ inputStream.read(data);
Review Comment:
thanks!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on a diff in pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
kerneltime commented on code in PR #3754:
URL: https://github.com/apache/ozone/pull/3754#discussion_r981720285
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-m", "--read-metadata-only"},
+ description = "If only read key's metadata. " +
+ "Supported values are Y, F.",
+ defaultValue = "false")
+ private boolean readMetadataOnly;
+ @CommandLine.Option(names = {"-s", "--start-index"},
+ description = "Start index of keys of read/write operation.",
+ defaultValue = "0")
+ private int startIndex;
+
+ @CommandLine.Option(names = {"-r", "--range"},
+ description = "index range of read/write operations.",
+ defaultValue = "0")
+ private int range;
+ @CommandLine.Option(names = {"--size"},
+ description = "Object size (in bytes) " +
+ "to be generated.",
+ defaultValue = "1")
+ private int objectSizeInBytes;
+
+ @CommandLine.Option(names = {"--keySorted"},
+ description = "Generated sorted key or not. The key name " +
+ "will be generated via md5 hash if choose " +
+ "to use unsorted key.",
+ defaultValue = "false")
+ private boolean keySorted;
+
+ @CommandLine.Option(names = {"--percentage-read"},
+ description = "Percentage of read tasks in mix workload.",
Review Comment:
The remainder of the percentage will writes to keys. Example --percentage-read 90 will result in 10% writes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on a diff in pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
kerneltime commented on code in PR #3754:
URL: https://github.com/apache/ozone/pull/3754#discussion_r981716096
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-m", "--read-metadata-only"},
+ description = "If only read key's metadata. " +
+ "Supported values are Y, F.",
+ defaultValue = "false")
+ private boolean readMetadataOnly;
+ @CommandLine.Option(names = {"-s", "--start-index"},
+ description = "Start index of keys of read/write operation.",
+ defaultValue = "0")
+ private int startIndex;
+
+ @CommandLine.Option(names = {"-r", "--range"},
+ description = "index range of read/write operations.",
Review Comment:
```suggestion
description = "Range of read/write operations. This in co-ordination with --start-index can specify the range to read. Example: Readom from --start-index 1000 and read --range 1000 keys",
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on a diff in pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on code in PR #3754:
URL: https://github.com/apache/ozone/pull/3754#discussion_r991691245
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-m", "--read-metadata-only"},
+ description = "If only read key's metadata. " +
+ "Supported values are Y, F.",
+ defaultValue = "false")
+ private boolean readMetadataOnly;
+ @CommandLine.Option(names = {"-s", "--start-index"},
+ description = "Start index of keys of read/write operation.",
+ defaultValue = "0")
+ private int startIndex;
+
+ @CommandLine.Option(names = {"-r", "--range"},
+ description = "index range of read/write operations.",
+ defaultValue = "0")
+ private int range;
+ @CommandLine.Option(names = {"--size"},
+ description = "Object size (in bytes) " +
+ "to be generated.",
+ defaultValue = "1")
+ private int objectSizeInBytes;
+
+ @CommandLine.Option(names = {"--keySorted"},
+ description = "Generated sorted key or not. The key name " +
+ "will be generated via md5 hash if choose " +
+ "to use unsorted key.",
+ defaultValue = "false")
Review Comment:
Updated!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on a diff in pull request #3754: HDDS-7199. Implement new mix workload Read/Write Freon command which meets specific test requirements
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on code in PR #3754:
URL: https://github.com/apache/ozone/pull/3754#discussion_r991689798
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyReadWriteOps.java:
##########
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+
+import com.codahale.metrics.Timer;
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.client.OzoneBucket;
+import org.apache.hadoop.ozone.client.OzoneClient;
+import org.apache.hadoop.ozone.client.io.OzoneInputStream;
+import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.hadoop.ozone.freon.KeyGeneratorUtil.FILE_DIR_SEPARATOR;
+
+/**
+ * Ozone key generator/reader for performance test.
+ */
+
+@CommandLine.Command(name = "ockrw",
+ aliases = "ozone-client-key-read-write-ops",
+ description = "Read and write keys with the help of the ozone clients.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class OzoneClientKeyReadWriteOps extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ @CommandLine.Option(names = {"-v", "--volume"},
+ description = "Name of the volume which contains the test data. " +
+ "Will be created if missing.",
+ defaultValue = "vol1")
+ private String volumeName;
+
+ @CommandLine.Option(names = {"-b", "--bucket"},
+ description = "Name of the bucket which contains the test data.",
+ defaultValue = "bucket1")
+ private String bucketName;
+
+ @CommandLine.Option(names = {"-m", "--read-metadata-only"},
+ description = "If only read key's metadata. " +
+ "Supported values are Y, F.",
+ defaultValue = "false")
+ private boolean readMetadataOnly;
+ @CommandLine.Option(names = {"-s", "--start-index"},
+ description = "Start index of keys of read/write operation.",
+ defaultValue = "0")
+ private int startIndex;
+
+ @CommandLine.Option(names = {"-r", "--range"},
+ description = "index range of read/write operations.",
Review Comment:
updated!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org