You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/01/12 13:32:51 UTC
[GitHub] [hadoop-ozone] mukul1987 opened a new pull request #438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load generator…
mukul1987 opened a new pull request #438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load generator…
URL: https://github.com/apache/hadoop-ozone/pull/438
…s to chaos testing.
## What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
## What is the link to the Apache JIRA
(Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HDDS-XXXX. Fix a typo in YYY.)
Please replace this section with the link to the Apache JIRA)
## How was this patch tested?
(Please explain how this patch was tested. Ex: unit tests, manual tests)
(If this patch involves UI changes, please attach a screen-shot; otherwise, remove this)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] nandakumar131 commented on a change in pull request
#438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load
generators to chaos testing.
Posted by GitBox <gi...@apache.org>.
nandakumar131 commented on a change in pull request #438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load generators to chaos testing.
URL: https://github.com/apache/hadoop-ozone/pull/438#discussion_r373041258
##########
File path: hadoop-ozone/fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/loadgenerators/AgedLoadGenerator.java
##########
@@ -0,0 +1,110 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.loadgenerators;
+
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.ozone.MiniOzoneLoadGenerator;
+import org.apache.hadoop.ozone.utils.LoadBucket;
+import org.apache.hadoop.ozone.utils.TestProbability;
+import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.util.Time;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.util.Optional;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * Aged Load Generator for Ozone.
+ *
+ * This Load Generator reads and write key to an Ozone bucket.
+ *
+ * The defautl writes to read ratio is 10:90.
+ */
+public class AgedLoadGenerator implements LoadGenerator {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(AgedLoadGenerator.class);
+ private static String fileSuffex = "aged";
Review comment:
Typo: Suffex
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on issue #438: HDDS-2878. Refactor
MiniOzoneLoadGenerator to add more load generators to chaos testing.
Posted by GitBox <gi...@apache.org>.
elek commented on issue #438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load generators to chaos testing.
URL: https://github.com/apache/hadoop-ozone/pull/438#issuecomment-587378501
/pending Pending on author to address the comments
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request
#438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load
generators to chaos testing.
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load generators to chaos testing.
URL: https://github.com/apache/hadoop-ozone/pull/438#discussion_r394933743
##########
File path: hadoop-ozone/fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/loadgenerators/AgedLoadGenerator.java
##########
@@ -0,0 +1,110 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.loadgenerators;
+
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.ozone.MiniOzoneLoadGenerator;
+import org.apache.hadoop.ozone.utils.LoadBucket;
+import org.apache.hadoop.ozone.utils.TestProbability;
+import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.util.Time;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.util.Optional;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * Aged Load Generator for Ozone.
+ *
+ * This Load Generator reads and write key to an Ozone bucket.
+ *
+ * The default writes to read ratio is 10:90.
+ */
+public class AgedLoadGenerator implements LoadGenerator {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(AgedLoadGenerator.class);
+ private static String agedSuffix = "aged";
+
+ private final AtomicInteger agedFileWrittenIndex;
+ private final AtomicInteger agedFileAllocationIndex;
+ private final LoadBucket agedLoadBucket;
+ private final TestProbability agedWriteProbability;
+ private final DataBuffer dataBuffer;
+
+ public AgedLoadGenerator(DataBuffer data, LoadBucket agedLoadBucket) {
+ this.dataBuffer = data;
+ this.agedFileWrittenIndex = new AtomicInteger(0);
+ this.agedFileAllocationIndex = new AtomicInteger(0);
+ this.agedLoadBucket = agedLoadBucket;
+ this.agedWriteProbability = TestProbability.valueOf(10);
+ }
+
+ public void startLoad(long runTimeMillis) {
+ long threadID = Thread.currentThread().getId();
+ LOG.info("AGED LOADGEN: Started Aged IO Thread:{}.", threadID);
+ long startTime = Time.monotonicNow();
+
+ while (Time.monotonicNow() < startTime + runTimeMillis) {
Review comment:
`System#nanoTime` says timestamps should be compared in the form: `t1 - t0 < ...`, so I think this should be:
```suggestion
while (Time.monotonicNow() - startTime < runTimeMillis) {
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] nandakumar131 commented on a change in pull request
#438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load
generators to chaos testing.
Posted by GitBox <gi...@apache.org>.
nandakumar131 commented on a change in pull request #438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load generators to chaos testing.
URL: https://github.com/apache/hadoop-ozone/pull/438#discussion_r373057922
##########
File path: hadoop-ozone/fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/MiniOzoneLoadGenerator.java
##########
@@ -47,212 +42,61 @@
LoggerFactory.getLogger(MiniOzoneLoadGenerator.class);
private static String keyNameDelimiter = "_";
-
- private ThreadPoolExecutor writeExecutor;
- private int numThreads;
- // number of buffer to be allocated, each is allocated with length which
- // is multiple of 2, each buffer is populated with random data.
- private int numBuffers;
- private List<ByteBuffer> buffers;
-
- private AtomicBoolean isIOThreadRunning;
-
- private final List<LoadBucket> ozoneBuckets;
-
- private final AtomicInteger agedFileWrittenIndex;
- private final ExecutorService agedFileExecutor;
- private final LoadBucket agedLoadBucket;
- private final TestProbability agedWriteProbability;
-
- private final ThreadPoolExecutor fsExecutor;
- private final LoadBucket fsBucket;
-
- MiniOzoneLoadGenerator(List<LoadBucket> bucket,
- LoadBucket agedLoadBucket, LoadBucket fsBucket,
- int numThreads, int numBuffers) {
- this.ozoneBuckets = bucket;
- this.numThreads = numThreads;
- this.numBuffers = numBuffers;
- this.writeExecutor = createExecutor();
-
- this.agedFileWrittenIndex = new AtomicInteger(0);
- this.agedFileExecutor = Executors.newSingleThreadExecutor();
- this.agedLoadBucket = agedLoadBucket;
- this.agedWriteProbability = TestProbability.valueOf(10);
-
- this.fsExecutor = createExecutor();
- this.fsBucket = fsBucket;
-
- this.isIOThreadRunning = new AtomicBoolean(false);
-
- // allocate buffers and populate random data.
- buffers = new ArrayList<>();
- for (int i = 0; i < numBuffers; i++) {
- int size = (int) StorageUnit.KB.toBytes(1 << i);
- ByteBuffer buffer = ByteBuffer.allocate(size);
- buffer.put(RandomUtils.nextBytes(size));
- buffers.add(buffer);
- }
- }
-
- private ThreadPoolExecutor createExecutor() {
- ThreadPoolExecutor executor = new ThreadPoolExecutor(numThreads, numThreads,
- 100, TimeUnit.SECONDS, new ArrayBlockingQueue<>(1024),
- new ThreadPoolExecutor.CallerRunsPolicy());
- executor.prestartAllCoreThreads();
- return executor;
-
- }
-
- // Start IO load on an Ozone bucket.
- private void load(long runTimeMillis) {
- long threadID = Thread.currentThread().getId();
- LOG.info("Started Mixed IO Thread:{}.", threadID);
- String threadName = Thread.currentThread().getName();
- long startTime = Time.monotonicNow();
-
- while (isIOThreadRunning.get() &&
- (Time.monotonicNow() < startTime + runTimeMillis)) {
- LoadBucket bucket =
- ozoneBuckets.get((int) (Math.random() * ozoneBuckets.size()));
- try {
- int index = RandomUtils.nextInt();
- ByteBuffer buffer = getBuffer(index);
- String keyName = getKeyName(index, threadName);
- bucket.writeKey(buffer, keyName);
-
- bucket.readKey(buffer, keyName);
-
- bucket.deleteKey(keyName);
- } catch (Exception e) {
- LOG.error("LOADGEN: Exiting due to exception", e);
- break;
- }
- }
- // This will terminate other threads too.
- isIOThreadRunning.set(false);
- LOG.info("Terminating IO thread:{}.", threadID);
- }
-
- private Optional<Integer> randomKeyToRead() {
- int currentIndex = agedFileWrittenIndex.get();
- return currentIndex != 0
- ? Optional.of(RandomUtils.nextInt(0, currentIndex))
- : Optional.empty();
- }
-
- private void startAgedLoad(long runTimeMillis) {
- long threadID = Thread.currentThread().getId();
- LOG.info("AGED LOADGEN: Started Aged IO Thread:{}.", threadID);
- String threadName = Thread.currentThread().getName();
- long startTime = Time.monotonicNow();
-
- while (isIOThreadRunning.get() &&
- (Time.monotonicNow() < startTime + runTimeMillis)) {
-
- String keyName = null;
- try {
- if (agedWriteProbability.isTrue()) {
- int index = agedFileWrittenIndex.getAndIncrement();
- ByteBuffer buffer = getBuffer(index);
- keyName = getKeyName(index, threadName);
-
- agedLoadBucket.writeKey(buffer, keyName);
- } else {
- Optional<Integer> index = randomKeyToRead();
- if (index.isPresent()) {
- ByteBuffer buffer = getBuffer(index.get());
- keyName = getKeyName(index.get(), threadName);
- agedLoadBucket.readKey(buffer, keyName);
- }
- }
- } catch (Throwable t) {
- LOG.error("AGED LOADGEN: {} Exiting due to exception", keyName, t);
- break;
- }
- }
- // This will terminate other threads too.
- isIOThreadRunning.set(false);
- LOG.info("Terminating IO thread:{}.", threadID);
- }
-
- // Start IO load on an Ozone bucket.
- private void startFsLoad(long runTimeMillis) {
- long threadID = Thread.currentThread().getId();
- LOG.info("Started Filesystem IO Thread:{}.", threadID);
- String threadName = Thread.currentThread().getName();
- long startTime = Time.monotonicNow();
-
- while (isIOThreadRunning.get() &&
- (Time.monotonicNow() < startTime + runTimeMillis)) {
- try {
- int index = RandomUtils.nextInt();
- ByteBuffer buffer = getBuffer(index);
- String keyName = getKeyName(index, threadName);
- fsBucket.writeKey(true, buffer, keyName);
-
- fsBucket.readKey(true, buffer, keyName);
-
- fsBucket.deleteKey(true, keyName);
- } catch (Exception e) {
- LOG.error("LOADGEN: Exiting due to exception", e);
- break;
- }
+ private final List<LoadExecutors> loadExecutors;
+
Review comment:
The `numThreads` option given is used by all the generators separately. So if the `numThreads` value is 5, we will be ideally starting
5 - `RandomLoadGenerator` Threads
5 - `AgedLoadGenerator` Threads
5 - `FilesystemLoadGenerator` Threads
Total of 15 threads, everything parallel?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] mukul1987 commented on a change in pull request
#438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load
generators to chaos testing.
Posted by GitBox <gi...@apache.org>.
mukul1987 commented on a change in pull request #438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load generators to chaos testing.
URL: https://github.com/apache/hadoop-ozone/pull/438#discussion_r392697327
##########
File path: hadoop-ozone/fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/loadgenerators/AgedLoadGenerator.java
##########
@@ -0,0 +1,110 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.loadgenerators;
+
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.ozone.MiniOzoneLoadGenerator;
+import org.apache.hadoop.ozone.utils.LoadBucket;
+import org.apache.hadoop.ozone.utils.TestProbability;
+import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.util.Time;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.util.Optional;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * Aged Load Generator for Ozone.
+ *
+ * This Load Generator reads and write key to an Ozone bucket.
+ *
+ * The defautl writes to read ratio is 10:90.
+ */
+public class AgedLoadGenerator implements LoadGenerator {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(AgedLoadGenerator.class);
+ private static String fileSuffex = "aged";
+
+ private final AtomicInteger agedFileWrittenIndex;
+ private final AtomicInteger agedFileAllocationIndex;
+ private final LoadBucket agedLoadBucket;
+ private final TestProbability agedWriteProbability;
+ private final DataBuffer dataBuffer;
+
+ public AgedLoadGenerator(DataBuffer data, LoadBucket agedLoadBucket) {
+ this.dataBuffer = data;
+ this.agedFileWrittenIndex = new AtomicInteger(0);
+ this.agedFileAllocationIndex = new AtomicInteger(0);
+ this.agedLoadBucket = agedLoadBucket;
+ this.agedWriteProbability = TestProbability.valueOf(10);
+ }
+
+ public void startLoad(long runTimeMillis) {
+ long threadID = Thread.currentThread().getId();
+ LOG.info("AGED LOADGEN: Started Aged IO Thread:{}.", threadID);
+ long startTime = Time.monotonicNow();
+
+ while (Time.monotonicNow() < startTime + runTimeMillis) {
+
+ String keyName = null;
+ try {
+ if (agedWriteProbability.isTrue()) {
+ synchronized (agedFileAllocationIndex) {
Review comment:
This agedFileWrittenIndex can increment for lets say file-4 before file-1 and that causes the read to fail as the file can not been written to already.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request
#438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load
generators to chaos testing.
Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on a change in pull request #438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load generators to chaos testing.
URL: https://github.com/apache/hadoop-ozone/pull/438#discussion_r394471359
##########
File path: hadoop-ozone/fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/MiniOzoneLoadGenerator.java
##########
@@ -47,212 +42,61 @@
LoggerFactory.getLogger(MiniOzoneLoadGenerator.class);
private static String keyNameDelimiter = "_";
-
- private ThreadPoolExecutor writeExecutor;
- private int numThreads;
- // number of buffer to be allocated, each is allocated with length which
- // is multiple of 2, each buffer is populated with random data.
- private int numBuffers;
- private List<ByteBuffer> buffers;
-
- private AtomicBoolean isIOThreadRunning;
-
- private final List<LoadBucket> ozoneBuckets;
-
- private final AtomicInteger agedFileWrittenIndex;
- private final ExecutorService agedFileExecutor;
- private final LoadBucket agedLoadBucket;
- private final TestProbability agedWriteProbability;
-
- private final ThreadPoolExecutor fsExecutor;
- private final LoadBucket fsBucket;
-
- MiniOzoneLoadGenerator(List<LoadBucket> bucket,
- LoadBucket agedLoadBucket, LoadBucket fsBucket,
- int numThreads, int numBuffers) {
- this.ozoneBuckets = bucket;
- this.numThreads = numThreads;
- this.numBuffers = numBuffers;
- this.writeExecutor = createExecutor();
-
- this.agedFileWrittenIndex = new AtomicInteger(0);
- this.agedFileExecutor = Executors.newSingleThreadExecutor();
- this.agedLoadBucket = agedLoadBucket;
- this.agedWriteProbability = TestProbability.valueOf(10);
-
- this.fsExecutor = createExecutor();
- this.fsBucket = fsBucket;
-
- this.isIOThreadRunning = new AtomicBoolean(false);
-
- // allocate buffers and populate random data.
- buffers = new ArrayList<>();
- for (int i = 0; i < numBuffers; i++) {
- int size = (int) StorageUnit.KB.toBytes(1 << i);
- ByteBuffer buffer = ByteBuffer.allocate(size);
- buffer.put(RandomUtils.nextBytes(size));
- buffers.add(buffer);
- }
- }
-
- private ThreadPoolExecutor createExecutor() {
- ThreadPoolExecutor executor = new ThreadPoolExecutor(numThreads, numThreads,
- 100, TimeUnit.SECONDS, new ArrayBlockingQueue<>(1024),
- new ThreadPoolExecutor.CallerRunsPolicy());
- executor.prestartAllCoreThreads();
- return executor;
-
- }
-
- // Start IO load on an Ozone bucket.
- private void load(long runTimeMillis) {
- long threadID = Thread.currentThread().getId();
- LOG.info("Started Mixed IO Thread:{}.", threadID);
- String threadName = Thread.currentThread().getName();
- long startTime = Time.monotonicNow();
-
- while (isIOThreadRunning.get() &&
- (Time.monotonicNow() < startTime + runTimeMillis)) {
- LoadBucket bucket =
- ozoneBuckets.get((int) (Math.random() * ozoneBuckets.size()));
- try {
- int index = RandomUtils.nextInt();
- ByteBuffer buffer = getBuffer(index);
- String keyName = getKeyName(index, threadName);
- bucket.writeKey(buffer, keyName);
-
- bucket.readKey(buffer, keyName);
-
- bucket.deleteKey(keyName);
- } catch (Exception e) {
- LOG.error("LOADGEN: Exiting due to exception", e);
- break;
- }
- }
- // This will terminate other threads too.
- isIOThreadRunning.set(false);
- LOG.info("Terminating IO thread:{}.", threadID);
- }
-
- private Optional<Integer> randomKeyToRead() {
- int currentIndex = agedFileWrittenIndex.get();
- return currentIndex != 0
- ? Optional.of(RandomUtils.nextInt(0, currentIndex))
- : Optional.empty();
- }
-
- private void startAgedLoad(long runTimeMillis) {
- long threadID = Thread.currentThread().getId();
- LOG.info("AGED LOADGEN: Started Aged IO Thread:{}.", threadID);
- String threadName = Thread.currentThread().getName();
- long startTime = Time.monotonicNow();
-
- while (isIOThreadRunning.get() &&
- (Time.monotonicNow() < startTime + runTimeMillis)) {
-
- String keyName = null;
- try {
- if (agedWriteProbability.isTrue()) {
- int index = agedFileWrittenIndex.getAndIncrement();
- ByteBuffer buffer = getBuffer(index);
- keyName = getKeyName(index, threadName);
-
- agedLoadBucket.writeKey(buffer, keyName);
- } else {
- Optional<Integer> index = randomKeyToRead();
- if (index.isPresent()) {
- ByteBuffer buffer = getBuffer(index.get());
- keyName = getKeyName(index.get(), threadName);
- agedLoadBucket.readKey(buffer, keyName);
- }
- }
- } catch (Throwable t) {
- LOG.error("AGED LOADGEN: {} Exiting due to exception", keyName, t);
- break;
- }
- }
- // This will terminate other threads too.
- isIOThreadRunning.set(false);
- LOG.info("Terminating IO thread:{}.", threadID);
- }
-
- // Start IO load on an Ozone bucket.
- private void startFsLoad(long runTimeMillis) {
- long threadID = Thread.currentThread().getId();
- LOG.info("Started Filesystem IO Thread:{}.", threadID);
- String threadName = Thread.currentThread().getName();
- long startTime = Time.monotonicNow();
-
- while (isIOThreadRunning.get() &&
- (Time.monotonicNow() < startTime + runTimeMillis)) {
- try {
- int index = RandomUtils.nextInt();
- ByteBuffer buffer = getBuffer(index);
- String keyName = getKeyName(index, threadName);
- fsBucket.writeKey(true, buffer, keyName);
-
- fsBucket.readKey(true, buffer, keyName);
-
- fsBucket.deleteKey(true, keyName);
- } catch (Exception e) {
- LOG.error("LOADGEN: Exiting due to exception", e);
- break;
- }
+ private final List<LoadExecutors> loadExecutors;
+
+ MiniOzoneLoadGenerator(OzoneVolume volume, int numClients, int numThreads,
+ int numBuffers, OzoneConfiguration conf)
+ throws Exception {
+ DataBuffer buffer = new DataBuffer(numBuffers);
+ loadExecutors = new ArrayList<>();
+
+ // Random Load
+ String mixBucketName = RandomStringUtils.randomAlphabetic(10).toLowerCase();
+ volume.createBucket(mixBucketName);
+ List<LoadBucket> ozoneBuckets = new ArrayList<>(numClients);
+ for (int i = 0; i < numClients; i++) {
+ ozoneBuckets.add(new LoadBucket(volume.getBucket(mixBucketName),
+ conf));
}
- // This will terminate other threads too.
- isIOThreadRunning.set(false);
- LOG.info("Terminating IO thread:{}.", threadID);
+ RandomLoadGenerator loadGenerator =
+ new RandomLoadGenerator(buffer, ozoneBuckets);
+ loadExecutors.add(new LoadExecutors(numThreads, loadGenerator));
+
+ // Aged Load
Review comment:
NIT: line 65-84 can be refactored to dedup the same create loadgenerator and added to load executors.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request
#438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load
generators to chaos testing.
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load generators to chaos testing.
URL: https://github.com/apache/hadoop-ozone/pull/438#discussion_r394942892
##########
File path: hadoop-ozone/fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/loadgenerators/LoadExecutors.java
##########
@@ -0,0 +1,81 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.loadgenerators;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Load executors for Ozone, this class provides a plugable
+ * executor for different load generators.
+ */
+public class LoadExecutors {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(LoadExecutors.class);
+
+ private final LoadGenerator generator;
+ private final int numThreads;
+ private final ThreadPoolExecutor executor;
+ private final List<CompletableFuture<Void>> futures = new ArrayList<>();
+
+ public LoadExecutors(int numThreads, LoadGenerator generator) {
+ this.numThreads = numThreads;
+ this.generator = generator;
+ this.executor = new ThreadPoolExecutor(numThreads, numThreads,
+ 100, TimeUnit.SECONDS, new ArrayBlockingQueue<>(1024),
+ new ThreadPoolExecutor.CallerRunsPolicy());
Review comment:
Since the number of threads (both core and max) and number of tasks is the same, do we need to specify keepAliveTime, queue limit and policy? I don't think these parameters will be applied in any case. I think using `Executors.newFixedThreadPool(numThreads)` would be simpler. Sure, `ExecutorService` provides no way to prestart threads, but this is not a short-running performance test, so the overhead of starting threads as tasks are added does not seem to matter.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] mukul1987 commented on a change in pull request
#438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load
generators to chaos testing.
Posted by GitBox <gi...@apache.org>.
mukul1987 commented on a change in pull request #438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load generators to chaos testing.
URL: https://github.com/apache/hadoop-ozone/pull/438#discussion_r396467880
##########
File path: hadoop-ozone/fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/loadgenerators/DataBuffer.java
##########
@@ -0,0 +1,53 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.loadgenerators;
+
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.conf.StorageUnit;
+
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * List of buffers used by the load generators.
+ */
+public class DataBuffer {
+ private List<ByteBuffer> buffers;
+ // number of buffer to be allocated, each is allocated with length which
+ // is multiple of 2, each buffer is populated with random data.
+ private int numBuffers;
+
+ public DataBuffer(int numBuffers) {
+ // allocate buffers and populate random data.
+ this.numBuffers = numBuffers;
+ this.buffers = new ArrayList<>();
+ for (int i = 0; i < numBuffers; i++) {
Review comment:
I am skipping his change for now as this will need a significant refactoring. Can I do this as a followup please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] mukul1987 commented on a change in pull request
#438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load
generators to chaos testing.
Posted by GitBox <gi...@apache.org>.
mukul1987 commented on a change in pull request #438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load generators to chaos testing.
URL: https://github.com/apache/hadoop-ozone/pull/438#discussion_r396458781
##########
File path: hadoop-ozone/fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/loadgenerators/FilesystemLoadGenerator.java
##########
@@ -0,0 +1,82 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.loadgenerators;
+
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.ozone.MiniOzoneLoadGenerator;
+import org.apache.hadoop.ozone.utils.LoadBucket;
+import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.util.Time;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+
+/**
+ * Filesystem load generator for Ozone.
+ *
+ * This load generator read, writes and deletes data using the filesystem
+ * apis.
+ */
+public class FilesystemLoadGenerator implements LoadGenerator {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(FilesystemLoadGenerator.class);
+
+
+ private final LoadBucket fsBucket;
+ private final DataBuffer dataBuffer;
+
+ public FilesystemLoadGenerator(DataBuffer dataBuffer, LoadBucket fsBucket) {
+ this.dataBuffer = dataBuffer;
+ this.fsBucket = fsBucket;
+ }
+
+ // Start IO load on an Ozone bucket.
+ public void startLoad(long runTimeMillis) {
+ long threadID = Thread.currentThread().getId();
+ LOG.info("Started Filesystem IO Thread:{}.", threadID);
+ String threadName = Thread.currentThread().getName();
+ long startTime = Time.monotonicNow();
+
+ while (Time.monotonicNow() < startTime + runTimeMillis) {
+ try {
+ int index = RandomUtils.nextInt();
Review comment:
I haven't seen the contention anytime in any run. Also, Chaos cluster is a correctness check and not a performance run. so skipping this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] nandakumar131 commented on a change in pull request
#438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load
generators to chaos testing.
Posted by GitBox <gi...@apache.org>.
nandakumar131 commented on a change in pull request #438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load generators to chaos testing.
URL: https://github.com/apache/hadoop-ozone/pull/438#discussion_r373045976
##########
File path: hadoop-ozone/fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/loadgenerators/LoadExecutors.java
##########
@@ -0,0 +1,81 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.loadgenerators;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Load executors for Ozone, this class provides a pluggable
Review comment:
Typo: pluggable
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] mukul1987 commented on a change in pull request
#438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load
generators to chaos testing.
Posted by GitBox <gi...@apache.org>.
mukul1987 commented on a change in pull request #438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load generators to chaos testing.
URL: https://github.com/apache/hadoop-ozone/pull/438#discussion_r396299867
##########
File path: hadoop-ozone/fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/loadgenerators/AgedLoadGenerator.java
##########
@@ -0,0 +1,110 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.loadgenerators;
+
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.ozone.MiniOzoneLoadGenerator;
+import org.apache.hadoop.ozone.utils.LoadBucket;
+import org.apache.hadoop.ozone.utils.TestProbability;
+import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.util.Time;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.util.Optional;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * Aged Load Generator for Ozone.
+ *
+ * This Load Generator reads and write key to an Ozone bucket.
+ *
+ * The default writes to read ratio is 10:90.
+ */
+public class AgedLoadGenerator implements LoadGenerator {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(AgedLoadGenerator.class);
+ private static String agedSuffix = "aged";
+
+ private final AtomicInteger agedFileWrittenIndex;
+ private final AtomicInteger agedFileAllocationIndex;
+ private final LoadBucket agedLoadBucket;
+ private final TestProbability agedWriteProbability;
+ private final DataBuffer dataBuffer;
+
+ public AgedLoadGenerator(DataBuffer data, LoadBucket agedLoadBucket) {
+ this.dataBuffer = data;
+ this.agedFileWrittenIndex = new AtomicInteger(0);
+ this.agedFileAllocationIndex = new AtomicInteger(0);
+ this.agedLoadBucket = agedLoadBucket;
+ this.agedWriteProbability = TestProbability.valueOf(10);
+ }
+
+ public void startLoad(long runTimeMillis) {
+ long threadID = Thread.currentThread().getId();
+ LOG.info("AGED LOADGEN: Started Aged IO Thread:{}.", threadID);
+ long startTime = Time.monotonicNow();
+
+ while (Time.monotonicNow() < startTime + runTimeMillis) {
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] mukul1987 commented on a change in pull request
#438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load
generators to chaos testing.
Posted by GitBox <gi...@apache.org>.
mukul1987 commented on a change in pull request #438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load generators to chaos testing.
URL: https://github.com/apache/hadoop-ozone/pull/438#discussion_r396454779
##########
File path: hadoop-ozone/fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/MiniOzoneLoadGenerator.java
##########
@@ -47,212 +42,61 @@
LoggerFactory.getLogger(MiniOzoneLoadGenerator.class);
private static String keyNameDelimiter = "_";
-
- private ThreadPoolExecutor writeExecutor;
- private int numThreads;
- // number of buffer to be allocated, each is allocated with length which
- // is multiple of 2, each buffer is populated with random data.
- private int numBuffers;
- private List<ByteBuffer> buffers;
-
- private AtomicBoolean isIOThreadRunning;
-
- private final List<LoadBucket> ozoneBuckets;
-
- private final AtomicInteger agedFileWrittenIndex;
- private final ExecutorService agedFileExecutor;
- private final LoadBucket agedLoadBucket;
- private final TestProbability agedWriteProbability;
-
- private final ThreadPoolExecutor fsExecutor;
- private final LoadBucket fsBucket;
-
- MiniOzoneLoadGenerator(List<LoadBucket> bucket,
- LoadBucket agedLoadBucket, LoadBucket fsBucket,
- int numThreads, int numBuffers) {
- this.ozoneBuckets = bucket;
- this.numThreads = numThreads;
- this.numBuffers = numBuffers;
- this.writeExecutor = createExecutor();
-
- this.agedFileWrittenIndex = new AtomicInteger(0);
- this.agedFileExecutor = Executors.newSingleThreadExecutor();
- this.agedLoadBucket = agedLoadBucket;
- this.agedWriteProbability = TestProbability.valueOf(10);
-
- this.fsExecutor = createExecutor();
- this.fsBucket = fsBucket;
-
- this.isIOThreadRunning = new AtomicBoolean(false);
-
- // allocate buffers and populate random data.
- buffers = new ArrayList<>();
- for (int i = 0; i < numBuffers; i++) {
- int size = (int) StorageUnit.KB.toBytes(1 << i);
- ByteBuffer buffer = ByteBuffer.allocate(size);
- buffer.put(RandomUtils.nextBytes(size));
- buffers.add(buffer);
- }
- }
-
- private ThreadPoolExecutor createExecutor() {
- ThreadPoolExecutor executor = new ThreadPoolExecutor(numThreads, numThreads,
- 100, TimeUnit.SECONDS, new ArrayBlockingQueue<>(1024),
- new ThreadPoolExecutor.CallerRunsPolicy());
- executor.prestartAllCoreThreads();
- return executor;
-
- }
-
- // Start IO load on an Ozone bucket.
- private void load(long runTimeMillis) {
- long threadID = Thread.currentThread().getId();
- LOG.info("Started Mixed IO Thread:{}.", threadID);
- String threadName = Thread.currentThread().getName();
- long startTime = Time.monotonicNow();
-
- while (isIOThreadRunning.get() &&
- (Time.monotonicNow() < startTime + runTimeMillis)) {
- LoadBucket bucket =
- ozoneBuckets.get((int) (Math.random() * ozoneBuckets.size()));
- try {
- int index = RandomUtils.nextInt();
- ByteBuffer buffer = getBuffer(index);
- String keyName = getKeyName(index, threadName);
- bucket.writeKey(buffer, keyName);
-
- bucket.readKey(buffer, keyName);
-
- bucket.deleteKey(keyName);
- } catch (Exception e) {
- LOG.error("LOADGEN: Exiting due to exception", e);
- break;
- }
- }
- // This will terminate other threads too.
- isIOThreadRunning.set(false);
- LOG.info("Terminating IO thread:{}.", threadID);
- }
-
- private Optional<Integer> randomKeyToRead() {
- int currentIndex = agedFileWrittenIndex.get();
- return currentIndex != 0
- ? Optional.of(RandomUtils.nextInt(0, currentIndex))
- : Optional.empty();
- }
-
- private void startAgedLoad(long runTimeMillis) {
- long threadID = Thread.currentThread().getId();
- LOG.info("AGED LOADGEN: Started Aged IO Thread:{}.", threadID);
- String threadName = Thread.currentThread().getName();
- long startTime = Time.monotonicNow();
-
- while (isIOThreadRunning.get() &&
- (Time.monotonicNow() < startTime + runTimeMillis)) {
-
- String keyName = null;
- try {
- if (agedWriteProbability.isTrue()) {
- int index = agedFileWrittenIndex.getAndIncrement();
- ByteBuffer buffer = getBuffer(index);
- keyName = getKeyName(index, threadName);
-
- agedLoadBucket.writeKey(buffer, keyName);
- } else {
- Optional<Integer> index = randomKeyToRead();
- if (index.isPresent()) {
- ByteBuffer buffer = getBuffer(index.get());
- keyName = getKeyName(index.get(), threadName);
- agedLoadBucket.readKey(buffer, keyName);
- }
- }
- } catch (Throwable t) {
- LOG.error("AGED LOADGEN: {} Exiting due to exception", keyName, t);
- break;
- }
- }
- // This will terminate other threads too.
- isIOThreadRunning.set(false);
- LOG.info("Terminating IO thread:{}.", threadID);
- }
-
- // Start IO load on an Ozone bucket.
- private void startFsLoad(long runTimeMillis) {
- long threadID = Thread.currentThread().getId();
- LOG.info("Started Filesystem IO Thread:{}.", threadID);
- String threadName = Thread.currentThread().getName();
- long startTime = Time.monotonicNow();
-
- while (isIOThreadRunning.get() &&
- (Time.monotonicNow() < startTime + runTimeMillis)) {
- try {
- int index = RandomUtils.nextInt();
- ByteBuffer buffer = getBuffer(index);
- String keyName = getKeyName(index, threadName);
- fsBucket.writeKey(true, buffer, keyName);
-
- fsBucket.readKey(true, buffer, keyName);
-
- fsBucket.deleteKey(true, keyName);
- } catch (Exception e) {
- LOG.error("LOADGEN: Exiting due to exception", e);
- break;
- }
+ private final List<LoadExecutors> loadExecutors;
+
+ MiniOzoneLoadGenerator(OzoneVolume volume, int numClients, int numThreads,
+ int numBuffers, OzoneConfiguration conf)
+ throws Exception {
+ DataBuffer buffer = new DataBuffer(numBuffers);
+ loadExecutors = new ArrayList<>();
+
+ // Random Load
+ String mixBucketName = RandomStringUtils.randomAlphabetic(10).toLowerCase();
+ volume.createBucket(mixBucketName);
+ List<LoadBucket> ozoneBuckets = new ArrayList<>(numClients);
+ for (int i = 0; i < numClients; i++) {
+ ozoneBuckets.add(new LoadBucket(volume.getBucket(mixBucketName),
+ conf));
}
- // This will terminate other threads too.
- isIOThreadRunning.set(false);
- LOG.info("Terminating IO thread:{}.", threadID);
+ RandomLoadGenerator loadGenerator =
+ new RandomLoadGenerator(buffer, ozoneBuckets);
+ loadExecutors.add(new LoadExecutors(numThreads, loadGenerator));
+
+ // Aged Load
+ String agedBucketName =
+ RandomStringUtils.randomAlphabetic(10).toLowerCase();
+ volume.createBucket(agedBucketName);
+ LoadBucket agedLoadBucket =
+ new LoadBucket(volume.getBucket(agedBucketName), conf);
+ AgedLoadGenerator agedLoadGenerator =
+ new AgedLoadGenerator(buffer, agedLoadBucket);
+ loadExecutors.add(new LoadExecutors(numThreads, agedLoadGenerator));
+
+ //Filesystem Load
+ String fsBucketName =
+ RandomStringUtils.randomAlphabetic(10).toLowerCase();
+
+ volume.createBucket(fsBucketName);
+ LoadBucket fsBucket =
+ new LoadBucket(volume.getBucket(fsBucketName), conf);
+ FilesystemLoadGenerator filesystemLoadGenerator =
+ new FilesystemLoadGenerator(buffer, fsBucket);
+ loadExecutors.add(new LoadExecutors(numThreads, filesystemLoadGenerator));
}
void startIO(long time, TimeUnit timeUnit) {
- List<CompletableFuture<Void>> writeFutures = new ArrayList<>();
- LOG.info("Starting MiniOzoneLoadGenerator for time {}:{} with {} buffers " +
- "and {} threads", time, timeUnit, numBuffers, numThreads);
- if (isIOThreadRunning.compareAndSet(false, true)) {
- // Start the IO thread
- for (int i = 0; i < numThreads; i++) {
- writeFutures.add(
- CompletableFuture.runAsync(() -> load(timeUnit.toMillis(time)),
- writeExecutor));
- }
-
- for (int i = 0; i < numThreads; i++) {
- writeFutures.add(
- CompletableFuture.runAsync(() -> startAgedLoad(
- timeUnit.toMillis(time)), agedFileExecutor));
- }
-
- for (int i = 0; i < numThreads; i++) {
- writeFutures.add(
- CompletableFuture.runAsync(() -> startFsLoad(
- timeUnit.toMillis(time)), fsExecutor));
- }
-
- // Wait for IO to complete
- for (CompletableFuture<Void> f : writeFutures) {
- try {
- f.get();
- } catch (Throwable t) {
- LOG.error("startIO failed with exception", t);
- }
- }
- }
- }
-
- public void shutdownLoadGenerator() {
- try {
- writeExecutor.shutdown();
- writeExecutor.awaitTermination(1, TimeUnit.DAYS);
- } catch (Exception e) {
- LOG.error("error while closing ", e);
- }
+ LOG.info("Starting MiniOzoneLoadGenerator for time {}:{}", time, timeUnit);
+ long runTime = timeUnit.toMillis(time);
+ // start and wait for executors to finish
+ loadExecutors.forEach(le -> le.startLoad(runTime));
+ loadExecutors.forEach(LoadExecutors::waitForCompletion);
}
- private ByteBuffer getBuffer(int keyIndex) {
- return buffers.get(keyIndex % numBuffers);
+ void shutdownLoadGenerator() {
+ loadExecutors.forEach(LoadExecutors::shutdown);
}
- private String getKeyName(int keyIndex, String threadName) {
- return threadName + keyNameDelimiter + keyIndex;
+ public static String getKeyName(int keyIndex, String prefix) {
+ return prefix + keyNameDelimiter + keyIndex;
}
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request
#438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load
generators to chaos testing.
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load generators to chaos testing.
URL: https://github.com/apache/hadoop-ozone/pull/438#discussion_r394937622
##########
File path: hadoop-ozone/fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/loadgenerators/RandomLoadGenerator.java
##########
@@ -0,0 +1,82 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.loadgenerators;
+
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.ozone.MiniOzoneLoadGenerator;
+import org.apache.hadoop.ozone.utils.LoadBucket;
+import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.util.Time;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.util.List;
+
+/**
+ * Random load generator which writes, read and deletes keys from
+ * the bucket.
+ */
+public class RandomLoadGenerator implements LoadGenerator {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(RandomLoadGenerator.class);
+
+ private final List<LoadBucket> ozoneBuckets;
+ private final DataBuffer dataBuffer;
+
+ public RandomLoadGenerator(DataBuffer dataBuffer, List<LoadBucket> buckets) {
+ this.ozoneBuckets = buckets;
+ this.dataBuffer = dataBuffer;
+ }
+
+ // Start IO load on an Ozone bucket.
+ public void startLoad(long runTimeMillis) {
+ long threadID = Thread.currentThread().getId();
+ LOG.info("Started Mixed IO Thread:{}.", threadID);
+ String threadName = Thread.currentThread().getName();
+ long startTime = Time.monotonicNow();
+
+ while (Time.monotonicNow() < startTime + runTimeMillis) {
+ LoadBucket bucket =
+ ozoneBuckets.get((int) (Math.random() * ozoneBuckets.size()));
+ try {
+ int index = RandomUtils.nextInt();
+ ByteBuffer buffer = dataBuffer.getBuffer(index);
+ String keyName = MiniOzoneLoadGenerator.getKeyName(index, threadName);
+ bucket.writeKey(buffer, keyName);
+
+ bucket.readKey(buffer, keyName);
+
+ bucket.deleteKey(keyName);
+ } catch (Throwable t) {
+ LOG.error("LOADGEN: Exiting due to exception", t);
+ ExitUtil.terminate(new ExitUtil.ExitException(3, t));
+ }
+ }
+ }
Review comment:
Can you please extract common structure from all 3 generators? Only `<perform operation>` is changing.
```
public void startLoad(long runTimeMillis) {
...
while (...) {
try {
<perform operation>
} catch (...) {
..
}
}
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] nandakumar131 commented on a change in pull request
#438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load
generators to chaos testing.
Posted by GitBox <gi...@apache.org>.
nandakumar131 commented on a change in pull request #438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load generators to chaos testing.
URL: https://github.com/apache/hadoop-ozone/pull/438#discussion_r373054791
##########
File path: hadoop-ozone/fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/MiniOzoneLoadGenerator.java
##########
@@ -47,212 +42,61 @@
LoggerFactory.getLogger(MiniOzoneLoadGenerator.class);
private static String keyNameDelimiter = "_";
-
- private ThreadPoolExecutor writeExecutor;
- private int numThreads;
- // number of buffer to be allocated, each is allocated with length which
- // is multiple of 2, each buffer is populated with random data.
- private int numBuffers;
- private List<ByteBuffer> buffers;
-
- private AtomicBoolean isIOThreadRunning;
-
- private final List<LoadBucket> ozoneBuckets;
-
- private final AtomicInteger agedFileWrittenIndex;
- private final ExecutorService agedFileExecutor;
- private final LoadBucket agedLoadBucket;
- private final TestProbability agedWriteProbability;
-
- private final ThreadPoolExecutor fsExecutor;
- private final LoadBucket fsBucket;
-
- MiniOzoneLoadGenerator(List<LoadBucket> bucket,
- LoadBucket agedLoadBucket, LoadBucket fsBucket,
- int numThreads, int numBuffers) {
- this.ozoneBuckets = bucket;
- this.numThreads = numThreads;
- this.numBuffers = numBuffers;
- this.writeExecutor = createExecutor();
-
- this.agedFileWrittenIndex = new AtomicInteger(0);
- this.agedFileExecutor = Executors.newSingleThreadExecutor();
- this.agedLoadBucket = agedLoadBucket;
- this.agedWriteProbability = TestProbability.valueOf(10);
-
- this.fsExecutor = createExecutor();
- this.fsBucket = fsBucket;
-
- this.isIOThreadRunning = new AtomicBoolean(false);
-
- // allocate buffers and populate random data.
- buffers = new ArrayList<>();
- for (int i = 0; i < numBuffers; i++) {
- int size = (int) StorageUnit.KB.toBytes(1 << i);
- ByteBuffer buffer = ByteBuffer.allocate(size);
- buffer.put(RandomUtils.nextBytes(size));
- buffers.add(buffer);
- }
- }
-
- private ThreadPoolExecutor createExecutor() {
- ThreadPoolExecutor executor = new ThreadPoolExecutor(numThreads, numThreads,
- 100, TimeUnit.SECONDS, new ArrayBlockingQueue<>(1024),
- new ThreadPoolExecutor.CallerRunsPolicy());
- executor.prestartAllCoreThreads();
- return executor;
-
- }
-
- // Start IO load on an Ozone bucket.
- private void load(long runTimeMillis) {
- long threadID = Thread.currentThread().getId();
- LOG.info("Started Mixed IO Thread:{}.", threadID);
- String threadName = Thread.currentThread().getName();
- long startTime = Time.monotonicNow();
-
- while (isIOThreadRunning.get() &&
- (Time.monotonicNow() < startTime + runTimeMillis)) {
- LoadBucket bucket =
- ozoneBuckets.get((int) (Math.random() * ozoneBuckets.size()));
- try {
- int index = RandomUtils.nextInt();
- ByteBuffer buffer = getBuffer(index);
- String keyName = getKeyName(index, threadName);
- bucket.writeKey(buffer, keyName);
-
- bucket.readKey(buffer, keyName);
-
- bucket.deleteKey(keyName);
- } catch (Exception e) {
- LOG.error("LOADGEN: Exiting due to exception", e);
- break;
- }
- }
- // This will terminate other threads too.
- isIOThreadRunning.set(false);
- LOG.info("Terminating IO thread:{}.", threadID);
- }
-
- private Optional<Integer> randomKeyToRead() {
- int currentIndex = agedFileWrittenIndex.get();
- return currentIndex != 0
- ? Optional.of(RandomUtils.nextInt(0, currentIndex))
- : Optional.empty();
- }
-
- private void startAgedLoad(long runTimeMillis) {
- long threadID = Thread.currentThread().getId();
- LOG.info("AGED LOADGEN: Started Aged IO Thread:{}.", threadID);
- String threadName = Thread.currentThread().getName();
- long startTime = Time.monotonicNow();
-
- while (isIOThreadRunning.get() &&
- (Time.monotonicNow() < startTime + runTimeMillis)) {
-
- String keyName = null;
- try {
- if (agedWriteProbability.isTrue()) {
- int index = agedFileWrittenIndex.getAndIncrement();
- ByteBuffer buffer = getBuffer(index);
- keyName = getKeyName(index, threadName);
-
- agedLoadBucket.writeKey(buffer, keyName);
- } else {
- Optional<Integer> index = randomKeyToRead();
- if (index.isPresent()) {
- ByteBuffer buffer = getBuffer(index.get());
- keyName = getKeyName(index.get(), threadName);
- agedLoadBucket.readKey(buffer, keyName);
- }
- }
- } catch (Throwable t) {
- LOG.error("AGED LOADGEN: {} Exiting due to exception", keyName, t);
- break;
- }
- }
- // This will terminate other threads too.
- isIOThreadRunning.set(false);
- LOG.info("Terminating IO thread:{}.", threadID);
- }
-
- // Start IO load on an Ozone bucket.
- private void startFsLoad(long runTimeMillis) {
- long threadID = Thread.currentThread().getId();
- LOG.info("Started Filesystem IO Thread:{}.", threadID);
- String threadName = Thread.currentThread().getName();
- long startTime = Time.monotonicNow();
-
- while (isIOThreadRunning.get() &&
- (Time.monotonicNow() < startTime + runTimeMillis)) {
- try {
- int index = RandomUtils.nextInt();
- ByteBuffer buffer = getBuffer(index);
- String keyName = getKeyName(index, threadName);
- fsBucket.writeKey(true, buffer, keyName);
-
- fsBucket.readKey(true, buffer, keyName);
-
- fsBucket.deleteKey(true, keyName);
- } catch (Exception e) {
- LOG.error("LOADGEN: Exiting due to exception", e);
- break;
- }
+ private final List<LoadExecutors> loadExecutors;
+
+ MiniOzoneLoadGenerator(OzoneVolume volume, int numClients, int numThreads,
Review comment:
`numClients` actually mean the max number of buckets that will be used by `RandomLoadGenerator` ?
Correct me if I'm wrong, we got `numThreads` which actually means the number of client threads.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] nandakumar131 commented on a change in pull request
#438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load
generators to chaos testing.
Posted by GitBox <gi...@apache.org>.
nandakumar131 commented on a change in pull request #438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load generators to chaos testing.
URL: https://github.com/apache/hadoop-ozone/pull/438#discussion_r373040115
##########
File path: hadoop-ozone/fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/loadgenerators/AgedLoadGenerator.java
##########
@@ -0,0 +1,110 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.loadgenerators;
+
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.ozone.MiniOzoneLoadGenerator;
+import org.apache.hadoop.ozone.utils.LoadBucket;
+import org.apache.hadoop.ozone.utils.TestProbability;
+import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.util.Time;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.util.Optional;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * Aged Load Generator for Ozone.
+ *
+ * This Load Generator reads and write key to an Ozone bucket.
+ *
+ * The defautl writes to read ratio is 10:90.
Review comment:
Typo: defautl
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] mukul1987 commented on a change in pull request
#438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load
generators to chaos testing.
Posted by GitBox <gi...@apache.org>.
mukul1987 commented on a change in pull request #438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load generators to chaos testing.
URL: https://github.com/apache/hadoop-ozone/pull/438#discussion_r392697193
##########
File path: hadoop-ozone/fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/MiniOzoneLoadGenerator.java
##########
@@ -47,212 +42,61 @@
LoggerFactory.getLogger(MiniOzoneLoadGenerator.class);
private static String keyNameDelimiter = "_";
-
- private ThreadPoolExecutor writeExecutor;
- private int numThreads;
- // number of buffer to be allocated, each is allocated with length which
- // is multiple of 2, each buffer is populated with random data.
- private int numBuffers;
- private List<ByteBuffer> buffers;
-
- private AtomicBoolean isIOThreadRunning;
-
- private final List<LoadBucket> ozoneBuckets;
-
- private final AtomicInteger agedFileWrittenIndex;
- private final ExecutorService agedFileExecutor;
- private final LoadBucket agedLoadBucket;
- private final TestProbability agedWriteProbability;
-
- private final ThreadPoolExecutor fsExecutor;
- private final LoadBucket fsBucket;
-
- MiniOzoneLoadGenerator(List<LoadBucket> bucket,
- LoadBucket agedLoadBucket, LoadBucket fsBucket,
- int numThreads, int numBuffers) {
- this.ozoneBuckets = bucket;
- this.numThreads = numThreads;
- this.numBuffers = numBuffers;
- this.writeExecutor = createExecutor();
-
- this.agedFileWrittenIndex = new AtomicInteger(0);
- this.agedFileExecutor = Executors.newSingleThreadExecutor();
- this.agedLoadBucket = agedLoadBucket;
- this.agedWriteProbability = TestProbability.valueOf(10);
-
- this.fsExecutor = createExecutor();
- this.fsBucket = fsBucket;
-
- this.isIOThreadRunning = new AtomicBoolean(false);
-
- // allocate buffers and populate random data.
- buffers = new ArrayList<>();
- for (int i = 0; i < numBuffers; i++) {
- int size = (int) StorageUnit.KB.toBytes(1 << i);
- ByteBuffer buffer = ByteBuffer.allocate(size);
- buffer.put(RandomUtils.nextBytes(size));
- buffers.add(buffer);
- }
- }
-
- private ThreadPoolExecutor createExecutor() {
- ThreadPoolExecutor executor = new ThreadPoolExecutor(numThreads, numThreads,
- 100, TimeUnit.SECONDS, new ArrayBlockingQueue<>(1024),
- new ThreadPoolExecutor.CallerRunsPolicy());
- executor.prestartAllCoreThreads();
- return executor;
-
- }
-
- // Start IO load on an Ozone bucket.
- private void load(long runTimeMillis) {
- long threadID = Thread.currentThread().getId();
- LOG.info("Started Mixed IO Thread:{}.", threadID);
- String threadName = Thread.currentThread().getName();
- long startTime = Time.monotonicNow();
-
- while (isIOThreadRunning.get() &&
- (Time.monotonicNow() < startTime + runTimeMillis)) {
- LoadBucket bucket =
- ozoneBuckets.get((int) (Math.random() * ozoneBuckets.size()));
- try {
- int index = RandomUtils.nextInt();
- ByteBuffer buffer = getBuffer(index);
- String keyName = getKeyName(index, threadName);
- bucket.writeKey(buffer, keyName);
-
- bucket.readKey(buffer, keyName);
-
- bucket.deleteKey(keyName);
- } catch (Exception e) {
- LOG.error("LOADGEN: Exiting due to exception", e);
- break;
- }
- }
- // This will terminate other threads too.
- isIOThreadRunning.set(false);
- LOG.info("Terminating IO thread:{}.", threadID);
- }
-
- private Optional<Integer> randomKeyToRead() {
- int currentIndex = agedFileWrittenIndex.get();
- return currentIndex != 0
- ? Optional.of(RandomUtils.nextInt(0, currentIndex))
- : Optional.empty();
- }
-
- private void startAgedLoad(long runTimeMillis) {
- long threadID = Thread.currentThread().getId();
- LOG.info("AGED LOADGEN: Started Aged IO Thread:{}.", threadID);
- String threadName = Thread.currentThread().getName();
- long startTime = Time.monotonicNow();
-
- while (isIOThreadRunning.get() &&
- (Time.monotonicNow() < startTime + runTimeMillis)) {
-
- String keyName = null;
- try {
- if (agedWriteProbability.isTrue()) {
- int index = agedFileWrittenIndex.getAndIncrement();
- ByteBuffer buffer = getBuffer(index);
- keyName = getKeyName(index, threadName);
-
- agedLoadBucket.writeKey(buffer, keyName);
- } else {
- Optional<Integer> index = randomKeyToRead();
- if (index.isPresent()) {
- ByteBuffer buffer = getBuffer(index.get());
- keyName = getKeyName(index.get(), threadName);
- agedLoadBucket.readKey(buffer, keyName);
- }
- }
- } catch (Throwable t) {
- LOG.error("AGED LOADGEN: {} Exiting due to exception", keyName, t);
- break;
- }
- }
- // This will terminate other threads too.
- isIOThreadRunning.set(false);
- LOG.info("Terminating IO thread:{}.", threadID);
- }
-
- // Start IO load on an Ozone bucket.
- private void startFsLoad(long runTimeMillis) {
- long threadID = Thread.currentThread().getId();
- LOG.info("Started Filesystem IO Thread:{}.", threadID);
- String threadName = Thread.currentThread().getName();
- long startTime = Time.monotonicNow();
-
- while (isIOThreadRunning.get() &&
- (Time.monotonicNow() < startTime + runTimeMillis)) {
- try {
- int index = RandomUtils.nextInt();
- ByteBuffer buffer = getBuffer(index);
- String keyName = getKeyName(index, threadName);
- fsBucket.writeKey(true, buffer, keyName);
-
- fsBucket.readKey(true, buffer, keyName);
-
- fsBucket.deleteKey(true, keyName);
- } catch (Exception e) {
- LOG.error("LOADGEN: Exiting due to exception", e);
- break;
- }
+ private final List<LoadExecutors> loadExecutors;
+
+ MiniOzoneLoadGenerator(OzoneVolume volume, int numClients, int numThreads,
Review comment:
yes, thats write. numthreads is the number of threads used across all the buckets
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] mukul1987 commented on a change in pull request
#438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load
generators to chaos testing.
Posted by GitBox <gi...@apache.org>.
mukul1987 commented on a change in pull request #438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load generators to chaos testing.
URL: https://github.com/apache/hadoop-ozone/pull/438#discussion_r395213932
##########
File path: hadoop-ozone/fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/MiniOzoneLoadGenerator.java
##########
@@ -47,212 +42,61 @@
LoggerFactory.getLogger(MiniOzoneLoadGenerator.class);
private static String keyNameDelimiter = "_";
-
- private ThreadPoolExecutor writeExecutor;
- private int numThreads;
- // number of buffer to be allocated, each is allocated with length which
- // is multiple of 2, each buffer is populated with random data.
- private int numBuffers;
- private List<ByteBuffer> buffers;
-
- private AtomicBoolean isIOThreadRunning;
-
- private final List<LoadBucket> ozoneBuckets;
-
- private final AtomicInteger agedFileWrittenIndex;
- private final ExecutorService agedFileExecutor;
- private final LoadBucket agedLoadBucket;
- private final TestProbability agedWriteProbability;
-
- private final ThreadPoolExecutor fsExecutor;
- private final LoadBucket fsBucket;
-
- MiniOzoneLoadGenerator(List<LoadBucket> bucket,
- LoadBucket agedLoadBucket, LoadBucket fsBucket,
- int numThreads, int numBuffers) {
- this.ozoneBuckets = bucket;
- this.numThreads = numThreads;
- this.numBuffers = numBuffers;
- this.writeExecutor = createExecutor();
-
- this.agedFileWrittenIndex = new AtomicInteger(0);
- this.agedFileExecutor = Executors.newSingleThreadExecutor();
- this.agedLoadBucket = agedLoadBucket;
- this.agedWriteProbability = TestProbability.valueOf(10);
-
- this.fsExecutor = createExecutor();
- this.fsBucket = fsBucket;
-
- this.isIOThreadRunning = new AtomicBoolean(false);
-
- // allocate buffers and populate random data.
- buffers = new ArrayList<>();
- for (int i = 0; i < numBuffers; i++) {
- int size = (int) StorageUnit.KB.toBytes(1 << i);
- ByteBuffer buffer = ByteBuffer.allocate(size);
- buffer.put(RandomUtils.nextBytes(size));
- buffers.add(buffer);
- }
- }
-
- private ThreadPoolExecutor createExecutor() {
- ThreadPoolExecutor executor = new ThreadPoolExecutor(numThreads, numThreads,
- 100, TimeUnit.SECONDS, new ArrayBlockingQueue<>(1024),
- new ThreadPoolExecutor.CallerRunsPolicy());
- executor.prestartAllCoreThreads();
- return executor;
-
- }
-
- // Start IO load on an Ozone bucket.
- private void load(long runTimeMillis) {
- long threadID = Thread.currentThread().getId();
- LOG.info("Started Mixed IO Thread:{}.", threadID);
- String threadName = Thread.currentThread().getName();
- long startTime = Time.monotonicNow();
-
- while (isIOThreadRunning.get() &&
- (Time.monotonicNow() < startTime + runTimeMillis)) {
- LoadBucket bucket =
- ozoneBuckets.get((int) (Math.random() * ozoneBuckets.size()));
- try {
- int index = RandomUtils.nextInt();
- ByteBuffer buffer = getBuffer(index);
- String keyName = getKeyName(index, threadName);
- bucket.writeKey(buffer, keyName);
-
- bucket.readKey(buffer, keyName);
-
- bucket.deleteKey(keyName);
- } catch (Exception e) {
- LOG.error("LOADGEN: Exiting due to exception", e);
- break;
- }
- }
- // This will terminate other threads too.
- isIOThreadRunning.set(false);
- LOG.info("Terminating IO thread:{}.", threadID);
- }
-
- private Optional<Integer> randomKeyToRead() {
- int currentIndex = agedFileWrittenIndex.get();
- return currentIndex != 0
- ? Optional.of(RandomUtils.nextInt(0, currentIndex))
- : Optional.empty();
- }
-
- private void startAgedLoad(long runTimeMillis) {
- long threadID = Thread.currentThread().getId();
- LOG.info("AGED LOADGEN: Started Aged IO Thread:{}.", threadID);
- String threadName = Thread.currentThread().getName();
- long startTime = Time.monotonicNow();
-
- while (isIOThreadRunning.get() &&
- (Time.monotonicNow() < startTime + runTimeMillis)) {
-
- String keyName = null;
- try {
- if (agedWriteProbability.isTrue()) {
- int index = agedFileWrittenIndex.getAndIncrement();
- ByteBuffer buffer = getBuffer(index);
- keyName = getKeyName(index, threadName);
-
- agedLoadBucket.writeKey(buffer, keyName);
- } else {
- Optional<Integer> index = randomKeyToRead();
- if (index.isPresent()) {
- ByteBuffer buffer = getBuffer(index.get());
- keyName = getKeyName(index.get(), threadName);
- agedLoadBucket.readKey(buffer, keyName);
- }
- }
- } catch (Throwable t) {
- LOG.error("AGED LOADGEN: {} Exiting due to exception", keyName, t);
- break;
- }
- }
- // This will terminate other threads too.
- isIOThreadRunning.set(false);
- LOG.info("Terminating IO thread:{}.", threadID);
- }
-
- // Start IO load on an Ozone bucket.
- private void startFsLoad(long runTimeMillis) {
- long threadID = Thread.currentThread().getId();
- LOG.info("Started Filesystem IO Thread:{}.", threadID);
- String threadName = Thread.currentThread().getName();
- long startTime = Time.monotonicNow();
-
- while (isIOThreadRunning.get() &&
- (Time.monotonicNow() < startTime + runTimeMillis)) {
- try {
- int index = RandomUtils.nextInt();
- ByteBuffer buffer = getBuffer(index);
- String keyName = getKeyName(index, threadName);
- fsBucket.writeKey(true, buffer, keyName);
-
- fsBucket.readKey(true, buffer, keyName);
-
- fsBucket.deleteKey(true, keyName);
- } catch (Exception e) {
- LOG.error("LOADGEN: Exiting due to exception", e);
- break;
- }
+ private final List<LoadExecutors> loadExecutors;
+
+ MiniOzoneLoadGenerator(OzoneVolume volume, int numClients, int numThreads,
+ int numBuffers, OzoneConfiguration conf)
+ throws Exception {
+ DataBuffer buffer = new DataBuffer(numBuffers);
+ loadExecutors = new ArrayList<>();
+
+ // Random Load
+ String mixBucketName = RandomStringUtils.randomAlphabetic(10).toLowerCase();
+ volume.createBucket(mixBucketName);
+ List<LoadBucket> ozoneBuckets = new ArrayList<>(numClients);
+ for (int i = 0; i < numClients; i++) {
+ ozoneBuckets.add(new LoadBucket(volume.getBucket(mixBucketName),
+ conf));
}
- // This will terminate other threads too.
- isIOThreadRunning.set(false);
- LOG.info("Terminating IO thread:{}.", threadID);
+ RandomLoadGenerator loadGenerator =
+ new RandomLoadGenerator(buffer, ozoneBuckets);
+ loadExecutors.add(new LoadExecutors(numThreads, loadGenerator));
+
+ // Aged Load
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] nandakumar131 commented on a change in pull request
#438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load
generators to chaos testing.
Posted by GitBox <gi...@apache.org>.
nandakumar131 commented on a change in pull request #438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load generators to chaos testing.
URL: https://github.com/apache/hadoop-ozone/pull/438#discussion_r373058998
##########
File path: hadoop-ozone/fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/loadgenerators/AgedLoadGenerator.java
##########
@@ -0,0 +1,110 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.loadgenerators;
+
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.ozone.MiniOzoneLoadGenerator;
+import org.apache.hadoop.ozone.utils.LoadBucket;
+import org.apache.hadoop.ozone.utils.TestProbability;
+import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.util.Time;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.util.Optional;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * Aged Load Generator for Ozone.
+ *
+ * This Load Generator reads and write key to an Ozone bucket.
+ *
+ * The defautl writes to read ratio is 10:90.
+ */
+public class AgedLoadGenerator implements LoadGenerator {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(AgedLoadGenerator.class);
+ private static String fileSuffex = "aged";
+
+ private final AtomicInteger agedFileWrittenIndex;
+ private final AtomicInteger agedFileAllocationIndex;
+ private final LoadBucket agedLoadBucket;
+ private final TestProbability agedWriteProbability;
+ private final DataBuffer dataBuffer;
+
+ public AgedLoadGenerator(DataBuffer data, LoadBucket agedLoadBucket) {
+ this.dataBuffer = data;
+ this.agedFileWrittenIndex = new AtomicInteger(0);
+ this.agedFileAllocationIndex = new AtomicInteger(0);
+ this.agedLoadBucket = agedLoadBucket;
+ this.agedWriteProbability = TestProbability.valueOf(10);
+ }
+
+ public void startLoad(long runTimeMillis) {
+ long threadID = Thread.currentThread().getId();
+ LOG.info("AGED LOADGEN: Started Aged IO Thread:{}.", threadID);
+ long startTime = Time.monotonicNow();
+
+ while (Time.monotonicNow() < startTime + runTimeMillis) {
+
+ String keyName = null;
+ try {
+ if (agedWriteProbability.isTrue()) {
+ synchronized (agedFileAllocationIndex) {
Review comment:
Why do we need to synchronize here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] mukul1987 commented on a change in pull request
#438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load
generators to chaos testing.
Posted by GitBox <gi...@apache.org>.
mukul1987 commented on a change in pull request #438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load generators to chaos testing.
URL: https://github.com/apache/hadoop-ozone/pull/438#discussion_r392696766
##########
File path: hadoop-ozone/fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/MiniOzoneLoadGenerator.java
##########
@@ -47,212 +42,61 @@
LoggerFactory.getLogger(MiniOzoneLoadGenerator.class);
private static String keyNameDelimiter = "_";
-
- private ThreadPoolExecutor writeExecutor;
- private int numThreads;
- // number of buffer to be allocated, each is allocated with length which
- // is multiple of 2, each buffer is populated with random data.
- private int numBuffers;
- private List<ByteBuffer> buffers;
-
- private AtomicBoolean isIOThreadRunning;
-
- private final List<LoadBucket> ozoneBuckets;
-
- private final AtomicInteger agedFileWrittenIndex;
- private final ExecutorService agedFileExecutor;
- private final LoadBucket agedLoadBucket;
- private final TestProbability agedWriteProbability;
-
- private final ThreadPoolExecutor fsExecutor;
- private final LoadBucket fsBucket;
-
- MiniOzoneLoadGenerator(List<LoadBucket> bucket,
- LoadBucket agedLoadBucket, LoadBucket fsBucket,
- int numThreads, int numBuffers) {
- this.ozoneBuckets = bucket;
- this.numThreads = numThreads;
- this.numBuffers = numBuffers;
- this.writeExecutor = createExecutor();
-
- this.agedFileWrittenIndex = new AtomicInteger(0);
- this.agedFileExecutor = Executors.newSingleThreadExecutor();
- this.agedLoadBucket = agedLoadBucket;
- this.agedWriteProbability = TestProbability.valueOf(10);
-
- this.fsExecutor = createExecutor();
- this.fsBucket = fsBucket;
-
- this.isIOThreadRunning = new AtomicBoolean(false);
-
- // allocate buffers and populate random data.
- buffers = new ArrayList<>();
- for (int i = 0; i < numBuffers; i++) {
- int size = (int) StorageUnit.KB.toBytes(1 << i);
- ByteBuffer buffer = ByteBuffer.allocate(size);
- buffer.put(RandomUtils.nextBytes(size));
- buffers.add(buffer);
- }
- }
-
- private ThreadPoolExecutor createExecutor() {
- ThreadPoolExecutor executor = new ThreadPoolExecutor(numThreads, numThreads,
- 100, TimeUnit.SECONDS, new ArrayBlockingQueue<>(1024),
- new ThreadPoolExecutor.CallerRunsPolicy());
- executor.prestartAllCoreThreads();
- return executor;
-
- }
-
- // Start IO load on an Ozone bucket.
- private void load(long runTimeMillis) {
- long threadID = Thread.currentThread().getId();
- LOG.info("Started Mixed IO Thread:{}.", threadID);
- String threadName = Thread.currentThread().getName();
- long startTime = Time.monotonicNow();
-
- while (isIOThreadRunning.get() &&
- (Time.monotonicNow() < startTime + runTimeMillis)) {
- LoadBucket bucket =
- ozoneBuckets.get((int) (Math.random() * ozoneBuckets.size()));
- try {
- int index = RandomUtils.nextInt();
- ByteBuffer buffer = getBuffer(index);
- String keyName = getKeyName(index, threadName);
- bucket.writeKey(buffer, keyName);
-
- bucket.readKey(buffer, keyName);
-
- bucket.deleteKey(keyName);
- } catch (Exception e) {
- LOG.error("LOADGEN: Exiting due to exception", e);
- break;
- }
- }
- // This will terminate other threads too.
- isIOThreadRunning.set(false);
- LOG.info("Terminating IO thread:{}.", threadID);
- }
-
- private Optional<Integer> randomKeyToRead() {
- int currentIndex = agedFileWrittenIndex.get();
- return currentIndex != 0
- ? Optional.of(RandomUtils.nextInt(0, currentIndex))
- : Optional.empty();
- }
-
- private void startAgedLoad(long runTimeMillis) {
- long threadID = Thread.currentThread().getId();
- LOG.info("AGED LOADGEN: Started Aged IO Thread:{}.", threadID);
- String threadName = Thread.currentThread().getName();
- long startTime = Time.monotonicNow();
-
- while (isIOThreadRunning.get() &&
- (Time.monotonicNow() < startTime + runTimeMillis)) {
-
- String keyName = null;
- try {
- if (agedWriteProbability.isTrue()) {
- int index = agedFileWrittenIndex.getAndIncrement();
- ByteBuffer buffer = getBuffer(index);
- keyName = getKeyName(index, threadName);
-
- agedLoadBucket.writeKey(buffer, keyName);
- } else {
- Optional<Integer> index = randomKeyToRead();
- if (index.isPresent()) {
- ByteBuffer buffer = getBuffer(index.get());
- keyName = getKeyName(index.get(), threadName);
- agedLoadBucket.readKey(buffer, keyName);
- }
- }
- } catch (Throwable t) {
- LOG.error("AGED LOADGEN: {} Exiting due to exception", keyName, t);
- break;
- }
- }
- // This will terminate other threads too.
- isIOThreadRunning.set(false);
- LOG.info("Terminating IO thread:{}.", threadID);
- }
-
- // Start IO load on an Ozone bucket.
- private void startFsLoad(long runTimeMillis) {
- long threadID = Thread.currentThread().getId();
- LOG.info("Started Filesystem IO Thread:{}.", threadID);
- String threadName = Thread.currentThread().getName();
- long startTime = Time.monotonicNow();
-
- while (isIOThreadRunning.get() &&
- (Time.monotonicNow() < startTime + runTimeMillis)) {
- try {
- int index = RandomUtils.nextInt();
- ByteBuffer buffer = getBuffer(index);
- String keyName = getKeyName(index, threadName);
- fsBucket.writeKey(true, buffer, keyName);
-
- fsBucket.readKey(true, buffer, keyName);
-
- fsBucket.deleteKey(true, keyName);
- } catch (Exception e) {
- LOG.error("LOADGEN: Exiting due to exception", e);
- break;
- }
+ private final List<LoadExecutors> loadExecutors;
+
Review comment:
yes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] mukul1987 commented on a change in pull request
#438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load
generators to chaos testing.
Posted by GitBox <gi...@apache.org>.
mukul1987 commented on a change in pull request #438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load generators to chaos testing.
URL: https://github.com/apache/hadoop-ozone/pull/438#discussion_r396297765
##########
File path: hadoop-ozone/fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/loadgenerators/LoadExecutors.java
##########
@@ -0,0 +1,81 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.loadgenerators;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Load executors for Ozone, this class provides a plugable
+ * executor for different load generators.
+ */
+public class LoadExecutors {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(LoadExecutors.class);
+
+ private final LoadGenerator generator;
+ private final int numThreads;
+ private final ThreadPoolExecutor executor;
+ private final List<CompletableFuture<Void>> futures = new ArrayList<>();
+
+ public LoadExecutors(int numThreads, LoadGenerator generator) {
+ this.numThreads = numThreads;
+ this.generator = generator;
+ this.executor = new ThreadPoolExecutor(numThreads, numThreads,
+ 100, TimeUnit.SECONDS, new ArrayBlockingQueue<>(1024),
+ new ThreadPoolExecutor.CallerRunsPolicy());
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request
#438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load
generators to chaos testing.
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load generators to chaos testing.
URL: https://github.com/apache/hadoop-ozone/pull/438#discussion_r394929910
##########
File path: hadoop-ozone/fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/MiniOzoneLoadGenerator.java
##########
@@ -47,212 +42,61 @@
LoggerFactory.getLogger(MiniOzoneLoadGenerator.class);
private static String keyNameDelimiter = "_";
-
- private ThreadPoolExecutor writeExecutor;
- private int numThreads;
- // number of buffer to be allocated, each is allocated with length which
- // is multiple of 2, each buffer is populated with random data.
- private int numBuffers;
- private List<ByteBuffer> buffers;
-
- private AtomicBoolean isIOThreadRunning;
-
- private final List<LoadBucket> ozoneBuckets;
-
- private final AtomicInteger agedFileWrittenIndex;
- private final ExecutorService agedFileExecutor;
- private final LoadBucket agedLoadBucket;
- private final TestProbability agedWriteProbability;
-
- private final ThreadPoolExecutor fsExecutor;
- private final LoadBucket fsBucket;
-
- MiniOzoneLoadGenerator(List<LoadBucket> bucket,
- LoadBucket agedLoadBucket, LoadBucket fsBucket,
- int numThreads, int numBuffers) {
- this.ozoneBuckets = bucket;
- this.numThreads = numThreads;
- this.numBuffers = numBuffers;
- this.writeExecutor = createExecutor();
-
- this.agedFileWrittenIndex = new AtomicInteger(0);
- this.agedFileExecutor = Executors.newSingleThreadExecutor();
- this.agedLoadBucket = agedLoadBucket;
- this.agedWriteProbability = TestProbability.valueOf(10);
-
- this.fsExecutor = createExecutor();
- this.fsBucket = fsBucket;
-
- this.isIOThreadRunning = new AtomicBoolean(false);
-
- // allocate buffers and populate random data.
- buffers = new ArrayList<>();
- for (int i = 0; i < numBuffers; i++) {
- int size = (int) StorageUnit.KB.toBytes(1 << i);
- ByteBuffer buffer = ByteBuffer.allocate(size);
- buffer.put(RandomUtils.nextBytes(size));
- buffers.add(buffer);
- }
- }
-
- private ThreadPoolExecutor createExecutor() {
- ThreadPoolExecutor executor = new ThreadPoolExecutor(numThreads, numThreads,
- 100, TimeUnit.SECONDS, new ArrayBlockingQueue<>(1024),
- new ThreadPoolExecutor.CallerRunsPolicy());
- executor.prestartAllCoreThreads();
- return executor;
-
- }
-
- // Start IO load on an Ozone bucket.
- private void load(long runTimeMillis) {
- long threadID = Thread.currentThread().getId();
- LOG.info("Started Mixed IO Thread:{}.", threadID);
- String threadName = Thread.currentThread().getName();
- long startTime = Time.monotonicNow();
-
- while (isIOThreadRunning.get() &&
- (Time.monotonicNow() < startTime + runTimeMillis)) {
- LoadBucket bucket =
- ozoneBuckets.get((int) (Math.random() * ozoneBuckets.size()));
- try {
- int index = RandomUtils.nextInt();
- ByteBuffer buffer = getBuffer(index);
- String keyName = getKeyName(index, threadName);
- bucket.writeKey(buffer, keyName);
-
- bucket.readKey(buffer, keyName);
-
- bucket.deleteKey(keyName);
- } catch (Exception e) {
- LOG.error("LOADGEN: Exiting due to exception", e);
- break;
- }
- }
- // This will terminate other threads too.
- isIOThreadRunning.set(false);
- LOG.info("Terminating IO thread:{}.", threadID);
- }
-
- private Optional<Integer> randomKeyToRead() {
- int currentIndex = agedFileWrittenIndex.get();
- return currentIndex != 0
- ? Optional.of(RandomUtils.nextInt(0, currentIndex))
- : Optional.empty();
- }
-
- private void startAgedLoad(long runTimeMillis) {
- long threadID = Thread.currentThread().getId();
- LOG.info("AGED LOADGEN: Started Aged IO Thread:{}.", threadID);
- String threadName = Thread.currentThread().getName();
- long startTime = Time.monotonicNow();
-
- while (isIOThreadRunning.get() &&
- (Time.monotonicNow() < startTime + runTimeMillis)) {
-
- String keyName = null;
- try {
- if (agedWriteProbability.isTrue()) {
- int index = agedFileWrittenIndex.getAndIncrement();
- ByteBuffer buffer = getBuffer(index);
- keyName = getKeyName(index, threadName);
-
- agedLoadBucket.writeKey(buffer, keyName);
- } else {
- Optional<Integer> index = randomKeyToRead();
- if (index.isPresent()) {
- ByteBuffer buffer = getBuffer(index.get());
- keyName = getKeyName(index.get(), threadName);
- agedLoadBucket.readKey(buffer, keyName);
- }
- }
- } catch (Throwable t) {
- LOG.error("AGED LOADGEN: {} Exiting due to exception", keyName, t);
- break;
- }
- }
- // This will terminate other threads too.
- isIOThreadRunning.set(false);
- LOG.info("Terminating IO thread:{}.", threadID);
- }
-
- // Start IO load on an Ozone bucket.
- private void startFsLoad(long runTimeMillis) {
- long threadID = Thread.currentThread().getId();
- LOG.info("Started Filesystem IO Thread:{}.", threadID);
- String threadName = Thread.currentThread().getName();
- long startTime = Time.monotonicNow();
-
- while (isIOThreadRunning.get() &&
- (Time.monotonicNow() < startTime + runTimeMillis)) {
- try {
- int index = RandomUtils.nextInt();
- ByteBuffer buffer = getBuffer(index);
- String keyName = getKeyName(index, threadName);
- fsBucket.writeKey(true, buffer, keyName);
-
- fsBucket.readKey(true, buffer, keyName);
-
- fsBucket.deleteKey(true, keyName);
- } catch (Exception e) {
- LOG.error("LOADGEN: Exiting due to exception", e);
- break;
- }
+ private final List<LoadExecutors> loadExecutors;
+
+ MiniOzoneLoadGenerator(OzoneVolume volume, int numClients, int numThreads,
+ int numBuffers, OzoneConfiguration conf)
+ throws Exception {
+ DataBuffer buffer = new DataBuffer(numBuffers);
+ loadExecutors = new ArrayList<>();
+
+ // Random Load
+ String mixBucketName = RandomStringUtils.randomAlphabetic(10).toLowerCase();
+ volume.createBucket(mixBucketName);
+ List<LoadBucket> ozoneBuckets = new ArrayList<>(numClients);
+ for (int i = 0; i < numClients; i++) {
+ ozoneBuckets.add(new LoadBucket(volume.getBucket(mixBucketName),
+ conf));
}
- // This will terminate other threads too.
- isIOThreadRunning.set(false);
- LOG.info("Terminating IO thread:{}.", threadID);
+ RandomLoadGenerator loadGenerator =
+ new RandomLoadGenerator(buffer, ozoneBuckets);
+ loadExecutors.add(new LoadExecutors(numThreads, loadGenerator));
+
+ // Aged Load
+ String agedBucketName =
+ RandomStringUtils.randomAlphabetic(10).toLowerCase();
+ volume.createBucket(agedBucketName);
+ LoadBucket agedLoadBucket =
+ new LoadBucket(volume.getBucket(agedBucketName), conf);
+ AgedLoadGenerator agedLoadGenerator =
+ new AgedLoadGenerator(buffer, agedLoadBucket);
+ loadExecutors.add(new LoadExecutors(numThreads, agedLoadGenerator));
+
+ //Filesystem Load
+ String fsBucketName =
+ RandomStringUtils.randomAlphabetic(10).toLowerCase();
+
+ volume.createBucket(fsBucketName);
+ LoadBucket fsBucket =
+ new LoadBucket(volume.getBucket(fsBucketName), conf);
+ FilesystemLoadGenerator filesystemLoadGenerator =
+ new FilesystemLoadGenerator(buffer, fsBucket);
+ loadExecutors.add(new LoadExecutors(numThreads, filesystemLoadGenerator));
}
void startIO(long time, TimeUnit timeUnit) {
- List<CompletableFuture<Void>> writeFutures = new ArrayList<>();
- LOG.info("Starting MiniOzoneLoadGenerator for time {}:{} with {} buffers " +
- "and {} threads", time, timeUnit, numBuffers, numThreads);
- if (isIOThreadRunning.compareAndSet(false, true)) {
- // Start the IO thread
- for (int i = 0; i < numThreads; i++) {
- writeFutures.add(
- CompletableFuture.runAsync(() -> load(timeUnit.toMillis(time)),
- writeExecutor));
- }
-
- for (int i = 0; i < numThreads; i++) {
- writeFutures.add(
- CompletableFuture.runAsync(() -> startAgedLoad(
- timeUnit.toMillis(time)), agedFileExecutor));
- }
-
- for (int i = 0; i < numThreads; i++) {
- writeFutures.add(
- CompletableFuture.runAsync(() -> startFsLoad(
- timeUnit.toMillis(time)), fsExecutor));
- }
-
- // Wait for IO to complete
- for (CompletableFuture<Void> f : writeFutures) {
- try {
- f.get();
- } catch (Throwable t) {
- LOG.error("startIO failed with exception", t);
- }
- }
- }
- }
-
- public void shutdownLoadGenerator() {
- try {
- writeExecutor.shutdown();
- writeExecutor.awaitTermination(1, TimeUnit.DAYS);
- } catch (Exception e) {
- LOG.error("error while closing ", e);
- }
+ LOG.info("Starting MiniOzoneLoadGenerator for time {}:{}", time, timeUnit);
+ long runTime = timeUnit.toMillis(time);
+ // start and wait for executors to finish
+ loadExecutors.forEach(le -> le.startLoad(runTime));
+ loadExecutors.forEach(LoadExecutors::waitForCompletion);
}
- private ByteBuffer getBuffer(int keyIndex) {
- return buffers.get(keyIndex % numBuffers);
+ void shutdownLoadGenerator() {
+ loadExecutors.forEach(LoadExecutors::shutdown);
}
- private String getKeyName(int keyIndex, String threadName) {
- return threadName + keyNameDelimiter + keyIndex;
+ public static String getKeyName(int keyIndex, String prefix) {
+ return prefix + keyNameDelimiter + keyIndex;
}
Review comment:
Nit: can be moved to `LoadGenerator`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request
#438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load
generators to chaos testing.
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load generators to chaos testing.
URL: https://github.com/apache/hadoop-ozone/pull/438#discussion_r394911832
##########
File path: hadoop-ozone/fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/loadgenerators/FilesystemLoadGenerator.java
##########
@@ -0,0 +1,82 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.loadgenerators;
+
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.ozone.MiniOzoneLoadGenerator;
+import org.apache.hadoop.ozone.utils.LoadBucket;
+import org.apache.hadoop.util.ExitUtil;
+import org.apache.hadoop.util.Time;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+
+/**
+ * Filesystem load generator for Ozone.
+ *
+ * This load generator read, writes and deletes data using the filesystem
+ * apis.
+ */
+public class FilesystemLoadGenerator implements LoadGenerator {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(FilesystemLoadGenerator.class);
+
+
+ private final LoadBucket fsBucket;
+ private final DataBuffer dataBuffer;
+
+ public FilesystemLoadGenerator(DataBuffer dataBuffer, LoadBucket fsBucket) {
+ this.dataBuffer = dataBuffer;
+ this.fsBucket = fsBucket;
+ }
+
+ // Start IO load on an Ozone bucket.
+ public void startLoad(long runTimeMillis) {
+ long threadID = Thread.currentThread().getId();
+ LOG.info("Started Filesystem IO Thread:{}.", threadID);
+ String threadName = Thread.currentThread().getName();
+ long startTime = Time.monotonicNow();
+
+ while (Time.monotonicNow() < startTime + runTimeMillis) {
+ try {
+ int index = RandomUtils.nextInt();
Review comment:
Would it make sense to use `ThreadLocalRandom` (also in other generators) to avoid contention among load generator threads?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] mukul1987 merged pull request #438: HDDS-2878.
Refactor MiniOzoneLoadGenerator to add more load generators to chaos testing.
Posted by GitBox <gi...@apache.org>.
mukul1987 merged pull request #438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load generators to chaos testing.
URL: https://github.com/apache/hadoop-ozone/pull/438
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] mukul1987 commented on issue #438: HDDS-2878.
Refactor MiniOzoneLoadGenerator to add more load generators to chaos testing.
Posted by GitBox <gi...@apache.org>.
mukul1987 commented on issue #438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load generators to chaos testing.
URL: https://github.com/apache/hadoop-ozone/pull/438#issuecomment-599243753
/ready
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request
#438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load
generators to chaos testing.
Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on a change in pull request #438: HDDS-2878. Refactor MiniOzoneLoadGenerator to add more load generators to chaos testing.
URL: https://github.com/apache/hadoop-ozone/pull/438#discussion_r394483511
##########
File path: hadoop-ozone/fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/loadgenerators/DataBuffer.java
##########
@@ -0,0 +1,53 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.loadgenerators;
+
+import org.apache.commons.lang3.RandomUtils;
+import org.apache.hadoop.conf.StorageUnit;
+
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * List of buffers used by the load generators.
+ */
+public class DataBuffer {
+ private List<ByteBuffer> buffers;
+ // number of buffer to be allocated, each is allocated with length which
+ // is multiple of 2, each buffer is populated with random data.
+ private int numBuffers;
+
+ public DataBuffer(int numBuffers) {
+ // allocate buffers and populate random data.
+ this.numBuffers = numBuffers;
+ this.buffers = new ArrayList<>();
+ for (int i = 0; i < numBuffers; i++) {
Review comment:
Can we generate only 1 random byte array of size 1<<(numBuffers+10) and use List of ByteBuffer to wrap 0~1<<i part of the same array with wrap(byte[] array, int offset, int length)?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org