You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/05/04 15:50:59 UTC
[GitHub] [hadoop-ozone] rakeshadr opened a new pull request #895: HDDS-3518: Add a freon generator to create directory tree and create …
rakeshadr opened a new pull request #895:
URL: https://github.com/apache/hadoop-ozone/pull/895
## What changes were proposed in this pull request?
This Jira proposes to add a functionality to freon tool to create directories in tree structure.
**Proposed structure:**
/*
Nested directories will be created like this,
suppose you pass depth=3, span=3 and number of tests=2
Directory Structure:-
|-- Dir111
|
|-- Dir11 --|-- Dir112
| |
| |-- Dir113
|
|
| |-- Dir121
| |
Dir1 --|-- Dir12 --|-- Dir122
| |
| |-- Dir123
|
|
| |-- Dir131
| |
|-- Dir13 --|-- Dir132
|
|-- Dir133
In each directory 'c' number of files to be written in each directory with file size 'g' will be created.
*/
This is basically useful to determine the performance evaluation of ozone, when creating defined number of nested directories and also create given number of files into each directory.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-3518
## How was this patch tested?
Added unit test case..
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] rakeshadr commented on a change in pull request #895: HDDS-3518: Add a freon generator to create directory tree and create …
Posted by GitBox <gi...@apache.org>.
rakeshadr commented on a change in pull request #895:
URL: https://github.com/apache/hadoop-ozone/pull/895#discussion_r420521952
##########
File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/HadoopDirTreeGenerator.java
##########
@@ -0,0 +1,218 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+import org.apache.commons.lang3.RandomStringUtils;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine.Command;
+import picocli.CommandLine.Option;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Random;
+import java.util.concurrent.Callable;
+
+/**
+ * Directory & File Generator tool to test OM performance.
+ */
+@Command(name = "dtsg",
+ aliases = "dfs-tree-generator",
+ description =
+ "Create nested directories and create given number of files in each " +
+ "dir in any dfs compatible file system.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class HadoopDirTreeGenerator extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(HadoopDirTreeGenerator.class);
+
+ @Option(names = {"-r", "--rpath"},
+ description = "Hadoop FS directory system path",
Review comment:
Thanks @xiaoyuyao for the reviews.
Yes, its "Hadoop FS root path". I will modify to make it clear.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on pull request #895: HDDS-3518: Add a freon generator to create directory tree and create …
Posted by GitBox <gi...@apache.org>.
elek commented on pull request #895:
URL: https://github.com/apache/hadoop-ozone/pull/895#issuecomment-638241382
Thanks @rakeshadr the update.
Unfortunately when I checked your update 3 more opinions came in to my mind (Sorry). Let me share them here:
1. You introduced an atomic counter `totalDirsCnt`. But we already have multiple counters `successCounter`, `failureCounter`, `failureCounter` in the base class. The only problem is that his patch doesn't follow the existing approach where the overall test can be splitted to small tasks (which can be measured one by one). Personally I think it's possible even in this use case but the generation should be refactored.
2. For example you used some `System.out.println` in this patch. We had some problems with different printing earlier (sometimes freon is executed from shell sometimes from command line) `BaseFreonGenerator.printReport` already handle this, but this functionality is not used.
3. We already have a `HadoopNestedDirGenerator`, half of the functionality of this patch is already there. For me it seems to be better to combine the two tests.
But I have two additional notes:
1. I might be wrong with any point, I can be convinced.
2. Usually I am more interested about the agreement in long-term directions, I think we can merge this patch. It works well and (if we agree) we can further improve it (maybe merge with the `HadoopNestedDirGenerator`, maybe use the same structure as the other Freon tests.) --> **Let me merge it now, and let's continue the discussion**
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] rakeshadr edited a comment on pull request #895: HDDS-3518: Add a freon generator to create directory tree and create …
Posted by GitBox <gi...@apache.org>.
rakeshadr edited a comment on pull request #895:
URL: https://github.com/apache/hadoop-ozone/pull/895#issuecomment-635135367
Thanks a lot @elek for the comments with detailed explanation.
> 1. I guess you would like to test the correctness and not interested about the performance. But I love the numbers and I think it's always good to have.
Sure, I will add the timer and helpful during perf evaluation.
> 2. There is a `prefix` field in the `BaseFreonGenerator`. This is a random string, which can be specified.
> (One easy way to use it to generate dirs in `rpath` + `/` + `prefix`)
This tool provides '--nameLen' option to the users to define the length of the random name of the directory. Another point is, `/` + `prefix` will add additional depth(sub-dir), which HadoopDirTreeGenerator#makeDirWithGivenNumberOfFiles() is already doing it.
I am not finding any usage of 'prefix' in this tool and skip this comment. Does that make sense to you? Thanks!
> 3. We have a `ContentGenerator` class which can do the the random key generation. I think it can be used here... If you see some limitation of it, we can improve it as well...
Yes, I will use ContentGenerator for writing the content.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #895: HDDS-3518: Add a freon generator to create directory tree and create …
Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on a change in pull request #895:
URL: https://github.com/apache/hadoop-ozone/pull/895#discussion_r420316026
##########
File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/HadoopDirTreeGenerator.java
##########
@@ -0,0 +1,218 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+import org.apache.commons.lang3.RandomStringUtils;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine.Command;
+import picocli.CommandLine.Option;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Random;
+import java.util.concurrent.Callable;
+
+/**
+ * Directory & File Generator tool to test OM performance.
+ */
+@Command(name = "dtsg",
+ aliases = "dfs-tree-generator",
+ description =
+ "Create nested directories and create given number of files in each " +
+ "dir in any dfs compatible file system.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class HadoopDirTreeGenerator extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(HadoopDirTreeGenerator.class);
+
+ @Option(names = {"-r", "--rpath"},
+ description = "Hadoop FS directory system path",
Review comment:
"Hadoop FS directory system path" is a bit confusing, do you mean "Hadoop FS root path"?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on pull request #895: HDDS-3518: Add a freon generator to create directory tree and create …
Posted by GitBox <gi...@apache.org>.
elek commented on pull request #895:
URL: https://github.com/apache/hadoop-ozone/pull/895#issuecomment-638242436
Merged. Thank you the contribution @rakeshadr (and thanks for the patience for my long comments ;-) )
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] rakeshadr commented on pull request #895: HDDS-3518: Add a freon generator to create directory tree and create …
Posted by GitBox <gi...@apache.org>.
rakeshadr commented on pull request #895:
URL: https://github.com/apache/hadoop-ozone/pull/895#issuecomment-635135367
Thanks a lot @elek for the comments with detailed explanation.
> 1. I guess you would like to test the correctness and not interested about the performance. But I love the numbers and I think it's always good to have.
Sure, I will add the timer and helpful during perf evaluation.
> 1. There is a `prefix` field in the `BaseFreonGenerator`. This is a random string, which can be specified.
> (One easy way to use it to generate dirs in `rpath` + `/` + `prefix`)
This tool provides '--nameLen' option to the users to define the length of the random name of the directory. Another point is, `/` + `prefix` will add additional depth(sub-dir), which HadoopDirTreeGenerator#makeDirWithGivenNumberOfFiles() is already doing it.
I am not finding any usage of 'prefix' in this tool and skip this comment. Does that make sense to you? Thanks!
> 1. We have a `ContentGenerator` class which can do the the random key generation. I think it can be used here... If you see some limitation of it, we can improve it as well...
Yes, will use ContentGenerator for writing the content.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] rakeshadr commented on a change in pull request #895: HDDS-3518: Add a freon generator to create directory tree and create …
Posted by GitBox <gi...@apache.org>.
rakeshadr commented on a change in pull request #895:
URL: https://github.com/apache/hadoop-ozone/pull/895#discussion_r421199394
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/freon/TestHadoopDirTreeGenerator.java
##########
@@ -0,0 +1,182 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.MiniOzoneCluster;
+import org.apache.hadoop.ozone.client.ObjectStore;
+import org.apache.hadoop.ozone.client.OzoneClientFactory;
+import org.apache.hadoop.ozone.client.OzoneVolume;
+import org.apache.hadoop.ozone.om.OzoneManager;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.apache.ratis.server.impl.RaftServerImpl;
+import org.apache.ratis.server.raftlog.RaftLog;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.slf4j.event.Level;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.net.URI;
+
+/**
+ * Test for HadoopNestedDirTreeGenerator.
+ */
+public class TestHadoopDirTreeGenerator {
+
+ private String path;
+
+ /**
+ * Create a MiniDFSCluster for testing.
+ *
+ * @throws IOException
+ */
+ @Before
+ public void setup() {
+ path = GenericTestUtils
+ .getTempPath(TestOzoneClientKeyGenerator.class.getSimpleName());
+ GenericTestUtils.setLogLevel(RaftLog.LOG, Level.DEBUG);
+ GenericTestUtils.setLogLevel(RaftServerImpl.LOG, Level.DEBUG);
+ File baseDir = new File(path);
+ baseDir.mkdirs();
+ }
+
+ /**
+ * Shutdown MiniDFSCluster.
+ */
+ private void shutdown(MiniOzoneCluster cluster) throws IOException {
+ if (cluster != null) {
+ cluster.shutdown();
+ FileUtils.deleteDirectory(new File(path));
+ }
+ }
+
+ private MiniOzoneCluster startCluster(OzoneConfiguration conf)
+ throws Exception {
+ if (conf == null) {
+ conf = new OzoneConfiguration();
+ }
+ MiniOzoneCluster cluster = MiniOzoneCluster.newBuilder(conf)
+ .setNumDatanodes(5)
+ .build();
+
+ cluster.waitForClusterToBeReady();
+ cluster.waitTobeOutOfSafeMode();
+ return cluster;
+ }
+
+ @Test
+ public void testNestedDirTreeGeneration() throws Exception {
+ OzoneConfiguration conf = new OzoneConfiguration();
+ MiniOzoneCluster cluster = startCluster(conf);
+ ObjectStore store =
Review comment:
Done!
##########
File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/HadoopDirTreeGenerator.java
##########
@@ -0,0 +1,218 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+import org.apache.commons.lang3.RandomStringUtils;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine.Command;
+import picocli.CommandLine.Option;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Random;
+import java.util.concurrent.Callable;
+
+/**
+ * Directory & File Generator tool to test OM performance.
+ */
+@Command(name = "dtsg",
+ aliases = "dfs-tree-generator",
+ description =
+ "Create nested directories and create given number of files in each " +
+ "dir in any dfs compatible file system.",
+ versionProvider = HddsVersionProvider.class,
+ mixinStandardHelpOptions = true,
+ showDefaultValues = true)
+public class HadoopDirTreeGenerator extends BaseFreonGenerator
+ implements Callable<Void> {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(HadoopDirTreeGenerator.class);
+
+ @Option(names = {"-r", "--rpath"},
+ description = "Hadoop FS directory system path",
Review comment:
Done!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] rakeshadr commented on a change in pull request #895: HDDS-3518: Add a freon generator to create directory tree and create …
Posted by GitBox <gi...@apache.org>.
rakeshadr commented on a change in pull request #895:
URL: https://github.com/apache/hadoop-ozone/pull/895#discussion_r420522148
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/freon/TestHadoopDirTreeGenerator.java
##########
@@ -0,0 +1,182 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.MiniOzoneCluster;
+import org.apache.hadoop.ozone.client.ObjectStore;
+import org.apache.hadoop.ozone.client.OzoneClientFactory;
+import org.apache.hadoop.ozone.client.OzoneVolume;
+import org.apache.hadoop.ozone.om.OzoneManager;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.apache.ratis.server.impl.RaftServerImpl;
+import org.apache.ratis.server.raftlog.RaftLog;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.slf4j.event.Level;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.net.URI;
+
+/**
+ * Test for HadoopNestedDirTreeGenerator.
+ */
+public class TestHadoopDirTreeGenerator {
+
+ private String path;
+
+ /**
+ * Create a MiniDFSCluster for testing.
+ *
+ * @throws IOException
+ */
+ @Before
+ public void setup() {
+ path = GenericTestUtils
+ .getTempPath(TestOzoneClientKeyGenerator.class.getSimpleName());
+ GenericTestUtils.setLogLevel(RaftLog.LOG, Level.DEBUG);
+ GenericTestUtils.setLogLevel(RaftServerImpl.LOG, Level.DEBUG);
+ File baseDir = new File(path);
+ baseDir.mkdirs();
+ }
+
+ /**
+ * Shutdown MiniDFSCluster.
+ */
+ private void shutdown(MiniOzoneCluster cluster) throws IOException {
+ if (cluster != null) {
+ cluster.shutdown();
+ FileUtils.deleteDirectory(new File(path));
+ }
+ }
+
+ private MiniOzoneCluster startCluster(OzoneConfiguration conf)
+ throws Exception {
+ if (conf == null) {
+ conf = new OzoneConfiguration();
+ }
+ MiniOzoneCluster cluster = MiniOzoneCluster.newBuilder(conf)
+ .setNumDatanodes(5)
+ .build();
+
+ cluster.waitForClusterToBeReady();
+ cluster.waitTobeOutOfSafeMode();
+ return cluster;
+ }
+
+ @Test
+ public void testNestedDirTreeGeneration() throws Exception {
+ OzoneConfiguration conf = new OzoneConfiguration();
+ MiniOzoneCluster cluster = startCluster(conf);
+ ObjectStore store =
Review comment:
Sure, I will update this in next PR commit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek merged pull request #895: HDDS-3518: Add a freon generator to create directory tree and create …
Posted by GitBox <gi...@apache.org>.
elek merged pull request #895:
URL: https://github.com/apache/hadoop-ozone/pull/895
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #895: HDDS-3518: Add a freon generator to create directory tree and create …
Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on a change in pull request #895:
URL: https://github.com/apache/hadoop-ozone/pull/895#discussion_r420312123
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/freon/TestHadoopDirTreeGenerator.java
##########
@@ -0,0 +1,182 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.MiniOzoneCluster;
+import org.apache.hadoop.ozone.client.ObjectStore;
+import org.apache.hadoop.ozone.client.OzoneClientFactory;
+import org.apache.hadoop.ozone.client.OzoneVolume;
+import org.apache.hadoop.ozone.om.OzoneManager;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.apache.ratis.server.impl.RaftServerImpl;
+import org.apache.ratis.server.raftlog.RaftLog;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.slf4j.event.Level;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.net.URI;
+
+/**
+ * Test for HadoopNestedDirTreeGenerator.
+ */
+public class TestHadoopDirTreeGenerator {
+
+ private String path;
+
+ /**
+ * Create a MiniDFSCluster for testing.
+ *
+ * @throws IOException
+ */
+ @Before
+ public void setup() {
+ path = GenericTestUtils
+ .getTempPath(TestOzoneClientKeyGenerator.class.getSimpleName());
+ GenericTestUtils.setLogLevel(RaftLog.LOG, Level.DEBUG);
+ GenericTestUtils.setLogLevel(RaftServerImpl.LOG, Level.DEBUG);
+ File baseDir = new File(path);
+ baseDir.mkdirs();
+ }
+
+ /**
+ * Shutdown MiniDFSCluster.
+ */
+ private void shutdown(MiniOzoneCluster cluster) throws IOException {
+ if (cluster != null) {
+ cluster.shutdown();
+ FileUtils.deleteDirectory(new File(path));
+ }
+ }
+
+ private MiniOzoneCluster startCluster(OzoneConfiguration conf)
+ throws Exception {
+ if (conf == null) {
+ conf = new OzoneConfiguration();
+ }
+ MiniOzoneCluster cluster = MiniOzoneCluster.newBuilder(conf)
+ .setNumDatanodes(5)
+ .build();
+
+ cluster.waitForClusterToBeReady();
+ cluster.waitTobeOutOfSafeMode();
+ return cluster;
+ }
+
+ @Test
+ public void testNestedDirTreeGeneration() throws Exception {
+ OzoneConfiguration conf = new OzoneConfiguration();
+ MiniOzoneCluster cluster = startCluster(conf);
+ ObjectStore store =
Review comment:
Can we wrap this with try {} finally {shutdown (cluster)} so that the cluster is ensured to shutdown if any verification failed?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org