You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/07/29 07:43:04 UTC

[GitHub] [hadoop-ozone] aryangupta1998 opened a new pull request #1266: HDDS-4034. Add Unit Test for HadoopNestedDirGenerator.

aryangupta1998 opened a new pull request #1266:
URL: https://github.com/apache/hadoop-ozone/pull/1266


   ## What changes were proposed in this pull request?
   
   Added Unit Test for HadoopNestedDirGenerator.
   
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-4034
   
   ## How was this patch tested?
   
   Tested Manually.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org

[GitHub] [hadoop-ozone] bshashikant commented on a change in pull request #1266: HDDS-4034. Add Unit Test for HadoopNestedDirGenerator.

Posted by GitBox <gi...@apache.org>.
bshashikant commented on a change in pull request #1266:
URL: https://github.com/apache/hadoop-ozone/pull/1266#discussion_r462877851



##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/freon/TestHadoopNestedDirGenerator.java
##########
@@ -0,0 +1,208 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership.  The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.MiniOzoneCluster;
+import org.apache.hadoop.ozone.client.ObjectStore;
+import org.apache.hadoop.ozone.client.OzoneClientFactory;
+import org.apache.hadoop.ozone.client.OzoneVolume;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.apache.ratis.server.impl.RaftServerImpl;
+import org.apache.ratis.server.raftlog.RaftLog;
+import java.util.LinkedList;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.net.URI;
+
+/**
+ * Test for HadoopNestedDirGenerator.
+ */
+
+public class TestHadoopNestedDirGenerator {
+
+  private String path;
+  private OzoneConfiguration conf = null;
+  private MiniOzoneCluster cluster = null;
+  private ObjectStore store = null;
+  private static final Logger LOG =
+          LoggerFactory.getLogger(TestHadoopNestedDirGenerator.class);
+  @Before
+    public void setup() {
+    path = GenericTestUtils
+                .getTempPath(TestOzoneClientKeyGenerator.class.getSimpleName());
+    GenericTestUtils.setLogLevel(RaftLog.LOG, Level.DEBUG);
+    GenericTestUtils.setLogLevel(RaftServerImpl.LOG, Level.DEBUG);
+    File baseDir = new File(path);
+    baseDir.mkdirs();
+  }
+
+    /**
+     * Shutdown MiniDFSCluster.
+     */
+
+  private void shutdown() throws IOException {
+    if (cluster != null) {
+      cluster.shutdown();
+      FileUtils.deleteDirectory(new File(path));
+    }
+  }
+
+    /**
+     * Create a MiniDFSCluster for testing.
+     *
+     * @throws IOException
+     */
+
+    /**
+     * Below we have 5 datanodes because only 5 tests are activated
+     * As you need more tests increase the datanodes accordingly.

Review comment:
       Thanks @aryangupta1998 . Why do the no of datanodes matter here? can you explain?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org

[GitHub] [hadoop-ozone] bshashikant commented on pull request #1266: HDDS-4034. Add Unit Test for HadoopNestedDirGenerator.

Posted by GitBox <gi...@apache.org>.
bshashikant commented on pull request #1266:
URL: https://github.com/apache/hadoop-ozone/pull/1266#issuecomment-671850092


   Thanks @aryangupta1998 for the contribution.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org

[GitHub] [hadoop-ozone] bshashikant merged pull request #1266: HDDS-4034. Add Unit Test for HadoopNestedDirGenerator.

Posted by GitBox <gi...@apache.org>.
bshashikant merged pull request #1266:
URL: https://github.com/apache/hadoop-ozone/pull/1266


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org

[GitHub] [hadoop-ozone] aryangupta1998 commented on a change in pull request #1266: HDDS-4034. Add Unit Test for HadoopNestedDirGenerator.

Posted by GitBox <gi...@apache.org>.
aryangupta1998 commented on a change in pull request #1266:
URL: https://github.com/apache/hadoop-ozone/pull/1266#discussion_r462904139



##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/freon/TestHadoopNestedDirGenerator.java
##########
@@ -0,0 +1,208 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership.  The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.MiniOzoneCluster;
+import org.apache.hadoop.ozone.client.ObjectStore;
+import org.apache.hadoop.ozone.client.OzoneClientFactory;
+import org.apache.hadoop.ozone.client.OzoneVolume;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.apache.ratis.server.impl.RaftServerImpl;
+import org.apache.ratis.server.raftlog.RaftLog;
+import java.util.LinkedList;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.net.URI;
+
+/**
+ * Test for HadoopNestedDirGenerator.
+ */
+
+public class TestHadoopNestedDirGenerator {
+
+  private String path;
+  private OzoneConfiguration conf = null;
+  private MiniOzoneCluster cluster = null;
+  private ObjectStore store = null;
+  private static final Logger LOG =
+          LoggerFactory.getLogger(TestHadoopNestedDirGenerator.class);
+  @Before
+    public void setup() {
+    path = GenericTestUtils
+                .getTempPath(TestOzoneClientKeyGenerator.class.getSimpleName());
+    GenericTestUtils.setLogLevel(RaftLog.LOG, Level.DEBUG);
+    GenericTestUtils.setLogLevel(RaftServerImpl.LOG, Level.DEBUG);
+    File baseDir = new File(path);
+    baseDir.mkdirs();
+  }
+
+    /**
+     * Shutdown MiniDFSCluster.
+     */
+
+  private void shutdown() throws IOException {
+    if (cluster != null) {
+      cluster.shutdown();
+      FileUtils.deleteDirectory(new File(path));
+    }
+  }
+
+    /**
+     * Create a MiniDFSCluster for testing.
+     *
+     * @throws IOException
+     */
+
+    /**
+     * Below we have 5 datanodes because only 5 tests are activated
+     * As you need more tests increase the datanodes accordingly.

Review comment:
       Thanks @bshashikant. Thanks for pointing out and sorry for the mistake. I have removed this javadoc.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org

[GitHub] [hadoop-ozone] codecov-commenter commented on pull request #1266: HDDS-4034. Add Unit Test for HadoopNestedDirGenerator.

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #1266:
URL: https://github.com/apache/hadoop-ozone/pull/1266#issuecomment-665093227


   # [Codecov](https://codecov.io/gh/apache/hadoop-ozone/pull/1266?src=pr&el=h1) Report
   > Merging [#1266](https://codecov.io/gh/apache/hadoop-ozone/pull/1266?src=pr&el=desc) into [master](https://codecov.io/gh/apache/hadoop-ozone/commit/715aed2d158d2c6708af8b6a9b8270766103ee52&el=desc) will **increase** coverage by `0.30%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/hadoop-ozone/pull/1266/graphs/tree.svg?width=650&height=150&src=pr&token=5YeeptJMby)](https://codecov.io/gh/apache/hadoop-ozone/pull/1266?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #1266      +/-   ##
   ============================================
   + Coverage     73.54%   73.85%   +0.30%     
   - Complexity    10043    10148     +105     
   ============================================
     Files           978      974       -4     
     Lines         49894    50186     +292     
     Branches       4845     4891      +46     
   ============================================
   + Hits          36696    37066     +370     
   + Misses        10887    10781     -106     
   - Partials       2311     2339      +28     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/hadoop-ozone/pull/1266?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [...x509/certificates/utils/SelfSignedCertificate.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1266/diff?src=pr&el=tree#diff-aGFkb29wLWhkZHMvZnJhbWV3b3JrL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9oYWRvb3AvaGRkcy9zZWN1cml0eS94NTA5L2NlcnRpZmljYXRlcy91dGlscy9TZWxmU2lnbmVkQ2VydGlmaWNhdGUuamF2YQ==) | `76.59% <0.00%> (-15.95%)` | `7.00% <0.00%> (+1.00%)` | :arrow_down: |
   | [...er/common/transport/server/GrpcXceiverService.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1266/diff?src=pr&el=tree#diff-aGFkb29wLWhkZHMvY29udGFpbmVyLXNlcnZpY2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2hhZG9vcC9vem9uZS9jb250YWluZXIvY29tbW9uL3RyYW5zcG9ydC9zZXJ2ZXIvR3JwY1hjZWl2ZXJTZXJ2aWNlLmphdmE=) | `70.00% <0.00%> (-10.00%)` | `3.00% <0.00%> (ø%)` | |
   | [...g/apache/hadoop/hdds/utils/ResourceLimitCache.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1266/diff?src=pr&el=tree#diff-aGFkb29wLWhkZHMvY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9oYWRvb3AvaGRkcy91dGlscy9SZXNvdXJjZUxpbWl0Q2FjaGUuamF2YQ==) | `84.37% <0.00%> (-9.38%)` | `7.00% <0.00%> (-1.00%)` | |
   | [.../transport/server/ratis/ContainerStateMachine.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1266/diff?src=pr&el=tree#diff-aGFkb29wLWhkZHMvY29udGFpbmVyLXNlcnZpY2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2hhZG9vcC9vem9uZS9jb250YWluZXIvY29tbW9uL3RyYW5zcG9ydC9zZXJ2ZXIvcmF0aXMvQ29udGFpbmVyU3RhdGVNYWNoaW5lLmphdmE=) | `71.07% <0.00%> (-5.61%)` | `62.00% <0.00%> (-4.00%)` | |
   | [...ozone/container/ozoneimpl/ContainerController.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1266/diff?src=pr&el=tree#diff-aGFkb29wLWhkZHMvY29udGFpbmVyLXNlcnZpY2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2hhZG9vcC9vem9uZS9jb250YWluZXIvb3pvbmVpbXBsL0NvbnRhaW5lckNvbnRyb2xsZXIuamF2YQ==) | `63.15% <0.00%> (-5.27%)` | `11.00% <0.00%> (-1.00%)` | |
   | [...p/ozone/container/common/utils/ContainerCache.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1266/diff?src=pr&el=tree#diff-aGFkb29wLWhkZHMvY29udGFpbmVyLXNlcnZpY2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2hhZG9vcC9vem9uZS9jb250YWluZXIvY29tbW9uL3V0aWxzL0NvbnRhaW5lckNhY2hlLmphdmE=) | `91.54% <0.00%> (-4.82%)` | `11.00% <0.00%> (ø%)` | |
   | [.../common/volume/RoundRobinVolumeChoosingPolicy.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1266/diff?src=pr&el=tree#diff-aGFkb29wLWhkZHMvY29udGFpbmVyLXNlcnZpY2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2hhZG9vcC9vem9uZS9jb250YWluZXIvY29tbW9uL3ZvbHVtZS9Sb3VuZFJvYmluVm9sdW1lQ2hvb3NpbmdQb2xpY3kuamF2YQ==) | `80.95% <0.00%> (-4.77%)` | `5.00% <0.00%> (-1.00%)` | |
   | [...rg/apache/hadoop/hdds/client/ContainerBlockID.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1266/diff?src=pr&el=tree#diff-aGFkb29wLWhkZHMvY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9oYWRvb3AvaGRkcy9jbGllbnQvQ29udGFpbmVyQmxvY2tJRC5qYXZh) | `73.91% <0.00%> (-4.35%)` | `9.00% <0.00%> (-1.00%)` | |
   | [...che/hadoop/ozone/om/helpers/OmKeyLocationInfo.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1266/diff?src=pr&el=tree#diff-aGFkb29wLW96b25lL2NvbW1vbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaGFkb29wL296b25lL29tL2hlbHBlcnMvT21LZXlMb2NhdGlvbkluZm8uamF2YQ==) | `80.95% <0.00%> (-4.24%)` | `24.00% <0.00%> (-2.00%)` | |
   | [...apache/hadoop/hdds/server/events/EventWatcher.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1266/diff?src=pr&el=tree#diff-aGFkb29wLWhkZHMvZnJhbWV3b3JrL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9oYWRvb3AvaGRkcy9zZXJ2ZXIvZXZlbnRzL0V2ZW50V2F0Y2hlci5qYXZh) | `77.77% <0.00%> (-4.17%)` | `14.00% <0.00%> (ø%)` | |
   | ... and [76 more](https://codecov.io/gh/apache/hadoop-ozone/pull/1266/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/hadoop-ozone/pull/1266?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/hadoop-ozone/pull/1266?src=pr&el=footer). Last update [715aed2...a0430d2](https://codecov.io/gh/apache/hadoop-ozone/pull/1266?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org