You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "adoroszlai (via GitHub)" <gi...@apache.org> on 2024/02/07 09:04:53 UTC

[PR] HDDS-10291. Set simple properties in MiniOzoneCluster via Configuration [ozone]

adoroszlai opened a new pull request, #6186:
URL: https://github.com/apache/ozone/pull/6186

   ## What changes were proposed in this pull request?
   
   Set the following properties directly in `Configuration` instead of using methods of `MiniOzoneCluster.Builder`:
   
   - `hbInterval`
   - `hbProcessorInterval`
   - `omLayoutVersion`
   - `pipelineNumLimit`
   - `scmLayoutVersion`
   
   Default values are added to `ozone-site.xml` for `integration-test`, where applicable.
   
   Remove heartbeat config from `Test2WayCommitInRatis`, `TestQueryNode`.  Previously these set it via `Configuration`, but not via `MiniOzoneCluster`, which overwrote them with its defaults.  Now after removing the config from `MiniOzoneCluster`, these tests suddenly started failing due to the custom heartbeat intervals.
   
   https://issues.apache.org/jira/browse/HDDS-10291
   
   ## How was this patch tested?
   
   CI:
   https://github.com/adoroszlai/ozone/actions/runs/7806174237


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10291. Set simple properties in MiniOzoneCluster via Configuration [ozone]

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #6186:
URL: https://github.com/apache/ozone/pull/6186#issuecomment-1933362852

   Thanks @hemantk-12 for the review.
   
   > One more thing, we can also remove `setNumDatanodes(3)` because 3 is [default value](https://github.com/apache/ozone/pull/6186/files#diff-4a7db71bf162b78f4726b5c7888966ff40ee52e0652fcac9d188cd754f44e5feL333).
   
   I agree, but this patch is only part of a multi-step cleanup task (HDDS-6739), it does not try to fix all items.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10291. Set simple properties in MiniOzoneCluster via Configuration [ozone]

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai merged PR #6186:
URL: https://github.com/apache/ozone/pull/6186


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10291. Set simple properties in MiniOzoneCluster via Configuration [ozone]

Posted by "hemantk-12 (via GitHub)" <gi...@apache.org>.
hemantk-12 commented on code in PR #6186:
URL: https://github.com/apache/ozone/pull/6186#discussion_r1482291660


##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestSCMRestart.java:
##########
@@ -60,17 +61,17 @@ public class TestSCMRestart {
    */
   @BeforeAll
   public static void init() throws Exception {
+    final int numOfNodes = 4;
     conf = new OzoneConfiguration();
     conf.setTimeDuration(HDDS_PIPELINE_REPORT_INTERVAL, 1000,
             TimeUnit.MILLISECONDS);
+    conf.setTimeDuration(HDDS_HEARTBEAT_INTERVAL, 1000, TimeUnit.MILLISECONDS);
+    conf.setTimeDuration(ScmConfigKeys.OZONE_SCM_HEARTBEAT_PROCESS_INTERVAL, 1000, TimeUnit.MILLISECONDS);
     conf.setInt(ScmConfigKeys.OZONE_DATANODE_PIPELINE_LIMIT, 1);
-    int numOfNodes = 4;
+    // allow only one FACTOR THREE pipeline.

Review Comment:
   nit: not related to this PR, but this comment doesn't make sense to me.



##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/dn/ratis/TestDnRatisLogParser.java:
##########
@@ -50,8 +51,9 @@ public class TestDnRatisLogParser {
   @BeforeEach
   public void setup() throws Exception {
     OzoneConfiguration conf = new OzoneConfiguration();
+    conf.setInt(OZONE_SCM_RATIS_PIPELINE_LIMIT, 2);

Review Comment:
   I don't think this line is needed and default value, from `ozone-site.xml`, should be used.
   
   If I'm not wrong, default value 3 was used when `totalPipelineNumLimit` was set less than 3 previously.
   
   https://github.com/apache/ozone/blob/68c6f71b2d37bcee40957a8faa9083c9c9ef044f/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneClusterImpl.java#L711



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10291. Set simple properties in MiniOzoneCluster via Configuration [ozone]

Posted by "hemantk-12 (via GitHub)" <gi...@apache.org>.
hemantk-12 commented on code in PR #6186:
URL: https://github.com/apache/ozone/pull/6186#discussion_r1482290710


##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/dn/ratis/TestDnRatisLogParser.java:
##########
@@ -50,8 +51,9 @@ public class TestDnRatisLogParser {
   @BeforeEach
   public void setup() throws Exception {
     OzoneConfiguration conf = new OzoneConfiguration();
+    conf.setInt(OZONE_SCM_RATIS_PIPELINE_LIMIT, 2);

Review Comment:
   I don't think this line is needed and default value, from `ozone-site.xml`, should be used.
   
   If I'm not wrong, default value 3 was used when `totalPipelineNumLimit` was is less than 3 previously.
   
   https://github.com/apache/ozone/blob/68c6f71b2d37bcee40957a8faa9083c9c9ef044f/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneClusterImpl.java#L711



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10291. Set simple properties in MiniOzoneCluster via Configuration [ozone]

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #6186:
URL: https://github.com/apache/ozone/pull/6186#discussion_r1482407507


##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/dn/ratis/TestDnRatisLogParser.java:
##########
@@ -50,8 +51,9 @@ public class TestDnRatisLogParser {
   @BeforeEach
   public void setup() throws Exception {
     OzoneConfiguration conf = new OzoneConfiguration();
+    conf.setInt(OZONE_SCM_RATIS_PIPELINE_LIMIT, 2);

Review Comment:
   > If I'm not wrong, default value 3 was used when `totalPipelineNumLimit` was less than 3 previously.
   
   You are right.  I didn't notice this.
   
   But this test uses only one datanode, so it doesn't make any difference.
   
   https://github.com/apache/ozone/blob/59ca4d35cd0060f2bf704147fcb7a3b546937154/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/dn/ratis/TestDnRatisLogParser.java#L56



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10291. Set simple properties in MiniOzoneCluster via Configuration [ozone]

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #6186:
URL: https://github.com/apache/ozone/pull/6186#discussion_r1482408259


##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestSCMRestart.java:
##########
@@ -60,17 +61,17 @@ public class TestSCMRestart {
    */
   @BeforeAll
   public static void init() throws Exception {
+    final int numOfNodes = 4;
     conf = new OzoneConfiguration();
     conf.setTimeDuration(HDDS_PIPELINE_REPORT_INTERVAL, 1000,
             TimeUnit.MILLISECONDS);
+    conf.setTimeDuration(HDDS_HEARTBEAT_INTERVAL, 1000, TimeUnit.MILLISECONDS);
+    conf.setTimeDuration(ScmConfigKeys.OZONE_SCM_HEARTBEAT_PROCESS_INTERVAL, 1000, TimeUnit.MILLISECONDS);
     conf.setInt(ScmConfigKeys.OZONE_DATANODE_PIPELINE_LIMIT, 1);
-    int numOfNodes = 4;
+    // allow only one FACTOR THREE pipeline.

Review Comment:
   I think the idea is that each datanode will also have a single-node pipeline.  So with N datanodes a total limit of N + 1 will allow only one regular, 3-node pipeline.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10291. Set simple properties in MiniOzoneCluster via Configuration [ozone]

Posted by "hemantk-12 (via GitHub)" <gi...@apache.org>.
hemantk-12 commented on code in PR #6186:
URL: https://github.com/apache/ozone/pull/6186#discussion_r1482290710


##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/dn/ratis/TestDnRatisLogParser.java:
##########
@@ -50,8 +51,9 @@ public class TestDnRatisLogParser {
   @BeforeEach
   public void setup() throws Exception {
     OzoneConfiguration conf = new OzoneConfiguration();
+    conf.setInt(OZONE_SCM_RATIS_PIPELINE_LIMIT, 2);

Review Comment:
   I don't think this line is needed and default value, from `ozone-site.xml`, should be used.
   
   If I'm not wrong, default value 3 was used when `totalPipelineNumLimit` was is less than 3 previously.
   
   https://github.com/apache/ozone/blob/68c6f71b2d37bcee40957a8faa9083c9c9ef044f/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneClusterImpl.java#L711-L713



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10291. Set simple properties in MiniOzoneCluster via Configuration [ozone]

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #6186:
URL: https://github.com/apache/ozone/pull/6186#issuecomment-1934649176

   Thanks a lot @hemantk-12 for the review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org