You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2022/02/16 12:02:58 UTC

[GitHub] [ozone] kaijchen opened a new pull request #3099: HDDS-6332. Reduce SCM downtime in testDNStartAfterSCM for faster and stabler CI

kaijchen opened a new pull request #3099:
URL: https://github.com/apache/ozone/pull/3099


   ## What changes were proposed in this pull request?
   
   Reduce SCM downtime from 20s to 10s in testDNStartAfterSCM for faster and stabler CI.
   There is a 100s time limit in this test, sometimes it will fail due to timeover.
   Instead of increasing the time limit, decreasing the SCM downtime can make this test run faster.
   
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-6332
   
   ## How was this patch tested?
   
   intergration test


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] kaijchen commented on pull request #3099: HDDS-6332. Reduce SCM downtime in testDNStartAfterSCM for faster and stabler CI

Posted by GitBox <gi...@apache.org>.
kaijchen commented on pull request #3099:
URL: https://github.com/apache/ozone/pull/3099#issuecomment-1050466233


   For example:
   
   https://github.com/apache/ozone/runs/5325087931?check_suite_focus=true


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] kaijchen removed a comment on pull request #3099: HDDS-6332. Make TestMiniOzoneCluster more stable

Posted by GitBox <gi...@apache.org>.
kaijchen removed a comment on pull request #3099:
URL: https://github.com/apache/ozone/pull/3099#issuecomment-1053472689


   [Reducing SCM downtime](https://github.com/kaijchen/ozone/actions/runs/1905341634) also helps. Let's include this change also.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] kaijchen commented on pull request #3099: HDDS-6332. Increase timeout in TestMiniOzoneCluster for stabler CI

Posted by GitBox <gi...@apache.org>.
kaijchen commented on pull request #3099:
URL: https://github.com/apache/ozone/pull/3099#issuecomment-1053472689


   [Reducing SCM downtime](https://github.com/kaijchen/ozone/actions/runs/1905341634) also helps. Let's include this change also.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] kaijchen commented on pull request #3099: HDDS-6332. Make TestMiniOzoneCluster more stable

Posted by GitBox <gi...@apache.org>.
kaijchen commented on pull request #3099:
URL: https://github.com/apache/ozone/pull/3099#issuecomment-1069799914


   Closing this due to HDDS-6095 (#3148), thanks for the work @adoroszlai.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] kaijchen commented on pull request #3099: HDDS-6332. Reduce SCM downtime in testDNStartAfterSCM for faster and stabler CI

Posted by GitBox <gi...@apache.org>.
kaijchen commented on pull request #3099:
URL: https://github.com/apache/ozone/pull/3099#issuecomment-1053414190


   > Thanks @kaijchen for the patch. Tested 100x: [with the patch](https://github.com/adoroszlai/hadoop-ozone/actions/runs/1899164608) `testDN[Ss]tartAfterSCM` failed 2 times, while there were 3 failures [without it](https://github.com/adoroszlai/hadoop-ozone/actions/runs/1900411280).
   
   I noticed there are timeout failures in `testStartMultipleDatanodes` also. I've tried multiple different time limits and here's the results.
   
   | testStartMultipleDatanodes | testDNStartAfterSCM | Failures | Reason |
   | --- | --- | --- | --- |
   | 60s | 100s | [3/100](https://github.com/kaijchen/ozone/actions/runs/1902607746) | - |
   | 100s | 150s | [1/400](https://github.com/kaijchen/ozone/actions/runs/1902640160) | testStartMultipleDatanodes: timeout |
   | 120s | 180s | [1/1000](https://github.com/kaijchen/ozone/actions/runs/1903385947) | testDNStartAfterSCM: timeout |
   | 120s | 200s | [2/1000](https://github.com/kaijchen/ozone/runs/5347846823) | testDNStartAfterSCM: assertion failure + timed out waiting for condition |
   | 120s | 240s | [1/1000](https://github.com/kaijchen/ozone/runs/5347813636) | testDNStartAfterSCM: assertion failure |
   
   So instead of reducing SCM downtime, I suggest increasing the timeout of both tests. 
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] kaijchen edited a comment on pull request #3099: HDDS-6332. Increase timeout in TestMiniOzoneCluster for stabler CI

Posted by GitBox <gi...@apache.org>.
kaijchen edited a comment on pull request #3099:
URL: https://github.com/apache/ozone/pull/3099#issuecomment-1053414190


   > Thanks @kaijchen for the patch. Tested 100x: [with the patch](https://github.com/adoroszlai/hadoop-ozone/actions/runs/1899164608) `testDN[Ss]tartAfterSCM` failed 2 times, while there were 3 failures [without it](https://github.com/adoroszlai/hadoop-ozone/actions/runs/1900411280).
   
   I noticed there are timeout failures in `testStartMultipleDatanodes` also. I've tried multiple different time limits and here's the results.
   
   | testStartMultipleDatanodes | testDNStartAfterSCM | Failures | Reason |
   | --- | --- | --- | --- |
   | 60s | 100s | [3/100](https://github.com/kaijchen/ozone/actions/runs/1902607746) | - |
   | 100s | 150s | [1/400](https://github.com/kaijchen/ozone/actions/runs/1902640160) | testStartMultipleDatanodes: timeout |
   | 120s | 180s | [1/1000](https://github.com/kaijchen/ozone/actions/runs/1903385947) | testDNStartAfterSCM: timeout |
   | 120s | 200s | [2/1000](https://github.com/kaijchen/ozone/actions/runs/1904832682) | testDNStartAfterSCM: assertion failure + timed out waiting for condition |
   | 120s | 240s | [1/1000](https://github.com/kaijchen/ozone/actions/runs/1904835449) | testDNStartAfterSCM: assertion failure |
   
   So instead of reducing SCM downtime, I suggest increasing the timeout of both tests. 
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] kaijchen edited a comment on pull request #3099: HDDS-6332. Make TestMiniOzoneCluster more stable

Posted by GitBox <gi...@apache.org>.
kaijchen edited a comment on pull request #3099:
URL: https://github.com/apache/ozone/pull/3099#issuecomment-1053414190


   > Thanks @kaijchen for the patch. Tested 100x: [with the patch](https://github.com/adoroszlai/hadoop-ozone/actions/runs/1899164608) `testDN[Ss]tartAfterSCM` failed 2 times, while there were 3 failures [without it](https://github.com/adoroszlai/hadoop-ozone/actions/runs/1900411280).
   
   I noticed there are timeout failures in `testStartMultipleDatanodes` also. I've tried multiple different time limits and here's the results.
   
   | testStartMultipleDatanodes | testDNStartAfterSCM | Failures | Reason |
   | --- | --- | --- | --- |
   | 60s | 100s | [3/100](https://github.com/kaijchen/ozone/actions/runs/1902607746) | - |
   | 100s | 150s | [1/400](https://github.com/kaijchen/ozone/actions/runs/1902640160) | testStartMultipleDatanodes: timeout |
   | 120s | 180s | [1/1000](https://github.com/kaijchen/ozone/actions/runs/1903385947) | testDNStartAfterSCM: timeout |
   | 120s | 200s | [2/1000](https://github.com/kaijchen/ozone/actions/runs/1904832682) | testDNStartAfterSCM: assertion failure + timed out waiting for condition |
   | 120s | 240s | [1/1000](https://github.com/kaijchen/ozone/actions/runs/1904835449) | testDNStartAfterSCM: assertion failure |
   
   SCM downtime was 20s in the tests above.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] kaijchen commented on pull request #3099: HDDS-6332. Reduce SCM downtime in testDNStartAfterSCM for faster and stabler CI

Posted by GitBox <gi...@apache.org>.
kaijchen commented on pull request #3099:
URL: https://github.com/apache/ozone/pull/3099#issuecomment-1051440107


   Thanks for the test @adoroszlai, I will try other ways to improve this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] kaijchen closed pull request #3099: HDDS-6332. Make TestMiniOzoneCluster more stable

Posted by GitBox <gi...@apache.org>.
kaijchen closed pull request #3099:
URL: https://github.com/apache/ozone/pull/3099


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] kaijchen edited a comment on pull request #3099: HDDS-6332. Make TestMiniOzoneCluster more stable

Posted by GitBox <gi...@apache.org>.
kaijchen edited a comment on pull request #3099:
URL: https://github.com/apache/ozone/pull/3099#issuecomment-1053414190


   > Thanks @kaijchen for the patch. Tested 100x: [with the patch](https://github.com/adoroszlai/hadoop-ozone/actions/runs/1899164608) `testDN[Ss]tartAfterSCM` failed 2 times, while there were 3 failures [without it](https://github.com/adoroszlai/hadoop-ozone/actions/runs/1900411280).
   
   I noticed there are timeout failures in `testStartMultipleDatanodes` also. I've tried multiple different time limits and here's the results.
   
   | testStartMultipleDatanodes | testDNStartAfterSCM | Failures | Reason |
   | --- | --- | --- | --- |
   | 60s | 100s | [3/100](https://github.com/kaijchen/ozone/actions/runs/1902607746) | testDNstartAfterSCM:  `org.junit.runners.model.TestTimedOutException` + `java.lang.AssertionError` |
   | 100s | 150s | [1/400](https://github.com/kaijchen/ozone/actions/runs/1902640160) | testStartMultipleDatanodes: `org.junit.runners.model.TestTimedOutException` |
   | 120s | 180s | [1/1000](https://github.com/kaijchen/ozone/actions/runs/1903385947) | testDNStartAfterSCM: `org.junit.runners.model.TestTimedOutException` |
   | 120s | 200s | [2/1000](https://github.com/kaijchen/ozone/actions/runs/1904832682/attempts/1) | testDNStartAfterSCM: `java.lang.AssertionError` + `java.util.concurrent.TimeoutException ` |
   | 120s | 240s | [1/1000](https://github.com/kaijchen/ozone/actions/runs/1904835449) | testDNStartAfterSCM: `java.lang.AssertionError` |
   
   SCM downtime was 20s in the tests above.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] kaijchen edited a comment on pull request #3099: HDDS-6332. Make TestMiniOzoneCluster more stable

Posted by GitBox <gi...@apache.org>.
kaijchen edited a comment on pull request #3099:
URL: https://github.com/apache/ozone/pull/3099#issuecomment-1053414190


   > Thanks @kaijchen for the patch. Tested 100x: [with the patch](https://github.com/adoroszlai/hadoop-ozone/actions/runs/1899164608) `testDN[Ss]tartAfterSCM` failed 2 times, while there were 3 failures [without it](https://github.com/adoroszlai/hadoop-ozone/actions/runs/1900411280).
   
   I noticed there are timeout failures in `testStartMultipleDatanodes` also. I've tried multiple different time limits and here's the results.
   
   | testStartMultipleDatanodes | testDNStartAfterSCM | Failures | Reason |
   | --- | --- | --- | --- |
   | 60s | 100s | [3/100](https://github.com/kaijchen/ozone/actions/runs/1902607746) | testDNstartAfterSCM:  `org.junit.runners.model.TestTimedOutException` + `java.lang.AssertionError` |
   | 100s | 150s | [1/400](https://github.com/kaijchen/ozone/actions/runs/1902640160) | testStartMultipleDatanodes: `org.junit.runners.model.TestTimedOutException` |
   | 120s | 180s | [1/1000](https://github.com/kaijchen/ozone/actions/runs/1903385947) | testDNStartAfterSCM: `org.junit.runners.model.TestTimedOutException` |
   | 120s | 200s | [2/1000](https://github.com/kaijchen/ozone/actions/runs/1904832682) | testDNStartAfterSCM: `java.lang.AssertionError` + `java.util.concurrent.TimeoutException ` |
   | 120s | 240s | [1/1000](https://github.com/kaijchen/ozone/actions/runs/1904835449) | testDNStartAfterSCM: `java.lang.AssertionError` |
   
   SCM downtime was 20s in the tests above.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] kaijchen commented on pull request #3099: HDDS-6332. Make TestMiniOzoneCluster more stable

Posted by GitBox <gi...@apache.org>.
kaijchen commented on pull request #3099:
URL: https://github.com/apache/ozone/pull/3099#issuecomment-1053811981


   There were `java.lang.AssertionError: expected:<HEARTBEAT> but was:<REGISTER>` errors in [run 1](https://github.com/kaijchen/ozone/actions/runs/1904832682/attempts/1) and [run 2](https://github.com/kaijchen/ozone/actions/runs/1904832682/attempts/2).
   Where SCM thought all DataNodes have registered in `cluster.waitForClusterToBeReady()`.
   But the DataNode haven't changed state.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org