You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "raju-balpande (via GitHub)" <gi...@apache.org> on 2024/04/24 17:32:36 UTC
[PR] HDDS-10582. Intermittent timeout during waitForReplicaCount in TestReconAndAdminContainerCLI [ozone]
raju-balpande opened a new pull request, #6585:
URL: https://github.com/apache/ozone/pull/6585
## What changes were proposed in this pull request?
Intermittent timeout during waitForReplicaCount in TestReconAndAdminContainerCLI
I observed two wait condition having similar checks and were intermittently failing. Observed the condition was true for a very smaller span and hence increased the frequency to check. Earlier frequency was 1000ms, tested it with 500ms and 200ms. 500ms still have few failures. 200ms is working with no failure.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-10582
## How was this patch tested?
Test it in CI by creating flaky tests check : https://github.com/raju-balpande/apache_ozone/actions/runs/8817120192
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10582. Intermittent timeout during waitForReplicaCount in TestReconAndAdminContainerCLI [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai merged PR #6585:
URL: https://github.com/apache/ozone/pull/6585
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10582. Intermittent timeout during waitForReplicaCount in TestReconAndAdminContainerCLI [ozone]
Posted by "raju-balpande (via GitHub)" <gi...@apache.org>.
raju-balpande commented on code in PR #6585:
URL: https://github.com/apache/ozone/pull/6585#discussion_r1579252937
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/recon/TestReconAndAdminContainerCLI.java:
##########
@@ -261,7 +261,8 @@ void testNodesInDecommissionOrMaintenance(
// a new replica-copy is made to another node.
// For maintenance, there is no replica-copy in this case.
if (!isMaintenance) {
- TestHelper.waitForReplicaCount(containerIdR3, 4, cluster);
+ GenericTestUtils.waitFor(() -> TestHelper.countReplicas(containerIdR3, cluster) == 4,
+ 200, 30000);
Review Comment:
I agree to your point. Even though the TestContainerReplication is not breaking now but the following two lines are indicating the condition is true for short span which if missed will fail the test method and hence thought to extend the change for this class as well. Hence making change in TestHelper.waitForReplicaCount only.
<img width="783" alt="image" src="https://github.com/apache/ozone/assets/146973984/91753787-4dfd-436b-a570-61b28af16844">
Will be observing the result for both the flaky classes under https://github.com/raju-balpande/apache_ozone/actions/runs/8831034232
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10582. Intermittent timeout during waitForReplicaCount in TestReconAndAdminContainerCLI [ozone]
Posted by "raju-balpande (via GitHub)" <gi...@apache.org>.
raju-balpande commented on code in PR #6585:
URL: https://github.com/apache/ozone/pull/6585#discussion_r1578997148
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/recon/TestReconAndAdminContainerCLI.java:
##########
@@ -261,7 +261,8 @@ void testNodesInDecommissionOrMaintenance(
// a new replica-copy is made to another node.
// For maintenance, there is no replica-copy in this case.
if (!isMaintenance) {
- TestHelper.waitForReplicaCount(containerIdR3, 4, cluster);
+ GenericTestUtils.waitFor(() -> TestHelper.countReplicas(containerIdR3, cluster) == 4,
+ 200, 30000);
Review Comment:
That is why I cross checked the uses of waitForReplicaCount which I found in TestContainerReplication, And I tried to see its flakiness..
- Worked 100% success for 10x10 https://github.com/raju-balpande/apache_ozone/actions/runs/8710014704
- Worked 99.75% success for 20x20 https://github.com/raju-balpande/apache_ozone/actions/runs/8720977974
And seems no change require for this class.
Please suggest if I shall make this change in TestHelper.waitForReplicaCount instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10582. Intermittent timeout during waitForReplicaCount in TestReconAndAdminContainerCLI [ozone]
Posted by "raju-balpande (via GitHub)" <gi...@apache.org>.
raju-balpande commented on code in PR #6585:
URL: https://github.com/apache/ozone/pull/6585#discussion_r1579252937
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/recon/TestReconAndAdminContainerCLI.java:
##########
@@ -261,7 +261,8 @@ void testNodesInDecommissionOrMaintenance(
// a new replica-copy is made to another node.
// For maintenance, there is no replica-copy in this case.
if (!isMaintenance) {
- TestHelper.waitForReplicaCount(containerIdR3, 4, cluster);
+ GenericTestUtils.waitFor(() -> TestHelper.countReplicas(containerIdR3, cluster) == 4,
+ 200, 30000);
Review Comment:
I agree to your point. Even though the TestContainerReplication is not breaking now but the following two lines are indicating the temporary condition and hence thought to extend the change for this class as well. Hence making change in TestHelper.waitForReplicaCount only.
<img width="783" alt="image" src="https://github.com/apache/ozone/assets/146973984/91753787-4dfd-436b-a570-61b28af16844">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10582. Intermittent timeout during waitForReplicaCount in TestReconAndAdminContainerCLI [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #6585:
URL: https://github.com/apache/ozone/pull/6585#discussion_r1578359531
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/recon/TestReconAndAdminContainerCLI.java:
##########
@@ -261,7 +261,8 @@ void testNodesInDecommissionOrMaintenance(
// a new replica-copy is made to another node.
// For maintenance, there is no replica-copy in this case.
if (!isMaintenance) {
- TestHelper.waitForReplicaCount(containerIdR3, 4, cluster);
+ GenericTestUtils.waitFor(() -> TestHelper.countReplicas(containerIdR3, cluster) == 4,
+ 200, 30000);
Review Comment:
> Observed the condition was true for a very smaller span
The same problem may affect other tests (currently there is only one other usage). I think the new, reduced check interval can be set in `waitForReplicaCount`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10582. Intermittent timeout during waitForReplicaCount in TestReconAndAdminContainerCLI [ozone]
Posted by "raju-balpande (via GitHub)" <gi...@apache.org>.
raju-balpande commented on code in PR #6585:
URL: https://github.com/apache/ozone/pull/6585#discussion_r1578997148
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/recon/TestReconAndAdminContainerCLI.java:
##########
@@ -261,7 +261,8 @@ void testNodesInDecommissionOrMaintenance(
// a new replica-copy is made to another node.
// For maintenance, there is no replica-copy in this case.
if (!isMaintenance) {
- TestHelper.waitForReplicaCount(containerIdR3, 4, cluster);
+ GenericTestUtils.waitFor(() -> TestHelper.countReplicas(containerIdR3, cluster) == 4,
+ 200, 30000);
Review Comment:
That is why I cross checked the uses of waitForReplicaCount which I found in TestContainerReplication, And I tried to see its flakiness..
- Worked 100% success for 10x10 https://github.com/raju-balpande/apache_ozone/actions/runs/8710014704
- Worked 99.5% success for 20x20 https://github.com/raju-balpande/apache_ozone/actions/runs/8720977974
And seems no change require for this class.
Please suggest if I shall make this change in TestHelper.waitForReplicaCount instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10582. Intermittent timeout during waitForReplicaCount in TestReconAndAdminContainerCLI [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #6585:
URL: https://github.com/apache/ozone/pull/6585#discussion_r1578999675
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/recon/TestReconAndAdminContainerCLI.java:
##########
@@ -261,7 +261,8 @@ void testNodesInDecommissionOrMaintenance(
// a new replica-copy is made to another node.
// For maintenance, there is no replica-copy in this case.
if (!isMaintenance) {
- TestHelper.waitForReplicaCount(containerIdR3, 4, cluster);
+ GenericTestUtils.waitFor(() -> TestHelper.countReplicas(containerIdR3, cluster) == 4,
+ 200, 30000);
Review Comment:
> no change require for this class
If it's not getting worse due to the change, please make it in `waitForReplicaCount`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10582. Intermittent timeout during waitForReplicaCount in TestReconAndAdminContainerCLI [ozone]
Posted by "raju-balpande (via GitHub)" <gi...@apache.org>.
raju-balpande commented on code in PR #6585:
URL: https://github.com/apache/ozone/pull/6585#discussion_r1579252937
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/recon/TestReconAndAdminContainerCLI.java:
##########
@@ -261,7 +261,8 @@ void testNodesInDecommissionOrMaintenance(
// a new replica-copy is made to another node.
// For maintenance, there is no replica-copy in this case.
if (!isMaintenance) {
- TestHelper.waitForReplicaCount(containerIdR3, 4, cluster);
+ GenericTestUtils.waitFor(() -> TestHelper.countReplicas(containerIdR3, cluster) == 4,
+ 200, 30000);
Review Comment:
I agree to your point. Even though the TestContainerReplication is not breaking now but the following two lines are indicating the condition is true for short span which if missed will fail the test method and hence thought to extend the change for this class as well. Hence making change in TestHelper.waitForReplicaCount only.
<img width="783" alt="image" src="https://github.com/apache/ozone/assets/146973984/91753787-4dfd-436b-a570-61b28af16844">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org