You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "raju-balpande (via GitHub)" <gi...@apache.org> on 2024/06/10 06:30:23 UTC
[PR] HDDS-10778. Intermittent failure in testWatchForCommitForRetryfailure [ozone]
raju-balpande opened a new pull request, #6797:
URL: https://github.com/apache/ozone/pull/6797
## What changes were proposed in this pull request?
The intermittent failure in TestWatchForCommit.testWatchForCommitForRetryfailure
Fixed the changes and tested it for 1000 runs.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-10788
## How was this patch tested?
Reproduced and tested the scenario in local system. Tested it in CI build. Earlier there were 4.5% failure ratio for 1000 runs https://github.com/raju-balpande/apache_ozone/actions/runs/9398762434
Now it succeed completely here https://github.com/raju-balpande/apache_ozone/actions/runs/9427808268
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10788. Intermittent failure in testWatchForCommitForRetryfailure [ozone]
Posted by "raju-balpande (via GitHub)" <gi...@apache.org>.
raju-balpande commented on PR #6797:
URL: https://github.com/apache/ozone/pull/6797#issuecomment-2158173827
> @raju-balpande Thank you for the patch.
>
> I agree with @adoroszlai that shutting down all DNs are not right. This would defeat the purpose of the test.
>
> A bit of background, in Ozone's XceiverClientRatis#watchForCommit, there are two watch commits request with different ReplicationLevel
>
> 1. Watch for ALL_COMMITTED
>
> 2. Watch for MAJORITY_COMMITTED (If the previous watch threw an exception)
>
>
> Based on the second watch request, the client will remove some failed datanode UUID from the commitInfoMap.
>
> From my understanding `testWatchForCommitFoRetryFailure` is there to test what happens if two DNs (i.e. majority DNs) are down. Which means that both watch requests (ALL_COMMITTED and MAJORITY_COMMITTED) will both fail.
>
> From the ticket comment, I believe issue might be that #5727 was changing the wrong test. It should have updated `testWatchForCommitForGroupMismatchException` instead of `testWatchForCommitForRetryfailure`.
>
> So I suggest these changes:
>
> 1. Revert the changes for [HDDS-9826. Fix handling of GroupMismatchException in XceiverClientRatis.watchForCommit #5727](https://github.com/apache/ozone/pull/5727) on `TestWatchForCommit#testWatchForCommitForRetryFailure`
>
> 2. Apply the changes for [HDDS-9826. Fix handling of GroupMismatchException in XceiverClientRatis.watchForCommit #5727](https://github.com/apache/ozone/pull/5727) to `TestWatchForCommit#testWatchForCommitForGroupMismatchException`
>
>
> ```java
> // client should not attempt to watch with
> // MAJORITY_COMMITTED replication level, except the grpc IO issue
> if (!logCapturer.getOutput().contains("Connection refused")) {
> Assert.assertFalse(
> e.getMessage().contains("Watch-MAJORITY_COMMITTED"));
> }
> ```
>
> Please take a look at #5727 for context and let me know what you think.
Thanks for quick explanation, This is giving the picture on the logic behind the recent changes. I will check for possible solution here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10788. Intermittent failure in testWatchForCommitForRetryfailure [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on PR #6797:
URL: https://github.com/apache/ozone/pull/6797#issuecomment-2157947278
@ibrusentsev Could you help to review since it might be related to your patch in https://github.com/apache/ozone/pull/5727?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10788. Intermittent failure in testWatchForCommitForRetryfailure [ozone]
Posted by "raju-balpande (via GitHub)" <gi...@apache.org>.
raju-balpande commented on PR #6797:
URL: https://github.com/apache/ozone/pull/6797#issuecomment-2162690966
@ivandika3 , As you suggested the changes,
- I checked the changes been done under https://github.com/apache/ozone/pull/5727 and shifted those changes into method TestWatchForCommit#testWatchForCommitForGroupMismatchException from TestWatchForCommit#testWatchForCommitForRetryFailure.
- In my system it is working fine. CI bulid is passed.
- I checked the flakiness for `TestWatchForCommit#testWatchForCommitForRetryFailure` and is 100% for 1000 runs as in https://github.com/raju-balpande/apache_ozone/actions/runs/9477452421
- I checked the flakiness for `TestWatchForCommit#testWatchForCommitForGroupMismatchException` and is 100% for 1000 runs as in https://github.com/raju-balpande/apache_ozone/actions/runs/9465541138
Can we go ahead to merge this pull request?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10788. Intermittent failure in testWatchForCommitForRetryfailure [ozone]
Posted by "raju-balpande (via GitHub)" <gi...@apache.org>.
raju-balpande commented on code in PR #6797:
URL: https://github.com/apache/ozone/pull/6797#discussion_r1633001023
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestWatchForCommit.java:
##########
@@ -274,8 +272,9 @@ public void testWatchForCommitForRetryfailure() throws Exception {
xceiverClient.getPipeline()));
reply.getResponse().get();
long index = reply.getLogIndex();
- cluster.shutdownHddsDatanode(pipeline.getNodes().get(0));
- cluster.shutdownHddsDatanode(pipeline.getNodes().get(1));
+ for (int i = 0; i < pipeline.getNodes().size(); i++) {
+ cluster.shutdownHddsDatanode(pipeline.getNodes().get(i));
+ }
Review Comment:
Only in few cases if the execution took little longer the pipeline was not getting closed even after few minutes wait. I see out of three nodes only two nodes were closed. If all the nodes are closed all the steps are working fine.
Not sure, I am right or wrong. Whom do you suggest can validate this logic?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10778. Intermittent failure in testWatchForCommitForRetryfailure [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #6797:
URL: https://github.com/apache/ozone/pull/6797#discussion_r1632754952
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestWatchForCommit.java:
##########
@@ -274,8 +272,9 @@ public void testWatchForCommitForRetryfailure() throws Exception {
xceiverClient.getPipeline()));
reply.getResponse().get();
long index = reply.getLogIndex();
- cluster.shutdownHddsDatanode(pipeline.getNodes().get(0));
- cluster.shutdownHddsDatanode(pipeline.getNodes().get(1));
+ for (int i = 0; i < pipeline.getNodes().size(); i++) {
+ cluster.shutdownHddsDatanode(pipeline.getNodes().get(i));
+ }
Review Comment:
I don't exactly understand the purpose of this test, but stopping all 3 datanodes doesn't seem right. Maybe the test should wait until SCM detects dead nodes, instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10778. Intermittent failure in testWatchForCommitForRetryfailure [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #6797:
URL: https://github.com/apache/ozone/pull/6797#discussion_r1632754952
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestWatchForCommit.java:
##########
@@ -274,8 +272,9 @@ public void testWatchForCommitForRetryfailure() throws Exception {
xceiverClient.getPipeline()));
reply.getResponse().get();
long index = reply.getLogIndex();
- cluster.shutdownHddsDatanode(pipeline.getNodes().get(0));
- cluster.shutdownHddsDatanode(pipeline.getNodes().get(1));
+ for (int i = 0; i < pipeline.getNodes().size(); i++) {
+ cluster.shutdownHddsDatanode(pipeline.getNodes().get(i));
+ }
Review Comment:
I don't exactly understand the purpose of this test, but stopping all 3 datanodes doesn't seem right. Maybe we should wait until SCM detects dead nodes, instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10788. Intermittent failure in testWatchForCommitForRetryfailure [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai closed pull request #6797: HDDS-10788. Intermittent failure in testWatchForCommitForRetryfailure
URL: https://github.com/apache/ozone/pull/6797
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10788. Intermittent failure in testWatchForCommitForRetryfailure [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #6797:
URL: https://github.com/apache/ozone/pull/6797#issuecomment-2162701725
@raju-balpande single test case results look good, but there are 34 failures when running the whole class: https://github.com/raju-balpande/apache_ozone/actions/runs/9450084317/job/26038239702
Can you please check?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10778. Intermittent failure in testWatchForCommitForRetryfailure [ozone]
Posted by "Galsza (via GitHub)" <gi...@apache.org>.
Galsza commented on PR #6797:
URL: https://github.com/apache/ozone/pull/6797#issuecomment-2157853587
Minor: the jira link in the description points to HDDS-10788 instead of HDDS-10778
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org