You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "peterxcli (via GitHub)" <gi...@apache.org> on 2025/03/12 15:21:53 UTC
[PR] HDDS-12535. Intermittent failure in TestContainerReportHandling [ozone]
peterxcli opened a new pull request, #8060:
URL: https://github.com/apache/ozone/pull/8060
## What changes were proposed in this pull request?
```
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 82.54 s <<< FAILURE! -- in org.apache.hadoop.ozone.container.TestContainerReportHandling
org.apache.hadoop.ozone.container.TestContainerReportHandling.testDeletingOrDeletedContainerTransitionsToClosedWhenNonEmptyReplicaIsReported(LifeCycleState)[2] -- Time elapsed: 33.84 s <<< ERROR!
org.apache.hadoop.hdds.scm.exceptions.SCMException: org.apache.ratis.protocol.exceptions.NotLeaderException: Server a4f85781-650a-46e8-940e-a45bfdaa2a01@group-BBAD22E09632 is not the leader
at org.apache.hadoop.hdds.scm.ha.SCMHAInvocationHandler.translateException(SCMHAInvocationHandler.java:164)
at org.apache.hadoop.hdds.scm.ha.SCMHAInvocationHandler.invokeRatis(SCMHAInvocationHandler.java:114)
at org.apache.hadoop.hdds.scm.ha.SCMHAInvocationHandler.invoke(SCMHAInvocationHandler.java:73)
at jdk.proxy2/jdk.proxy2.$Proxy42.updateContainerState(Unknown Source)
at org.apache.hadoop.hdds.scm.container.ContainerManagerImpl.updateContainerState(ContainerManagerImpl.java:283)
at org.apache.hadoop.ozone.container.TestContainerReportHandling.testDeletingOrDeletedContainerTransitionsToClosedWhenNonEmptyReplicaIsReported(TestContainerReportHandling.java:100)
...
Caused by: org.apache.ratis.protocol.exceptions.NotLeaderException: Server a4f85781-650a-46e8-940e-a45bfdaa2a01@group-BBAD22E09632 is not the leader
at org.apache.ratis.server.impl.RaftServerImpl.generateNotLeaderException(RaftServerImpl.java:780)
at org.apache.ratis.server.impl.LeaderStateImpl.stop(LeaderStateImpl.java:437)
at org.apache.ratis.server.impl.RoleInfo.shutdownLeaderState(RoleInfo.java:104)
at org.apache.ratis.server.impl.RaftServerImpl.lambda$close$1(RaftServerImpl.java:530)
at org.apache.ratis.util.LifeCycle.lambda$checkStateAndClose$7(LifeCycle.java:306)
at org.apache.ratis.util.LifeCycle.checkStateAndClose(LifeCycle.java:326)
at org.apache.ratis.util.LifeCycle.checkStateAndClose(LifeCycle.java:304)
at org.apache.ratis.server.impl.RaftServerImpl.close(RaftServerImpl.java:512)
at org.apache.ratis.server.impl.StateMachineUpdater.run(StateMachineUpdater.java:207)
```
Same problem affects TestContainerReportHandlingWithHA
---
See `Terminating with exit status 1: Invalid event: DELETE at CLOSING state.` in test result log.
```
2025-03-12 20:19:41,402 [scmNode-3-FixedThreadPoolWithAffinityExecutor-0-0] INFO container.IncrementalContainerReportHandler (IncrementalContainerReportHandler.java:onMessage(109)) - Failed to process CLOSED container #1: org.apache.ratis.protocol.exceptions.NotLeaderException: Server cd03248d-9309-426c-ac05-3168de666b12@group-727E36EF7571 is not the leader, suggested leader is: 5bbab233-360a-4332-b5e3-d5fdfa6c8f19|localhost:15076
2025-03-12 20:19:41,402 [scmNode-2-FixedThreadPoolWithAffinityExecutor-0-0] INFO container.IncrementalContainerReportHandler (IncrementalContainerReportHandler.java:onMessage(109)) - Failed to process CLOSED container #1: org.apache.ratis.protocol.exceptions.NotLeaderException: Server 416483bd-419d-4dc1-bc92-de8c5f70f57c@group-727E36EF7571 is not the leader, suggested leader is: 5bbab233-360a-4332-b5e3-d5fdfa6c8f19|localhost:15076
2025-03-12 20:19:41,409 [5bbab233-360a-4332-b5e3-d5fdfa6c8f19@group-727E36EF7571-StateMachineUpdater] ERROR statemachine.StateMachine (ExitUtils.java:terminate(133)) - Terminating with exit status 1: Invalid event: DELETE at CLOSING state.
org.apache.hadoop.ozone.common.statemachine.InvalidStateTransitionException: Invalid event: DELETE at CLOSING state.
at org.apache.hadoop.ozone.common.statemachine.StateMachine.getNextState(StateMachine.java:58)
at org.apache.hadoop.hdds.scm.container.ContainerStateManagerImpl.updateContainerState(ContainerStateManagerImpl.java:354)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.hadoop.hdds.scm.ha.SCMStateMachine.process(SCMStateMachine.java:192)
at org.apache.hadoop.hdds.scm.ha.SCMStateMachine.applyTransaction(SCMStateMachine.java:155)
at org.apache.ratis.server.impl.RaftServerImpl.applyLogToStateMachine(RaftServerImpl.java:1832)
at org.apache.ratis.server.impl.StateMachineUpdater.applyLog(StateMachineUpdater.java:252)
at org.apache.ratis.server.impl.StateMachineUpdater.run(StateMachineUpdater.java:193)
at java.base/java.lang.Thread.run(Thread.java:829)
2025-03-12 20:19:41,409 [5bbab233-360a-4332-b5e3-d5fdfa6c8f19@group-727E36EF7571-StateMachineUpdater] ERROR impl.StateMachineUpdater (StateMachineUpdater.java:run(206)) - 5bbab233-360a-4332-b5e3-d5fdfa6c8f19@group-727E36EF7571-StateMachineUpdater caught a Throwable.
org.apache.ratis.server.raftlog.RaftLogIOException: org.apache.ratis.util.ExitUtils$ExitException: Invalid event: DELETE at CLOSING state.
at org.apache.ratis.server.impl.RaftServerImpl.applyLogToStateMachine(RaftServerImpl.java:1835)
at org.apache.ratis.server.impl.StateMachineUpdater.applyLog(StateMachineUpdater.java:252)
at org.apache.ratis.server.impl.StateMachineUpdater.run(StateMachineUpdater.java:193)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.ratis.util.ExitUtils$ExitException: Invalid event: DELETE at CLOSING state.
at org.apache.ratis.util.ExitUtils.terminate(ExitUtils.java:141)
at org.apache.ratis.util.ExitUtils.terminate(ExitUtils.java:151)
at org.apache.hadoop.hdds.scm.ha.SCMStateMachine.applyTransaction(SCMStateMachine.java:176)
at org.apache.ratis.server.impl.RaftServerImpl.applyLogToStateMachine(RaftServerImpl.java:1832)
... 3 more
Caused by: org.apache.hadoop.ozone.common.statemachine.InvalidStateTransitionException: Invalid event: DELETE at CLOSING state.
at org.apache.hadoop.ozone.common.statemachine.StateMachine.getNextState(StateMachine.java:58)
at org.apache.hadoop.hdds.scm.container.ContainerStateManagerImpl.updateContainerState(ContainerStateManagerImpl.java:354)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.hadoop.hdds.scm.ha.SCMStateMachine.process(SCMStateMachine.java:192)
at org.apache.hadoop.hdds.scm.ha.SCMStateMachine.applyTransaction(SCMStateMachine.java:155)
... 4 more
```
## What has been done?
Wait till scm think that container is closed after datanodes report their containers are closed before update container with DELETE event
## What is the link to the Apache JIRA
CI:
- build-branch: https://github.com/peterxcli/ozone/actions/runs/13812189282
- flakey-check
- TestContainerReportHandling: https://github.com/peterxcli/ozone/actions/runs/13812243728
- TestContainerReportHandlingWithHA: https://github.com/peterxcli/ozone/actions/runs/13812233483
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12535. Intermittent failure in TestContainerReportHandling [ozone]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on PR #8060:
URL: https://github.com/apache/ozone/pull/8060#issuecomment-2722049534
Thanks for the review.
New flaky test check:
TestContainerReportHandlingWithHA: https://github.com/peterxcli/ozone/actions/runs/13840306197
TestContainerReportHandling: https://github.com/peterxcli/ozone/actions/runs/13840325468
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12535. Intermittent failure in TestContainerReportHandling [ozone]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on PR #8060:
URL: https://github.com/apache/ozone/pull/8060#issuecomment-2719779332
New flakey test check:
TestContainerReportHandlingWithHA: https://github.com/peterxcli/ozone/actions/runs/13826252713
TestContainerReportHandling: https://github.com/peterxcli/ozone/actions/runs/13826248187
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12535. Intermittent failure in TestContainerReportHandling [ozone]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on PR #8060:
URL: https://github.com/apache/ozone/pull/8060#issuecomment-2718331056
cc @adoroszlai, please take a look. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12535. Intermittent failure in TestContainerReportHandling [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #8060:
URL: https://github.com/apache/ozone/pull/8060#issuecomment-2722357021
Thanks @peterxcli for the patch, @siddhantsangwan for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12535. Intermittent failure in TestContainerReportHandling [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #8060:
URL: https://github.com/apache/ozone/pull/8060#discussion_r1992900595
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/TestContainerReportHandlingWithHA.java:
##########
@@ -113,23 +110,7 @@ void testDeletingOrDeletedContainerTransitionsToClosedWhenNonEmptyReplicaIsRepor
// restart a DN and wait for the container to get CLOSED in all SCMs
HddsDatanodeService dn = cluster.getHddsDatanode(keyLocation.getPipeline().getFirstNode());
cluster.restartHddsDatanode(dn.getDatanodeDetails(), false);
- ContainerManager[] array = new ContainerManager[numSCM];
- for (int i = 0; i < numSCM; i++) {
- array[i] = cluster.getStorageContainerManager(i).getContainerManager();
- }
Review Comment:
Previously the HA test checked container state in all 3 SCMs, but I guess it is enough to check only the leader, as the test does with the patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12535. Intermittent failure in TestContainerReportHandling [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #8060:
URL: https://github.com/apache/ozone/pull/8060#discussion_r1992334858
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/TestContainerReportHandling.java:
##########
@@ -97,6 +95,19 @@ void testDeletingOrDeletedContainerTransitionsToClosedWhenNonEmptyReplicaIsRepor
ContainerID containerID = ContainerID.valueOf(keyLocation.getContainerID());
waitForContainerClose(cluster, containerID.getId());
+ // also wait till the container is closed in SCM
+ GenericTestUtils.waitFor(() -> {
+ ContainerManager containerManager = cluster.getStorageContainerManager().getContainerManager();
+ try {
+ if (containerManager.getContainer(containerID).getState() != HddsProtos.LifeCycleState.CLOSED) {
+ return false;
+ }
+ } catch (ContainerNotFoundException e) {
+ return false;
+ }
+ return true;
+ }, 2000, 20000);
+
Review Comment:
I guess at this point SCM should already have the container, `ContainerNotFoundException` can be considered a problem. If so, we could reuse the check (extracted to a method in `TestHelper`) from the test's end:
https://github.com/apache/ozone/blob/6648965b5ce224cff535fd3bf633fcd251c54167/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/TestContainerReportHandling.java#L125-L132
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12535. Intermittent failure in TestContainerReportHandling [ozone]
Posted by "siddhantsangwan (via GitHub)" <gi...@apache.org>.
siddhantsangwan commented on code in PR #8060:
URL: https://github.com/apache/ozone/pull/8060#discussion_r1993374518
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/TestContainerReportHandlingWithHA.java:
##########
@@ -113,23 +110,7 @@ void testDeletingOrDeletedContainerTransitionsToClosedWhenNonEmptyReplicaIsRepor
// restart a DN and wait for the container to get CLOSED in all SCMs
HddsDatanodeService dn = cluster.getHddsDatanode(keyLocation.getPipeline().getFirstNode());
cluster.restartHddsDatanode(dn.getDatanodeDetails(), false);
- ContainerManager[] array = new ContainerManager[numSCM];
- for (int i = 0; i < numSCM; i++) {
- array[i] = cluster.getStorageContainerManager(i).getContainerManager();
- }
- GenericTestUtils.waitFor(() -> {
- try {
- for (ContainerManager manager : array) {
- if (manager.getContainer(containerID).getState() != HddsProtos.LifeCycleState.CLOSED) {
- return false;
- }
- }
- return true;
- } catch (ContainerNotFoundException e) {
- fail(e);
- }
- return false;
- }, 2000, 20000);
+ waitForContainerStateInSCM(cluster.getScmLeader(), containerID, HddsProtos.LifeCycleState.CLOSED);
Review Comment:
The point of this test is that the container should eventually move to `CLOSED` in all three SCMs, so we should check the other SCMs as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12535. Intermittent failure in TestContainerReportHandling [ozone]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on PR #8060:
URL: https://github.com/apache/ozone/pull/8060#issuecomment-2723413522
Thanks @adoroszlai @siddhantsangwan for the review, and @adoroszlai for merging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12535. Intermittent failure in TestContainerReportHandling [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai merged PR #8060:
URL: https://github.com/apache/ozone/pull/8060
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org