You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "symious (via GitHub)" <gi...@apache.org> on 2024/08/07 23:50:40 UTC
[PR] HDDS-11291. Fix timeout logic in XceiverServerRatis.submitRequest [ozone]
symious opened a new pull request, #7046:
URL: https://github.com/apache/ozone/pull/7046
## What changes were proposed in this pull request?
We met the following issue: Datanode command handler executing close container request, but the timeout logic is not correct, so it blocks all requests from SCM.
The jstack shows as follows:
```
"Command processor thread" #215 daemon prio=5 os_prio=0 tid=0x00007fcef3262000 nid=0xa56 waiting on condition [0x00007fcf63f9d000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00007fd4ab6dcd38> (a java.util.concurrent.CompletableFuture$Signaller)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
at java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1947)
at org.apache.ratis.server.impl.RaftServerImpl.executeSubmitClientRequestAsync(RaftServerImpl.java:816)
at org.apache.ratis.server.impl.RaftServerProxy.lambda$submitClientRequestAsync$7(RaftServerProxy.java:436)
at org.apache.ratis.server.impl.RaftServerProxy$$Lambda$827/1961332062.apply(Unknown Source)
at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:995)
at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2137)
at org.apache.ratis.server.impl.RaftServerProxy.submitClientRequestAsync(RaftServerProxy.java:436)
at org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.submitRequest(XceiverServerRatis.java:611)
at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.CloseContainerCommandHandler.handle(CloseContainerCommandHandler.java:105)
at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.CommandDispatcher.handle(CommandDispatcher.java:103)
at org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.lambda$initCommandHandlerThread$3(DatanodeStateMachine.java:593)
at org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine$$Lambda$270/1788388131.run(Unknown Source)
at java.lang.Thread.run(Thread.java:748)
```
The direct reason is the timeout logic is not working, because in Ratis the executeSubmitClientRequestAsync is a join() operation, and it will block the timeout on the outer CompletableFuture.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-11291
## How was this patch tested?
(Please explain how this patch was tested. Ex: unit tests, manual tests, workflow run on the fork git repo.)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11291. Fix timeout logic in XceiverServerRatis.submitRequest [ozone]
Posted by "symious (via GitHub)" <gi...@apache.org>.
symious commented on PR #7046:
URL: https://github.com/apache/ozone/pull/7046#issuecomment-2282186193
> We should check why ContainerStateMachine.startTransaction(RaftClientRequest request) was blocked. Are you able to reproduce the problem?
Not sure if it can be reproduced, but we suspect it's related to peer removal when leader is doing container close.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11291. Fix timeout logic in XceiverServerRatis.submitRequest [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai closed pull request #7046: HDDS-11291. Fix timeout logic in XceiverServerRatis.submitRequest
URL: https://github.com/apache/ozone/pull/7046
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11291. Fix timeout logic in XceiverServerRatis.submitRequest [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on PR #7046:
URL: https://github.com/apache/ozone/pull/7046#issuecomment-2276331658
> ... in Ratis the executeSubmitClientRequestAsync is a join() operation, and it will block the timeout on the outer CompletableFuture.
The `join()` to join the request submission but not the future completion. So, it is correct. It seems that the require submission did not go through in Datanode. We should check why. Are you able to reproduce the problem?
For Ratis, I would see if there is a better way to eliminate `join()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11291. Fix timeout logic in XceiverServerRatis.submitRequest [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #7046:
URL: https://github.com/apache/ozone/pull/7046#issuecomment-2421445796
@symious Based on @szetszwo's comment, I think the root cause needs further investigation. Can this be closed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org