[jira] [Updated] (RATIS-776) Handle ResourceUnavailabeException properly in Ratis Server

["Lokesh Jain (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/RATIS-776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Lokesh Jain updated RATIS-776:
------------------------------
    Attachment: RATIS-776.001.patch

> Handle ResourceUnavailabeException properly in Ratis Server
> -----------------------------------------------------------
>
>                 Key: RATIS-776
>                 URL: https://issues.apache.org/jira/browse/RATIS-776
>             Project: Ratis
>          Issue Type: Bug
>          Components: server
>            Reporter: Lokesh Jain
>            Assignee: Lokesh Jain
>            Priority: Major
>         Attachments: RATIS-776.001.patch
>
>
> Ratis leader while processing a client request tries to create a pending request. If it is not able to do so it fails the request with ResourceUnavailableException. But the server keeps processing the other requests from the same client. The resources can be released when the other client requests are processed, resulting in out of order processing of client requests. On failure the server should ideally fail all the client requests which need to be processed.
> {code:java}
> 2019-12-10 19:50:31,846 [grpc-default-executor-5] INFO  ratis.ContainerStateMachine (ContainerStateMachine.java:preAppendTransaction(311)) - append seqNum:2 WriteChunk
> 2019-12-10 19:50:31,860 [grpc-default-executor-5] INFO  ratis.ContainerStateMachine (ContainerStateMachine.java:preAppendTransaction(311)) - append seqNum:3 WriteChunk
> Caused by: org.apache.ratis.protocol.exceptions.ResourceUnavailableException: 164293f2-68e3-4851-bc46-4a828bd79ffa@group-03010B1A5718: Failed to acquire a pending write request for RaftClientRequest:client-38E7254A5AF1->164293f2-68e3-4851-bc46-4a828bd79ffa@group-03010B1A5718, cid=3, seq=4, RW, Message:000000b2080612343362...(size=182)Caused by: org.apache.ratis.protocol.exceptions.ResourceUnavailableException: 164293f2-68e3-4851-bc46-4a828bd79ffa@group-03010B1A5718: Failed to acquire a pending write request for RaftClientRequest:client-38E7254A5AF1->164293f2-68e3-4851-bc46-4a828bd79ffa@group-03010B1A5718, cid=3, seq=4, RW, Message:000000b2080612343362...(size=182) at org.apache.ratis.server.impl.RaftServerImpl.appendTransaction(RaftServerImpl.java:514) at org.apache.ratis.server.impl.RaftServerImpl.submitClientRequestAsync(RaftServerImpl.java:589) at org.apache.ratis.server.impl.RaftServerProxy.lambda$submitClientRequestAsync$7(RaftServerProxy.java:333) at org.apache.ratis.server.impl.RaftServerProxy.lambda$null$5(RaftServerProxy.java:328) at org.apache.ratis.util.JavaUtils.callAsUnchecked(JavaUtils.java:109) at org.apache.ratis.server.impl.RaftServerProxy.lambda$submitRequest$6(RaftServerProxy.java:328) at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:981) at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2124) at org.apache.ratis.server.impl.RaftServerProxy.submitRequest(RaftServerProxy.java:327) at org.apache.ratis.server.impl.RaftServerProxy.submitClientRequestAsync(RaftServerProxy.java:333) at org.apache.ratis.grpc.client.GrpcClientProtocolService$RequestStreamObserver.processClientRequest(GrpcClientProtocolService.java:221) at org.apache.ratis.grpc.client.GrpcClientProtocolService$OrderedRequestStreamObserver.processClientRequest(GrpcClientProtocolService.java:327) at org.apache.ratis.util.SlidingWindow$Server.processRequestsFromHead(SlidingWindow.java:429) at org.apache.ratis.util.SlidingWindow$Server.receivedRequest(SlidingWindow.java:421) at org.apache.ratis.grpc.client.GrpcClientProtocolService$OrderedRequestStreamObserver.processClientRequest(GrpcClientProtocolService.java:346) at org.apache.ratis.grpc.client.GrpcClientProtocolService$RequestStreamObserver.onNext(GrpcClientProtocolService.java:241) at org.apache.ratis.grpc.client.GrpcClientProtocolService$RequestStreamObserver.onNext(GrpcClientProtocolService.java:168) at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:251) at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailableInternal(ServerCallImpl.java:309) at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:292) at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:779) ... 5 more
> 2019-12-10 19:50:31,860 [grpc-default-executor-5] INFO ratis.ContainerStateMachine (ContainerStateMachine.java:preAppendTransaction(311)) - append seqNum:5 WriteChunk 
> {code}
> Further while failing the client request server does not invalidate the retry cache. Any retries from the client are therefore ignored in the server.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)