[jira] [Created] (RATIS-776) Handle ResourceUnavailabeException properly in Ratis Server

["Lokesh Jain (Jira)" <ji...@apache.org>]

Lokesh Jain created RATIS-776:
---------------------------------

             Summary: Handle ResourceUnavailabeException properly in Ratis Server
                 Key: RATIS-776
                 URL: https://issues.apache.org/jira/browse/RATIS-776
             Project: Ratis
          Issue Type: Bug
          Components: server
            Reporter: Lokesh Jain


Ratis leader while processing a client request tries to create a pending request. If it is not able to do so it fails the request with ResourceUnavailableException. But the server keeps processing the other requests from the same client. The resources can be released when the other client requests are processed, resulting in out of order processing of client requests. On failure the server should ideally fail all the client requests which need to be processed.
{code:java}
2019-12-10 19:50:31,846 [grpc-default-executor-5] INFO  ratis.ContainerStateMachine (ContainerStateMachine.java:preAppendTransaction(311)) - append seqNum:2 WriteChunk
2019-12-10 19:50:31,860 [grpc-default-executor-5] INFO  ratis.ContainerStateMachine (ContainerStateMachine.java:preAppendTransaction(311)) - append seqNum:3 WriteChunk

Caused by: org.apache.ratis.protocol.exceptions.ResourceUnavailableException: 164293f2-68e3-4851-bc46-4a828bd79ffa@group-03010B1A5718: Failed to acquire a pending write request for RaftClientRequest:client-38E7254A5AF1->164293f2-68e3-4851-bc46-4a828bd79ffa@group-03010B1A5718, cid=3, seq=4, RW, Message:000000b2080612343362...(size=182)Caused by: org.apache.ratis.protocol.exceptions.ResourceUnavailableException: 164293f2-68e3-4851-bc46-4a828bd79ffa@group-03010B1A5718: Failed to acquire a pending write request for RaftClientRequest:client-38E7254A5AF1->164293f2-68e3-4851-bc46-4a828bd79ffa@group-03010B1A5718, cid=3, seq=4, RW, Message:000000b2080612343362...(size=182) at org.apache.ratis.server.impl.RaftServerImpl.appendTransaction(RaftServerImpl.java:514) at org.apache.ratis.server.impl.RaftServerImpl.submitClientRequestAsync(RaftServerImpl.java:589) at org.apache.ratis.server.impl.RaftServerProxy.lambda$submitClientRequestAsync$7(RaftServerProxy.java:333) at org.apache.ratis.server.impl.RaftServerProxy.lambda$null$5(RaftServerProxy.java:328) at org.apache.ratis.util.JavaUtils.callAsUnchecked(JavaUtils.java:109) at org.apache.ratis.server.impl.RaftServerProxy.lambda$submitRequest$6(RaftServerProxy.java:328) at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:981) at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2124) at org.apache.ratis.server.impl.RaftServerProxy.submitRequest(RaftServerProxy.java:327) at org.apache.ratis.server.impl.RaftServerProxy.submitClientRequestAsync(RaftServerProxy.java:333) at org.apache.ratis.grpc.client.GrpcClientProtocolService$RequestStreamObserver.processClientRequest(GrpcClientProtocolService.java:221) at org.apache.ratis.grpc.client.GrpcClientProtocolService$OrderedRequestStreamObserver.processClientRequest(GrpcClientProtocolService.java:327) at org.apache.ratis.util.SlidingWindow$Server.processRequestsFromHead(SlidingWindow.java:429) at org.apache.ratis.util.SlidingWindow$Server.receivedRequest(SlidingWindow.java:421) at org.apache.ratis.grpc.client.GrpcClientProtocolService$OrderedRequestStreamObserver.processClientRequest(GrpcClientProtocolService.java:346) at org.apache.ratis.grpc.client.GrpcClientProtocolService$RequestStreamObserver.onNext(GrpcClientProtocolService.java:241) at org.apache.ratis.grpc.client.GrpcClientProtocolService$RequestStreamObserver.onNext(GrpcClientProtocolService.java:168) at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:251) at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailableInternal(ServerCallImpl.java:309) at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:292) at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:779) ... 5 more

2019-12-10 19:50:31,860 [grpc-default-executor-5] INFO ratis.ContainerStateMachine (ContainerStateMachine.java:preAppendTransaction(311)) - append seqNum:5 WriteChunk 
{code}
Further while failing the client request server does not invalidate the retry cache. Any retries from the client are therefore ignored in the server.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)