[jira] [Commented] (RATIS-2315) Ignore ExecutionException during take checkpoint check

["Tsz-wo Sze (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/RATIS-2315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18000778#comment-18000778 ] 

Tsz-wo Sze commented on RATIS-2315:
-----------------------------------

[~Sammi] and Hao Guo, you are right that Ratis should ignore the exception from StateMachine.

RATIS-2245 should has fixed it.  Ozone in the master branch has updated to Ratis 3.2.0.  Would you be able to check if the problem still exists with Ratis 3.2.0 ?

> Ignore ExecutionException during take checkpoint check
> ------------------------------------------------------
>
>                 Key: RATIS-2315
>                 URL: https://issues.apache.org/jira/browse/RATIS-2315
>             Project: Ratis
>          Issue Type: Improvement
>            Reporter: Sammi Chen
>            Priority: Major
>
> When SCM raft reapply during restart, SCMStateMachine#applyTransaction could execute 
> {code:java}
> "applyTransactionFuture.completeExceptionally(ex);"
> {code}
>  for ContainerStateManagerImpl#addContainer operation, once it fails at 
> {code:java}
> pipelineManager.addContainerToPipeline(pipelineID, containerID);
> {code}
> The failure message is likes "Cannot add container to pipeline=PipelineID=b2f717d8-3912-424c-b42a-e0b52c305c97 in closed state". 
> This didn't crash the SCM if it happens after SCM has started and running. It also did't crash every peer of SCM in the raft group.  The root cause is StateMachineUpdater#run -> StateMachineUpdater#checkAndTakeSnapshot
> {code:java}
> private void checkAndTakeSnapshot(MemoizedSupplier<List<CompletableFuture<Message>>> futures)
>       throws ExecutionException, InterruptedException {
>     // check if need to trigger a snapshot
>     if (shouldTakeSnapshot()) {
>       if (futures.isInitialized()) {
>         JavaUtils.allOf(futures.get()).get();
>       }
>       takeSnapshot();
>     }
>   }
> {code}
> When shouldTakeSnapshot() is false, it doesn't care about the futures result. When shouldTakeSnapshot is true, if one of futures throws exception, checkAndTakeSnapshot will throws ExecutionException, which in turn shutdown the raft server in  StateMachineUpdater#run. 
> So the behavior when shouldTakeSnapshot false, and true are different. It's better have the aligned behavior. The proposal of this JIRA is to ignore the ExecutionException exception when shouldTakeSnapshot() is true. 
> The above problem is reported and co-analyzed by "Hao Guo". 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)