[jira] [Created] (RATIS-1695) Improve observability and internal error detection in RaftServer

["Jiacheng Liu (Jira)" <ji...@apache.org>]

Jiacheng Liu created RATIS-1695:
-----------------------------------

             Summary: Improve observability and internal error detection in RaftServer
                 Key: RATIS-1695
                 URL: https://issues.apache.org/jira/browse/RATIS-1695
             Project: Ratis
          Issue Type: Improvement
          Components: server
            Reporter: Jiacheng Liu


In Ratis many threads are created using `Daemon` class manually. For threads like this, if there's an uncaught exception, the thread will just crash silently without other components knowing. If the thread happens to be a critical component then some part of the RaftServer is essentially down, whereas the RaftServer's lifecycle is still RUNNING (not set to EXCEPTION because the thread didn't have a chance).

One example where this can happen is [https://github.com/apache/ratis/pull/417/files] Before this change is in, the StateMachineUpdater thread can throw NPE and exit, so the follower RaftServer stays stale forever. The RaftServer's lifecycle is RUNNING and there's no way for the external party to know by `RaftServer.getLifeCycleState()`.

The proposal is to improve observability on RaftServer to ensure an uncaught exception can be caught and propagated to the external user, by multiple folds:
 # For all `Daemon` threads, they should have UncaughtExceptionHandler set.
 # Add an extra field to the RaftServer to store an exception, and that field can be set by the UncaughtExceptionHandler instances.
 # The UncaughtExceptionHandler also transitions the RaftServer to EXCEPTION state.

So external users can 
{code:java}
RaftServer server = RaftServer.newBuilder().build();
// Periodically check
if (server.getLifeCycleState() == State.EXCEPTION) {
  Throwable t = server.getError();
  // Deal with the throwable
}{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)