[GitHub] [incubator-doris] morningman opened a new issue #2051: FE Startup Logic Optimization

[GitBox <gi...@apache.org>]

morningman opened a new issue #2051: FE Startup Logic Optimization
URL: https://github.com/apache/incubator-doris/issues/2051
 
 
   The current FE startup logic is confusing. The main problem is that when the metadata is not completely replayed, it will start to provide services externally, which may cause some unexpected problems. Here we reorganize the FE startup process, and also sort out the processing logic when the FE role changes.
   
   ## Startup logic
   
   The FE startup mainly completes the two parts of `replay metadata` and `startup service`, which are explained below.
   
   ### Replay metadata
   
   The initialization process of the catalog is the replay process of the metadata. All services should not be started until the entire metadata replay is complete and the role status is normal.
   
   The Catalog first gets the meta image file from the local or helper node and reads it. After that, initialize the BDBJE environment and start the StateListener to start listening to the node state change event of BDBJE.
   
   In previous implementations, the StateListener set the `feType` parameter in the Catalog after listening for state changes. And in the Catalog, there is a replay thread that continuously checks whether `feType` has changed. If there is any change, the corresponding processing is performed. This logic may cause some state change event to be missed.
   
   In the new implementation, the StateListener will send the state to a queue of the Catalog each time it listens for a state change, and the background thread of the Catalog will continue to take the state from the queue for processing, thus ensuring that no state change event will be missed.
   
   **FE's role status change**
   
   FE will only change between the following states
   
   * INIT/UNKNOWN/FOLLOWER -> MASTER
   
       To change the node to Master, first turn off the replayer thread, then re-open the BDBJE environment and doing fencing. After success, the metadata is replayed and the Master-related daemon threads are started.
       
   * INIT/UNKNOWN -> FOLLOWER/OBSERVER
   
       Start the replayer thread and start the Non Master related daemon threads.
       
   * MASTER -> other
   
       From MASTER to any other state, we need to exit the FE process directly. Because the memory state cannot be rolled back, not exiting the process will result in inconsistent memory state and metadata logs.
       
   * OBSERVER/FOLLOWER -> UNKNOWN
   
       The FE node is disconnected with MASTER. In this case, we just keep everything unchanged. And if it is disconnected with MASTER for a long enough time, this FE will become unreadable.
   
   **FE Ready Status**
   
   FE requires two flags to identify if FE is ready.
   
   * `isReady`
   
       `isReady` is true, indicating that the FE is ready, that is, FE's states is in one of MASTER/FOLLOWER/OBSERVER, and the metadata is replayed normally.
       
   * `canRead`
   
       `canRead` is true, indicating that FE can provide read service. When the FOLLOWER/OBSERVER status changes to UNKNOWN, `isReady` is false, but FE can still provide read service. `canRead` will be set to false only after the node and Master have lost contact for more than a certain amount of time.
       
   `isReady` is false, indicating that FE is in a state change, or an error occurs when replaying metadata.
   
   When `canRead` is false, the FE can no longer continue to provide read service.
   
   Subsequent implementations of `isReady` and `canRead` can be used to disable the execution of certain operations.
       
   ### Start service
   
   FE ready, then start the external service port to ensure data integrity. External service ports include http services, thrift services, and MySQL services.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org