[jira] [Created] (HDDS-5529) scm should be terminated when IOException is throw by rocksdb

["Jackson Yao (Jira)" <ji...@apache.org>]

Jackson Yao created HDDS-5529:
---------------------------------

             Summary: scm should be terminated when IOException is throw by rocksdb
                 Key: HDDS-5529
                 URL: https://issues.apache.org/jira/browse/HDDS-5529
             Project: Apache Ozone
          Issue Type: Sub-task
            Reporter: Jackson Yao


@Replicate
  void updatePipelineState(
      HddsProtos.PipelineID pipelineIDProto,
      HddsProtos.PipelineState newState
  )
      throws IOException;  @Override
  public void updatePipelineState(
      HddsProtos.PipelineID pipelineIDProto, HddsProtos.PipelineState newState)
      throws IOException {
    PipelineID pipelineID = PipelineID.getFromProtobuf(pipelineIDProto);
    Pipeline.PipelineState oldState = null;
    lock.writeLock().lock();
    try {
      oldState = getPipeline(pipelineID).getPipelineState();
      // null check is here to prevent the case where SCM store
      // is closed but the staleNode handlers/pipeline creations
      // still try to access it.
      if (pipelineStore != null) {
        pipelineStateMap.updatePipelineState(pipelineID,
            Pipeline.PipelineState.fromProtobuf(newState));
        transactionBuffer
            .addToBuffer(pipelineStore, pipelineID, getPipeline(pipelineID));
      }
    } catch (PipelineNotFoundException pnfe) {
      LOG.warn("Pipeline {} is not found in the pipeline Map. Pipeline"
          + " may have been deleted already.", pipelineID);
    } catch (IOException ex) {
      LOG.warn("Pipeline {} state update failed", pipelineID);
      // revert back to old state in memory
      pipelineStateMap.updatePipelineState(pipelineID, oldState);
    } finally {
      lock.writeLock().unlock();
    }
  }
it is annotated with {{@Replicate}} , so it will be replicated by ratis if HA is enabled and it is actually called when the raft log is applied. suppose we have one leader L1 , and two followes F1 and F2. if the rocksdb of L1 is ok , and the methoed is excuted successfully at L1. when L1 crashed , F1 is elected as the new leader, but the rocksdb of F1 got some error(for example , can not be written) , so when F1 applied the raft log entry,  it will get an exception, and the code of e{{xception}} will be excuted. so the point here is that , current HA implementation seems could not guarantee the consistence of different SCM, because although different scm apply the same raft logs in the same order, but the result of  {{@Replicate}}  functions may be different by other factors. leader may succeed writting DB. but follower may fail when it is elected as a new leader , this may cause the inconsistence



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org