You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Jackson Yao (Jira)" <ji...@apache.org> on 2021/08/03 08:26:00 UTC
[jira] [Assigned] (HDDS-5529) scm should be terminated when
IOException is throw by rocksdb
[ https://issues.apache.org/jira/browse/HDDS-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jackson Yao reassigned HDDS-5529:
---------------------------------
Assignee: Jackson Yao
> scm should be terminated when IOException is throw by rocksdb
> -------------------------------------------------------------
>
> Key: HDDS-5529
> URL: https://issues.apache.org/jira/browse/HDDS-5529
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Jackson Yao
> Assignee: Jackson Yao
> Priority: Major
>
> @Replicate
> void updatePipelineState(
> HddsProtos.PipelineID pipelineIDProto,
> HddsProtos.PipelineState newState
> )
> throws IOException; @Override
> public void updatePipelineState(
> HddsProtos.PipelineID pipelineIDProto, HddsProtos.PipelineState newState)
> throws IOException {
> PipelineID pipelineID = PipelineID.getFromProtobuf(pipelineIDProto);
> Pipeline.PipelineState oldState = null;
> lock.writeLock().lock();
> try {
> oldState = getPipeline(pipelineID).getPipelineState();
> // null check is here to prevent the case where SCM store
> // is closed but the staleNode handlers/pipeline creations
> // still try to access it.
> if (pipelineStore != null) {
> pipelineStateMap.updatePipelineState(pipelineID,
> Pipeline.PipelineState.fromProtobuf(newState));
> transactionBuffer
> .addToBuffer(pipelineStore, pipelineID, getPipeline(pipelineID));
> }
> } catch (PipelineNotFoundException pnfe) {
> LOG.warn("Pipeline {} is not found in the pipeline Map. Pipeline"
> + " may have been deleted already.", pipelineID);
> } catch (IOException ex) {
> LOG.warn("Pipeline {} state update failed", pipelineID);
> // revert back to old state in memory
> pipelineStateMap.updatePipelineState(pipelineID, oldState);
> } finally {
> lock.writeLock().unlock();
> }
> }
> it is annotated with {{@Replicate}} , so it will be replicated by ratis if HA is enabled and it is actually called when the raft log is applied. suppose we have one leader L1 , and two followes F1 and F2. if the rocksdb of L1 is ok , and the methoed is excuted successfully at L1. when L1 crashed , F1 is elected as the new leader, but the rocksdb of F1 got some error(for example , can not be written) , so when F1 applied the raft log entry, it will get an exception, and the code of e{{xception}} will be excuted. so the point here is that , current HA implementation seems could not guarantee the consistence of different SCM, because although different scm apply the same raft logs in the same order, but the result of {{@Replicate}} functions may be different by other factors. leader may succeed writting DB. but follower may fail when it is elected as a new leader , this may cause the inconsistence
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org