You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Jackson Yao (Jira)" <ji...@apache.org> on 2021/08/03 08:26:00 UTC
[jira] [Updated] (HDDS-5529) scm should be terminated when
IOException is throw by rocksdb
[ https://issues.apache.org/jira/browse/HDDS-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jackson Yao updated HDDS-5529:
------------------------------
Description:
```
@Replicate
void updatePipelineState(
HddsProtos.PipelineID pipelineIDProto,
HddsProtos.PipelineState newState
)
throws IOException; @Override
public void updatePipelineState(
HddsProtos.PipelineID pipelineIDProto, HddsProtos.PipelineState newState)
throws IOException {
PipelineID pipelineID = PipelineID.getFromProtobuf(pipelineIDProto);
Pipeline.PipelineState oldState = null;
lock.writeLock().lock();
try {
oldState = getPipeline(pipelineID).getPipelineState();
// null check is here to prevent the case where SCM store
// is closed but the staleNode handlers/pipeline creations
// still try to access it.
if (pipelineStore != null)
{ pipelineStateMap.updatePipelineState(pipelineID, Pipeline.PipelineState.fromProtobuf(newState)); transactionBuffer .addToBuffer(pipelineStore, pipelineID, getPipeline(pipelineID)); }
} catch (PipelineNotFoundException pnfe) {
LOG.warn("Pipeline {} is not found in the pipeline Map. Pipeline"
+ " may have been deleted already.", pipelineID);
} catch (IOException ex) {
LOG.warn("Pipeline {} state update failed", pipelineID);
// revert back to old state in memory
pipelineStateMap.updatePipelineState(pipelineID, oldState);
} finally
{ lock.writeLock().unlock(); }
}
```
it is annotated with {{@Replicate}} , so it will be replicated by ratis if HA is enabled and it is actually called when the raft log is applied. suppose we have one leader L1 , and two followes F1 and F2. if the rocksdb of L1 is ok , and the methoed is excuted successfully at L1. when L1 crashed , F1 is elected as the new leader, but the rocksdb of F1 got some error(for example , can not be written) , so when F1 applied the raft log entry, it will get an exception, and the code of e\{{xception}} will be excuted. so the point here is that , current HA implementation seems could not guarantee the consistence of different SCM, because although different scm apply the same raft logs in the same order, but the result of {{@Replicate}} functions may be different by other factors. leader may succeed writting DB. but follower may fail when it is elected as a new leader , this may cause the inconsistence
was:
@Replicate
void updatePipelineState(
HddsProtos.PipelineID pipelineIDProto,
HddsProtos.PipelineState newState
)
throws IOException; @Override
public void updatePipelineState(
HddsProtos.PipelineID pipelineIDProto, HddsProtos.PipelineState newState)
throws IOException {
PipelineID pipelineID = PipelineID.getFromProtobuf(pipelineIDProto);
Pipeline.PipelineState oldState = null;
lock.writeLock().lock();
try {
oldState = getPipeline(pipelineID).getPipelineState();
// null check is here to prevent the case where SCM store
// is closed but the staleNode handlers/pipeline creations
// still try to access it.
if (pipelineStore != null) {
pipelineStateMap.updatePipelineState(pipelineID,
Pipeline.PipelineState.fromProtobuf(newState));
transactionBuffer
.addToBuffer(pipelineStore, pipelineID, getPipeline(pipelineID));
}
} catch (PipelineNotFoundException pnfe) {
LOG.warn("Pipeline {} is not found in the pipeline Map. Pipeline"
+ " may have been deleted already.", pipelineID);
} catch (IOException ex) {
LOG.warn("Pipeline {} state update failed", pipelineID);
// revert back to old state in memory
pipelineStateMap.updatePipelineState(pipelineID, oldState);
} finally {
lock.writeLock().unlock();
}
}
it is annotated with {{@Replicate}} , so it will be replicated by ratis if HA is enabled and it is actually called when the raft log is applied. suppose we have one leader L1 , and two followes F1 and F2. if the rocksdb of L1 is ok , and the methoed is excuted successfully at L1. when L1 crashed , F1 is elected as the new leader, but the rocksdb of F1 got some error(for example , can not be written) , so when F1 applied the raft log entry, it will get an exception, and the code of e{{xception}} will be excuted. so the point here is that , current HA implementation seems could not guarantee the consistence of different SCM, because although different scm apply the same raft logs in the same order, but the result of {{@Replicate}} functions may be different by other factors. leader may succeed writting DB. but follower may fail when it is elected as a new leader , this may cause the inconsistence
> scm should be terminated when IOException is throw by rocksdb
> -------------------------------------------------------------
>
> Key: HDDS-5529
> URL: https://issues.apache.org/jira/browse/HDDS-5529
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Jackson Yao
> Assignee: Jackson Yao
> Priority: Major
>
> ```
> @Replicate
> void updatePipelineState(
> HddsProtos.PipelineID pipelineIDProto,
> HddsProtos.PipelineState newState
> )
> throws IOException; @Override
> public void updatePipelineState(
> HddsProtos.PipelineID pipelineIDProto, HddsProtos.PipelineState newState)
> throws IOException {
> PipelineID pipelineID = PipelineID.getFromProtobuf(pipelineIDProto);
> Pipeline.PipelineState oldState = null;
> lock.writeLock().lock();
> try {
> oldState = getPipeline(pipelineID).getPipelineState();
> // null check is here to prevent the case where SCM store
> // is closed but the staleNode handlers/pipeline creations
> // still try to access it.
> if (pipelineStore != null)
> { pipelineStateMap.updatePipelineState(pipelineID, Pipeline.PipelineState.fromProtobuf(newState)); transactionBuffer .addToBuffer(pipelineStore, pipelineID, getPipeline(pipelineID)); }
> } catch (PipelineNotFoundException pnfe) {
> LOG.warn("Pipeline {} is not found in the pipeline Map. Pipeline"
> + " may have been deleted already.", pipelineID);
> } catch (IOException ex) {
> LOG.warn("Pipeline {} state update failed", pipelineID);
> // revert back to old state in memory
> pipelineStateMap.updatePipelineState(pipelineID, oldState);
> } finally
> { lock.writeLock().unlock(); }
> }
> ```
> it is annotated with {{@Replicate}} , so it will be replicated by ratis if HA is enabled and it is actually called when the raft log is applied. suppose we have one leader L1 , and two followes F1 and F2. if the rocksdb of L1 is ok , and the methoed is excuted successfully at L1. when L1 crashed , F1 is elected as the new leader, but the rocksdb of F1 got some error(for example , can not be written) , so when F1 applied the raft log entry, it will get an exception, and the code of e\{{xception}} will be excuted. so the point here is that , current HA implementation seems could not guarantee the consistence of different SCM, because although different scm apply the same raft logs in the same order, but the result of {{@Replicate}} functions may be different by other factors. leader may succeed writting DB. but follower may fail when it is elected as a new leader , this may cause the inconsistence
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org