You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Ashish Kumar (Jira)" <ji...@apache.org> on 2025/10/28 09:13:00 UTC
[jira] [Updated] (HDDS-13846) Container CLOSED with sequence ID lower than a replica
[ https://issues.apache.org/jira/browse/HDDS-13846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashish Kumar updated HDDS-13846:
--------------------------------
Description:
In below case SCM closes container when sequence id of container is lesser than replica sequence id.
{code:java}
ERROR [node1-FixedThreadPoolWithAffinityExecutor-9-0]-org.apache.hadoop.hdds.scm.container.ContainerReportHandler: Container CLOSED with sequence ID lower than a replica: Container #2006 (CLOSED, sid=5408) r0 (CLOSED, bcsid=6556, origin=858ea3bc-f045-442f-a34e-6a3e3c6e38f4, non-empty) from dn 858ea3bc-f045-442f-a34e-6a3e3c6e38f4(xxxxxx), proto=containerID: 2006 state: CLOSED used: 4364173816 keyCount: 26 readCount: 411 writeCount: 1215 readBytes: 1723858944 writeBytes: 4988468861 deleteTransactionId: 7006 blockCommitSequenceId: 6304 originNodeId: "858ea3bc-f045-442f-a34e-6a3e3c6e38f4" replicaIndex: 0 isEmpty: false dataChecksum: 121871427{code}
h4. Assume below scenario:
h4. Initial State:
* Leader SCM: {{containerInfo(state=CLOSING, sequenceId=6556)}} (in memory)
* Follower SCM: {{containerInfo(state=CLOSING, sequenceId=5408)}} (in memory)
h4. When CLOSE Event Triggers on Leader:
*Step 1 - Leader Processing:*
{code:java}
// Leader SCM:
final ContainerInfo oldInfo = containers.getContainerInfo(id);
// sequenceId=6556
containers.updateState(id, CLOSING, CLOSED); // State: CLOSING → CLOSED transactionBuffer.addToBuffer(containerStore, id, containers.getContainerInfo(id));
// Persists: state=CLOSED, sequenceId=6556{code}
*Step 2 - Raft Replication:*
{code:java}
// Raft log entry contains: CLOSE event for containerID
// NO sequenceId information in the Raft log!{code}
*Step 3 - Follower Processing:*
{code:java}
// Follower SCM applies the same transaction:
final ContainerInfo oldInfo = containers.getContainerInfo(id);
// sequenceId=5408
containers.updateState(id, CLOSING, CLOSED); // State: CLOSING → CLOSED
transactionBuffer.addToBuffer(containerStore, id, containers.getContainerInfo(id));
// Persists: state=CLOSED, sequenceId=5408 (STALE){code}
After the CLOSE operation:
* Leader DB: {{state=CLOSED, sequenceId=6556}}
* Follower DB: {{state=CLOSED, sequenceId=5408}}
Since recently log is added in HDDS-12409 it is more visible.
But eventually in follower as well sequence id is getting [updated in memory|https://github.infra.cloudera.com/CDH/ozone/blob/020aa2248702aa7402becc7b5229dce096a153e0/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/AbstractContainerReportHandler.java#L142](In DB it remains old).
This is wrt current code logic, as in follower sequence id update is eventual but state is consistent across all 3 SCMs.
was:
In below case SCM closes container when sequence id of container is lesser than replica sequence id.
{{}}
{code:java}
ERROR [node1-FixedThreadPoolWithAffinityExecutor-9-0]-org.apache.hadoop.hdds.scm.container.ContainerReportHandler: Container CLOSED with sequence ID lower than a replica: Container #2006 (CLOSED, sid=5408) r0 (CLOSED, bcsid=6556, origin=858ea3bc-f045-442f-a34e-6a3e3c6e38f4, non-empty) from dn 858ea3bc-f045-442f-a34e-6a3e3c6e38f4(xxxxxx), proto=containerID: 2006 state: CLOSED used: 4364173816 keyCount: 26 readCount: 411 writeCount: 1215 readBytes: 1723858944 writeBytes: 4988468861 deleteTransactionId: 7006 blockCommitSequenceId: 6304 originNodeId: "858ea3bc-f045-442f-a34e-6a3e3c6e38f4" replicaIndex: 0 isEmpty: false dataChecksum: 121871427{code}
{{}}
h4. Assume below scenario:
h4. Initial State:
* Leader SCM: {{containerInfo(state=CLOSING, sequenceId=6556)}} (in memory)
* Follower SCM: {{containerInfo(state=CLOSING, sequenceId=5408)}} (in memory)
h4. When CLOSE Event Triggers on Leader:
*Step 1 - Leader Processing:*
{{}}
{code:java}
// Leader SCM:
final ContainerInfo oldInfo = containers.getContainerInfo(id);
// sequenceId=6556
containers.updateState(id, CLOSING, CLOSED); // State: CLOSING → CLOSED transactionBuffer.addToBuffer(containerStore, id, containers.getContainerInfo(id));
// Persists: state=CLOSED, sequenceId=6556{code}
{{}}
*Step 2 - Raft Replication:*
{{}}
{code:java}
// Raft log entry contains: CLOSE event for containerID
// NO sequenceId information in the Raft log!{code}
{{}}
*Step 3 - Follower Processing:*
{{}}
{code:java}
// Follower SCM applies the same transaction:
final ContainerInfo oldInfo = containers.getContainerInfo(id);
// sequenceId=5408
containers.updateState(id, CLOSING, CLOSED); // State: CLOSING → CLOSED
transactionBuffer.addToBuffer(containerStore, id, containers.getContainerInfo(id));
// Persists: state=CLOSED, sequenceId=5408 (STALE){code}
{{}}
After the CLOSE operation:
* Leader DB: {{state=CLOSED, sequenceId=6556}}
* Follower DB: {{state=CLOSED, sequenceId=5408}}
Since recently log is added in HDDS-12409 it is more visible.
But eventually in follower as well sequence id is getting [updated in memory|https://github.infra.cloudera.com/CDH/ozone/blob/020aa2248702aa7402becc7b5229dce096a153e0/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/AbstractContainerReportHandler.java#L142](In DB it remains old).
This is wrt current code logic, as in follower sequence id update is eventual but state is consistent across all 3 SCMs.
> Container CLOSED with sequence ID lower than a replica
> ------------------------------------------------------
>
> Key: HDDS-13846
> URL: https://issues.apache.org/jira/browse/HDDS-13846
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Ashish Kumar
> Assignee: Ashish Kumar
> Priority: Major
>
> In below case SCM closes container when sequence id of container is lesser than replica sequence id.
> {code:java}
> ERROR [node1-FixedThreadPoolWithAffinityExecutor-9-0]-org.apache.hadoop.hdds.scm.container.ContainerReportHandler: Container CLOSED with sequence ID lower than a replica: Container #2006 (CLOSED, sid=5408) r0 (CLOSED, bcsid=6556, origin=858ea3bc-f045-442f-a34e-6a3e3c6e38f4, non-empty) from dn 858ea3bc-f045-442f-a34e-6a3e3c6e38f4(xxxxxx), proto=containerID: 2006 state: CLOSED used: 4364173816 keyCount: 26 readCount: 411 writeCount: 1215 readBytes: 1723858944 writeBytes: 4988468861 deleteTransactionId: 7006 blockCommitSequenceId: 6304 originNodeId: "858ea3bc-f045-442f-a34e-6a3e3c6e38f4" replicaIndex: 0 isEmpty: false dataChecksum: 121871427{code}
> h4. Assume below scenario:
> h4. Initial State:
> * Leader SCM: {{containerInfo(state=CLOSING, sequenceId=6556)}} (in memory)
> * Follower SCM: {{containerInfo(state=CLOSING, sequenceId=5408)}} (in memory)
> h4. When CLOSE Event Triggers on Leader:
> *Step 1 - Leader Processing:*
> {code:java}
> // Leader SCM:
> final ContainerInfo oldInfo = containers.getContainerInfo(id);
> // sequenceId=6556
> containers.updateState(id, CLOSING, CLOSED); // State: CLOSING → CLOSED transactionBuffer.addToBuffer(containerStore, id, containers.getContainerInfo(id));
> // Persists: state=CLOSED, sequenceId=6556{code}
> *Step 2 - Raft Replication:*
> {code:java}
> // Raft log entry contains: CLOSE event for containerID
> // NO sequenceId information in the Raft log!{code}
> *Step 3 - Follower Processing:*
> {code:java}
> // Follower SCM applies the same transaction:
> final ContainerInfo oldInfo = containers.getContainerInfo(id);
> // sequenceId=5408
> containers.updateState(id, CLOSING, CLOSED); // State: CLOSING → CLOSED
> transactionBuffer.addToBuffer(containerStore, id, containers.getContainerInfo(id));
> // Persists: state=CLOSED, sequenceId=5408 (STALE){code}
> After the CLOSE operation:
> * Leader DB: {{state=CLOSED, sequenceId=6556}}
> * Follower DB: {{state=CLOSED, sequenceId=5408}}
> Since recently log is added in HDDS-12409 it is more visible.
> But eventually in follower as well sequence id is getting [updated in memory|https://github.infra.cloudera.com/CDH/ozone/blob/020aa2248702aa7402becc7b5229dce096a153e0/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/AbstractContainerReportHandler.java#L142](In DB it remains old).
> This is wrt current code logic, as in follower sequence id update is eventual but state is consistent across all 3 SCMs.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org