You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "navinko (via GitHub)" <gi...@apache.org> on 2026/05/19 18:59:13 UTC
[PR] HDDS-15288. Use SequenceIdType in StateManagerImpl. [ozone]
navinko opened a new pull request, #10313:
URL: https://github.com/apache/ozone/pull/10313
## What changes were proposed in this pull request?
Used SequenceIdType for sequenceIdToLastIdMap in StateManagerImpl
## Please describe your PR in detail:
Refactored sequenceIdToLastIdMap in StateManagerImpl from <String, Long> to <SequenceIdType, Long>.
Updated allocateBatch(), getLastId(), and initialize() to safely convert DB key of string type to SequenceIdType Enums using SequenceIdType.valueOf().
Also as per the Jira, the @Replicate method signatures were NOT changed to maintain RPC compatibility.
Regarding the change related to other in-memory data structure -sequenceIdTable, will create a separate PR with SequenceIdType codec by moving away from default StringCodec to support serialization and adapt SequenceIdType
for Table<String, Long> --> Table<SequenceIdType, Long>
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-15288
## How was this patch tested?
Added new test cases under TestSequenceIDGenerator and validated changes.
Ran locally, docker-compose and validated the flow for reinitialise with scm failover
> ozonesecure-ha % docker compose logs -f scm1.org scm2.org 2>&1 | grep -iE "SequenceIdGenerator|allocateBatch"
scm2.org-1 | 2026-05-19 11:07:13,743 [main] INFO ha.SequenceIdGenerator: upgrade localId to 117883640217600000
scm2.org-1 | 2026-05-19 11:07:13,744 [main] INFO ha.SequenceIdGenerator: upgrade delTxnId to 0
scm2.org-1 | 2026-05-19 11:07:13,747 [main] INFO ha.SequenceIdGenerator: upgrade containerId to 0
scm2.org-1 | 2026-05-19 11:07:13,748 [main] INFO ha.SequenceIdGenerator: upgrade CertificateId to 2
scm2.org-1 | 2026-05-19 11:07:13,748 [main] INFO ha.SequenceIdGenerator: Init the HA SequenceIdGenerator.
scm2.org-1 | 2026-05-19 11:07:14,412 [a0c35110-bb90-45b4-862f-e7d338f1a711@group-DA1743C8D70B-StateMachineUpdater] INFO ha.SequenceIdGenerator: reinitialize SequenceIdGenerator.
scm2.org-1 | 2026-05-19 11:14:28,423 [a0c35110-bb90-45b4-862f-e7d338f1a711@group-DA1743C8D70B-StateMachineUpdater] WARN ha.SequenceIdGenerator: Failed to allocate a batch for localId, expected lastId is 0, actual lastId is 117883640217600000.
scm1.org-1 | 2026-05-19 11:07:06,181 [main] INFO ha.SequenceIdGenerator: upgrade localId to 117883640217600000
scm1.org-1 | 2026-05-19 11:07:06,181 [main] INFO ha.SequenceIdGenerator: upgrade delTxnId to 0
scm1.org-1 | 2026-05-19 11:07:06,183 [main] INFO ha.SequenceIdGenerator: upgrade containerId to 0
scm1.org-1 | 2026-05-19 11:07:06,183 [main] INFO ha.SequenceIdGenerator: upgrade CertificateId to 2
scm1.org-1 | 2026-05-19 11:07:06,184 [main] INFO ha.SequenceIdGenerator: Init the HA SequenceIdGenerator.
scm1.org-1 | 2026-05-19 11:07:06,296 [main] INFO ha.SequenceIdGenerator: upgrade CertificateId to 2
scm1.org-1 | 2026-05-19 11:07:12,154 [9c364a4e-6060-4490-a237-6a9e96102d15@group-DA1743C8D70B-StateMachineUpdater] WARN ha.SequenceIdGenerator: Failed to allocate a batch for CertificateId, expected lastId is 0, actual lastId is 2.
scm1.org-1 | 2026-05-19 11:07:12,157 [IPC Server handler 0 on default port 9961] INFO ha.SequenceIdGenerator: Allocate a batch for CertificateId, change lastId from 2 to 3.
scm1.org-1 | 2026-05-19 11:07:15,999 [IPC Server handler 0 on default port 9961] INFO ha.SequenceIdGenerator: Allocate a batch for CertificateId, change lastId from 3 to 4.
scm1.org-1 | 2026-05-19 11:07:16,134 [IPC Server handler 1 on default port 9961] INFO ha.SequenceIdGenerator: Allocate a batch for CertificateId, change lastId from 4 to 5.
scm1.org-1 | 2026-05-19 11:07:21,456 [IPC Server handler 1 on default port 9961] INFO ha.SequenceIdGenerator: Allocate a batch for CertificateId, change lastId from 5 to 6.
scm1.org-1 | 2026-05-19 11:07:21,541 [IPC Server handler 0 on default port 9961] INFO ha.SequenceIdGenerator: Allocate a batch for CertificateId, change lastId from 6 to 7.
scm1.org-1 | 2026-05-19 11:07:21,594 [IPC Server handler 1 on default port 9961] INFO ha.SequenceIdGenerator: Allocate a batch for CertificateId, change lastId from 7 to 8.
scm1.org-1 | 2026-05-19 11:07:21,632 [IPC Server handler 0 on default port 9961] INFO ha.SequenceIdGenerator: Allocate a batch for CertificateId, change lastId from 8 to 9.
scm1.org-1 | 2026-05-19 11:07:21,666 [IPC Server handler 1 on default port 9961] INFO ha.SequenceIdGenerator: Allocate a batch for CertificateId, change lastId from 9 to 10.
scm1.org-1 | 2026-05-19 11:07:21,696 [IPC Server handler 0 on default port 9961] INFO ha.SequenceIdGenerator: Allocate a batch for CertificateId, change lastId from 10 to 11.
scm1.org-1 | 2026-05-19 11:14:28,398 [IPC Server handler 20 on default port 9863] INFO ha.SequenceIdGenerator: Allocate a batch for containerId, change lastId from 0 to 1000.
scm1.org-1 | 2026-05-19 11:14:28,411 [9c364a4e-6060-4490-a237-6a9e96102d15@group-DA1743C8D70B-StateMachineUpdater] WARN ha.SequenceIdGenerator: Failed to allocate a batch for localId, expected lastId is 0, actual lastId is 117883640217600000.
scm1.org-1 | 2026-05-19 11:14:28,416 [IPC Server handler 20 on default port 9863] INFO ha.SequenceIdGenerator: Allocate a batch for localId, change lastId from 117883640217600000 to 117883640217601000.
scm1.org-1 | 2026-05-19 11:19:06,367 [main] INFO ha.SequenceIdGenerator: Init the HA SequenceIdGenerator.
scm1.org-1 | 2026-05-19 11:19:06,486 [main] INFO ha.SequenceIdGenerator: upgrade CertificateId to 11
Successful CI build:
https://github.com/navinko/ozone/actions/runs/26101162654
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15288. Use SequenceIdType in StateManagerImpl. [ozone]
Posted by "navinko (via GitHub)" <gi...@apache.org>.
navinko commented on PR #10313:
URL: https://github.com/apache/ozone/pull/10313#issuecomment-4491101990
Hi @szetszwo , added the changes. Once you get some time please review it .
Regards
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15288. Use SequenceIdType in StateManagerImpl. [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo merged PR #10313:
URL: https://github.com/apache/ozone/pull/10313
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15288. Use SequenceIdType in StateManagerImpl. [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on code in PR #10313:
URL: https://github.com/apache/ozone/pull/10313#discussion_r3283450326
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SequenceIdGenerator.java:
##########
@@ -259,18 +254,18 @@ public Boolean allocateBatch(String sequenceIdName,
try {
transactionBuffer
- .addToBuffer(sequenceIdTable, sequenceIdName, newLastId);
+ .addToBuffer(sequenceIdTable, idType.name(), newLastId);
} catch (IOException ioe) {
throw new RuntimeException("Failed to put lastId to Batch", ioe);
}
- sequenceIdToLastIdMap.put(sequenceIdName, newLastId);
+ sequenceIdToLastIdMap.put(idType, newLastId);
return true;
}
@Override
public Long getLastId(String sequenceIdName) {
Review Comment:
Since getLastId is a local method, we can change it to use SequenceIdType.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15288. Use SequenceIdType in StateManagerImpl. [ozone]
Posted by "navinko (via GitHub)" <gi...@apache.org>.
navinko commented on code in PR #10313:
URL: https://github.com/apache/ozone/pull/10313#discussion_r3284205493
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SequenceIdGenerator.java:
##########
@@ -259,18 +254,18 @@ public Boolean allocateBatch(String sequenceIdName,
try {
transactionBuffer
- .addToBuffer(sequenceIdTable, sequenceIdName, newLastId);
+ .addToBuffer(sequenceIdTable, idType.name(), newLastId);
} catch (IOException ioe) {
throw new RuntimeException("Failed to put lastId to Batch", ioe);
}
- sequenceIdToLastIdMap.put(sequenceIdName, newLastId);
+ sequenceIdToLastIdMap.put(idType, newLastId);
return true;
}
@Override
public Long getLastId(String sequenceIdName) {
Review Comment:
Thanks @szetszwo updated the changes
- updated interface method signature and impl
- updated the existing test cases
- ran test locally and build success
- CI build progressing : https://github.com/navinko/ozone/actions/runs/26251573876
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15288. Use SequenceIdType in StateManagerImpl. [ozone]
Posted by "navinko (via GitHub)" <gi...@apache.org>.
navinko commented on PR #10313:
URL: https://github.com/apache/ozone/pull/10313#issuecomment-4529463357
> @navinko , we also need to update SequenceIdGeneratorStateManagerInvoker. It does not compile with master.
Thanks @szetszwo , updated the changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15288. Use SequenceIdType in StateManagerImpl. [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on PR #10313:
URL: https://github.com/apache/ozone/pull/10313#issuecomment-4513079933
@navinko , we also need to update SequenceIdGeneratorStateManagerInvoker. It does not compile with master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org