You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "ss77892 (via GitHub)" <gi...@apache.org> on 2026/07/31 15:06:19 UTC
[PR] HDDS-16035. Secure SCM HA cluster cannot start: the primary SCM refuses to sign certificates for the other SCMs when there is no leader [ozone]
ss77892 opened a new pull request, #10916:
URL: https://github.com/apache/ozone/pull/10916
## What changes were proposed in this pull request?
HDDS-16035. Secure SCM HA cluster cannot start: the primary SCM refuses to sign certificates for the other SCMs when there is no leader
HDDS-16035. Secure SCM HA cluster cannot start: the primary SCM refuses to sign certificates for the other SCMs when there is no leader.
Let a single narrow case skip the leader check: a GetSCMCertificate request that is not a renewal, served by the primary SCM (the one holding the root CA), at a moment when no leader is known. Signing on that path also needs a serial number and a database write, which normally go through Ratis, so it would use a direct serial allocator next to the existing SequenceIdGenerator.upgradeToCertificateSequenceId() and the existing unreplicated storeValidScmCertificate() write that persistPrimarySCMCerts() already uses on every primary start. The current checkValidCertID() check stays, so a repeated serial is refused and never issued, and the path logs a warning so an operator can see it ran. Everything else keeps the leader check: renewals, non-primary SCMs, and OM and datanode certificates. When a leader exists, nothing changes, so this is not a revert of [HDDS-8286](https://issues.apache.org/jira/browse/HDDS-8286). It needs no proto change, no new config key, and no interface change.
One known gap to note: a certificate signed this way is written only in the primary's local database, which is the same exposure persistPrimarySCMCerts() already has, and each SCM keeps its own certificate on local disk, so startup does not depend on the replicated copy.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-16035
## How was this patch tested?
1. UTs for all changed parts (Protocol translator, Leaderless SCM, sequence generator)
2. The situation of the cluster before and after enabling security (Kerberos).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16035. Secure SCM HA cluster cannot start: the primary SCM refuses to sign certificates for the other SCMs when there is no leader [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10916:
URL: https://github.com/apache/ozone/pull/10916#issuecomment-5502317857
Thank you for your contribution. This PR is being closed due to inactivity. Please contact a maintainer if you would like to reopen it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16035. Secure SCM HA cluster cannot start: the primary SCM refuses to sign certificates for the other SCMs when there is no leader [ozone]
Posted by "ss77892 (via GitHub)" <gi...@apache.org>.
ss77892 commented on code in PR #10916:
URL: https://github.com/apache/ozone/pull/10916#discussion_r3716302883
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SequenceIdGenerator.java:
##########
@@ -180,6 +180,44 @@ public void reinitialize(Table<SequenceIdType, Long> sequenceIdTable)
}
}
+ /**
+ * Allocate the next CertificateId directly against the CertificateId row
+ * of {@link SCMMetadataStore#getSequenceIdTable()}, bypassing Ratis. This
+ * is used for leaderless bootstrap certificate signing, where no Ratis
+ * leader is available yet.
+ *
+ * The live StateManager cache and any un-exhausted batch are refreshed
+ * under the same lock, so that a later Ratis-based allocation of
+ * CertificateId (e.g. after a leader election) cannot CAS a stale cached
+ * lastId and reissue the value handed out here.
+ *
+ * @param scmMetadataStore : the SCMMetadataStore to allocate against.
+ * @return the newly allocated CertificateId.
+ */
+ public long getNextCertificateIdWithoutRatis(SCMMetadataStore scmMetadataStore)
+ throws IOException {
+ lock.lock();
+ try {
+ // Re-derive the CertificateId row from existing certificates if missing.
+ upgradeToCertificateSequenceId(scmMetadataStore, false);
+
+ Table<SequenceIdType, Long> sequenceIdTable = scmMetadataStore.getSequenceIdTable();
+ Long lastId = sequenceIdTable.get(SequenceIdType.CertificateId);
Review Comment:
This code is no longer relevant. But you are right; under certain conditions, there was a possibility of losing track of some certificates (still, I can hardly imagine what we would need to do to get into that).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16035. Secure SCM HA cluster cannot start: the primary SCM refuses to sign certificates for the other SCMs when there is no leader [ozone]
Posted by "ss77892 (via GitHub)" <gi...@apache.org>.
ss77892 commented on code in PR #10916:
URL: https://github.com/apache/ozone/pull/10916#discussion_r3716256330
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMSecurityProtocolServer.java:
##########
@@ -326,7 +334,29 @@ public String getSCMCertificate(ScmNodeDetailsProto scmNodeDetails,
LOGGER.info("Processing CSR for scm {}, nodeId: {}",
scmNodeDetails.getHostName(), scmNodeDetails.getScmNodeId());
- return getEncodedCertToString(certSignReq, NodeType.SCM);
+ boolean leaderless = !storageContainerManager.checkLeader()
+ && isLeaderlessPrimaryScmSigner(storageContainerManager,
+ storageContainerManager.getScmHAManager().getRatisServer().triggerNotLeaderException(),
+ isRenew);
+
+ return getEncodedCertToString(certSignReq, NodeType.SCM, leaderless, scmNodeDetails.getScmNodeId());
+ }
+
+ /**
+ * Single source of truth for whether this SCM should sign its own leaderless bootstrap SCM
+ * certificate: no Ratis leader is known cluster-wide (not merely that this node isn't leader),
+ * this is not a renewal, and this SCM hosts the primary root CA.
+ *
+ * @param scm - the serving StorageContainerManager.
+ * @param nle - the NotLeaderException produced by the local Ratis server, or null.
+ * @param isRenew - whether this request is a certificate renewal.
+ * @return true iff the leaderless SCM-certificate signing path should be used.
+ */
+ @VisibleForTesting
+ public static boolean isLeaderlessPrimaryScmSigner(StorageContainerManager scm, NotLeaderException nle,
+ boolean isRenew) {
+ return !isRenew && nle != null && nle.getSuggestedLeader() == null
Review Comment:
Well, for those who never had a certificate, there is a case where the SCM requested a certificate and died almost immediately. So, in the we track: since the start, there has been no leader elected; the request comes from an existing quorum member, and it's not a renewal; and we are the primary SCM.
As for the disjoint ID ranges, my idea was just to limit IDs for Ratis-issued certificates to something like 1<<62 and use bigger numbers for leaderless certificates. One note: we are not talking about certificates for everyone. All this code is for the SCM certificates only, so everything can be handled in one place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16035. Secure SCM HA cluster cannot start: the primary SCM refuses to sign certificates for the other SCMs when there is no leader [ozone]
Posted by "chihsuan (via GitHub)" <gi...@apache.org>.
chihsuan commented on code in PR #10916:
URL: https://github.com/apache/ozone/pull/10916#discussion_r3714155614
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMSecurityProtocolServer.java:
##########
@@ -326,7 +334,29 @@ public String getSCMCertificate(ScmNodeDetailsProto scmNodeDetails,
LOGGER.info("Processing CSR for scm {}, nodeId: {}",
scmNodeDetails.getHostName(), scmNodeDetails.getScmNodeId());
- return getEncodedCertToString(certSignReq, NodeType.SCM);
+ boolean leaderless = !storageContainerManager.checkLeader()
+ && isLeaderlessPrimaryScmSigner(storageContainerManager,
+ storageContainerManager.getScmHAManager().getRatisServer().triggerNotLeaderException(),
+ isRenew);
+
+ return getEncodedCertToString(certSignReq, NodeType.SCM, leaderless, scmNodeDetails.getScmNodeId());
+ }
+
+ /**
+ * Single source of truth for whether this SCM should sign its own leaderless bootstrap SCM
+ * certificate: no Ratis leader is known cluster-wide (not merely that this node isn't leader),
+ * this is not a renewal, and this SCM hosts the primary root CA.
+ *
+ * @param scm - the serving StorageContainerManager.
+ * @param nle - the NotLeaderException produced by the local Ratis server, or null.
+ * @param isRenew - whether this request is a certificate renewal.
+ * @return true iff the leaderless SCM-certificate signing path should be used.
+ */
+ @VisibleForTesting
+ public static boolean isLeaderlessPrimaryScmSigner(StorageContainerManager scm, NotLeaderException nle,
+ boolean isRenew) {
+ return !isRenew && nle != null && nle.getSuggestedLeader() == null
+ && scm.getRootCertificateServer() != null;
Review Comment:
I just noticed that when the primary ID is absent, the upgrade branch initializes one without an identity check:
https://github.com/apache/ozone/blob/8e3d5e6ae9bf3f74ab4a99a05b631298c2460068/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java#L940-L962
Could a non-primary SCM therefore enter this bypass? If it did, it wouldn't have any root CA material on disk, so I think it would end up generating a fresh self-signed one that no other node trusts.
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SequenceIdGenerator.java:
##########
@@ -180,6 +180,44 @@ public void reinitialize(Table<SequenceIdType, Long> sequenceIdTable)
}
}
+ /**
+ * Allocate the next CertificateId directly against the CertificateId row
+ * of {@link SCMMetadataStore#getSequenceIdTable()}, bypassing Ratis. This
+ * is used for leaderless bootstrap certificate signing, where no Ratis
+ * leader is available yet.
+ *
+ * The live StateManager cache and any un-exhausted batch are refreshed
+ * under the same lock, so that a later Ratis-based allocation of
+ * CertificateId (e.g. after a leader election) cannot CAS a stale cached
+ * lastId and reissue the value handed out here.
+ *
+ * @param scmMetadataStore : the SCMMetadataStore to allocate against.
+ * @return the newly allocated CertificateId.
+ */
+ public long getNextCertificateIdWithoutRatis(SCMMetadataStore scmMetadataStore)
+ throws IOException {
+ lock.lock();
+ try {
+ // Re-derive the CertificateId row from existing certificates if missing.
+ upgradeToCertificateSequenceId(scmMetadataStore, false);
+
+ Table<SequenceIdType, Long> sequenceIdTable = scmMetadataStore.getSequenceIdTable();
+ Long lastId = sequenceIdTable.get(SequenceIdType.CertificateId);
+ long newId = (lastId != null ? lastId : INVALID_SEQUENCE_ID) + 1;
+ sequenceIdTable.put(SequenceIdType.CertificateId, newId);
Review Comment:
One more question: if a leader checkpoint later replaces the local DB, would these updates be lost?
The primary's own certs seem to survive that because they're re-created on every start, and followers pull anything missing. A cert issued to another node doesn't look like it has either of those. Would a reconciliation step be needed once a leader becomes available?
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMSecurityProtocolServer.java:
##########
@@ -326,7 +334,29 @@ public String getSCMCertificate(ScmNodeDetailsProto scmNodeDetails,
LOGGER.info("Processing CSR for scm {}, nodeId: {}",
scmNodeDetails.getHostName(), scmNodeDetails.getScmNodeId());
- return getEncodedCertToString(certSignReq, NodeType.SCM);
+ boolean leaderless = !storageContainerManager.checkLeader()
+ && isLeaderlessPrimaryScmSigner(storageContainerManager,
+ storageContainerManager.getScmHAManager().getRatisServer().triggerNotLeaderException(),
+ isRenew);
+
+ return getEncodedCertToString(certSignReq, NodeType.SCM, leaderless, scmNodeDetails.getScmNodeId());
+ }
+
+ /**
+ * Single source of truth for whether this SCM should sign its own leaderless bootstrap SCM
+ * certificate: no Ratis leader is known cluster-wide (not merely that this node isn't leader),
+ * this is not a renewal, and this SCM hosts the primary root CA.
+ *
+ * @param scm - the serving StorageContainerManager.
+ * @param nle - the NotLeaderException produced by the local Ratis server, or null.
+ * @param isRenew - whether this request is a certificate renewal.
+ * @return true iff the leaderless SCM-certificate signing path should be used.
+ */
+ @VisibleForTesting
+ public static boolean isLeaderlessPrimaryScmSigner(StorageContainerManager scm, NotLeaderException nle,
+ boolean isRenew) {
+ return !isRenew && nle != null && nle.getSuggestedLeader() == null
Review Comment:
Your three checks sound right to me, and I think "has never had a certificate" may be the strongest one, since it bounds the bypass without depending on correct leader detection. For the membership check, would the existing peer lookup on `StorageContainerManager` work here? It reads the live Raft configuration rather than a local snapshot.
On disjoint ID ranges, I'm not sure it's cheap. The counter gets re-derived from the highest serial across both cert tables on every primary start, so a reserved high value would just become the new floor. Clients also pick their current CA by highest serial, so such a cert might get pinned everywhere. Did you have a different layout in mind?
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMSecurityProtocolServer.java:
##########
@@ -326,7 +334,29 @@ public String getSCMCertificate(ScmNodeDetailsProto scmNodeDetails,
LOGGER.info("Processing CSR for scm {}, nodeId: {}",
scmNodeDetails.getHostName(), scmNodeDetails.getScmNodeId());
- return getEncodedCertToString(certSignReq, NodeType.SCM);
+ boolean leaderless = !storageContainerManager.checkLeader()
+ && isLeaderlessPrimaryScmSigner(storageContainerManager,
+ storageContainerManager.getScmHAManager().getRatisServer().triggerNotLeaderException(),
+ isRenew);
+
+ return getEncodedCertToString(certSignReq, NodeType.SCM, leaderless, scmNodeDetails.getScmNodeId());
+ }
+
+ /**
+ * Single source of truth for whether this SCM should sign its own leaderless bootstrap SCM
+ * certificate: no Ratis leader is known cluster-wide (not merely that this node isn't leader),
+ * this is not a renewal, and this SCM hosts the primary root CA.
+ *
+ * @param scm - the serving StorageContainerManager.
+ * @param nle - the NotLeaderException produced by the local Ratis server, or null.
+ * @param isRenew - whether this request is a certificate renewal.
+ * @return true iff the leaderless SCM-certificate signing path should be used.
+ */
+ @VisibleForTesting
+ public static boolean isLeaderlessPrimaryScmSigner(StorageContainerManager scm, NotLeaderException nle,
+ boolean isRenew) {
+ return !isRenew && nle != null && nle.getSuggestedLeader() == null
Review Comment:
Thanks for the explanation. That helps. 🙏 I took another look and noticed that `checkLeader()` checks `isLeaderReady()`, while the suggested leader is read from `followerInfo`.
https://github.com/apache/ozone/blob/8e3d5e6ae9bf3f74ab4a99a05b631298c2460068/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java#L1962-L1966
https://github.com/apache/ozone/blob/8e3d5e6ae9bf3f74ab4a99a05b631298c2460068/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SCMRatisServerImpl.java#L309-L310
Ratis seems to populate only the field matching the current role, so this could also be null for a candidate, or for a leader that hasn't become ready yet. Since the bootstrap client retries indefinitely, it would keep hitting this node right through an election. I wonder how often it lands in that window. Perhaps the Javadoc could also say "no leader is known locally" rather than "cluster-wide"?
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SequenceIdGenerator.java:
##########
@@ -180,6 +180,44 @@ public void reinitialize(Table<SequenceIdType, Long> sequenceIdTable)
}
}
+ /**
+ * Allocate the next CertificateId directly against the CertificateId row
+ * of {@link SCMMetadataStore#getSequenceIdTable()}, bypassing Ratis. This
+ * is used for leaderless bootstrap certificate signing, where no Ratis
+ * leader is available yet.
+ *
+ * The live StateManager cache and any un-exhausted batch are refreshed
+ * under the same lock, so that a later Ratis-based allocation of
+ * CertificateId (e.g. after a leader election) cannot CAS a stale cached
+ * lastId and reissue the value handed out here.
+ *
+ * @param scmMetadataStore : the SCMMetadataStore to allocate against.
+ * @return the newly allocated CertificateId.
+ */
+ public long getNextCertificateIdWithoutRatis(SCMMetadataStore scmMetadataStore)
+ throws IOException {
+ lock.lock();
+ try {
+ // Re-derive the CertificateId row from existing certificates if missing.
+ upgradeToCertificateSequenceId(scmMetadataStore, false);
+
+ Table<SequenceIdType, Long> sequenceIdTable = scmMetadataStore.getSequenceIdTable();
+ Long lastId = sequenceIdTable.get(SequenceIdType.CertificateId);
Review Comment:
Thanks, that makes sense for the initial conversion. I was wondering about the disaster-recovery case, though, where an existing secure cluster loses a node and reboots it. There, the counter has already advanced, and the other SCMs still hold their own cached value. Is DR meant to be in scope here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16035. Secure SCM HA cluster cannot start: the primary SCM refuses to sign certificates for the other SCMs when there is no leader [ozone]
Posted by "ss77892 (via GitHub)" <gi...@apache.org>.
ss77892 commented on code in PR #10916:
URL: https://github.com/apache/ozone/pull/10916#discussion_r3709291939
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMSecurityProtocolServer.java:
##########
@@ -326,7 +334,29 @@ public String getSCMCertificate(ScmNodeDetailsProto scmNodeDetails,
LOGGER.info("Processing CSR for scm {}, nodeId: {}",
scmNodeDetails.getHostName(), scmNodeDetails.getScmNodeId());
- return getEncodedCertToString(certSignReq, NodeType.SCM);
+ boolean leaderless = !storageContainerManager.checkLeader()
+ && isLeaderlessPrimaryScmSigner(storageContainerManager,
+ storageContainerManager.getScmHAManager().getRatisServer().triggerNotLeaderException(),
+ isRenew);
+
+ return getEncodedCertToString(certSignReq, NodeType.SCM, leaderless, scmNodeDetails.getScmNodeId());
+ }
+
+ /**
+ * Single source of truth for whether this SCM should sign its own leaderless bootstrap SCM
+ * certificate: no Ratis leader is known cluster-wide (not merely that this node isn't leader),
+ * this is not a renewal, and this SCM hosts the primary root CA.
+ *
+ * @param scm - the serving StorageContainerManager.
+ * @param nle - the NotLeaderException produced by the local Ratis server, or null.
+ * @param isRenew - whether this request is a certificate renewal.
+ * @return true iff the leaderless SCM-certificate signing path should be used.
+ */
+ @VisibleForTesting
+ public static boolean isLeaderlessPrimaryScmSigner(StorageContainerManager scm, NotLeaderException nle,
+ boolean isRenew) {
+ return !isRenew && nle != null && nle.getSuggestedLeader() == null
Review Comment:
Good question. Well, it might definitely happen that during the network split, a new SCM is bootstrapping (we skip the renewal process) and requesting a new certificate. That might create some mess. I think we might make the check stricter. We should also check that the request came from the ratis member, that it has never had a certificate, and that this node is the primary SCM. There is still a case with an unsecured 5-node cluster that is being secured, bootstrapped, and started. 4 nodes are started; a network split occurs for the primary SCM and the 5th node only, and the 5th node is bootstrapped. Not sure whether this case justifies significantly more complicated logic (disjoint the ranges of IDs for leaderless and ratis as an option)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16035. Secure SCM HA cluster cannot start: the primary SCM refuses to sign certificates for the other SCMs when there is no leader [ozone]
Posted by "chihsuan (via GitHub)" <gi...@apache.org>.
chihsuan commented on code in PR #10916:
URL: https://github.com/apache/ozone/pull/10916#discussion_r3705462773
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMSecurityProtocolServer.java:
##########
@@ -326,7 +334,29 @@ public String getSCMCertificate(ScmNodeDetailsProto scmNodeDetails,
LOGGER.info("Processing CSR for scm {}, nodeId: {}",
scmNodeDetails.getHostName(), scmNodeDetails.getScmNodeId());
- return getEncodedCertToString(certSignReq, NodeType.SCM);
+ boolean leaderless = !storageContainerManager.checkLeader()
+ && isLeaderlessPrimaryScmSigner(storageContainerManager,
+ storageContainerManager.getScmHAManager().getRatisServer().triggerNotLeaderException(),
+ isRenew);
+
+ return getEncodedCertToString(certSignReq, NodeType.SCM, leaderless, scmNodeDetails.getScmNodeId());
+ }
+
+ /**
+ * Single source of truth for whether this SCM should sign its own leaderless bootstrap SCM
+ * certificate: no Ratis leader is known cluster-wide (not merely that this node isn't leader),
+ * this is not a renewal, and this SCM hosts the primary root CA.
+ *
+ * @param scm - the serving StorageContainerManager.
+ * @param nle - the NotLeaderException produced by the local Ratis server, or null.
+ * @param isRenew - whether this request is a certificate renewal.
+ * @return true iff the leaderless SCM-certificate signing path should be used.
+ */
+ @VisibleForTesting
+ public static boolean isLeaderlessPrimaryScmSigner(StorageContainerManager scm, NotLeaderException nle,
+ boolean isRenew) {
+ return !isRenew && nle != null && nle.getSuggestedLeader() == null
Review Comment:
I was curious about this condition. Could `getSuggestedLeader() == null` also occur briefly during an election, or on an isolated SCM while the other peers have a leader?
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SequenceIdGenerator.java:
##########
@@ -180,6 +180,44 @@ public void reinitialize(Table<SequenceIdType, Long> sequenceIdTable)
}
}
+ /**
+ * Allocate the next CertificateId directly against the CertificateId row
+ * of {@link SCMMetadataStore#getSequenceIdTable()}, bypassing Ratis. This
+ * is used for leaderless bootstrap certificate signing, where no Ratis
+ * leader is available yet.
+ *
+ * The live StateManager cache and any un-exhausted batch are refreshed
+ * under the same lock, so that a later Ratis-based allocation of
+ * CertificateId (e.g. after a leader election) cannot CAS a stale cached
+ * lastId and reissue the value handed out here.
+ *
+ * @param scmMetadataStore : the SCMMetadataStore to allocate against.
+ * @return the newly allocated CertificateId.
+ */
+ public long getNextCertificateIdWithoutRatis(SCMMetadataStore scmMetadataStore)
+ throws IOException {
+ lock.lock();
+ try {
+ // Re-derive the CertificateId row from existing certificates if missing.
+ upgradeToCertificateSequenceId(scmMetadataStore, false);
+
+ Table<SequenceIdType, Long> sequenceIdTable = scmMetadataStore.getSequenceIdTable();
+ Long lastId = sequenceIdTable.get(SequenceIdType.CertificateId);
Review Comment:
Would we also need to account for an older write already queued in the transaction buffer, or is there a guarantee that the buffer is empty on this path?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16035. Secure SCM HA cluster cannot start: the primary SCM refuses to sign certificates for the other SCMs when there is no leader [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #10916: HDDS-16035. Secure SCM HA cluster cannot start: the primary SCM refuses to sign certificates for the other SCMs when there is no leader
URL: https://github.com/apache/ozone/pull/10916
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16035. Secure SCM HA cluster cannot start: the primary SCM refuses to sign certificates for the other SCMs when there is no leader [ozone]
Posted by "ss77892 (via GitHub)" <gi...@apache.org>.
ss77892 commented on code in PR #10916:
URL: https://github.com/apache/ozone/pull/10916#discussion_r3709291065
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SequenceIdGenerator.java:
##########
@@ -180,6 +180,44 @@ public void reinitialize(Table<SequenceIdType, Long> sequenceIdTable)
}
}
+ /**
+ * Allocate the next CertificateId directly against the CertificateId row
+ * of {@link SCMMetadataStore#getSequenceIdTable()}, bypassing Ratis. This
+ * is used for leaderless bootstrap certificate signing, where no Ratis
+ * leader is available yet.
+ *
+ * The live StateManager cache and any un-exhausted batch are refreshed
+ * under the same lock, so that a later Ratis-based allocation of
+ * CertificateId (e.g. after a leader election) cannot CAS a stale cached
+ * lastId and reissue the value handed out here.
+ *
+ * @param scmMetadataStore : the SCMMetadataStore to allocate against.
+ * @return the newly allocated CertificateId.
+ */
+ public long getNextCertificateIdWithoutRatis(SCMMetadataStore scmMetadataStore)
+ throws IOException {
+ lock.lock();
+ try {
+ // Re-derive the CertificateId row from existing certificates if missing.
+ upgradeToCertificateSequenceId(scmMetadataStore, false);
+
+ Table<SequenceIdType, Long> sequenceIdTable = scmMetadataStore.getSequenceIdTable();
+ Long lastId = sequenceIdTable.get(SequenceIdType.CertificateId);
Review Comment:
Well, the expectation is that it's the first time certificates are issued.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16035. Secure SCM HA cluster cannot start: the primary SCM refuses to sign certificates for the other SCMs when there is no leader [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10916:
URL: https://github.com/apache/ozone/pull/10916#issuecomment-5418705868
This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org