You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2021/08/30 08:15:16 UTC
[GitHub] [ozone] bharatviswa504 opened a new pull request #2596: HDDS-5678. Handle unsecure SCM HA converted to secure SCM HA.
bharatviswa504 opened a new pull request #2596:
URL: https://github.com/apache/ozone/pull/2596
## What changes were proposed in this pull request?
Handle unsecure SCM HA converted to secure SCM HA. This PR covers for bootstrapped SCMs, unlike HDDS-5273 which has taken care only for primordial/single node SCM.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-5678
## How was this patch tested?
Tested following scenario.
1. Setup SCM HA Cluster with out security
2. Enabled Kerberos
3. Started Ozone Services
**Logs from testing** (Skipped unncessary part of logs to show actual logs to show this PR has fixed the problem.
```
STARTUP_MSG: Starting StorageContainerManager
STARTUP_MSG: host = bv-oz-4.bv-oz.root.hwx.site/172.27.99.0
STARTUP_MSG: args = [--bootstrap]
STARTUP_MSG: version = 1.1.0.7.2.12.0-200
2021-08-30 07:50:55,866 INFO org.apache.hadoop.hdds.scm.server.StorageContainerManager: Skipping clusterId validation during bootstrap command. ClusterId id CID-6fa9d66e-d857-4bd9-86a9-f293654cec20, SCM id b9285a61-463e-4eea-8d60-62fa62fcfeb8
2021-08-30 07:50:55,869 INFO org.apache.hadoop.hdds.scm.ha.HASecurityUtils: Initializing secure StorageContainerManager.
2021-08-30 07:50:56,193 ERROR org.apache.hadoop.hdds.security.x509.certificate.client.SCMCertificateClient: Default certificate serial id is not set. Can't locate the default certificate for this client.
2021-08-30 07:50:56,193 INFO org.apache.hadoop.hdds.security.x509.certificate.client.SCMCertificateClient: Certificate client init case: 0
2021-08-30 07:50:56,194 INFO org.apache.hadoop.hdds.security.x509.certificate.client.SCMCertificateClient: Creating keypair for client as keypair and certificate not found.
2021-08-30 07:50:56,628 INFO org.apache.hadoop.hdds.scm.ha.HASecurityUtils: Init response: GETCERT
2021-08-30 07:50:57,207 INFO org.apache.hadoop.ozone.OzoneSecurityUtil: ip:fe80:0:0:0:42:acff:fe1b:6300%eth0 not returned.
2021-08-30 07:50:57,207 INFO org.apache.hadoop.ozone.OzoneSecurityUtil: Adding ip:172.27.99.0,host:bv-oz-4.bv-oz.root.hwx.site
2021-08-30 07:50:57,207 INFO org.apache.hadoop.ozone.OzoneSecurityUtil: ip:0:0:0:0:0:0:0:1%lo not returned.
2021-08-30 07:50:57,207 INFO org.apache.hadoop.ozone.OzoneSecurityUtil: ip:127.0.0.1 not returned.
2021-08-30 07:50:57,294 INFO org.apache.hadoop.ozone.OzoneSecurityUtil: ip:fe80:0:0:0:42:acff:fe1b:6300%eth0 not returned.
2021-08-30 07:50:57,294 INFO org.apache.hadoop.ozone.OzoneSecurityUtil: Adding ip:172.27.99.0,host:bv-oz-4.bv-oz.root.hwx.site
2021-08-30 07:50:57,295 INFO org.apache.hadoop.ozone.OzoneSecurityUtil: ip:0:0:0:0:0:0:0:1%lo not returned.
2021-08-30 07:50:57,295 INFO org.apache.hadoop.ozone.OzoneSecurityUtil: ip:127.0.0.1 not returned.
2021-08-30 07:50:57,296 INFO org.apache.hadoop.hdds.scm.ha.HASecurityUtils: Creating csr for SCM->hostName:bv-oz-4.bv-oz.root.hwx.site,scmId:b9285a61-463e-4eea-8d60-62fa62fcfeb8,clusterId:CID-6fa9d66e-d857-4bd9-86a9-f293654cec20,subject:scm-sub@bv-oz-4.bv-oz.root.hwx.site
2021-08-30 07:50:57,351 INFO org.apache.hadoop.hdds.scm.ha.HASecurityUtils: Successfully stored SCM signed certificate.
2021-08-30 07:50:57,356 INFO org.apache.hadoop.hdds.scm.server.StorageContainerManager: SCM unsecure cluster is converted to secure cluster. Persisted SCM Certificate SerialID 49003249912697594
2021-08-30 07:50:57,358 INFO org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter: SHUTDOWN_MSG:
```
**Before this fix for reference:**
```
2021-08-30 07:40:53,822 ERROR org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter: SCM start failed with exception
java.lang.IllegalStateException: INTERMEDIARY_CA Should not be in Initialize State during startup.
at org.apache.hadoop.hdds.security.x509.certificate.authority.DefaultCAServer.lambda$processVerificationStatus$4(DefaultCAServer.java:496)
at org.apache.hadoop.hdds.security.x509.certificate.authority.DefaultCAServer.init(DefaultCAServer.java:170)
at org.apache.hadoop.hdds.scm.server.StorageContainerManager.initializeCAnSecurityProtocol(StorageContainerManager.java:616)
at org.apache.hadoop.hdds.scm.server.StorageContainerManager.<init>(StorageContainerManager.java:327)
at org.apache.hadoop.hdds.scm.server.StorageContainerManager.createSCM(StorageContainerManager.java:440)
at org.apache.hadoop.hdds.scm.server.StorageContainerManager.createSCM(StorageContainerManager.java:452)
at org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter$SCMStarterHelper.start(StorageContainerManagerStarter.java:165)
at org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.startScm(StorageContainerManagerStarter.java:139)
at org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.call(StorageContainerManagerStarter.java:68)
at org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.call(StorageContainerManagerStarter.java:44)
at picocli.CommandLine.executeUserObject(CommandLine.java:1933)
at picocli.CommandLine.access$1100(CommandLine.java:145)
at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2332)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2326)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2291)
at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:2152)
at picocli.CommandLine.parseWithHandlers(CommandLine.java:2530)
at picocli.CommandLine.parseWithHandler(CommandLine.java:2465)
at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:96)
at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:87)
at org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.main(StorageContainerManagerStarter.java:57)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bharatviswa504 edited a comment on pull request #2596: HDDS-5678. Handle unsecure SCM HA converted to secure SCM HA.
Posted by GitBox <gi...@apache.org>.
bharatviswa504 edited a comment on pull request #2596:
URL: https://github.com/apache/ozone/pull/2596#issuecomment-908230410
> @bharatviswa504 , the fix looks good. Can we consider adding an acceptance test for the same ?
@bshashikant I think it is not that straightforward with our acceptance test suite. (Because it needs a cluster to be started with out security config, and later same should be started with newly updated config of security)
But Once we have the MiniOzoneSecureHA Cluster we can try to simulate these scenarios.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bharatviswa504 commented on pull request #2596: HDDS-5678. Handle unsecure SCM HA converted to secure SCM HA.
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on pull request #2596:
URL: https://github.com/apache/ozone/pull/2596#issuecomment-915131223
Thank You @mukul1987 @bshashikant and @JacksonYao287 for the review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bharatviswa504 edited a comment on pull request #2596: HDDS-5678. Handle unsecure SCM HA converted to secure SCM HA.
Posted by GitBox <gi...@apache.org>.
bharatviswa504 edited a comment on pull request #2596:
URL: https://github.com/apache/ozone/pull/2596#issuecomment-908230410
> @bharatviswa504 , the fix looks good. Can we consider adding an acceptance test for the same ?
@bshashikant I think it is not that straightforward with our acceptance test suite. (Because it needs a cluster to be started with out security config, and later same should be started with newly updated config of security and also with same mounted paths for metadata)
But Once we have the MiniOzoneSecureHA Cluster we can try to simulate these scenarios.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] mukul1987 commented on a change in pull request #2596: HDDS-5678. Handle unsecure SCM HA converted to secure SCM HA.
Posted by GitBox <gi...@apache.org>.
mukul1987 commented on a change in pull request #2596:
URL: https://github.com/apache/ozone/pull/2596#discussion_r698365202
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
##########
@@ -934,6 +946,18 @@ public static boolean scmBootstrap(OzoneConfiguration conf)
+ "is {}", persistedClusterId, fetchedId);
return false;
}
+
+ // Initialize security if security is enabled later.
+ if (OzoneSecurityUtil.isSecurityEnabled(conf)
+ && scmStorageConfig.getScmCertSerialId() == null) {
+ HASecurityUtils.initializeSecurity(scmStorageConfig, conf,
Review comment:
the two code sections are same, can we move them to a common function ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bharatviswa504 merged pull request #2596: HDDS-5678. Handle unsecure SCM HA converted to secure SCM HA.
Posted by GitBox <gi...@apache.org>.
bharatviswa504 merged pull request #2596:
URL: https://github.com/apache/ozone/pull/2596
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2596: HDDS-5678. Handle unsecure SCM HA converted to secure SCM HA.
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2596:
URL: https://github.com/apache/ozone/pull/2596#discussion_r698374179
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
##########
@@ -934,6 +946,18 @@ public static boolean scmBootstrap(OzoneConfiguration conf)
+ "is {}", persistedClusterId, fetchedId);
return false;
}
+
+ // Initialize security if security is enabled later.
+ if (OzoneSecurityUtil.isSecurityEnabled(conf)
+ && scmStorageConfig.getScmCertSerialId() == null) {
+ HASecurityUtils.initializeSecurity(scmStorageConfig, conf,
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bharatviswa504 commented on pull request #2596: HDDS-5678. Handle unsecure SCM HA converted to secure SCM HA.
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on pull request #2596:
URL: https://github.com/apache/ozone/pull/2596#issuecomment-908230410
> @bharatviswa504 , the fix looks good. Can we consider adding an acceptance test for the same ?
@bshashikant I think it is not that straightforward with our acceptance test suite. (Because it needs a cluster to be started with out security config, and later same should be started with newly updated config of security)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bshashikant commented on a change in pull request #2596: HDDS-5678. Handle unsecure SCM HA converted to secure SCM HA.
Posted by GitBox <gi...@apache.org>.
bshashikant commented on a change in pull request #2596:
URL: https://github.com/apache/ozone/pull/2596#discussion_r698369978
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
##########
@@ -934,6 +946,18 @@ public static boolean scmBootstrap(OzoneConfiguration conf)
+ "is {}", persistedClusterId, fetchedId);
return false;
}
+
+ // Initialize security if security is enabled later.
+ if (OzoneSecurityUtil.isSecurityEnabled(conf)
+ && scmStorageConfig.getScmCertSerialId() == null) {
+ HASecurityUtils.initializeSecurity(scmStorageConfig, conf,
Review comment:
Its better to move this a a static utility function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bshashikant commented on pull request #2596: HDDS-5678. Handle unsecure SCM HA converted to secure SCM HA.
Posted by GitBox <gi...@apache.org>.
bshashikant commented on pull request #2596:
URL: https://github.com/apache/ozone/pull/2596#issuecomment-908226395
@bharatviswa504 , the fix looks good. Can we consider adding an acceptance test for the same ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org