You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2021/03/08 07:52:09 UTC

[GitHub] [ozone] bharatviswa504 opened a new pull request #2000: HDDS-4915. Integrate CertClient.

bharatviswa504 opened a new pull request #2000:
URL: https://github.com/apache/ozone/pull/2000


   ## What changes were proposed in this pull request?
   
   This PR is to implement
   1. Use RootCertificate server to issue certs for SCM
   2. Use scmCertificatServer to issue certs for DN/OM. (This cert server got certs from RootCertificate Server)
   3. Start RootCertificate server only on primary SCM.
   
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-4915
   
   ## How was this patch tested?
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r603003080



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/HASecurityUtils.java
##########
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership.  The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.hdds.scm.ha;
+
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.protocol.proto.SCMSecurityProtocolProtos.SCMGetCertResponseProto;
+import org.apache.hadoop.hdds.protocolPB.SCMSecurityProtocolClientSideTranslatorPB;
+import org.apache.hadoop.hdds.scm.server.SCMCertStore;
+import org.apache.hadoop.hdds.scm.server.SCMStorageConfig;
+import org.apache.hadoop.hdds.security.x509.SecurityConfig;
+import org.apache.hadoop.hdds.security.x509.certificate.authority.CertificateServer;
+import org.apache.hadoop.hdds.security.x509.certificate.authority.DefaultCAServer;
+import org.apache.hadoop.hdds.security.x509.certificate.authority.PKIProfiles.DefaultCAProfile;
+import org.apache.hadoop.hdds.security.x509.certificate.client.CertificateClient;
+import org.apache.hadoop.hdds.security.x509.certificate.client.SCMCertificateClient;
+import org.apache.hadoop.hdds.security.x509.certificate.utils.CertificateCodec;
+import org.apache.hadoop.hdds.security.x509.certificates.utils.CertificateSignRequest;
+import org.apache.hadoop.hdds.utils.HddsServerUtil;
+import org.apache.ratis.conf.Parameters;
+import org.apache.ratis.grpc.GrpcConfigKeys;
+import org.apache.ratis.grpc.GrpcTlsConfig;
+import org.bouncycastle.cert.X509CertificateHolder;
+import org.bouncycastle.pkcs.PKCS10CertificationRequest;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.net.InetAddress;
+import java.net.InetSocketAddress;
+import java.nio.file.Paths;
+import java.security.KeyPair;
+import java.security.cert.CertificateException;
+import java.security.cert.X509Certificate;
+import java.util.concurrent.ExecutionException;
+
+import static org.apache.hadoop.hdds.protocol.proto.HddsProtos.NodeType.SCM;
+import static org.apache.hadoop.hdds.security.x509.certificate.authority.CertificateApprover.ApprovalType.KERBEROS_TRUSTED;
+import static org.apache.hadoop.hdds.security.x509.certificates.utils.CertificateSignRequest.getEncodedString;
+import static org.apache.hadoop.ozone.OzoneConsts.SCM_CA_CERT_STORAGE_DIR;
+import static org.apache.hadoop.ozone.OzoneConsts.SCM_CA_PATH;
+
+public final class HASecurityUtils {
+
+  private HASecurityUtils() {
+  }
+
+  public static final Logger LOG =
+      LoggerFactory.getLogger(HASecurityUtils.class);
+
+  /**
+   * Initialize Security which generates public, private key pair and get SCM
+   * signed certificate and persist to local disk.
+   * @param scmStorageConfig
+   * @param fetchedScmId
+   * @param conf
+   * @param scmAddress
+   * @throws IOException
+   */
+  public static void initializeSecurity(SCMStorageConfig scmStorageConfig,
+      String fetchedScmId, OzoneConfiguration conf,
+      InetSocketAddress scmAddress, boolean primaryscm)
+      throws IOException {
+    LOG.info("Initializing secure StorageContainerManager.");
+
+    CertificateClient certClient =
+        new SCMCertificateClient(new SecurityConfig(conf));
+    CertificateClient.InitResponse response = certClient.init();
+    LOG.info("Init response: {}", response);
+    switch (response) {
+    case SUCCESS:
+      LOG.info("Initialization successful.");
+      break;
+    case GETCERT:
+      if (!primaryscm) {
+        getSCMSignedCert(certClient, conf, fetchedScmId, scmStorageConfig,
+            scmAddress);
+      } else {
+        getPrimarySCMSignedCert(certClient, conf, fetchedScmId,
+            scmStorageConfig, scmAddress);
+      }
+      LOG.info("Successfully stored SCM signed certificate.");
+      break;
+    case FAILURE:
+      LOG.error("SCM security initialization failed.");
+      throw new RuntimeException("OM security initialization failed.");
+    case RECOVER:
+      LOG.error("SCM security initialization failed. SCM certificate is " +
+          "missing.");
+      throw new RuntimeException("SCM security initialization failed.");
+    default:
+      LOG.error("SCM security initialization failed. Init response: {}",
+          response);
+      throw new RuntimeException("SCM security initialization failed.");
+    }
+  }
+
+  /**
+   * Get SCM signed certificate and store it using certificate client.
+   */
+  private static void getSCMSignedCert(CertificateClient client,
+      OzoneConfiguration config, String fetchedSCMId,
+      SCMStorageConfig scmStorageConfig, InetSocketAddress scmAddress)
+      throws IOException {
+
+    PKCS10CertificationRequest csr = generateCSR(client, scmStorageConfig,
+        config, scmAddress, fetchedSCMId);
+
+    HddsProtos.ScmNodeDetailsProto scmNodeDetailsProto =
+        HddsProtos.ScmNodeDetailsProto.newBuilder()
+            .setClusterId(scmStorageConfig.getClusterID())
+            .setHostName(scmAddress.getHostName())
+            .setScmNodeId(fetchedSCMId).build();
+
+    SCMSecurityProtocolClientSideTranslatorPB secureScmClient =
+        HddsServerUtil.getScmSecurityClient(config);
+
+    SCMGetCertResponseProto response = secureScmClient.
+        getSCMCertChain(scmNodeDetailsProto, getEncodedString(csr));
+    String pemEncodedCert = response.getX509Certificate();
+
+    try {
+
+      // Store SCM CA certificate.
+      if (response.hasX509CACertificate()) {
+        String pemEncodedRootCert = response.getX509CACertificate();
+        client.storeCertificate(pemEncodedRootCert, true, true);
+        client.storeCertificate(pemEncodedCert, true);
+
+
+        // Write to the location, so that Default CA server can read the
+        // certificate generated by cert client.
+        CertificateCodec certCodec =
+            new CertificateCodec(new SecurityConfig(config),
+                client.getComponentName());
+
+        X509Certificate certificate =
+            CertificateCodec.getX509Certificate(pemEncodedCert);
+        certCodec.writeCertificate(certCodec.getCertificateHolder(certificate));
+
+        // Persist scm cert serial ID.
+        scmStorageConfig.setScmCertSerialId(certificate.getSerialNumber()
+            .toString());
+      } else {
+        throw new RuntimeException("Unable to retrieve SCM certificate chain");
+      }
+    } catch (IOException | CertificateException e) {
+      LOG.error("Error while storing SCM signed certificate.", e);
+      throw new RuntimeException(e);
+    }
+  }
+
+  private static void getPrimarySCMSignedCert(CertificateClient client,
+      OzoneConfiguration config, String fetchedSCMId,
+      SCMStorageConfig scmStorageConfig, InetSocketAddress scmAddress)
+      throws IOException {
+
+    PKCS10CertificationRequest csr = generateCSR(client, scmStorageConfig,
+        config, scmAddress, fetchedSCMId);
+
+    CertificateServer rootCAServer =
+        initializeRootCertificateServer(scmStorageConfig.getClusterID(),
+            scmStorageConfig.getScmId(), null);
+
+    rootCAServer.init(new SecurityConfig(config),
+        CertificateServer.CAType.SELF_SIGNED_CA);
+
+    try {
+      X509CertificateHolder certificateHolder = rootCAServer.
+          requestCertificate(csr, KERBEROS_TRUSTED, SCM).get();
+
+      X509CertificateHolder rootCAServerCACertificateHolder =
+          rootCAServer.getCACertificate();
+
+      String pemEncodedCert =
+          CertificateCodec.getPEMEncodedString(certificateHolder);
+
+      String pemEncodedRootCert =
+          CertificateCodec.getPEMEncodedString(rootCAServerCACertificateHolder);
+
+
+      client.storeCertificate(pemEncodedRootCert, true, true);
+      client.storeCertificate(pemEncodedCert, true);
+
+      // Write to the location, so that Default CA server can read the
+      // certificate generated by cert client.
+      CertificateCodec certCodec =
+          new CertificateCodec(new SecurityConfig(config),
+              client.getComponentName());
+
+      certCodec.writeCertificate(certificateHolder);
+
+        // Persist scm cert serial ID.
+      scmStorageConfig.setScmCertSerialId(certificateHolder.getSerialNumber()
+          .toString());
+    } catch (InterruptedException e) {
+      LOG.error("Error while storing SCM signed certificate.", e);
+      Thread.currentThread().interrupt();
+      throw new RuntimeException(e);
+    } catch (ExecutionException e) {
+      LOG.error("Error while storing SCM signed certificate.", e);
+      throw new RuntimeException(e);
+    } catch (IOException | CertificateException  e) {
+      LOG.error("Error while storing SCM signed certificate.", e);
+      throw new RuntimeException(e);
+    }
+
+  }
+
+  /**
+   * This function creates/initializes a certificate server as needed.
+   * This function is idempotent, so calling this again and again after the
+   * server is initialized is not a problem.
+   *
+   * @param clusterID - Cluster ID
+   * @param scmID     - SCM ID
+   */
+  public static CertificateServer initializeRootCertificateServer(
+      String clusterID, String scmID, SCMCertStore scmCertStore)
+      throws IOException {
+    String subject = "scm-rootca@" + InetAddress.getLocalHost().getHostName();
+
+    return new DefaultCAServer(subject, clusterID, scmID, scmCertStore,
+        new DefaultCAProfile(),
+        Paths.get(SCM_CA_CERT_STORAGE_DIR, SCM_CA_PATH).toString());
+  }
+
+  /**
+   * Create Server TLS parameters required for Ratis Server.
+   * @param conf
+   * @param caClient
+   * @return
+   */
+  public static Parameters createServerTlsParameters(SecurityConfig conf,
+      CertificateClient caClient) {
+    Parameters parameters = new Parameters();
+
+    if (conf.isSecurityEnabled() && conf.isGrpcTlsEnabled()) {
+      GrpcTlsConfig config = new GrpcTlsConfig(
+          caClient.getPrivateKey(), caClient.getCertificate(),
+          caClient.getCACertificate(), true);
+      GrpcConfigKeys.Server.setTlsConf(parameters, config);
+      GrpcConfigKeys.Admin.setTlsConf(parameters, config);
+      GrpcConfigKeys.Client.setTlsConf(parameters, config);
+      GrpcConfigKeys.TLS.setConf(parameters, config);
+    }
+
+    return parameters;
+  }
+
+  private static PKCS10CertificationRequest generateCSR(
+      CertificateClient client, SCMStorageConfig scmStorageConfig,
+      OzoneConfiguration config, InetSocketAddress scmAddress,
+      String fetchedSCMId) throws IOException {
+    CertificateSignRequest.Builder builder = client.getCSRBuilder();
+    KeyPair keyPair = new KeyPair(client.getPublicKey(),

Review comment:
       For generating CSR, we generate keyPair here.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r600395507



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
##########
@@ -793,7 +880,18 @@ public static boolean scmBootstrap(OzoneConfiguration conf)
         // SCM Node info containing hostname to scm Id mappings
         // will be persisted into the version file once this node gets added
         // to existing SCM ring post node regular start up.
+
+        if(OzoneSecurityUtil.isSecurityEnabled(conf)) {
+          HASecurityUtils.initializeSecurity(scmStorageConfig,
+              scmInfo.getScmId(), config, getScmAddress(scmhaNodeDetails, conf),
+              false);
+        }
+        scmStorageConfig.setPrimaryScmNodeId(scmInfo.getScmId());

Review comment:
       Updated to fix the issue gracefully.
   
   **The current solution  in PR is** 
   1. Removed isLeader check and added a check if it is primary SCM it can issue SCM signed certificate. So, in case leader changes still newly bootstrapped SCM can get signed cert.
   2. But if the primary is down, it will fail after failover retry count, instead of after 3 retries to SCM. (Will open a new Jira to address this)
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#issuecomment-810834926


   Thank You @xiaoyuyao for the review. I have addressed review comments.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] xiaoyuyao commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r595517926



##########
File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
##########
@@ -416,4 +418,7 @@ private OzoneConsts() {
 
   // CRL Sequence Id
   public static final String CRL_SEQUENCE_ID_KEY = "CRL_SEQUENCE_ID";
+
+  public static final String SCM_CA_PATH = "ca";
+  public static final String SCM_CA_CERT_STORAGE_DIR = "scm";

Review comment:
       Do we differentiate the primary scm root and the the scm ca metadata locations? Are they all stored under {metadir}/ca/scm/keys (or certs)?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r600354804



##########
File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ha/SCMNodeInfo.java
##########
@@ -246,4 +247,20 @@ public String getScmSecurityAddress() {
   public String getScmDatanodeAddress() {
     return scmDatanodeAddress;
   }
+
+  public static SCMNodeInfo getScmNodeInfo(ConfigurationSource conf,

Review comment:
       Removed it




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r603013863



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/protocol/SCMSecurityProtocolServerSideTranslatorPB.java
##########
@@ -271,6 +278,7 @@ public SCMGetCertResponseProto getRootCACertificate() throws IOException {
         SCMGetCertResponseProto
             .newBuilder()
             .setResponseCode(ResponseCode.success)
+            .setX509Certificate(rootCACertificate)

Review comment:
       As x509Certificate is mandatory in SCMGetCertResponseProto.
   required string x509Certificate = 2; 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r600590639



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
##########
@@ -793,7 +880,18 @@ public static boolean scmBootstrap(OzoneConfiguration conf)
         // SCM Node info containing hostname to scm Id mappings
         // will be persisted into the version file once this node gets added
         // to existing SCM ring post node regular start up.
+
+        if(OzoneSecurityUtil.isSecurityEnabled(conf)) {
+          HASecurityUtils.initializeSecurity(scmStorageConfig,
+              scmInfo.getScmId(), config, getScmAddress(scmhaNodeDetails, conf),
+              false);
+        }
+        scmStorageConfig.setPrimaryScmNodeId(scmInfo.getScmId());

Review comment:
       Opened https://issues.apache.org/jira/browse/HDDS-5027 for this




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] xiaoyuyao commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r600778695



##########
File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/OzoneContainer.java
##########
@@ -167,8 +171,23 @@ public OzoneContainer(
     blockDeletingService =
         new BlockDeletingService(this, svcInterval.toMillis(), serviceTimeout,
             TimeUnit.MILLISECONDS, config);
-    tlsClientConfig = RatisHelper.createTlsClientConfig(
-        secConf, certClient != null ? certClient.getCACertificate() : null);
+
+    List< X509Certificate > x509Certificates = null;
+    if (certClient != null) {
+      List<String> pemEncodedCerts = HAUtils.buildCAList(certClient, conf);
+      x509Certificates = new ArrayList<>(pemEncodedCerts.size());

Review comment:
       NIT: can we wrap line 178-186 into HAUtils.buildCAList?

##########
File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/XceiverServerRatis.java
##########
@@ -465,9 +466,14 @@ private static Parameters createTlsParameters(SecurityConfig conf,
     Parameters parameters = new Parameters();
 
     if (conf.isSecurityEnabled() && conf.isGrpcTlsEnabled()) {
+      ArrayList< X509Certificate > listCA = new ArrayList<>();

Review comment:
       NIT: List<X509Certificate>

##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
##########
@@ -793,7 +880,18 @@ public static boolean scmBootstrap(OzoneConfiguration conf)
         // SCM Node info containing hostname to scm Id mappings
         // will be persisted into the version file once this node gets added
         // to existing SCM ring post node regular start up.
+
+        if(OzoneSecurityUtil.isSecurityEnabled(conf)) {
+          HASecurityUtils.initializeSecurity(scmStorageConfig,
+              scmInfo.getScmId(), config, getScmAddress(scmhaNodeDetails, conf),
+              false);
+        }
+        scmStorageConfig.setPrimaryScmNodeId(scmInfo.getScmId());

Review comment:
       bq. will be no rootCA running on that SCM, so bootstrap won't be successful
   
   If I remember correctly, the primary SCM(rootCA) does not have to be the same as the Ratis leader. But when primary SCM issue SCM certificate, it will go though via Ratis. 

##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMStorageConfig.java
##########
@@ -58,6 +60,28 @@ public void setScmId(String scmId) throws IOException {
     }
   }
 
+  public void setScmCertSerialId(String certSerialId) throws IOException {

Review comment:
       Do we need the primary cert serialId saved separately?

##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMSecurityProtocolServer.java
##########
@@ -268,7 +286,7 @@ public String getCACertificate() throws IOException {
   public List<String> listCertificate(NodeType role,
       long startSerialId, int count, boolean isRevoked) throws IOException {
     List<X509Certificate> certificates =
-        certificateServer.listCertificate(role, startSerialId, count,
+        rootCertificateServer.listCertificate(role, startSerialId, count,

Review comment:
       the API is changed to list all SCM certificates. Can we add new API for this instead of changing the semantics of existing ones? 

##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMSecurityProtocolServer.java
##########
@@ -225,7 +231,7 @@ public String getCertificate(String certSerialId) throws IOException {
         certSerialId);
     try {
       X509Certificate certificate =
-          certificateServer.getCertificate(certSerialId);
+          rootCertificateServer.getCertificate(certSerialId);

Review comment:
       here we change to only get SCM CA certificate from this API?

##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
##########
@@ -558,44 +579,105 @@ private void initializeSystemManagers(OzoneConfiguration conf,
    */
   private void initializeCAnSecurityProtocol(OzoneConfiguration conf,
       SCMConfigurator configurator) throws IOException {
-    if(configurator.getCertificateServer() != null) {
-      this.certificateServer = configurator.getCertificateServer();
+
+    // TODO: Support Certificate Server loading via Class Name loader.
+    // So it is easy to use different Certificate Servers if needed.
+    if(this.scmMetadataStore == null) {
+      LOG.error("Cannot initialize Certificate Server without a valid meta " +
+          "data layer.");
+      throw new SCMException("Cannot initialize CA without a valid metadata " +
+          "store", ResultCodes.SCM_NOT_INITIALIZED);
+    }
+
+    certificateStore =
+        new SCMCertStore.Builder().setMetadaStore(scmMetadataStore)
+            .setRatisServer(scmHAManager.getRatisServer())
+            .setCRLSequenceId(getLastSequenceIdForCRL()).build();
+
+
+    // If primary SCM node Id is set it means this is a cluster which has
+    // performed init with SCM HA version code.
+    if (primaryScmNodeId != null) {

Review comment:
       Can we use explicit state to represent the init vs. normal run on primary?

##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/HASecurityUtils.java
##########
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership.  The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.hdds.scm.ha;
+
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.protocol.proto.SCMSecurityProtocolProtos.SCMGetCertResponseProto;
+import org.apache.hadoop.hdds.protocolPB.SCMSecurityProtocolClientSideTranslatorPB;
+import org.apache.hadoop.hdds.scm.server.SCMCertStore;
+import org.apache.hadoop.hdds.scm.server.SCMStorageConfig;
+import org.apache.hadoop.hdds.security.x509.SecurityConfig;
+import org.apache.hadoop.hdds.security.x509.certificate.authority.CertificateServer;
+import org.apache.hadoop.hdds.security.x509.certificate.authority.DefaultCAServer;
+import org.apache.hadoop.hdds.security.x509.certificate.authority.PKIProfiles.DefaultCAProfile;
+import org.apache.hadoop.hdds.security.x509.certificate.client.CertificateClient;
+import org.apache.hadoop.hdds.security.x509.certificate.client.SCMCertificateClient;
+import org.apache.hadoop.hdds.security.x509.certificate.utils.CertificateCodec;
+import org.apache.hadoop.hdds.security.x509.certificates.utils.CertificateSignRequest;
+import org.apache.hadoop.hdds.utils.HddsServerUtil;
+import org.apache.ratis.conf.Parameters;
+import org.apache.ratis.grpc.GrpcConfigKeys;
+import org.apache.ratis.grpc.GrpcTlsConfig;
+import org.bouncycastle.cert.X509CertificateHolder;
+import org.bouncycastle.pkcs.PKCS10CertificationRequest;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.net.InetAddress;
+import java.net.InetSocketAddress;
+import java.nio.file.Paths;
+import java.security.KeyPair;
+import java.security.cert.CertificateException;
+import java.security.cert.X509Certificate;
+import java.util.concurrent.ExecutionException;
+
+import static org.apache.hadoop.hdds.protocol.proto.HddsProtos.NodeType.SCM;
+import static org.apache.hadoop.hdds.security.x509.certificate.authority.CertificateApprover.ApprovalType.KERBEROS_TRUSTED;
+import static org.apache.hadoop.hdds.security.x509.certificates.utils.CertificateSignRequest.getEncodedString;
+import static org.apache.hadoop.ozone.OzoneConsts.SCM_CA_CERT_STORAGE_DIR;
+import static org.apache.hadoop.ozone.OzoneConsts.SCM_CA_PATH;
+
+public final class HASecurityUtils {
+
+  private HASecurityUtils() {
+  }
+
+  public static final Logger LOG =
+      LoggerFactory.getLogger(HASecurityUtils.class);
+
+  /**
+   * Initialize Security which generates public, private key pair and get SCM
+   * signed certificate and persist to local disk.
+   * @param scmStorageConfig
+   * @param fetchedScmId
+   * @param conf
+   * @param scmAddress
+   * @throws IOException
+   */
+  public static void initializeSecurity(SCMStorageConfig scmStorageConfig,
+      String fetchedScmId, OzoneConfiguration conf,
+      InetSocketAddress scmAddress, boolean primaryscm)
+      throws IOException {
+    LOG.info("Initializing secure StorageContainerManager.");
+
+    CertificateClient certClient =
+        new SCMCertificateClient(new SecurityConfig(conf));
+    CertificateClient.InitResponse response = certClient.init();
+    LOG.info("Init response: {}", response);
+    switch (response) {
+    case SUCCESS:
+      LOG.info("Initialization successful.");
+      break;
+    case GETCERT:
+      if (!primaryscm) {
+        getSCMSignedCert(certClient, conf, fetchedScmId, scmStorageConfig,
+            scmAddress);
+      } else {
+        getPrimarySCMSignedCert(certClient, conf, fetchedScmId,
+            scmStorageConfig, scmAddress);
+      }
+      LOG.info("Successfully stored SCM signed certificate.");
+      break;
+    case FAILURE:
+      LOG.error("SCM security initialization failed.");
+      throw new RuntimeException("OM security initialization failed.");
+    case RECOVER:
+      LOG.error("SCM security initialization failed. SCM certificate is " +
+          "missing.");
+      throw new RuntimeException("SCM security initialization failed.");
+    default:
+      LOG.error("SCM security initialization failed. Init response: {}",
+          response);
+      throw new RuntimeException("SCM security initialization failed.");
+    }
+  }
+
+  /**
+   * Get SCM signed certificate and store it using certificate client.
+   */
+  private static void getSCMSignedCert(CertificateClient client,
+      OzoneConfiguration config, String fetchedSCMId,
+      SCMStorageConfig scmStorageConfig, InetSocketAddress scmAddress)
+      throws IOException {
+
+    PKCS10CertificationRequest csr = generateCSR(client, scmStorageConfig,
+        config, scmAddress, fetchedSCMId);
+
+    HddsProtos.ScmNodeDetailsProto scmNodeDetailsProto =
+        HddsProtos.ScmNodeDetailsProto.newBuilder()
+            .setClusterId(scmStorageConfig.getClusterID())
+            .setHostName(scmAddress.getHostName())
+            .setScmNodeId(fetchedSCMId).build();
+
+    SCMSecurityProtocolClientSideTranslatorPB secureScmClient =
+        HddsServerUtil.getScmSecurityClient(config);
+
+    SCMGetCertResponseProto response = secureScmClient.
+        getSCMCertChain(scmNodeDetailsProto, getEncodedString(csr));
+    String pemEncodedCert = response.getX509Certificate();
+
+    try {
+
+      // Store SCM CA certificate.
+      if (response.hasX509CACertificate()) {
+        String pemEncodedRootCert = response.getX509CACertificate();
+        client.storeCertificate(pemEncodedRootCert, true, true);
+        client.storeCertificate(pemEncodedCert, true);
+
+
+        // Write to the location, so that Default CA server can read the
+        // certificate generated by cert client.
+        CertificateCodec certCodec =
+            new CertificateCodec(new SecurityConfig(config),
+                client.getComponentName());
+
+        X509Certificate certificate =
+            CertificateCodec.getX509Certificate(pemEncodedCert);
+        certCodec.writeCertificate(certCodec.getCertificateHolder(certificate));
+
+        // Persist scm cert serial ID.
+        scmStorageConfig.setScmCertSerialId(certificate.getSerialNumber()
+            .toString());
+      } else {
+        throw new RuntimeException("Unable to retrieve SCM certificate chain");
+      }
+    } catch (IOException | CertificateException e) {
+      LOG.error("Error while storing SCM signed certificate.", e);
+      throw new RuntimeException(e);
+    }
+  }
+
+  private static void getPrimarySCMSignedCert(CertificateClient client,
+      OzoneConfiguration config, String fetchedSCMId,
+      SCMStorageConfig scmStorageConfig, InetSocketAddress scmAddress)
+      throws IOException {
+
+    PKCS10CertificationRequest csr = generateCSR(client, scmStorageConfig,
+        config, scmAddress, fetchedSCMId);
+
+    CertificateServer rootCAServer =
+        initializeRootCertificateServer(scmStorageConfig.getClusterID(),
+            scmStorageConfig.getScmId(), null);
+
+    rootCAServer.init(new SecurityConfig(config),
+        CertificateServer.CAType.SELF_SIGNED_CA);
+
+    try {
+      X509CertificateHolder certificateHolder = rootCAServer.
+          requestCertificate(csr, KERBEROS_TRUSTED, SCM).get();
+
+      X509CertificateHolder rootCAServerCACertificateHolder =
+          rootCAServer.getCACertificate();
+
+      String pemEncodedCert =
+          CertificateCodec.getPEMEncodedString(certificateHolder);
+
+      String pemEncodedRootCert =
+          CertificateCodec.getPEMEncodedString(rootCAServerCACertificateHolder);
+
+
+      client.storeCertificate(pemEncodedRootCert, true, true);
+      client.storeCertificate(pemEncodedCert, true);
+
+      // Write to the location, so that Default CA server can read the
+      // certificate generated by cert client.
+      CertificateCodec certCodec =
+          new CertificateCodec(new SecurityConfig(config),
+              client.getComponentName());
+
+      certCodec.writeCertificate(certificateHolder);
+
+        // Persist scm cert serial ID.
+      scmStorageConfig.setScmCertSerialId(certificateHolder.getSerialNumber()
+          .toString());
+    } catch (InterruptedException e) {
+      LOG.error("Error while storing SCM signed certificate.", e);
+      Thread.currentThread().interrupt();
+      throw new RuntimeException(e);
+    } catch (ExecutionException e) {
+      LOG.error("Error while storing SCM signed certificate.", e);
+      throw new RuntimeException(e);
+    } catch (IOException | CertificateException  e) {
+      LOG.error("Error while storing SCM signed certificate.", e);
+      throw new RuntimeException(e);
+    }
+
+  }
+
+  /**
+   * This function creates/initializes a certificate server as needed.
+   * This function is idempotent, so calling this again and again after the
+   * server is initialized is not a problem.
+   *
+   * @param clusterID - Cluster ID
+   * @param scmID     - SCM ID
+   */
+  public static CertificateServer initializeRootCertificateServer(
+      String clusterID, String scmID, SCMCertStore scmCertStore)
+      throws IOException {
+    String subject = "scm-rootca@" + InetAddress.getLocalHost().getHostName();
+
+    return new DefaultCAServer(subject, clusterID, scmID, scmCertStore,
+        new DefaultCAProfile(),
+        Paths.get(SCM_CA_CERT_STORAGE_DIR, SCM_CA_PATH).toString());
+  }
+
+  /**
+   * Create Server TLS parameters required for Ratis Server.
+   * @param conf
+   * @param caClient
+   * @return
+   */
+  public static Parameters createServerTlsParameters(SecurityConfig conf,
+      CertificateClient caClient) {
+    Parameters parameters = new Parameters();
+
+    if (conf.isSecurityEnabled() && conf.isGrpcTlsEnabled()) {
+      GrpcTlsConfig config = new GrpcTlsConfig(
+          caClient.getPrivateKey(), caClient.getCertificate(),
+          caClient.getCACertificate(), true);
+      GrpcConfigKeys.Server.setTlsConf(parameters, config);
+      GrpcConfigKeys.Admin.setTlsConf(parameters, config);
+      GrpcConfigKeys.Client.setTlsConf(parameters, config);
+      GrpcConfigKeys.TLS.setConf(parameters, config);
+    }
+
+    return parameters;
+  }
+
+  private static PKCS10CertificationRequest generateCSR(
+      CertificateClient client, SCMStorageConfig scmStorageConfig,
+      OzoneConfiguration config, InetSocketAddress scmAddress,
+      String fetchedSCMId) throws IOException {
+    CertificateSignRequest.Builder builder = client.getCSRBuilder();
+    KeyPair keyPair = new KeyPair(client.getPublicKey(),

Review comment:
       why do we create a new keypair even non-primary already has a keypair and the action is to GetCert from Primary SCM?

##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/protocol/SCMSecurityProtocolServerSideTranslatorPB.java
##########
@@ -271,6 +278,7 @@ public SCMGetCertResponseProto getRootCACertificate() throws IOException {
         SCMGetCertResponseProto
             .newBuilder()
             .setResponseCode(ResponseCode.success)
+            .setX509Certificate(rootCACertificate)

Review comment:
       why do we need to set rootCACertificate twice? setX509RootCACertificate should be sufficient as this is a new API. 

##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/HASecurityUtils.java
##########
@@ -281,4 +284,26 @@ private static void persistSubCACertificate(OzoneConfiguration config,
     certCodec.writeCertificate(certificateHolder);
   }
 
+  /**
+   * Create Server TLS parameters required for Ratis Server.
+   * @param conf
+   * @param caClient
+   * @return
+   */
+  public static Parameters createServerTlsParameters(SecurityConfig conf,
+      CertificateClient caClient) {
+    Parameters parameters = new Parameters();
+
+    if (conf.isSecurityEnabled() && conf.isGrpcTlsEnabled()) {
+      GrpcTlsConfig config = new GrpcTlsConfig(
+          caClient.getPrivateKey(), caClient.getCertificate(),
+          caClient.getCACertificate(), true);

Review comment:
       I think we need to include the RootCA certificate as well. 

##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMSecurityProtocolServer.java
##########
@@ -155,18 +155,27 @@ public String getOMCertificate(OzoneManagerDetailsProto omDetails,
   public String getSCMCertificate(ScmNodeDetailsProto scmNodeDetails,
       String certSignReq) throws IOException {
     Objects.requireNonNull(scmNodeDetails);
-    LOGGER.info("Processing CSR for scm {}, nodeId: {}",
-        scmNodeDetails.getHostName(), scmNodeDetails.getScmNodeId());
+    String primaryScmId =
+        storageContainerManager.getScmStorageConfig().getPrimaryScmNodeId();
 
-    // Check clusterID
-    if (storageContainerManager.getClusterId().equals(
-        scmNodeDetails.getClusterId())) {
-      throw new IOException("SCM ClusterId mismatch. Peer SCM ClusterId " +
-          scmNodeDetails.getClusterId() + ", primary SCM ClusterId "
-          + storageContainerManager.getClusterId());
-    }
+    if (primaryScmId != null &&
+        primaryScmId.equals(storageContainerManager.getScmId())) {
+      LOGGER.info("Processing CSR for scm {}, nodeId: {}",
+          scmNodeDetails.getHostName(), scmNodeDetails.getScmNodeId());
+
+      // Check clusterID
+      if (!storageContainerManager.getClusterId().equals(
+          scmNodeDetails.getClusterId())) {
+        throw new IOException("SCM ClusterId mismatch. Peer SCM ClusterId " +
+            scmNodeDetails.getClusterId() + ", primary SCM ClusterId "
+            + storageContainerManager.getClusterId());
+      }
 
-    return getEncodedCertToString(certSignReq, NodeType.SCM);
+      return getEncodedCertToString(certSignReq, NodeType.SCM);
+    } else {
+      throw new SCMSecurityException("Get SCM Certificate can be run only " +
+          "primary SCM", PRIMARY_SCM_IS_NOT_LEADER);

Review comment:
       This error code is confusing as we don't require the PRIMARY to be the leader. Just the persist of the issued sub-scm certificate needs to go through Ratis leader via the Ratis client API. 

##########
File path: hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/authority/DefaultApprover.java
##########
@@ -76,12 +77,12 @@ public DefaultApprover(PKIProfile pkiProfile, SecurityConfig config) {
    * @param scmId - SCM id.
    * @param clusterId - Cluster id.
    * @return Signed Certificate.
-   * @throws IOException - On Error
+   * @throws IOException               - On Error

Review comment:
       NIT: unnecessary change. 

##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
##########
@@ -860,6 +966,40 @@ public static boolean scmInit(OzoneConfiguration conf,
     }
   }
 
+  private static InetSocketAddress getScmAddress(SCMHANodeDetails haDetails,

Review comment:
       Can we move this to the HAUtils class?

##########
File path: hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/HAUtils.java
##########
@@ -326,4 +350,123 @@ public static void checkSecurityAndSCMHAEnabled(OzoneConfiguration conf) {
       }
     }
   }
+
+  /**
+   * Build CA list which need to be passed to client.
+   *
+   * If certificate client is null, obtain the list of CA using SCM security
+   * client, else it uses certificate client.
+   * @param certClient
+   * @param configuration
+   * @return list of CA
+   * @throws IOException
+   */
+  public static List<String> buildCAList(CertificateClient certClient,
+      ConfigurationSource configuration) throws IOException {
+    //TODO: make it configurable.
+    long waitTime = 5 * 60 * 1000L;
+    long retryTime = 10 * 1000L;
+    long currentTime = Time.monotonicNow();
+    List<String> caCertPemList = null;
+    if (certClient != null) {
+      caCertPemList = new ArrayList<>();
+      if (!SCMHAUtils.isSCMHAEnabled(configuration)) {
+        if (certClient.getRootCACertificate() != null) {
+          caCertPemList.add(CertificateCodec.getPEMEncodedString(

Review comment:
       Can we bypass the encode as string here and as the caller will need the X509Certificate format? The encode/decode seems redundant. 

##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMSecurityProtocolServer.java
##########
@@ -70,7 +70,7 @@
 
   private static final Logger LOGGER = LoggerFactory
       .getLogger(SCMSecurityProtocolServer.class);
-  private final CertificateServer certificateServer;
+  private final CertificateServer rootCertificateServer;

Review comment:
       NIT: SecurityProtocolServer also serves CSR for OM and DN, rename the CertificateSever inside as rootCertificateServer is confusing. 

##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMSecurityProtocolServer.java
##########
@@ -179,9 +188,15 @@ public String getSCMCertificate(ScmNodeDetailsProto scmNodeDetails,
    */
   private String getEncodedCertToString(String certSignReq, NodeType nodeType)
       throws IOException {
-    Future<X509CertificateHolder> future =
-        certificateServer.requestCertificate(certSignReq,
-            KERBEROS_TRUSTED, nodeType);
+
+    Future<X509CertificateHolder> future;
+    if (nodeType == NodeType.SCM) {
+      future = rootCertificateServer.requestCertificate(certSignReq,
+              KERBEROS_TRUSTED, nodeType);
+    } else {
+      future = storageContainerManager.getScmCertificateServer()

Review comment:
       Should we move the certificate server from SCM back to SecurityProtocolServer and keep two certificate server inside the security server for the primary SCM?

##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
##########
@@ -319,7 +331,8 @@ private StorageContainerManager(OzoneConfiguration conf,
       // if no Security, we do not create a Certificate Server at all.
       // This allows user to boot SCM without security temporarily
       // and then come back and enable it without any impact.
-      certificateServer = null;
+      rootCertificateServer = null;

Review comment:
       can we wrap the cert servers into the security protocol server?

##########
File path: hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/scm/proxy/SCMBlockLocationFailoverProxyProvider.java
##########
@@ -70,11 +70,20 @@
   private final int maxRetryCount;
   private final long retryInterval;
 
+  private final UserGroupInformation ugi;
+
 
   public SCMBlockLocationFailoverProxyProvider(ConfigurationSource conf) {
     this.conf = conf;
     this.scmVersion = RPC.getProtocolVersion(ScmBlockLocationProtocolPB.class);
 
+    try {
+      this.ugi = UserGroupInformation.getCurrentUser();

Review comment:
       Can you enable the Debug log of UGI class to see which UGI is used if we don't cache the UGI at the time of provider creation?

##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerRatisServer.java
##########
@@ -651,9 +652,14 @@ private static Parameters createServerTlsParameters(SecurityConfig conf,
     Parameters parameters = new Parameters();
 
     if (conf.isSecurityEnabled() && conf.isGrpcTlsEnabled()) {
+      ArrayList< X509Certificate > listCA = new ArrayList<>();
+      listCA.add(caClient.getCACertificate());

Review comment:
       This needs to include the ROOT ca certificate  with getCaCertPemList. 

##########
File path: hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/ScmOption.java
##########
@@ -61,6 +61,7 @@ public ScmClient createScmClient() {
 
       return new ContainerOperationClient(conf);
     } catch (IOException ex) {
+      ex.printStackTrace();

Review comment:
       Can we put this into LOG.error instead of print on the console?

##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
##########
@@ -558,44 +579,105 @@ private void initializeSystemManagers(OzoneConfiguration conf,
    */
   private void initializeCAnSecurityProtocol(OzoneConfiguration conf,

Review comment:
       I feel the logic inside this initializeCAnSecurityProtocol can be refactored. Let's discuss offline... 

##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
##########
@@ -400,6 +413,14 @@ private StorageContainerManager(OzoneConfiguration conf,
     registerMetricsSource(this);
   }
 
+  private void initializeCertificateClient() {
+    if (primaryScmNodeId != null) {

Review comment:
       should we check the local scm node id!=primaryScmNodeId here insead?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r593009482



##########
File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/OzoneContainer.java
##########
@@ -167,8 +168,16 @@ public OzoneContainer(
     blockDeletingService =
         new BlockDeletingService(this, svcInterval.toMillis(), serviceTimeout,
             TimeUnit.MILLISECONDS, config);
+
+    List<X509Certificate> x509Certificates = new ArrayList<>();
+    if (certClient != null) {
+      x509Certificates.add(certClient.getCACertificate());
+      if (certClient.getRootCACertificate() != null) {

Review comment:
       This is not making an RPC Call, it is getting from in-mem map.
   
   (It gets Root Cert when it makes a call getDatanodeCertificate)




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r600392963



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
##########
@@ -793,7 +880,18 @@ public static boolean scmBootstrap(OzoneConfiguration conf)
         // SCM Node info containing hostname to scm Id mappings
         // will be persisted into the version file once this node gets added
         // to existing SCM ring post node regular start up.
+
+        if(OzoneSecurityUtil.isSecurityEnabled(conf)) {
+          HASecurityUtils.initializeSecurity(scmStorageConfig,
+              scmInfo.getScmId(), config, getScmAddress(scmhaNodeDetails, conf),
+              false);
+        }
+        scmStorageConfig.setPrimaryScmNodeId(scmInfo.getScmId());

Review comment:
       To add more info, the main reason for persistence to all SCM is, so that each SCM knows who is primary and it will help during debugging customer issues. And also in code it is used to start RootCA only on primary SCM when it is matching with SCMID.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r603025764



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMSecurityProtocolServer.java
##########
@@ -268,7 +286,7 @@ public String getCACertificate() throws IOException {
   public List<String> listCertificate(NodeType role,
       long startSerialId, int count, boolean isRevoked) throws IOException {
     List<X509Certificate> certificates =
-        certificateServer.listCertificate(role, startSerialId, count,
+        rootCertificateServer.listCertificate(role, startSerialId, count,

Review comment:
       This API semantics is not changed it can return list of certificates for all roles.
   As scmcertserver runs on all SCM's used the scmcertserver
   
   Underlying both CertificateServers use the same certstore. So, the semantics of this API is not changed, it can get any certificate.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 edited a comment on pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 edited a comment on pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#issuecomment-800999900


   @xiaoyuyao Addressed/replied review comments these are addressed in HDDS-4897 under PR #2041 
   Can we continue the review #2041 
   
   I will update the PR to use all the Jira's which went in and integrate it.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r603019860



##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerRatisServer.java
##########
@@ -651,9 +652,14 @@ private static Parameters createServerTlsParameters(SecurityConfig conf,
     Parameters parameters = new Parameters();
 
     if (conf.isSecurityEnabled() && conf.isGrpcTlsEnabled()) {
+      ArrayList< X509Certificate > listCA = new ArrayList<>();
+      listCA.add(caClient.getCACertificate());

Review comment:
       Question: why do we need to complete the sub SCM ca list here, as this is here it is for MTLS?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r600366956



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
##########
@@ -793,7 +880,18 @@ public static boolean scmBootstrap(OzoneConfiguration conf)
         // SCM Node info containing hostname to scm Id mappings
         // will be persisted into the version file once this node gets added
         // to existing SCM ring post node regular start up.
+
+        if(OzoneSecurityUtil.isSecurityEnabled(conf)) {
+          HASecurityUtils.initializeSecurity(scmStorageConfig,
+              scmInfo.getScmId(), config, getScmAddress(scmhaNodeDetails, conf),
+              false);
+        }
+        scmStorageConfig.setPrimaryScmNodeId(scmInfo.getScmId());

Review comment:
       Primary SCM only starts Root CA that is the only one can issue scm sub ca certificates.
   
   But if you add later on, and the leader changes there will be no rootCA running on that SCM, so bootstrap won't be successful. Let me try to handle that.

##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
##########
@@ -793,7 +880,18 @@ public static boolean scmBootstrap(OzoneConfiguration conf)
         // SCM Node info containing hostname to scm Id mappings
         // will be persisted into the version file once this node gets added
         // to existing SCM ring post node regular start up.
+
+        if(OzoneSecurityUtil.isSecurityEnabled(conf)) {
+          HASecurityUtils.initializeSecurity(scmStorageConfig,
+              scmInfo.getScmId(), config, getScmAddress(scmhaNodeDetails, conf),
+              false);
+        }
+        scmStorageConfig.setPrimaryScmNodeId(scmInfo.getScmId());

Review comment:
       Primary SCM only starts Root CA that is the only one can issue scm sub ca certificates.
   
   But if you add later on, and the leader changes there will be no rootCA running on that SCM, so bootstrap won't be successful. Let me try to handle that gracefully.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] xiaoyuyao commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r604438524



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMSecurityProtocolServer.java
##########
@@ -288,22 +306,18 @@ public String getCACertificate() throws IOException {
   public List<String> listCACertificate() throws IOException {
     List<String> caCerts =
         listCertificate(NodeType.SCM, 0, 10, false);
-    caCerts.add(getRootCACertificate());
     return caCerts;
   }
 
   @Override
   public String getRootCACertificate() throws IOException {
     LOGGER.debug("Getting Root CA certificate.");
-    //TODO: This code will be modified after HDDS-4897 is merged and
-    // integrated. For now getting RootCA cert from certificateServer.
-    try {
+    if (storageContainerManager.getScmStorageConfig()

Review comment:
       Can we pass in the Root CA certificate to the security protocol server without calling back into the cert client from scm?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r603016162



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMSecurityProtocolServer.java
##########
@@ -179,9 +188,15 @@ public String getSCMCertificate(ScmNodeDetailsProto scmNodeDetails,
    */
   private String getEncodedCertToString(String certSignReq, NodeType nodeType)
       throws IOException {
-    Future<X509CertificateHolder> future =
-        certificateServer.requestCertificate(certSignReq,
-            KERBEROS_TRUSTED, nodeType);
+
+    Future<X509CertificateHolder> future;
+    if (nodeType == NodeType.SCM) {
+      future = rootCertificateServer.requestCertificate(certSignReq,
+              KERBEROS_TRUSTED, nodeType);
+    } else {
+      future = storageContainerManager.getScmCertificateServer()

Review comment:
       Done




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] xiaoyuyao commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r604435275



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMSecurityProtocolServer.java
##########
@@ -70,7 +70,7 @@
 
   private static final Logger LOGGER = LoggerFactory
       .getLogger(SCMSecurityProtocolServer.class);
-  private final CertificateServer certificateServer;
+  private final CertificateServer rootCertificateServer;

Review comment:
       The SecurityProtocolServer is much cleaner now.  




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r600382827



##########
File path: hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/client/DefaultCertificateClient.java
##########
@@ -818,29 +830,64 @@ public Logger getLogger() {
     return logger;
   }
 
-  /**
-   * Create a scm security client, used to get SCM signed certificate.
-   *
-   * @return {@link SCMSecurityProtocol}
-   */
-  private static SCMSecurityProtocol getScmSecurityClient(
-      OzoneConfiguration conf) throws IOException {
-    RPC.setProtocolEngine(conf, SCMSecurityProtocolPB.class,
-        ProtobufRpcEngine.class);
-    long scmVersion =
-        RPC.getProtocolVersion(ScmBlockLocationProtocolPB.class);
-    InetSocketAddress scmSecurityProtoAdd =
-        HddsServerUtil.getScmAddressForSecurityProtocol(conf);
-    SCMSecurityProtocolClientSideTranslatorPB scmSecurityClient =
-        new SCMSecurityProtocolClientSideTranslatorPB(
-            RPC.getProxy(SCMSecurityProtocolPB.class, scmVersion,
-                scmSecurityProtoAdd, UserGroupInformation.getCurrentUser(),
-                conf, NetUtils.getDefaultSocketFactory(conf),
-                Client.getRpcTimeout(conf)));
-    return scmSecurityClient;
+  public String getComponentName() {
+    return null;
   }
 
-  public String getComponentName() {
+  @Override
+  public X509Certificate getRootCACertificate() {
+    if (rootCaCertId != null) {
+      return certificateMap.get(rootCaCertId);
+    }
     return null;
   }
+
+  @Override
+  public void storeRootCACertificate(String pemEncodedCert, boolean force)
+      throws CertificateException {
+    CertificateCodec certificateCodec = new CertificateCodec(securityConfig,
+        component);
+    try {
+      Path basePath = securityConfig.getCertificateLocation(component);
+
+      X509Certificate cert =
+          CertificateCodec.getX509Certificate(pemEncodedCert);
+      String certName = String.format(CERT_FILE_NAME_FORMAT,
+          cert.getSerialNumber().toString());
+
+      certName = ROOT_CA_CERT_PREFIX + certName;
+      rootCaCertId = cert.getSerialNumber().toString();
+
+      certificateCodec.writeCertificate(basePath, certName,
+          pemEncodedCert, force);
+      certificateMap.putIfAbsent(cert.getSerialNumber().toString(), cert);
+    } catch (IOException | java.security.cert.CertificateException e) {
+      throw new CertificateException("Error while storing Root CA " +
+          "certificate.", e, CERTIFICATE_ERROR);
+    }
+  }
+
+  @Override
+  public List<String> listCA() throws IOException {
+    if (pemEncodedCACerts == null) {
+      updateCAList();

Review comment:
       Done, used locks




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] xiaoyuyao commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r604430441



##########
File path: hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/HAUtils.java
##########
@@ -326,4 +351,139 @@ public static void checkSecurityAndSCMHAEnabled(OzoneConfiguration conf) {
       }
     }
   }
+
+  /**
+   * Build CA list which need to be passed to client.
+   *
+   * If certificate client is null, obtain the list of CA using SCM security
+   * client, else it uses certificate client.
+   * @param certClient
+   * @param configuration
+   * @return list of CA
+   * @throws IOException
+   */
+  public static List<String> buildCAList(CertificateClient certClient,
+      ConfigurationSource configuration) throws IOException {
+    //TODO: make it configurable.
+    long waitTime = 5 * 60 * 1000L;
+    long retryTime = 10 * 1000L;
+    long currentTime = Time.monotonicNow();
+    List<String> caCertPemList = null;
+    if (certClient != null) {
+      caCertPemList = new ArrayList<>();
+      if (!SCMHAUtils.isSCMHAEnabled(configuration)) {
+        if (certClient.getRootCACertificate() != null) {
+          caCertPemList.add(CertificateCodec.getPEMEncodedString(
+              certClient.getRootCACertificate()));
+        }
+        caCertPemList.add(CertificateCodec.getPEMEncodedString(
+            certClient.getCACertificate()));
+      } else {
+        // TODO: If SCMs are bootstrapped later, then listCA need to be
+        //  refetched if listCA size is less than scm ha config node list size.
+        // For now when Client of SCM's are started we compare their node list
+        // size and ca list size if it is as expected, we return the ca list.
+        boolean caListUpToDate;
+        Collection<String> scmNodes = SCMHAUtils.getSCMNodeIds(configuration);
+        // TODO: make them configurable.
+        if (scmNodes.size() > 1) {
+          do {
+            caCertPemList = certClient.updateCAList();
+            caListUpToDate =
+                caCertPemList.size() == scmNodes.size() + 1 ? true : false;
+            if (!caListUpToDate) {
+              try {
+                Thread.sleep(retryTime);
+              } catch (InterruptedException ex) {
+                Thread.currentThread().interrupt();
+              }
+            }
+          } while (!caListUpToDate &&
+              Time.monotonicNow() - currentTime < waitTime);
+          checkCertCount(caCertPemList.size(), scmNodes.size() + 1);
+        } else {
+          caCertPemList = certClient.updateCAList();
+        }
+      }
+    } else {
+      if (!SCMHAUtils.isSCMHAEnabled(configuration)) {
+        caCertPemList = new ArrayList<>();
+        SCMSecurityProtocolClientSideTranslatorPB scmSecurityProtocolClient =

Review comment:
       Can we move the scmsecurityprotocol client creation up and dedup the same logic between them?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r600395507



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
##########
@@ -793,7 +880,18 @@ public static boolean scmBootstrap(OzoneConfiguration conf)
         // SCM Node info containing hostname to scm Id mappings
         // will be persisted into the version file once this node gets added
         // to existing SCM ring post node regular start up.
+
+        if(OzoneSecurityUtil.isSecurityEnabled(conf)) {
+          HASecurityUtils.initializeSecurity(scmStorageConfig,
+              scmInfo.getScmId(), config, getScmAddress(scmhaNodeDetails, conf),
+              false);
+        }
+        scmStorageConfig.setPrimaryScmNodeId(scmInfo.getScmId());

Review comment:
       Updated to fix the issue gracefully.
   
   **The current solution  in PR is** 
   1. Removed isLeader check and added a check if it is primary SCM it can issue SCM signed certificate. So, in case leader changes still newly bootstrapped SCM can get signed cert.
   2. But if the primary is down, it will fail after failover retry count, instead of after 3 retries to SCM. (Will open a new Jira to address this)
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] GlenGeng commented on pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
GlenGeng commented on pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#issuecomment-797360236


   Please hold on the merge util we finish merging master back to 2823. Thanks!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] xiaoyuyao commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r604431084



##########
File path: hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/HAUtils.java
##########
@@ -326,4 +350,123 @@ public static void checkSecurityAndSCMHAEnabled(OzoneConfiguration conf) {
       }
     }
   }
+
+  /**
+   * Build CA list which need to be passed to client.
+   *
+   * If certificate client is null, obtain the list of CA using SCM security
+   * client, else it uses certificate client.
+   * @param certClient
+   * @param configuration
+   * @return list of CA
+   * @throws IOException
+   */
+  public static List<String> buildCAList(CertificateClient certClient,
+      ConfigurationSource configuration) throws IOException {
+    //TODO: make it configurable.
+    long waitTime = 5 * 60 * 1000L;
+    long retryTime = 10 * 1000L;
+    long currentTime = Time.monotonicNow();
+    List<String> caCertPemList = null;
+    if (certClient != null) {
+      caCertPemList = new ArrayList<>();
+      if (!SCMHAUtils.isSCMHAEnabled(configuration)) {
+        if (certClient.getRootCACertificate() != null) {
+          caCertPemList.add(CertificateCodec.getPEMEncodedString(

Review comment:
       I would suggest we have a method to get a list of X509Certificate inside SCM and make a conversion helper for OM like you did in buildCAX509List.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r604648420



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
##########
@@ -319,7 +331,8 @@ private StorageContainerManager(OzoneConfiguration conf,
       // if no Security, we do not create a Certificate Server at all.
       // This allows user to boot SCM without security temporarily
       // and then come back and enable it without any impact.
-      certificateServer = null;
+      rootCertificateServer = null;

Review comment:
       Done. securityProtocolServer is needed, as it is used in other places in SCM.
   Removed scmCertificateServer and rootCACertificateserver




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r600366956



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
##########
@@ -793,7 +880,18 @@ public static boolean scmBootstrap(OzoneConfiguration conf)
         // SCM Node info containing hostname to scm Id mappings
         // will be persisted into the version file once this node gets added
         // to existing SCM ring post node regular start up.
+
+        if(OzoneSecurityUtil.isSecurityEnabled(conf)) {
+          HASecurityUtils.initializeSecurity(scmStorageConfig,
+              scmInfo.getScmId(), config, getScmAddress(scmhaNodeDetails, conf),
+              false);
+        }
+        scmStorageConfig.setPrimaryScmNodeId(scmInfo.getScmId());

Review comment:
       Primary SCM only starts Root CA that is the only one can issue scm sub ca certificates.
   
   But if you add later on, and the leader changes there will be no rootCA running on that SCM, so bootstrap won't be successful, we will not hit this code line at all. Let me try to handle that gracefully.
   

##########
File path: hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/client/DefaultCertificateClient.java
##########
@@ -818,29 +830,64 @@ public Logger getLogger() {
     return logger;
   }
 
-  /**
-   * Create a scm security client, used to get SCM signed certificate.
-   *
-   * @return {@link SCMSecurityProtocol}
-   */
-  private static SCMSecurityProtocol getScmSecurityClient(
-      OzoneConfiguration conf) throws IOException {
-    RPC.setProtocolEngine(conf, SCMSecurityProtocolPB.class,
-        ProtobufRpcEngine.class);
-    long scmVersion =
-        RPC.getProtocolVersion(ScmBlockLocationProtocolPB.class);
-    InetSocketAddress scmSecurityProtoAdd =
-        HddsServerUtil.getScmAddressForSecurityProtocol(conf);
-    SCMSecurityProtocolClientSideTranslatorPB scmSecurityClient =
-        new SCMSecurityProtocolClientSideTranslatorPB(
-            RPC.getProxy(SCMSecurityProtocolPB.class, scmVersion,
-                scmSecurityProtoAdd, UserGroupInformation.getCurrentUser(),
-                conf, NetUtils.getDefaultSocketFactory(conf),
-                Client.getRpcTimeout(conf)));
-    return scmSecurityClient;
+  public String getComponentName() {
+    return null;
   }
 
-  public String getComponentName() {
+  @Override
+  public X509Certificate getRootCACertificate() {
+    if (rootCaCertId != null) {
+      return certificateMap.get(rootCaCertId);
+    }
     return null;
   }
+
+  @Override
+  public void storeRootCACertificate(String pemEncodedCert, boolean force)
+      throws CertificateException {
+    CertificateCodec certificateCodec = new CertificateCodec(securityConfig,
+        component);
+    try {
+      Path basePath = securityConfig.getCertificateLocation(component);
+
+      X509Certificate cert =
+          CertificateCodec.getX509Certificate(pemEncodedCert);
+      String certName = String.format(CERT_FILE_NAME_FORMAT,
+          cert.getSerialNumber().toString());
+
+      certName = ROOT_CA_CERT_PREFIX + certName;
+      rootCaCertId = cert.getSerialNumber().toString();
+
+      certificateCodec.writeCertificate(basePath, certName,
+          pemEncodedCert, force);
+      certificateMap.putIfAbsent(cert.getSerialNumber().toString(), cert);
+    } catch (IOException | java.security.cert.CertificateException e) {
+      throw new CertificateException("Error while storing Root CA " +
+          "certificate.", e, CERTIFICATE_ERROR);
+    }
+  }
+
+  @Override
+  public List<String> listCA() throws IOException {
+    if (pemEncodedCACerts == null) {
+      updateCAList();

Review comment:
       Done

##########
File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/OzoneContainer.java
##########
@@ -167,8 +171,25 @@ public OzoneContainer(
     blockDeletingService =
         new BlockDeletingService(this, svcInterval.toMillis(), serviceTimeout,
             TimeUnit.MILLISECONDS, config);
-    tlsClientConfig = RatisHelper.createTlsClientConfig(
-        secConf, certClient != null ? certClient.getCACertificate() : null);
+
+    List< X509Certificate > x509Certificates;

Review comment:
       Done




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] xiaoyuyao commented on pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#issuecomment-799589638


   > I have split the changes into 2 Jiras [HDDS-4897](https://issues.apache.org/jira/browse/HDDS-4897) and [HDDS-4952](https://issues.apache.org/jira/browse/HDDS-4952).
   > 
   > Integration to code will do in this Jira, once the above Jiras are merged.
   
   Thanks for the efforts. I will look into those dependency JIRAs. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r603029843



##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerRatisServer.java
##########
@@ -651,9 +652,14 @@ private static Parameters createServerTlsParameters(SecurityConfig conf,
     Parameters parameters = new Parameters();
 
     if (conf.isSecurityEnabled() && conf.isGrpcTlsEnabled()) {
+      ArrayList< X509Certificate > listCA = new ArrayList<>();
+      listCA.add(caClient.getCACertificate());

Review comment:
       As XceiverServerRatis also has similar kind of logic when starting ratis server in DN.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r603016336



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
##########
@@ -319,7 +331,8 @@ private StorageContainerManager(OzoneConfiguration conf,
       // if no Security, we do not create a Certificate Server at all.
       // This allows user to boot SCM without security temporarily
       // and then come back and enable it without any impact.
-      certificateServer = null;
+      rootCertificateServer = null;

Review comment:
       Passed both cert servers to security protocol server




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 closed pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 closed pull request #2000:
URL: https://github.com/apache/ozone/pull/2000


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] xiaoyuyao commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r595517926



##########
File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
##########
@@ -416,4 +418,7 @@ private OzoneConsts() {
 
   // CRL Sequence Id
   public static final String CRL_SEQUENCE_ID_KEY = "CRL_SEQUENCE_ID";
+
+  public static final String SCM_CA_PATH = "ca";
+  public static final String SCM_CA_CERT_STORAGE_DIR = "scm";

Review comment:
       Do we differentiate the primary scm root certificate and the the scm ca certificate? I think they are all stored under {metadir}/ca/scm with CA-{certid}.pem?
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 merged pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 merged pull request #2000:
URL: https://github.com/apache/ozone/pull/2000


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#issuecomment-811663348


   Thank You @xiaoyuyao and @GlenGeng for the review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] xiaoyuyao commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r595399151



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/HASecurityUtils.java
##########
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership.  The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.hdds.scm.ha;
+
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.protocol.proto.SCMSecurityProtocolProtos.SCMGetCertResponseProto;
+import org.apache.hadoop.hdds.protocolPB.SCMSecurityProtocolClientSideTranslatorPB;
+import org.apache.hadoop.hdds.scm.server.SCMCertStore;
+import org.apache.hadoop.hdds.scm.server.SCMStorageConfig;
+import org.apache.hadoop.hdds.security.x509.SecurityConfig;
+import org.apache.hadoop.hdds.security.x509.certificate.authority.CertificateServer;
+import org.apache.hadoop.hdds.security.x509.certificate.authority.DefaultCAServer;
+import org.apache.hadoop.hdds.security.x509.certificate.authority.PKIProfiles.DefaultCAProfile;
+import org.apache.hadoop.hdds.security.x509.certificate.client.CertificateClient;
+import org.apache.hadoop.hdds.security.x509.certificate.client.SCMCertificateClient;
+import org.apache.hadoop.hdds.security.x509.certificate.utils.CertificateCodec;
+import org.apache.hadoop.hdds.security.x509.certificates.utils.CertificateSignRequest;
+import org.apache.hadoop.hdds.utils.HddsServerUtil;
+import org.apache.ratis.conf.Parameters;
+import org.apache.ratis.grpc.GrpcConfigKeys;
+import org.apache.ratis.grpc.GrpcTlsConfig;
+import org.bouncycastle.cert.X509CertificateHolder;
+import org.bouncycastle.pkcs.PKCS10CertificationRequest;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.net.InetAddress;
+import java.net.InetSocketAddress;
+import java.nio.file.Paths;
+import java.security.KeyPair;
+import java.security.cert.CertificateException;
+import java.security.cert.X509Certificate;
+import java.util.concurrent.ExecutionException;
+
+import static org.apache.hadoop.hdds.protocol.proto.HddsProtos.NodeType.SCM;
+import static org.apache.hadoop.hdds.security.x509.certificate.authority.CertificateApprover.ApprovalType.KERBEROS_TRUSTED;
+import static org.apache.hadoop.hdds.security.x509.certificates.utils.CertificateSignRequest.getEncodedString;
+import static org.apache.hadoop.ozone.OzoneConsts.SCM_CA_CERT_STORAGE_DIR;
+import static org.apache.hadoop.ozone.OzoneConsts.SCM_CA_PATH;
+
+public final class HASecurityUtils {
+
+  private HASecurityUtils() {
+  }
+
+  public static final Logger LOG =
+      LoggerFactory.getLogger(HASecurityUtils.class);
+
+  /**
+   * Initialize Security which generates public, private key pair and get SCM
+   * signed certificate and persist to local disk.
+   * @param scmStorageConfig
+   * @param fetchedScmId
+   * @param conf
+   * @param scmAddress
+   * @throws IOException
+   */
+  public static void initializeSecurity(SCMStorageConfig scmStorageConfig,
+      String fetchedScmId, OzoneConfiguration conf,
+      InetSocketAddress scmAddress, boolean primaryscm)
+      throws IOException {
+    LOG.info("Initializing secure StorageContainerManager.");
+
+    CertificateClient certClient =
+        new SCMCertificateClient(new SecurityConfig(conf));
+    CertificateClient.InitResponse response = certClient.init();
+    LOG.info("Init response: {}", response);
+    switch (response) {
+    case SUCCESS:
+      LOG.info("Initialization successful.");
+      break;
+    case GETCERT:
+      if (!primaryscm) {
+        getSCMSignedCert(certClient, conf, fetchedScmId, scmStorageConfig,
+            scmAddress);
+      } else {
+        getPrimarySCMSignedCert(certClient, conf, fetchedScmId,
+            scmStorageConfig, scmAddress);
+      }
+      LOG.info("Successfully stored SCM signed certificate.");
+      break;
+    case FAILURE:
+      LOG.error("SCM security initialization failed.");
+      throw new RuntimeException("OM security initialization failed.");
+    case RECOVER:
+      LOG.error("SCM security initialization failed. SCM certificate is " +
+          "missing.");
+      throw new RuntimeException("SCM security initialization failed.");
+    default:
+      LOG.error("SCM security initialization failed. Init response: {}",
+          response);
+      throw new RuntimeException("SCM security initialization failed.");
+    }
+  }
+
+  /**
+   * Get SCM signed certificate and store it using certificate client.
+   */
+  private static void getSCMSignedCert(CertificateClient client,
+      OzoneConfiguration config, String fetchedSCMId,
+      SCMStorageConfig scmStorageConfig, InetSocketAddress scmAddress)
+      throws IOException {
+
+    PKCS10CertificationRequest csr = generateCSR(client, scmStorageConfig,
+        config, scmAddress, fetchedSCMId);
+
+    HddsProtos.ScmNodeDetailsProto scmNodeDetailsProto =
+        HddsProtos.ScmNodeDetailsProto.newBuilder()
+            .setClusterId(scmStorageConfig.getClusterID())
+            .setHostName(scmAddress.getHostName())
+            .setScmNodeId(fetchedSCMId).build();
+
+    SCMSecurityProtocolClientSideTranslatorPB secureScmClient =
+        HddsServerUtil.getScmSecurityClient(config);
+
+    SCMGetCertResponseProto response = secureScmClient.
+        getSCMCertChain(scmNodeDetailsProto, getEncodedString(csr));
+    String pemEncodedCert = response.getX509Certificate();
+
+    try {
+
+      // Store SCM CA certificate.
+      if (response.hasX509CACertificate()) {
+        String pemEncodedRootCert = response.getX509CACertificate();
+        client.storeCertificate(pemEncodedRootCert, true, true);
+        client.storeCertificate(pemEncodedCert, true);
+
+
+        // Write to the location, so that Default CA server can read the
+        // certificate generated by cert client.
+        CertificateCodec certCodec =
+            new CertificateCodec(new SecurityConfig(config),
+                client.getComponentName());
+
+        X509Certificate certificate =
+            CertificateCodec.getX509Certificate(pemEncodedCert);
+        certCodec.writeCertificate(certCodec.getCertificateHolder(certificate));
+
+        // Persist scm cert serial ID.
+        scmStorageConfig.setScmCertSerialId(certificate.getSerialNumber()

Review comment:
       On primary SCM(Root CA), the non-root SCM cert will overwrite the cert serial id in the normal run. If it is expected, should we skip write the root CA cert serial id to the version file?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] GlenGeng commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
GlenGeng commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r601006956



##########
File path: hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/scm/proxy/SCMBlockLocationFailoverProxyProvider.java
##########
@@ -70,11 +70,20 @@
   private final int maxRetryCount;
   private final long retryInterval;
 
+  private final UserGroupInformation ugi;
+
 
   public SCMBlockLocationFailoverProxyProvider(ConfigurationSource conf) {
     this.conf = conf;
     this.scmVersion = RPC.getProtocolVersion(ScmBlockLocationProtocolPB.class);
 
+    try {
+      this.ugi = UserGroupInformation.getCurrentUser();

Review comment:
       Thanks for the clarification!




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on pull request #2000: HDDS-4915. Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#issuecomment-792551033


   PR based on #1988 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#issuecomment-804545998


   Rebased with latest HDDS-2823.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] xiaoyuyao commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r604472651



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
##########
@@ -558,44 +577,99 @@ private void initializeSystemManagers(OzoneConfiguration conf,
    */
   private void initializeCAnSecurityProtocol(OzoneConfiguration conf,
       SCMConfigurator configurator) throws IOException {
-    if(configurator.getCertificateServer() != null) {
-      this.certificateServer = configurator.getCertificateServer();
+
+    // TODO: Support Certificate Server loading via Class Name loader.
+    // So it is easy to use different Certificate Servers if needed.
+    if(this.scmMetadataStore == null) {
+      LOG.error("Cannot initialize Certificate Server without a valid meta " +
+          "data layer.");
+      throw new SCMException("Cannot initialize CA without a valid metadata " +
+          "store", ResultCodes.SCM_NOT_INITIALIZED);
+    }
+
+    certificateStore =
+        new SCMCertStore.Builder().setMetadaStore(scmMetadataStore)
+            .setRatisServer(scmHAManager.getRatisServer())
+            .setCRLSequenceId(getLastSequenceIdForCRL()).build();
+
+
+    // If primary SCM node Id is set it means this is a cluster which has
+    // performed init with SCM HA version code.
+    if (scmStorageConfig.checkPrimarySCMIdInitialized()) {
+      // Start specific instance SCM CA server.
+      String subject = SCM_SUB_CA_PREFIX +
+          InetAddress.getLocalHost().getHostName();
+      if (configurator.getCertificateServer() != null) {
+        this.scmCertificateServer = configurator.getCertificateServer();
+      } else {
+        scmCertificateServer = new DefaultCAServer(subject,
+            scmStorageConfig.getClusterID(), scmStorageConfig.getScmId(),
+            certificateStore, new DefaultProfile(),
+            scmCertificateClient.getComponentName());
+        // INTERMEDIARY_CA which issues certs to DN and OM.
+        scmCertificateServer.init(new SecurityConfig(configuration),
+            CertificateServer.CAType.INTERMEDIARY_CA);
+      }
+
+      if (primaryScmNodeId.equals(scmStorageConfig.getScmId())) {
+        if (configurator.getCertificateServer() != null) {
+          this.rootCertificateServer = configurator.getCertificateServer();
+        } else {
+          rootCertificateServer =
+              HASecurityUtils.initializeRootCertificateServer(
+              conf, certificateStore, scmStorageConfig);
+        }
+
+        BigInteger certSerial =

Review comment:
       Can we wrap line 623 and 641 into separate helper, this seems to be a long function already. 

##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
##########
@@ -558,44 +577,99 @@ private void initializeSystemManagers(OzoneConfiguration conf,
    */
   private void initializeCAnSecurityProtocol(OzoneConfiguration conf,
       SCMConfigurator configurator) throws IOException {
-    if(configurator.getCertificateServer() != null) {
-      this.certificateServer = configurator.getCertificateServer();
+
+    // TODO: Support Certificate Server loading via Class Name loader.
+    // So it is easy to use different Certificate Servers if needed.
+    if(this.scmMetadataStore == null) {
+      LOG.error("Cannot initialize Certificate Server without a valid meta " +
+          "data layer.");
+      throw new SCMException("Cannot initialize CA without a valid metadata " +
+          "store", ResultCodes.SCM_NOT_INITIALIZED);
+    }
+
+    certificateStore =
+        new SCMCertStore.Builder().setMetadaStore(scmMetadataStore)
+            .setRatisServer(scmHAManager.getRatisServer())
+            .setCRLSequenceId(getLastSequenceIdForCRL()).build();
+
+
+    // If primary SCM node Id is set it means this is a cluster which has
+    // performed init with SCM HA version code.
+    if (scmStorageConfig.checkPrimarySCMIdInitialized()) {
+      // Start specific instance SCM CA server.
+      String subject = SCM_SUB_CA_PREFIX +
+          InetAddress.getLocalHost().getHostName();
+      if (configurator.getCertificateServer() != null) {
+        this.scmCertificateServer = configurator.getCertificateServer();
+      } else {
+        scmCertificateServer = new DefaultCAServer(subject,
+            scmStorageConfig.getClusterID(), scmStorageConfig.getScmId(),
+            certificateStore, new DefaultProfile(),
+            scmCertificateClient.getComponentName());
+        // INTERMEDIARY_CA which issues certs to DN and OM.
+        scmCertificateServer.init(new SecurityConfig(configuration),
+            CertificateServer.CAType.INTERMEDIARY_CA);
+      }
+
+      if (primaryScmNodeId.equals(scmStorageConfig.getScmId())) {
+        if (configurator.getCertificateServer() != null) {
+          this.rootCertificateServer = configurator.getCertificateServer();
+        } else {
+          rootCertificateServer =
+              HASecurityUtils.initializeRootCertificateServer(
+              conf, certificateStore, scmStorageConfig);
+        }
+
+        BigInteger certSerial =
+            scmCertificateClient.getCertificate().getSerialNumber();
+        // Store the certificate in DB. On primary SCM when init happens, the
+        // certificate is not persisted to DB. As we don't have Metadatstore
+        // and ratis server initialized with statemachine. We need to do only
+        // for primary scm, for other bootstrapped scm's certificates will be
+        // persisted via ratis.
+        if (certificateStore.getCertificateByID(certSerial,
+            VALID_CERTS) == null) {
+          LOG.info("Storing certSerial {}", certSerial);

Review comment:
       The log message can be improved. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#issuecomment-804063090


   Re-opened pull request based on HDDS-4897.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r603019036



##########
File path: hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/ScmOption.java
##########
@@ -61,6 +61,7 @@ public ScmClient createScmClient() {
 
       return new ContainerOperationClient(conf);
     } catch (IOException ex) {
+      ex.printStackTrace();

Review comment:
       This is for my debugging I have added, will remove it.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] xiaoyuyao commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r604441095



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
##########
@@ -319,7 +331,8 @@ private StorageContainerManager(OzoneConfiguration conf,
       // if no Security, we do not create a Certificate Server at all.
       // This allows user to boot SCM without security temporarily
       // and then come back and enable it without any impact.
-      certificateServer = null;
+      rootCertificateServer = null;

Review comment:
       can we get rid of scmCertificateServer and securityProtocolServer as member variable from SCM and maintain them as local variable of initializeCAnSecurityProtocol?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r603013863



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/protocol/SCMSecurityProtocolServerSideTranslatorPB.java
##########
@@ -271,6 +278,7 @@ public SCMGetCertResponseProto getRootCACertificate() throws IOException {
         SCMGetCertResponseProto
             .newBuilder()
             .setResponseCode(ResponseCode.success)
+            .setX509Certificate(rootCACertificate)

Review comment:
       As x509Certificate is mandatory in SCMGetCertResponseProto.
   required string x509Certificate = 2; 
   
   So, set the result in both of them.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r603012814



##########
File path: hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/HAUtils.java
##########
@@ -326,4 +350,123 @@ public static void checkSecurityAndSCMHAEnabled(OzoneConfiguration conf) {
       }
     }
   }
+
+  /**
+   * Build CA list which need to be passed to client.
+   *
+   * If certificate client is null, obtain the list of CA using SCM security
+   * client, else it uses certificate client.
+   * @param certClient
+   * @param configuration
+   * @return list of CA
+   * @throws IOException
+   */
+  public static List<String> buildCAList(CertificateClient certClient,
+      ConfigurationSource configuration) throws IOException {
+    //TODO: make it configurable.
+    long waitTime = 5 * 60 * 1000L;
+    long retryTime = 10 * 1000L;
+    long currentTime = Time.monotonicNow();
+    List<String> caCertPemList = null;
+    if (certClient != null) {
+      caCertPemList = new ArrayList<>();
+      if (!SCMHAUtils.isSCMHAEnabled(configuration)) {
+        if (certClient.getRootCACertificate() != null) {
+          caCertPemList.add(CertificateCodec.getPEMEncodedString(

Review comment:
       Refactored this to make it work for both OM/DN/client.
   
   As OM needs list of pemEncodedcerts where as OzoneContainer/ContainerOperationClientt needs X509Certificates.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#issuecomment-800999900


   @xiaoyuyao Addressed/replied review comments these are addressed in HDDS-4897. 
   Can we continue the review #2041 
   
   I will update the PR to use all the Jira's which went in and integrate it.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] GlenGeng commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
GlenGeng commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r599250302



##########
File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/ratis/RatisHelper.java
##########
@@ -294,7 +294,7 @@ public static void createRaftServerProperties(ConfigurationSource ozoneConf,
   // For External gRPC client to server with gRPC TLS.
   // No mTLS for external client as SCM CA does not issued certificates for them
   public static GrpcTlsConfig createTlsClientConfig(SecurityConfig conf,
-      X509Certificate caCert) {
+      List<X509Certificate> caCert) {

Review comment:
       caCerts

##########
File path: hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientGrpc.java
##########
@@ -96,7 +96,7 @@
    * @param caCert   - SCM ca certificate.
    */
   public XceiverClientGrpc(Pipeline pipeline, ConfigurationSource config,
-      X509Certificate caCert) {
+      List<X509Certificate> caCert) {

Review comment:
       typo `caCerts`

##########
File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ha/SCMHAUtils.java
##########
@@ -173,4 +173,15 @@ public static OzoneConfiguration removeSelfId(
     }
     return conf;
   }
+
+  /**
+   * Get SCM Node Id list.
+   * @param configuration
+   * @return list of node ids.
+   */
+  public static Collection<String> getSCMNodeIds(
+      ConfigurationSource configuration) {
+    String scmServiceId = SCMHAUtils.getScmServiceId(configuration);

Review comment:
       no need `SCMHAUtils.` prefix for `getScmServiceId` and `getSCMNodeIds`

##########
File path: hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/scm/proxy/SCMContainerLocationFailoverProxyProvider.java
##########
@@ -70,9 +70,18 @@
   private final int maxRetryCount;
   private final long retryInterval;
 
+  private final UserGroupInformation ugi;
+
 
   public SCMContainerLocationFailoverProxyProvider(ConfigurationSource conf) {
     this.conf = conf;
+
+    try {
+      this.ugi = UserGroupInformation.getCurrentUser();

Review comment:
       same as block client.

##########
File path: hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/client/DefaultCertificateClient.java
##########
@@ -818,29 +830,64 @@ public Logger getLogger() {
     return logger;
   }
 
-  /**
-   * Create a scm security client, used to get SCM signed certificate.
-   *
-   * @return {@link SCMSecurityProtocol}
-   */
-  private static SCMSecurityProtocol getScmSecurityClient(
-      OzoneConfiguration conf) throws IOException {
-    RPC.setProtocolEngine(conf, SCMSecurityProtocolPB.class,
-        ProtobufRpcEngine.class);
-    long scmVersion =
-        RPC.getProtocolVersion(ScmBlockLocationProtocolPB.class);
-    InetSocketAddress scmSecurityProtoAdd =
-        HddsServerUtil.getScmAddressForSecurityProtocol(conf);
-    SCMSecurityProtocolClientSideTranslatorPB scmSecurityClient =
-        new SCMSecurityProtocolClientSideTranslatorPB(
-            RPC.getProxy(SCMSecurityProtocolPB.class, scmVersion,
-                scmSecurityProtoAdd, UserGroupInformation.getCurrentUser(),
-                conf, NetUtils.getDefaultSocketFactory(conf),
-                Client.getRpcTimeout(conf)));
-    return scmSecurityClient;
+  public String getComponentName() {
+    return null;
   }
 
-  public String getComponentName() {
+  @Override
+  public X509Certificate getRootCACertificate() {
+    if (rootCaCertId != null) {
+      return certificateMap.get(rootCaCertId);
+    }
     return null;
   }
+
+  @Override
+  public void storeRootCACertificate(String pemEncodedCert, boolean force)
+      throws CertificateException {
+    CertificateCodec certificateCodec = new CertificateCodec(securityConfig,
+        component);
+    try {
+      Path basePath = securityConfig.getCertificateLocation(component);
+
+      X509Certificate cert =
+          CertificateCodec.getX509Certificate(pemEncodedCert);
+      String certName = String.format(CERT_FILE_NAME_FORMAT,
+          cert.getSerialNumber().toString());
+
+      certName = ROOT_CA_CERT_PREFIX + certName;
+      rootCaCertId = cert.getSerialNumber().toString();
+
+      certificateCodec.writeCertificate(basePath, certName,
+          pemEncodedCert, force);
+      certificateMap.putIfAbsent(cert.getSerialNumber().toString(), cert);
+    } catch (IOException | java.security.cert.CertificateException e) {
+      throw new CertificateException("Error while storing Root CA " +
+          "certificate.", e, CERTIFICATE_ERROR);
+    }
+  }
+
+  @Override
+  public List<String> listCA() throws IOException {
+    if (pemEncodedCACerts == null) {
+      updateCAList();

Review comment:
       Will there be contention of the client ? Since there is no lock protection here, need make sure the modifications are properly serialized.

##########
File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ha/SCMNodeInfo.java
##########
@@ -246,4 +247,20 @@ public String getScmSecurityAddress() {
   public String getScmDatanodeAddress() {
     return scmDatanodeAddress;
   }
+
+  public static SCMNodeInfo getScmNodeInfo(ConfigurationSource conf,
+      String nodeId) {
+    List<SCMNodeInfo> scmNodeInfoList = SCMNodeInfo.buildNodeInfo(conf);
+    Preconditions.checkState(scmNodeInfoList.size() >=1);
+    if (SCMHAUtils.getScmServiceId(conf) != null) {
+      for (SCMNodeInfo scmNodeInfo : scmNodeInfoList) {
+        if (scmNodeInfo.getNodeId().equals(nodeId)) {
+          return scmNodeInfo;
+        }
+      }

Review comment:
       remove line 264, return null here, and print a warn ?

##########
File path: hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/scm/proxy/SCMBlockLocationFailoverProxyProvider.java
##########
@@ -70,11 +70,20 @@
   private final int maxRetryCount;
   private final long retryInterval;
 
+  private final UserGroupInformation ugi;
+
 
   public SCMBlockLocationFailoverProxyProvider(ConfigurationSource conf) {
     this.conf = conf;
     this.scmVersion = RPC.getProtocolVersion(ScmBlockLocationProtocolPB.class);
 
+    try {
+      this.ugi = UserGroupInformation.getCurrentUser();

Review comment:
       why need change this file ? The logic is unchanged.

##########
File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/OzoneContainer.java
##########
@@ -167,8 +171,25 @@ public OzoneContainer(
     blockDeletingService =
         new BlockDeletingService(this, svcInterval.toMillis(), serviceTimeout,
             TimeUnit.MILLISECONDS, config);
-    tlsClientConfig = RatisHelper.createTlsClientConfig(
-        secConf, certClient != null ? certClient.getCACertificate() : null);
+
+    List< X509Certificate > x509Certificates;

Review comment:
       ```
   List< X509Certificate > x509Certificates = null;
   if (certClient != null) {
     ///
   }
   ```

##########
File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ha/SCMNodeInfo.java
##########
@@ -246,4 +247,20 @@ public String getScmSecurityAddress() {
   public String getScmDatanodeAddress() {
     return scmDatanodeAddress;
   }
+
+  public static SCMNodeInfo getScmNodeInfo(ConfigurationSource conf,

Review comment:
       This function seems not used.

##########
File path: hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientRatis.java
##########
@@ -81,7 +81,7 @@ public static XceiverClientRatis newXceiverClientRatis(
 
   public static XceiverClientRatis newXceiverClientRatis(
       org.apache.hadoop.hdds.scm.pipeline.Pipeline pipeline,
-      ConfigurationSource ozoneConf, X509Certificate caCert) {
+      ConfigurationSource ozoneConf, List<X509Certificate> caCert) {

Review comment:
       caCerts




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r603005291



##########
File path: hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/authority/DefaultApprover.java
##########
@@ -76,12 +77,12 @@ public DefaultApprover(PKIProfile pkiProfile, SecurityConfig config) {
    * @param scmId - SCM id.
    * @param clusterId - Cluster id.
    * @return Signed Certificate.
-   * @throws IOException - On Error
+   * @throws IOException               - On Error

Review comment:
       Done




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r601974239



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
##########
@@ -793,7 +880,18 @@ public static boolean scmBootstrap(OzoneConfiguration conf)
         // SCM Node info containing hostname to scm Id mappings
         // will be persisted into the version file once this node gets added
         // to existing SCM ring post node regular start up.
+
+        if(OzoneSecurityUtil.isSecurityEnabled(conf)) {
+          HASecurityUtils.initializeSecurity(scmStorageConfig,
+              scmInfo.getScmId(), config, getScmAddress(scmhaNodeDetails, conf),
+              false);
+        }
+        scmStorageConfig.setPrimaryScmNodeId(scmInfo.getScmId());

Review comment:
       Yes that is the plan as discussed offline, will do that change in HDDS-5027




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] xiaoyuyao commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r604438524



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMSecurityProtocolServer.java
##########
@@ -288,22 +306,18 @@ public String getCACertificate() throws IOException {
   public List<String> listCACertificate() throws IOException {
     List<String> caCerts =
         listCertificate(NodeType.SCM, 0, 10, false);
-    caCerts.add(getRootCACertificate());
     return caCerts;
   }
 
   @Override
   public String getRootCACertificate() throws IOException {
     LOGGER.debug("Getting Root CA certificate.");
-    //TODO: This code will be modified after HDDS-4897 is merged and
-    // integrated. For now getting RootCA cert from certificateServer.
-    try {
+    if (storageContainerManager.getScmStorageConfig()

Review comment:
       Can we pass in the Root CA certificate in constructor to the security protocol server without calling back into the cert client from scm?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] xiaoyuyao commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r595401448



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/HASecurityUtils.java
##########
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership.  The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.hdds.scm.ha;
+
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.protocol.proto.SCMSecurityProtocolProtos.SCMGetCertResponseProto;
+import org.apache.hadoop.hdds.protocolPB.SCMSecurityProtocolClientSideTranslatorPB;
+import org.apache.hadoop.hdds.scm.server.SCMCertStore;
+import org.apache.hadoop.hdds.scm.server.SCMStorageConfig;
+import org.apache.hadoop.hdds.security.x509.SecurityConfig;
+import org.apache.hadoop.hdds.security.x509.certificate.authority.CertificateServer;
+import org.apache.hadoop.hdds.security.x509.certificate.authority.DefaultCAServer;
+import org.apache.hadoop.hdds.security.x509.certificate.authority.PKIProfiles.DefaultCAProfile;
+import org.apache.hadoop.hdds.security.x509.certificate.client.CertificateClient;
+import org.apache.hadoop.hdds.security.x509.certificate.client.SCMCertificateClient;
+import org.apache.hadoop.hdds.security.x509.certificate.utils.CertificateCodec;
+import org.apache.hadoop.hdds.security.x509.certificates.utils.CertificateSignRequest;
+import org.apache.hadoop.hdds.utils.HddsServerUtil;
+import org.apache.ratis.conf.Parameters;
+import org.apache.ratis.grpc.GrpcConfigKeys;
+import org.apache.ratis.grpc.GrpcTlsConfig;
+import org.bouncycastle.cert.X509CertificateHolder;
+import org.bouncycastle.pkcs.PKCS10CertificationRequest;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.net.InetAddress;
+import java.net.InetSocketAddress;
+import java.nio.file.Paths;
+import java.security.KeyPair;
+import java.security.cert.CertificateException;
+import java.security.cert.X509Certificate;
+import java.util.concurrent.ExecutionException;
+
+import static org.apache.hadoop.hdds.protocol.proto.HddsProtos.NodeType.SCM;
+import static org.apache.hadoop.hdds.security.x509.certificate.authority.CertificateApprover.ApprovalType.KERBEROS_TRUSTED;
+import static org.apache.hadoop.hdds.security.x509.certificates.utils.CertificateSignRequest.getEncodedString;
+import static org.apache.hadoop.ozone.OzoneConsts.SCM_CA_CERT_STORAGE_DIR;
+import static org.apache.hadoop.ozone.OzoneConsts.SCM_CA_PATH;
+
+public final class HASecurityUtils {
+
+  private HASecurityUtils() {
+  }
+
+  public static final Logger LOG =
+      LoggerFactory.getLogger(HASecurityUtils.class);
+
+  /**
+   * Initialize Security which generates public, private key pair and get SCM
+   * signed certificate and persist to local disk.
+   * @param scmStorageConfig
+   * @param fetchedScmId
+   * @param conf
+   * @param scmAddress
+   * @throws IOException
+   */
+  public static void initializeSecurity(SCMStorageConfig scmStorageConfig,
+      String fetchedScmId, OzoneConfiguration conf,
+      InetSocketAddress scmAddress, boolean primaryscm)
+      throws IOException {
+    LOG.info("Initializing secure StorageContainerManager.");
+
+    CertificateClient certClient =
+        new SCMCertificateClient(new SecurityConfig(conf));
+    CertificateClient.InitResponse response = certClient.init();
+    LOG.info("Init response: {}", response);
+    switch (response) {
+    case SUCCESS:
+      LOG.info("Initialization successful.");
+      break;
+    case GETCERT:
+      if (!primaryscm) {
+        getSCMSignedCert(certClient, conf, fetchedScmId, scmStorageConfig,
+            scmAddress);
+      } else {
+        getPrimarySCMSignedCert(certClient, conf, fetchedScmId,
+            scmStorageConfig, scmAddress);
+      }
+      LOG.info("Successfully stored SCM signed certificate.");
+      break;
+    case FAILURE:
+      LOG.error("SCM security initialization failed.");
+      throw new RuntimeException("OM security initialization failed.");
+    case RECOVER:
+      LOG.error("SCM security initialization failed. SCM certificate is " +
+          "missing.");
+      throw new RuntimeException("SCM security initialization failed.");
+    default:
+      LOG.error("SCM security initialization failed. Init response: {}",
+          response);
+      throw new RuntimeException("SCM security initialization failed.");
+    }
+  }
+
+  /**
+   * Get SCM signed certificate and store it using certificate client.
+   */
+  private static void getSCMSignedCert(CertificateClient client,
+      OzoneConfiguration config, String fetchedSCMId,
+      SCMStorageConfig scmStorageConfig, InetSocketAddress scmAddress)
+      throws IOException {
+
+    PKCS10CertificationRequest csr = generateCSR(client, scmStorageConfig,
+        config, scmAddress, fetchedSCMId);
+
+    HddsProtos.ScmNodeDetailsProto scmNodeDetailsProto =
+        HddsProtos.ScmNodeDetailsProto.newBuilder()
+            .setClusterId(scmStorageConfig.getClusterID())
+            .setHostName(scmAddress.getHostName())
+            .setScmNodeId(fetchedSCMId).build();
+
+    SCMSecurityProtocolClientSideTranslatorPB secureScmClient =
+        HddsServerUtil.getScmSecurityClient(config);
+
+    SCMGetCertResponseProto response = secureScmClient.
+        getSCMCertChain(scmNodeDetailsProto, getEncodedString(csr));
+    String pemEncodedCert = response.getX509Certificate();
+
+    try {
+
+      // Store SCM CA certificate.
+      if (response.hasX509CACertificate()) {
+        String pemEncodedRootCert = response.getX509CACertificate();
+        client.storeCertificate(pemEncodedRootCert, true, true);
+        client.storeCertificate(pemEncodedCert, true);
+
+
+        // Write to the location, so that Default CA server can read the
+        // certificate generated by cert client.
+        CertificateCodec certCodec =
+            new CertificateCodec(new SecurityConfig(config),
+                client.getComponentName());
+
+        X509Certificate certificate =
+            CertificateCodec.getX509Certificate(pemEncodedCert);
+        certCodec.writeCertificate(certCodec.getCertificateHolder(certificate));
+
+        // Persist scm cert serial ID.
+        scmStorageConfig.setScmCertSerialId(certificate.getSerialNumber()
+            .toString());
+      } else {
+        throw new RuntimeException("Unable to retrieve SCM certificate chain");
+      }
+    } catch (IOException | CertificateException e) {
+      LOG.error("Error while storing SCM signed certificate.", e);
+      throw new RuntimeException(e);
+    }
+  }
+
+  private static void getPrimarySCMSignedCert(CertificateClient client,

Review comment:
       NIT: Can we rename this to getPrimarySCMSelfSignedCert?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r603025531



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMSecurityProtocolServer.java
##########
@@ -225,7 +231,7 @@ public String getCertificate(String certSerialId) throws IOException {
         certSerialId);
     try {
       X509Certificate certificate =
-          certificateServer.getCertificate(certSerialId);
+          rootCertificateServer.getCertificate(certSerialId);

Review comment:
       No, as scmcertserver runs on all SCM's used the scmcertserver
   
   Underlying both CertificateServers use the same certstore. So, the semantics of this API is not changed, it can get any certificate.
   
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 edited a comment on pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 edited a comment on pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#issuecomment-800999900


   @xiaoyuyao Addressed/replied review comments these are addressed in HDDS-4897 under PR #2041 
   Can we continue the review in #2041 
   
   I will update the PR to use all the Jira's which went in and integrate it.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r603029489



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMSecurityProtocolServer.java
##########
@@ -70,7 +70,7 @@
 
   private static final Logger LOGGER = LoggerFactory
       .getLogger(SCMSecurityProtocolServer.class);
-  private final CertificateServer certificateServer;
+  private final CertificateServer rootCertificateServer;

Review comment:
       Done. Now SecurityProtocolServer has both rootCertificateServer and scmCertificateServer




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r600366956



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
##########
@@ -793,7 +880,18 @@ public static boolean scmBootstrap(OzoneConfiguration conf)
         // SCM Node info containing hostname to scm Id mappings
         // will be persisted into the version file once this node gets added
         // to existing SCM ring post node regular start up.
+
+        if(OzoneSecurityUtil.isSecurityEnabled(conf)) {
+          HASecurityUtils.initializeSecurity(scmStorageConfig,
+              scmInfo.getScmId(), config, getScmAddress(scmhaNodeDetails, conf),
+              false);
+        }
+        scmStorageConfig.setPrimaryScmNodeId(scmInfo.getScmId());

Review comment:
       Primary SCM only starts Root CA that is the only one can issue scm sub ca certificates.
   
   But if an SCM is added later on, and the leader changes there will be no rootCA running on that SCM, so bootstrap won't be successful, we will not hit this code line at all. Let me try to handle that gracefully, as with current PR we will hit NPE and retry failOverretry count and finally fail.
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r600356252



##########
File path: hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/scm/proxy/SCMBlockLocationFailoverProxyProvider.java
##########
@@ -70,11 +70,20 @@
   private final int maxRetryCount;
   private final long retryInterval;
 
+  private final UserGroupInformation ugi;
+
 
   public SCMBlockLocationFailoverProxyProvider(ConfigurationSource conf) {
     this.conf = conf;
     this.scmVersion = RPC.getProtocolVersion(ScmBlockLocationProtocolPB.class);
 
+    try {
+      this.ugi = UserGroupInformation.getCurrentUser();

Review comment:
       This is to make use of UGI during the creation of FailOverProxyProvider, otherwise, we will use UGI during create proxy time which might not be correct one.
   Without this change, we shall see this error.
   
   
   ```
   om1_1        | 2021-03-23 05:59:25,420 [IPC Server handler 7 on default port 9862] WARN ipc.Client: Exception encountered while connecting to the server
   om1_1        | javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
   om1_1        | 	at jdk.security.jgss/com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
   om1_1        | 	at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:408)
   om1_1        | 	at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:622)
   om1_1        | 	at org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:413)
   om1_1        | 	at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:822)
   om1_1        | 	at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:818)
   om1_1        | 	at java.base/java.security.AccessController.doPrivileged(Native Method)
   om1_1        | 	at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
   om1_1        | 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
   om1_1        | 	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:818)
   om1_1        | 	at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:413)
   om1_1        | 	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1636)
   om1_1        | 	at org.apache.hadoop.ipc.Client.call(Client.java:1452)
   om1_1        | 	at org.apache.hadoop.ipc.Client.call(Client.java:1405)
   om1_1        | 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
   om1_1        | 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
   om1_1        | 	at com.sun.proxy.$Proxy32.send(Unknown Source)
   om1_1        | 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   om1_1        | 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   om1_1        | 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   om1_1        | 	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
   om1_1        | 	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
   om1_1        | 	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
   om1_1        | 	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
   om1_1        | 	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
   om1_1        | 	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
   om1_1        | 	at com.sun.proxy.$Proxy32.send(Unknown Source)
   om1_1        | 	at org.apache.hadoop.hdds.scm.protocolPB.ScmBlockLocationProtocolClientSideTranslatorPB.submitRequest(ScmBlockLocationProtocolClientSideTranslatorPB.java:118)
   om1_1        | 	at org.apache.hadoop.hdds.scm.protocolPB.ScmBlockLocationProtocolClientSideTranslatorPB.allocateBlock(ScmBlockLocationProtocolClientSideTranslatorPB.java:172)
   om1_1        | 	at org.apache.hadoop.ozone.om.request.key.OMKeyRequest.allocateBlock(OMKeyRequest.java:128)
   om1_1        | 	at org.apache.hadoop.ozone.om.request.key.OMKeyCreateRequest.preExecute(OMKeyCreateRequest.java:151)
   om1_1        | 	at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.processRequest(OzoneManagerProtocolServerSideTranslatorPB.java:139)
   om1_1        | 	at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:87)
   om1_1        | 	at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:122)
   om1_1        | 	at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java)
   om1_1        | 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:528)
   om1_1        | 	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1086)
   om1_1        | 	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1029)
   om1_1        | 	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:957)
   om1_1        | 	at java.base/java.security.AccessController.doPrivileged(Native Method)
   om1_1        | 	at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
   om1_1        | 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
   om1_1        | 	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2957)
   om1_1        | Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
   om1_1        | 	at java.security.jgss/sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:162)
   om1_1        | 	at java.security.jgss/sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:126)
   om1_1        | 	at java.security.jgss/sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:193)
   om1_1        | 	at java.security.jgss/sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:218)
   om1_1        | 	at java.security.jgss/sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:230)
   om1_1        | 	at java.security.jgss/sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:196)
   om1_1        | 	at jdk.security.jgss/com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
   om1_1        | 	... 42 more
   ```

##########
File path: hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/scm/proxy/SCMContainerLocationFailoverProxyProvider.java
##########
@@ -70,9 +70,18 @@
   private final int maxRetryCount;
   private final long retryInterval;
 
+  private final UserGroupInformation ugi;
+
 
   public SCMContainerLocationFailoverProxyProvider(ConfigurationSource conf) {
     this.conf = conf;
+
+    try {
+      this.ugi = UserGroupInformation.getCurrentUser();

Review comment:
       Same as above 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] xiaoyuyao commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r604473546



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
##########
@@ -860,6 +966,40 @@ public static boolean scmInit(OzoneConfiguration conf,
     }
   }
 
+  private static InetSocketAddress getScmAddress(SCMHANodeDetails haDetails,

Review comment:
       Ok




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r603015314



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMSecurityProtocolServer.java
##########
@@ -155,18 +155,27 @@ public String getOMCertificate(OzoneManagerDetailsProto omDetails,
   public String getSCMCertificate(ScmNodeDetailsProto scmNodeDetails,
       String certSignReq) throws IOException {
     Objects.requireNonNull(scmNodeDetails);
-    LOGGER.info("Processing CSR for scm {}, nodeId: {}",
-        scmNodeDetails.getHostName(), scmNodeDetails.getScmNodeId());
+    String primaryScmId =
+        storageContainerManager.getScmStorageConfig().getPrimaryScmNodeId();
 
-    // Check clusterID
-    if (storageContainerManager.getClusterId().equals(
-        scmNodeDetails.getClusterId())) {
-      throw new IOException("SCM ClusterId mismatch. Peer SCM ClusterId " +
-          scmNodeDetails.getClusterId() + ", primary SCM ClusterId "
-          + storageContainerManager.getClusterId());
-    }
+    if (primaryScmId != null &&
+        primaryScmId.equals(storageContainerManager.getScmId())) {
+      LOGGER.info("Processing CSR for scm {}, nodeId: {}",
+          scmNodeDetails.getHostName(), scmNodeDetails.getScmNodeId());
+
+      // Check clusterID
+      if (!storageContainerManager.getClusterId().equals(
+          scmNodeDetails.getClusterId())) {
+        throw new IOException("SCM ClusterId mismatch. Peer SCM ClusterId " +
+            scmNodeDetails.getClusterId() + ", primary SCM ClusterId "
+            + storageContainerManager.getClusterId());
+      }
 
-    return getEncodedCertToString(certSignReq, NodeType.SCM);
+      return getEncodedCertToString(certSignReq, NodeType.SCM);
+    } else {
+      throw new SCMSecurityException("Get SCM Certificate can be run only " +
+          "primary SCM", PRIMARY_SCM_IS_NOT_LEADER);

Review comment:
       Renamed it to NOT_A_PRIMARY_SCM
   Complete handling of it will be done in HDDS-5027




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r600366956



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
##########
@@ -793,7 +880,18 @@ public static boolean scmBootstrap(OzoneConfiguration conf)
         // SCM Node info containing hostname to scm Id mappings
         // will be persisted into the version file once this node gets added
         // to existing SCM ring post node regular start up.
+
+        if(OzoneSecurityUtil.isSecurityEnabled(conf)) {
+          HASecurityUtils.initializeSecurity(scmStorageConfig,
+              scmInfo.getScmId(), config, getScmAddress(scmhaNodeDetails, conf),
+              false);
+        }
+        scmStorageConfig.setPrimaryScmNodeId(scmInfo.getScmId());

Review comment:
       Primary SCM only starts Root CA that is the only one can issue scm sub ca certificates.
   
   But if an SCM is added later on, and the leader changes there will be no rootCA running on that SCM, so bootstrap won't be successful, we will not hit this code line at all. Let me try to handle that gracefully.
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#issuecomment-799196325


   I have split the changes into 2 Jiras HDDS-4897 and HDDS-4952.
   
   Integration to code will do in this Jira, once the above Jiras are merged.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r603003603



##########
File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/XceiverServerRatis.java
##########
@@ -465,9 +466,14 @@ private static Parameters createTlsParameters(SecurityConfig conf,
     Parameters parameters = new Parameters();
 
     if (conf.isSecurityEnabled() && conf.isGrpcTlsEnabled()) {
+      ArrayList< X509Certificate > listCA = new ArrayList<>();

Review comment:
       done




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] GlenGeng commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
GlenGeng commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r599427607



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
##########
@@ -793,7 +880,18 @@ public static boolean scmBootstrap(OzoneConfiguration conf)
         // SCM Node info containing hostname to scm Id mappings
         // will be persisted into the version file once this node gets added
         // to existing SCM ring post node regular start up.
+
+        if(OzoneSecurityUtil.isSecurityEnabled(conf)) {
+          HASecurityUtils.initializeSecurity(scmStorageConfig,
+              scmInfo.getScmId(), config, getScmAddress(scmhaNodeDetails, conf),
+              false);
+        }
+        scmStorageConfig.setPrimaryScmNodeId(scmInfo.getScmId());

Review comment:
       the scmId here may not be the primaryScmId, not sure if it is OK.
   
   say current cluster is scm1 and scm2, scm2 is the already bootstrapped scm.
   now bootstrap scm3, scm3 may talk with scm2 to get the scmInfo, and treate scm2 as the primary SCM.
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r600383272



##########
File path: hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/client/DefaultCertificateClient.java
##########
@@ -818,29 +830,64 @@ public Logger getLogger() {
     return logger;
   }
 
-  /**
-   * Create a scm security client, used to get SCM signed certificate.
-   *
-   * @return {@link SCMSecurityProtocol}
-   */
-  private static SCMSecurityProtocol getScmSecurityClient(
-      OzoneConfiguration conf) throws IOException {
-    RPC.setProtocolEngine(conf, SCMSecurityProtocolPB.class,
-        ProtobufRpcEngine.class);
-    long scmVersion =
-        RPC.getProtocolVersion(ScmBlockLocationProtocolPB.class);
-    InetSocketAddress scmSecurityProtoAdd =
-        HddsServerUtil.getScmAddressForSecurityProtocol(conf);
-    SCMSecurityProtocolClientSideTranslatorPB scmSecurityClient =
-        new SCMSecurityProtocolClientSideTranslatorPB(
-            RPC.getProxy(SCMSecurityProtocolPB.class, scmVersion,
-                scmSecurityProtoAdd, UserGroupInformation.getCurrentUser(),
-                conf, NetUtils.getDefaultSocketFactory(conf),
-                Client.getRpcTimeout(conf)));
-    return scmSecurityClient;
+  public String getComponentName() {
+    return null;
   }
 
-  public String getComponentName() {
+  @Override
+  public X509Certificate getRootCACertificate() {
+    if (rootCaCertId != null) {
+      return certificateMap.get(rootCaCertId);
+    }
     return null;
   }
+
+  @Override
+  public void storeRootCACertificate(String pemEncodedCert, boolean force)
+      throws CertificateException {
+    CertificateCodec certificateCodec = new CertificateCodec(securityConfig,
+        component);
+    try {
+      Path basePath = securityConfig.getCertificateLocation(component);
+
+      X509Certificate cert =
+          CertificateCodec.getX509Certificate(pemEncodedCert);
+      String certName = String.format(CERT_FILE_NAME_FORMAT,
+          cert.getSerialNumber().toString());
+
+      certName = ROOT_CA_CERT_PREFIX + certName;
+      rootCaCertId = cert.getSerialNumber().toString();
+
+      certificateCodec.writeCertificate(basePath, certName,
+          pemEncodedCert, force);
+      certificateMap.putIfAbsent(cert.getSerialNumber().toString(), cert);
+    } catch (IOException | java.security.cert.CertificateException e) {
+      throw new CertificateException("Error while storing Root CA " +
+          "certificate.", e, CERTIFICATE_ERROR);
+    }
+  }
+
+  @Override
+  public List<String> listCA() throws IOException {
+    if (pemEncodedCACerts == null) {
+      updateCAList();

Review comment:
       And on a side note listCA() is not used, added that for future purpose.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] xiaoyuyao commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r595402392



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/HASecurityUtils.java
##########
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership.  The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.hdds.scm.ha;
+
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.protocol.proto.SCMSecurityProtocolProtos.SCMGetCertResponseProto;
+import org.apache.hadoop.hdds.protocolPB.SCMSecurityProtocolClientSideTranslatorPB;
+import org.apache.hadoop.hdds.scm.server.SCMCertStore;
+import org.apache.hadoop.hdds.scm.server.SCMStorageConfig;
+import org.apache.hadoop.hdds.security.x509.SecurityConfig;
+import org.apache.hadoop.hdds.security.x509.certificate.authority.CertificateServer;
+import org.apache.hadoop.hdds.security.x509.certificate.authority.DefaultCAServer;
+import org.apache.hadoop.hdds.security.x509.certificate.authority.PKIProfiles.DefaultCAProfile;
+import org.apache.hadoop.hdds.security.x509.certificate.client.CertificateClient;
+import org.apache.hadoop.hdds.security.x509.certificate.client.SCMCertificateClient;
+import org.apache.hadoop.hdds.security.x509.certificate.utils.CertificateCodec;
+import org.apache.hadoop.hdds.security.x509.certificates.utils.CertificateSignRequest;
+import org.apache.hadoop.hdds.utils.HddsServerUtil;
+import org.apache.ratis.conf.Parameters;
+import org.apache.ratis.grpc.GrpcConfigKeys;
+import org.apache.ratis.grpc.GrpcTlsConfig;
+import org.bouncycastle.cert.X509CertificateHolder;
+import org.bouncycastle.pkcs.PKCS10CertificationRequest;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.net.InetAddress;
+import java.net.InetSocketAddress;
+import java.nio.file.Paths;
+import java.security.KeyPair;
+import java.security.cert.CertificateException;
+import java.security.cert.X509Certificate;
+import java.util.concurrent.ExecutionException;
+
+import static org.apache.hadoop.hdds.protocol.proto.HddsProtos.NodeType.SCM;
+import static org.apache.hadoop.hdds.security.x509.certificate.authority.CertificateApprover.ApprovalType.KERBEROS_TRUSTED;
+import static org.apache.hadoop.hdds.security.x509.certificates.utils.CertificateSignRequest.getEncodedString;
+import static org.apache.hadoop.ozone.OzoneConsts.SCM_CA_CERT_STORAGE_DIR;
+import static org.apache.hadoop.ozone.OzoneConsts.SCM_CA_PATH;
+
+public final class HASecurityUtils {
+
+  private HASecurityUtils() {
+  }
+
+  public static final Logger LOG =
+      LoggerFactory.getLogger(HASecurityUtils.class);
+
+  /**
+   * Initialize Security which generates public, private key pair and get SCM
+   * signed certificate and persist to local disk.
+   * @param scmStorageConfig
+   * @param fetchedScmId
+   * @param conf
+   * @param scmAddress
+   * @throws IOException
+   */
+  public static void initializeSecurity(SCMStorageConfig scmStorageConfig,
+      String fetchedScmId, OzoneConfiguration conf,
+      InetSocketAddress scmAddress, boolean primaryscm)
+      throws IOException {
+    LOG.info("Initializing secure StorageContainerManager.");
+
+    CertificateClient certClient =
+        new SCMCertificateClient(new SecurityConfig(conf));
+    CertificateClient.InitResponse response = certClient.init();
+    LOG.info("Init response: {}", response);
+    switch (response) {
+    case SUCCESS:
+      LOG.info("Initialization successful.");
+      break;
+    case GETCERT:
+      if (!primaryscm) {
+        getSCMSignedCert(certClient, conf, fetchedScmId, scmStorageConfig,
+            scmAddress);
+      } else {
+        getPrimarySCMSignedCert(certClient, conf, fetchedScmId,
+            scmStorageConfig, scmAddress);
+      }
+      LOG.info("Successfully stored SCM signed certificate.");
+      break;
+    case FAILURE:
+      LOG.error("SCM security initialization failed.");
+      throw new RuntimeException("OM security initialization failed.");
+    case RECOVER:
+      LOG.error("SCM security initialization failed. SCM certificate is " +
+          "missing.");
+      throw new RuntimeException("SCM security initialization failed.");
+    default:
+      LOG.error("SCM security initialization failed. Init response: {}",
+          response);
+      throw new RuntimeException("SCM security initialization failed.");
+    }
+  }
+
+  /**
+   * Get SCM signed certificate and store it using certificate client.
+   */
+  private static void getSCMSignedCert(CertificateClient client,
+      OzoneConfiguration config, String fetchedSCMId,
+      SCMStorageConfig scmStorageConfig, InetSocketAddress scmAddress)
+      throws IOException {
+
+    PKCS10CertificationRequest csr = generateCSR(client, scmStorageConfig,
+        config, scmAddress, fetchedSCMId);
+
+    HddsProtos.ScmNodeDetailsProto scmNodeDetailsProto =
+        HddsProtos.ScmNodeDetailsProto.newBuilder()
+            .setClusterId(scmStorageConfig.getClusterID())
+            .setHostName(scmAddress.getHostName())
+            .setScmNodeId(fetchedSCMId).build();
+
+    SCMSecurityProtocolClientSideTranslatorPB secureScmClient =
+        HddsServerUtil.getScmSecurityClient(config);
+
+    SCMGetCertResponseProto response = secureScmClient.
+        getSCMCertChain(scmNodeDetailsProto, getEncodedString(csr));
+    String pemEncodedCert = response.getX509Certificate();
+
+    try {
+
+      // Store SCM CA certificate.
+      if (response.hasX509CACertificate()) {
+        String pemEncodedRootCert = response.getX509CACertificate();
+        client.storeCertificate(pemEncodedRootCert, true, true);
+        client.storeCertificate(pemEncodedCert, true);
+
+
+        // Write to the location, so that Default CA server can read the
+        // certificate generated by cert client.
+        CertificateCodec certCodec =
+            new CertificateCodec(new SecurityConfig(config),
+                client.getComponentName());
+
+        X509Certificate certificate =
+            CertificateCodec.getX509Certificate(pemEncodedCert);
+        certCodec.writeCertificate(certCodec.getCertificateHolder(certificate));
+
+        // Persist scm cert serial ID.
+        scmStorageConfig.setScmCertSerialId(certificate.getSerialNumber()
+            .toString());
+      } else {
+        throw new RuntimeException("Unable to retrieve SCM certificate chain");
+      }
+    } catch (IOException | CertificateException e) {
+      LOG.error("Error while storing SCM signed certificate.", e);
+      throw new RuntimeException(e);
+    }
+  }
+
+  private static void getPrimarySCMSignedCert(CertificateClient client,
+      OzoneConfiguration config, String fetchedSCMId,
+      SCMStorageConfig scmStorageConfig, InetSocketAddress scmAddress)
+      throws IOException {
+
+    PKCS10CertificationRequest csr = generateCSR(client, scmStorageConfig,
+        config, scmAddress, fetchedSCMId);
+
+    CertificateServer rootCAServer =
+        initializeRootCertificateServer(scmStorageConfig.getClusterID(),
+            scmStorageConfig.getScmId(), null);
+
+    rootCAServer.init(new SecurityConfig(config),
+        CertificateServer.CAType.SELF_SIGNED_CA);
+
+    try {
+      X509CertificateHolder certificateHolder = rootCAServer.
+          requestCertificate(csr, KERBEROS_TRUSTED, SCM).get();
+
+      X509CertificateHolder rootCAServerCACertificateHolder =
+          rootCAServer.getCACertificate();
+
+      String pemEncodedCert =
+          CertificateCodec.getPEMEncodedString(certificateHolder);
+
+      String pemEncodedRootCert =
+          CertificateCodec.getPEMEncodedString(rootCAServerCACertificateHolder);
+
+
+      client.storeCertificate(pemEncodedRootCert, true, true);
+      client.storeCertificate(pemEncodedCert, true);
+
+      // Write to the location, so that Default CA server can read the
+      // certificate generated by cert client.
+      CertificateCodec certCodec =
+          new CertificateCodec(new SecurityConfig(config),
+              client.getComponentName());
+
+      certCodec.writeCertificate(certificateHolder);
+
+        // Persist scm cert serial ID.
+      scmStorageConfig.setScmCertSerialId(certificateHolder.getSerialNumber()

Review comment:
       Should we persist this to version file as root CA certID, which is different from normal SCM certid.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r600354608



##########
File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ha/SCMNodeInfo.java
##########
@@ -246,4 +247,20 @@ public String getScmSecurityAddress() {
   public String getScmDatanodeAddress() {
     return scmDatanodeAddress;
   }
+
+  public static SCMNodeInfo getScmNodeInfo(ConfigurationSource conf,
+      String nodeId) {
+    List<SCMNodeInfo> scmNodeInfoList = SCMNodeInfo.buildNodeInfo(conf);
+    Preconditions.checkState(scmNodeInfoList.size() >=1);
+    if (SCMHAUtils.getScmServiceId(conf) != null) {
+      for (SCMNodeInfo scmNodeInfo : scmNodeInfoList) {
+        if (scmNodeInfo.getNodeId().equals(nodeId)) {
+          return scmNodeInfo;
+        }
+      }

Review comment:
       Removed this method, as it is not used anywhere




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] xiaoyuyao commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r604416469



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
##########
@@ -793,7 +880,18 @@ public static boolean scmBootstrap(OzoneConfiguration conf)
         // SCM Node info containing hostname to scm Id mappings
         // will be persisted into the version file once this node gets added
         // to existing SCM ring post node regular start up.
+
+        if(OzoneSecurityUtil.isSecurityEnabled(conf)) {
+          HASecurityUtils.initializeSecurity(scmStorageConfig,
+              scmInfo.getScmId(), config, getScmAddress(scmhaNodeDetails, conf),
+              false);
+        }
+        scmStorageConfig.setPrimaryScmNodeId(scmInfo.getScmId());

Review comment:
       Ok, sounds good to me. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r603013388



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/HASecurityUtils.java
##########
@@ -281,4 +284,26 @@ private static void persistSubCACertificate(OzoneConfiguration config,
     certCodec.writeCertificate(certificateHolder);
   }
 
+  /**
+   * Create Server TLS parameters required for Ratis Server.
+   * @param conf
+   * @param caClient
+   * @return
+   */
+  public static Parameters createServerTlsParameters(SecurityConfig conf,
+      CertificateClient caClient) {
+    Parameters parameters = new Parameters();
+
+    if (conf.isSecurityEnabled() && conf.isGrpcTlsEnabled()) {
+      GrpcTlsConfig config = new GrpcTlsConfig(
+          caClient.getPrivateKey(), caClient.getCertificate(),
+          caClient.getCACertificate(), true);

Review comment:
       This method is used by SCM, for SCM getCACert returns rootCA, as for SCM it got signed cert from rootCA.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r604648797



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
##########
@@ -558,44 +577,99 @@ private void initializeSystemManagers(OzoneConfiguration conf,
    */
   private void initializeCAnSecurityProtocol(OzoneConfiguration conf,
       SCMConfigurator configurator) throws IOException {
-    if(configurator.getCertificateServer() != null) {
-      this.certificateServer = configurator.getCertificateServer();
+
+    // TODO: Support Certificate Server loading via Class Name loader.
+    // So it is easy to use different Certificate Servers if needed.
+    if(this.scmMetadataStore == null) {
+      LOG.error("Cannot initialize Certificate Server without a valid meta " +
+          "data layer.");
+      throw new SCMException("Cannot initialize CA without a valid metadata " +
+          "store", ResultCodes.SCM_NOT_INITIALIZED);
+    }
+
+    certificateStore =
+        new SCMCertStore.Builder().setMetadaStore(scmMetadataStore)
+            .setRatisServer(scmHAManager.getRatisServer())
+            .setCRLSequenceId(getLastSequenceIdForCRL()).build();
+
+
+    // If primary SCM node Id is set it means this is a cluster which has
+    // performed init with SCM HA version code.
+    if (scmStorageConfig.checkPrimarySCMIdInitialized()) {
+      // Start specific instance SCM CA server.
+      String subject = SCM_SUB_CA_PREFIX +
+          InetAddress.getLocalHost().getHostName();
+      if (configurator.getCertificateServer() != null) {
+        this.scmCertificateServer = configurator.getCertificateServer();
+      } else {
+        scmCertificateServer = new DefaultCAServer(subject,
+            scmStorageConfig.getClusterID(), scmStorageConfig.getScmId(),
+            certificateStore, new DefaultProfile(),
+            scmCertificateClient.getComponentName());
+        // INTERMEDIARY_CA which issues certs to DN and OM.
+        scmCertificateServer.init(new SecurityConfig(configuration),
+            CertificateServer.CAType.INTERMEDIARY_CA);
+      }
+
+      if (primaryScmNodeId.equals(scmStorageConfig.getScmId())) {
+        if (configurator.getCertificateServer() != null) {
+          this.rootCertificateServer = configurator.getCertificateServer();
+        } else {
+          rootCertificateServer =
+              HASecurityUtils.initializeRootCertificateServer(
+              conf, certificateStore, scmStorageConfig);
+        }
+
+        BigInteger certSerial =
+            scmCertificateClient.getCertificate().getSerialNumber();
+        // Store the certificate in DB. On primary SCM when init happens, the
+        // certificate is not persisted to DB. As we don't have Metadatstore
+        // and ratis server initialized with statemachine. We need to do only
+        // for primary scm, for other bootstrapped scm's certificates will be
+        // persisted via ratis.
+        if (certificateStore.getCertificateByID(certSerial,
+            VALID_CERTS) == null) {
+          LOG.info("Storing certSerial {}", certSerial);

Review comment:
       done, Let me know if that looks okay to you?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r603136179



##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerRatisServer.java
##########
@@ -651,9 +652,14 @@ private static Parameters createServerTlsParameters(SecurityConfig conf,
     Parameters parameters = new Parameters();
 
     if (conf.isSecurityEnabled() && conf.isGrpcTlsEnabled()) {
+      ArrayList< X509Certificate > listCA = new ArrayList<>();
+      listCA.add(caClient.getCACertificate());

Review comment:
       Done. Updated in both OM/DN. Also tested when OM1 got cert from SCM1(leader), OM2,OM3 got cert from new leader(SCM2/SCM3) OM is able to form a ratis quorum, and Key create is successful.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r604648529



##########
File path: hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/HAUtils.java
##########
@@ -326,4 +351,139 @@ public static void checkSecurityAndSCMHAEnabled(OzoneConfiguration conf) {
       }
     }
   }
+
+  /**
+   * Build CA list which need to be passed to client.
+   *
+   * If certificate client is null, obtain the list of CA using SCM security
+   * client, else it uses certificate client.
+   * @param certClient
+   * @param configuration
+   * @return list of CA
+   * @throws IOException
+   */
+  public static List<String> buildCAList(CertificateClient certClient,
+      ConfigurationSource configuration) throws IOException {
+    //TODO: make it configurable.
+    long waitTime = 5 * 60 * 1000L;
+    long retryTime = 10 * 1000L;
+    long currentTime = Time.monotonicNow();
+    List<String> caCertPemList = null;
+    if (certClient != null) {
+      caCertPemList = new ArrayList<>();
+      if (!SCMHAUtils.isSCMHAEnabled(configuration)) {
+        if (certClient.getRootCACertificate() != null) {
+          caCertPemList.add(CertificateCodec.getPEMEncodedString(
+              certClient.getRootCACertificate()));
+        }
+        caCertPemList.add(CertificateCodec.getPEMEncodedString(
+            certClient.getCACertificate()));
+      } else {
+        // TODO: If SCMs are bootstrapped later, then listCA need to be
+        //  refetched if listCA size is less than scm ha config node list size.
+        // For now when Client of SCM's are started we compare their node list
+        // size and ca list size if it is as expected, we return the ca list.
+        boolean caListUpToDate;
+        Collection<String> scmNodes = SCMHAUtils.getSCMNodeIds(configuration);
+        // TODO: make them configurable.
+        if (scmNodes.size() > 1) {
+          do {
+            caCertPemList = certClient.updateCAList();
+            caListUpToDate =
+                caCertPemList.size() == scmNodes.size() + 1 ? true : false;
+            if (!caListUpToDate) {
+              try {
+                Thread.sleep(retryTime);
+              } catch (InterruptedException ex) {
+                Thread.currentThread().interrupt();
+              }
+            }
+          } while (!caListUpToDate &&
+              Time.monotonicNow() - currentTime < waitTime);
+          checkCertCount(caCertPemList.size(), scmNodes.size() + 1);
+        } else {
+          caCertPemList = certClient.updateCAList();
+        }
+      }
+    } else {
+      if (!SCMHAUtils.isSCMHAEnabled(configuration)) {
+        caCertPemList = new ArrayList<>();
+        SCMSecurityProtocolClientSideTranslatorPB scmSecurityProtocolClient =

Review comment:
       Done.

##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMSecurityProtocolServer.java
##########
@@ -288,22 +306,18 @@ public String getCACertificate() throws IOException {
   public List<String> listCACertificate() throws IOException {
     List<String> caCerts =
         listCertificate(NodeType.SCM, 0, 10, false);
-    caCerts.add(getRootCACertificate());
     return caCerts;
   }
 
   @Override
   public String getRootCACertificate() throws IOException {
     LOGGER.debug("Getting Root CA certificate.");
-    //TODO: This code will be modified after HDDS-4897 is merged and
-    // integrated. For now getting RootCA cert from certificateServer.
-    try {
+    if (storageContainerManager.getScmStorageConfig()

Review comment:
       Done

##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
##########
@@ -558,44 +577,99 @@ private void initializeSystemManagers(OzoneConfiguration conf,
    */
   private void initializeCAnSecurityProtocol(OzoneConfiguration conf,
       SCMConfigurator configurator) throws IOException {
-    if(configurator.getCertificateServer() != null) {
-      this.certificateServer = configurator.getCertificateServer();
+
+    // TODO: Support Certificate Server loading via Class Name loader.
+    // So it is easy to use different Certificate Servers if needed.
+    if(this.scmMetadataStore == null) {
+      LOG.error("Cannot initialize Certificate Server without a valid meta " +
+          "data layer.");
+      throw new SCMException("Cannot initialize CA without a valid metadata " +
+          "store", ResultCodes.SCM_NOT_INITIALIZED);
+    }
+
+    certificateStore =
+        new SCMCertStore.Builder().setMetadaStore(scmMetadataStore)
+            .setRatisServer(scmHAManager.getRatisServer())
+            .setCRLSequenceId(getLastSequenceIdForCRL()).build();
+
+
+    // If primary SCM node Id is set it means this is a cluster which has
+    // performed init with SCM HA version code.
+    if (scmStorageConfig.checkPrimarySCMIdInitialized()) {
+      // Start specific instance SCM CA server.
+      String subject = SCM_SUB_CA_PREFIX +
+          InetAddress.getLocalHost().getHostName();
+      if (configurator.getCertificateServer() != null) {
+        this.scmCertificateServer = configurator.getCertificateServer();
+      } else {
+        scmCertificateServer = new DefaultCAServer(subject,
+            scmStorageConfig.getClusterID(), scmStorageConfig.getScmId(),
+            certificateStore, new DefaultProfile(),
+            scmCertificateClient.getComponentName());
+        // INTERMEDIARY_CA which issues certs to DN and OM.
+        scmCertificateServer.init(new SecurityConfig(configuration),
+            CertificateServer.CAType.INTERMEDIARY_CA);
+      }
+
+      if (primaryScmNodeId.equals(scmStorageConfig.getScmId())) {
+        if (configurator.getCertificateServer() != null) {
+          this.rootCertificateServer = configurator.getCertificateServer();
+        } else {
+          rootCertificateServer =
+              HASecurityUtils.initializeRootCertificateServer(
+              conf, certificateStore, scmStorageConfig);
+        }
+
+        BigInteger certSerial =

Review comment:
       Done




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] xiaoyuyao commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r595395838



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/HASecurityUtils.java
##########
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership.  The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.hdds.scm.ha;
+
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.protocol.proto.SCMSecurityProtocolProtos.SCMGetCertResponseProto;
+import org.apache.hadoop.hdds.protocolPB.SCMSecurityProtocolClientSideTranslatorPB;
+import org.apache.hadoop.hdds.scm.server.SCMCertStore;
+import org.apache.hadoop.hdds.scm.server.SCMStorageConfig;
+import org.apache.hadoop.hdds.security.x509.SecurityConfig;
+import org.apache.hadoop.hdds.security.x509.certificate.authority.CertificateServer;
+import org.apache.hadoop.hdds.security.x509.certificate.authority.DefaultCAServer;
+import org.apache.hadoop.hdds.security.x509.certificate.authority.PKIProfiles.DefaultCAProfile;
+import org.apache.hadoop.hdds.security.x509.certificate.client.CertificateClient;
+import org.apache.hadoop.hdds.security.x509.certificate.client.SCMCertificateClient;
+import org.apache.hadoop.hdds.security.x509.certificate.utils.CertificateCodec;
+import org.apache.hadoop.hdds.security.x509.certificates.utils.CertificateSignRequest;
+import org.apache.hadoop.hdds.utils.HddsServerUtil;
+import org.apache.ratis.conf.Parameters;
+import org.apache.ratis.grpc.GrpcConfigKeys;
+import org.apache.ratis.grpc.GrpcTlsConfig;
+import org.bouncycastle.cert.X509CertificateHolder;
+import org.bouncycastle.pkcs.PKCS10CertificationRequest;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.net.InetAddress;
+import java.net.InetSocketAddress;
+import java.nio.file.Paths;
+import java.security.KeyPair;
+import java.security.cert.CertificateException;
+import java.security.cert.X509Certificate;
+import java.util.concurrent.ExecutionException;
+
+import static org.apache.hadoop.hdds.protocol.proto.HddsProtos.NodeType.SCM;
+import static org.apache.hadoop.hdds.security.x509.certificate.authority.CertificateApprover.ApprovalType.KERBEROS_TRUSTED;
+import static org.apache.hadoop.hdds.security.x509.certificates.utils.CertificateSignRequest.getEncodedString;
+import static org.apache.hadoop.ozone.OzoneConsts.SCM_CA_CERT_STORAGE_DIR;
+import static org.apache.hadoop.ozone.OzoneConsts.SCM_CA_PATH;
+
+public final class HASecurityUtils {
+
+  private HASecurityUtils() {
+  }
+
+  public static final Logger LOG =
+      LoggerFactory.getLogger(HASecurityUtils.class);
+
+  /**
+   * Initialize Security which generates public, private key pair and get SCM
+   * signed certificate and persist to local disk.
+   * @param scmStorageConfig
+   * @param fetchedScmId
+   * @param conf
+   * @param scmAddress
+   * @throws IOException
+   */
+  public static void initializeSecurity(SCMStorageConfig scmStorageConfig,
+      String fetchedScmId, OzoneConfiguration conf,
+      InetSocketAddress scmAddress, boolean primaryscm)
+      throws IOException {
+    LOG.info("Initializing secure StorageContainerManager.");
+
+    CertificateClient certClient =
+        new SCMCertificateClient(new SecurityConfig(conf));
+    CertificateClient.InitResponse response = certClient.init();
+    LOG.info("Init response: {}", response);
+    switch (response) {
+    case SUCCESS:
+      LOG.info("Initialization successful.");
+      break;
+    case GETCERT:
+      if (!primaryscm) {
+        getSCMSignedCert(certClient, conf, fetchedScmId, scmStorageConfig,
+            scmAddress);
+      } else {
+        getPrimarySCMSignedCert(certClient, conf, fetchedScmId,
+            scmStorageConfig, scmAddress);
+      }
+      LOG.info("Successfully stored SCM signed certificate.");
+      break;
+    case FAILURE:
+      LOG.error("SCM security initialization failed.");
+      throw new RuntimeException("OM security initialization failed.");
+    case RECOVER:
+      LOG.error("SCM security initialization failed. SCM certificate is " +
+          "missing.");
+      throw new RuntimeException("SCM security initialization failed.");
+    default:
+      LOG.error("SCM security initialization failed. Init response: {}",
+          response);
+      throw new RuntimeException("SCM security initialization failed.");
+    }
+  }
+
+  /**
+   * Get SCM signed certificate and store it using certificate client.
+   */
+  private static void getSCMSignedCert(CertificateClient client,

Review comment:
       NIT: can we rename this to getRootCASignedSCMCert()?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r603019860



##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerRatisServer.java
##########
@@ -651,9 +652,14 @@ private static Parameters createServerTlsParameters(SecurityConfig conf,
     Parameters parameters = new Parameters();
 
     if (conf.isSecurityEnabled() && conf.isGrpcTlsEnabled()) {
+      ArrayList< X509Certificate > listCA = new ArrayList<>();
+      listCA.add(caClient.getCACertificate());

Review comment:
       Question: why do we need to complete the sub SCM ca list here, as this is here it is for MTLS?
   
   Right now for trust store passing both RootCA and subSCM CA cert.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r603005080



##########
File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/OzoneContainer.java
##########
@@ -167,8 +171,23 @@ public OzoneContainer(
     blockDeletingService =
         new BlockDeletingService(this, svcInterval.toMillis(), serviceTimeout,
             TimeUnit.MILLISECONDS, config);
-    tlsClientConfig = RatisHelper.createTlsClientConfig(
-        secConf, certClient != null ? certClient.getCACertificate() : null);
+
+    List< X509Certificate > x509Certificates = null;
+    if (certClient != null) {
+      List<String> pemEncodedCerts = HAUtils.buildCAList(certClient, conf);
+      x509Certificates = new ArrayList<>(pemEncodedCerts.size());

Review comment:
       Done




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#issuecomment-797858821


   > @bharatviswa504 thanks for working on this. Can you separate the patch into smaller ones? For example
   > separate the bootstrap of the non-primary from this patch and focus only on the init of primary SCM.
   > Otherwise, the review will take longer time.
   
   @xiaoyuyao I understand it is a big change, sorry for the trouble. Bootstrap changes will be less than 100 LOC, but if it is still needed I can split it into a new Jira.  
   
   Actually, I have planned to implement CertClient and Integration in two parts, later when integrating found issues and closed the other one. Let me see if I can split again, but 2nd PR integration one will be dependent on this. Can we also have a call for a review?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 edited a comment on pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 edited a comment on pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#issuecomment-800999900


   @xiaoyuyao Addressed/replied review comments these are addressed in HDDS-4897 under PR #2041 
   Can we continue the review in #2041 
   
   I will update this PR to use all the Jira's which went in and integrate it.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r601975030



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMStorageConfig.java
##########
@@ -58,6 +60,28 @@ public void setScmId(String scmId) throws IOException {
     }
   }
 
+  public void setScmCertSerialId(String certSerialId) throws IOException {

Review comment:
       Currently, primary cert serialid is "1". (It is not generated value)
   Let me know if you see any value in adding this to the version file, I can do it.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r603016962



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
##########
@@ -400,6 +413,14 @@ private StorageContainerManager(OzoneConfiguration conf,
     registerMetricsSource(this);
   }
 
+  private void initializeCertificateClient() {
+    if (primaryScmNodeId != null) {

Review comment:
       We should initialize CertificateClient on all SCM's.
   So had this check primaryScmNodeId != null, as we don't want to initializeCertClient on upgraded SCM where init has not happened.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r595720314



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/HASecurityUtils.java
##########
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership.  The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.hdds.scm.ha;
+
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.protocol.proto.SCMSecurityProtocolProtos.SCMGetCertResponseProto;
+import org.apache.hadoop.hdds.protocolPB.SCMSecurityProtocolClientSideTranslatorPB;
+import org.apache.hadoop.hdds.scm.server.SCMCertStore;
+import org.apache.hadoop.hdds.scm.server.SCMStorageConfig;
+import org.apache.hadoop.hdds.security.x509.SecurityConfig;
+import org.apache.hadoop.hdds.security.x509.certificate.authority.CertificateServer;
+import org.apache.hadoop.hdds.security.x509.certificate.authority.DefaultCAServer;
+import org.apache.hadoop.hdds.security.x509.certificate.authority.PKIProfiles.DefaultCAProfile;
+import org.apache.hadoop.hdds.security.x509.certificate.client.CertificateClient;
+import org.apache.hadoop.hdds.security.x509.certificate.client.SCMCertificateClient;
+import org.apache.hadoop.hdds.security.x509.certificate.utils.CertificateCodec;
+import org.apache.hadoop.hdds.security.x509.certificates.utils.CertificateSignRequest;
+import org.apache.hadoop.hdds.utils.HddsServerUtil;
+import org.apache.ratis.conf.Parameters;
+import org.apache.ratis.grpc.GrpcConfigKeys;
+import org.apache.ratis.grpc.GrpcTlsConfig;
+import org.bouncycastle.cert.X509CertificateHolder;
+import org.bouncycastle.pkcs.PKCS10CertificationRequest;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.net.InetAddress;
+import java.net.InetSocketAddress;
+import java.nio.file.Paths;
+import java.security.KeyPair;
+import java.security.cert.CertificateException;
+import java.security.cert.X509Certificate;
+import java.util.concurrent.ExecutionException;
+
+import static org.apache.hadoop.hdds.protocol.proto.HddsProtos.NodeType.SCM;
+import static org.apache.hadoop.hdds.security.x509.certificate.authority.CertificateApprover.ApprovalType.KERBEROS_TRUSTED;
+import static org.apache.hadoop.hdds.security.x509.certificates.utils.CertificateSignRequest.getEncodedString;
+import static org.apache.hadoop.ozone.OzoneConsts.SCM_CA_CERT_STORAGE_DIR;
+import static org.apache.hadoop.ozone.OzoneConsts.SCM_CA_PATH;
+
+public final class HASecurityUtils {
+
+  private HASecurityUtils() {
+  }
+
+  public static final Logger LOG =
+      LoggerFactory.getLogger(HASecurityUtils.class);
+
+  /**
+   * Initialize Security which generates public, private key pair and get SCM
+   * signed certificate and persist to local disk.
+   * @param scmStorageConfig
+   * @param fetchedScmId
+   * @param conf
+   * @param scmAddress
+   * @throws IOException
+   */
+  public static void initializeSecurity(SCMStorageConfig scmStorageConfig,
+      String fetchedScmId, OzoneConfiguration conf,
+      InetSocketAddress scmAddress, boolean primaryscm)
+      throws IOException {
+    LOG.info("Initializing secure StorageContainerManager.");
+
+    CertificateClient certClient =
+        new SCMCertificateClient(new SecurityConfig(conf));
+    CertificateClient.InitResponse response = certClient.init();
+    LOG.info("Init response: {}", response);
+    switch (response) {
+    case SUCCESS:
+      LOG.info("Initialization successful.");
+      break;
+    case GETCERT:
+      if (!primaryscm) {
+        getSCMSignedCert(certClient, conf, fetchedScmId, scmStorageConfig,
+            scmAddress);
+      } else {
+        getPrimarySCMSignedCert(certClient, conf, fetchedScmId,
+            scmStorageConfig, scmAddress);
+      }
+      LOG.info("Successfully stored SCM signed certificate.");
+      break;
+    case FAILURE:
+      LOG.error("SCM security initialization failed.");
+      throw new RuntimeException("OM security initialization failed.");
+    case RECOVER:
+      LOG.error("SCM security initialization failed. SCM certificate is " +
+          "missing.");
+      throw new RuntimeException("SCM security initialization failed.");
+    default:
+      LOG.error("SCM security initialization failed. Init response: {}",
+          response);
+      throw new RuntimeException("SCM security initialization failed.");
+    }
+  }
+
+  /**
+   * Get SCM signed certificate and store it using certificate client.
+   */
+  private static void getSCMSignedCert(CertificateClient client,
+      OzoneConfiguration config, String fetchedSCMId,
+      SCMStorageConfig scmStorageConfig, InetSocketAddress scmAddress)
+      throws IOException {
+
+    PKCS10CertificationRequest csr = generateCSR(client, scmStorageConfig,
+        config, scmAddress, fetchedSCMId);
+
+    HddsProtos.ScmNodeDetailsProto scmNodeDetailsProto =
+        HddsProtos.ScmNodeDetailsProto.newBuilder()
+            .setClusterId(scmStorageConfig.getClusterID())
+            .setHostName(scmAddress.getHostName())
+            .setScmNodeId(fetchedSCMId).build();
+
+    SCMSecurityProtocolClientSideTranslatorPB secureScmClient =
+        HddsServerUtil.getScmSecurityClient(config);
+
+    SCMGetCertResponseProto response = secureScmClient.
+        getSCMCertChain(scmNodeDetailsProto, getEncodedString(csr));
+    String pemEncodedCert = response.getX509Certificate();
+
+    try {
+
+      // Store SCM CA certificate.
+      if (response.hasX509CACertificate()) {
+        String pemEncodedRootCert = response.getX509CACertificate();
+        client.storeCertificate(pemEncodedRootCert, true, true);
+        client.storeCertificate(pemEncodedCert, true);
+
+
+        // Write to the location, so that Default CA server can read the
+        // certificate generated by cert client.
+        CertificateCodec certCodec =
+            new CertificateCodec(new SecurityConfig(config),
+                client.getComponentName());
+
+        X509Certificate certificate =
+            CertificateCodec.getX509Certificate(pemEncodedCert);
+        certCodec.writeCertificate(certCodec.getCertificateHolder(certificate));
+
+        // Persist scm cert serial ID.
+        scmStorageConfig.setScmCertSerialId(certificate.getSerialNumber()

Review comment:
       On all SCM's only persisting sub SCM cert SerialID to version file. This is to find out after a restart, what is the Certificate SerialID and so that getCertificate() will work. In all SCMS (primary/bootstrap) followed same approach
   
   So in SCMCertClient
   
   certSerialID -> Certificate got from root CA
   caCertId -> Root CA Certificate 
   
   Not persisted rootCA cert again, as for SCM caCertId and rootCaCertID are same.
   
   Whereas for other roles like OM/DN
   
   certSerialID -> Certificate got from sub SCM CA(any SCM which is leader at that point)
   caCertId -> Sub SCM CA Certificate ID
   rootCaCertId -> root SCM CA Certificate ID




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r603018275



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
##########
@@ -558,44 +579,105 @@ private void initializeSystemManagers(OzoneConfiguration conf,
    */
   private void initializeCAnSecurityProtocol(OzoneConfiguration conf,
       SCMConfigurator configurator) throws IOException {
-    if(configurator.getCertificateServer() != null) {
-      this.certificateServer = configurator.getCertificateServer();
+
+    // TODO: Support Certificate Server loading via Class Name loader.
+    // So it is easy to use different Certificate Servers if needed.
+    if(this.scmMetadataStore == null) {
+      LOG.error("Cannot initialize Certificate Server without a valid meta " +
+          "data layer.");
+      throw new SCMException("Cannot initialize CA without a valid metadata " +
+          "store", ResultCodes.SCM_NOT_INITIALIZED);
+    }
+
+    certificateStore =
+        new SCMCertStore.Builder().setMetadaStore(scmMetadataStore)
+            .setRatisServer(scmHAManager.getRatisServer())
+            .setCRLSequenceId(getLastSequenceIdForCRL()).build();
+
+
+    // If primary SCM node Id is set it means this is a cluster which has
+    // performed init with SCM HA version code.
+    if (primaryScmNodeId != null) {

Review comment:
       Done Moved this to ScmStorageconfig not used a state, because we have already initialized state whether SCM has version fille or not. But for primaryScmNodeID there are 2 scenarios on upgraded cluster where init will not happen, it will not have primaryScmNodeIID set, on freshly installed clusters where init has happened it will be set.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#issuecomment-809093861


   Thank You @xiaoyuyao for the review. Addressed/replied review comments.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] xiaoyuyao commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r592726011



##########
File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/OzoneContainer.java
##########
@@ -167,8 +168,16 @@ public OzoneContainer(
     blockDeletingService =
         new BlockDeletingService(this, svcInterval.toMillis(), serviceTimeout,
             TimeUnit.MILLISECONDS, config);
+
+    List<X509Certificate> x509Certificates = new ArrayList<>();
+    if (certClient != null) {
+      x509Certificates.add(certClient.getCACertificate());
+      if (certClient.getRootCACertificate() != null) {

Review comment:
       we should avoid making the same RPC call   getRootCACertificate twice. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r600382827



##########
File path: hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/client/DefaultCertificateClient.java
##########
@@ -818,29 +830,64 @@ public Logger getLogger() {
     return logger;
   }
 
-  /**
-   * Create a scm security client, used to get SCM signed certificate.
-   *
-   * @return {@link SCMSecurityProtocol}
-   */
-  private static SCMSecurityProtocol getScmSecurityClient(
-      OzoneConfiguration conf) throws IOException {
-    RPC.setProtocolEngine(conf, SCMSecurityProtocolPB.class,
-        ProtobufRpcEngine.class);
-    long scmVersion =
-        RPC.getProtocolVersion(ScmBlockLocationProtocolPB.class);
-    InetSocketAddress scmSecurityProtoAdd =
-        HddsServerUtil.getScmAddressForSecurityProtocol(conf);
-    SCMSecurityProtocolClientSideTranslatorPB scmSecurityClient =
-        new SCMSecurityProtocolClientSideTranslatorPB(
-            RPC.getProxy(SCMSecurityProtocolPB.class, scmVersion,
-                scmSecurityProtoAdd, UserGroupInformation.getCurrentUser(),
-                conf, NetUtils.getDefaultSocketFactory(conf),
-                Client.getRpcTimeout(conf)));
-    return scmSecurityClient;
+  public String getComponentName() {
+    return null;
   }
 
-  public String getComponentName() {
+  @Override
+  public X509Certificate getRootCACertificate() {
+    if (rootCaCertId != null) {
+      return certificateMap.get(rootCaCertId);
+    }
     return null;
   }
+
+  @Override
+  public void storeRootCACertificate(String pemEncodedCert, boolean force)
+      throws CertificateException {
+    CertificateCodec certificateCodec = new CertificateCodec(securityConfig,
+        component);
+    try {
+      Path basePath = securityConfig.getCertificateLocation(component);
+
+      X509Certificate cert =
+          CertificateCodec.getX509Certificate(pemEncodedCert);
+      String certName = String.format(CERT_FILE_NAME_FORMAT,
+          cert.getSerialNumber().toString());
+
+      certName = ROOT_CA_CERT_PREFIX + certName;
+      rootCaCertId = cert.getSerialNumber().toString();
+
+      certificateCodec.writeCertificate(basePath, certName,
+          pemEncodedCert, force);
+      certificateMap.putIfAbsent(cert.getSerialNumber().toString(), cert);
+    } catch (IOException | java.security.cert.CertificateException e) {
+      throw new CertificateException("Error while storing Root CA " +
+          "certificate.", e, CERTIFICATE_ERROR);
+    }
+  }
+
+  @Override
+  public List<String> listCA() throws IOException {
+    if (pemEncodedCACerts == null) {
+      updateCAList();

Review comment:
       Done




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 edited a comment on pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 edited a comment on pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#issuecomment-810834926






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r601974239



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
##########
@@ -793,7 +880,18 @@ public static boolean scmBootstrap(OzoneConfiguration conf)
         // SCM Node info containing hostname to scm Id mappings
         // will be persisted into the version file once this node gets added
         // to existing SCM ring post node regular start up.
+
+        if(OzoneSecurityUtil.isSecurityEnabled(conf)) {
+          HASecurityUtils.initializeSecurity(scmStorageConfig,
+              scmInfo.getScmId(), config, getScmAddress(scmhaNodeDetails, conf),
+              false);
+        }
+        scmStorageConfig.setPrimaryScmNodeId(scmInfo.getScmId());

Review comment:
       Yes that is the plan as discussed, will do that change in HDDS-5027




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r595921581



##########
File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
##########
@@ -416,4 +418,7 @@ private OzoneConsts() {
 
   // CRL Sequence Id
   public static final String CRL_SEQUENCE_ID_KEY = "CRL_SEQUENCE_ID";
+
+  public static final String SCM_CA_PATH = "ca";
+  public static final String SCM_CA_CERT_STORAGE_DIR = "scm";

Review comment:
       SCM Root CA location is 
   {metadir}/scm/ca/keys
   {metadir}/scm/ca/certs
   
   And SCM Sub CA location is
    {metadir}/scm/sub-ca/keys 
    {metadir}/scm/sub-ca/certs 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r595717104



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/HASecurityUtils.java
##########
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership.  The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.hdds.scm.ha;
+
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.protocol.proto.SCMSecurityProtocolProtos.SCMGetCertResponseProto;
+import org.apache.hadoop.hdds.protocolPB.SCMSecurityProtocolClientSideTranslatorPB;
+import org.apache.hadoop.hdds.scm.server.SCMCertStore;
+import org.apache.hadoop.hdds.scm.server.SCMStorageConfig;
+import org.apache.hadoop.hdds.security.x509.SecurityConfig;
+import org.apache.hadoop.hdds.security.x509.certificate.authority.CertificateServer;
+import org.apache.hadoop.hdds.security.x509.certificate.authority.DefaultCAServer;
+import org.apache.hadoop.hdds.security.x509.certificate.authority.PKIProfiles.DefaultCAProfile;
+import org.apache.hadoop.hdds.security.x509.certificate.client.CertificateClient;
+import org.apache.hadoop.hdds.security.x509.certificate.client.SCMCertificateClient;
+import org.apache.hadoop.hdds.security.x509.certificate.utils.CertificateCodec;
+import org.apache.hadoop.hdds.security.x509.certificates.utils.CertificateSignRequest;
+import org.apache.hadoop.hdds.utils.HddsServerUtil;
+import org.apache.ratis.conf.Parameters;
+import org.apache.ratis.grpc.GrpcConfigKeys;
+import org.apache.ratis.grpc.GrpcTlsConfig;
+import org.bouncycastle.cert.X509CertificateHolder;
+import org.bouncycastle.pkcs.PKCS10CertificationRequest;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.net.InetAddress;
+import java.net.InetSocketAddress;
+import java.nio.file.Paths;
+import java.security.KeyPair;
+import java.security.cert.CertificateException;
+import java.security.cert.X509Certificate;
+import java.util.concurrent.ExecutionException;
+
+import static org.apache.hadoop.hdds.protocol.proto.HddsProtos.NodeType.SCM;
+import static org.apache.hadoop.hdds.security.x509.certificate.authority.CertificateApprover.ApprovalType.KERBEROS_TRUSTED;
+import static org.apache.hadoop.hdds.security.x509.certificates.utils.CertificateSignRequest.getEncodedString;
+import static org.apache.hadoop.ozone.OzoneConsts.SCM_CA_CERT_STORAGE_DIR;
+import static org.apache.hadoop.ozone.OzoneConsts.SCM_CA_PATH;
+
+public final class HASecurityUtils {
+
+  private HASecurityUtils() {
+  }
+
+  public static final Logger LOG =
+      LoggerFactory.getLogger(HASecurityUtils.class);
+
+  /**
+   * Initialize Security which generates public, private key pair and get SCM
+   * signed certificate and persist to local disk.
+   * @param scmStorageConfig
+   * @param fetchedScmId
+   * @param conf
+   * @param scmAddress
+   * @throws IOException
+   */
+  public static void initializeSecurity(SCMStorageConfig scmStorageConfig,
+      String fetchedScmId, OzoneConfiguration conf,
+      InetSocketAddress scmAddress, boolean primaryscm)
+      throws IOException {
+    LOG.info("Initializing secure StorageContainerManager.");
+
+    CertificateClient certClient =
+        new SCMCertificateClient(new SecurityConfig(conf));
+    CertificateClient.InitResponse response = certClient.init();
+    LOG.info("Init response: {}", response);
+    switch (response) {
+    case SUCCESS:
+      LOG.info("Initialization successful.");
+      break;
+    case GETCERT:
+      if (!primaryscm) {
+        getSCMSignedCert(certClient, conf, fetchedScmId, scmStorageConfig,
+            scmAddress);
+      } else {
+        getPrimarySCMSignedCert(certClient, conf, fetchedScmId,
+            scmStorageConfig, scmAddress);
+      }
+      LOG.info("Successfully stored SCM signed certificate.");
+      break;
+    case FAILURE:
+      LOG.error("SCM security initialization failed.");
+      throw new RuntimeException("OM security initialization failed.");
+    case RECOVER:
+      LOG.error("SCM security initialization failed. SCM certificate is " +
+          "missing.");
+      throw new RuntimeException("SCM security initialization failed.");
+    default:
+      LOG.error("SCM security initialization failed. Init response: {}",
+          response);
+      throw new RuntimeException("SCM security initialization failed.");
+    }
+  }
+
+  /**
+   * Get SCM signed certificate and store it using certificate client.
+   */
+  private static void getSCMSignedCert(CertificateClient client,
+      OzoneConfiguration config, String fetchedSCMId,
+      SCMStorageConfig scmStorageConfig, InetSocketAddress scmAddress)
+      throws IOException {
+
+    PKCS10CertificationRequest csr = generateCSR(client, scmStorageConfig,
+        config, scmAddress, fetchedSCMId);
+
+    HddsProtos.ScmNodeDetailsProto scmNodeDetailsProto =
+        HddsProtos.ScmNodeDetailsProto.newBuilder()
+            .setClusterId(scmStorageConfig.getClusterID())
+            .setHostName(scmAddress.getHostName())
+            .setScmNodeId(fetchedSCMId).build();
+
+    SCMSecurityProtocolClientSideTranslatorPB secureScmClient =
+        HddsServerUtil.getScmSecurityClient(config);
+
+    SCMGetCertResponseProto response = secureScmClient.
+        getSCMCertChain(scmNodeDetailsProto, getEncodedString(csr));
+    String pemEncodedCert = response.getX509Certificate();
+
+    try {
+
+      // Store SCM CA certificate.
+      if (response.hasX509CACertificate()) {
+        String pemEncodedRootCert = response.getX509CACertificate();
+        client.storeCertificate(pemEncodedRootCert, true, true);
+        client.storeCertificate(pemEncodedCert, true);
+
+
+        // Write to the location, so that Default CA server can read the
+        // certificate generated by cert client.
+        CertificateCodec certCodec =
+            new CertificateCodec(new SecurityConfig(config),
+                client.getComponentName());
+
+        X509Certificate certificate =
+            CertificateCodec.getX509Certificate(pemEncodedCert);
+        certCodec.writeCertificate(certCodec.getCertificateHolder(certificate));
+
+        // Persist scm cert serial ID.
+        scmStorageConfig.setScmCertSerialId(certificate.getSerialNumber()
+            .toString());
+      } else {
+        throw new RuntimeException("Unable to retrieve SCM certificate chain");
+      }
+    } catch (IOException | CertificateException e) {
+      LOG.error("Error while storing SCM signed certificate.", e);
+      throw new RuntimeException(e);
+    }
+  }
+
+  private static void getPrimarySCMSignedCert(CertificateClient client,
+      OzoneConfiguration config, String fetchedSCMId,
+      SCMStorageConfig scmStorageConfig, InetSocketAddress scmAddress)
+      throws IOException {
+
+    PKCS10CertificationRequest csr = generateCSR(client, scmStorageConfig,
+        config, scmAddress, fetchedSCMId);
+
+    CertificateServer rootCAServer =
+        initializeRootCertificateServer(scmStorageConfig.getClusterID(),
+            scmStorageConfig.getScmId(), null);
+
+    rootCAServer.init(new SecurityConfig(config),
+        CertificateServer.CAType.SELF_SIGNED_CA);
+
+    try {
+      X509CertificateHolder certificateHolder = rootCAServer.
+          requestCertificate(csr, KERBEROS_TRUSTED, SCM).get();
+
+      X509CertificateHolder rootCAServerCACertificateHolder =
+          rootCAServer.getCACertificate();
+
+      String pemEncodedCert =
+          CertificateCodec.getPEMEncodedString(certificateHolder);
+
+      String pemEncodedRootCert =
+          CertificateCodec.getPEMEncodedString(rootCAServerCACertificateHolder);
+
+
+      client.storeCertificate(pemEncodedRootCert, true, true);
+      client.storeCertificate(pemEncodedCert, true);
+
+      // Write to the location, so that Default CA server can read the
+      // certificate generated by cert client.
+      CertificateCodec certCodec =
+          new CertificateCodec(new SecurityConfig(config),
+              client.getComponentName());
+
+      certCodec.writeCertificate(certificateHolder);
+
+        // Persist scm cert serial ID.
+      scmStorageConfig.setScmCertSerialId(certificateHolder.getSerialNumber()

Review comment:
       This is not rootCA CertID, this is the signed cert SCM got from root CA.
   Similar to other roles like OM/DN only persisting cert serial ID which it got from CA. (In case of SCM it is from root CA, where as from OM/DN it is from Sub SCM CA)
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2000: HDDS-4915. [SCM HA Security] Integrate CertClient.

Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2000:
URL: https://github.com/apache/ozone/pull/2000#discussion_r603018839



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
##########
@@ -860,6 +966,40 @@ public static boolean scmInit(OzoneConfiguration conf,
     }
   }
 
+  private static InetSocketAddress getScmAddress(SCMHANodeDetails haDetails,

Review comment:
       As it has dependency on SCMHANodeDetails which is in server-scm (And server-scm depends on framework) and HAUtills class is in server-framework, we cannot move this to HAUtils. I will leave it as it is for now.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org