You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Galsza (via GitHub)" <gi...@apache.org> on 2023/08/07 15:41:51 UTC

[GitHub] [ozone] Galsza opened a new pull request, #5157: HDDS-9046. [ozone-cert-rotation] cert clean is unable to cleanup certificates LOCK error

Galsza opened a new pull request, #5157:
URL: https://github.com/apache/ozone/pull/5157

   The previous version of CertClean command required 2 uncomfortable things from the user: 
   1. specifying the database path to be cleared
   2. shutting down the SCMs before the expired certs can be removed
   
   This new version uses functionality added in https://github.com/apache/ozone/pull/5089 .
   
   ## What is the link to the Apache JIRA
   
   [HDDS-9046](https://issues.apache.org/jira/browse/HDDS-9046)
   
   ## How was this patch tested?
   
   Tested manually in a docker cluster, by disabling the automatic cert removal and giving out the cert clean command.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] ChenSammi merged pull request #5157: HDDS-9046. [ozone-cert-rotation] cert clean is unable to cleanup certificates LOCK error

Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi merged PR #5157:
URL: https://github.com/apache/ozone/pull/5157


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] Galsza commented on pull request #5157: HDDS-9046. [ozone-cert-rotation] cert clean is unable to cleanup certificates LOCK error

Posted by "Galsza (via GitHub)" <gi...@apache.org>.
Galsza commented on PR #5157:
URL: https://github.com/apache/ozone/pull/5157#issuecomment-1671410211

   @fapifta Thanks for notifying me, I've addressed the rat failure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] ChenSammi commented on a diff in pull request #5157: HDDS-9046. [ozone-cert-rotation] cert clean is unable to cleanup certificates LOCK error

Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #5157:
URL: https://github.com/apache/ozone/pull/5157#discussion_r1288250956


##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMSecurityProtocolServer.java:
##########
@@ -508,6 +508,16 @@ public long revokeCertificates(List<String> certIds, int reason,
     }
   }
 
+  @Override
+  public List<String> clearExpiredCertificates() throws IOException {
+    List<String> pemEncodedCerts = new ArrayList<>();

Review Comment:
   I would suggest add the isAdmin check here. Here is an example.
   
    storageContainerManager.checkAdminAccess(getRpcRemoteUser(), false);



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] fapifta commented on a diff in pull request #5157: HDDS-9046. [ozone-cert-rotation] cert clean is unable to cleanup certificates LOCK error

Posted by "fapifta (via GitHub)" <gi...@apache.org>.
fapifta commented on code in PR #5157:
URL: https://github.com/apache/ozone/pull/5157#discussion_r1287792674


##########
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/cert/CertificatePrintUtils.java:
##########


Review Comment:
   Instead of having a utility class for this, as this is used only in the CLI, can we put this logic to the SCMCertSubCommand as non-static protected methods, and then use it from the children CleanExpired and ListSubCommand? If you see any immediate or in future use of having these methods as static utility in this package, then let's discuss it, I am open to have it this way if there is a reason, but I would prefer to just leave this internal to the certificate sub commands based on my current knowledge.



##########
hadoop-hdds/tools/src/test/java/org/apache/hadoop/hdds/scm/cli/cert/TestCleanExpired.java:
##########


Review Comment:
   Should we have some kind of test at least for the happy path?
   I can imagine testing server side implementation with a mocked certificate store or something like that, look at ScmConfigurator usages to see how you can inject one in a mocked ScmHAManager.
   
   Also we should test client side I believe with a mocked server side behind, so that we can ensure that it is visible via the test change when the output format changes for any reason.
   
   But there might be better ways to test the functonality all I am sure about is that it is definitelly better to have some tests than no tests at all.



##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMCertStore.java:
##########
@@ -223,33 +223,41 @@ public void removeExpiredCertificate(BigInteger serialID)
   }
 
   @Override
-  public void removeAllExpiredCertificates() throws IOException {
+  public List<X509Certificate> removeAllExpiredCertificates()
+      throws IOException {
+    List<X509Certificate> removedCerts = new ArrayList<>();
     lock.lock();
     try (BatchOperation batchOperation =
              scmMetadataStore.getBatchHandler().initBatchOperation()) {
-      addExpiredCertsToBeRemoved(batchOperation,
-          scmMetadataStore.getValidCertsTable());
-      addExpiredCertsToBeRemoved(batchOperation,
-          scmMetadataStore.getValidSCMCertsTable());
+      removedCerts.addAll(addExpiredCertsToBeRemoved(batchOperation,
+          scmMetadataStore.getValidCertsTable()));
+      removedCerts.addAll(addExpiredCertsToBeRemoved(batchOperation,
+          scmMetadataStore.getValidSCMCertsTable()));
       scmMetadataStore.getStore().commitBatchOperation(batchOperation);
     } finally {
       lock.unlock();
     }
+    return removedCerts;
   }
 
-  private void addExpiredCertsToBeRemoved(BatchOperation batchOperation,
-      Table<BigInteger, X509Certificate> certTable) throws IOException {
+  private List<X509Certificate> addExpiredCertsToBeRemoved(
+      BatchOperation batchOperation, Table<BigInteger,
+      X509Certificate> certTable) throws IOException {
+    List<X509Certificate> removedCerts = new ArrayList<>();
     try (TableIterator<BigInteger, ? extends Table.KeyValue<BigInteger,
         X509Certificate>> certsIterator = certTable.iterator()) {
       Instant now = Instant.now();
       while (certsIterator.hasNext()) {
         Table.KeyValue<BigInteger, X509Certificate> certEntry =
             certsIterator.next();
-        if (certEntry.getValue().getNotAfter().toInstant().isBefore(now)) {
+        X509Certificate cert = certEntry.getValue();
+        if (cert.getNotAfter().toInstant().isBefore(now)) {

Review Comment:
   We don't need toInstant, you have Date#before for this check, with a new Date() instead of Instant.now().



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] ChenSammi commented on pull request #5157: HDDS-9046. [ozone-cert-rotation] cert clean is unable to cleanup certificates LOCK error

Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on PR #5157:
URL: https://github.com/apache/ozone/pull/5157#issuecomment-1672505139

   @Galsza , thanks for the changes.  There is more one comment,  OUTPUT_FORMAT and printCert in ListSubcommand can be removed since it's not used any more.  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] Galsza commented on pull request #5157: HDDS-9046. [ozone-cert-rotation] cert clean is unable to cleanup certificates LOCK error

Posted by "Galsza (via GitHub)" <gi...@apache.org>.
Galsza commented on PR #5157:
URL: https://github.com/apache/ozone/pull/5157#issuecomment-1671289374

   @ChenSammi Thank you for your review. Regarding the clean/clear/remove I've changed it to be clean on client side and remove on SCM side. Let me know if you are okay with this solution.
   
   @fapifta Thank you for your comments as well, I've addressed them.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] ChenSammi commented on a diff in pull request #5157: HDDS-9046. [ozone-cert-rotation] cert clean is unable to cleanup certificates LOCK error

Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #5157:
URL: https://github.com/apache/ozone/pull/5157#discussion_r1288267884


##########
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/protocol/SCMSecurityProtocol.java:
##########
@@ -190,4 +190,5 @@ String getCertificate(NodeDetailsProto nodeDetails,
    */
   List<String> getAllRootCaCertificates() throws IOException;
 
+  List<String> clearExpiredCertificates() throws IOException;

Review Comment:
   Can you add the java doc for this API?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] Galsza commented on a diff in pull request #5157: HDDS-9046. [ozone-cert-rotation] cert clean is unable to cleanup certificates LOCK error

Posted by "Galsza (via GitHub)" <gi...@apache.org>.
Galsza commented on code in PR #5157:
URL: https://github.com/apache/ozone/pull/5157#discussion_r1288446963


##########
hadoop-hdds/tools/src/test/java/org/apache/hadoop/hdds/scm/cli/cert/TestCleanExpired.java:
##########


Review Comment:
   Thank you for the suggestion, the client is tested now. SCMCertStore is tested already and the rest of the changes are on the client side/protocol messages, which are tested separately.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] fapifta commented on pull request #5157: HDDS-9046. [ozone-cert-rotation] cert clean is unable to cleanup certificates LOCK error

Posted by "fapifta (via GitHub)" <gi...@apache.org>.
fapifta commented on PR #5157:
URL: https://github.com/apache/ozone/pull/5157#issuecomment-1671346464

   @Galsza thank you for the changes, the patch looks good to me, I have kicked off the CI, and I will let @ChenSammi to comment on the changes before we merge.
   
   rat already failed, please take a look at the CI results when done.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] Galsza commented on pull request #5157: HDDS-9046. [ozone-cert-rotation] cert clean is unable to cleanup certificates LOCK error

Posted by "Galsza (via GitHub)" <gi...@apache.org>.
Galsza commented on PR #5157:
URL: https://github.com/apache/ozone/pull/5157#issuecomment-1672737558

   @ChenSammi I've addressed that comment now and also did some more manual testing.
   
   The exit code when the process is successful is 0.
   When server side throws an exception it's non 0, so this is working as intended.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] Galsza commented on pull request #5157: HDDS-9046. [ozone-cert-rotation] cert clean is unable to cleanup certificates LOCK error

Posted by "Galsza (via GitHub)" <gi...@apache.org>.
Galsza commented on PR #5157:
URL: https://github.com/apache/ozone/pull/5157#issuecomment-1669267633

   @fapifta , @ChenSammi the pipeline failed on a findbugs issue which has been fixed since. Please take a look!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] Galsza commented on a diff in pull request #5157: HDDS-9046. [ozone-cert-rotation] cert clean is unable to cleanup certificates LOCK error

Posted by "Galsza (via GitHub)" <gi...@apache.org>.
Galsza commented on code in PR #5157:
URL: https://github.com/apache/ozone/pull/5157#discussion_r1288448154


##########
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/cert/CertificatePrintUtils.java:
##########


Review Comment:
   Thank you that's a nice catch. At first my instinct was to create a utility class because other subcommands might use it elsewhere, but they should all originate from the cert commands anyway.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] ChenSammi commented on pull request #5157: HDDS-9046. [ozone-cert-rotation] cert clean is unable to cleanup certificates LOCK error

Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on PR #5157:
URL: https://github.com/apache/ozone/pull/5157#issuecomment-1671064690

   Hi @Galsza , overall the patch looks good to me.  I would like to discuss about the wording used in the patch.
   
   There are words "remove", "clear" and "clean" used in client side CLI,  SCM protocol file, and logic in SCM which delete the expired certificates from the rocksDB. 
   
   Can we consolidate the words? for example, use "remove" in SCM side and "clean" in CLI?  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] Galsza commented on a diff in pull request #5157: HDDS-9046. [ozone-cert-rotation] cert clean is unable to cleanup certificates LOCK error

Posted by "Galsza (via GitHub)" <gi...@apache.org>.
Galsza commented on code in PR #5157:
URL: https://github.com/apache/ozone/pull/5157#discussion_r1288445693


##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMCertStore.java:
##########
@@ -223,33 +223,41 @@ public void removeExpiredCertificate(BigInteger serialID)
   }
 
   @Override
-  public void removeAllExpiredCertificates() throws IOException {
+  public List<X509Certificate> removeAllExpiredCertificates()
+      throws IOException {
+    List<X509Certificate> removedCerts = new ArrayList<>();
     lock.lock();
     try (BatchOperation batchOperation =
              scmMetadataStore.getBatchHandler().initBatchOperation()) {
-      addExpiredCertsToBeRemoved(batchOperation,
-          scmMetadataStore.getValidCertsTable());
-      addExpiredCertsToBeRemoved(batchOperation,
-          scmMetadataStore.getValidSCMCertsTable());
+      removedCerts.addAll(addExpiredCertsToBeRemoved(batchOperation,
+          scmMetadataStore.getValidCertsTable()));
+      removedCerts.addAll(addExpiredCertsToBeRemoved(batchOperation,
+          scmMetadataStore.getValidSCMCertsTable()));
       scmMetadataStore.getStore().commitBatchOperation(batchOperation);
     } finally {
       lock.unlock();
     }
+    return removedCerts;
   }
 
-  private void addExpiredCertsToBeRemoved(BatchOperation batchOperation,
-      Table<BigInteger, X509Certificate> certTable) throws IOException {
+  private List<X509Certificate> addExpiredCertsToBeRemoved(
+      BatchOperation batchOperation, Table<BigInteger,
+      X509Certificate> certTable) throws IOException {
+    List<X509Certificate> removedCerts = new ArrayList<>();
     try (TableIterator<BigInteger, ? extends Table.KeyValue<BigInteger,
         X509Certificate>> certsIterator = certTable.iterator()) {
       Instant now = Instant.now();
       while (certsIterator.hasNext()) {
         Table.KeyValue<BigInteger, X509Certificate> certEntry =
             certsIterator.next();
-        if (certEntry.getValue().getNotAfter().toInstant().isBefore(now)) {
+        X509Certificate cert = certEntry.getValue();
+        if (cert.getNotAfter().toInstant().isBefore(now)) {

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] Galsza commented on a diff in pull request #5157: HDDS-9046. [ozone-cert-rotation] cert clean is unable to cleanup certificates LOCK error

Posted by "Galsza (via GitHub)" <gi...@apache.org>.
Galsza commented on code in PR #5157:
URL: https://github.com/apache/ozone/pull/5157#discussion_r1288444302


##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMSecurityProtocolServer.java:
##########
@@ -508,6 +508,16 @@ public long revokeCertificates(List<String> certIds, int reason,
     }
   }
 
+  @Override
+  public List<String> clearExpiredCertificates() throws IOException {
+    List<String> pemEncodedCerts = new ArrayList<>();

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org