You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Galsza (via GitHub)" <gi...@apache.org> on 2023/05/09 06:24:30 UTC
[GitHub] [ozone] Galsza opened a new pull request, #4681: HDDS-8266. Create tests that cover certificate hotswap mechanisms
Galsza opened a new pull request, #4681:
URL: https://github.com/apache/ozone/pull/4681
## What changes were proposed in this pull request?
Create a test that covers the basic functionality of the certificate hotswap mechanism.
The test itself covers 2 cases. Both of them tries the cluster with a short certificate duration before and after a certificate rotation happens.
1. Put operations still work
2. Connection to the datanode via openssl is still possible (And the certificate id has changed)
## What is the link to the Apache JIRA
[HDDS-8266](https://issues.apache.org/jira/browse/HDDS-8266)
## How was this patch tested?
This is a testing improvement. Both locally and on github actions all tests still work:
https://github.com/Galsza/ozone/actions/runs/4916996599
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on pull request #4681: HDDS-8266. Create tests that cover certificate hotswap mechanisms
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #4681:
URL: https://github.com/apache/ozone/pull/4681#issuecomment-1547772895
Thanks @Galsza for updating the patch. It seems secure HA cluster/test needs some tweak:
```
Verify SCM4 Certificate | FAIL |
Resolving variable '${scm4_cert_split[4]}' failed: IndexError: list index out of range
```
https://github.com/apache/ozone/actions/runs/4978851064/jobs/8910534321?pr=4681#step:5:1006
https://github.com/Galsza/ozone/actions/runs/4978850737/jobs/8910141281#step:5:1011
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai merged pull request #4681: HDDS-8266. Create tests that cover certificate hotswap mechanisms
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai merged PR #4681:
URL: https://github.com/apache/ozone/pull/4681
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] Galsza commented on pull request #4681: HDDS-8266. Create tests that cover certificate hotswap mechanisms
Posted by "Galsza (via GitHub)" <gi...@apache.org>.
Galsza commented on PR #4681:
URL: https://github.com/apache/ozone/pull/4681#issuecomment-1547524566
Using @adoroszlai 's suggestion of disabling block token lifetime checks against the certificate lifetime it was possible to add the test case to HA clusters as well. The small block token lifetime would have caused flakiness before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] Galsza commented on a diff in pull request #4681: HDDS-8266. Create tests that cover certificate hotswap mechanisms
Posted by "Galsza (via GitHub)" <gi...@apache.org>.
Galsza commented on code in PR #4681:
URL: https://github.com/apache/ozone/pull/4681#discussion_r1196048459
##########
hadoop-ozone/dist/src/main/smoketest/security/cert-rotation.robot:
##########
@@ -0,0 +1,51 @@
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+*** Settings ***
+Documentation Smoketest ozone cluster startup
+Library OperatingSystem
+Resource ../commonlib.robot
+Resource ../ozone-lib/freon.robot
+Suite Setup Setup Test
+Test Timeout 5 minutes
+
+*** Variables ***
+${datanode} datanode
+${port} 9859
+
+*** Keywords ***
+Setup Test
+ Run Keyword if '${SECURITY_ENABLED}' == 'true' Kinit test user testuser testuser.keytab
+
+Basic key generation and validation
+ ${random} = Generate Random String 10
+ Freon OCKG prefix=${random}
+ Freon OCKV prefix=${random}
+
+Find wait time
+ ${waitTime} = Execute printenv | grep hdds.x509.default.duration | sed 's/OZONE-SITE.XML_hdds.x509.default.duration=//' | sed 's/PT//'
+ ${result} = Set Variable if "${waitTime}" != "${EMPTY}" ${waitTime} 0s
+ [return] ${result}
+
+Get datanode cert serial
+ ${certSerial} Execute openssl s_client -connect "${datanode}":"${port}" -showcerts | openssl x509 -noout -serial | grep serial | sed 's/serial=//'
+ [return] ${certSerial}
+
+*** Test Cases ***
+Test datanode functions before and after certificate rotation
+ Basic key generation and validation
+ ${sleepTime} = Find wait time
+ ${certId1} = Get datanode cert serial
+ Sleep ${sleepTime}
+ Basic key generation and validation
+ ${certId2} = Get datanode cert serial
+ Should Not Be Equal ${certId1} ${certId2}
Review Comment:
Thanks for both suggestions! I've also increased the timeout to double the cert expiration time just in case. (This test might start at a time when a cert-rotation has just finished, and then waiting only for the duration in the config might mean that the next cert rotation hasn't finished by the time the test expects it)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on pull request #4681: HDDS-8266. Create tests that cover certificate hotswap mechanisms
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #4681:
URL: https://github.com/apache/ozone/pull/4681#issuecomment-1548031502
@Galsza Since the test is [still failing](https://github.com/Galsza/ozone/actions/runs/4980695369/jobs/8914625086#step:5:1005), I think there must be more than 1000 certificates. All 9 services apparently renew every 20 seconds. Actual number will depend on how long it takes for the earlier tests to execute (about an hour currently). You can find the output in `log.html` in the [bundle](https://github.com/Galsza/ozone/suites/12901778895/artifacts/696615673) for the failed acceptance test.
For now you may want to set a very high limit. Later it can be improved by adding options for the cert list command to filter by issuer, subject etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] Galsza commented on a diff in pull request #4681: HDDS-8266. Create tests that cover certificate hotswap mechanisms
Posted by "Galsza (via GitHub)" <gi...@apache.org>.
Galsza commented on code in PR #4681:
URL: https://github.com/apache/ozone/pull/4681#discussion_r1193589916
##########
hadoop-ozone/dist/src/main/compose/ozonesecure/docker-config:
##########
@@ -41,6 +41,12 @@ OZONE-SITE.XML_ozone.scm.client.address=scm
OZONE-SITE.XML_hdds.block.token.enabled=true
OZONE-SITE.XML_hdds.container.token.enabled=true
OZONE-SITE.XML_hdds.grpc.tls.enabled=true
+
+OZONE-SITE.XML_hdds.x509.default.duration=PT40s
+OZONE-SITE.XML_hdds.x509.renew.grace.duration=PT30s
+OZONE-SITE.XML_hdds.block.token.expiry.time=15s
+OZONE-SITE.XML_ozone.manager.delegation.token.max-lifetime=15s
Review Comment:
Thank you Attila, I've learned something new from you. Please take another look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on pull request #4681: HDDS-8266. Create tests that cover certificate hotswap mechanisms
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #4681:
URL: https://github.com/apache/ozone/pull/4681#issuecomment-1558941187
@fapifta I'd like to extract the certificate rotation tests to a separate script after #4750 (which allows doing this without duplicating the environment).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a diff in pull request #4681: HDDS-8266. Create tests that cover certificate hotswap mechanisms
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #4681:
URL: https://github.com/apache/ozone/pull/4681#discussion_r1189015358
##########
hadoop-ozone/dist/src/main/compose/ozonesecure/docker-config:
##########
@@ -41,6 +41,12 @@ OZONE-SITE.XML_ozone.scm.client.address=scm
OZONE-SITE.XML_hdds.block.token.enabled=true
OZONE-SITE.XML_hdds.container.token.enabled=true
OZONE-SITE.XML_hdds.grpc.tls.enabled=true
+
+OZONE-SITE.XML_hdds.x509.default.duration=PT40s
+OZONE-SITE.XML_hdds.x509.renew.grace.duration=PT30s
+OZONE-SITE.XML_hdds.block.token.expiry.time=15s
+OZONE-SITE.XML_ozone.manager.delegation.token.max-lifetime=15s
Review Comment:
15 seconds may be too short for the token tests to pass:
```
TOKEN_EXPIRED testuser tried to renew an expired token (OzoneToken owner=testuser/scm@EXAMPLE.COM, renewer=testuser, realUser=, issueDate=2023-05-09T14:28:19.801Z, maxDate=2023-05-09T14:28:34.801Z, sequenceNumber=1, masterKeyId=131, strToSign=null, signature=null, awsAccessKeyId=null, omServiceId=omServiceIdDefault, omCertSerialId=2513321810675) max expiration date: 2023-05-09 14:28:34,801+0000 currentTime: 2023-05-09 14:28:36,793+0000 |
```
https://github.com/apache/ozone/actions/runs/4923738644/jobs/8802170708?pr=4681#step:5:623
Does token expiry matter for the cert rotation tests, anyway?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] Galsza commented on a diff in pull request #4681: HDDS-8266. Create tests that cover certificate hotswap mechanisms
Posted by "Galsza (via GitHub)" <gi...@apache.org>.
Galsza commented on code in PR #4681:
URL: https://github.com/apache/ozone/pull/4681#discussion_r1219175995
##########
hadoop-ozone/dist/src/main/compose/ozonesecure-ha/docker-config:
##########
@@ -54,6 +54,11 @@ OZONE-SITE.XML_ozone.scm.client.address=scm
OZONE-SITE.XML_hdds.block.token.enabled=true
OZONE-SITE.XML_hdds.container.token.enabled=true
OZONE-SITE.XML_hdds.grpc.tls.enabled=true
+
+OZONE-SITE.XML_hdds.x509.default.duration=PT40s
+OZONE-SITE.XML_hdds.x509.renew.grace.duration=PT30s
+OZONE-SITE.XML_hdds.x509.grace.duration.token.checks.enabled=false
Review Comment:
Thank you, I have removed the unused configs. In the end we are not going to run this test in ha for now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a diff in pull request #4681: HDDS-8266. Create tests that cover certificate hotswap mechanisms
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #4681:
URL: https://github.com/apache/ozone/pull/4681#discussion_r1217706463
##########
hadoop-ozone/dist/src/main/compose/ozonesecure-ha/docker-config:
##########
@@ -54,6 +54,11 @@ OZONE-SITE.XML_ozone.scm.client.address=scm
OZONE-SITE.XML_hdds.block.token.enabled=true
OZONE-SITE.XML_hdds.container.token.enabled=true
OZONE-SITE.XML_hdds.grpc.tls.enabled=true
+
+OZONE-SITE.XML_hdds.x509.default.duration=PT40s
+OZONE-SITE.XML_hdds.x509.renew.grace.duration=PT30s
+OZONE-SITE.XML_hdds.x509.grace.duration.token.checks.enabled=false
Review Comment:
Please remove these. Currently certificate rotation is not tested in `ozonesecure-ha`. If you would like to add that, please add them as separate config+script.
##########
hadoop-ozone/dist/src/main/compose/ozonesecure/docker-config:
##########
@@ -35,6 +35,11 @@ OZONE-SITE.XML_ozone.scm.client.address=scm
OZONE-SITE.XML_hdds.block.token.enabled=true
OZONE-SITE.XML_hdds.container.token.enabled=true
OZONE-SITE.XML_hdds.grpc.tls.enabled=true
+
+OZONE-SITE.XML_hdds.x509.default.duration=PT40s
+OZONE-SITE.XML_hdds.x509.renew.grace.duration=PT30s
+OZONE-SITE.XML_hdds.x509.grace.duration.token.checks.enabled=false
Review Comment:
These configs are now added by `certificate-rotation.yaml`, so they should be removed from the general `docker-config`.
##########
hadoop-ozone/dist/src/main/compose/ozonesecure/test-certificate-rotation.sh:
##########
@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
Review Comment:
Test scripts are assigned to various suites (`unsecure`, `secure`, `MR`, etc.). Uncategorized scripts, like this one, are run as part of the `misc` suite.
https://github.com/Galsza/ozone/actions/runs/5173931561/jobs/9320052865#step:5:1356
I think we should assign it to the `secure` suite.
```suggestion
#suite:secure
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a diff in pull request #4681: HDDS-8266. Create tests that cover certificate hotswap mechanisms
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #4681:
URL: https://github.com/apache/ozone/pull/4681#discussion_r1189514511
##########
hadoop-ozone/dist/src/main/compose/ozonesecure/docker-config:
##########
@@ -41,6 +41,12 @@ OZONE-SITE.XML_ozone.scm.client.address=scm
OZONE-SITE.XML_hdds.block.token.enabled=true
OZONE-SITE.XML_hdds.container.token.enabled=true
OZONE-SITE.XML_hdds.grpc.tls.enabled=true
+
+OZONE-SITE.XML_hdds.x509.default.duration=PT40s
+OZONE-SITE.XML_hdds.x509.renew.grace.duration=PT30s
+OZONE-SITE.XML_hdds.block.token.expiry.time=15s
+OZONE-SITE.XML_ozone.manager.delegation.token.max-lifetime=15s
Review Comment:
One option is to add an undocumented config property that disables the sanity check you mention. An example of such a config property:
https://github.com/apache/ozone/blob/e8fb1744e63b0f87379f7cb0eb1cb8a98520bd95/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsConfigKeys.java#L268-L272
https://github.com/apache/ozone/blob/e8fb1744e63b0f87379f7cb0eb1cb8a98520bd95/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestOzoneConfigurationFields.java#L88-L90
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a diff in pull request #4681: HDDS-8266. Create tests that cover certificate hotswap mechanisms
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #4681:
URL: https://github.com/apache/ozone/pull/4681#discussion_r1194870696
##########
hadoop-ozone/dist/src/main/smoketest/security/cert-rotation.robot:
##########
@@ -0,0 +1,51 @@
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+*** Settings ***
+Documentation Smoketest ozone cluster startup
+Library OperatingSystem
+Resource ../commonlib.robot
+Resource ../ozone-lib/freon.robot
+Suite Setup Setup Test
+Test Timeout 5 minutes
+
+*** Variables ***
+${datanode} datanode
+${port} 9859
+
+*** Keywords ***
+Setup Test
+ Run Keyword if '${SECURITY_ENABLED}' == 'true' Kinit test user testuser testuser.keytab
+
+Basic key generation and validation
+ ${random} = Generate Random String 10
+ Freon OCKG prefix=${random}
+ Freon OCKV prefix=${random}
+
+Find wait time
+ ${waitTime} = Execute printenv | grep hdds.x509.default.duration | sed 's/OZONE-SITE.XML_hdds.x509.default.duration=//' | sed 's/PT//'
Review Comment:
```suggestion
${waitTime} = Execute ozone getconf confKey hdds.x509.default.duration | sed 's/PT//'
```
##########
hadoop-ozone/dist/src/main/smoketest/security/cert-rotation.robot:
##########
@@ -0,0 +1,51 @@
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+*** Settings ***
+Documentation Smoketest ozone cluster startup
+Library OperatingSystem
+Resource ../commonlib.robot
+Resource ../ozone-lib/freon.robot
+Suite Setup Setup Test
+Test Timeout 5 minutes
+
+*** Variables ***
+${datanode} datanode
+${port} 9859
+
+*** Keywords ***
+Setup Test
+ Run Keyword if '${SECURITY_ENABLED}' == 'true' Kinit test user testuser testuser.keytab
+
+Basic key generation and validation
+ ${random} = Generate Random String 10
+ Freon OCKG prefix=${random}
+ Freon OCKV prefix=${random}
+
+Find wait time
+ ${waitTime} = Execute printenv | grep hdds.x509.default.duration | sed 's/OZONE-SITE.XML_hdds.x509.default.duration=//' | sed 's/PT//'
+ ${result} = Set Variable if "${waitTime}" != "${EMPTY}" ${waitTime} 0s
+ [return] ${result}
+
+Get datanode cert serial
+ ${certSerial} Execute openssl s_client -connect "${datanode}":"${port}" -showcerts | openssl x509 -noout -serial | grep serial | sed 's/serial=//'
+ [return] ${certSerial}
+
+*** Test Cases ***
+Test datanode functions before and after certificate rotation
+ Basic key generation and validation
+ ${sleepTime} = Find wait time
+ ${certId1} = Get datanode cert serial
+ Sleep ${sleepTime}
+ Basic key generation and validation
+ ${certId2} = Get datanode cert serial
+ Should Not Be Equal ${certId1} ${certId2}
Review Comment:
This could be improved by repeatedly testing if datanode has new certificate:
```
Wait Until Keyword Succeeds ${sleepTime} 5sec Datanode has new certificate ${certId1}
Basic key generation and validation
```
and move the certificate check to a new keyword:
```
Datanode has new certificate
[arguments] ${certId}
${newCertId} = Get datanode cert serial
Should Not Be Equal ${certId} ${newCertId}
```
(I assume certificate should be rotated even if we don't generate keys. And hope I got the syntax right.)
This is similar to `GenericTestUtils.waitFor()` vs. `Thread.sleep()` + `assert...()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on pull request #4681: HDDS-8266. Create tests that cover certificate hotswap mechanisms
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #4681:
URL: https://github.com/apache/ozone/pull/4681#issuecomment-1550870080
@Galsza Container replication [failed](https://github.com/Galsza/ozone/actions/runs/4995706186/jobs/8948418334#step:5:692) in _acceptance (secure)_ in your fork. Please check if the change introduces flakiness.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on pull request #4681: HDDS-8266. Create tests that cover certificate hotswap mechanisms
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #4681:
URL: https://github.com/apache/ozone/pull/4681#issuecomment-1558570518
@fapifta can you please take a look at the changes since your approval?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] Galsza commented on a diff in pull request #4681: HDDS-8266. Create tests that cover certificate hotswap mechanisms
Posted by "Galsza (via GitHub)" <gi...@apache.org>.
Galsza commented on code in PR #4681:
URL: https://github.com/apache/ozone/pull/4681#discussion_r1219175097
##########
hadoop-ozone/dist/src/main/compose/ozonesecure/docker-config:
##########
@@ -35,6 +35,11 @@ OZONE-SITE.XML_ozone.scm.client.address=scm
OZONE-SITE.XML_hdds.block.token.enabled=true
OZONE-SITE.XML_hdds.container.token.enabled=true
OZONE-SITE.XML_hdds.grpc.tls.enabled=true
+
+OZONE-SITE.XML_hdds.x509.default.duration=PT40s
+OZONE-SITE.XML_hdds.x509.renew.grace.duration=PT30s
+OZONE-SITE.XML_hdds.x509.grace.duration.token.checks.enabled=false
Review Comment:
Thank you, I have removed these unused configs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] Galsza commented on a diff in pull request #4681: HDDS-8266. Create tests that cover certificate hotswap mechanisms
Posted by "Galsza (via GitHub)" <gi...@apache.org>.
Galsza commented on code in PR #4681:
URL: https://github.com/apache/ozone/pull/4681#discussion_r1189495830
##########
hadoop-ozone/dist/src/main/compose/ozonesecure/docker-config:
##########
@@ -41,6 +41,12 @@ OZONE-SITE.XML_ozone.scm.client.address=scm
OZONE-SITE.XML_hdds.block.token.enabled=true
OZONE-SITE.XML_hdds.container.token.enabled=true
OZONE-SITE.XML_hdds.grpc.tls.enabled=true
+
+OZONE-SITE.XML_hdds.x509.default.duration=PT40s
+OZONE-SITE.XML_hdds.x509.renew.grace.duration=PT30s
+OZONE-SITE.XML_hdds.block.token.expiry.time=15s
+OZONE-SITE.XML_ozone.manager.delegation.token.max-lifetime=15s
Review Comment:
It does not matter directly, but we have a sanity check in [StorageContainerManager](https://github.com/apache/ozone/blob/e8fb1744e63b0f87379f7cb0eb1cb8a98520bd95/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java#L951)
checking if the block token expiry time is shorter than the certificate renew grace duration, therefore we need to also change it here. I'm not sure how to avoid flakiness here other than increasing the token expiry time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] fapifta commented on pull request #4681: HDDS-8266. Create tests that cover certificate hotswap mechanisms
Posted by "fapifta (via GitHub)" <gi...@apache.org>.
fapifta commented on PR #4681:
URL: https://github.com/apache/ozone/pull/4681#issuecomment-1558900443
I am not sure how to approach the following tradeoff here:
- on one hand, if we have a secure test environment, where there is a certificate rotation often, even if it becomes flaky that is a result for us, and we possibly find bugs in the rotation mechanism, however it is hard to debug, and most likely it won't happen all the time (knowing and extrapolating from my attitude against flaky tests). There is one interesting cause for a flaky behaviour here, when the box that runs the test runs out of entropy and with that introduces a higher delay that might lead to timeouts. (I am not sure if that is the case with the first attempt on the fork, I did not found logs to confirm.)
- on the other hand, if we separate the rotation related testing, we do not need to to crazy things like that -c gazillion in the current version of the patch, but we loose a more wide correctness check regarding the rotation itself.
As the test runs roughly for an hour or a bit more (in the success case it was 1 hour in the fail case it was over 1.5 hours), I think we should temper the situation, by allowing 10 minutes of timeout for the newly added rotation test, and work with a 5 minute certificate duration. As I understand (and @Galsza correct me if I am wrong), the newly added test actually checks every 5 seconds if the DN has a new certificate and with a 5 minute duration it would wait for 10 minutes tops for a rotation to happen, but if everything goes well, the rotation should happen in about 4.5 minutes the worst case with a grace duration of 30 seconds.
With these settings counting with 9 services for an hours and some we are at about 120-140 certificates once the test is finished, for which we can have a reasonable count around 300 for a large safety net, and perhaps we are somewhere in the middle of the two extremes.
Later on as part of HDDS-7401 we should revisit this, and see if we can define enough tests around the rotation (I believe we will) to pull off this change from here to a separate test set concentrating on rotation and functionality during and after rotation.
What do you think @Galsza and @adoroszlai about the proposed defaults, and the idea in general to temper the amount of certificates via their lifetime to avoid requesting for potentially thousands of certificates in other tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] Galsza commented on pull request #4681: HDDS-8266. Create tests that cover certificate hotswap mechanisms
Posted by "Galsza (via GitHub)" <gi...@apache.org>.
Galsza commented on PR #4681:
URL: https://github.com/apache/ozone/pull/4681#issuecomment-1559027156
I think separating this to a separate script as Attila mentioned is a good idea, because tempering these values right now would probably result in having to reconfigure them again later, once more certificate related features are added. (Like root ca rotation, or certificate revocation).
I also agree that having cert rotation in the longer tests is very beneficial, but that can probably be added with [HDDS-7401](https://issues.apache.org/jira/browse/HDDS-7401) or its subtasks, once we have more features to test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a diff in pull request #4681: HDDS-8266. Create tests that cover certificate hotswap mechanisms
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #4681:
URL: https://github.com/apache/ozone/pull/4681#discussion_r1214163548
##########
hadoop-ozone/dist/src/main/compose/ozonesecure-ha/docker-config:
##########
@@ -54,6 +54,11 @@ OZONE-SITE.XML_ozone.scm.client.address=scm
OZONE-SITE.XML_hdds.block.token.enabled=true
OZONE-SITE.XML_hdds.container.token.enabled=true
OZONE-SITE.XML_hdds.grpc.tls.enabled=true
+
+OZONE-SITE.XML_hdds.x509.default.duration=PT40s
+OZONE-SITE.XML_hdds.x509.renew.grace.duration=PT30s
+OZONE-SITE.XML_hdds.x509.grace.duration.token.checks.enabled=false
Review Comment:
@Galsza now that HDDS-8640 is merged, we can extract these into an add-on config, and the cert-rotation test into an extra shell script.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org