You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "ss77892 (via GitHub)" <gi...@apache.org> on 2026/05/05 18:30:04 UTC
[PR] HDDS-15177. acceptance test race condition in generating keytab and service start. [ozone]
ss77892 opened a new pull request, #10194:
URL: https://github.com/apache/ozone/pull/10194
## What changes were proposed in this pull request?
HDDS-15177. acceptance test race condition in generating keytab and service start.
Fix intermittent xcompat failures where the new-cluster Compose stack could start SCM and other Kerberos-backed services before init-kdc finished exporting keytabs and starting the KDC. Add the same KDC healthcheck used on the old cluster (verify scm.keytab exists and port 88 accepts connections) and depend on the KDC being healthy.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-15177
## How was this patch tested?
CI + local tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15177. acceptance test race condition in generating keytab and service start. [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #10194:
URL: https://github.com/apache/ozone/pull/10194#issuecomment-4439143208
> I think other secure clusters need the same fix (only upgrade has it currently). `xcompat-old` is special only because it starts multiple clusters (one per Ozone version), increasing the probability of hitting the race condition.
Adding to remaining tests in #10256.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15177. acceptance test race condition in generating keytab and service start. [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #10194:
URL: https://github.com/apache/ozone/pull/10194#discussion_r3203968764
##########
hadoop-ozone/dist/src/main/compose/xcompat/new-cluster.yaml:
##########
@@ -35,6 +38,12 @@ services:
- ../..:/opt/hadoop
- ../_keytabs:/etc/security/keytabs
command: ["/opt/hadoop/compose/common/init-kdc.sh"]
+ healthcheck:
+ test: ["CMD-SHELL", "test -s /etc/security/keytabs/scm.keytab && nc -z 127.0.0.1 88"]
+ interval: 2s
+ timeout: 3s
+ retries: 45
+ start_period: 5s
Review Comment:
Thanks @ss77892 for moving the healthcheck to the image definition. New image with that change is now available (`ghcr.io/apache/ozone-testkrb5:20260507-1`) for use in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15177. acceptance test race condition in generating keytab and service start. [ozone]
Posted by "ss77892 (via GitHub)" <gi...@apache.org>.
ss77892 commented on PR #10194:
URL: https://github.com/apache/ozone/pull/10194#issuecomment-4402158430
@adoroszlai Thank you for your review and suggestions. I've updated the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15177. acceptance test race condition in generating keytab and service start. [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #10194:
URL: https://github.com/apache/ozone/pull/10194#discussion_r3193779260
##########
hadoop-ozone/dist/src/main/compose/xcompat/new-cluster.yaml:
##########
@@ -48,6 +57,9 @@ services:
volumes:
- ../../libexec/transformation.py:/opt/transformation.py
command: [ "hadoop", "kms" ]
+ depends_on:
+ kdc:
+ condition: service_healthy
Review Comment:
This seems to be unnecessary, since `/etc/security/keytabs` is not mounted, so KMS cannot use keytabs created by KDC.
##########
hadoop-ozone/dist/src/main/compose/xcompat/new-cluster.yaml:
##########
@@ -35,6 +38,12 @@ services:
- ../..:/opt/hadoop
- ../_keytabs:/etc/security/keytabs
command: ["/opt/hadoop/compose/common/init-kdc.sh"]
+ healthcheck:
+ test: ["CMD-SHELL", "test -s /etc/security/keytabs/scm.keytab && nc -z 127.0.0.1 88"]
+ interval: 2s
+ timeout: 3s
+ retries: 45
+ start_period: 5s
Review Comment:
- Checking existence of `scm.keytab` is not necessary, since KDC is started (in `init-kdc.sh`) only after exporting all keytabs.
- I think it would be better to add the [healthcheck](https://docs.docker.com/reference/dockerfile/#healthcheck) in the [image](https://github.com/apache/ozone-docker-testkrb5/blob/master/Dockerfile) itself to reduce duplication. Base command may need to be set (`krb5kdc -n`), too. With that, we only need `depends_on` in compose files (and update [`docker.ozone-testkr5b.image`](https://github.com/apache/ozone/blob/c3ffd0562e86d7188b128b2f7503fc69dee3bdf9/hadoop-ozone/dist/pom.xml#L33))
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15177. acceptance test race condition in generating keytab and service start. [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #10194:
URL: https://github.com/apache/ozone/pull/10194#issuecomment-4388819927
Example for failure: https://github.com/apache/ozone/actions/runs/25426543750/job/74600076227
I think other secure clusters need the same fix (only upgrade has it currently). `xcompat-old` is special only because it starts multiple clusters (one per Ozone version).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15177. acceptance test race condition in generating keytab and service start. [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai merged PR #10194:
URL: https://github.com/apache/ozone/pull/10194
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org