[jira] [Updated] (HDDS-17236) Intermittent failure in ozonesecure public key and certificate recovery test

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-17236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated HDDS-17236:
----------------------------------
    Labels: pull-request-available  (was: )

> Intermittent failure in ozonesecure public key and certificate recovery test
> ----------------------------------------------------------------------------
>
>                 Key: HDDS-17236
>                 URL: https://issues.apache.org/jira/browse/HDDS-17236
>             Project: Apache Ozone
>          Issue Type: Sub-task
>          Components: docker-compose, test
>            Reporter: Siyao Meng
>            Assignee: Siyao Meng
>            Priority: Major
>              Labels: pull-request-available
>
> {code}
> Port 9888 is available on recon
> find /data/metadata/recon/keys/public.pem -delete && find /data/metadata/recon/certs/*.crt -delete succeed
> Ncat: Could not resolve hostname "recon": Name or service not known. QUITTING.
> Port 9888 is not available on recon yet
> ...
> Timed out waiting on recon 9888 to become available
> ERROR: Test execution of ozonesecure/test.sh is FAILED!!!!
> {code}
> h3. Root cause
> {{ozonesecure/public-key-cert-recovery-test.sh}} restarts Recon (and later the datanodes) with {{docker-compose stop recon}} followed by {{docker-compose start recon}}. The Recon container log shows the SIGTERM and a clean shutdown, but no second {{STARTUP_MSG}}, and the hostname stays unresolvable for the whole 120s wait. So the container was never started again, yet {{docker-compose start}} exited 0 (the script runs with {{set -e}} and kept going).
> {{docker compose start}} (v2.38.2 on the runner) lists the project containers once and skips any whose state is {{running}}. The Docker daemon wakes up the {{docker stop}} caller in {{handleContainerExit}} via {{SetStopped}} before it writes the new state to the replica that {{ContainerList}} reads ({{CheckpointTo(containersReplica)}}). If {{start}} lists containers in that window, it still sees Recon as running and silently does nothing.
> The same failure appears on master on 2026-07-13, 2026-07-15 and 2026-08-31 (ozone-build-results), always with the identical log pattern.
> h3. Fix
> Use {{docker-compose restart}} instead of {{stop}} + {{start}}. It calls {{ContainerRestart}} for each container unconditionally, so the stop and start happen inside the daemon with no stale state check in between.
> - https://github.com/apache/ozone/actions/runs/39561350318/job/119114972849



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org