[jira] [Created] (HDDS-16638) Fix kubernetes CI ozone-ha SCM HA startup (DNS / safe mode wait)

["Wei-Chiu Chuang (Jira)" <ji...@apache.org>]

Wei-Chiu Chuang created HDDS-16638:
--------------------------------------

             Summary: Fix kubernetes CI ozone-ha SCM HA startup (DNS / safe mode wait)
                 Key: HDDS-16638
                 URL: https://issues.apache.org/jira/browse/HDDS-16638
             Project: Apache Ozone
          Issue Type: Bug
            Reporter: Wei-Chiu Chuang


h2. Problem

The {{kubernetes}} CI job fails on the {{ozone-ha}} example (see GitHub Actions). {{wait_for_startup}} never sees {{SCM exiting safe mode.}} in {{scm-0}} logs, so Robot tests never run and {{output.xml}} is missing.

h2. Root cause

SCM HA pods start with Ratis configured for {{scm-0}}, {{scm-1}}, and {{scm-2}} while the StatefulSet uses default {{OrderedReady}} scheduling. {{scm-0}} starts before peer pods (and their headless-service DNS records) exist, which produces {{UnknownHostException}} for {{scm-1.scm...}} / {{scm-2.scm...}} and prevents the SCM HA quorum from forming.

A secondary issue: {{all_pods_are_running}} counted every pod in the namespace, so leftover {{prometheus}}/{{jaeger}} pods from {{ozone-dev}} could block startup checks even when Ozone pods were healthy.

h2. Fix

* Set {{podManagementPolicy: Parallel}} on the SCM StatefulSet and add a {{wait-for-scm-peers}} init container that waits for {{scm-0.scm}}, {{scm-1.scm}}, and {{scm-2.scm}} DNS before {{init}}/{{bootstrap}}.
* Scope {{all_pods_are_running}} to {{app=ozone}} and wait for pod deletion after {{reset_k8s_env}}.
* Treat any SCM pod logging safe-mode exit as cluster ready (HA leader may not be {{scm-0}}).

h2. Testing

* {{bash -n}} on {{testlib.sh}}
* {{./hadoop-ozone/dev-support/checks/bats.sh}}
* {{kubernetes}} CI job on PR



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org