[jira] [Created] (HDDS-16172) scaling up OM replicas leaves the new OM pod stuck in Init CrashLoopBackOff

["Meng-Shuan Tsai (Jira)" <ji...@apache.org>]

Meng-Shuan Tsai created HDDS-16172:
--------------------------------------

             Summary: scaling up OM replicas leaves the new OM pod stuck in Init CrashLoopBackOff
                 Key: HDDS-16172
                 URL: https://issues.apache.org/jira/browse/HDDS-16172
             Project: Apache Ozone
          Issue Type: Sub-task
            Reporter: Meng-Shuan Tsai


h3. Problem

Scaling up OM replicas (e.g. {{om.replicas: 2 -> 3}} with {{{}om.persistence.enabled=true{}}}) leaves the new OM pod stuck in {{Init:0/1}} CrashLoopBackOff forever, and {{helm upgrade --wait}} times out.

The {{om-bootstrap}} init container script (charts/ozone/templates/om/om-bootstrap-configmap.yaml) runs:
{code:java}
ozone admin om --set "ozone.om.nodes.$OZONE_CLUSTER_ID=$overwriteCmd" --bootstrap
{code}
but {{ozone admin om}} has no {{{}-{}}}set or {{-bootstrap}} option (neither in 2.0.0, the chart's appVersion when the script was introduced in HDDS-11618, nor in the current 2.2.0). The command fails with exit code 2, the script retries 3 times and gives up, and the init container crash-loops:
{code:java}
Unknown options: '--set', 'ozone.om.nodes.cluster1=ozone-om-0,ozone-om-1,ozone-om-2', '--bootstrap'
Usage: ozone admin om [-hV] [--verbose] [COMMAND]
Bootstrap failed after 3 attempts with exit code 2
{code}
Note that simply fixing the command to {{ozone om --bootstrap}} is not enough: that command keeps running as the OM daemon on success, so it can never complete inside an init container, and the config check it performs ({{{}checkConfigBeforeBootstrap{}}}) requires the existing OMs to be restarted with the new node list first - a rolling restart which is itself blocked by the new pod being stuck in Init. Details and fix proposals will follow in the comments/PR.
h3. Steps to reproduce (kind)
 # Install with {{{}om.replicas=3{}}}, persistence enabled for om/scm/datanode
 # {{helm upgrade}} to {{om.replicas=2}} (scale-down works)
 # {{helm upgrade}} back to {{om.replicas=3}}
 # {{ozone-om-2}} is stuck in {{Init:0/1}} CrashLoopBackOff with the log above

This path has never worked since it was introduced in HDDS-11618; chart CI ({{{}ct lint{}}} / {{{}ct install{}}}) never exercises a scale-up against a live cluster, so it went unnoticed.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org