You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Siyao Meng (Jira)" <ji...@apache.org> on 2026/07/10 21:11:00 UTC
[jira] [Updated] (HDDS-15824) Flaky debug replicas acceptance test: wait for RATIS THREE pipeline before writing keys
[ https://issues.apache.org/jira/browse/HDDS-15824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siyao Meng updated HDDS-15824:
------------------------------
Description:
h2. Problem
The {{ozonesecure-ha}} debug replicas acceptance suite ({{Ozone-Debug-Tests :: Test ozone debug CLI}} in {{smoketest/debug/ozone-debug-tests.robot}}) is flaky. Its Suite Setup ({{Write keys}}) intermittently fails with {{255 != 0}}, which marks every test in the suite as failed even though no test logic ran.
Root cause is in {{hadoop-ozone/dist/src/main/compose/common/replicas-test.sh}}. The container.db backup flow introduced by HDDS-14225 (the RocksDB 7.7.3 to 10.10.1 upgrade) stops all datanodes for a consistent backup, restarts them, and waits only for {{wait_for_datanode HEALTHY}} before running {{debug/ozone-debug-tests.robot}}. {{wait_for_datanode}} confirms the SCM heartbeat state (HEALTHY) but not that write pipelines have been re-established. {{Write keys}} then writes a default RATIS THREE key, a RATIS ONE key, a RATIS THREE key, and an EC rs-3-2 key; if it runs before an OPEN pipeline is available after the mass restart, the {{ozone sh key put}} exits non-zero (255) and the suite setup fails.
Before HDDS-14225 the test backed up only datanode1 via {{docker cp}} without stopping any datanode, and did not restart datanodes before {{Write keys}}, so this race did not exist.
h2. Fix
After the datanode restart and before running the debug test, wait (bounded at 180s) for at least one OPEN RATIS THREE pipeline via {{ozone admin pipeline list --state OPEN --type RATIS --replication THREE --json}}. RATIS ONE and EC writes are covered by all five datanodes already being HEALTHY; the RATIS THREE pipeline being OPEN is the gap this closes.
was:Placeholder, set in follow-up update.
> Flaky debug replicas acceptance test: wait for RATIS THREE pipeline before writing keys
> ---------------------------------------------------------------------------------------
>
> Key: HDDS-15824
> URL: https://issues.apache.org/jira/browse/HDDS-15824
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
>
> h2. Problem
> The {{ozonesecure-ha}} debug replicas acceptance suite ({{Ozone-Debug-Tests :: Test ozone debug CLI}} in {{smoketest/debug/ozone-debug-tests.robot}}) is flaky. Its Suite Setup ({{Write keys}}) intermittently fails with {{255 != 0}}, which marks every test in the suite as failed even though no test logic ran.
> Root cause is in {{hadoop-ozone/dist/src/main/compose/common/replicas-test.sh}}. The container.db backup flow introduced by HDDS-14225 (the RocksDB 7.7.3 to 10.10.1 upgrade) stops all datanodes for a consistent backup, restarts them, and waits only for {{wait_for_datanode HEALTHY}} before running {{debug/ozone-debug-tests.robot}}. {{wait_for_datanode}} confirms the SCM heartbeat state (HEALTHY) but not that write pipelines have been re-established. {{Write keys}} then writes a default RATIS THREE key, a RATIS ONE key, a RATIS THREE key, and an EC rs-3-2 key; if it runs before an OPEN pipeline is available after the mass restart, the {{ozone sh key put}} exits non-zero (255) and the suite setup fails.
> Before HDDS-14225 the test backed up only datanode1 via {{docker cp}} without stopping any datanode, and did not restart datanodes before {{Write keys}}, so this race did not exist.
> h2. Fix
> After the datanode restart and before running the debug test, wait (bounded at 180s) for at least one OPEN RATIS THREE pipeline via {{ozone admin pipeline list --state OPEN --type RATIS --replication THREE --json}}. RATIS ONE and EC writes are covered by all five datanodes already being HEALTHY; the RATIS THREE pipeline being OPEN is the gap this closes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org