You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Siyao Meng (Jira)" <ji...@apache.org> on 2026/08/11 07:08:00 UTC

[jira] [Updated] (HDDS-16120) A single MiniOzoneCluster build timeout cascades into whole-suite failures in MiniOzoneClusterProvider

     [ https://issues.apache.org/jira/browse/HDDS-16120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Siyao Meng updated HDDS-16120:
------------------------------
    Attachment: HDDS-16120.001.patch

> A single MiniOzoneCluster build timeout cascades into whole-suite failures in MiniOzoneClusterProvider
> ------------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-16120
>                 URL: https://issues.apache.org/jira/browse/HDDS-16120
>             Project: Apache Ozone
>          Issue Type: Bug
>            Reporter: Siyao Meng
>            Assignee: Chia-Chuan Ho
>            Priority: Major
>         Attachments: HDDS-16120.001.patch
>
>
> {{MiniOzoneClusterProvider.createClusters()}} builds clusters on a background thread and hands them to consumers through a small blocking queue. When a build times out ({{waitForClusterToBeReady()}} throws {{TimeoutException}}) or fails with {{IOException}}, the background thread rethrows it as an unchecked {{RuntimeException("Unable to build cluster")}}. That exception is uncaught on the create thread, so the thread dies and the queue is never refilled. Every subsequent {{provide()}} then blocks until its own timeout and fails with "Failed to obtain available cluster in time".
> The effect is that a single slow or failed cluster build turns into failures for every remaining test that shares the provider. This was observed on master where one method timed out during cluster startup and the following seven methods in the same class all failed with "Failed to obtain available cluster in time". It also makes any transient startup slowdown disproportionately expensive, since one timeout can push an entire integration split to its 90 minute job limit.
> h3. Proposed fix
> Do not let one build failure kill the create thread. Tear down the partial cluster and continue (retry) so a single slow build is not fatal to the suite, or record the failure and surface the original cause from {{provide()}} while keeping the thread alive. Either way removes the "one timeout becomes many failures" behavior.
> h3. Testing
> Unit test for {{MiniOzoneClusterProvider}} where the builder throws on one build: assert that later {{provide()}} calls still succeed (retry path) or fail fast with the original cause, and that the create thread remains alive.
> h3. Note
> This is an amplifier that bounds the blast radius of any single slow cluster build; it is valuable independently of whatever triggers a slow build. It is not itself the root trigger of the recent master integration-job timeouts, and that root trigger is currently unconfirmed.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org