You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@maka.apache.org by "Astro-Han (via GitHub)" <gi...@apache.org> on 2026/08/26 02:21:07 UTC
[I] test(runtime-host): flaky 'startup failure preserves its cause when shutdown reaches the active deadline' [maka]
Astro-Han opened a new issue, #3840:
URL: https://github.com/apache/maka/issues/3840
`packages/runtime-host/src/__tests__/host-kernel.test.ts` intermittently fails on CI with:
```
✖ startup failure preserves its cause when shutdown reaches the active deadline
Error: composition close did not begin
```
## Observed occurrences
Both on unrelated branches, so the failure is not caused by the changes under test:
- https://github.com/apache/maka/actions/runs/32921736550/job/98036487525 (`fix/canonical-message-projection`, 2026-08-26T02:15Z, 1080ms)
- https://github.com/apache/maka/actions/runs/32884873070 (`feat/workhub-action-gate`, 2026-08-25T18:54Z, 1053ms)
In both cases the whole `non-serving Runtime Host kernel` suite ran 27–33s, i.e. the machine was heavily loaded.
## Analysis
The test arms a real Runtime Host kernel, forces a recovery failure, and then waits for the composition `close` hook with a 1s wall-clock deadline:
https://github.com/apache/maka/blob/main/packages/runtime-host/src/__tests__/host-kernel.test.ts#L1463
Reaching that hook requires real storage-root resolution, owner acquisition, drain request, and the recover rejection to propagate. On a loaded CI runner that chain can exceed 1s, so the timeout fires before the kernel is actually wrong. The reported durations (1053ms / 1080ms) sit right at the deadline, which is consistent with a too-tight timeout rather than a real lifecycle regression.
## Possible fixes
- Raise the `withTimeout` budget for this test (and the sibling startup-deadline assertion) to something that tolerates loaded runners, while keeping `shutdownGraceMs` semantics unchanged.
- Or drive the deadline off a fake/injected clock so the assertion no longer depends on wall-clock scheduling.
Whoever picks this up should confirm the fix keeps the test failing when the kernel genuinely skips `close`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [I] test(runtime-host): flaky 'startup failure preserves its cause when shutdown reaches the active deadline' [maka]
Posted by "Sma1lboy (via GitHub)" <gi...@apache.org>.
Sma1lboy commented on issue #3840:
URL: https://github.com/apache/maka/issues/3840#issuecomment-5419850473
I can take this, and I have a local reproduction plus some measurements that narrow down where the time actually goes.
## Reproduced locally
I could not reproduce this on an idle machine — the phase is roughly 10x under the deadline there. It reproduces reliably once the host is both **CPU-starved** and **running the suite concurrently**, which matches the loaded-runner theory in the issue.
Setup on an M-series mac: clamp everything to the efficiency cores with `taskpolicy -c background`, run `@maka/runtime-host`, `@maka/core` and `maka-agent` suites concurrently (mirroring `--concurrency=3`), and time the same arrangement the test uses — real kernel, real storage root, real owner lock, asserting the same `AggregateError` / cause / lifecycle the test asserts.
`closeEntered` latency, deadline is 1000ms:
| Condition | median | max | over 1000ms |
|---|---|---|---|
| idle | 100ms | 122ms | 0/8 |
| efficiency cores only | 436ms | 468ms | 0/6 |
| efficiency cores + 3 suites concurrent | — | 10035ms | **9/10** |
The individual overruns in that last row: 1000.3, 1158.3, 1269.1, 1400.2, 1545.1, 1722.2, 3214.8, 3465.1, and one that never arrived within 10s. The run that passed came in at 934.9ms — 65ms of headroom.
The CI-reported 1053ms and 1080ms sit right inside that distribution, which I think confirms the diagnosis: the deadline is too tight for a loaded runner, not a lifecycle regression.
## Where the time goes
Per-phase breakdown from the passing 934.9ms run:
```
root resolve 11.6ms
owner acquire 60.2ms
-> factory 849.5ms <-- the whole overrun lives here
-> close 13.6ms
```
Storage-root resolution and owner acquisition stay cheap under load. Almost all of the variance is in reaching the composition factory. Idle, that same phase is ~75ms.
## On the fix
I verified the acceptance criterion from the issue before proposing anything — that a raised budget must still catch a kernel that genuinely skips `close`:
```
budget = 10000ms
normal composition -> timed out: false OK: passes
composition skips close -> timed out: true OK: still caught
```
So raising the budget does not blunt the assertion. That said, I would rather go with the injected-clock option than pick a bigger wall-clock number — the numbers above spread across more than a 10x range under load, so any fixed budget is a guess about the slowest runner rather than a statement about the kernel.
**Two scope questions for a maintainer before I open a PR:**
1. `withTimeout(..., 1_000, ...)` appears at 20+ call sites in this file, and the issue mentions "the sibling startup-deadline assertion". Should this PR fix only the two sites named, or take the whole file? I would rather not quietly expand the blast radius.
2. Separately, I have hit a *different* flake in the same file on a loaded machine: `does not take over a generation-mismatched Host before it is ready` (`host-kernel.test.ts:1183`). Different failure mode — a state assertion on `handshake?.state`, not a `withTimeout` — so likely the same load sensitivity but not the same fix. Happy to fold it in or file it separately, whichever you prefer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [I] test(runtime-host): flaky 'startup failure preserves its cause when shutdown reaches the active deadline' [maka]
Posted by "sunheyi6 (via GitHub)" <gi...@apache.org>.
sunheyi6 commented on issue #3840:
URL: https://github.com/apache/maka/issues/3840#issuecomment-5419816039
Fixed in #3844: raised both `withTimeout` budgets in the test to 5s (matching the loaded-CI-tolerant waits already used in this file), keeping `shutdownGraceMs: 100` unchanged. Verified the raised budget still fails with `composition close did not begin` when the `close` hook is genuinely skipped (removed `markCloseEntered()` as a mutation check).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [I] test(runtime-host): flaky 'startup failure preserves its cause when shutdown reaches the active deadline' [maka]
Posted by "Astro-Han (via GitHub)" <gi...@apache.org>.
Astro-Han closed issue #3840: test(runtime-host): flaky 'startup failure preserves its cause when shutdown reaches the active deadline'
URL: https://github.com/apache/maka/issues/3840
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org