You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@systemds.apache.org by "Baunsgaard (via GitHub)" <gi...@apache.org> on 2026/06/23 23:16:57 UTC
[PR] [MINOR][CI] Raise federated request timeout for multitenant Spark tests [systemds]
Baunsgaard opened a new pull request, #2505:
URL: https://github.com/apache/systemds/pull/2505
The `**.functions.federated.monitoring.**,**.functions.federated.multitenant.**` CI shard has been intermittently failing and, when reruns pile up, timing out at the 30-minute job cap (e.g. [run 28041416088](https://github.com/apache/systemds/actions/runs/28041416088/job/83008345704)).
Root cause: the multi-tenant test config pins `sysds.federated.timeout` to **16s**. This value bounds *both* federated instruction execution (`FederationMap`/`FederatedData`) and end-of-run stats collection (`FederatedStatistics.collectFedStats`). For the Spark-backed (`*SP`) variants of the reuse tests, Spark context creation alone is ~14s, so under shared CI load a single federated request regularly exceeds 16s and throws `TimeoutException`:
- `SPARK°rightIndex°… DMLRuntimeException -- java.util.concurrent.TimeoutException`
- `Exception … thrown while getting the federated stats of the federated response`
The spurious timeouts make `FederatedReuseReadTest.testModifiedValLineageSP` and `FederatedSerializationReuseTest.testRowSumsSP` fail; surefire then reruns each failing test, and the accumulated rerun time repeatedly pushes the shard past the 30-minute cap, cancelling the whole job.
This bumps the timeout to **60s**: still a hard bound on a genuinely runaway request (the suite cannot hang silently — the reason it was lowered from 128 → 16 in `8f5a42c0`), but enough headroom for the Spark variants to pass on the first attempt, which also removes the expensive reruns and brings the shard comfortably back under the time cap.
## Evidence (recurring flake, last ~2 weeks)
Same two tests, same `TimeoutException` signature, across independent runs:
| Run | Date | Failing test(s) |
|---|---|---|
| [27910160773](https://github.com/apache/systemds/actions/runs/27910160773) | Jun 21 | `testModifiedValLineageSP`, `testRowSumsSP` |
| [27645002589](https://github.com/apache/systemds/actions/runs/27645002589) | Jun 16 | `testModifiedValLineageSP` |
| [27542041413](https://github.com/apache/systemds/actions/runs/27542041413) | Jun 15 | `testModifiedValLineageSP`, `testPlusScalarCP`, `testRowSumsSP` |
| [27363185073](https://github.com/apache/systemds/actions/runs/27363185073) | Jun 11 | `testModifiedValLineageSP`, `testRow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR][CI] Raise federated request timeout for multitenant Spark tests [systemds]
Posted by "Baunsgaard (via GitHub)" <gi...@apache.org>.
Baunsgaard merged PR #2505:
URL: https://github.com/apache/systemds/pull/2505
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR][CI] Raise federated request timeout for multitenant Spark tests [systemds]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #2505:
URL: https://github.com/apache/systemds/pull/2505#issuecomment-4784641963
## [Codecov](https://app.codecov.io/gh/apache/systemds/pull/2505?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 71.51%. Comparing base ([`3871809`](https://app.codecov.io/gh/apache/systemds/commit/38718092696497aa354db73e828d68f94ad51620?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`f02420a`](https://app.codecov.io/gh/apache/systemds/commit/f02420a53176cf5b388bfa7b29821ac0b8e5878b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 3 commits behind head on main.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #2505 +/- ##
============================================
+ Coverage 71.45% 71.51% +0.06%
- Complexity 48864 48947 +83
============================================
Files 1573 1573
Lines 189239 189334 +95
Branches 37128 37149 +21
============================================
+ Hits 135215 135406 +191
+ Misses 43576 43456 -120
- Partials 10448 10472 +24
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/systemds/pull/2505?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org