You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@systemds.apache.org by "janniklinde (via GitHub)" <gi...@apache.org> on 2026/03/03 09:42:26 UTC
[PR] [SYSTEMDS-3891] OOC k-Means Support [systemds]
janniklinde opened a new pull request, #2442:
URL: https://github.com/apache/systemds/pull/2442
This patch adds support for the k-means builtin script in the out-of-core backend by introducing new instructions, handling `parfor`.
Additionally, this patch improves performance of the `OOCLRUCacheScheduler` under memory pressure by debouncing cleanup runs, which previously caused stalls due to long synchronization barriers.
Finally, this patch provides bugfixes for various instructions and stream lifecycle handling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [SYSTEMDS-3891] OOC k-Means Support [systemds]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #2442:
URL: https://github.com/apache/systemds/pull/2442#issuecomment-3990088997
## [Codecov](https://app.codecov.io/gh/apache/systemds/pull/2442?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `58.14932%` with `398 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 71.78%. Comparing base ([`743c902`](https://app.codecov.io/gh/apache/systemds/commit/743c9022f6a2a7c6701e7431f09d53210f8853ea?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`6e7439f`](https://app.codecov.io/gh/apache/systemds/commit/6e7439fc880fa7b520e269e3bd30788689ce051e?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 3 commits behind head on main.
| [Files with missing lines](https://app.codecov.io/gh/apache/systemds/pull/2442?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...instructions/ooc/MatrixIndexingOOCInstruction.java](https://app.codecov.io/gh/apache/systemds/pull/2442?src=pr&el=tree&filepath=src%2Fmain%2Fjava%2Forg%2Fapache%2Fsysds%2Fruntime%2Finstructions%2Fooc%2FMatrixIndexingOOCInstruction.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3N5c2RzL3J1bnRpbWUvaW5zdHJ1Y3Rpb25zL29vYy9NYXRyaXhJbmRleGluZ09PQ0luc3RydWN0aW9uLmphdmE=) | 60.96% | [63 Missing and 26 partials :warning: ](https://app.codecov.io/gh/apache/systemds/pull/2442?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../sysds/runtime/ooc/cache/OOCLRUCacheScheduler.java](https://app.codecov.io/gh/apache/systemds/pull/2442?src=pr&el=tree&filepath=src%2Fmain%2Fjava%2Forg%2Fapache%2Fsysds%2Fruntime%2Fooc%2Fcache%2FOOCLRUCacheScheduler.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3N5c2RzL3J1bnRpbWUvb29jL2NhY2hlL09PQ0xSVUNhY2hlU2NoZWR1bGVyLmphdmE=) | 64.32% | [36 Missing and 25 partials :warning: ](https://app.codecov.io/gh/apache/systemds/pull/2442?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...he/sysds/runtime/instructions/ooc/OOCWatchdog.java](https://app.codecov.io/gh/apache/systemds/pull/2442?src=pr&el=tree&filepath=src%2Fmain%2Fjava%2Forg%2Fapache%2Fsysds%2Fruntime%2Finstructions%2Fooc%2FOOCWatchdog.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3N5c2RzL3J1bnRpbWUvaW5zdHJ1Y3Rpb25zL29vYy9PT0NXYXRjaGRvZy5qYXZh) | 0.00% | [56 Missing :warning: ](https://app.codecov.io/gh/apache/systemds/pull/2442?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...sysds/runtime/instructions/ooc/OOCInstruction.java](https://app.codecov.io/gh/apache/systemds/pull/2442?src=pr&el=tree&filepath=src%2Fmain%2Fjava%2Forg%2Fapache%2Fsysds%2Fruntime%2Finstructions%2Fooc%2FOOCInstruction.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3N5c2RzL3J1bnRpbWUvaW5zdHJ1Y3Rpb25zL29vYy9PT0NJbnN0cnVjdGlvbi5qYXZh) | 58.47% | [33 Missing and 16 partials :warning: ](https://app.codecov.io/gh/apache/systemds/pull/2442?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...rg/apache/sysds/runtime/ooc/stats/OOCEventLog.java](https://app.codecov.io/gh/apache/systemds/pull/2442?src=pr&el=tree&filepath=src%2Fmain%2Fjava%2Forg%2Fapache%2Fsysds%2Fruntime%2Fooc%2Fstats%2FOOCEventLog.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3N5c2RzL3J1bnRpbWUvb29jL3N0YXRzL09PQ0V2ZW50TG9nLmphdmE=) | 0.00% | [37 Missing :warning: ](https://app.codecov.io/gh/apache/systemds/pull/2442?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../sysds/runtime/instructions/ooc/CachingStream.java](https://app.codecov.io/gh/apache/systemds/pull/2442?src=pr&el=tree&filepath=src%2Fmain%2Fjava%2Forg%2Fapache%2Fsysds%2Fruntime%2Finstructions%2Fooc%2FCachingStream.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3N5c2RzL3J1bnRpbWUvaW5zdHJ1Y3Rpb25zL29vYy9DYWNoaW5nU3RyZWFtLmphdmE=) | 68.42% | [20 Missing and 10 partials :warning: ](https://app.codecov.io/gh/apache/systemds/pull/2442?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../runtime/instructions/ooc/UnaryOOCInstruction.java](https://app.codecov.io/gh/apache/systemds/pull/2442?src=pr&el=tree&filepath=src%2Fmain%2Fjava%2Forg%2Fapache%2Fsysds%2Fruntime%2Finstructions%2Fooc%2FUnaryOOCInstruction.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3N5c2RzL3J1bnRpbWUvaW5zdHJ1Y3Rpb25zL29vYy9VbmFyeU9PQ0luc3RydWN0aW9uLmphdmE=) | 78.57% | [7 Missing and 8 partials :warning: ](https://app.codecov.io/gh/apache/systemds/pull/2442?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...runtime/instructions/ooc/BinaryOOCInstruction.java](https://app.codecov.io/gh/apache/systemds/pull/2442?src=pr&el=tree&filepath=src%2Fmain%2Fjava%2Forg%2Fapache%2Fsysds%2Fruntime%2Finstructions%2Fooc%2FBinaryOOCInstruction.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3N5c2RzL3J1bnRpbWUvaW5zdHJ1Y3Rpb25zL29vYy9CaW5hcnlPT0NJbnN0cnVjdGlvbi5qYXZh) | 0.00% | [11 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/systemds/pull/2442?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...sds/runtime/controlprogram/ParForProgramBlock.java](https://app.codecov.io/gh/apache/systemds/pull/2442?src=pr&el=tree&filepath=src%2Fmain%2Fjava%2Forg%2Fapache%2Fsysds%2Fruntime%2Fcontrolprogram%2FParForProgramBlock.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3N5c2RzL3J1bnRpbWUvY29udHJvbHByb2dyYW0vUGFyRm9yUHJvZ3JhbUJsb2NrLmphdmE=) | 72.97% | [4 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/systemds/pull/2442?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ds/runtime/instructions/ooc/TeeOOCInstruction.java](https://app.codecov.io/gh/apache/systemds/pull/2442?src=pr&el=tree&filepath=src%2Fmain%2Fjava%2Forg%2Fapache%2Fsysds%2Fruntime%2Finstructions%2Fooc%2FTeeOOCInstruction.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3N5c2RzL3J1bnRpbWUvaW5zdHJ1Y3Rpb25zL29vYy9UZWVPT0NJbnN0cnVjdGlvbi5qYXZh) | 0.00% | [6 Missing :warning: ](https://app.codecov.io/gh/apache/systemds/pull/2442?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| ... and [15 more](https://app.codecov.io/gh/apache/systemds/pull/2442?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #2442 +/- ##
============================================
+ Coverage 71.72% 71.78% +0.06%
- Complexity 48011 48219 +208
============================================
Files 1554 1554
Lines 184671 185352 +681
Branches 36248 36381 +133
============================================
+ Hits 132453 133058 +605
- Misses 41964 41998 +34
- Partials 10254 10296 +42
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/systemds/pull/2442?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [SYSTEMDS-3891] OOC k-Means Support [systemds]
Posted by "janniklinde (via GitHub)" <gi...@apache.org>.
janniklinde merged PR #2442:
URL: https://github.com/apache/systemds/pull/2442
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org