You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@systemds.apache.org by "Baunsgaard (via GitHub)" <gi...@apache.org> on 2026/06/26 12:56:02 UTC
[PR] [MINOR] Reference-count shared Spark context to prevent concurrent shutdown [systemds]
Baunsgaard opened a new pull request, #2516:
URL: https://github.com/apache/systemds/pull/2516
The JVM-wide singleton `SparkContext` in `SparkExecutionContext` was stopped unconditionally at the end of every `DMLScript.execute()`. With surefire parallel tests (`parallel=classes`, `threadCount=2`), one finishing execution would stop the context while a concurrent execution still had an in-flight Spark job.
## Change
- Make `close()` reference-counted. `SparkExecutionContext.enterSparkExecution()` is called from `DMLScript.execute()` when the execution context is a `SparkExecutionContext`, and `close()` only stops the shared context once the active-execution count reaches zero.
## Notes
- Single-run behaviour is unchanged (enter -> 1, close -> 0 -> stop).
- The counter is balanced: enter and close run under the same condition at the same single site, with `close()` always in the `finally` block.
- Only the `DMLScript.execute()` path that owns the context is covered; MLContext manages an externally-owned context separately and is unaffected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Reference-count shared Spark context to prevent concurrent shutdown [systemds]
Posted by "Baunsgaard (via GitHub)" <gi...@apache.org>.
Baunsgaard merged PR #2516:
URL: https://github.com/apache/systemds/pull/2516
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Reference-count shared Spark context to prevent concurrent shutdown [systemds]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #2516:
URL: https://github.com/apache/systemds/pull/2516#issuecomment-4809979272
## [Codecov](https://app.codecov.io/gh/apache/systemds/pull/2516?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `81.81818%` with `2 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 71.57%. Comparing base ([`d4e7def`](https://app.codecov.io/gh/apache/systemds/commit/d4e7defc9b1dab1562ff8325c8dad6b83c1dbd50?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`9391528`](https://app.codecov.io/gh/apache/systemds/commit/9391528c2bbd95a63ee9c01d98df8d35c22d9357?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/systemds/pull/2516?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [.../controlprogram/context/SparkExecutionContext.java](https://app.codecov.io/gh/apache/systemds/pull/2516?src=pr&el=tree&filepath=src%2Fmain%2Fjava%2Forg%2Fapache%2Fsysds%2Fruntime%2Fcontrolprogram%2Fcontext%2FSparkExecutionContext.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3N5c2RzL3J1bnRpbWUvY29udHJvbHByb2dyYW0vY29udGV4dC9TcGFya0V4ZWN1dGlvbkNvbnRleHQuamF2YQ==) | 77.77% | [1 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/systemds/pull/2516?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #2516 +/- ##
=========================================
Coverage 71.56% 71.57%
+ Complexity 49125 49118 -7
=========================================
Files 1575 1575
Lines 189784 189795 +11
Branches 37232 37235 +3
=========================================
+ Hits 135823 135844 +21
+ Misses 43470 43462 -8
+ Partials 10491 10489 -2
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/systemds/pull/2516?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org