You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "akashchamp (via GitHub)" <gi...@apache.org> on 2026/09/24 16:24:47 UTC
[PR] [Bug]: Unpersist cached Datasets after evaluation in Spark Structured Streaming runner [beam]
akashchamp opened a new pull request, #40265:
URL: https://github.com/apache/beam/pull/40265
Fixes #40243
### Problem
In the Spark Structured Streaming runner, `ParDoTranslatorBatch` explicitly
`persist()`s the intermediate, wide-row `Dataset` it builds for a multi-output
`ParDo` (so the per-`mapPartitions` transformation backing every tagged output
isn't re-evaluated once per tag). The code even carried a `FIXME` acknowledging
the gap:
```java
// FIXME What's the strategy to unpersist Datasets / RDDs?
```
No corresponding `unpersist()` was ever called, so every multi-output `ParDo`
in a pipeline leaves its cached `Dataset` in Spark's storage memory for the
lifetime of the `SparkSession`. On a large batch pipeline with several such
transforms this accumulates without bound and leads to storage-memory
exhaustion, disk spilling, and OOMs.
### Fix
- Added `PipelineTranslator.TranslationState#cacheDataset(Dataset, StorageLevel)`,
a small helper translators use instead of calling `Dataset#persist` directly
when they need to cache a dataset that isn't going through the normal
`putDataset` path. It persists the dataset and registers it for later
cleanup.
- `PipelineTranslator.TranslatingVisitor` now tracks every dataset cached this
way and threads that collection through to `EvaluationContext`.
- `EvaluationContext#evaluate()` now unpersists all such datasets in a
`finally` block once evaluation of the leaves has finished — whether it
completed normally, was stopped early, or an evaluation failed with an
exception — so a cached dataset never outlives the pipeline run that created
it.
- `ParDoTranslatorBatch` now calls `cxt.cacheDataset(...)` instead of calling
`persist()` directly on the wide-row `Dataset`, removing the `FIXME`.
### Testing
- `EvaluationContextTest`: added two new tests asserting that datasets
registered via the new cache-tracking collection are `unpersist()`-ed once
`evaluate()` finishes, both on the happy path and when evaluating a leaf
throws.
- `ParDoTest`: updated the two existing tests that assert a multi-output
`ParDo` produces a cached RDD/Dataset (`testPardoWithOutputTagsCachedRDD` /
`...CachedDataset`, renamed to `...UnpersistsRDDAfterRun` /
`...UnpersistsDatasetAfterRun`) so that, after the pipeline run completes,
they assert the opposite: `SESSION.hasCachedData()` is now `false`, i.e. the
dataset was unpersisted rather than left cached.
- Reproduced the leak first: ran the two updated `ParDoTest` regression tests
(with their new "unpersisted after run" assertions) against the unfixed
production code — both fail with
`AssertionError: Expected cached data to be unpersisted after run`,
confirming `SESSION.hasCachedData()` is still `true` after the pipeline run
completes, i.e. the `Dataset` really is left cached.
- Re-applied the fix and ran the full `EvaluationContextTest` and `ParDoTest`
suites via `./gradlew :runners:spark:3:test --tests
...EvaluationContextTest --tests ...batch.ParDoTest`: 14 tests, 14 passed,
0 failed. I did not run the `runners:spark:4` variant locally (shares the
same `src/` sources via `runners/spark/4`); CI covers it.
### Notes
This only changes the Spark Structured Streaming runner's translation-time
bookkeeping; it does not change what gets computed or written, only when the
intermediate cached `Dataset` used to fan out a multi-output `ParDo` into its
per-tag `Dataset`s is released.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Bug]: Unpersist cached Datasets after evaluation in Spark Structured Streaming runner [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #40265:
URL: https://github.com/apache/beam/pull/40265#issuecomment-6215564497
Reminder, please take a look at this pr: @chamikaramj
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Bug]: Unpersist cached Datasets after evaluation in Spark Structured Streaming runner [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #40265:
URL: https://github.com/apache/beam/pull/40265#issuecomment-5819707474
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Bug]: Unpersist cached Datasets after evaluation in Spark Structured Streaming runner [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #40265:
URL: https://github.com/apache/beam/pull/40265#issuecomment-5969821220
Reminder, please take a look at this pr: @chamikaramj
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Bug]: Unpersist cached Datasets after evaluation in Spark Structured Streaming runner [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #40265:
URL: https://github.com/apache/beam/pull/40265#issuecomment-5822639369
Assigning reviewers:
R: @chamikaramj added as fallback since no labels match configuration
Note: If you would like to opt out of this review, comment `assign to next reviewer`.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Bug]: Unpersist cached Datasets after evaluation in Spark Structured Streaming runner [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #40265:
URL: https://github.com/apache/beam/pull/40265#issuecomment-6038032094
Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @chamikaramj added as fallback since no labels match configuration
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Bug]: Unpersist cached Datasets after evaluation in Spark Structured Streaming runner [beam]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #40265:
URL: https://github.com/apache/beam/pull/40265#issuecomment-6296397536
## [Codecov](https://app.codecov.io/gh/apache/beam/pull/40265?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `74.07407%` with `7 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 58.73%. Comparing base ([`76f6e6d`](https://app.codecov.io/gh/apache/beam/commit/76f6e6d7c0c1b2ce582c73d068ad9260cf15c5f3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`b20bf04`](https://app.codecov.io/gh/apache/beam/commit/b20bf0489be52d19a8723b3128095ced72fd5b51?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 249 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/beam/pull/40265?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...cturedstreaming/translation/EvaluationContext.java](https://app.codecov.io/gh/apache/beam/pull/40265?src=pr&el=tree&filepath=runners%2Fspark%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fbeam%2Frunners%2Fspark%2Fstructuredstreaming%2Ftranslation%2FEvaluationContext.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cnVubmVycy9zcGFyay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvYmVhbS9ydW5uZXJzL3NwYXJrL3N0cnVjdHVyZWRzdHJlYW1pbmcvdHJhbnNsYXRpb24vRXZhbHVhdGlvbkNvbnRleHQuamF2YQ==) | 61.11% | [5 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/beam/pull/40265?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #40265 +/- ##
============================================
+ Coverage 56.16% 58.73% +2.56%
- Complexity 2288 13920 +11632
============================================
Files 1124 2585 +1461
Lines 178336 271705 +93369
Branches 1489 11243 +9754
============================================
+ Hits 100170 159589 +59419
- Misses 75645 106112 +30467
- Partials 2521 6004 +3483
```
| [Flag](https://app.codecov.io/gh/apache/beam/pull/40265/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [java](https://app.codecov.io/gh/apache/beam/pull/40265/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `65.26% <74.07%> (-10.52%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/beam/pull/40265?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Bug]: Unpersist cached Datasets after evaluation in Spark Structured Streaming runner [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #40265:
URL: https://github.com/apache/beam/pull/40265#issuecomment-6290911486
Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @kennknowles added as fallback since no labels match configuration
Backup reviewer: damccorm (if the main reviewer is unresponsive, you can escalate to them by tagging their username)
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org