You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "rluvaton (via GitHub)" <gi...@apache.org> on 2026/07/29 09:21:06 UTC
[PR] feat: drop generator on error [datafusion]
rluvaton opened a new pull request, #23967:
URL: https://github.com/apache/datafusion/pull/23967
## Which issue does this PR close?
N/A
## Rationale for this change
Drop generator on error so the memory is freed faster
## What changes are included in this PR?
Replaced done with Option and set
## Are these changes tested?
Yes
## Are there any user-facing changes?
Clear faster
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: drop generator on error [datafusion]
Posted by "Weijun-H (via GitHub)" <gi...@apache.org>.
Weijun-H commented on code in PR #23967:
URL: https://github.com/apache/datafusion/pull/23967#discussion_r3672932208
##########
datafusion/execution/src/async_stream.rs:
##########
@@ -655,6 +665,59 @@ mod test {
);
}
+ struct DropGuard(Arc<AtomicUsize>);
+
+ impl Drop for DropGuard {
+ fn drop(&mut self) {
+ self.0.fetch_add(1, Ordering::SeqCst);
+ }
+ }
+
+ #[tokio::test]
+ async fn generator_freed_on_done() {
+ let drops = Arc::new(AtomicUsize::new(0));
+ let guard = DropGuard(Arc::clone(&drops));
+
+ let s = async_stream(|mut emitter| async move {
+ let _guard = guard;
+ emitter.emit(1).await;
+ });
+ pin_mut!(s);
+
+ assert_eq!(s.next().await, Some(1));
+ assert_eq!(s.next().await, None);
+
+ // State captured by the generator is dropped as soon as it completes
+ // (async blocks drop their locals on return), even though the stream
+ // itself is still alive
+ assert_eq!(drops.load(Ordering::SeqCst), 1);
+ assert_eq!(s.next().await, None);
+ }
+
+ #[tokio::test]
+ async fn generator_freed_on_emitted_error() {
+ let drops = Arc::new(AtomicUsize::new(0));
+ let guard = DropGuard(Arc::clone(&drops));
+
+ let s = async_try_stream(|mut emitter| async move {
+ let _guard = guard;
+ emitter.emit(1).await;
+ Err("boom")
+ });
+ pin_mut!(s);
+
+ assert_eq!(s.next().await, Some(Ok(1)));
+ assert_eq!(s.next().await, Some(Err("boom")));
+
+ // The stream terminates in the same poll that yields the error, so the
+ // generator state is freed even if the consumer never polls again
+ assert!(s.is_terminated());
+ assert_eq!(drops.load(Ordering::SeqCst), 1);
Review Comment:
The inner generator drops its locals when it returns Err, before the outer emission future suspends. So the drop count reaches 1 on the old path too; only is_terminated() differs.
Suggest reframing the PR and test around the real change: is_terminated()/size_hint flipping in the same poll as the final error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: drop generator on error [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #23967:
URL: https://github.com/apache/datafusion/pull/23967#issuecomment-5117308192
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/23967?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 80.75%. Comparing base ([`1c3232c`](https://app.codecov.io/gh/apache/datafusion/commit/1c3232ce2262523fb4043e7264185b2f8f929b0d?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`a4a15a6`](https://app.codecov.io/gh/apache/datafusion/commit/a4a15a6897ba5c4c109e967cab76006dd15af2fd?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 23 commits behind head on main.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #23967 +/- ##
==========================================
+ Coverage 80.67% 80.75% +0.08%
==========================================
Files 1095 1096 +1
Lines 372376 373320 +944
Branches 372376 373320 +944
==========================================
+ Hits 300397 301459 +1062
+ Misses 54052 53881 -171
- Partials 17927 17980 +53
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/23967?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: drop generator on error to free memory faster [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #23967:
URL: https://github.com/apache/datafusion/pull/23967#issuecomment-5147841449
FYI @pepijnve in case you are interested
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: drop generator on error to free memory faster [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #23967:
URL: https://github.com/apache/datafusion/pull/23967
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: drop generator on error to free memory faster [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #23967:
URL: https://github.com/apache/datafusion/pull/23967#issuecomment-5170458039
Thanks everyone
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: drop generator on error to free memory faster [datafusion]
Posted by "pepijnve (via GitHub)" <gi...@apache.org>.
pepijnve commented on PR #23967:
URL: https://github.com/apache/datafusion/pull/23967#issuecomment-5150212791
Thanks for the ping. Proposed change looks good to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org