You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2026/04/25 19:34:39 UTC
[PR] Draft: implement non-blocking morsel API [datafusion]
alamb opened a new pull request, #21852:
URL: https://github.com/apache/datafusion/pull/21852
TODO
- [ ] Tests
- [ ] Make a PR to refactor the large `poll_scan` loop into separate functions to reduce the indent level / control flow
## Which issue does this PR close?
- Closes https://github.com/apache/datafusion/issues/21598
## Rationale for this change
In https://github.com/apache/datafusion/pull/21342#discussion_r3070184104, @adriangb pointed out that the current [Morsel](https://github.com/apache/datafusion/blob/04dbbbf6694a4b162f76aee0091fdc3a47d2f9f0/datafusion/datasource/src/morsel/mod.rs#L52) API relied on a comment rather than they typesystem to separate IO and CPU.
Also, it should be pointed out that the current Parquet opener actually does now do IO in the stream reader. This makes overlapping the IO and CPU harder
## What changes are included in this PR?
It would be nice to
2. Make the morsel API harder to misuse
1. Avoid IO after morsels are ready
I don't expect this change will have much of an actual impact (yet) but I do expect that it will set us up for better IO interleaving
## Are these changes tested?
yes by CI and new tests (to be written)
## Are there any user-facing changes?
The unreleased Morsel API is slightly different
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Draft: implement non-blocking morsel API [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #21852:
URL: https://github.com/apache/datafusion/pull/21852#issuecomment-4672694734
> @alamb is this something I could help push forward, if you still think it's the right direction? I'm happy to review, etc. Just going through your open PRs to see what I can help with.
Thank you for the ping.
Basically I think to be motivated to move this along I need to find some query where it improves performance. I didn't have such evidence and thus is dropped off my radar.
So in other words, I am not sure what to do with this -- I think it basically "realizes the vision" of morsel driven API but just makes the code more complicated and has no direct benefit yet 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Draft: implement non-blocking morsel API [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #21852:
URL: https://github.com/apache/datafusion/pull/21852#issuecomment-4667070454
@alamb is this something I could help push forward, if you still think it's the right direction? I'm happy to review, etc. Just going through your open PRs to see what I can help with.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Draft: implement non-blocking morsel API [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed pull request #21852: Draft: implement non-blocking morsel API
URL: https://github.com/apache/datafusion/pull/21852
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Draft: implement non-blocking morsel API [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21852:
URL: https://github.com/apache/datafusion/pull/21852#issuecomment-5235155909
Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org