You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alexanderbianchi (via GitHub)" <gi...@apache.org> on 2026/10/23 19:24:34 UTC
[PR] feat(substrait): add a hook for consuming relation inputs [datafusion]
alexanderbianchi opened a new pull request, #26679:
URL: https://github.com/apache/datafusion/pull/26679
## Which issue does this PR close?
No linked issue. This extracts the extension point from a larger concurrent-consumer change; parallel default-consumer behavior will be proposed separately.
## Rationale for this change
Custom Substrait consumers cannot overlap independent JOIN/UNION input consumption while reusing upstream operator assembly. Those helpers await each child directly. Downstream consumers currently need to override the operators and wrap already-consumed children just to control scheduling.
## What changes are included in this PR?
Add `SubstraitConsumer::consume_rel_inputs(&[&Rel]) -> Result<Vec<LogicalPlan>>` and call it from JOIN and UNION assembly. The default consumes inputs sequentially and stops on the first consumption error. Overrides must return one plan per input in input order; assembly checks the count.
This PR adds no branch cloning, concurrency, caching, semaphore, or dependency. JOIN/UNION assembly still handles predicates, qualification, and distinctness. Invalid JOINs with missing children return an error instead of panicking.
## What is the testing strategy for this PR?
Tests exercise the public consumer trait: sequential default consumption with a gated first input, error short-circuiting, custom-hook dispatch for JOIN/UNION ALL/UNION DISTINCT, input order, and rejection of incorrect result counts. Existing Substrait integration tests cover operator semantics.
`cargo fmt --all` and workspace `cargo clippy --profile ci --all-targets --all-features -- -D warnings` pass on base `6bb92bb3f`. The focused Substrait suite passed 70 unit tests, 228 integration tests (6 ignored), and 3 doctests before the base refresh; the hook code is unchanged. Extended workspace tests are running. The standalone lint suite could not start because system Python lacks PyYAML.
## Are there any user-facing changes?
Adds a provided trait method, so existing consumer implementations remain source-compatible and keep sequential input consumption. Custom consumers can override scheduling without replacing operator assembly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(substrait): add a hook for consuming relation inputs [datafusion]
Posted by "alexanderbianchi (via GitHub)" <gi...@apache.org>.
alexanderbianchi closed pull request #26679: feat(substrait): add a hook for consuming relation inputs
URL: https://github.com/apache/datafusion/pull/26679
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org