You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Dandandan (via GitHub)" <gi...@apache.org> on 2026/08/04 10:33:54 UTC
[PR] fix(proto): preserve empty projection when ser/de MemoryScanExec [datafusion]
Dandandan opened a new pull request, #24084:
URL: https://github.com/apache/datafusion/pull/24084
## Which issue does this PR close?
Finishes the work started in #23082 (`HashJoinExec`, `NestedLoopJoinExec`) and #21885 (`FilterExec`) — `MemoryScanExec` was the last physical plan node left with an ambiguous projection.
## Rationale for this change
`MemorySourceConfig`'s projection is `Option<Vec<usize>>` but is encoded as a bare `repeated uint32`:
```rust
// encode
let proto_projection = source_conf.projection().as_ref().map_or_else(Vec::new, |v| ...);
// decode
let projection = if !scan.projection.is_empty() { Some(...) } else { None };
```
So `Some(vec![])` — project away every column — serializes exactly like `None` and always decodes back as `None`, and the decoded scan reports every column again.
That silently changes a plan's output schema mid-flight. In a distributed engine it also breaks the parent: a `UnionExec` over a zero-column scan and a zero-column sibling fails on the worker as soon as the plan is decoded, because `UnionExec::try_new` validates field counts:
```
UnionExec/InterleaveExec requires all inputs to have the same number of fields.
Input 0 has 0 fields, but input 1 has 2 fields
```
We hit exactly this on the join side in production (a `count`-style query over a union), which is what #23082 fixed; the memory-scan path has the same defect.
## What changes are included in this PR?
Encode an empty projection with the single-element `[u32::MAX]` sentinel — the same convention `HashJoinExec` and `NestedLoopJoinExec` already use — and decode it back to `Some(vec![])`. Every other state is unchanged on the wire, so `None` and non-empty projections round-trip exactly as before.
## Are these changes tested?
Yes — `roundtrip_memory_source_projections` covers `None`, `Some(vec![])` and a partial projection. Without the fix the `Some(vec![])` case fails with `projection: Some([])` vs `projection: None` and a schema that grows from `[]` back to every column.
(The 6 pre-existing `roundtrip_parquet_*` / `roundtrip_empty_projection` / `roundtrip_physical_plan_node` failures in my environment are unrelated — they fail identically on unmodified `main`. Passing count goes 205 → 206.)
## Are there any user-facing changes?
No API change. The wire format changes only for the previously-unrepresentable `Some(vec![])` case, matching what the joins already do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix(proto): preserve empty projection when ser/de MemoryScanExec [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #24084:
URL: https://github.com/apache/datafusion/pull/24084#issuecomment-5178528680
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/24084?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 80.89%. Comparing base ([`db0c31b`](https://app.codecov.io/gh/apache/datafusion/commit/db0c31bfe16628ed6118ea3fc5d4002c8f2365fb?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`d298fba`](https://app.codecov.io/gh/apache/datafusion/commit/d298fba0319a8f697e30255e8a57a91a7ae00cf3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #24084 +/- ##
==========================================
- Coverage 80.89% 80.89% -0.01%
==========================================
Files 1102 1102
Lines 376111 376107 -4
Branches 376111 376107 -4
==========================================
- Hits 304251 304246 -5
+ Misses 53753 53750 -3
- Partials 18107 18111 +4
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/24084?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix(proto): preserve empty projection when ser/de MemoryScanExec [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan closed pull request #24084: fix(proto): preserve empty projection when ser/de MemoryScanExec
URL: https://github.com/apache/datafusion/pull/24084
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org