You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "yihua (via GitHub)" <gi...@apache.org> on 2026/09/05 18:47:26 UTC
[PR] perf(datafusion): let session parquet options drive the cow scan [hudi-rs]
yihua opened a new pull request, #763:
URL: https://github.com/apache/hudi-rs/pull/763
## Description
The COW parquet scan now takes row-filter pushdown, filter reordering, and page-index use from the session's parquet options instead of hardcoding them on, so a Hudi table scan behaves like a plain parquet registration of the same files under the same session config. The scan also only installs predicates for conjuncts DataFusion's physical filter-pushdown rule cannot absorb itself: pushable conjuncts reach the scan through that rule exactly once (for stats pruning, and as a row filter when the session enables pushdown), so the merged scan predicate now carries each conjunct once in both modes. Plan test anchors are updated for the resulting plan shape (the struct-field access is still computed inside the scan, under an optimizer-extracted name). Part of the query analysis work in #713.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
New test pins that the filter stays in a `FilterExec` under default session config, moves into the scan when `pushdown_filters` is enabled, and that the scan predicate carries each conjunct exactly once in both modes (the once-only assertions fail on the previous behavior). Existing plan tests keep pinning struct-field predicate and projection pushdown; the full hudi-datafusion suite passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(datafusion): let session parquet options drive the cow scan [hudi-rs]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #763:
URL: https://github.com/apache/hudi-rs/pull/763#issuecomment-5554106005
## [Codecov](https://app.codecov.io/gh/apache/hudi-rs/pull/763?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `60.00000%` with `4 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 79.47%. Comparing base ([`ed3653b`](https://app.codecov.io/gh/apache/hudi-rs/commit/ed3653b42fab43a6747b1031c5d5e0bfde4fac67?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`91d7778`](https://app.codecov.io/gh/apache/hudi-rs/commit/91d7778afd84b6a193986411edcf7048b8c79494?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 7 commits behind head on main.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi-rs/pull/763?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [crates/datafusion/src/lib.rs](https://app.codecov.io/gh/apache/hudi-rs/pull/763?src=pr&el=tree&filepath=crates%2Fdatafusion%2Fsrc%2Flib.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y3JhdGVzL2RhdGFmdXNpb24vc3JjL2xpYi5ycw==) | 42.85% | [4 Missing :warning: ](https://app.codecov.io/gh/apache/hudi-rs/pull/763?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #763 +/- ##
==========================================
- Coverage 79.49% 79.47% -0.03%
==========================================
Files 126 126
Lines 12369 12374 +5
==========================================
+ Hits 9833 9834 +1
- Misses 2536 2540 +4
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/hudi-rs/pull/763?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(datafusion): let session parquet options drive the cow scan [hudi-rs]
Posted by "yihua (via GitHub) via commits" <co...@hudi.apache.org>.
yihua commented on PR #763:
URL: https://github.com/apache/hudi-rs/pull/763#issuecomment-6717065393
Pushed a minor revision in the latest commit, after merging main: the provider no longer installs its own scan predicate (the pushdown rule supplies it once in every mode), partition equality filters are `Inexact` so a `FilterExec` still rejects rows partition pruning keeps, with a regression test for NULLs in the default partition, and the docs note that late materialization now needs `pushdown_filters`. LGTM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org