You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "kosiew (via GitHub)" <gi...@apache.org> on 2026/09/28 11:37:04 UTC
[PR] Return provider extensions directly from the statistics walk [datafusion]
kosiew opened a new pull request, #25826:
URL: https://github.com/apache/datafusion/pull/25826
## Which issue does this PR close?
Part of #25141.
## Rationale for this change
`compute_extended` currently reads a root node’s provider extensions back from the cache after computing its statistics. A planned change will stop caching borrowed roots, so that read would lose their extensions. Returning the extensions from the statistics walk preserves the result without depending on a root cache entry.
## What changes are included in this PR?
- Return statistics and provider extensions together from the private statistics walk. `compute_extended` uses that result; `compute` keeps returning only the statistics.
- Continue caching provider extensions for parent nodes. On a cache hit, read them only when the caller requests them.
- Replace `ExtendedStatistics::extensions()` with `into_parts()` to move its statistics and extensions into the walk’s result.
The public signatures of `compute` and `compute_extended` are unchanged.
## Are these changes tested?
The patch adds `cached_node_keeps_extensions` and `walk_returns_provider_extensions_directly`. They cover extensions returned after a child is cached, extensions returned from a fresh provider result, and cache reads that occur only when requested. The provided diff does not show test execution results.
## Are there any user-facing changes?
No user-facing change is intended. This refactor preserves provider extension behavior ahead of the planned cache change.
## LLM-generated code disclosure
This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Return provider extensions directly from the statistics walk [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew merged PR #25826:
URL: https://github.com/apache/datafusion/pull/25826
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Return provider extensions directly from the statistics walk [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #25826:
URL: https://github.com/apache/datafusion/pull/25826#issuecomment-5869452442
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/25826?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `88.23529%` with `4 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 82.55%. Comparing base ([`6a792c6`](https://app.codecov.io/gh/apache/datafusion/commit/6a792c671376c6c91f778530220358eb70a91271?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`c27f226`](https://app.codecov.io/gh/apache/datafusion/commit/c27f2261fb48c176c3f49c9d626f6e86486ced50?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 20 commits behind head on main.
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/25826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [datafusion/physical-plan/src/statistics.rs](https://app.codecov.io/gh/apache/datafusion/pull/25826?src=pr&el=tree&filepath=datafusion%2Fphysical-plan%2Fsrc%2Fstatistics.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9waHlzaWNhbC1wbGFuL3NyYy9zdGF0aXN0aWNzLnJz) | 87.50% | [1 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/25826?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #25826 +/- ##
==========================================
+ Coverage 82.49% 82.55% +0.05%
==========================================
Files 1140 1141 +1
Lines 438555 440219 +1664
Branches 438555 440219 +1664
==========================================
+ Hits 361803 363402 +1599
+ Misses 54880 54840 -40
- Partials 21872 21977 +105
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/25826?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Return provider extensions directly from the statistics walk [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on PR #25826:
URL: https://github.com/apache/datafusion/pull/25826#issuecomment-5902472663
🚀
@asolimando
Thank you for your review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Return provider extensions directly from the statistics walk [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on PR #25826:
URL: https://github.com/apache/datafusion/pull/25826#issuecomment-5925534434
🚀
@zhuqi-lucas
Thank you for your contribution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org