You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "asolimando (via GitHub)" <gi...@apache.org> on 2026/09/24 08:38:52 UTC
[PR] docs(physical-plan): add doctest examples for StatisticsRegistry, ExtendedStatistics, and compute_extended [datafusion]
asolimando opened a new pull request, #25699:
URL: https://github.com/apache/datafusion/pull/25699
## Which issue does this PR close?
Closes #25698.
## Rationale for this change
`StatisticsRegistry` and `ExtendedStatistics` each had a `# Example` doc block wrapped in
` ```ignore `, so it was never compiled or checked by `cargo test --doc`.
`StatisticsContext::compute_extended` had no example at all.
## What changes are included in this PR?
- Add a real, compiled `# Example` to `StatisticsContext::compute_extended`.
- Convert the `StatisticsRegistry` module doc example from `ignore` to a real doctest.
- Convert the `ExtendedStatistics` struct doc example from `ignore` to a real doctest.
- The shared running example is a cross-column correlation extension
(`ColumnCorrelation`), since it is a genuinely node-level statistic, unlike a
per-column histogram, which has no home in the current single node-level
`Extensions` field.
## Are these changes tested?
Yes, each doc comment is now a doctest, verified by `cargo test --doc`.
## Are there any user-facing changes?
No, this is a documentation-only change to public API doc comments.
----
Disclaimer: I used AI to assist in the code generation, I have manually reviewed the output and it matches my intention and understanding.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs(physical-plan): add doctest examples for StatisticsRegistry, ExtendedStatistics, and compute_extended [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew merged PR #25699:
URL: https://github.com/apache/datafusion/pull/25699
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs(physical-plan): add doctest examples for StatisticsRegistry, ExtendedStatistics, and compute_extended [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on code in PR #25699:
URL: https://github.com/apache/datafusion/pull/25699#discussion_r4192584360
##########
datafusion/physical-plan/src/statistics.rs:
##########
@@ -307,6 +307,73 @@ impl StatisticsContext {
/// for how extensions propagate up the tree). As with [`Self::compute`],
/// the root is not memoized and must be a standalone plan node; see
/// [`Self::compute_extended_arc`].
+ ///
+ /// # Example
+ ///
+ /// ```
+ /// # use std::sync::Arc;
+ /// # use arrow::datatypes::{DataType, Field, Schema};
+ /// # use datafusion_common::{ColumnStatistics, Result, Statistics};
+ /// # use datafusion_common::stats::Precision;
+ /// # use datafusion_physical_plan::ExecutionPlan;
+ /// # use datafusion_physical_plan::operator_statistics::{
+ /// # ExtendedStatistics, StatisticsProvider, StatisticsRegistry, StatisticsResult,
+ /// # };
+ /// # use datafusion_physical_plan::statistics::{StatisticsArgs, StatisticsContext};
+ /// # use datafusion_physical_plan::test::exec::StatisticsExec;
+ ///
+ /// // Pearson correlation between two columns, identified by schema index.
+ /// #[derive(Debug, Clone)]
+ /// struct ColumnCorrelation {
+ /// columns: (usize, usize),
+ /// coefficient: f64,
+ /// }
+ ///
+ /// #[derive(Debug)]
+ /// struct CorrelationProvider;
+ ///
+ /// impl StatisticsProvider for CorrelationProvider {
+ /// fn matches(&self, plan: &dyn ExecutionPlan) -> bool {
+ /// plan.downcast_ref::<StatisticsExec>().is_some()
+ /// }
+ ///
+ /// fn compute_statistics(
+ /// &self,
+ /// plan: &dyn ExecutionPlan,
+ /// _child_stats: &[ExtendedStatistics],
+ /// ) -> Result<StatisticsResult> {
+ /// let base = plan.statistics_from_inputs(&[], &StatisticsArgs::new())?;
+ /// let mut extended = ExtendedStatistics::new_arc(base);
+ /// extended.set_extension(ColumnCorrelation {
+ /// columns: (0, 1),
+ /// coefficient: 0.92,
+ /// });
+ /// Ok(StatisticsResult::Computed(extended))
+ /// }
+ /// }
+ ///
+ /// let schema = Schema::new(vec![
+ /// Field::new("city", DataType::Utf8, false),
+ /// Field::new("zip_code", DataType::Utf8, false),
+ /// ]);
+ /// let stats = Statistics {
+ /// num_rows: Precision::Exact(1000),
+ /// total_byte_size: Precision::Absent,
+ /// column_statistics: vec![ColumnStatistics::new_unknown(); 2],
+ /// };
+ /// let plan = StatisticsExec::new(stats, schema);
+ ///
+ /// let mut registry = StatisticsRegistry::new();
+ /// registry.register(Arc::new(CorrelationProvider));
+ /// let context = StatisticsContext::new_with_registry(registry);
+ ///
+ /// let extended = context.compute_extended(&plan, &StatisticsArgs::new())?;
+ /// assert_eq!(extended.base().num_rows, Precision::Exact(1000));
+ ///
Review Comment:
Optional: could we also assert `correlation.coefficient == 0.92` here? That would make this example show how to read the correlation value as well as the column indexes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs(physical-plan): add doctest examples for StatisticsRegistry, ExtendedStatistics, and compute_extended [datafusion]
Posted by "asolimando (via GitHub)" <gi...@apache.org>.
asolimando commented on PR #25699:
URL: https://github.com/apache/datafusion/pull/25699#issuecomment-5992761100
@kosiew I wonder if you would be interested into reviewing this, since you were the main reviewer of this code and you have recently touched `StatisticsRegistry` doc after your follow-up improvements over the cache life-time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs(physical-plan): add doctest examples for StatisticsRegistry, ExtendedStatistics, and compute_extended [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on PR #25699:
URL: https://github.com/apache/datafusion/pull/25699#issuecomment-6031865744
🚀
@asolimando
Thank you for your contribution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs(physical-plan): add doctest examples for StatisticsRegistry, ExtendedStatistics, and compute_extended [datafusion]
Posted by "asolimando (via GitHub)" <gi...@apache.org>.
asolimando commented on code in PR #25699:
URL: https://github.com/apache/datafusion/pull/25699#discussion_r4192945398
##########
datafusion/physical-plan/src/statistics.rs:
##########
@@ -307,6 +307,73 @@ impl StatisticsContext {
/// for how extensions propagate up the tree). As with [`Self::compute`],
/// the root is not memoized and must be a standalone plan node; see
/// [`Self::compute_extended_arc`].
+ ///
+ /// # Example
+ ///
+ /// ```
+ /// # use std::sync::Arc;
+ /// # use arrow::datatypes::{DataType, Field, Schema};
+ /// # use datafusion_common::{ColumnStatistics, Result, Statistics};
+ /// # use datafusion_common::stats::Precision;
+ /// # use datafusion_physical_plan::ExecutionPlan;
+ /// # use datafusion_physical_plan::operator_statistics::{
+ /// # ExtendedStatistics, StatisticsProvider, StatisticsRegistry, StatisticsResult,
+ /// # };
+ /// # use datafusion_physical_plan::statistics::{StatisticsArgs, StatisticsContext};
+ /// # use datafusion_physical_plan::test::exec::StatisticsExec;
+ ///
+ /// // Pearson correlation between two columns, identified by schema index.
+ /// #[derive(Debug, Clone)]
+ /// struct ColumnCorrelation {
+ /// columns: (usize, usize),
+ /// coefficient: f64,
+ /// }
+ ///
+ /// #[derive(Debug)]
+ /// struct CorrelationProvider;
+ ///
+ /// impl StatisticsProvider for CorrelationProvider {
+ /// fn matches(&self, plan: &dyn ExecutionPlan) -> bool {
+ /// plan.downcast_ref::<StatisticsExec>().is_some()
+ /// }
+ ///
+ /// fn compute_statistics(
+ /// &self,
+ /// plan: &dyn ExecutionPlan,
+ /// _child_stats: &[ExtendedStatistics],
+ /// ) -> Result<StatisticsResult> {
+ /// let base = plan.statistics_from_inputs(&[], &StatisticsArgs::new())?;
+ /// let mut extended = ExtendedStatistics::new_arc(base);
+ /// extended.set_extension(ColumnCorrelation {
+ /// columns: (0, 1),
+ /// coefficient: 0.92,
+ /// });
+ /// Ok(StatisticsResult::Computed(extended))
+ /// }
+ /// }
+ ///
+ /// let schema = Schema::new(vec![
+ /// Field::new("city", DataType::Utf8, false),
+ /// Field::new("zip_code", DataType::Utf8, false),
+ /// ]);
+ /// let stats = Statistics {
+ /// num_rows: Precision::Exact(1000),
+ /// total_byte_size: Precision::Absent,
+ /// column_statistics: vec![ColumnStatistics::new_unknown(); 2],
+ /// };
+ /// let plan = StatisticsExec::new(stats, schema);
+ ///
+ /// let mut registry = StatisticsRegistry::new();
+ /// registry.register(Arc::new(CorrelationProvider));
+ /// let context = StatisticsContext::new_with_registry(registry);
+ ///
+ /// let extended = context.compute_extended(&plan, &StatisticsArgs::new())?;
+ /// assert_eq!(extended.base().num_rows, Precision::Exact(1000));
+ ///
Review Comment:
Good suggestion, it's indeed nice to check the content too, done in https://github.com/apache/datafusion/pull/25699/commits/2bc67a13e3c933b6f7963bcd1782dbe86e6b1ead
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs(physical-plan): add doctest examples for StatisticsRegistry, ExtendedStatistics, and compute_extended [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #25699:
URL: https://github.com/apache/datafusion/pull/25699#issuecomment-5811124536
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/25699?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 82.49%. Comparing base ([`424cfd8`](https://app.codecov.io/gh/apache/datafusion/commit/424cfd8eb7e5d73229227324f02b39179babeeb1?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`1d243eb`](https://app.codecov.io/gh/apache/datafusion/commit/1d243ebe19957732c3ddd12e1c03d17ae3754188?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #25699 +/- ##
==========================================
- Coverage 82.49% 82.49% -0.01%
==========================================
Files 1140 1140
Lines 437675 437675
Branches 437675 437675
==========================================
- Hits 361054 361044 -10
- Misses 54830 54834 +4
- Partials 21791 21797 +6
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/25699?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org