You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/07/05 05:15:46 UTC
(datafusion) branch gh-readonly-queue/main/pr-23277-85de1367630b9e74e65962485c656b495a44431d created (now 7b1bffaaff)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a change to branch gh-readonly-queue/main/pr-23277-85de1367630b9e74e65962485c656b495a44431d
in repository https://gitbox.apache.org/repos/asf/datafusion.git
at 7b1bffaaff fix: Avoid panicing when stats are not available for a file group split (#23277)
This branch includes the following new commits:
new 7b1bffaaff fix: Avoid panicing when stats are not available for a file group split (#23277)
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion) 01/01: fix: Avoid panicing when stats are not available for a file group split (#23277)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-23277-85de1367630b9e74e65962485c656b495a44431d
in repository https://gitbox.apache.org/repos/asf/datafusion.git
commit 7b1bffaaffa3ace1c059722ee5493e3f289f203a
Author: Michael Kleen <mk...@gmail.com>
AuthorDate: Sun Jul 5 07:15:27 2026 +0200
fix: Avoid panicing when stats are not available for a file group split (#23277)
## Which issue does this PR close?
- Closes https://github.com/apache/datafusion/issues/23219.
## Rationale for this change
The query from the issue:
```sql
SELECT (((Cast(id AS BIGINT) % 1024) + 1024) % 1024) AS computed_bucket
FROM profile
ORDER BY computed_bucket,
Cast(id AS BIGINT) limit 10;
```
panics:
```
thread 'main' panicked at .../datafusion-datasource-54.0.0/src/statistics.rs:100:48:
index out of bounds: the len is 0 but the index is 0
```
The underlying issue is that the current code panics when files are
split by statistics and there are no statistics available for the column
where the sort order is defined in this case `computed_bucket`.
## What changes are included in this PR?
- Fix in `MinMaxStatistics` to check if there are stats available for a
given column
- Test
## Are these changes tested?
Yes
## Are there any user-facing changes?
No
---
datafusion/datasource/src/statistics.rs | 39 +++++++++++++++++++++++++++++++--
1 file changed, 37 insertions(+), 2 deletions(-)
diff --git a/datafusion/datasource/src/statistics.rs b/datafusion/datasource/src/statistics.rs
index 781cf3dbce..491b4e54e8 100644
--- a/datafusion/datasource/src/statistics.rs
+++ b/datafusion/datasource/src/statistics.rs
@@ -97,8 +97,16 @@ impl MinMaxStatistics {
.zip(s.column_statistics[i].max_value.get_value().cloned())
.ok_or_else(|| plan_datafusion_err!("statistics not found"))
} else {
- let partition_value = &pv[i - s.column_statistics.len()];
- Ok((partition_value.clone(), partition_value.clone()))
+ if let Some(partition_value) =
+ pv.get(i - s.column_statistics.len())
+ {
+ Ok((partition_value.clone(), partition_value.clone()))
+ } else {
+ Err(plan_datafusion_err!(
+ "statistics not found for partition, expected at most {}",
+ s.column_statistics.len()
+ ))
+ }
}
})
.collect::<Result<Vec<_>>>()?
@@ -882,4 +890,31 @@ mod tests {
Ok(())
}
+
+ #[test]
+ fn min_max_statistics_missing_column_stats_returns_error() {
+ let schema = test_schema();
+ let sort_order =
+ [PhysicalSortExpr::new_default(Arc::new(Column::new("a", 0)))].into();
+ let files = [
+ file_with_stats("f1.parquet", Statistics::default()),
+ file_with_stats("f2.parquet", Statistics::default()),
+ ];
+
+ let err = match MinMaxStatistics::new_from_files(
+ &sort_order,
+ &schema,
+ None,
+ files.iter(),
+ ) {
+ Ok(_) => panic!("expected missing statistics error"),
+ Err(err) => err,
+ };
+
+ assert!(
+ err.to_string()
+ .contains("statistics not found for partition"),
+ "unexpected error: {err:?}"
+ );
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org