You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Mark1626 (via GitHub)" <gi...@apache.org> on 2025/11/25 05:04:23 UTC
[I] Incorrectly result from PartitionPruningStatistics when multiple values are present [datafusion]
Mark1626 opened a new issue, #18922:
URL: https://github.com/apache/datafusion/issues/18922
### Describe the bug
I have a hive partitioned TPC-DS dataset and I'm using a custom table provider where I'm doing some pre-scan partition pruning using `PartitionPruningStatistics`
A query with a single value in the filter expr works
```
select ss_list_price
from store_sales
where ss_sold_date_sk = 2451529 limit 10;
```
But when there are multiple values in the filter expr it fails
```
select ss_list_price
from store_sales
where ss_sold_date_sk in (2451529, 2452570, 2452596) limit 10;
```
This part of the code seems to be the problem
https://github.com/apache/datafusion/blob/d24eb4a23156b7814836e765d5890186ab40682f/datafusion/common/src/pruning.rs#L240-L250
I think `arrow::compute::kernels::boolean::or` should be used here instead of the `arrow::compute::kernels::boolean::and`
This query works from the `datafusion-cli` as I suspect that the file statistics prevents the accidental pruning
```
CREATE EXTERNAL TABLE store_sales
STORED AS PARQUET
LOCATION '/path/to/tpcds_1_delta/store_sales/';
select ss_list_price
from store_sales
where ss_sold_date_sk in (2451529, 2452570, 2452596) limit 10;
```
### To Reproduce
The following unit test will fail in `pruning.rs`
```
#[test]
fn test_partition_pruning_statistics_multiple_values() {
let partition_values = vec![
vec![ScalarValue::from(1i32), ScalarValue::from(2i32)],
vec![ScalarValue::from(3i32), ScalarValue::from(4i32)],
];
let partition_fields = vec![
Arc::new(Field::new("a", DataType::Int32, false)),
Arc::new(Field::new("b", DataType::Int32, false)),
];
let partition_stats =
PartitionPruningStatistics::try_new(partition_values, partition_fields)
.unwrap();
let column_a = Column::new_unqualified("a");
let column_b = Column::new_unqualified("b");
// Corresponds to
// select * from table where a in (1, 3);
let values = HashSet::from([ScalarValue::from(1i32), ScalarValue::from(3i32)]);
let contained_a = partition_stats.contained(&column_a, &values).unwrap();
let expected_contained_a = BooleanArray::from(vec![true, true]);
assert_eq!(contained_a, expected_contained_a);
}
```
### Expected behavior
The unit test mentioned above should pass
### Additional context
I can raise a PR for this, let me know if my analysis was correct and if there is any background context behind why the condition is an `AND` in the `contained` function
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Incorrectly result from PartitionPruningStatistics when multiple values are present [datafusion]
Posted by "Mark1626 (via GitHub)" <gi...@apache.org>.
Mark1626 commented on issue #18922:
URL: https://github.com/apache/datafusion/issues/18922#issuecomment-3579324569
I've changed the `arrow::compute::kernels::boolean::or` to a `arrow::compute::kernels::boolean::or_kleene` to take into account the behaviour of returing `null`.
There seems to be a different issue end-to-end in my case, all the partitions are selected when `null` is in the filter expr. This seems to be an issue in the `PruningPredicate` (it creates a `literal(true)` expr auto qualifying all the partitions) rather than the stats, I'll file a separate issue for it
```
select ss_list_price
from store_sales
where ss_sold_date_sk in (2451529, 2452570, NULL) limit 10;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Incorrectly result from PartitionPruningStatistics when multiple values are present [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #18922:
URL: https://github.com/apache/datafusion/issues/18922#issuecomment-3593755005
> But when there are multiple values in the filter expr it fails
Could you clarify what is "failing" / how the query fails?
Is the query getting the wrong answer? Is it failing to prune the files as you expect?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Incorrectly result from PartitionPruningStatistics when multiple values are present [datafusion]
Posted by "Mark1626 (via GitHub)" <gi...@apache.org>.
Mark1626 commented on issue #18922:
URL: https://github.com/apache/datafusion/issues/18922#issuecomment-3594303681
I was using `PartitionStatistics` in isolation and I noticed that it's pruning all the files, `PartitionStatistics` is incorrectly marking the files to be pruned. Let's say all the available partitions for `ss_sold_date_sk` are `[(2451000), (2451001), (2451002), (245103)]`. The current behaviour of `contained` of `PartitionStatistics` is
1. returns `[true]` for `select ss_list_price from store_sales where ss_sold_date_sk = 2451000`
2. returns `[false, false]` for `select ss_list_price from store_sales where ss_sold_date_sk in (2451000, 2451001)`
The second scenario should return `[true, true]`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Incorrectly result from PartitionPruningStatistics when multiple values are present [datafusion]
Posted by "dqkqd (via GitHub)" <gi...@apache.org>.
dqkqd commented on issue #18922:
URL: https://github.com/apache/datafusion/issues/18922#issuecomment-3577546042
> I think arrow::compute::kernels::boolean::or should be used here instead of the arrow::compute::kernels::boolean::and
The [docs for contained](https://docs.rs/datafusion/latest/datafusion/common/pruning/trait.PruningStatistics.html#tymethod.contained) states that it should return:
- true if the values in column ONLY contain values from values.
- false if the values in column are NOT ANY of values
- null if the neither of the above holds or is unknown.
This explains why `arrow::compute::kernels::boolean::and` being used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Incorrectly result from PartitionPruningStatistics when multiple values are present [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #18922: Incorrectly result from PartitionPruningStatistics when multiple values are present
URL: https://github.com/apache/datafusion/issues/18922
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org