You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "yashrb24 (via GitHub)" <gi...@apache.org> on 2026/09/18 07:49:25 UTC
[PR] feat: derive per-column pruning guarantees from tuple IN lists [datafusion]
yashrb24 opened a new pull request, #25469:
URL: https://github.com/apache/datafusion/pull/25469
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: derive per-column pruning guarantees from tuple IN lists [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on PR #25469:
URL: https://github.com/apache/datafusion/pull/25469#issuecomment-6160914338
@yashrb24
I would love to review this after you resolve the merge conflicts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: derive per-column pruning guarantees from tuple IN lists [datafusion]
Posted by "rgbuilds (via GitHub)" <gi...@apache.org>.
rgbuilds commented on code in PR #25469:
URL: https://github.com/apache/datafusion/pull/25469#discussion_r4174252842
##########
datafusion/physical-expr/src/utils/guarantee.rs:
##########
@@ -377,6 +388,85 @@ impl<'a> GuaranteeBuilder<'a> {
}
}
+/// Project necessary per-column guarantees; the original predicate retains tuple correlation.
+fn project_struct_in_list(
+ inlist: &crate::expressions::InListExpr,
+) -> Option<Vec<(&crate::expressions::Column, Vec<ScalarValue>)>> {
+ if inlist.negated() || inlist.is_empty() {
+ return None;
+ }
+ let expr = inlist.expr().downcast_ref::<crate::ScalarFunctionExpr>()?;
+ let literal_args = expr
+ .args()
+ .iter()
+ .map(|arg| {
+ arg.downcast_ref::<crate::expressions::Literal>()
+ .map(|lit| lit.value().clone())
+ })
+ .collect::<Vec<_>>();
+ let mapping = expr.fun().struct_field_mapping(&literal_args)?;
+ if mapping.field_accessor.signature().volatility != Volatility::Immutable {
+ return None;
+ }
+ let tuples = inlist
+ .list()
+ .iter()
+ .map(|value| {
+ let literal = value.downcast_ref::<crate::expressions::Literal>()?;
+ let ScalarValue::Struct(array) = literal.value() else {
+ return None;
+ };
+ (array.len() == 1).then_some(literal.value())
+ })
+ .collect::<Option<Vec<_>>>()?;
+ // Null tuples cannot make a positive IN predicate true.
+ let tuples = ScalarValue::iter_to_array(
+ tuples.into_iter().filter(|tuple| !tuple.is_null()).cloned(),
+ )
+ .ok()?;
+ let batch = RecordBatch::try_from_iter([("tuple", tuples)]).ok()?;
+
+ let mut projected = Vec::new();
+ for (accessor_args, source_index) in mapping.fields {
Review Comment:
Thanks. Returning no mapping when a field name is repeated prevents the ambiguous inference, and the focused regression test covers the case I raised. This resolves my concern.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: derive per-column pruning guarantees from tuple IN lists [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew merged PR #25469:
URL: https://github.com/apache/datafusion/pull/25469
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: derive per-column pruning guarantees from tuple IN lists [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on code in PR #25469:
URL: https://github.com/apache/datafusion/pull/25469#discussion_r4310453061
##########
datafusion/physical-expr/src/utils/guarantee.rs:
##########
@@ -377,6 +388,85 @@ impl<'a> GuaranteeBuilder<'a> {
}
}
+/// Project necessary per-column guarantees; the original predicate retains tuple correlation.
+fn project_struct_in_list(
+ inlist: &crate::expressions::InListExpr,
+) -> Option<Vec<(&crate::expressions::Column, Vec<ScalarValue>)>> {
+ if inlist.negated() || inlist.is_empty() {
+ return None;
+ }
+ let expr = inlist.expr().downcast_ref::<crate::ScalarFunctionExpr>()?;
+ let literal_args = expr
+ .args()
+ .iter()
+ .map(|arg| {
+ arg.downcast_ref::<crate::expressions::Literal>()
+ .map(|lit| lit.value().clone())
+ })
+ .collect::<Vec<_>>();
+ let mapping = expr.fun().struct_field_mapping(&literal_args)?;
+ if mapping.field_accessor.signature().volatility != Volatility::Immutable {
+ return None;
+ }
+ let tuples = inlist
+ .list()
+ .iter()
+ .map(|value| {
+ let literal = value.downcast_ref::<crate::expressions::Literal>()?;
+ let ScalarValue::Struct(array) = literal.value() else {
+ return None;
+ };
+ (array.len() == 1).then_some(literal.value())
+ })
+ .collect::<Option<Vec<_>>>()?;
+ // Null tuples cannot make a positive IN predicate true.
+ let tuples = ScalarValue::iter_to_array(
Review Comment:
Could we add a test where all RHS tuples are NULL, using `Some(vec![false, false, false].into())`, and verify that no guarantees are extracted? The current fallback is safe, but this would help ensure it stays that way.
##########
datafusion/physical-expr/src/utils/guarantee.rs:
##########
@@ -377,6 +388,85 @@ impl<'a> GuaranteeBuilder<'a> {
}
}
+/// Project necessary per-column guarantees; the original predicate retains tuple correlation.
+fn project_struct_in_list(
+ inlist: &crate::expressions::InListExpr,
+) -> Option<Vec<(&crate::expressions::Column, Vec<ScalarValue>)>> {
+ if inlist.negated() || inlist.is_empty() {
+ return None;
+ }
+ let expr = inlist.expr().downcast_ref::<crate::ScalarFunctionExpr>()?;
+ let literal_args = expr
+ .args()
+ .iter()
+ .map(|arg| {
+ arg.downcast_ref::<crate::expressions::Literal>()
+ .map(|lit| lit.value().clone())
+ })
+ .collect::<Vec<_>>();
+ let mapping = expr.fun().struct_field_mapping(&literal_args)?;
+ if mapping.field_accessor.signature().volatility != Volatility::Immutable {
+ return None;
+ }
+ let tuples = inlist
+ .list()
+ .iter()
+ .map(|value| {
+ let literal = value.downcast_ref::<crate::expressions::Literal>()?;
+ let ScalarValue::Struct(array) = literal.value() else {
+ return None;
+ };
+ (array.len() == 1).then_some(literal.value())
+ })
+ .collect::<Option<Vec<_>>>()?;
+ // Null tuples cannot make a positive IN predicate true.
+ let tuples = ScalarValue::iter_to_array(
+ tuples.into_iter().filter(|tuple| !tuple.is_null()).cloned(),
+ )
+ .ok()?;
+ let batch = RecordBatch::try_from_iter([("tuple", tuples)]).ok()?;
+
+ let mut projected = Vec::new();
+ for (accessor_args, source_index) in mapping.fields {
+ let column = expr
+ .args()
+ .get(source_index)?
+ .downcast_ref::<crate::expressions::Column>()?;
Review Comment:
As a possible follow-up, could we skip computed arguments instead of abandoning all projections? For `struct(a, b + 1) IN ((1, 10), (2, 20))`, we could still extract `a IN (1, 2)`. Invalid mapping indices should still abort extraction, and tests could cover computed arguments before and after a direct column. No change is needed for this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: derive per-column pruning guarantees from tuple IN lists [datafusion]
Posted by "rgbuilds (via GitHub)" <gi...@apache.org>.
rgbuilds commented on code in PR #25469:
URL: https://github.com/apache/datafusion/pull/25469#discussion_r4057567297
##########
datafusion/datasource-parquet/src/bloom_filter.rs:
##########
@@ -270,6 +270,114 @@ mod tests {
.unwrap()
}
+ #[tokio::test]
+ async fn test_tuple_in_bloom_pruning_preserves_correlation() -> Result<()> {
Review Comment:
The unit test nicely covers both pruning and the crossed-pair correlation case, while the SLT exercises the real Parquet scan path. Would it be valuable to combine these in one end-to-end test with multiple row groups? For example, the test could include:
1. A row group containing an exact tuple match.
2. A row group that statistics cannot eliminate but Bloom filters can, verifying that it is counted as pruned in row_groups_pruned_bloom_filter.
3. A crossed-pair row group that satisfies the derived per-column guarantees and therefore survives Bloom pruning, but produces no rows after evaluation of the original tuple predicate.
This would verify that the derived per-column conditions are used only for pruning, while the original tuple predicate is still applied for exact row filtering.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: derive per-column pruning guarantees from tuple IN lists [datafusion]
Posted by "yashrb24 (via GitHub)" <gi...@apache.org>.
yashrb24 commented on code in PR #25469:
URL: https://github.com/apache/datafusion/pull/25469#discussion_r4046713099
##########
datafusion/functions/src/core/struct.rs:
##########
@@ -150,4 +152,16 @@ impl ScalarUDFImpl for StructFunc {
fn documentation(&self) -> Option<&Documentation> {
self.doc()
}
+
+ fn struct_field_mapping(
+ &self,
+ literal_args: &[Option<ScalarValue>],
+ ) -> Option<StructFieldMapping> {
+ Some(StructFieldMapping {
+ field_accessor: Arc::new(ScalarUDF::from(GetFieldFunc::new())),
+ fields: (0..literal_args.len())
+ .map(|i| (vec![ScalarValue::Utf8(Some(format!("c{i}")))], i))
+ .collect(),
+ })
+ }
Review Comment:
`StructFunc` from datafusion-functions crate already provides support for this, but instead of adding another dependency I thought it might be a better idea to have it defined separately here instead
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: derive per-column pruning guarantees from tuple IN lists [datafusion]
Posted by "rgbuilds (via GitHub)" <gi...@apache.org>.
rgbuilds commented on PR #25469:
URL: https://github.com/apache/datafusion/pull/25469#issuecomment-5843616836
Thanks for adding the three-row-group end-to-end scan case. This addresses my earlier review comment: the revised test now exercises:
- an exact-match row group,
- a Bloom-pruned row group, and
- a crossed-pair row group that survives the derived per-column guarantees but is rejected by the original tuple predicate.
I also ran `test_struct_inlist_guarantees` and the `push_down_filter_parquet` SLT locally; both passed.
While reviewing the revised guarantee-extraction path, I found a duplicate-field-name case in `named_struct` that appears capable of deriving an invalid necessary guarantee and pruning a row group containing a valid match. I left the concrete counterexample inline.
Iām requesting changes pending resolution of that correctness case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: derive per-column pruning guarantees from tuple IN lists [datafusion]
Posted by "yashrb24 (via GitHub)" <gi...@apache.org>.
yashrb24 commented on PR #25469:
URL: https://github.com/apache/datafusion/pull/25469#issuecomment-6209723382
@kosiew thank you. i've resolved the conflicts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: derive per-column pruning guarantees from tuple IN lists [datafusion]
Posted by "yashrb24 (via GitHub)" <gi...@apache.org>.
yashrb24 commented on code in PR #25469:
URL: https://github.com/apache/datafusion/pull/25469#discussion_r4340999898
##########
datafusion/physical-expr/src/utils/guarantee.rs:
##########
@@ -377,6 +388,85 @@ impl<'a> GuaranteeBuilder<'a> {
}
}
+/// Project necessary per-column guarantees; the original predicate retains tuple correlation.
+fn project_struct_in_list(
+ inlist: &crate::expressions::InListExpr,
+) -> Option<Vec<(&crate::expressions::Column, Vec<ScalarValue>)>> {
+ if inlist.negated() || inlist.is_empty() {
+ return None;
+ }
+ let expr = inlist.expr().downcast_ref::<crate::ScalarFunctionExpr>()?;
+ let literal_args = expr
+ .args()
+ .iter()
+ .map(|arg| {
+ arg.downcast_ref::<crate::expressions::Literal>()
+ .map(|lit| lit.value().clone())
+ })
+ .collect::<Vec<_>>();
+ let mapping = expr.fun().struct_field_mapping(&literal_args)?;
+ if mapping.field_accessor.signature().volatility != Volatility::Immutable {
+ return None;
+ }
+ let tuples = inlist
+ .list()
+ .iter()
+ .map(|value| {
+ let literal = value.downcast_ref::<crate::expressions::Literal>()?;
+ let ScalarValue::Struct(array) = literal.value() else {
+ return None;
+ };
+ (array.len() == 1).then_some(literal.value())
+ })
+ .collect::<Option<Vec<_>>>()?;
+ // Null tuples cannot make a positive IN predicate true.
+ let tuples = ScalarValue::iter_to_array(
Review Comment:
This one should be useful! adding it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: derive per-column pruning guarantees from tuple IN lists [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #25469:
URL: https://github.com/apache/datafusion/pull/25469#issuecomment-5727216742
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/25469?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `89.38776%` with `26 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 82.35%. Comparing base ([`3b16a3d`](https://app.codecov.io/gh/apache/datafusion/commit/3b16a3d0eef765b81ce5aad20997b193a42ad02a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`e351986`](https://app.codecov.io/gh/apache/datafusion/commit/e3519868c9ead026d8c116796bc64930ca528433?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 1 commits behind head on main.
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/25469?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [datafusion/datasource-parquet/src/bloom\_filter.rs](https://app.codecov.io/gh/apache/datafusion/pull/25469?src=pr&el=tree&filepath=datafusion%2Fdatasource-parquet%2Fsrc%2Fbloom_filter.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9kYXRhc291cmNlLXBhcnF1ZXQvc3JjL2Jsb29tX2ZpbHRlci5ycw==) | 84.26% | [3 Missing and 11 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/25469?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [datafusion/physical-expr/src/utils/guarantee.rs](https://app.codecov.io/gh/apache/datafusion/pull/25469?src=pr&el=tree&filepath=datafusion%2Fphysical-expr%2Fsrc%2Futils%2Fguarantee.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9waHlzaWNhbC1leHByL3NyYy91dGlscy9ndWFyYW50ZWUucnM=) | 91.83% | [3 Missing and 9 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/25469?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #25469 +/- ##
========================================
Coverage 82.34% 82.35%
========================================
Files 1137 1137
Lines 432713 432989 +276
Branches 432713 432989 +276
========================================
+ Hits 356338 356584 +246
- Misses 54842 54855 +13
- Partials 21533 21550 +17
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/25469?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: derive per-column pruning guarantees from tuple IN lists [datafusion]
Posted by "yashrb24 (via GitHub)" <gi...@apache.org>.
yashrb24 commented on code in PR #25469:
URL: https://github.com/apache/datafusion/pull/25469#discussion_r4059977262
##########
datafusion/datasource-parquet/src/bloom_filter.rs:
##########
@@ -270,6 +270,114 @@ mod tests {
.unwrap()
}
+ #[tokio::test]
+ async fn test_tuple_in_bloom_pruning_preserves_correlation() -> Result<()> {
Review Comment:
thanks for the review! this makes sense to me, will extend this test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: derive per-column pruning guarantees from tuple IN lists [datafusion]
Posted by "rgbuilds (via GitHub)" <gi...@apache.org>.
rgbuilds commented on code in PR #25469:
URL: https://github.com/apache/datafusion/pull/25469#discussion_r4110344145
##########
datafusion/physical-expr/src/utils/guarantee.rs:
##########
@@ -377,6 +388,85 @@ impl<'a> GuaranteeBuilder<'a> {
}
}
+/// Project necessary per-column guarantees; the original predicate retains tuple correlation.
+fn project_struct_in_list(
+ inlist: &crate::expressions::InListExpr,
+) -> Option<Vec<(&crate::expressions::Column, Vec<ScalarValue>)>> {
+ if inlist.negated() || inlist.is_empty() {
+ return None;
+ }
+ let expr = inlist.expr().downcast_ref::<crate::ScalarFunctionExpr>()?;
+ let literal_args = expr
+ .args()
+ .iter()
+ .map(|arg| {
+ arg.downcast_ref::<crate::expressions::Literal>()
+ .map(|lit| lit.value().clone())
+ })
+ .collect::<Vec<_>>();
+ let mapping = expr.fun().struct_field_mapping(&literal_args)?;
+ if mapping.field_accessor.signature().volatility != Volatility::Immutable {
+ return None;
+ }
+ let tuples = inlist
+ .list()
+ .iter()
+ .map(|value| {
+ let literal = value.downcast_ref::<crate::expressions::Literal>()?;
+ let ScalarValue::Struct(array) = literal.value() else {
+ return None;
+ };
+ (array.len() == 1).then_some(literal.value())
+ })
+ .collect::<Option<Vec<_>>>()?;
+ // Null tuples cannot make a positive IN predicate true.
+ let tuples = ScalarValue::iter_to_array(
+ tuples.into_iter().filter(|tuple| !tuple.is_null()).cloned(),
+ )
+ .ok()?;
+ let batch = RecordBatch::try_from_iter([("tuple", tuples)]).ok()?;
+
+ let mut projected = Vec::new();
+ for (accessor_args, source_index) in mapping.fields {
Review Comment:
I think duplicate `named_struct` field names can make this inference unsound.
`named_struct` currently permits duplicate names and emits a mapping entry for each field, while `get_field` uses `StructArray::column_by_name`, which selects the first matching field when names are duplicated.
For example:
```sql
named_struct('k', x, 'k', y)
IN (named_struct('k', 1, 'k', 10))
```
Both mapping entries evaluate `get_field(..., 'k')` against the first literal field. This appears to derive:
```text
x IN (1)
y IN (1)
```
However, `(x=1, y=10)` satisfies the original tuple predicate.
A row group containing the matching row `(1,10)` and a filler row `(0,0)` would have `y_min=0` and `y_max=10`, so statistics cannot eliminate `y=1`. Its Bloom filter could nevertheless prove that `y=1` is absent and incorrectly prune the row group containing the valid `(1,10)` match.
Would it be safer for `named_struct::struct_field_mapping` to return no mapping when an accessor name is repeated, with a focused regression test for this case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: derive per-column pruning guarantees from tuple IN lists [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on PR #25469:
URL: https://github.com/apache/datafusion/pull/25469#issuecomment-6334546335
š
@yashrb24
Thank you for your contribution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org