You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "neilconway (via GitHub)" <gi...@apache.org> on 2026/09/17 19:06:27 UTC
[PR] perf: Improve efficiency of `array_has_any` on sliced lists [datafusion]
neilconway opened a new pull request, #25427:
URL: https://github.com/apache/datafusion/pull/25427
## Which issue does this PR close?
- Closes #25426.
## Rationale for this change
When one argument of `array_has_any` is a scalar and the element type is not a string, the fast path `array_has_any_with_scalar_general` row-converts the list column's full child array, which does a lot of unnecessary work for sliced lists.
Benchmarks:
- array_has_any_scalar/i64_no_match_sliced/1: 4.14 ms -> 167.4 µs, −96.0%
- array_has_any_scalar/i64_no_match_sliced/10: 4.11 ms -> 125.9 µs, −96.9%
- array_has_any_scalar/i64_no_match_sliced/100: 4.25 ms -> 257.8 µs, −93.9%
- array_has_any_scalar/i64_no_match_sliced/1000: 4.12 ms -> 141.1 µs, −96.6%
Results on unsliced inputs were unchanged, within run-to-run noise.
## What changes are included in this PR?
* Improve `array_has_any_with_scalar_general` to only touch the visible portion of the input list
* Add a benchmark
* Add unit tests for correct behavior on sliced lists
## What is the testing strategy for this PR?
Existing tests pass, new tests added. Benchmark confirms that the inefficiency has been addressed.
## Are there any user-facing changes?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Improve efficiency of `array_has_any` on sliced lists [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #25427:
URL: https://github.com/apache/datafusion/pull/25427#discussion_r4046750965
##########
datafusion/functions-nested/src/array_has.rs:
##########
@@ -1683,4 +1690,79 @@ mod tests {
vec![Some(true), Some(false)]
);
}
+
+ /// Invoke `array_has_any` with an array haystack and a scalar list of
+ /// `Int32` elements.
+ fn invoke_array_has_any_scalar(haystack: ArrayRef, scalar: Vec<i32>) -> ArrayRef {
+ let num_rows = haystack.len();
+ let haystack_type = haystack.data_type().clone();
+ let scalar = SingleRowListArrayBuilder::new(Arc::new(Int32Array::from(scalar)))
+ .build_list_scalar();
+ let scalar_type = scalar.data_type();
+ ArrayHasAny::new()
+ .invoke_with_args(ScalarFunctionArgs {
+ args: vec![
+ ColumnarValue::Array(haystack),
+ ColumnarValue::Scalar(scalar),
+ ],
+ arg_fields: vec![
+ Arc::new(Field::new("haystack", haystack_type, true)),
+ Arc::new(Field::new("scalar", scalar_type, true)),
+ ],
+ number_rows: num_rows,
+ return_field: Arc::new(Field::new("return", DataType::Boolean, true)),
+ config_options: Arc::new(ConfigOptions::default()),
+ })
+ .unwrap()
+ .into_array(num_rows)
+ .unwrap()
+ }
+
+ #[test]
+ fn test_array_has_any_scalar_sliced() {
Review Comment:
Sliced test covers `List` + `FixedSizeList`; add `LargeList` for the third `ArrayWrapper` arm.
```rs
let large = LargeListArray::from_iter_primitive::<Int32Type, _, _>(vec![
Some(vec![Some(1), Some(2)]),
Some(vec![Some(10), None, Some(30)]),
Some(vec![Some(70)]),
]);
let sliced: ArrayRef = Arc::new(large.slice(1, 1));
let result = invoke_array_has_any_scalar(sliced, vec![1, 70, 30]);
assert_eq!(
result.as_boolean().iter().collect::<Vec<_>>(),
vec![Some(true)]
);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Improve efficiency of `array_has_any` on sliced lists [datafusion]
Posted by "neilconway (via GitHub)" <gi...@apache.org>.
neilconway merged PR #25427:
URL: https://github.com/apache/datafusion/pull/25427
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Improve efficiency of `array_has_any` on sliced lists [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #25427:
URL: https://github.com/apache/datafusion/pull/25427#issuecomment-5720094476
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/25427?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `98.41270%` with `1 line` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 82.33%. Comparing base ([`98b26c5`](https://app.codecov.io/gh/apache/datafusion/commit/98b26c5f9e83f3a21624c97dfaecf37904ff0452?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`11f4099`](https://app.codecov.io/gh/apache/datafusion/commit/11f4099d0dd52f0738dfb0efb00ff5d9b127863a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 1 commits behind head on main.
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/25427?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [datafusion/functions-nested/src/array\_has.rs](https://app.codecov.io/gh/apache/datafusion/pull/25427?src=pr&el=tree&filepath=datafusion%2Ffunctions-nested%2Fsrc%2Farray_has.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9mdW5jdGlvbnMtbmVzdGVkL3NyYy9hcnJheV9oYXMucnM=) | 98.41% | [0 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/25427?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #25427 +/- ##
=======================================
Coverage 82.33% 82.33%
=======================================
Files 1137 1137
Lines 432049 432107 +58
Branches 432049 432107 +58
=======================================
+ Hits 355723 355789 +66
+ Misses 54826 54816 -10
- Partials 21500 21502 +2
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/25427?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Improve efficiency of `array_has_any` on sliced lists [datafusion]
Posted by "neilconway (via GitHub)" <gi...@apache.org>.
neilconway commented on code in PR #25427:
URL: https://github.com/apache/datafusion/pull/25427#discussion_r4047615959
##########
datafusion/functions-nested/src/array_has.rs:
##########
@@ -1683,4 +1690,79 @@ mod tests {
vec![Some(true), Some(false)]
);
}
+
+ /// Invoke `array_has_any` with an array haystack and a scalar list of
+ /// `Int32` elements.
+ fn invoke_array_has_any_scalar(haystack: ArrayRef, scalar: Vec<i32>) -> ArrayRef {
+ let num_rows = haystack.len();
+ let haystack_type = haystack.data_type().clone();
+ let scalar = SingleRowListArrayBuilder::new(Arc::new(Int32Array::from(scalar)))
+ .build_list_scalar();
+ let scalar_type = scalar.data_type();
+ ArrayHasAny::new()
+ .invoke_with_args(ScalarFunctionArgs {
+ args: vec![
+ ColumnarValue::Array(haystack),
+ ColumnarValue::Scalar(scalar),
+ ],
+ arg_fields: vec![
+ Arc::new(Field::new("haystack", haystack_type, true)),
+ Arc::new(Field::new("scalar", scalar_type, true)),
+ ],
+ number_rows: num_rows,
+ return_field: Arc::new(Field::new("return", DataType::Boolean, true)),
+ config_options: Arc::new(ConfigOptions::default()),
+ })
+ .unwrap()
+ .into_array(num_rows)
+ .unwrap()
+ }
+
+ #[test]
+ fn test_array_has_any_scalar_sliced() {
Review Comment:
Thanks, good idea!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org