You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "neilconway (via GitHub)" <gi...@apache.org> on 2026/04/10 13:41:41 UTC
[PR] perf: Optimize NULL handling in `array_remove` [datafusion]
neilconway opened a new pull request, #21532:
URL: https://github.com/apache/datafusion/pull/21532
## Which issue does this PR close?
- Closes #21531.
## Rationale for this change
`array_remove` computes the result NULL bitmap incrementally (row-by-row). This is inefficient; it is faster to compute the result NULL bitmap via the bitwise AND of the input NULL bitmaps.
Benchmarks (Arm64):
```
- array_remove_binary/remove/10: 6.8ms → 6.6ms (-2.9%)
- array_remove_binary/remove/100: 13.0ms → 12.7ms (-2.3%)
- array_remove_binary/remove/500: 75.2ms → 76.1ms (+1.2%)
- array_remove_boolean/remove/10: 4.5ms → 4.5ms (-0.6%)
- array_remove_boolean/remove/100: 8.0ms → 7.8ms (-2.5%)
- array_remove_boolean/remove/500: 21.0ms → 21.3ms (+1.4%)
- array_remove_decimal64/remove/10: 5.6ms → 5.7ms (+1.8%)
- array_remove_decimal64/remove/100: 10.0ms → 10.2ms (+2.0%)
- array_remove_decimal64/remove/500: 58.6ms → 56.2ms (-4.1%)
- array_remove_f64/remove/10: 5.4ms → 5.4ms (0.0%)
- array_remove_f64/remove/100: 7.7ms → 7.8ms (+1.3%)
- array_remove_f64/remove/500: 36.5ms → 36.8ms (+0.8%)
- array_remove_fixed_size_binary/remove/10: 6.1ms → 5.4ms (-11.5%)
- array_remove_fixed_size_binary/remove/100: 12.4ms → 11.6ms (-6.5%)
- array_remove_fixed_size_binary/remove/500: 66.7ms → 64.4ms (-3.4%)
- array_remove_int64/remove/10: 5.6ms → 5.7ms (+1.8%)
- array_remove_int64/remove/100: 7.8ms → 7.8ms (-1.0%)
- array_remove_int64/remove/500: 35.5ms → 36.3ms (+2.3%)
- array_remove_strings/remove/10: 6.6ms → 6.8ms (+3.0%)
- array_remove_strings/remove/100: 18.0ms → 18.3ms (+1.7%)
- array_remove_strings/remove/500: 78.0ms → 79.7ms (+2.2%)
```
Not a massive win, but I think still worth making the change; the resulting code is also more idiomatic.
## What changes are included in this PR?
* Implement optimization
* Tweak `array_remove` benchmark for f64: the previous code resulted in never actually removing anything from the array, which was inconsistent with the other benchmarks and probably unintentional.
## Are these changes tested?
Yes.
## Are there any user-facing changes?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Optimize NULL handling in `array_remove` [datafusion]
Posted by "neilconway (via GitHub)" <gi...@apache.org>.
neilconway commented on PR #21532:
URL: https://github.com/apache/datafusion/pull/21532#issuecomment-4224394411
@Jefffrey Thanks for the review! _Shameless plug_: If you enjoyed reviewing this PR, there are many such similar PRs you might also enjoy 😉
* https://github.com/apache/datafusion/pull/21471
* https://github.com/apache/datafusion/pull/21535
* https://github.com/apache/datafusion/pull/21464
* https://github.com/apache/datafusion/pull/21475
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Optimize NULL handling in `array_remove` [datafusion]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich merged PR #21532:
URL: https://github.com/apache/datafusion/pull/21532
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org