You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "neilconway (via GitHub)" <gi...@apache.org> on 2026/09/09 15:00:09 UTC
[PR] perf: Compute `array_length` of first dimension from offsets [datafusion]
neilconway opened a new pull request, #25121:
URL: https://github.com/apache/datafusion/pull/25121
## Which issue does this PR close?
- Closes ##25119.
## Rationale for this change
`array_length` worked by forming a slice of values for each logical row in the given dimension, and then appending the length of the slice to the result. That is inefficient; if computing the length of the first array dimension, we can get that from the offset buffer instead. That avoids a lot of per-row work and is significantly faster.
Benchmarks: (M4 Max)
- list/default/8192, 175.379 µs -> 1.137 µs, -99.35%
- list/dimension_1/8192, 173.643 µs -> 1.159 µs, -99.33%
- fixed_size_list/default/8192, 174.770 µs -> 0.963 µs, -99.45%
- list/default/1, 0.171 µs -> 0.088 µs, -48.48%
## What changes are included in this PR?
* Compute `array_length` of dimension 1 for `List` and `LargeList` from offsets
* Compute `array_length` of dimension 1 for `FixedSizeList` from the FSL's size
* Treat `array_length(x, 1)` as equivalent to `array_length(x)`
* Simplify the code to remove a no-longer-necessary macro
* Add unit tests
* Add benchmark for `array_length`
## What is the testing strategy for this PR?
Existing tests pass; new unit tests added.
## Are there any user-facing changes?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Compute `array_length` of first dimension from offsets [datafusion]
Posted by "neilconway (via GitHub)" <gi...@apache.org>.
neilconway commented on PR #25121:
URL: https://github.com/apache/datafusion/pull/25121#issuecomment-5620689695
@AdamGS @Rich-T-kid BTW, there is also #23922 with a closely related optimization for `cardinality`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Compute `array_length` of first dimension from offsets [datafusion]
Posted by "neilconway (via GitHub)" <gi...@apache.org>.
neilconway merged PR #25121:
URL: https://github.com/apache/datafusion/pull/25121
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Compute `array_length` of first dimension from offsets [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #25121:
URL: https://github.com/apache/datafusion/pull/25121#issuecomment-5609933706
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/25121?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `84.31373%` with `16 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 81.80%. Comparing base ([`4048898`](https://app.codecov.io/gh/apache/datafusion/commit/40488988ad596c9b093ad60e1453430d803ce33c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`8d918c3`](https://app.codecov.io/gh/apache/datafusion/commit/8d918c31ba9de598e90c7b751e981c79e123585f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/25121?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [datafusion/functions-nested/src/length.rs](https://app.codecov.io/gh/apache/datafusion/pull/25121?src=pr&el=tree&filepath=datafusion%2Ffunctions-nested%2Fsrc%2Flength.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9mdW5jdGlvbnMtbmVzdGVkL3NyYy9sZW5ndGgucnM=) | 84.31% | [8 Missing and 8 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/25121?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #25121 +/- ##
==========================================
- Coverage 81.80% 81.80% -0.01%
==========================================
Files 1130 1130
Lines 417754 417842 +88
Branches 417754 417842 +88
==========================================
+ Hits 341754 341823 +69
- Misses 55875 55883 +8
- Partials 20125 20136 +11
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/25121?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Compute `array_length` of first dimension from offsets [datafusion]
Posted by "neilconway (via GitHub)" <gi...@apache.org>.
neilconway commented on PR #25121:
URL: https://github.com/apache/datafusion/pull/25121#issuecomment-5607273049
@AdamGS @Rich-T-kid Thanks for the reviews!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org