You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "adriangb (via GitHub)" <gi...@apache.org> on 2026/05/15 20:16:15 UTC
[I] Optimize `arrays_zip` to avoid row-by-row copying in the perfect-zip case [datafusion]
adriangb opened a new issue, #22225:
URL: https://github.com/apache/datafusion/issues/22225
## Is your feature request related to a problem or challenge?
`arrays_zip` (`arrays_zip_inner_with_field` in `datafusion/functions-nested/src/arrays_zip.rs`) always assembles its output by walking every row through per-column `MutableArrayData` builders, copying each input slice one row at a time (`builder.extend(0, start, end)`) and padding shorter rows with NULLs (`builder.extend_nulls(...)`).
When the inputs form a **perfect zip** — every input array has identical per-row element lengths, no null list rows with non-zero element slots, and therefore no null padding is needed — this row-by-row copy is wasted work. In that case the resulting struct child columns are bit-identical to the (concatenated) input value arrays, and the list offsets are identical to the inputs' offsets.
## Describe the solution you'd like
Detect the perfect-zip case up front and skip the `MutableArrayData` path entirely:
- Build the output struct child columns directly from the original input value `ArrayRef`s (clone / concat, no per-row copy).
- Reuse an input array's offset buffer for the resulting `ListArray` instead of rebuilding it.
This keeps the existing general path as a fallback for the ragged / null-padded cases.
## Describe alternatives you've considered
Keep the current always-copy implementation. It is correct but does avoidable work for the common case where all zipped arrays line up.
## Additional context
Raised by @paleolimbot while reviewing #21984:
> Not here, but for the perfect zip (all value arrays the same length, no nulls with non-zero element slot lengths, no null padding needed) this should ideally be just clones of the original arrayrefs
Split out of #21984 (a metadata-propagation bugfix) since this is an orthogonal performance optimization that warrants its own benchmarks and edge-case tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Optimize `arrays_zip` to avoid row-by-row copying in the perfect-zip case [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #22225:
URL: https://github.com/apache/datafusion/issues/22225#issuecomment-4464551979
> No, I am happy to have someone else handle it!
Okie, we'll probably pick it up later next week as this function is heavily used in some of our workloads
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Optimize `arrays_zip` to avoid row-by-row copying in the perfect-zip case [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #22225:
URL: https://github.com/apache/datafusion/issues/22225#issuecomment-4467939994
Hi @BipashaBi, a link to the PR would be much more helpful than a screenshot. For what it’s worth I also suggest to not ask for things “ASAP”: reviewers have limited time and nowadays especially with so much AI generated code are under a lot of pressure. Wording like “when you have time” will land a lot smoother.
@puneetdixit200 I agree that adding benchmarks first as an isolated PR is the way to go.
Since @comphead already said they’d work on this I suggest we wait to see what they come up with. Maybe @puneetdixit200 @BipashaBi can help with code review of that PR? Time spent on code review is much more valuable than code submissions, it’s a much more limited resource.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Optimize `arrays_zip` to avoid row-by-row copying in the perfect-zip case [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #22225:
URL: https://github.com/apache/datafusion/issues/22225#issuecomment-4464403405
@adriangb are you planning to work on that? @hsiang-c cc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Optimize `arrays_zip` to avoid row-by-row copying in the perfect-zip case [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #22225:
URL: https://github.com/apache/datafusion/issues/22225#issuecomment-4464537213
No, I am happy to have someone else handle it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Optimize `arrays_zip` to avoid row-by-row copying in the perfect-zip case [datafusion]
Posted by "puneetdixit200 (via GitHub)" <gi...@apache.org>.
puneetdixit200 commented on issue #22225:
URL: https://github.com/apache/datafusion/issues/22225#issuecomment-4467919670
I’d like to work on this optimization. I can add a benchmark for the perfect-zip case, avoid row-by-row copying where input arrays align, and verify output compatibility with existing nested-array tests. Please assign this to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Optimize `arrays_zip` to avoid row-by-row copying in the perfect-zip case [datafusion]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey closed issue #22225: Optimize `arrays_zip` to avoid row-by-row copying in the perfect-zip case
URL: https://github.com/apache/datafusion/issues/22225
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Optimize `arrays_zip` to avoid row-by-row copying in the perfect-zip case [datafusion]
Posted by "BipashaBi (via GitHub)" <gi...@apache.org>.
BipashaBi commented on issue #22225:
URL: https://github.com/apache/datafusion/issues/22225#issuecomment-4466517441
greetings @adriangb !
i have a PR which solves this issue. Please kindly check it ASAP.
<img width="1136" height="908" alt="Image" src="https://github.com/user-attachments/assets/0a4b39fb-0ed8-419b-9960-0a76c404d037" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org