You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "0lai0 (via GitHub)" <gi...@apache.org> on 2026/08/05 04:10:29 UTC
[I] Use the offset buffer for Map in spark_size [datafusion-comet]
0lai0 opened a new issue, #5266:
URL: https://github.com/apache/datafusion-comet/issues/5266
### What is the problem the feature request solves?
Follow-up from #5233
`spark_size` for `List` / `LargeList` / `FixedSizeList` now reuses Arrow's `length` kernel.
The `Map` branch in `native/spark-expr/src/array_funcs/size.rs` is still a per-row loop that appends `MapArray::value_length(i)` into an `Int32Builder`.
`length` does not accept `MapArray`, so Map cannot reuse the kernel. It was kept out of #5233 on purpose: different technique, and its own correctness and benchmark surface.
### Describe the potential solution
`MapArray::offsets()` is the entry-range offset buffer; for each row, `offsets[i+1] - offsets[i]` is the entry count (same value `value_length(i)` returns one row at a time). Vectorize that and apply Spark's null → `-1` rewrite with the same `set_indices` patch the list path uses
### Additional context
Before merging:
- Add a Map shape (with and without nulls) to `benches/array_size.rs` — there is no Map coverage today. The win is expected to be smaller than for List, since `value_length` already reads the offsets; only per-row dispatch and builder overhead go away. If the benchmark comes back flat, closing this as not worth the churn is fine.
- Add a sliced-`MapArray` test, mirroring the sliced-List test added in #5233. `MapArray::offsets()` returns the sliced offsets so `windows(2)` stays correct, but pin the invariant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Use the offset buffer for Map in spark_size [datafusion-comet]
Posted by "0lai0 (via GitHub)" <gi...@apache.org>.
0lai0 commented on issue #5266:
URL: https://github.com/apache/datafusion-comet/issues/5266#issuecomment-5187446981
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Use the offset buffer for Map in spark_size [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao closed issue #5266: Use the offset buffer for Map in spark_size
URL: https://github.com/apache/datafusion-comet/issues/5266
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org