You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "skyon0522-ai (via GitHub)" <gi...@apache.org> on 2026/10/17 13:33:43 UTC
[PR] Fix IPC serialization of unions in sliced lists [arrow-rs]
skyon0522-ai opened a new pull request, #11651:
URL: https://github.com/apache/arrow-rs/pull/11651
# Which issue does this PR close?
- Closes #11586
# Rationale for this change
Writing a union inside a sliced List or LargeList currently writes unsliced union buffers despite the sliced row count. Dense unions can read back values from different rows, while sparse unions can fail with a child-length mismatch.
# What changes are included in this PR?
Slice union type-ID and dense-offset buffers to the selected row range, and apply that row slice to sparse children before encoding. Retain dense children and the absolute offset values within their existing child views.
# Are these changes tested?
A regression test round-trips List and LargeList arrays containing dense or sparse unions through both File and Stream writers. It covers full, nonzero-offset and empty slices, nullable values, custom type IDs, and nonmonotonic dense offsets, comparing decoded rows and validating input/output data.
AI assistance was used to analyze the serialization path and draft the code, regression test, and this description.
# Are there any significant user-facing changes?
No public API changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix IPC serialization of unions in sliced lists [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on code in PR #11651:
URL: https://github.com/apache/arrow-rs/pull/11651#discussion_r4305563312
##########
arrow-ipc/src/writer.rs:
##########
@@ -2835,6 +2867,92 @@ mod tests {
use super::*;
+ #[test]
+ fn test_roundtrip_sliced_list_of_union() {
Review Comment:
we can simplify the test to just test list, and not largelist too, since its the same code path
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix IPC serialization of unions in sliced lists [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey merged PR #11651:
URL: https://github.com/apache/arrow-rs/pull/11651
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org