You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "linhongyu510 (via GitHub)" <gi...@apache.org> on 2026/09/03 06:55:13 UTC

[PR] fix(arrow-avro): reset union offsets after flush [arrow-rs]

linhongyu510 opened a new pull request, #10971:
URL: https://github.com/apache/arrow-rs/pull/10971

   # Which issue does this PR close?
   
   - Closes #10876.
   
   # Rationale for this change
   
   `UnionDecoderBranches::counts` tracks the next dense-union offset for each branch. `flush` drained the type IDs, offsets, and child decoders but left these counters at the previous batch totals, so the next batch referenced positions beyond its newly emptied children.
   
   # What changes are included in this PR?
   
   - Reset all per-branch union counters after `UnionArray::try_new` succeeds.
   - Add a regression test that decodes and flushes two consecutive batches through the same dense union decoder and verifies each batch starts at offset zero.
   
   The reset intentionally happens only after successful array construction, so a failed flush does not additionally mutate the counters.
   
   # Are these changes tested?
   
   Yes. TDD evidence on current `main`:
   
   - Before the production change, the second flush failed with `Offsets must be non-negative and within the length of the Array`.
   - After the change, the focused regression passes.
   - `cargo test -p arrow-avro --all-features`: 487 passed.
   - Doc tests: 27 passed, 1 repository-marked ignored.
   - `cargo fmt --all -- --check`: passed.
   - `cargo clippy -p arrow-avro --all-targets --all-features -- -D warnings`: passed.
   - `git diff --check`: passed.
   
   The first all-features run had 399 passes and 88 failures because the official `testing` submodule was not initialized; every failure referenced missing `testing/data/avro` fixtures. After `git submodule update --init --depth 1 testing`, the complete suite passed as reported above.
   
   # Are there any user-facing changes?
   
   Yes. A decoder containing a multi-branch Avro union can now emit multiple batches without producing stale dense-union offsets. There are no API changes.
   
   # AI assistance
   
   I used AI assistance to investigate the decoder state, implement the focused regression and fix, and prepare this description. I reviewed the complete diff and verified all reported commands and outputs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] fix(arrow-avro): reset union offsets after flush [arrow-rs]

Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #10971:
URL: https://github.com/apache/arrow-rs/pull/10971#issuecomment-5525403668

   thanks @linhongyu510 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] fix(arrow-avro): reset union offsets after flush [arrow-rs]

Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey merged PR #10971:
URL: https://github.com/apache/arrow-rs/pull/10971


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org