You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "kentkwu (via GitHub)" <gi...@apache.org> on 2026/09/02 22:35:14 UTC
[PR] fix(ipc): body prefix must be uncompressed length [arrow-js]
kentkwu opened a new pull request, #470:
URL: https://github.com/apache/arrow-js/pull/470
## Summary
The Arrow columnar format spec requires the eight-byte prefix on each compressed IPC body buffer to hold the *uncompressed* length so a reader can size the decompression destination buffer.
> Spec: https://arrow.apache.org/docs/format/Columnar.html#compression
the body includes a flat sequence of compressed buffers together with the length of the uncompressed buffer as a 64-bit little-endian signed integer stored in the first 8 bytes of each buffer in the sequence. This uncompressed length can be set to -1 to indicate that that specific buffer is left uncompressed.
The writer was emitting the compressed length instead, so PyArrow rejects Arrow JS-produced ZSTD and LZ4_FRAME streams. JS-to-JS round trips missed the bug because the reader sizes buffers from the codec frame, not the prefix. The compression code path is shared between the stream and file writers, so the one-line fix covers both formats.
- `src/ipc/writer.ts`: write `byteBuf.length` in the prefix instead of `compressed.length`
- New byte-level regression test in both writer suites (two codecs × stream and file) decompresses each emitted body buffer and asserts the prefix equals the decompressed length
- Extract `registerCompressionCodecs` and the new prefix inspector into `test/unit/ipc/writer/compression-codecs.ts`, dropping a duplicate copy from `file-writer-tests.ts`
Cross-language interop verified against PyArrow 25.0.1.
## Test Plan
- [x] `npm test`
- [x] `npm run build`
- [x] `npm run lint:ci`
- [x] All four regression cases (two codecs × stream and file writers) fail against the unfixed writer and pass with the fix
## Related
- Arrow columnar format spec, compression section: https://arrow.apache.org/docs/format/Columnar.html#compression
- Cross-language reproducer by @ianmcook: https://gist.github.com/ianmcook/cae432c969498adf56f5b3d437eb7d92
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(ipc): body prefix must be uncompressed length [arrow-js]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #470:
URL: https://github.com/apache/arrow-js/pull/470#issuecomment-5519405437
Could you check CI failures?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(ipc): body prefix must be uncompressed length [arrow-js]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou merged PR #470:
URL: https://github.com/apache/arrow-js/pull/470
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(ipc): body prefix must be uncompressed length [arrow-js]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #470:
URL: https://github.com/apache/arrow-js/pull/470#issuecomment-5534109033
@trxcllnt Do you want to review this before we merge this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org