You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Wes McKinney (JIRA)" <ji...@apache.org> on 2019/08/01 22:33:00 UTC
[jira] [Updated] (ARROW-5908) [C#] ArrowStreamWriter doesn't align
buffers to 8 bytes
[ https://issues.apache.org/jira/browse/ARROW-5908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wes McKinney updated ARROW-5908:
--------------------------------
Fix Version/s: (was: 1.0.0)
0.15.0
> [C#] ArrowStreamWriter doesn't align buffers to 8 bytes
> -------------------------------------------------------
>
> Key: ARROW-5908
> URL: https://issues.apache.org/jira/browse/ARROW-5908
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C#
> Reporter: Eric Erhardt
> Assignee: Eric Erhardt
> Priority: Critical
> Labels: pull-request-available
> Fix For: 0.14.1, 0.15.0
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> When writing RecordBatches using ArrowStreamWriter, if the ArrowBuffers being written aren't all 8 byte aligned, the serialized RecordBatch won't conform to the Arrow specification. This leads to other languages' readers to throw an error when reading Arrow streams written by the C# writer.
> For example, if reading the stream from Python or C++, an error is raised here:
> [https://github.com/apache/arrow/blob/f77c3427ca801597b572fb197b92b0133269049b/cpp/src/arrow/ipc/reader.cc#L107-L110]
> A similar error is raised when Java tries to read the stream.
> We should be ensuring that the buffers being written to the stream are padded to 8 bytes, no matter their length, as specified in [https://arrow.apache.org/docs/format/Layout.html#requirements-goals-and-non-goals]
>
> {quote} * It is required to have all the contiguous memory buffers in an IPC payload aligned at 8-byte boundaries. In other words, each buffer must start at an aligned 8-byte offset. Additionally, each buffer should be padded to a multiple of 8 bytes.{quote}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)