You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "willtemperley (via GitHub)" <gi...@apache.org> on 2025/10/15 04:30:12 UTC
[I] Encapsulated message format metadata_size ambiguity. [arrow]
willtemperley opened a new issue, #47824:
URL: https://github.com/apache/arrow/issues/47824
### Describe the enhancement requested
This is a follow-up to a discussion started in https://github.com/apache/arrow-swift/pull/95 where we were fixing some issues in `ArrowWriter`.
@kou was asking how the fixes lined up with the spec, and that was a difficult question to answer. Looking at the [encapsulated-message-format](https://arrow.apache.org/docs/format/Columnar.html#encapsulated-message-format):
"The metadata_size includes the size of the Message plus padding. The metadata_flatbuffer contains a serialized Message Flatbuffer value"
It seems reasonable to assume that the `Message` probably refers to a FlatBuffers `Message` therefore metadata_size should be exactly the length of this message plus padding. However, according to PyArrow this includes the `Message` size plus 8 bytes.
Doing a small experiment with the testFileWriter_bool example in arrow-swift, writing the block metadata without the 8 byte prefix in the metadataLength:
offset: 120
metadataLength: 208
bodyLength: 296
PyArrow won't open the file, throwing an error:
pyarrow.lib.ArrowInvalid: flatbuffer size 8 invalid. File offset: 128, metadata length: 208
(Note that the `Message` size is confirmed to be 208 bytes).
However if the metadataLength includes the 8 byte prefix, i.e.:
offset: 120
metadataLength: 216
bodyLength: 296
The file is valid according to PyArrow.
### Component(s)
Documentation, Python, Swift
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Format] Encapsulated message format metadata_size ambiguity. [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #47824:
URL: https://github.com/apache/arrow/issues/47824#issuecomment-6314696247
This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Format] Encapsulated message format metadata_size ambiguity. [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #47824:
URL: https://github.com/apache/arrow/issues/47824#issuecomment-3420071912
If I understand right:
This refers to the length written as part of the message:
> The metadata_size includes the size of the Message plus padding. The metadata_flatbuffer contains a serialized Message Flatbuffer value, which internally includes:
But what we're concerned about is the length written to the IPC file footer, which also includes this size prefix?
I wish PyArrow included more ability to introspect files, but if I serialize a message, I get this:
```
b'\xff\xff\xff\xff0\x00\x00\x00\x10\x00\x00\x00\x00\x00\n\x00\x0c\x00\x06\x00\x05\x00\x08\x00\n\x00\x00\x00\x00\x01\x04\x00\x0c\x00\x00\x00\x08\x00\x08\x00\x00\x00\x04\x00\x08\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\xff\xff\xff\xff\x00\x00\x00\x00'
```
which has `metadata_size` = 48
```
In [12]: struct.unpack('<I', b'0\x00\x00\x00')
Out[12]: (48,)
```
which is consistent with the quote from the documentation. But for the file footer, I assume it needs to be `56` instead, and that's where the confusion stems from (there are two "message lengths" implicitly being discussed here, one is the actual size of the Flatbuf message, one is the size of the "Arrow IPC message"?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Format] Encapsulated message format metadata_size ambiguity. [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #47824:
URL: https://github.com/apache/arrow/issues/47824#issuecomment-6334118940
Perhaps we can clarify the comment in the .fbs, and try to be consistent about specifying "IPC message" (Flatbuf + prefix) or "Message flatbuf" (or some other way of differentiating between them)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Format] Encapsulated message format metadata_size ambiguity. [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #47824:
URL: https://github.com/apache/arrow/issues/47824#issuecomment-6334088771
I think we should try to clarify this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Format] Encapsulated message format metadata_size ambiguity. [arrow]
Posted by "willtemperley (via GitHub)" <gi...@apache.org>.
willtemperley commented on issue #47824:
URL: https://github.com/apache/arrow/issues/47824#issuecomment-3420684950
@lidavidm Thanks for clarifying this. Yes exactly, `metadata_size` in the encapsulated message format and `metadata_size` in the `Block` struct in the footer refer to _almost_ the same thing, except the one in the footer is the `metadata_size` plus the encapsulated message format prefix length.
I think this is definitely confusing! Reading [encapsulated-message-format](https://arrow.apache.org/docs/format/Columnar.html#encapsulated-message-format) :
> IPC File Format
>
> We define a “file format” supporting random access that is an extension of the stream format. The file starts and ends with a magic string ARROW1 (plus padding). What follows in the file is identical to the stream format. At the end of the file, we write a footer containing a redundant copy of the schema (which is a part of the streaming format) plus memory offsets and sizes for each of the data blocks in the file. This enables random access to any record batch in the file. See [File.fbs](https://github.com/apache/arrow/blob/main/format/File.fbs) for the precise details of the file footer.
So looking at File.fbs we have:
```
struct Block {
/// Index to the start of the RecordBlock (note this is past the Message header)
offset: long;
/// Length of the metadata
metaDataLength: int;
/// Length of the data (this is aligned so there can be a gap between this and
/// the metadata).
bodyLength: long;
}
```
We have `metaDataLength` aka `metadata_size` but no mention of the prefix. Perhaps this could be made explicit?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Format] Encapsulated message format metadata_size ambiguity. [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #47824:
URL: https://github.com/apache/arrow/issues/47824#issuecomment-3425152896
Also cc @alex-placet @hind-m who are working on an [Arrow IPC implementation for sparrow](https://github.com/QuantStack/sparrow-ipc/).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Format] Encapsulated message format metadata_size ambiguity. [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #47824:
URL: https://github.com/apache/arrow/issues/47824#issuecomment-3420696668
Yup, I think differentiating/documenting these sizes is worthwhile. And maybe considering the terminology here as well (both the lengths, and what a "message" is)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Format] Encapsulated message format metadata_size ambiguity. [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #47824:
URL: https://github.com/apache/arrow/issues/47824#issuecomment-3413538934
@pitrou @paleolimbot @lidavidm What do you think about this?
The related part in the C++ implementation:
https://github.com/apache/arrow/blob/7a38744e979def14885c9ced423771af0916090d/cpp/src/arrow/ipc/message.cc#L567-L579
The related part in the nanoarrow:
https://github.com/apache/arrow-nanoarrow/blob/02ebb803ccfe7cc398bc7839c1f2de80db132794/src/nanoarrow/ipc/encoder.c#L103-L106
Should we update the `metadata_size` documentation to `metadata_size = sizeof(continuation) + sizeof(metadata_size) + Message size + padding`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Format] Encapsulated message format metadata_size ambiguity. [arrow]
Posted by "willtemperley (via GitHub)" <gi...@apache.org>.
willtemperley commented on issue #47824:
URL: https://github.com/apache/arrow/issues/47824#issuecomment-3425113119
It seems quite delicate because "message" refers to the entire message, whereas `Message` refers to the FlatBuffers metadata.
So in this context, "Message" means the FlatBuffers `Message` (I believe):
```
/// Index to the start of the RecordBlock (note this is past the Message header)
offset: long;
```
and presumably "Message header" refers to the continuation marker plus `Message` size, however I don't think this is actually correct, based on the fact that arrrow-swift is writing the `Block` offset as the start of the entire message (uncapitalised) and PyArrow can read it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org