You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Rusty Conover <ru...@query.farm> on 2026/02/03 16:31:17 UTC
[DISCUSS] IPC MessageType - OpaqueBytes
Hi Arrow Friends,
I’ve really appreciated Arrow Flight’s ability to carry custom metadata messages alongside record batches. In some of my current work, however, I’m dealing with Arrow IPC streams that are *not* sent via Flight, and I’d like to have a comparable capability there as well.
To support this, I’d like to propose adding a new IPC message type—tentatively named `*OpaqueBytes*`—that would allow arbitrary bytes to be embedded directly within IPC streams. IPC readers that do not understand this message type could safely ignore it, preserving compatibility.
My motivation is to enable multiplexing of auxiliary messages within a stream that otherwise consists of schemas, dictionaries, and record batches. A concrete example would be interleaving logging or signaling messages with record batches. Today, I’m approximating this by emitting zero-row record batches with binary metadata, but this approach is awkward and incurs unnecessary overhead due to schema complexity.
An `OpaqueBytes` IPC message type could enable a range of use cases, including (but not limited to) logging, flow control, signaling, and other auxiliary communication needs that don’t naturally map to record batches.
I briefly discussed this idea a few weeks ago on the Apache Arrow call, but wanted to share it here to reach a broader audience and gather more feedback.
In addition to the message type itself, I’d also be interested in hearing thoughts on how PyArrow’s interfaces might be extended to allow users to read and write these arbitrary messages as part of existing IPC stream readers and writers.
Looking forward to your thoughts and discussion.
Kind regards,
Rusty
Re: [DISCUSS] IPC stream multiplexing
Posted by Vignesh Siva <vi...@gmail.com>.
Hi all,
I found this exchange very helpful in clarifying the requirements around
“multiplexing.”
It seems there are two different notions being discussed under the same
term:
1.
Transport-level multiplexing, where independent streams make progress
independently (as with QUIC), primarily to avoid head-of-line blocking.
2.
*Application-level interleaving*, where multiple logical data sources
are serialized into a single, totally ordered stream with explicit
sequencing semantics.
Antoine’s point makes a lot of sense for (1): if the goal is parallelism
and independence, this clearly belongs at the transport layer, and QUIC is
a good fit.
Rusty’s example, however, appears closer to (2): a single, ordered IPC
stream where messages from different logical producers are intentionally
interleaved, and later messages may depend on earlier ones across those
logical streams (even with different schemas). In that case, preserving a
single global ordering seems essential, and transport-level multiplexing
alone doesn’t provide that semantic.
From that perspective, it feels reasonable to keep IPC itself simple and
ordered and treat any “multiplexing” as an explicit, higher-level
convention encoded within the IPC stream (e.g., via message types or
annotations), rather than relying on transport behavior.
This also suggests that it may be useful to scope IPC changes narrowly
(e.g., ApplicationData / Empty messages with clear ordering semantics) and
leave broader multiplexing or parallelism concerns to Flight or
transport-layer choices.
Thanks for the thoughtful discussion; it’s been very illuminating to follow.
Best regards,
Vignesh
On Wed, 4 Feb 2026 at 22:35, Rusty Conover <ru...@query.farm> wrote:
> Hi Antoine,
>
> > Ideally IPC stream multiplexing should be done at the transport level,
> > for example using QUIC instead of TCP.
>
> I hoped QUIC would give me what I’m after, but I don’t think it does.
>
> My (admittedly non-expert) understanding is that QUIC streams are
> independently ordered: data on stream 2 is completely independent of data
> on stream 1, and either can make progress at any time. That’s great for
> avoiding head-of-line blocking, but it’s not the ordering model I desire.
>
> What I’m aiming for with interleaved/multiplexed IPC streams is a single,
> explicit ordering across record batches from different logical streams.
> That ordering allows data from IPC stream 2 to refer to previously
> processed data from IPC stream 1, even though the two streams may have
> different Arrow schemas.
>
> Because QUIC deliberately avoids imposing cross-stream ordering, I don’t
> think it provides the semantics I’d want in this case.
>
> Best,
>
> Rusty
>
Re: [DISCUSS] IPC stream multiplexing
Posted by Rusty Conover <ru...@query.farm>.
Hi Antoine,
> Ideally IPC stream multiplexing should be done at the transport level,
> for example using QUIC instead of TCP.
I hoped QUIC would give me what I’m after, but I don’t think it does.
My (admittedly non-expert) understanding is that QUIC streams are independently ordered: data on stream 2 is completely independent of data on stream 1, and either can make progress at any time. That’s great for avoiding head-of-line blocking, but it’s not the ordering model I desire.
What I’m aiming for with interleaved/multiplexed IPC streams is a single, explicit ordering across record batches from different logical streams. That ordering allows data from IPC stream 2 to refer to previously processed data from IPC stream 1, even though the two streams may have different Arrow schemas.
Because QUIC deliberately avoids imposing cross-stream ordering, I don’t think it provides the semantics I’d want in this case.
Best,
Rusty
Re: [DISCUSS] IPC stream multiplexing
Posted by Antoine Pitrou <an...@python.org>.
Hi Rusty,
Ideally IPC stream multiplexing should be done at the transport level,
for example using QUIC instead of TCP.
Regards
Antoine.
Le 03/02/2026 à 21:07, Rusty Conover a écrit :
> Hi Dewey,
>
> While thinking about this in a café in Amsterdam, another idea came to mind: most ApplicationData does have structure.
>
> It could be interesting to support multiplexing multiple IPC streams over the same socket. One way to do this would be to tag IPC messages with a destination stream ID, and have the IPC reader/writer emit messages annotated with that ID. On the receiving side, read_next() could yield messages from any active stream, leaving it to the user to interpret both the stream ID and the message itself.
>
> This is more of a thought experiment than a concrete proposal, since it would likely add significant complexity—but it felt related enough to mention.
>
> Best,
>
> Rusty
>
> On Tue, Feb 3, 2026, at 8:58 PM, Dewey Dunnington wrote:
>> Just a note that I think a dedicated ApplicationData message is a great
>> idea. Many spatial file formats embed a spatial index, which is too large
>> for schema metadata...in many cases it already exists in data I'd like to
>> stream over IPC and there's no good place to put it so it just gets dropped
>> by the producer and recomputed by the consumer. I had considered at one
>> point prototyping an arrow-based spatial format that placed this type of
>> data in an Arrow file with the extra spatial information after the EOS and
>> before the file footer; however, ApplicationData would be a much cleaner
>> approach. There are many instances of custom file formats built on top of
>> SQLite and I wonder if ApplicationData would open up something like that
>> for Arrow IPC (beyond just my spatial concept).
>>
>> Cheers,
>>
>> -dewey
>>
>> On Tue, Feb 3, 2026 at 1:28 PM Rusty Conover <ru...@query.farm> wrote:
>>
>>> Hi Antoine,
>>>
>>> It is nice to hear from you!
>>>
>>>> (I would perhaps also call it "application data" or something)
>>>
>>> I’m happy with ApplicationData as the name.
>>>
>>>> On the face of it, this looks like a reasonable idea, though I wonder if
>>>> it should be a separate message type *or* an optional field carried
>>>> together in RecordBatches.
>>>
>>> The main issue with carrying this in RecordBatch metadata is ordering.
>>> While IPC already supports `custom_metadata` via `write_batch` (which I’ve
>>> been using), that approach assumes the application data can be attached to
>>> a specific batch.
>>>
>>> In some cases, the application data and record batches are produced
>>> independently and cannot be cleanly associated. A concrete example is
>>> interleaving stderr output (arbitrary log messages) with record batches
>>> written to stdout, while preserving a single ordered IPC stream.
>>>
>>> I experimented with using zero-row record batches as a workaround, but
>>> this is inefficient: even with no rows, the serialized message size grows
>>> with schema complexity. I’ve measured this across several schemas; details
>>> and code are here:
>>>
>>> https://gist.github.com/rustyconover/6ff8cbd93369735287d80ae60436379e
>>>
>>> In short, zero-row batches can cost anywhere from ~120 bytes for simple
>>> schemas to ~450+ bytes for more complex ones, which makes this approach
>>> unattractive when trying to minimize bytes on the wire.
>>>
>>> For these reasons, a distinct IPC message type for application data seems
>>> like the cleanest solution. I’d be very interested in whether others have
>>> run into the need for this as well.
>>>
>>> Rusty
>>>
>>>
>>> On Tue, Feb 3, 2026, at 5:58 PM, Antoine Pitrou wrote:
>>>> Hi Rusty,
>>>>
>>>>
>>>>
>>>> Regards
>>>>
>>>> Antoine.
>>>>
>>>>
>>>> Le 03/02/2026 à 17:31, Rusty Conover a écrit :
>>>>> Hi Arrow Friends,
>>>>>
>>>>> I’ve really appreciated Arrow Flight’s ability to carry custom metadata
>>> messages alongside record batches. In some of my current work, however, I’m
>>> dealing with Arrow IPC streams that are *not* sent via Flight, and I’d like
>>> to have a comparable capability there as well.
>>>>>
>>>>> To support this, I’d like to propose adding a new IPC message
>>> type—tentatively named `*OpaqueBytes*`—that would allow arbitrary bytes to
>>> be embedded directly within IPC streams. IPC readers that do not understand
>>> this message type could safely ignore it, preserving compatibility.
>>>>>
>>>>> My motivation is to enable multiplexing of auxiliary messages within a
>>> stream that otherwise consists of schemas, dictionaries, and record
>>> batches. A concrete example would be interleaving logging or signaling
>>> messages with record batches. Today, I’m approximating this by emitting
>>> zero-row record batches with binary metadata, but this approach is awkward
>>> and incurs unnecessary overhead due to schema complexity.
>>>>>
>>>>> An `OpaqueBytes` IPC message type could enable a range of use cases,
>>> including (but not limited to) logging, flow control, signaling, and other
>>> auxiliary communication needs that don’t naturally map to record batches.
>>>>>
>>>>> I briefly discussed this idea a few weeks ago on the Apache Arrow call,
>>> but wanted to share it here to reach a broader audience and gather more
>>> feedback.
>>>>>
>>>>> In addition to the message type itself, I’d also be interested in
>>> hearing thoughts on how PyArrow’s interfaces might be extended to allow
>>> users to read and write these arbitrary messages as part of existing IPC
>>> stream readers and writers.
>>>>>
>>>>> Looking forward to your thoughts and discussion.
>>>>>
>>>>> Kind regards,
>>>>> Rusty
>>>
Re: [DISCUSS] IPC MessageType - OpaqueBytes
Posted by Rusty Conover <ru...@query.farm>.
Hi Dewey,
While thinking about this in a café in Amsterdam, another idea came to mind: most ApplicationData does have structure.
It could be interesting to support multiplexing multiple IPC streams over the same socket. One way to do this would be to tag IPC messages with a destination stream ID, and have the IPC reader/writer emit messages annotated with that ID. On the receiving side, read_next() could yield messages from any active stream, leaving it to the user to interpret both the stream ID and the message itself.
This is more of a thought experiment than a concrete proposal, since it would likely add significant complexity—but it felt related enough to mention.
Best,
Rusty
On Tue, Feb 3, 2026, at 8:58 PM, Dewey Dunnington wrote:
> Just a note that I think a dedicated ApplicationData message is a great
> idea. Many spatial file formats embed a spatial index, which is too large
> for schema metadata...in many cases it already exists in data I'd like to
> stream over IPC and there's no good place to put it so it just gets dropped
> by the producer and recomputed by the consumer. I had considered at one
> point prototyping an arrow-based spatial format that placed this type of
> data in an Arrow file with the extra spatial information after the EOS and
> before the file footer; however, ApplicationData would be a much cleaner
> approach. There are many instances of custom file formats built on top of
> SQLite and I wonder if ApplicationData would open up something like that
> for Arrow IPC (beyond just my spatial concept).
>
> Cheers,
>
> -dewey
>
> On Tue, Feb 3, 2026 at 1:28 PM Rusty Conover <ru...@query.farm> wrote:
>
>> Hi Antoine,
>>
>> It is nice to hear from you!
>>
>> > (I would perhaps also call it "application data" or something)
>>
>> I’m happy with ApplicationData as the name.
>>
>> > On the face of it, this looks like a reasonable idea, though I wonder if
>> > it should be a separate message type *or* an optional field carried
>> > together in RecordBatches.
>>
>> The main issue with carrying this in RecordBatch metadata is ordering.
>> While IPC already supports `custom_metadata` via `write_batch` (which I’ve
>> been using), that approach assumes the application data can be attached to
>> a specific batch.
>>
>> In some cases, the application data and record batches are produced
>> independently and cannot be cleanly associated. A concrete example is
>> interleaving stderr output (arbitrary log messages) with record batches
>> written to stdout, while preserving a single ordered IPC stream.
>>
>> I experimented with using zero-row record batches as a workaround, but
>> this is inefficient: even with no rows, the serialized message size grows
>> with schema complexity. I’ve measured this across several schemas; details
>> and code are here:
>>
>> https://gist.github.com/rustyconover/6ff8cbd93369735287d80ae60436379e
>>
>> In short, zero-row batches can cost anywhere from ~120 bytes for simple
>> schemas to ~450+ bytes for more complex ones, which makes this approach
>> unattractive when trying to minimize bytes on the wire.
>>
>> For these reasons, a distinct IPC message type for application data seems
>> like the cleanest solution. I’d be very interested in whether others have
>> run into the need for this as well.
>>
>> Rusty
>>
>>
>> On Tue, Feb 3, 2026, at 5:58 PM, Antoine Pitrou wrote:
>> > Hi Rusty,
>> >
>> >
>> >
>> > Regards
>> >
>> > Antoine.
>> >
>> >
>> > Le 03/02/2026 à 17:31, Rusty Conover a écrit :
>> >> Hi Arrow Friends,
>> >>
>> >> I’ve really appreciated Arrow Flight’s ability to carry custom metadata
>> messages alongside record batches. In some of my current work, however, I’m
>> dealing with Arrow IPC streams that are *not* sent via Flight, and I’d like
>> to have a comparable capability there as well.
>> >>
>> >> To support this, I’d like to propose adding a new IPC message
>> type—tentatively named `*OpaqueBytes*`—that would allow arbitrary bytes to
>> be embedded directly within IPC streams. IPC readers that do not understand
>> this message type could safely ignore it, preserving compatibility.
>> >>
>> >> My motivation is to enable multiplexing of auxiliary messages within a
>> stream that otherwise consists of schemas, dictionaries, and record
>> batches. A concrete example would be interleaving logging or signaling
>> messages with record batches. Today, I’m approximating this by emitting
>> zero-row record batches with binary metadata, but this approach is awkward
>> and incurs unnecessary overhead due to schema complexity.
>> >>
>> >> An `OpaqueBytes` IPC message type could enable a range of use cases,
>> including (but not limited to) logging, flow control, signaling, and other
>> auxiliary communication needs that don’t naturally map to record batches.
>> >>
>> >> I briefly discussed this idea a few weeks ago on the Apache Arrow call,
>> but wanted to share it here to reach a broader audience and gather more
>> feedback.
>> >>
>> >> In addition to the message type itself, I’d also be interested in
>> hearing thoughts on how PyArrow’s interfaces might be extended to allow
>> users to read and write these arbitrary messages as part of existing IPC
>> stream readers and writers.
>> >>
>> >> Looking forward to your thoughts and discussion.
>> >>
>> >> Kind regards,
>> >> Rusty
>>
Re: [DISCUSS] IPC MessageType - OpaqueBytes
Posted by Dewey Dunnington <de...@gmail.com>.
Just a note that I think a dedicated ApplicationData message is a great
idea. Many spatial file formats embed a spatial index, which is too large
for schema metadata...in many cases it already exists in data I'd like to
stream over IPC and there's no good place to put it so it just gets dropped
by the producer and recomputed by the consumer. I had considered at one
point prototyping an arrow-based spatial format that placed this type of
data in an Arrow file with the extra spatial information after the EOS and
before the file footer; however, ApplicationData would be a much cleaner
approach. There are many instances of custom file formats built on top of
SQLite and I wonder if ApplicationData would open up something like that
for Arrow IPC (beyond just my spatial concept).
Cheers,
-dewey
On Tue, Feb 3, 2026 at 1:28 PM Rusty Conover <ru...@query.farm> wrote:
> Hi Antoine,
>
> It is nice to hear from you!
>
> > (I would perhaps also call it "application data" or something)
>
> I’m happy with ApplicationData as the name.
>
> > On the face of it, this looks like a reasonable idea, though I wonder if
> > it should be a separate message type *or* an optional field carried
> > together in RecordBatches.
>
> The main issue with carrying this in RecordBatch metadata is ordering.
> While IPC already supports `custom_metadata` via `write_batch` (which I’ve
> been using), that approach assumes the application data can be attached to
> a specific batch.
>
> In some cases, the application data and record batches are produced
> independently and cannot be cleanly associated. A concrete example is
> interleaving stderr output (arbitrary log messages) with record batches
> written to stdout, while preserving a single ordered IPC stream.
>
> I experimented with using zero-row record batches as a workaround, but
> this is inefficient: even with no rows, the serialized message size grows
> with schema complexity. I’ve measured this across several schemas; details
> and code are here:
>
> https://gist.github.com/rustyconover/6ff8cbd93369735287d80ae60436379e
>
> In short, zero-row batches can cost anywhere from ~120 bytes for simple
> schemas to ~450+ bytes for more complex ones, which makes this approach
> unattractive when trying to minimize bytes on the wire.
>
> For these reasons, a distinct IPC message type for application data seems
> like the cleanest solution. I’d be very interested in whether others have
> run into the need for this as well.
>
> Rusty
>
>
> On Tue, Feb 3, 2026, at 5:58 PM, Antoine Pitrou wrote:
> > Hi Rusty,
> >
> >
> >
> > Regards
> >
> > Antoine.
> >
> >
> > Le 03/02/2026 à 17:31, Rusty Conover a écrit :
> >> Hi Arrow Friends,
> >>
> >> I’ve really appreciated Arrow Flight’s ability to carry custom metadata
> messages alongside record batches. In some of my current work, however, I’m
> dealing with Arrow IPC streams that are *not* sent via Flight, and I’d like
> to have a comparable capability there as well.
> >>
> >> To support this, I’d like to propose adding a new IPC message
> type—tentatively named `*OpaqueBytes*`—that would allow arbitrary bytes to
> be embedded directly within IPC streams. IPC readers that do not understand
> this message type could safely ignore it, preserving compatibility.
> >>
> >> My motivation is to enable multiplexing of auxiliary messages within a
> stream that otherwise consists of schemas, dictionaries, and record
> batches. A concrete example would be interleaving logging or signaling
> messages with record batches. Today, I’m approximating this by emitting
> zero-row record batches with binary metadata, but this approach is awkward
> and incurs unnecessary overhead due to schema complexity.
> >>
> >> An `OpaqueBytes` IPC message type could enable a range of use cases,
> including (but not limited to) logging, flow control, signaling, and other
> auxiliary communication needs that don’t naturally map to record batches.
> >>
> >> I briefly discussed this idea a few weeks ago on the Apache Arrow call,
> but wanted to share it here to reach a broader audience and gather more
> feedback.
> >>
> >> In addition to the message type itself, I’d also be interested in
> hearing thoughts on how PyArrow’s interfaces might be extended to allow
> users to read and write these arbitrary messages as part of existing IPC
> stream readers and writers.
> >>
> >> Looking forward to your thoughts and discussion.
> >>
> >> Kind regards,
> >> Rusty
>
Re: [DISCUSS] IPC MessageType - OpaqueBytes
Posted by Dewey Dunnington <de...@gmail.com>.
> I thought about that, but then it means the application data can only be
> transmitted in the new Empty message, not with a RecordBatch.
Agreed! Both are useful.
> I hope we can find a way of transporting statistics together with the
> corresponding RecordBatch message, as opposed to a separate message in
> the IPC stream.
That's a great point...I was mostly trying to validate Rusty's point that I
can see how that type of extra data might well be a table.
On Thu, Feb 5, 2026 at 2:46 AM Antoine Pitrou <an...@python.org> wrote:
>
> Le 05/02/2026 à 04:44, Dewey Dunnington a écrit :
> >> a new application_data field in the Message table to pass arbitrary
> >> opaque data with any kind of message
> >
> > I believe this could be done with the Empty message by putting the bytes
> in
> > the body instead of in the header. Probably the only place this
> > functionally makes a difference would be dissociated IPC where the body
> is
> > transported separately. Perhaps both are useful.
>
> I thought about that, but then it means the application data can only be
> transmitted in the new Empty message, not with a RecordBatch.
>
> That's not necessarily a problem, just a limitation to think about.
>
> >> It could be interesting to support multiplexing multiple IPC streams
> over
> > the same socket
> >
> > I agree that there are some applications of the Empty where it would be
> > tempting to have the payload of the Empty be serialized IPC (e.g., if
> used
> > for statistics and the statistics are encoded in the lovely Arrow spec we
> > have for that). Perhaps with Empty one could prototype that.
>
> I hope we can find a way of transporting statistics together with the
> corresponding RecordBatch message, as opposed to a separate message in
> the IPC stream.
>
> Regards
>
> Antoine.
>
>
Re: [DISCUSS] IPC MessageType - OpaqueBytes
Posted by Vignesh Siva <vi...@gmail.com>.
Hi all,
Thanks for the thoughtful discussion; this has been really helpful to
follow.
It seems to me that there are two slightly different (but related) needs
being discussed:
1.
The ability to carry *opaque, non-string bytes* in IPC, avoiding the
overhead and semantic mismatch of zero-row RecordBatches and the
string-only limitation of custom_metadata.
2.
The ability to control the association* and ordering* of such data with
respect to RecordBatch messages (sometimes tightly coupled, sometimes
intentionally independent).
From that perspective, the combination Antoine suggested of a lightweight
empty message type plus an application_data bytes field feels like a nice
decomposition:
-
Empty provides a low-overhead, ordering-preserving carrier when the data
is intentionally independent.
-
An application_data field on Message allows attaching bytes directly to
RecordBatch (or other messages) when tight association is desired (e.g.,
statistics, per-batch signals).
This also seems to align well with David’s point about avoiding base64 and
with Dewey’s use cases where the payload is meaningful but doesn’t
naturally fit schema metadata.
One thing I like about this direction is that it keeps the initial scope
focused: it doesn’t force multiplexing or structured interpretation up
front but still leaves room to experiment (e.g., embedding serialized IPC
in Empty or evolving higher-level conventions later).
From an implementation point of view, it also seems feasible to prototype
incrementally:
-
introduce Empty + application_data(bytes) in the IPC format
-
initially treat application_data as opaque and pass-through in
readers/writers
-
let higher-level libraries decide how (or whether) to interpret it
Happy to help with prototyping or reviewing pieces of this if that’s useful.
Best regards,
Vignesh
On Thu, 5 Feb 2026 at 14:17, Antoine Pitrou <an...@python.org> wrote:
>
> Le 05/02/2026 à 04:44, Dewey Dunnington a écrit :
> >> a new application_data field in the Message table to pass arbitrary
> >> opaque data with any kind of message
> >
> > I believe this could be done with the Empty message by putting the bytes
> in
> > the body instead of in the header. Probably the only place this
> > functionally makes a difference would be dissociated IPC where the body
> is
> > transported separately. Perhaps both are useful.
>
> I thought about that, but then it means the application data can only be
> transmitted in the new Empty message, not with a RecordBatch.
>
> That's not necessarily a problem, just a limitation to think about.
>
> >> It could be interesting to support multiplexing multiple IPC streams
> over
> > the same socket
> >
> > I agree that there are some applications of the Empty where it would be
> > tempting to have the payload of the Empty be serialized IPC (e.g., if
> used
> > for statistics and the statistics are encoded in the lovely Arrow spec we
> > have for that). Perhaps with Empty one could prototype that.
>
> I hope we can find a way of transporting statistics together with the
> corresponding RecordBatch message, as opposed to a separate message in
> the IPC stream.
>
> Regards
>
> Antoine.
>
>
Re: [DISCUSS] IPC MessageType - OpaqueBytes
Posted by Antoine Pitrou <an...@python.org>.
Le 05/02/2026 à 04:44, Dewey Dunnington a écrit :
>> a new application_data field in the Message table to pass arbitrary
>> opaque data with any kind of message
>
> I believe this could be done with the Empty message by putting the bytes in
> the body instead of in the header. Probably the only place this
> functionally makes a difference would be dissociated IPC where the body is
> transported separately. Perhaps both are useful.
I thought about that, but then it means the application data can only be
transmitted in the new Empty message, not with a RecordBatch.
That's not necessarily a problem, just a limitation to think about.
>> It could be interesting to support multiplexing multiple IPC streams over
> the same socket
>
> I agree that there are some applications of the Empty where it would be
> tempting to have the payload of the Empty be serialized IPC (e.g., if used
> for statistics and the statistics are encoded in the lovely Arrow spec we
> have for that). Perhaps with Empty one could prototype that.
I hope we can find a way of transporting statistics together with the
corresponding RecordBatch message, as opposed to a separate message in
the IPC stream.
Regards
Antoine.
Re: [DISCUSS] IPC MessageType - OpaqueBytes
Posted by Dewey Dunnington <de...@gmail.com>.
> a new Empty message type to avoid the overhead (and semantics) of
> empty record batches
I like this! I wonder if the name is a bit misleading (the messages will
contain *something*) but I'm happy with any name mapping to that concept.
> a new application_data field in the Message table to pass arbitrary
> opaque data with any kind of message
I believe this could be done with the Empty message by putting the bytes in
the body instead of in the header. Probably the only place this
functionally makes a difference would be dissociated IPC where the body is
transported separately. Perhaps both are useful.
> It could be interesting to support multiplexing multiple IPC streams over
the same socket
I agree that there are some applications of the Empty where it would be
tempting to have the payload of the Empty be serialized IPC (e.g., if used
for statistics and the statistics are encoded in the lovely Arrow spec we
have for that). Perhaps with Empty one could prototype that.
-dewey
On Wed, Feb 4, 2026 at 6:18 PM David Li <li...@apache.org> wrote:
> custom_metadata requires strings, so adding a bytes field may be useful to
> some applications, though. (This came up several years ago; you could
> always base64 encode but that takes time.)
>
> On Thu, Feb 5, 2026, at 02:33, Rusty Conover wrote:
> > Hi Antoine,
> >
> >> Ok, perhaps we can find a generic solution using two additions:
> >>
> >> 1) a new Empty message type to avoid the overhead (and semantics) of
> >> empty record batches
> >> 2) a new application_data field in the Message table to pass arbitrary
> >> opaque data with any kind of message
> >>
> >> Something like:
> >> https://gist.github.com/pitrou/363c4509706f56743f0ca0373f20949c
> >>
> >> What do you think?
> >
> > Thanks for making the gist. It looks good to me!
> >
> > One thought I've had about the `application_data` field is that it
> > could be considered to be duplicative since you could accomplish the
> > same thing with `custom_metadata`'s KeyValue. To do that you would need
> > a coordinated value for the "key" field from the writer to reader.
> >
> > How can I help with this?
> >
> > Thank you,
> >
> > Rusty
>
Re: [DISCUSS] IPC MessageType - OpaqueBytes
Posted by David Li <li...@apache.org>.
custom_metadata requires strings, so adding a bytes field may be useful to some applications, though. (This came up several years ago; you could always base64 encode but that takes time.)
On Thu, Feb 5, 2026, at 02:33, Rusty Conover wrote:
> Hi Antoine,
>
>> Ok, perhaps we can find a generic solution using two additions:
>>
>> 1) a new Empty message type to avoid the overhead (and semantics) of
>> empty record batches
>> 2) a new application_data field in the Message table to pass arbitrary
>> opaque data with any kind of message
>>
>> Something like:
>> https://gist.github.com/pitrou/363c4509706f56743f0ca0373f20949c
>>
>> What do you think?
>
> Thanks for making the gist. It looks good to me!
>
> One thought I've had about the `application_data` field is that it
> could be considered to be duplicative since you could accomplish the
> same thing with `custom_metadata`'s KeyValue. To do that you would need
> a coordinated value for the "key" field from the writer to reader.
>
> How can I help with this?
>
> Thank you,
>
> Rusty
Re: [DISCUSS] IPC MessageType - OpaqueBytes
Posted by Rusty Conover <ru...@query.farm>.
Hi Antoine,
> Ok, perhaps we can find a generic solution using two additions:
>
> 1) a new Empty message type to avoid the overhead (and semantics) of
> empty record batches
> 2) a new application_data field in the Message table to pass arbitrary
> opaque data with any kind of message
>
> Something like:
> https://gist.github.com/pitrou/363c4509706f56743f0ca0373f20949c
>
> What do you think?
Thanks for making the gist. It looks good to me!
One thought I've had about the `application_data` field is that it could be considered to be duplicative since you could accomplish the same thing with `custom_metadata`'s KeyValue. To do that you would need a coordinated value for the "key" field from the writer to reader.
How can I help with this?
Thank you,
Rusty
Re: [DISCUSS] IPC MessageType - OpaqueBytes
Posted by Antoine Pitrou <an...@python.org>.
Le 03/02/2026 à 20:27, Rusty Conover a écrit :
> Hi Antoine,
>
> It is nice to hear from you!
Ditto :-)
>> On the face of it, this looks like a reasonable idea, though I wonder if
>> it should be a separate message type *or* an optional field carried
>> together in RecordBatches.
>
> The main issue with carrying this in RecordBatch metadata is ordering. While IPC already supports `custom_metadata` via `write_batch` (which I’ve been using), that approach assumes the application data can be attached to a specific batch.
>
> In some cases, the application data and record batches are produced independently and cannot be cleanly associated. A concrete example is interleaving stderr output (arbitrary log messages) with record batches written to stdout, while preserving a single ordered IPC stream.
>
> I experimented with using zero-row record batches as a workaround, but this is inefficient: even with no rows, the serialized message size grows with schema complexity.
Ok, perhaps we can find a generic solution using two additions:
1) a new Empty message type to avoid the overhead (and semantics) of
empty record batches
2) a new application_data field in the Message table to pass arbitrary
opaque data with any kind of message
Something like:
https://gist.github.com/pitrou/363c4509706f56743f0ca0373f20949c
What do you think?
Regards
Antoine.
Re: [DISCUSS] IPC MessageType - OpaqueBytes
Posted by Rusty Conover <ru...@query.farm>.
Hi Antoine,
It is nice to hear from you!
> (I would perhaps also call it "application data" or something)
I’m happy with ApplicationData as the name.
> On the face of it, this looks like a reasonable idea, though I wonder if
> it should be a separate message type *or* an optional field carried
> together in RecordBatches.
The main issue with carrying this in RecordBatch metadata is ordering. While IPC already supports `custom_metadata` via `write_batch` (which I’ve been using), that approach assumes the application data can be attached to a specific batch.
In some cases, the application data and record batches are produced independently and cannot be cleanly associated. A concrete example is interleaving stderr output (arbitrary log messages) with record batches written to stdout, while preserving a single ordered IPC stream.
I experimented with using zero-row record batches as a workaround, but this is inefficient: even with no rows, the serialized message size grows with schema complexity. I’ve measured this across several schemas; details and code are here:
https://gist.github.com/rustyconover/6ff8cbd93369735287d80ae60436379e
In short, zero-row batches can cost anywhere from ~120 bytes for simple schemas to ~450+ bytes for more complex ones, which makes this approach unattractive when trying to minimize bytes on the wire.
For these reasons, a distinct IPC message type for application data seems like the cleanest solution. I’d be very interested in whether others have run into the need for this as well.
Rusty
On Tue, Feb 3, 2026, at 5:58 PM, Antoine Pitrou wrote:
> Hi Rusty,
>
>
>
> Regards
>
> Antoine.
>
>
> Le 03/02/2026 à 17:31, Rusty Conover a écrit :
>> Hi Arrow Friends,
>>
>> I’ve really appreciated Arrow Flight’s ability to carry custom metadata messages alongside record batches. In some of my current work, however, I’m dealing with Arrow IPC streams that are *not* sent via Flight, and I’d like to have a comparable capability there as well.
>>
>> To support this, I’d like to propose adding a new IPC message type—tentatively named `*OpaqueBytes*`—that would allow arbitrary bytes to be embedded directly within IPC streams. IPC readers that do not understand this message type could safely ignore it, preserving compatibility.
>>
>> My motivation is to enable multiplexing of auxiliary messages within a stream that otherwise consists of schemas, dictionaries, and record batches. A concrete example would be interleaving logging or signaling messages with record batches. Today, I’m approximating this by emitting zero-row record batches with binary metadata, but this approach is awkward and incurs unnecessary overhead due to schema complexity.
>>
>> An `OpaqueBytes` IPC message type could enable a range of use cases, including (but not limited to) logging, flow control, signaling, and other auxiliary communication needs that don’t naturally map to record batches.
>>
>> I briefly discussed this idea a few weeks ago on the Apache Arrow call, but wanted to share it here to reach a broader audience and gather more feedback.
>>
>> In addition to the message type itself, I’d also be interested in hearing thoughts on how PyArrow’s interfaces might be extended to allow users to read and write these arbitrary messages as part of existing IPC stream readers and writers.
>>
>> Looking forward to your thoughts and discussion.
>>
>> Kind regards,
>> Rusty
Re: [DISCUSS] IPC MessageType - OpaqueBytes
Posted by Antoine Pitrou <an...@python.org>.
Hi Rusty,
On the face of it, this looks like a reasonable idea, though I wonder if
it should be a separate message type *or* an optional field carried
together in RecordBatches.
(I would perhaps also call it "application data" or something)
Regards
Antoine.
Le 03/02/2026 à 17:31, Rusty Conover a écrit :
> Hi Arrow Friends,
>
> I’ve really appreciated Arrow Flight’s ability to carry custom metadata messages alongside record batches. In some of my current work, however, I’m dealing with Arrow IPC streams that are *not* sent via Flight, and I’d like to have a comparable capability there as well.
>
> To support this, I’d like to propose adding a new IPC message type—tentatively named `*OpaqueBytes*`—that would allow arbitrary bytes to be embedded directly within IPC streams. IPC readers that do not understand this message type could safely ignore it, preserving compatibility.
>
> My motivation is to enable multiplexing of auxiliary messages within a stream that otherwise consists of schemas, dictionaries, and record batches. A concrete example would be interleaving logging or signaling messages with record batches. Today, I’m approximating this by emitting zero-row record batches with binary metadata, but this approach is awkward and incurs unnecessary overhead due to schema complexity.
>
> An `OpaqueBytes` IPC message type could enable a range of use cases, including (but not limited to) logging, flow control, signaling, and other auxiliary communication needs that don’t naturally map to record batches.
>
> I briefly discussed this idea a few weeks ago on the Apache Arrow call, but wanted to share it here to reach a broader audience and gather more feedback.
>
> In addition to the message type itself, I’d also be interested in hearing thoughts on how PyArrow’s interfaces might be extended to allow users to read and write these arbitrary messages as part of existing IPC stream readers and writers.
>
> Looking forward to your thoughts and discussion.
>
> Kind regards,
> Rusty