You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "rustyconover (via GitHub)" <gi...@apache.org> on 2026/02/12 16:50:46 UTC
[PR] [C++][Python] Add public APIs for reading and serializing IPC dictionary messages [arrow]
rustyconover opened a new pull request, #49262:
URL: https://github.com/apache/arrow/pull/49262
This adds low-level APIs for working with IPC dictionary messages outside of the stream/file reader/writer context, enabling message-at-a-time IPC workflows with dictionary-encoded data. I'm trying to work with record batches that contain dictionaries with record batches being serialized to shared memory so I need additional methods to handle dictionary IPC messages.
I am addressing my issues from #49258.
C++ changes:
- Add public `ReadDictionary(Message, DictionaryMemo*, IpcReadOptions)` to read a single dictionary message into a memo
- Add `CollectAndSerializeDictionaries(RecordBatch, DictionaryMemo*, IpcWriteOptions)` to serialize dictionary messages with pointer-based deduplication
- Expose `dictionary_memo()` accessor on `RecordBatchStreamReader` and `RecordBatchFileReader`
- Refactor internal `ReadDictionary` to `ReadDictionaryMessage` in `StreamDecoderInternal`; make `dictionary_memo_` protected
Python changes:
- Add `ipc.read_dictionary_message()` to populate a `DictionaryMemo` from a dictionary `Message` or `Buffer`
- Add `RecordBatch.serialize_dictionaries()` to serialize dictionary IPC messages with memo-based deduplication
- Add dictionary_memo property on `RecordBatchStreamReader` and `RecordBatchFileReader`
- Add `DictionaryMemo.wrap()` for non-owning references to reader memos
- Add `read_dictionary_message` to API docs
- Comprehensive test coverage for all new APIs
**AI Disclosure: I used Claude help me prepare this diff and PR. I will be responsible for all bugs, problems or inconsistencies.**
### Are there any user-facing changes?
Yes they are documented above in the Python changes.
**This PR includes breaking changes to public APIs.**
I'm making `dictionary_memo_` on `StreamDecoderInternal`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49258: [C++][Python] Add public APIs for reading and serializing IPC dictionary messages [arrow]
Posted by "rustyconover (via GitHub)" <gi...@apache.org>.
rustyconover commented on PR #49262:
URL: https://github.com/apache/arrow/pull/49262#issuecomment-3892709151
Hi @rok, @raulcd,
I don't know if the CI failures are my fault. It doesn't seem so.
Rusty
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49258: [C++][Python] Add public APIs for reading and serializing IPC dictionary messages [arrow]
Posted by "rustyconover (via GitHub)" <gi...@apache.org>.
rustyconover commented on PR #49262:
URL: https://github.com/apache/arrow/pull/49262#issuecomment-3897978149
Hi @raulcd,
Thanks for looking at this.
My use case is "message-at-a-time" IPC over shared memory
I have two processes (client and server) transferring Arrow data via shared memory + a pipe. Shared memory holds the IPC message bodies; the pipe conveys (offset, length) pairs telling the other side where to read. On my MacBook Air, pipe throughput is ~4 GB/s vs ~20 GB/s for shared memory, so the payoff is significant.
The workflow looks roughly like this:
```python
# Producer
memo = pa.ipc.DictionaryMemo()
for batch in batches:
# Serialize any new dictionaries not yet in the memo
for dict_buf in batch.serialize_dictionaries(memo):
write_to_shm(dict_buf)
notify_via_pipe(offset, length)
# Serialize the record batch (indices only)
batch_buf = batch.serialize()
write_to_shm(batch_buf)
notify_via_pipe(offset, length)
```
```python
# Consumer
memo = pa.ipc.DictionaryMemo()
for offset, length in read_pipe():
msg = pa.ipc.read_message(shm[offset:offset+length])
if msg.type == 'dictionary':
pa.ipc.read_dictionary_message(msg, memo)
elif msg.type == 'record batch':
batch = pa.ipc.read_record_batch(msg, schema, memo)
```
Why not `ipc.serialize_record_batch_with_dictionaries()`?
In a streaming workflow, dictionaries are often written once and reused across many batches. A combined function would either re-serialize dictionaries unnecessarily or need the same DictionaryMemo tracking anyway. Keeping them separate gives the caller control over when dictionaries are emitted, which is exactly what the stream/file writers do internally. I'm just exposing that building block.
That said, I'm not attached to the method living on RecordBatch. If you'd prefer it as a free function like ipc.serialize_dictionaries(batch, memo), I'm happy to move it — the important thing is that the memo-based deduplication is available to users.
Happy to jump on a call if I can make this easier to understand or demo.
Rusty
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49258: [C++][Python] Add public APIs for reading and serializing IPC dictionary messages [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on PR #49262:
URL: https://github.com/apache/arrow/pull/49262#issuecomment-3896302992
BTW, I opened an issue for one of the failures (the segfault one) as I've seen this happening on other PRs:
- https://github.com/apache/arrow/issues/49272
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [C++][Python] Add public APIs for reading and serializing IPC dictionary messages [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #49262:
URL: https://github.com/apache/arrow/pull/49262#issuecomment-3892091301
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
Thanks for opening a pull request!
If this is not a [minor PR](https://github.com/apache/arrow/blob/main/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
or
MINOR: [${COMPONENT}] ${SUMMARY}
See also:
* [Other pull requests](https://github.com/apache/arrow/pulls/)
* [Contribution Guidelines - Contributing Overview](https://arrow.apache.org/docs/developers/overview.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49258: [C++][Python] Add public APIs for reading and serializing IPC dictionary messages [arrow]
Posted by "rustyconover (via GitHub)" <gi...@apache.org>.
rustyconover commented on PR #49262:
URL: https://github.com/apache/arrow/pull/49262#issuecomment-3898316603
@raulcd, I'll move `serialize_dictionaries` down to the `ipc` namespace.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49258: [C++][Python] Add public APIs for reading and serializing IPC dictionary messages [arrow]
Posted by "rustyconover (via GitHub)" <gi...@apache.org>.
rustyconover commented on PR #49262:
URL: https://github.com/apache/arrow/pull/49262#issuecomment-3898259720
@raulcd, I'm a little nervous about delta dictionaries I need to read more into them.
My anxiety comes from if a consumer keeps a record batch in memory, and then a delta is applied to its dictionary it references as a new record batch arrives. If that older record batch is used it would see the updated dictionary and potentially have incorrect values? Or are dictionary deltas append only?
I just don't know about dictionary deltas enough to not be anxious. But I'll learn.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49258: [C++][Python] Add public APIs for reading and serializing IPC dictionary messages [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on PR #49262:
URL: https://github.com/apache/arrow/pull/49262#issuecomment-3898089343
Hi @rustyconover no need to jump in a call (even though happy to have a chat and a coffee).
The use case makes sense to me. As per the API, personally, a function like `ipc.serialize_dictionaries(batch, memo)` makes more sense to me. On the C++ API everything already sits on the `ipc/{reader/writer}.h` so feels natural to have this into `pyarrow.ipc`.
I think dictionary deltas could shine on your use case. The current implementation does full dictionary replacement when the pointer changes. For large dictionaries over many batches that could be a significant improvement maybe you want to consider it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49258: [C++][Python] Add public APIs for reading and serializing IPC dictionary messages [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on PR #49262:
URL: https://github.com/apache/arrow/pull/49262#issuecomment-3892963506
> I don't know if the CI failures are my fault. It doesn't seem so.
They were successful on re-run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49258: [C++][Python] Add public APIs for reading and serializing IPC dictionary messages [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #49262:
URL: https://github.com/apache/arrow/pull/49262#issuecomment-3892098473
:warning: GitHub issue #49258 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org