You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Rusty Conover <ru...@query.farm> on 2026/02/20 17:21:08 UTC
Missing test coverage for RecordBatch based custom_metadata in IPC Test Files
Hi all,
I was looking at the arrow-testing repository for coverage of the IPC format's `custom_metadata` feature on `Message` flatbuffers and found that no files in the repo exercise this.
**What is Message-level `custom_metadata`?**
The Arrow IPC `Message` flatbuffer definition includes an optional `custom_metadata` field — a vector of `KeyValue` pairs that can be attached to any message, including `RecordBatch` messages. This is distinct from schema-level metadata and field-level metadata, which are already well-tested by the `generated_custom_metadata` integration files.
The relevant part of the flatbuffer definition (`Message.fbs`):
```
table Message {
version: org.apache.arrow.flatbuf.MetadataVersion;
header: MessageHeader;
bodyLength: long;
custom_metadata: [org.apache.arrow.flatbuf.KeyValue]; // <-- this field
}
```
**What I found**
I scanned all record batch messages across the integration gold files, the IPC fuzz regression files, and the forward-compatibility files. None of them populate `custom_metadata` on any `RecordBatch` message.
The existing `generated_custom_metadata` files in the integration directories only exercise metadata at the schema and field level — not at the per-message level.
**Why this matters**
Without integration test data for this feature, implementations may silently drop, corrupt, or fail to round-trip per-message `custom_metadata`. Adding test files would help ensure that:
- Readers correctly parse `custom_metadata` from `RecordBatch` messages
- Writers correctly serialize it
- Round-trip integrity is maintained across implementations
- The field is properly distinguished from schema/field-level metadata
Since issues aren't enabled on the arrow-testing repository, I'm raising this here. Would it make sense to add a `generated_custom_metadata_record_batch` (or similar) integration test case to cover this gap?
I've also opened PRs to various Arrow implementations to add this feature to their implementations but I'm commonly adding a test file to the PRs for testing.
Rust: https://github.com/apache/arrow-rs/pull/9445
Go: https://github.com/apache/arrow-go/pull/669
Javascript: https://github.com/apache/arrow-js/pull/361
Thank you,
Rusty