You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Rusty Conover <ru...@query.farm> on 2026/02/20 17:21:08 UTC

Missing test coverage for RecordBatch based custom_metadata in IPC Test Files

Hi all,

I was looking at the arrow-testing repository for coverage of the IPC format's `custom_metadata` feature on `Message` flatbuffers and found that no files in the repo exercise this.

**What is Message-level `custom_metadata`?**

The Arrow IPC `Message` flatbuffer definition includes an optional `custom_metadata` field — a vector of `KeyValue` pairs that can be attached to any message, including `RecordBatch` messages. This is distinct from schema-level metadata and field-level metadata, which are already well-tested by the `generated_custom_metadata` integration files.

The relevant part of the flatbuffer definition (`Message.fbs`):

```
table Message {
  version: org.apache.arrow.flatbuf.MetadataVersion;
  header: MessageHeader;
  bodyLength: long;
  custom_metadata: [org.apache.arrow.flatbuf.KeyValue];  // <-- this field
}
```

**What I found**

I scanned all record batch messages across the integration gold files, the IPC fuzz regression files, and the forward-compatibility files. None of them populate `custom_metadata` on any `RecordBatch` message.

The existing `generated_custom_metadata` files in the integration directories only exercise metadata at the schema and field level — not at the per-message level.

**Why this matters**

Without integration test data for this feature, implementations may silently drop, corrupt, or fail to round-trip per-message `custom_metadata`. Adding test files would help ensure that:

- Readers correctly parse `custom_metadata` from `RecordBatch` messages
- Writers correctly serialize it
- Round-trip integrity is maintained across implementations
- The field is properly distinguished from schema/field-level metadata

Since issues aren't enabled on the arrow-testing repository, I'm raising this here. Would it make sense to add a `generated_custom_metadata_record_batch` (or similar) integration test case to cover this gap?

I've also opened PRs to various Arrow implementations to add this feature to their implementations but I'm commonly adding a test file to the PRs for testing.

Rust: https://github.com/apache/arrow-rs/pull/9445
Go: https://github.com/apache/arrow-go/pull/669
Javascript: https://github.com/apache/arrow-js/pull/361

Thank you,

Rusty