You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "rustyconover (via GitHub)" <gi...@apache.org> on 2026/02/20 02:28:41 UTC
[PR] feat(arrow/ipc): support custom_metadata on RecordBatch messages [arrow-go]
rustyconover opened a new pull request, #669:
URL: https://github.com/apache/arrow-go/pull/669
### Rationale for this change
Arrow IPC Messages wrap RecordBatch data in a Message flatbuffer which has a custom_metadata field (vector of KeyValue pairs). PyArrow and other implementations use this to attach per-batch metadata, but the Go implementation previously ignored it on both read and write paths.
### What changes are included in this PR?
Add a RecordBatchWithMetadata optional interface to avoid breaking the existing RecordBatch interface. The simpleRecord implementation carries metadata through NewSlice, SetColumn, and IPC round-trips. Includes PyArrow-generated test fixtures for interoperability validation.
### Are these changes tested?
Yes tests are included.
### Are there any user-facing changes?
Yes custom_metadata is now supported on RecordBatches.
This change was developed with AI assistance and manually reviewed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(arrow/ipc): support custom_metadata on RecordBatch messages [arrow-go]
Posted by "rustyconover (via GitHub)" <gi...@apache.org>.
rustyconover commented on code in PR #669:
URL: https://github.com/apache/arrow-go/pull/669#discussion_r2834143786
##########
arrow/array/record.go:
##########
@@ -121,6 +121,7 @@ type simpleRecord struct {
refCount atomic.Int64
schema *arrow.Schema
+ meta arrow.Metadata
Review Comment:
Hi @zeroshade,
This isn't metadata associated with the schema its metadata on the record batch. Which may be completely different any metadata associated with the schema.
Rusty
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(arrow/ipc): support custom_metadata on RecordBatch messages [arrow-go]
Posted by "rustyconover (via GitHub)" <gi...@apache.org>.
rustyconover commented on code in PR #669:
URL: https://github.com/apache/arrow-go/pull/669#discussion_r2834251516
##########
arrow/ipc/testdata/custom_metadata_stream.arrows:
##########
Review Comment:
@zeroshade I've checked `arrow-testing` and there are no files that contain `custom_metadata` on record batches. I'll email the dev list since issues are not enabled for `arrow-testing`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(arrow/ipc): support custom_metadata on RecordBatch messages [arrow-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #669:
URL: https://github.com/apache/arrow-go/pull/669#discussion_r2833990261
##########
arrow/ipc/testdata/custom_metadata_stream.arrows:
##########
Review Comment:
Do we not have any tests in arrow-testing which use the custom metadata features?
##########
arrow/array/record.go:
##########
@@ -121,6 +121,7 @@ type simpleRecord struct {
refCount atomic.Int64
schema *arrow.Schema
+ meta arrow.Metadata
Review Comment:
is there any reason to not incorporate this extra metadata into the schema itself?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(arrow/ipc): support custom_metadata on RecordBatch messages [arrow-go]
Posted by "rustyconover (via GitHub)" <gi...@apache.org>.
rustyconover commented on code in PR #669:
URL: https://github.com/apache/arrow-go/pull/669#discussion_r2834145846
##########
arrow/ipc/testdata/custom_metadata_stream.arrows:
##########
Review Comment:
I was unaware of arrow-testing I will check it out now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(arrow/ipc): support custom_metadata on RecordBatch messages [arrow-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade merged PR #669:
URL: https://github.com/apache/arrow-go/pull/669
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org