[jira] [Commented] (ARROW-16430) [Python] Read/Write record batch custom metadata API in pyarrow

["Steve M. Kim (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-16430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17611673#comment-17611673 ] 

Steve M. Kim commented on ARROW-16430:
--------------------------------------

ARROW-2022 introduced a related problem with {{Schema}} messages. I am not sure whether the related problem ought to be tracked in this issue or in a separate issue, or discussed further on the mailing list.

The current [documentation|https://arrow.apache.org/docs/format/Columnar.html#custom-application-metadata] for the IPC format says
{quote}We provide a {{custom_metadata}} field at three levels to provide a mechanism for developers to pass application-specific metadata in Arrow protocol messages. This includes {{{}Field{}}}, {{{}Schema{}}}, and {{{}Message{}}}.
{quote}
Consistent with the documentation, the FlatBuffer definitions have two different {{custom_metadata}} fields that appear in an encapsulated message of type Schema:
* The {{custom_metadata}} field within the {{Schema}} table
* The {{custom_metadata}} field within the parent {{Message}} table

 

Currently, the pyarrow implementation recognizes only the custom_metadata field in the Schema table and is unaware of the custom_metadata field in the parent Message table. The proposed change [https://github.com/apache/arrow/pull/13041] will use the custom_metadata field in the parent Message table for RecordBatch messages, but it won't address this ambiguity with Schema messages.

I think that it is useful for {{pyarrow.Schema}} object and {{pyarrow.RecordBatch}} object to carry custom metadata, independent of their IPC message serialization. I also think that perhaps {{pyarrow.Table}} ought to carry its own custom metadata that is separate from the metadata of its {{{}Schema{}}}, because a {{Table}} is like a {{{}RecordBatch{}}}. In the current implementation, attempting to instantiate a {{Table}} with both a metadata-enriched {{Schema}} and a separate metadata dict raises {{{}ValueError: Cannot pass both schema and metadata{}}}. This behavior is inconsistent with the documentation, which distinguishes the metadata of a Schema from the metadata of a Message.

> [Python] Read/Write record batch custom metadata API in pyarrow
> ---------------------------------------------------------------
>
>                 Key: ARROW-16430
>                 URL: https://issues.apache.org/jira/browse/ARROW-16430
>             Project: Apache Arrow
>          Issue Type: Improvement
>          Components: Python
>    Affects Versions: 7.0.0
>            Reporter: Yue Ni
>            Assignee: Yue Ni
>            Priority: Major
>              Labels: pull-request-available
>          Time Spent: 7h 20m
>  Remaining Estimate: 0h
>
> In https://issues.apache.org/jira/browse/ARROW-16131, Arrow C++ APIs were added so that users can read/write record batch custom metadata for IPC file. But pyarrow still lacks corresponding APIs for doing this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)