You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@arrow.apache.org by "chaubold (via GitHub)" <gi...@apache.org> on 2025/04/24 12:15:45 UTC

[I] [Python] Allow to write footer into IPC file with pyarrow [arrow]

chaubold opened a new issue, #46222:
URL: https://github.com/apache/arrow/issues/46222

   ### Describe the enhancement requested
   
   In the Java and C/C++ API (at least, I didn't check others) there is a way to write to the footer in an Apache Arrow IPC file. In Python, this footer cannot be populated. We use the footer for some metadata about the stored batches in our Java implementation, but cannot fill that field from Python, preventing us from offering full interoperability (if the custom footer info is needed).
   
   Could this feature be added?
   
   **Suggestion:** Probably this could just be adding a key-value metadata parameter to the RecordBatchFileWriter, because the C API would accept this as parameter in the `MakeFileWriter` call [here](https://github.com/apache/arrow/blob/main/python/pyarrow/ipc.pxi#L1031).
   
   ### Component(s)
   
   Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Python] Allow to specify footer metadata when opening IPC file for writing [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.

rok closed issue #46222: [Python] Allow to specify footer metadata when opening IPC file for writing
URL: https://github.com/apache/arrow/issues/46222


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Python] Allow to write footer into IPC file with pyarrow [arrow]

Posted by "raulcd (via GitHub)" <gi...@apache.org>.

raulcd commented on issue #46222:
URL: https://github.com/apache/arrow/issues/46222#issuecomment-2827534667

   Hi,
   Just to be sure, when you say there is a way on C/C++ you mean using `arrow::ipc::internal::WriteFileFooter` ?
   https://github.com/apache/arrow/blob/520ae44272d491bbb52eb3c9b84864ed7088f11a/cpp/src/arrow/ipc/metadata_internal.cc#L1403-L1406
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Python] Allow to write footer into IPC file with pyarrow [arrow]

Posted by "chaubold (via GitHub)" <gi...@apache.org>.

chaubold commented on issue #46222:
URL: https://github.com/apache/arrow/issues/46222#issuecomment-2838865363

   Hi @raulcd,
   yes, I think that's the one I'm referring to -- I am not too familiar with the C/C++ code base. [This](https://github.com/knime/knime-core-columnar/blob/master/org.knime.core.columnar.arrow/src/main/java/org/knime/core/columnar/arrow/offheap/OffHeapArrowBatchWriter.java#L515-L523) is what we call on the Java side, and we'd like to be able to do the same in Python. Does that clarify what I mean?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Python] Allow to specify footer metadata when opening IPC file for writing [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.

rok commented on issue #46222:
URL: https://github.com/apache/arrow/issues/46222#issuecomment-2871714218

   Issue resolved by pull request 46354
   https://github.com/apache/arrow/pull/46354


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org