You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "chaubold (via GitHub)" <gi...@apache.org> on 2025/05/07 16:12:09 UTC
[PR] GH-46222: [Python] Allow to specify footer when opening IPC file for … [arrow]
chaubold opened a new pull request, #46354:
URL: https://github.com/apache/arrow/pull/46354
### Rationale for this change
The Arrow APIs for other languages offer to set the footer metadata of an IPC file, but `pyarrow` was lacking this. I opened #46222 because we need this feature at KNIME, and took a shot at suggesting an implementation. Happy to get your feedback on the change!
### What changes are included in this PR?
A new keyword argument `metadata` in `RecordBatchFileWriter.__init__` as well as in `ipc.new_file`. The value is `None` by default to not break backwards compatibility.
Also added a `metadata` property to the `RecordBatchFileReader` to be able to extract the metadata easily.
### Are these changes tested?
Yes, by a unit test.
### Are there any user-facing changes?
Yes, see above :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46222: [Python] Allow to specify footer metadata when opening IPC file for writing [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #46354:
URL: https://github.com/apache/arrow/pull/46354#issuecomment-2873211060
After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 41f1de092c642420599daba1cb7a444c35bd63ad.
There were no benchmark performance regressions. 🎉
The [full Conbench report](https://github.com/apache/arrow/runs/42073114401) has more details. It also includes information about 6 possible false positives for unstable benchmarks that are known to sometimes produce them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46222: [Python] Allow to specify footer metadata when opening IPC file for writing [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok merged PR #46354:
URL: https://github.com/apache/arrow/pull/46354
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46222: [Python] Allow to specify footer when opening IPC file for … [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #46354:
URL: https://github.com/apache/arrow/pull/46354#issuecomment-2859187375
:warning: GitHub issue #46222 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46222: [Python] Allow to specify footer when opening IPC file for writing [arrow]
Posted by "chaubold (via GitHub)" <gi...@apache.org>.
chaubold commented on PR #46354:
URL: https://github.com/apache/arrow/pull/46354#issuecomment-2866265450
Sorry for all the iterations here. But now the docstring complains are resolved and only the `gdb` issue remains.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46222: [Python] Allow to specify footer when opening IPC file for writing [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #46354:
URL: https://github.com/apache/arrow/pull/46354#issuecomment-2871359055
Revision: 97f8a5f56c2b1e194006d7a70e06c63232570825
Submitted crossbow builds: [ursacomputing/crossbow @ actions-3a520b18d3](https://github.com/ursacomputing/crossbow/branches/all?query=actions-3a520b18d3)
|Task|Status|
|----|------|
|example-python-minimal-build-fedora-conda|[](https://github.com/ursacomputing/crossbow/actions/runs/14967207118/job/42039744716)|
|example-python-minimal-build-ubuntu-venv|[](https://github.com/ursacomputing/crossbow/actions/runs/14967208142/job/42039748825)|
|test-conda-python-3.10|[](https://github.com/ursacomputing/crossbow/actions/runs/14967206830/job/42039743183)|
|test-conda-python-3.10-hdfs-2.9.2|[](https://github.com/ursacomputing/crossbow/actions/runs/14967207648/job/42039747579)|
|test-conda-python-3.10-hdfs-3.2.1|[](https://github.com/ursacomputing/crossbow/actions/runs/14967206658/job/42039742966)|
|test-conda-python-3.10-pandas-latest-numpy-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/14967207717/job/42039747921)|
|test-conda-python-3.11|[](https://github.com/ursacomputing/crossbow/actions/runs/14967207162/job/42039744702)|
|test-conda-python-3.11-dask-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/14967207411/job/42039746345)|
|test-conda-python-3.11-dask-upstream_devel|[](https://github.com/ursacomputing/crossbow/actions/runs/14967207422/job/42039746325)|
|test-conda-python-3.11-hypothesis|[](https://github.com/ursacomputing/crossbow/actions/runs/14967206684/job/42039742913)|
|test-conda-python-3.11-pandas-latest-numpy-1.26|[](https://github.com/ursacomputing/crossbow/actions/runs/14967207290/job/42039744175)|
|test-conda-python-3.11-pandas-latest-numpy-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/14967207849/job/42039748818)|
|test-conda-python-3.11-pandas-nightly-numpy-nightly|[](https://github.com/ursacomputing/crossbow/actions/runs/14967207527/job/42039748052)|
|test-conda-python-3.11-pandas-upstream_devel-numpy-nightly|[](https://github.com/ursacomputing/crossbow/actions/runs/14967207709/job/42039747743)|
|test-conda-python-3.11-spark-master|[](https://github.com/ursacomputing/crossbow/actions/runs/14967207319/job/42039744745)|
|test-conda-python-3.12|[](https://github.com/ursacomputing/crossbow/actions/runs/14967207871/job/42039748810)|
|test-conda-python-3.12-cpython-debug|[](https://github.com/ursacomputing/crossbow/actions/runs/14967207338/job/42039745193)|
|test-conda-python-3.13|[](https://github.com/ursacomputing/crossbow/actions/runs/14967207303/job/42039745100)|
|test-conda-python-3.9|[](https://github.com/ursacomputing/crossbow/actions/runs/14967207701/job/42039747617)|
|test-conda-python-3.9-pandas-1.1.3-numpy-1.19.5|[](https://github.com/ursacomputing/crossbow/actions/runs/14967206842/job/42039743661)|
|test-conda-python-emscripten|[](https://github.com/ursacomputing/crossbow/actions/runs/14967207455/job/42039746334)|
|test-cuda-python-ubuntu-22.04-cuda-11.7.1|[](https://github.com/ursacomputing/crossbow/actions/runs/14967207334/job/42039745159)|
|test-debian-12-python-3-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/14967207384/job/42039746290)|
|test-debian-12-python-3-i386|[](https://github.com/ursacomputing/crossbow/actions/runs/14967207207/job/42039744816)|
|test-fedora-39-python-3|[](https://github.com/ursacomputing/crossbow/actions/runs/14967206663/job/42039742936)|
|test-ubuntu-22.04-python-3|[](https://github.com/ursacomputing/crossbow/actions/runs/14967207694/job/42039747574)|
|test-ubuntu-22.04-python-313-freethreading|[](https://github.com/ursacomputing/crossbow/actions/runs/14967207356/job/42039745245)|
|test-ubuntu-24.04-python-3|[](https://github.com/ursacomputing/crossbow/actions/runs/14967207802/job/42039748827)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46222: [Python] Allow to specify footer when opening IPC file for writing [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on PR #46354:
URL: https://github.com/apache/arrow/pull/46354#issuecomment-2871351439
@github-actions crossbow submit -g python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46222: [Python] Allow to specify footer when opening IPC file for writing [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #46354:
URL: https://github.com/apache/arrow/pull/46354#issuecomment-2859238444
Hey @chaubold ! Thanks for doing this, PR looks pretty straightforward, I can take a look at it tomorrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46222: [Python] Allow to specify footer when opening IPC file for writing [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on PR #46354:
URL: https://github.com/apache/arrow/pull/46354#issuecomment-2871624843
> @raulcd CI looks good, shall I merge?
sure!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46222: [Python] Allow to specify footer when opening IPC file for writing [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on code in PR #46354:
URL: https://github.com/apache/arrow/pull/46354#discussion_r2081664745
##########
python/pyarrow/ipc.pxi:
##########
@@ -1192,6 +1198,13 @@ cdef class _RecordBatchFileReader(_Weakrefable):
raise ValueError("Operation on closed reader")
return _wrap_read_stats(self.reader.get().stats())
+ @property
+ def metadata(self):
+ """
+ File-level custom KeyValueMetadata written via ``ipc.new_file(..., metadata=)``.
+ """
+ return pyarrow_wrap_metadata(self.reader.get().metadata())
Review Comment:
In other metadata properties, i.e: https://github.com/apache/arrow/blob/88815a1f6ffcfbcecdfb7c0863795123ebf1a733/python/pyarrow/types.pxi#L2591-L2595
We return either a dict or None, I think we should do the same to follow consistency on our APIs.
```suggestion
wrapped = pyarrow_wrap_metadata(self.reader.get().metadata())
if wrapped is not None:
return wrapped.to_dict()
else:
return wrapped
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46222: [Python] Allow to specify footer when opening IPC file for writing [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on code in PR #46354:
URL: https://github.com/apache/arrow/pull/46354#discussion_r2081846780
##########
python/pyarrow/tests/test_ipc.py:
##########
@@ -1304,3 +1304,29 @@ def test_record_batch_reader_cast_nulls():
casted_reader = reader.cast(schema_dst)
with pytest.raises(pa.lib.ArrowInvalid, match="Can't cast array"):
casted_reader.read_all()
+
+
+def test_record_batch_file_writer_with_metadata():
+ # https://github.com/apache/arrow/issues/46222
+ tbl = pa.table({"a": [1, 2, 3]})
+ meta = {b"creator": b"test", b"version": b"0.1.0"}
+ sink = pa.BufferOutputStream()
+
+ with pa.ipc.new_file(sink, tbl.schema, metadata=meta) as w:
+ w.write_table(tbl)
+
+ buffer = sink.getvalue()
+ with pa.ipc.open_file(buffer) as r:
+ assert r.metadata == meta
+
Review Comment:
Could you add two empty lines to fix the linter failure?
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46222: [Python] Allow to specify footer when opening IPC file for writing [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #46354:
URL: https://github.com/apache/arrow/pull/46354#issuecomment-2866311204
Thanks for iterating @chaubold ! I think this is done, I'd just like to @AlenkaF or @raulcd to do another pass before merging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46222: [Python] Allow to specify footer when opening IPC file for writing [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #46354:
URL: https://github.com/apache/arrow/pull/46354#discussion_r2079577180
##########
python/pyarrow/ipc.py:
##########
@@ -109,11 +109,16 @@ class RecordBatchFileWriter(lib._RecordBatchFileWriter):
__doc__ = """Writer to create the Arrow binary file format
-{}""".format(_ipc_writer_class_doc)
+{}
+metadata : dict | pyarrow.KeyValueMetadata, optional
+ Key/value pairs (both **must** be bytes-like) that will be stored
+ in the file footer and are retrievable with
+ ``ipc.open_file(...).metadata``.
+""".format(_ipc_writer_class_doc)
Review Comment:
Docstring produced now doesn't seem to [sit well with the linter](https://github.com/apache/arrow/actions/runs/14888211833/job/41866015539?pr=46354#step:6:8647):
```python
INFO:archery:Running Python docstring linters
/opt/conda/envs/arrow/lib/python3.10/site-packages/numpydoc/docscrape.py:418: UserWarning: Unknown section Example
warn(msg)
pyarrow.ipc.new_file
-> pyarrow.ipc.Create an Arrow columnar IPC file writer instance
PR01: Parameters {'metadata'} not documented
Total number of docstring violations: 1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46222: [Python] Allow to specify footer when opening IPC file for writing [arrow]
Posted by "chaubold (via GitHub)" <gi...@apache.org>.
chaubold commented on code in PR #46354:
URL: https://github.com/apache/arrow/pull/46354#discussion_r2081730011
##########
python/pyarrow/ipc.pxi:
##########
@@ -1192,6 +1198,13 @@ cdef class _RecordBatchFileReader(_Weakrefable):
raise ValueError("Operation on closed reader")
return _wrap_read_stats(self.reader.get().stats())
+ @property
+ def metadata(self):
+ """
+ File-level custom KeyValueMetadata written via ``ipc.new_file(..., metadata=)``.
+ """
+ return pyarrow_wrap_metadata(self.reader.get().metadata())
Review Comment:
Thanks for the feedback @raulcd! I also adjusted the docstring and the test to reflect the change (and added a unit test to check that `pyarrow.ipc.open_file(...).metadata is None` in case nothing was set.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46222: [Python] Allow to specify footer when opening IPC file for writing [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #46354:
URL: https://github.com/apache/arrow/pull/46354#issuecomment-2871560750
@raulcd CI looks good, shall I merge?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org