You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "SoundBot (via GitHub)" <gi...@apache.org> on 2025/07/12 04:08:56 UTC
[PR] [Python][Parquet] Expose ParquetWriter properties and arrow_properties [arrow]
SoundBot opened a new pull request, #47087:
URL: https://github.com/apache/arrow/pull/47087
### Rationale for this change
Exposes `ParquetWriter` properties via `writer.properties` and `writer.arrow_properties`
(see https://github.com/apache/arrow/issues/45902)
### What changes are included in this PR?
See above
### Are these changes tested?
Yes
### Are there any user-facing changes?
Yes, properties are available via `writer.properties` and `writer.arrow_properties`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45902: [Python][Parquet] Expose ParquetWriter properties and arrow_properties [arrow]
Posted by "Reranko05 (via GitHub)" <gi...@apache.org>.
Reranko05 commented on PR #47087:
URL: https://github.com/apache/arrow/pull/47087#issuecomment-6529096974
@rok Should we consider closing this PR and unassign the author from the issue so that anyone interested can pick it up? He hasn't had any GitHub activity since August 2025, so it seems unlikely that the work will be continued.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45902: [Python][Parquet] Expose ParquetWriter properties and arrow_properties [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on PR #47087:
URL: https://github.com/apache/arrow/pull/47087#issuecomment-3097289620
@github-actions crossbow submit -g python -g wheel
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45902: [Python][Parquet] Expose ParquetWriter properties and arrow_properties [arrow]
Posted by "SoundBot (via GitHub)" <gi...@apache.org>.
SoundBot commented on PR #47087:
URL: https://github.com/apache/arrow/pull/47087#issuecomment-3212685793
@rok @wgtmac can any of you folks review this PR please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45902: [Python][Parquet] Expose ParquetWriter properties and arrow_properties [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47087:
URL: https://github.com/apache/arrow/pull/47087#discussion_r2314251651
##########
python/pyarrow/_parquet.pyx:
##########
@@ -2408,3 +2411,77 @@ cdef class ParquetWriter(_Weakrefable):
return result
raise RuntimeError(
'file metadata is only available after writer close')
+
+ @property
+ def properties(self):
+ cdef const WriterProperties* props_ptr
+ cdef WriterPropertiesWrapper result
+
+ props_ptr = &self.writer.get().properties()
+ result = WriterPropertiesWrapper.__new__(WriterPropertiesWrapper)
+ result.init(props_ptr)
+ return result
+
+ @property
+ def arrow_properties(self):
+ cdef ArrowWriterPropertiesWrapper result
+
+ result = ArrowWriterPropertiesWrapper.__new__(ArrowWriterPropertiesWrapper)
+ result.init(self._arrow_properties)
+ return result
+
+
+cdef class WriterPropertiesWrapper(_Weakrefable):
+ cdef:
+ const WriterProperties* props
+
+ def __cinit__(self):
+ self.props = NULL
+
+ cdef init(self, const WriterProperties* props):
+ self.props = props
+
+ @property
+ def version(self):
Review Comment:
Please add docstrings for all these properties.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45902: [Python][Parquet] Expose ParquetWriter properties and arrow_properties [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #47087:
URL: https://github.com/apache/arrow/pull/47087#issuecomment-5508928187
@SoundBot would you be interested in continuing this work?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45902: [Python][Parquet] Expose ParquetWriter properties and arrow_properties [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47087:
URL: https://github.com/apache/arrow/pull/47087#discussion_r2314253706
##########
python/pyarrow/tests/parquet/test_parquet_writer.py:
##########
@@ -448,3 +448,46 @@ def test_parquet_content_defined_chunking_parameters(tempdir):
# using min_chunk_size, max_chunk_size and norm_level
cdc_options = {"min_chunk_size": 32_768, "max_chunk_size": 65_536, "norm_level": 1}
pq.write_table(table, path, use_content_defined_chunking=cdc_options)
+
+
+@pytest.mark.pandas
+def test_parquet_writer_properties_access():
+ df = _test_dataframe(100)
+ table = pa.Table.from_pandas(df, preserve_index=False)
+
+ out = pa.BufferOutputStream()
+
+ with pq.ParquetWriter(
+ out, table.schema,
+ compression='snappy',
+ version='2.6',
+ use_dictionary=True,
+ write_statistics=True,
+ data_page_size=64 * 1024,
+ use_deprecated_int96_timestamps=False,
+ store_schema=True,
+ use_compliant_nested_type=True,
+ writer_engine_version='V2'
+ ) as writer:
+
+ props = writer.properties
+ assert props is not None
+ assert hasattr(props, 'version')
+ assert hasattr(props, 'created_by')
+ assert hasattr(props, 'dictionary_enabled')
+ assert hasattr(props, 'statistics_enabled')
+ assert props.version == 2
Review Comment:
Not sure what this 2 means... we passed "2.6" above and probably expect to get it back?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python][Parquet] Expose ParquetWriter properties and arrow_properties [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #47087:
URL: https://github.com/apache/arrow/pull/47087#issuecomment-3064634656
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
Thanks for opening a pull request!
If this is not a [minor PR](https://github.com/apache/arrow/blob/main/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
or
MINOR: [${COMPONENT}] ${SUMMARY}
See also:
* [Other pull requests](https://github.com/apache/arrow/pulls/)
* [Contribution Guidelines - Contributing Overview](https://arrow.apache.org/docs/developers/overview.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45902: [Python][Parquet] Expose ParquetWriter properties and arrow_properties [arrow]
Posted by "SoundBot (via GitHub)" <gi...@apache.org>.
SoundBot commented on code in PR #47087:
URL: https://github.com/apache/arrow/pull/47087#discussion_r2296996040
##########
python/pyarrow/_parquet.pyx:
##########
@@ -2408,3 +2411,77 @@ cdef class ParquetWriter(_Weakrefable):
return result
raise RuntimeError(
'file metadata is only available after writer close')
+
+ @property
+ def properties(self):
+ cdef const WriterProperties* props_ptr
+ cdef WriterPropertiesWrapper result
+
+ props_ptr = &self.writer.get().properties()
+ result = WriterPropertiesWrapper.__new__(WriterPropertiesWrapper)
+ result.init(props_ptr)
+ return result
+
+ @property
+ def arrow_properties(self):
+ cdef ArrowWriterPropertiesWrapper result
+
+ result = ArrowWriterPropertiesWrapper.__new__(ArrowWriterPropertiesWrapper)
+ result.init(self._arrow_properties)
+ return result
+
+
+cdef class WriterPropertiesWrapper(_Weakrefable):
Review Comment:
@wgtmac I'm not aware is there is a better way. I think generally you expose what the end-users might want to use.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45902: [Python][Parquet] Expose ParquetWriter properties and arrow_properties [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #47087:
URL: https://github.com/apache/arrow/pull/47087#discussion_r2266888042
##########
python/pyarrow/tests/parquet/test_parquet_writer.py:
##########
@@ -448,3 +448,46 @@ def test_parquet_content_defined_chunking_parameters(tempdir):
# using min_chunk_size, max_chunk_size and norm_level
cdc_options = {"min_chunk_size": 32_768, "max_chunk_size": 65_536, "norm_level": 1}
pq.write_table(table, path, use_content_defined_chunking=cdc_options)
+
+
+@pytest.mark.pandas
+def test_parquet_writer_properties_access():
+ df = _test_dataframe(100)
+ table = pa.Table.from_pandas(df, preserve_index=False)
Review Comment:
The PR that adds a utility function to create an Arrow table directly (without going through pandas) has been merged. It would be great to use that here now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45902: [Python][Parquet] Expose ParquetWriter properties and arrow_properties [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on code in PR #47087:
URL: https://github.com/apache/arrow/pull/47087#discussion_r2219521416
##########
python/pyarrow/tests/parquet/test_parquet_writer.py:
##########
@@ -448,3 +448,46 @@ def test_parquet_content_defined_chunking_parameters(tempdir):
# using min_chunk_size, max_chunk_size and norm_level
cdc_options = {"min_chunk_size": 32_768, "max_chunk_size": 65_536, "norm_level": 1}
pq.write_table(table, path, use_content_defined_chunking=cdc_options)
+
+
+@pytest.mark.pandas
+def test_parquet_writer_properties_access():
+ df = _test_dataframe(100)
+ table = pa.Table.from_pandas(df, preserve_index=False)
Review Comment:
not specifically related to this PR but would be nice to have an utility function to create an Arrow table directly instead of having to go through pandas. This test doesn't really have to require pandas but the utility function is handy. We can create an issue to move some of those tests to use a new utility function and remove the `@pytest.mark.pandas` for some of those tests. @AlenkaF what do you think? It could be a `good-first-issue`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45902: [Python][Parquet] Expose ParquetWriter properties and arrow_properties [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47087:
URL: https://github.com/apache/arrow/pull/47087#discussion_r2314220120
##########
python/pyarrow/_parquet.pyx:
##########
@@ -2408,3 +2411,77 @@ cdef class ParquetWriter(_Weakrefable):
return result
raise RuntimeError(
'file metadata is only available after writer close')
+
+ @property
+ def properties(self):
+ cdef const WriterProperties* props_ptr
+ cdef WriterPropertiesWrapper result
+
+ props_ptr = &self.writer.get().properties()
+ result = WriterPropertiesWrapper.__new__(WriterPropertiesWrapper)
+ result.init(props_ptr)
+ return result
+
+ @property
+ def arrow_properties(self):
+ cdef ArrowWriterPropertiesWrapper result
+
+ result = ArrowWriterPropertiesWrapper.__new__(ArrowWriterPropertiesWrapper)
+ result.init(self._arrow_properties)
+ return result
+
+
+cdef class WriterPropertiesWrapper(_Weakrefable):
+ cdef:
+ const WriterProperties* props
+
+ def __cinit__(self):
+ self.props = NULL
+
+ cdef init(self, const WriterProperties* props):
+ self.props = props
+
+ @property
+ def version(self):
+ return self.props.version()
+
+ @property
+ def created_by(self):
+ return frombytes(self.props.created_by())
+
+ @property
+ def dictionary_enabled(self):
+ return self.props.default_column_properties().dictionary_enabled()
+
+ @property
+ def statistics_enabled(self):
+ return self.props.default_column_properties().statistics_enabled()
+
+
+cdef class ArrowWriterPropertiesWrapper(_Weakrefable):
+ cdef:
+ shared_ptr[ArrowWriterProperties] props
+
+ def __cinit__(self):
+ self.props.reset()
+
+ cdef init(self, shared_ptr[ArrowWriterProperties] props):
+ self.props = props
+
+ @property
+ def support_deprecated_int96_timestamps(self):
+ return self.props.get().support_deprecated_int96_timestamps()
+
+ @property
+ def store_schema(self):
+ return self.props.get().store_schema()
+
+ @property
+ def engine_version(self):
+ cdef int version_int = self.props.get().engine_version()
+ if version_int == 0: # V1
+ return "V1"
+ elif version_int == 1: # V2
+ return "V2"
Review Comment:
Why not return integers 1 or 2, respectively?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45902: [Python][Parquet] Expose ParquetWriter properties and arrow_properties [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47087:
URL: https://github.com/apache/arrow/pull/47087#discussion_r2314217533
##########
python/pyarrow/_parquet.pyx:
##########
@@ -2408,3 +2411,77 @@ cdef class ParquetWriter(_Weakrefable):
return result
raise RuntimeError(
'file metadata is only available after writer close')
+
+ @property
+ def properties(self):
+ cdef const WriterProperties* props_ptr
+ cdef WriterPropertiesWrapper result
+
+ props_ptr = &self.writer.get().properties()
+ result = WriterPropertiesWrapper.__new__(WriterPropertiesWrapper)
+ result.init(props_ptr)
+ return result
+
+ @property
+ def arrow_properties(self):
+ cdef ArrowWriterPropertiesWrapper result
+
+ result = ArrowWriterPropertiesWrapper.__new__(ArrowWriterPropertiesWrapper)
+ result.init(self._arrow_properties)
+ return result
+
+
+cdef class WriterPropertiesWrapper(_Weakrefable):
Review Comment:
Why call it "Wrapper"? We should just call it `WriterProperties`. If that produces a name clash with the C++ class, then we should rename the latter in the Cython declarations to `CWriterProperties` as we do for many other such cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45902: [Python][Parquet] Expose ParquetWriter properties and arrow_properties [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #47087:
URL: https://github.com/apache/arrow/pull/47087#issuecomment-3067771325
:warning: GitHub issue #45902 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45902: [Python][Parquet] Expose ParquetWriter properties and arrow_properties [arrow]
Posted by "SoundBot (via GitHub)" <gi...@apache.org>.
SoundBot commented on PR #47087:
URL: https://github.com/apache/arrow/pull/47087#issuecomment-3099713213
Is there anything else needed to merge this PR? @AlenkaF @raulcd
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45902: [Python][Parquet] Expose ParquetWriter properties and arrow_properties [arrow]
Posted by "SoundBot (via GitHub)" <gi...@apache.org>.
SoundBot commented on PR #47087:
URL: https://github.com/apache/arrow/pull/47087#issuecomment-3071487441
@AlenkaF thank you, I added `@pytest.mark.pandas` and ran `pre-commit run --all-files`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45902: [Python][Parquet] Expose ParquetWriter properties and arrow_properties [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #47087:
URL: https://github.com/apache/arrow/pull/47087#discussion_r2292529044
##########
python/pyarrow/_parquet.pyx:
##########
@@ -2408,3 +2411,77 @@ cdef class ParquetWriter(_Weakrefable):
return result
raise RuntimeError(
'file metadata is only available after writer close')
+
+ @property
+ def properties(self):
+ cdef const WriterProperties* props_ptr
+ cdef WriterPropertiesWrapper result
+
+ props_ptr = &self.writer.get().properties()
+ result = WriterPropertiesWrapper.__new__(WriterPropertiesWrapper)
+ result.init(props_ptr)
+ return result
+
+ @property
+ def arrow_properties(self):
+ cdef ArrowWriterPropertiesWrapper result
+
+ result = ArrowWriterPropertiesWrapper.__new__(ArrowWriterPropertiesWrapper)
+ result.init(self._arrow_properties)
+ return result
+
+
+cdef class WriterPropertiesWrapper(_Weakrefable):
Review Comment:
I'm not familiar with CPython. Does it mean that we only support a subset of attributes from these properties?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45902: [Python][Parquet] Expose ParquetWriter properties and arrow_properties [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #47087:
URL: https://github.com/apache/arrow/pull/47087#discussion_r2295448143
##########
python/pyarrow/_parquet.pyx:
##########
@@ -2408,3 +2411,77 @@ cdef class ParquetWriter(_Weakrefable):
return result
raise RuntimeError(
'file metadata is only available after writer close')
+
+ @property
+ def properties(self):
+ cdef const WriterProperties* props_ptr
+ cdef WriterPropertiesWrapper result
+
+ props_ptr = &self.writer.get().properties()
+ result = WriterPropertiesWrapper.__new__(WriterPropertiesWrapper)
+ result.init(props_ptr)
+ return result
+
+ @property
+ def arrow_properties(self):
+ cdef ArrowWriterPropertiesWrapper result
+
+ result = ArrowWriterPropertiesWrapper.__new__(ArrowWriterPropertiesWrapper)
+ result.init(self._arrow_properties)
+ return result
+
+
+cdef class WriterPropertiesWrapper(_Weakrefable):
Review Comment:
Why is that? Is it a convention on the pyarrow side? Isn't it annoying to add a missing property in the future? cc @rok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45902: [Python][Parquet] Expose ParquetWriter properties and arrow_properties [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #47087:
URL: https://github.com/apache/arrow/pull/47087#issuecomment-3097301673
Revision: 3adfc51f651604fadc664b9ead2d681a31d8552f
Submitted crossbow builds: [ursacomputing/crossbow @ actions-623832f922](https://github.com/ursacomputing/crossbow/branches/all?query=actions-623832f922)
|Task|Status|
|----|------|
|example-python-minimal-build-fedora-conda|[](https://github.com/ursacomputing/crossbow/actions/runs/16421551011/job/46401157565)|
|example-python-minimal-build-ubuntu-venv|[](https://github.com/ursacomputing/crossbow/actions/runs/16421551059/job/46401157584)|
|python-sdist|[](https://github.com/ursacomputing/crossbow/actions/runs/16421551138/job/46401158061)|
|test-conda-python-3.10|[](https://github.com/ursacomputing/crossbow/actions/runs/16421549246/job/46401151752)|
|test-conda-python-3.10-hdfs-2.9.2|[](https://github.com/ursacomputing/crossbow/actions/runs/16421551415/job/46401158630)|
|test-conda-python-3.10-hdfs-3.2.1|[](https://github.com/ursacomputing/crossbow/actions/runs/16421550033/job/46401154306)|
|test-conda-python-3.10-pandas-latest-numpy-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/16421549781/job/46401153545)|
|test-conda-python-3.11|[](https://github.com/ursacomputing/crossbow/actions/runs/16421549937/job/46401154002)|
|test-conda-python-3.11-dask-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/16421548769/job/46401150527)|
|test-conda-python-3.11-dask-upstream_devel|[](https://github.com/ursacomputing/crossbow/actions/runs/16421550586/job/46401156383)|
|test-conda-python-3.11-hypothesis|[](https://github.com/ursacomputing/crossbow/actions/runs/16421548938/job/46401150967)|
|test-conda-python-3.11-pandas-latest-numpy-1.26|[](https://github.com/ursacomputing/crossbow/actions/runs/16421549965/job/46401154077)|
|test-conda-python-3.11-pandas-latest-numpy-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/16421550971/job/46401157363)|
|test-conda-python-3.11-pandas-nightly-numpy-nightly|[](https://github.com/ursacomputing/crossbow/actions/runs/16421549262/job/46401151916)|
|test-conda-python-3.11-pandas-upstream_devel-numpy-nightly|[](https://github.com/ursacomputing/crossbow/actions/runs/16421550013/job/46401154775)|
|test-conda-python-3.11-spark-master|[](https://github.com/ursacomputing/crossbow/actions/runs/16421551201/job/46401157917)|
|test-conda-python-3.12|[](https://github.com/ursacomputing/crossbow/actions/runs/16421548793/job/46401150536)|
|test-conda-python-3.12-cpython-debug|[](https://github.com/ursacomputing/crossbow/actions/runs/16421550222/job/46401155084)|
|test-conda-python-3.13|[](https://github.com/ursacomputing/crossbow/actions/runs/16421549434/job/46401152544)|
|test-conda-python-3.9|[](https://github.com/ursacomputing/crossbow/actions/runs/16421551249/job/46401158326)|
|test-conda-python-3.9-pandas-1.1.3-numpy-1.19.5|[](https://github.com/ursacomputing/crossbow/actions/runs/16421551282/job/46401158259)|
|test-conda-python-emscripten|[](https://github.com/ursacomputing/crossbow/actions/runs/16421549248/job/46401151887)|
|test-cuda-python-ubuntu-22.04-cuda-11.7.1|[](https://github.com/ursacomputing/crossbow/actions/runs/16421551271/job/46401158402)|
|test-debian-12-python-3-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/16421549172/job/46401151563)|
|test-debian-12-python-3-i386|[](https://github.com/ursacomputing/crossbow/actions/runs/16421549895/job/46401153968)|
|test-fedora-39-python-3|[](https://github.com/ursacomputing/crossbow/actions/runs/16421551154/job/46401157839)|
|test-ubuntu-22.04-python-3|[](https://github.com/ursacomputing/crossbow/actions/runs/16421549546/job/46401152809)|
|test-ubuntu-22.04-python-313-freethreading|[](https://github.com/ursacomputing/crossbow/actions/runs/16421549316/job/46401152228)|
|test-ubuntu-24.04-python-3|[](https://github.com/ursacomputing/crossbow/actions/runs/16421548872/job/46401150799)|
|wheel-macos-monterey-cp310-cp310-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/16421550104/job/46401154579)|
|wheel-macos-monterey-cp310-cp310-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/16421551575/job/46401159157)|
|wheel-macos-monterey-cp311-cp311-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/16421549663/job/46401153191)|
|wheel-macos-monterey-cp311-cp311-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/16421551384/job/46401158556)|
|wheel-macos-monterey-cp312-cp312-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/16421550763/job/46401156612)|
|wheel-macos-monterey-cp312-cp312-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/16421550449/job/46401155709)|
|wheel-macos-monterey-cp313-cp313-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/16421549344/job/46401152240)|
|wheel-macos-monterey-cp313-cp313-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/16421549011/job/46401151125)|
|wheel-macos-monterey-cp313-cp313t-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/16421550083/job/46401154745)|
|wheel-macos-monterey-cp313-cp313t-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/16421549802/job/46401153590)|
|wheel-macos-monterey-cp39-cp39-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/16421551215/job/46401157931)|
|wheel-macos-monterey-cp39-cp39-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/16421550153/job/46401154946)|
|wheel-manylinux-2-28-cp310-cp310-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/16421550831/job/46401156962)|
|wheel-manylinux-2-28-cp310-cp310-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/16421548843/job/46401150587)|
|wheel-manylinux-2-28-cp311-cp311-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/16421549990/job/46401154227)|
|wheel-manylinux-2-28-cp311-cp311-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/16421550787/job/46401156726)|
|wheel-manylinux-2-28-cp312-cp312-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/16421549744/job/46401153452)|
|wheel-manylinux-2-28-cp312-cp312-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/16421548886/job/46401150853)|
|wheel-manylinux-2-28-cp313-cp313-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/16421550494/job/46401155977)|
|wheel-manylinux-2-28-cp313-cp313-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/16421550875/job/46401156992)|
|wheel-manylinux-2-28-cp313-cp313t-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/16421550392/job/46401155484)|
|wheel-manylinux-2-28-cp313-cp313t-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/16421549475/job/46401152547)|
|wheel-manylinux-2-28-cp39-cp39-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/16421548935/job/46401151006)|
|wheel-manylinux-2-28-cp39-cp39-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/16421550782/job/46401156671)|
|wheel-musllinux-1-2-cp310-cp310-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/16421550006/job/46401154249)|
|wheel-musllinux-1-2-cp310-cp310-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/16421551400/job/46401158670)|
|wheel-musllinux-1-2-cp311-cp311-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/16421550850/job/46401157128)|
|wheel-musllinux-1-2-cp311-cp311-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/16421550247/job/46401155072)|
|wheel-musllinux-1-2-cp312-cp312-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/16421551243/job/46401158163)|
|wheel-musllinux-1-2-cp312-cp312-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/16421550352/job/46401155517)|
|wheel-musllinux-1-2-cp313-cp313-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/16421550565/job/46401156123)|
|wheel-musllinux-1-2-cp313-cp313-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/16421549718/job/46401153390)|
|wheel-musllinux-1-2-cp313-cp313t-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/16421549483/job/46401152599)|
|wheel-musllinux-1-2-cp313-cp313t-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/16421549482/job/46401152551)|
|wheel-musllinux-1-2-cp39-cp39-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/16421550768/job/46401156590)|
|wheel-musllinux-1-2-cp39-cp39-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/16421550676/job/46401156434)|
|wheel-windows-cp310-cp310-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/16421549150/job/46401151508)|
|wheel-windows-cp311-cp311-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/16421549588/job/46401152956)|
|wheel-windows-cp312-cp312-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/16421551021/job/46401157619)|
|wheel-windows-cp313-cp313-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/16421550435/job/46401155588)|
|wheel-windows-cp313-cp313t-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/16421550122/job/46401155663)|
|wheel-windows-cp39-cp39-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/16421549306/job/46401152093)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45902: [Python][Parquet] Expose ParquetWriter properties and arrow_properties [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47087:
URL: https://github.com/apache/arrow/pull/47087#discussion_r2314213311
##########
python/pyarrow/tests/parquet/test_parquet_writer.py:
##########
@@ -448,3 +448,46 @@ def test_parquet_content_defined_chunking_parameters(tempdir):
# using min_chunk_size, max_chunk_size and norm_level
cdc_options = {"min_chunk_size": 32_768, "max_chunk_size": 65_536, "norm_level": 1}
pq.write_table(table, path, use_content_defined_chunking=cdc_options)
+
+
+@pytest.mark.pandas
+def test_parquet_writer_properties_access():
+ df = _test_dataframe(100)
+ table = pa.Table.from_pandas(df, preserve_index=False)
Review Comment:
That is not even required. Any hand-created Arrow table, however trivial, would probably be sufficient for this test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45902: [Python][Parquet] Expose ParquetWriter properties and arrow_properties [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47087:
URL: https://github.com/apache/arrow/pull/47087#discussion_r2314222537
##########
python/pyarrow/_parquet.pyx:
##########
@@ -2408,3 +2411,77 @@ cdef class ParquetWriter(_Weakrefable):
return result
raise RuntimeError(
'file metadata is only available after writer close')
+
+ @property
+ def properties(self):
+ cdef const WriterProperties* props_ptr
+ cdef WriterPropertiesWrapper result
+
+ props_ptr = &self.writer.get().properties()
+ result = WriterPropertiesWrapper.__new__(WriterPropertiesWrapper)
+ result.init(props_ptr)
+ return result
+
+ @property
+ def arrow_properties(self):
+ cdef ArrowWriterPropertiesWrapper result
+
+ result = ArrowWriterPropertiesWrapper.__new__(ArrowWriterPropertiesWrapper)
+ result.init(self._arrow_properties)
+ return result
+
+
+cdef class WriterPropertiesWrapper(_Weakrefable):
+ cdef:
+ const WriterProperties* props
+
+ def __cinit__(self):
+ self.props = NULL
+
+ cdef init(self, const WriterProperties* props):
+ self.props = props
+
+ @property
+ def version(self):
+ return self.props.version()
+
+ @property
+ def created_by(self):
+ return frombytes(self.props.created_by())
+
+ @property
+ def dictionary_enabled(self):
+ return self.props.default_column_properties().dictionary_enabled()
+
+ @property
+ def statistics_enabled(self):
+ return self.props.default_column_properties().statistics_enabled()
+
+
+cdef class ArrowWriterPropertiesWrapper(_Weakrefable):
+ cdef:
+ shared_ptr[ArrowWriterProperties] props
+
+ def __cinit__(self):
+ self.props.reset()
+
+ cdef init(self, shared_ptr[ArrowWriterProperties] props):
+ self.props = props
Review Comment:
If we're getting a const-ref from the C++ APIs, we should certainly not turn it into a `shared_ptr`. You should probably let the C++ APIs return a `shared_ptr`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45902: [Python][Parquet] Expose ParquetWriter properties and arrow_properties [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47087:
URL: https://github.com/apache/arrow/pull/47087#discussion_r2314251024
##########
python/pyarrow/tests/parquet/test_parquet_writer.py:
##########
@@ -448,3 +448,46 @@ def test_parquet_content_defined_chunking_parameters(tempdir):
# using min_chunk_size, max_chunk_size and norm_level
cdc_options = {"min_chunk_size": 32_768, "max_chunk_size": 65_536, "norm_level": 1}
pq.write_table(table, path, use_content_defined_chunking=cdc_options)
+
+
+@pytest.mark.pandas
+def test_parquet_writer_properties_access():
+ df = _test_dataframe(100)
+ table = pa.Table.from_pandas(df, preserve_index=False)
+
+ out = pa.BufferOutputStream()
+
+ with pq.ParquetWriter(
+ out, table.schema,
+ compression='snappy',
+ version='2.6',
+ use_dictionary=True,
+ write_statistics=True,
+ data_page_size=64 * 1024,
+ use_deprecated_int96_timestamps=False,
+ store_schema=True,
+ use_compliant_nested_type=True,
+ writer_engine_version='V2'
+ ) as writer:
+
+ props = writer.properties
+ assert props is not None
+ assert hasattr(props, 'version')
+ assert hasattr(props, 'created_by')
+ assert hasattr(props, 'dictionary_enabled')
+ assert hasattr(props, 'statistics_enabled')
+ assert props.version == 2
+ assert props.dictionary_enabled is True
+ assert props.statistics_enabled is True
+ assert isinstance(props.created_by, str)
Review Comment:
Why not test the actual value instead of this (or at least some substring of the value)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45902: [Python][Parquet] Expose ParquetWriter properties and arrow_properties [arrow]
Posted by "rok (via GitHub) via github" <gi...@arrow.apache.org>.
rok commented on PR #47087:
URL: https://github.com/apache/arrow/pull/47087#issuecomment-6533135041
@Reranko05 agreed. Let's unassign this issue to allow someone else complete it, but also leave room for @SoundBot to come back and do it in case they have time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45902: [Python][Parquet] Expose ParquetWriter properties and arrow_properties [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #47087:
URL: https://github.com/apache/arrow/pull/47087#discussion_r2220040785
##########
python/pyarrow/tests/parquet/test_parquet_writer.py:
##########
@@ -448,3 +448,46 @@ def test_parquet_content_defined_chunking_parameters(tempdir):
# using min_chunk_size, max_chunk_size and norm_level
cdc_options = {"min_chunk_size": 32_768, "max_chunk_size": 65_536, "norm_level": 1}
pq.write_table(table, path, use_content_defined_chunking=cdc_options)
+
+
+@pytest.mark.pandas
+def test_parquet_writer_properties_access():
+ df = _test_dataframe(100)
+ table = pa.Table.from_pandas(df, preserve_index=False)
Review Comment:
Agree, adding a new utility function to create Arrow tables and/or batches would be great! And perfect for `good-first-issue` 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45902: [Python][Parquet] Expose ParquetWriter properties and arrow_properties [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #47087:
URL: https://github.com/apache/arrow/pull/47087#discussion_r2224349784
##########
python/pyarrow/tests/parquet/test_parquet_writer.py:
##########
@@ -448,3 +448,46 @@ def test_parquet_content_defined_chunking_parameters(tempdir):
# using min_chunk_size, max_chunk_size and norm_level
cdc_options = {"min_chunk_size": 32_768, "max_chunk_size": 65_536, "norm_level": 1}
pq.write_table(table, path, use_content_defined_chunking=cdc_options)
+
+
+@pytest.mark.pandas
+def test_parquet_writer_properties_access():
+ df = _test_dataframe(100)
+ table = pa.Table.from_pandas(df, preserve_index=False)
Review Comment:
Created: https://github.com/apache/arrow/issues/47172
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45902: [Python][Parquet] Expose ParquetWriter properties and arrow_properties [arrow]
Posted by "SoundBot (via GitHub)" <gi...@apache.org>.
SoundBot commented on PR #47087:
URL: https://github.com/apache/arrow/pull/47087#issuecomment-3234965981
Hey folks, I suggest that if this code passes the sniff test, we merge it and address enhancements in later PRs.
If not, let's list the concrete changes I need to make to get this over the finish line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45902: [Python][Parquet] Expose ParquetWriter properties and arrow_properties [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47087:
URL: https://github.com/apache/arrow/pull/47087#discussion_r2314220120
##########
python/pyarrow/_parquet.pyx:
##########
@@ -2408,3 +2411,77 @@ cdef class ParquetWriter(_Weakrefable):
return result
raise RuntimeError(
'file metadata is only available after writer close')
+
+ @property
+ def properties(self):
+ cdef const WriterProperties* props_ptr
+ cdef WriterPropertiesWrapper result
+
+ props_ptr = &self.writer.get().properties()
+ result = WriterPropertiesWrapper.__new__(WriterPropertiesWrapper)
+ result.init(props_ptr)
+ return result
+
+ @property
+ def arrow_properties(self):
+ cdef ArrowWriterPropertiesWrapper result
+
+ result = ArrowWriterPropertiesWrapper.__new__(ArrowWriterPropertiesWrapper)
+ result.init(self._arrow_properties)
+ return result
+
+
+cdef class WriterPropertiesWrapper(_Weakrefable):
+ cdef:
+ const WriterProperties* props
+
+ def __cinit__(self):
+ self.props = NULL
+
+ cdef init(self, const WriterProperties* props):
+ self.props = props
+
+ @property
+ def version(self):
+ return self.props.version()
+
+ @property
+ def created_by(self):
+ return frombytes(self.props.created_by())
+
+ @property
+ def dictionary_enabled(self):
+ return self.props.default_column_properties().dictionary_enabled()
+
+ @property
+ def statistics_enabled(self):
+ return self.props.default_column_properties().statistics_enabled()
+
+
+cdef class ArrowWriterPropertiesWrapper(_Weakrefable):
+ cdef:
+ shared_ptr[ArrowWriterProperties] props
+
+ def __cinit__(self):
+ self.props.reset()
+
+ cdef init(self, shared_ptr[ArrowWriterProperties] props):
+ self.props = props
+
+ @property
+ def support_deprecated_int96_timestamps(self):
+ return self.props.get().support_deprecated_int96_timestamps()
+
+ @property
+ def store_schema(self):
+ return self.props.get().store_schema()
+
+ @property
+ def engine_version(self):
+ cdef int version_int = self.props.get().engine_version()
+ if version_int == 0: # V1
+ return "V1"
+ elif version_int == 1: # V2
+ return "V2"
Review Comment:
Why not return integers 1 or 2, respectively?
**Edit:** my bad, the correspond to the values that can be given when `ParquetWriter` is called.
##########
python/pyarrow/_parquet.pyx:
##########
@@ -2408,3 +2411,77 @@ cdef class ParquetWriter(_Weakrefable):
return result
raise RuntimeError(
'file metadata is only available after writer close')
+
+ @property
+ def properties(self):
+ cdef const WriterProperties* props_ptr
+ cdef WriterPropertiesWrapper result
+
+ props_ptr = &self.writer.get().properties()
+ result = WriterPropertiesWrapper.__new__(WriterPropertiesWrapper)
+ result.init(props_ptr)
+ return result
+
+ @property
+ def arrow_properties(self):
+ cdef ArrowWriterPropertiesWrapper result
+
+ result = ArrowWriterPropertiesWrapper.__new__(ArrowWriterPropertiesWrapper)
+ result.init(self._arrow_properties)
+ return result
+
+
+cdef class WriterPropertiesWrapper(_Weakrefable):
+ cdef:
+ const WriterProperties* props
+
+ def __cinit__(self):
+ self.props = NULL
+
+ cdef init(self, const WriterProperties* props):
+ self.props = props
+
+ @property
+ def version(self):
+ return self.props.version()
+
+ @property
+ def created_by(self):
+ return frombytes(self.props.created_by())
+
+ @property
+ def dictionary_enabled(self):
+ return self.props.default_column_properties().dictionary_enabled()
+
+ @property
+ def statistics_enabled(self):
+ return self.props.default_column_properties().statistics_enabled()
+
+
+cdef class ArrowWriterPropertiesWrapper(_Weakrefable):
+ cdef:
+ shared_ptr[ArrowWriterProperties] props
+
+ def __cinit__(self):
+ self.props.reset()
+
+ cdef init(self, shared_ptr[ArrowWriterProperties] props):
+ self.props = props
+
+ @property
+ def support_deprecated_int96_timestamps(self):
+ return self.props.get().support_deprecated_int96_timestamps()
+
+ @property
+ def store_schema(self):
+ return self.props.get().store_schema()
+
+ @property
+ def engine_version(self):
+ cdef int version_int = self.props.get().engine_version()
+ if version_int == 0: # V1
+ return "V1"
+ elif version_int == 1: # V2
+ return "V2"
Review Comment:
Why not return integers 1 or 2, respectively?
**Edit:** my bad, they correspond to the values that can be given when `ParquetWriter` is called.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45902: [Python][Parquet] Expose ParquetWriter properties and arrow_properties [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47087:
URL: https://github.com/apache/arrow/pull/47087#discussion_r2314218042
##########
python/pyarrow/_parquet.pyx:
##########
@@ -2408,3 +2411,77 @@ cdef class ParquetWriter(_Weakrefable):
return result
raise RuntimeError(
'file metadata is only available after writer close')
+
+ @property
+ def properties(self):
+ cdef const WriterProperties* props_ptr
+ cdef WriterPropertiesWrapper result
+
+ props_ptr = &self.writer.get().properties()
+ result = WriterPropertiesWrapper.__new__(WriterPropertiesWrapper)
+ result.init(props_ptr)
+ return result
+
+ @property
+ def arrow_properties(self):
+ cdef ArrowWriterPropertiesWrapper result
+
+ result = ArrowWriterPropertiesWrapper.__new__(ArrowWriterPropertiesWrapper)
+ result.init(self._arrow_properties)
+ return result
+
+
+cdef class WriterPropertiesWrapper(_Weakrefable):
+ cdef:
+ const WriterProperties* props
+
+ def __cinit__(self):
+ self.props = NULL
+
+ cdef init(self, const WriterProperties* props):
+ self.props = props
+
+ @property
+ def version(self):
+ return self.props.version()
+
+ @property
+ def created_by(self):
+ return frombytes(self.props.created_by())
+
+ @property
+ def dictionary_enabled(self):
+ return self.props.default_column_properties().dictionary_enabled()
+
+ @property
+ def statistics_enabled(self):
+ return self.props.default_column_properties().statistics_enabled()
+
+
+cdef class ArrowWriterPropertiesWrapper(_Weakrefable):
Review Comment:
Same here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45902: [Python][Parquet] Expose ParquetWriter properties and arrow_properties [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on PR #47087:
URL: https://github.com/apache/arrow/pull/47087#issuecomment-3097516414
The `example-python-minimal-build-*` failures are unrelated to the PR, known issue:
- https://github.com/apache/arrow/issues/44803
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45902: [Python][Parquet] Expose ParquetWriter properties and arrow_properties [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #47087:
URL: https://github.com/apache/arrow/pull/47087#discussion_r2308950455
##########
python/pyarrow/_parquet.pyx:
##########
@@ -2408,3 +2411,77 @@ cdef class ParquetWriter(_Weakrefable):
return result
raise RuntimeError(
'file metadata is only available after writer close')
+
+ @property
+ def properties(self):
+ cdef const WriterProperties* props_ptr
+ cdef WriterPropertiesWrapper result
+
+ props_ptr = &self.writer.get().properties()
+ result = WriterPropertiesWrapper.__new__(WriterPropertiesWrapper)
+ result.init(props_ptr)
+ return result
+
+ @property
+ def arrow_properties(self):
+ cdef ArrowWriterPropertiesWrapper result
+
+ result = ArrowWriterPropertiesWrapper.__new__(ArrowWriterPropertiesWrapper)
+ result.init(self._arrow_properties)
+ return result
+
+
+cdef class WriterPropertiesWrapper(_Weakrefable):
Review Comment:
Sounds good to me. WDYT? @pitrou
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45902: [Python][Parquet] Expose ParquetWriter properties and arrow_properties [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47087:
URL: https://github.com/apache/arrow/pull/47087#discussion_r2314250106
##########
python/pyarrow/tests/parquet/test_parquet_writer.py:
##########
@@ -448,3 +448,46 @@ def test_parquet_content_defined_chunking_parameters(tempdir):
# using min_chunk_size, max_chunk_size and norm_level
cdc_options = {"min_chunk_size": 32_768, "max_chunk_size": 65_536, "norm_level": 1}
pq.write_table(table, path, use_content_defined_chunking=cdc_options)
+
+
+@pytest.mark.pandas
+def test_parquet_writer_properties_access():
+ df = _test_dataframe(100)
+ table = pa.Table.from_pandas(df, preserve_index=False)
+
+ out = pa.BufferOutputStream()
+
+ with pq.ParquetWriter(
+ out, table.schema,
+ compression='snappy',
+ version='2.6',
+ use_dictionary=True,
+ write_statistics=True,
+ data_page_size=64 * 1024,
+ use_deprecated_int96_timestamps=False,
+ store_schema=True,
+ use_compliant_nested_type=True,
+ writer_engine_version='V2'
+ ) as writer:
+
+ props = writer.properties
+ assert props is not None
+ assert hasattr(props, 'version')
+ assert hasattr(props, 'created_by')
+ assert hasattr(props, 'dictionary_enabled')
+ assert hasattr(props, 'statistics_enabled')
Review Comment:
No need for all this, the attribute accesses below are sufficient.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45902: [Python][Parquet] Expose ParquetWriter properties and arrow_properties [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47087:
URL: https://github.com/apache/arrow/pull/47087#discussion_r2314219102
##########
python/pyarrow/_parquet.pyx:
##########
@@ -2408,3 +2411,77 @@ cdef class ParquetWriter(_Weakrefable):
return result
raise RuntimeError(
'file metadata is only available after writer close')
+
+ @property
+ def properties(self):
+ cdef const WriterProperties* props_ptr
+ cdef WriterPropertiesWrapper result
+
+ props_ptr = &self.writer.get().properties()
+ result = WriterPropertiesWrapper.__new__(WriterPropertiesWrapper)
+ result.init(props_ptr)
+ return result
+
+ @property
+ def arrow_properties(self):
+ cdef ArrowWriterPropertiesWrapper result
+
+ result = ArrowWriterPropertiesWrapper.__new__(ArrowWriterPropertiesWrapper)
+ result.init(self._arrow_properties)
+ return result
+
+
+cdef class WriterPropertiesWrapper(_Weakrefable):
+ cdef:
+ const WriterProperties* props
+
+ def __cinit__(self):
+ self.props = NULL
+
+ cdef init(self, const WriterProperties* props):
+ self.props = props
+
+ @property
+ def version(self):
+ return self.props.version()
+
+ @property
+ def created_by(self):
+ return frombytes(self.props.created_by())
+
+ @property
+ def dictionary_enabled(self):
+ return self.props.default_column_properties().dictionary_enabled()
+
+ @property
+ def statistics_enabled(self):
+ return self.props.default_column_properties().statistics_enabled()
+
+
+cdef class ArrowWriterPropertiesWrapper(_Weakrefable):
+ cdef:
+ shared_ptr[ArrowWriterProperties] props
+
+ def __cinit__(self):
+ self.props.reset()
+
+ cdef init(self, shared_ptr[ArrowWriterProperties] props):
+ self.props = props
+
+ @property
+ def support_deprecated_int96_timestamps(self):
+ return self.props.get().support_deprecated_int96_timestamps()
+
+ @property
+ def store_schema(self):
+ return self.props.get().store_schema()
+
+ @property
+ def engine_version(self):
+ cdef int version_int = self.props.get().engine_version()
+ if version_int == 0: # V1
+ return "V1"
+ elif version_int == 1: # V2
+ return "V2"
+ else:
+ return f"unknown({version_int})"
Review Comment:
We should probably raise `ValueError` here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45902: [Python][Parquet] Expose ParquetWriter properties and arrow_properties [arrow]
Posted by "SoundBot (via GitHub)" <gi...@apache.org>.
SoundBot commented on code in PR #47087:
URL: https://github.com/apache/arrow/pull/47087#discussion_r2292728008
##########
python/pyarrow/_parquet.pyx:
##########
@@ -2408,3 +2411,77 @@ cdef class ParquetWriter(_Weakrefable):
return result
raise RuntimeError(
'file metadata is only available after writer close')
+
+ @property
+ def properties(self):
+ cdef const WriterProperties* props_ptr
+ cdef WriterPropertiesWrapper result
+
+ props_ptr = &self.writer.get().properties()
+ result = WriterPropertiesWrapper.__new__(WriterPropertiesWrapper)
+ result.init(props_ptr)
+ return result
+
+ @property
+ def arrow_properties(self):
+ cdef ArrowWriterPropertiesWrapper result
+
+ result = ArrowWriterPropertiesWrapper.__new__(ArrowWriterPropertiesWrapper)
+ result.init(self._arrow_properties)
+ return result
+
+
+cdef class WriterPropertiesWrapper(_Weakrefable):
Review Comment:
That's correct. The full WriterProperties object is accessible in the Cython code, but the Python wrapper intentionally exposes only a subset.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45902: [Python][Parquet] Expose ParquetWriter properties and arrow_properties [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #47087:
URL: https://github.com/apache/arrow/pull/47087#issuecomment-5508736009
Thank you for your contribution. Unfortunately, this pull request has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this PR will be closed in 14 days. Feel free to re-open this if it has been closed in error. If you do not have repository permissions to reopen the PR, please tag a maintainer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org