You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "AlenkaF (via GitHub)" <gi...@apache.org> on 2026/07/08 13:58:00 UTC
[PR] GH-47686: [Docs][Python] Split the Python Parquet docs into separate items [arrow]
AlenkaF opened a new pull request, #50419:
URL: https://github.com/apache/arrow/pull/50419
### Rationale for this change
The current parquet section is one-page and covering lots of topics. As per https://github.com/apache/arrow/issues/46601 the parquet section is most visited and would therefore need a bit of maintenance/content update.
### What changes are included in this PR?
- Parquet documentation is split in 4 different subpages (index.rst, parquet.rst, parquet_datasets.rst and parquet_encryption.rst).
- Some additional content is added
### Are these changes tested?
Via the docs preview
### Are there any user-facing changes?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47686: [Docs][Python] Split the Python Parquet docs into separate items [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #50419:
URL: https://github.com/apache/arrow/pull/50419#issuecomment-5034459293
Revision: 706fa2966919845969a1f603f5f209cb6b0c7223
Submitted crossbow builds: [ursacomputing/crossbow @ actions-44053c8153](https://github.com/ursacomputing/crossbow/branches/all?query=actions-44053c8153)
|Task|Status|
|----|------|
|preview-docs|[](https://github.com/ursacomputing/crossbow/actions/runs/29833162263/job/88642682254)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47686: [Docs][Python] Split the Python Parquet docs into separate items [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #50419:
URL: https://github.com/apache/arrow/pull/50419#issuecomment-5025170362
Revision: 9773c76ebb5284fd7a3e7ba1afaf14de51809270
Submitted crossbow builds: [ursacomputing/crossbow @ actions-4a1d237853](https://github.com/ursacomputing/crossbow/branches/all?query=actions-4a1d237853)
|Task|Status|
|----|------|
|preview-docs|[](https://github.com/ursacomputing/crossbow/actions/runs/29763841319/job/88424898706)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47686: [Docs][Python] Split the Python Parquet docs into separate items [arrow]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #50419:
URL: https://github.com/apache/arrow/pull/50419#discussion_r3615036047
##########
docs/source/python/parquet/parquet.rst:
##########
@@ -0,0 +1,605 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. currentmodule:: pyarrow
+.. _parquet_single_file:
+
+
+Reading and Writing Single Files
+================================
+
+The functions :func:`~.parquet.read_table` and :func:`~.parquet.write_table`
+read and write the :ref:`pyarrow.Table <data.table>` object, respectively.
+
+Let's look at a simple table:
+
+.. code-block:: python
+
+ >>> import numpy as np
+ >>> import pandas as pd
+ >>> import pyarrow as pa
+ >>> df = pd.DataFrame({'one': [-1, np.nan, 2.5],
+ ... 'two': ['foo', 'bar', 'baz'],
+ ... 'three': [True, False, True]},
+ ... index=list('abc'))
+ >>> table = pa.Table.from_pandas(df)
+
+We write this to Parquet format with ``write_table``:
+
+.. code-block:: python
+
+ >>> import pyarrow.parquet as pq
+ >>> pq.write_table(table, 'example.parquet')
+
+This creates a single Parquet file. In practice, a Parquet dataset may consist
+of many files in many directories. We can read a single file back with
+``read_table``:
+
+.. code-block:: python
+
+ >>> table2 = pq.read_table('example.parquet')
+ >>> table2.to_pandas()
+ one two three
+ a -1.0 foo True
+ b NaN bar False
+ c 2.5 baz True
+
+You can pass a subset of columns to read, which can be much faster than reading
+the whole file (due to the columnar layout):
+
+.. code-block:: python
+
+ >>> pq.read_table('example.parquet', columns=['one', 'three'])
+ pyarrow.Table
+ one: double
+ three: bool
+ ----
+ one: [[-1,null,2.5]]
+ three: [[true,false,true]]
+
+When reading a subset of columns from a file that used a Pandas dataframe as the
+source, we use ``read_pandas`` to maintain any additional index column data:
+
+.. code-block:: python
+
+ >>> pq.read_pandas('example.parquet', columns=['two']).to_pandas()
+ two
+ a foo
+ b bar
+ c baz
+
+We do not need to use a string to specify the origin of the file. It can be any of:
+
+* A file path as a string
+* A :ref:`NativeFile <io.native_file>` from PyArrow
+* A Python file object
+
+In general, a Python file object will have the worst read performance, while a
+string file path or an instance of :class:`~.NativeFile` (especially memory
+maps) will perform the best.
+
+.. _parquet_mmap:
+
+Reading Parquet and Memory Mapping
+----------------------------------
+
+Because Parquet data needs to be decoded from the Parquet format
+and compression, it can't be directly mapped from disk.
+Thus the ``memory_map`` option might perform better on some systems
+but won't help much with resident memory consumption.
+
+.. code-block:: python
+
+ >>> pq_array = pa.parquet.read_table(path, memory_map=True) # doctest: +SKIP
+ >>> print("RSS: {}MB".format(pa.total_allocated_bytes() >> 20)) # doctest: +SKIP
+ RSS: 4299MB
+
+ >>> pq_array = pa.parquet.read_table(path, memory_map=False) # doctest: +SKIP
+ >>> print("RSS: {}MB".format(pa.total_allocated_bytes() >> 20)) # doctest: +SKIP
+ RSS: 4299MB
+
+If you need to deal with Parquet data bigger than memory,
+the :ref:`dataset` and partitioning is probably what you are looking for.
+
+Parquet file writing options
+----------------------------
+
+:func:`~pyarrow.parquet.write_table()` has a number of options to
+control various settings when writing a Parquet file.
+
+* ``version``, the Parquet format version to use. ``'1.0'`` ensures
+ compatibility with older readers, while ``'2.4'`` and greater values
+ enable more Parquet types and encodings.
+* ``data_page_size``, to control the approximate size of encoded data
+ pages within a column chunk. This currently defaults to 1MB.
+* ``max_rows_per_page``, to cap the number of rows per data page within
+ a column chunk (default 20000). Smaller values reduce memory usage
+ during reads at the cost of more page metadata.
+* ``flavor``, to set compatibility options particular to a Parquet
+ consumer like ``'spark'`` for Apache Spark.
+* ``store_decimal_as_integer``, to store decimals with precision 1–18
+ as ``int32`` or ``int64`` instead of ``fixed_len_byte_array``.
+ This produces more compact files but may not be supported by all readers.
+* ``write_time_adjusted_to_utc``, to mark ``TIME`` columns as
+ adjusted to UTC (``isAdjustedToUTC=True``). When ``False`` (the default),
+ the time is treated as local/unknown timezone.
+* ``write_page_index``, to write statistics to the page index
+ instead of writing it to each data page header.
+ Note that PyArrow does not yet use the page index on the read side.
+* ``write_page_checksum``, to write a page checksum. Use with
+ ``page_checksum_verification=True`` on read to detect data corruption.
+* ``sorting_columns``, to record the sort order of the data in each
+ row group's metadata. The writer does not sort the data nor does it verify
+ that the data is sorted. Readers can use this metadata to optimize queries.
+
+ Sort order is expressed as a sequence of :class:`~pyarrow.parquet.SortingColumn`
+ objects.
+
+See the :func:`~pyarrow.parquet.write_table()` docstring for more details.
+
+There are some additional data type handling-specific options
+described below.
+
+Omitting the DataFrame index
+----------------------------
+
+When using ``pa.Table.from_pandas`` to convert to an Arrow table, by default
+one or more special columns are added to keep track of the index (row
+labels). Storing the index takes extra space, so if your index is not valuable,
+you may choose to omit it by passing ``preserve_index=False``
+
+.. code-block:: python
+
+ >>> df = pd.DataFrame({'one': [-1, np.nan, 2.5],
+ ... 'two': ['foo', 'bar', 'baz'],
+ ... 'three': [True, False, True]},
+ ... index=list('abc'))
+ >>> table = pa.Table.from_pandas(df, preserve_index=False)
+
+Then we have:
+
+.. code-block:: python
+
+ >>> pq.write_table(table, 'example_noindex.parquet')
+ >>> t = pq.read_table('example_noindex.parquet')
+ >>> t.to_pandas()
+ one two three
+ 0 -1.0 foo True
+ 1 NaN bar False
+ 2 2.5 baz True
+
+Here you see the index did not survive the round trip.
+
+Finer-grained Reading and Writing
+---------------------------------
+
+``read_table`` uses the :class:`~.ParquetFile` class, which has other features:
Review Comment:
The cross-reference `:class:`~.ParquetFile`` resolves to `pyarrow.ParquetFile` (which is not exported) rather than `pyarrow.parquet.ParquetFile`. This likely produces an unresolved reference and is inconsistent with `parquet_type_handling.rst`, which uses fully-qualified `pyarrow.parquet.*` refs. Consider updating related refs in this page too (e.g. `:meth:`~.ParquetFile.iter_batches`` and `:func:`~parquet.read_metadata``) and similar occurrences in `parquet_datasets.rst` / `parquet_encryption.rst`.
##########
docs/source/python/parquet/parquet.rst:
##########
@@ -0,0 +1,605 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. currentmodule:: pyarrow
+.. _parquet_single_file:
+
+
+Reading and Writing Single Files
+================================
+
+The functions :func:`~.parquet.read_table` and :func:`~.parquet.write_table`
+read and write the :ref:`pyarrow.Table <data.table>` object, respectively.
+
+Let's look at a simple table:
+
+.. code-block:: python
+
+ >>> import numpy as np
+ >>> import pandas as pd
+ >>> import pyarrow as pa
+ >>> df = pd.DataFrame({'one': [-1, np.nan, 2.5],
+ ... 'two': ['foo', 'bar', 'baz'],
+ ... 'three': [True, False, True]},
+ ... index=list('abc'))
+ >>> table = pa.Table.from_pandas(df)
+
+We write this to Parquet format with ``write_table``:
+
+.. code-block:: python
+
+ >>> import pyarrow.parquet as pq
+ >>> pq.write_table(table, 'example.parquet')
+
+This creates a single Parquet file. In practice, a Parquet dataset may consist
+of many files in many directories. We can read a single file back with
+``read_table``:
+
+.. code-block:: python
+
+ >>> table2 = pq.read_table('example.parquet')
+ >>> table2.to_pandas()
+ one two three
+ a -1.0 foo True
+ b NaN bar False
+ c 2.5 baz True
+
+You can pass a subset of columns to read, which can be much faster than reading
+the whole file (due to the columnar layout):
+
+.. code-block:: python
+
+ >>> pq.read_table('example.parquet', columns=['one', 'three'])
+ pyarrow.Table
+ one: double
+ three: bool
+ ----
+ one: [[-1,null,2.5]]
+ three: [[true,false,true]]
+
+When reading a subset of columns from a file that used a Pandas dataframe as the
+source, we use ``read_pandas`` to maintain any additional index column data:
+
+.. code-block:: python
+
+ >>> pq.read_pandas('example.parquet', columns=['two']).to_pandas()
+ two
+ a foo
+ b bar
+ c baz
+
+We do not need to use a string to specify the origin of the file. It can be any of:
+
+* A file path as a string
+* A :ref:`NativeFile <io.native_file>` from PyArrow
+* A Python file object
+
+In general, a Python file object will have the worst read performance, while a
+string file path or an instance of :class:`~.NativeFile` (especially memory
+maps) will perform the best.
+
+.. _parquet_mmap:
+
+Reading Parquet and Memory Mapping
+----------------------------------
+
+Because Parquet data needs to be decoded from the Parquet format
+and compression, it can't be directly mapped from disk.
+Thus the ``memory_map`` option might perform better on some systems
+but won't help much with resident memory consumption.
+
+.. code-block:: python
+
+ >>> pq_array = pa.parquet.read_table(path, memory_map=True) # doctest: +SKIP
+ >>> print("RSS: {}MB".format(pa.total_allocated_bytes() >> 20)) # doctest: +SKIP
+ RSS: 4299MB
+
+ >>> pq_array = pa.parquet.read_table(path, memory_map=False) # doctest: +SKIP
Review Comment:
The memory-mapping example uses `pa.parquet.read_table`, but `ParquetFile`/`read_table` live under `pyarrow.parquet` (imported as `pq` above). As written, `pa.parquet` is not otherwise used in the docs and is likely incorrect/inconsistent for readers.
##########
docs/source/python/parquet/parquet_encryption.rst:
##########
@@ -0,0 +1,280 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. currentmodule:: pyarrow
+.. _parquet_encryption:
+
+
+Parquet Modular Encryption (Columnar Encryption)
+------------------------------------------------
+
+Columnar encryption is supported for Parquet files in C++ starting from
+Apache Arrow 4.0.0 and in PyArrow starting from Apache Arrow 6.0.0.
+
+Parquet uses the envelope encryption practice, where file parts are encrypted
+with "data encryption keys" (DEKs), and the DEKs are encrypted with "master
+encryption keys" (MEKs). The DEKs are randomly generated by Parquet for each
+encrypted file/column. The MEKs are generated, stored and managed in a Key
+Management Service (KMS) of user’s choice.
+
+Reading and writing encrypted Parquet files involves passing file encryption
+and decryption properties to :class:`~pyarrow.parquet.ParquetWriter` and to
+:class:`~.ParquetFile`, respectively.
+
+Writing an encrypted Parquet file:
+
+.. code-block:: python
+
+ >>> import pyarrow.parquet as pq
+ >>> encryption_properties = crypto_factory.file_encryption_properties( # doctest: +SKIP
+ ... kms_connection_config, encryption_config)
+ >>> with pq.ParquetWriter(filename, schema, # doctest: +SKIP
+ ... encryption_properties=encryption_properties) as writer:
+ ... writer.write_table(table)
+
+Reading an encrypted Parquet file:
+
+.. code-block:: python
+
+ >>> decryption_properties = crypto_factory.file_decryption_properties( # doctest: +SKIP
+ ... kms_connection_config)
+ >>> parquet_file = pq.ParquetFile(filename, # doctest: +SKIP
+ ... decryption_properties=decryption_properties)
+
+
+In order to create the encryption and decryption properties, a
+:class:`pyarrow.parquet.encryption.CryptoFactory` should be created and
+initialized with KMS Client details, as described below.
+
+
+KMS Client
+~~~~~~~~~~
+
+The master encryption keys should be kept and managed in a production-grade
+Key Management System (KMS), deployed in the user's organization. Using Parquet
+encryption requires implementation of a client class for the KMS server.
+Any KmsClient implementation should implement the informal interface
+defined by :class:`pyarrow.parquet.encryption.KmsClient` as following:
+
+.. code-block:: python
+
+ >>> import pyarrow.parquet.encryption as pe
+ >>> class MyKmsClient(pe.KmsClient):
+ ...
+ ... """An example KmsClient implementation skeleton"""
+ ... def __init__(self, kms_connection_configuration):
+ ... pe.KmsClient.__init__(self)
+ ... # Any KMS-specific initialization based on
+ ... # kms_connection_configuration comes here
+ ...
+ ... def wrap_key(self, key_bytes, master_key_identifier):
+ ... wrapped_key = ... # call KMS to wrap key_bytes with key specified by
+ ... # master_key_identifier
+ ... return wrapped_key
+ ...
+ ... def unwrap_key(self, wrapped_key, master_key_identifier):
+ ... key_bytes = ... # call KMS to unwrap wrapped_key with key specified by
+ ... # master_key_identifier
+ ... return key_bytes
+
+The concrete implementation will be loaded at runtime by a factory function
+provided by the user. This factory function will be used to initialize the
+:class:`pyarrow.parquet.encryption.CryptoFactory` for creating file encryption
+and decryption properties.
+
+For example, in order to use the ``MyKmsClient`` defined above:
+
+.. code-block:: python
+
+ >>> def kms_client_factory(kms_connection_configuration):
+ ... return MyKmsClient(kms_connection_configuration)
+
+ >>> crypto_factory = pe.CryptoFactory(kms_client_factory)
+
+An :download:`example <../../../python/examples/parquet_encryption/sample_vault_kms_client.py>`
+of such a class for an open source
+`KMS <https://www.vaultproject.io/api/secret/transit>`_ can be found in the Apache
+Arrow GitHub repository. The production KMS client should be designed in
+cooperation with an organization's security administrators, and built by
+developers with experience in access control management. Once such a class is
+created, it can be passed to applications via a factory method and leveraged
+by general PyArrow users as shown in the encrypted parquet write/read sample
+above.
+
+KMS connection configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Configuration of connection to KMS (:class:`pyarrow.parquet.encryption.KmsConnectionConfig`
+used when creating file encryption and decryption properties) includes the
+following options:
+
+* ``kms_instance_url``, URL of the KMS instance.
+* ``kms_instance_id``, ID of the KMS instance that will be used for encryption
+ (if multiple KMS instances are available).
+* ``key_access_token``, authorization token that will be passed to KMS.
+* ``custom_kms_conf``, a string dictionary with KMS-type-specific configuration.
+
+Encryption configuration
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+:class:`pyarrow.parquet.encryption.EncryptionConfiguration` (used when
+creating file encryption properties) includes the following options:
+
+* ``footer_key``, the ID of the master key for footer encryption/signing.
+* ``column_keys``, which columns to encrypt with which key. Dictionary with
+ master key IDs as the keys, and column name lists as the values,
+ e.g. ``{key1: [col1, col2], key2: [col3]}``. See notes on nested fields below.
+* ``uniform_encryption``, whether to encrypt the footer and all columns with
+ the same ``footer_key``, instead of specifying ``column_keys``
+ individually. Cannot be used together with ``column_keys``.
+* ``encryption_algorithm``, the Parquet encryption algorithm.
+ Can be ``AES_GCM_V1`` (default) or ``AES_GCM_CTR_V1``.
+* ``plaintext_footer``, whether to write the file footer in plain text (otherwise it is encrypted).
+* ``double_wrapping``, whether to use double wrapping - where data encryption keys (DEKs)
+ are encrypted with key encryption keys (KEKs), which in turn are encrypted
+ with master encryption keys (MEKs). If set to ``false``, single wrapping is
+ used - where DEKs are encrypted directly with MEKs.
+* ``cache_lifetime``, the lifetime of cached entities (key encryption keys,
+ local wrapping keys, KMS client objects) represented as a ``datetime.timedelta``.
+* ``internal_key_material``, whether to store key material inside Parquet file footers;
+ this mode doesn’t produce additional files. If set to ``false``, key material is
+ stored in separate files in the same folder, which enables key rotation for
+ immutable Parquet files.
+* ``data_key_length_bits``, the length of data encryption keys (DEKs), randomly
+ generated by Parquet key management tools. Can be 128, 192 or 256 bits.
+
+.. note::
+ When ``double_wrapping`` is true, Parquet implements a "double envelope
+ encryption" mode that minimizes the interaction of the program with a KMS
+ server. In this mode, the DEKs are encrypted with "key encryption keys"
+ (KEKs, randomly generated by Parquet). The KEKs are encrypted with "master
+ encryption keys" (MEKs) in the KMS; the result and the KEK itself are
+ cached in the process memory.
+
+An example encryption configuration:
+
+.. code-block:: python
+
+ >>> encryption_config = pe.EncryptionConfiguration(
+ ... footer_key="footer_key_name",
+ ... column_keys={
+ ... "column_key_name": ["Column1", "Column2"],
+ ... },
+ ... )
+
+.. note::
+
+ Columns with nested fields (struct or map data types) can be encrypted as a whole, or only
+ individual fields. Configure an encryption key for the root column name to encrypt all nested
+ fields with this key, or configure a key for individual leaf nested fields.
+
+ Conventionally, the key and value fields of a map column ``m`` have the names
+ ``m.key_value.key`` and ``m.key_value.value``, respectively.
+ An inner field ``f`` of a struct column ``s`` has the name ``s.f``.
+
+ With above example, *all* inner fields are encrypted with the same key by configuring that key
+ for column ``m`` and ``s``, respectively.
+
+An example encryption configuration for columns with nested fields, where
+all columns are encrypted with the same key identified by ``column_key_id``:
+
+.. code-block:: python
+
+ >>> import pyarrow as pa
+ >>> schema = pa.schema([
+ ... ("MapColumn", pa.map_(pa.string(), pa.int32())),
+ ... ("StructColumn", pa.struct([("f1", pa.int32()), ("f2", pa.string())])),
+ ... ])
+
+ >>> encryption_config = pe.EncryptionConfiguration(
+ ... footer_key="footer_key_name",
+ ... column_keys={
+ ... "column_key_id": [ "MapColumn", "StructColumn" ],
+ ... },
+ ... )
+
+An example encryption configuration for columns with nested fields, where
+some inner fields are encrypted with the same key identified by ``column_key_id``:
+
+.. code-block:: python
+
+ >>> schema = pa.schema([
+ ... ("MapColumn", pa.map_(pa.string(), pa.int32())),
+ ... ("StructColumn", pa.struct([("f1", pa.int32()), ("f2", pa.string())])),
+ ... ])
+
+ >>> encryption_config = pe.EncryptionConfiguration(
+ ... footer_key="footer_key_name",
+ ... column_keys={
+ ... "column_key_id": [ "MapColumn.key_value.value", "StructColumn.f1" ],
+ ... },
+ ... )
+
+Decryption configuration
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+:class:`pyarrow.parquet.encryption.DecryptionConfiguration` (used when creating
+file decryption properties) is optional and it includes the following options:
+
+* ``cache_lifetime``, the lifetime of cached entities (key encryption keys, local
+ wrapping keys, KMS client objects) represented as a ``datetime.timedelta``.
+
+External key material and key rotation
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+When ``internal_key_material=False`` is set on ``EncryptionConfiguration``,
+key material is stored in a separate file next to the Parquet file instead of
+in its footer.
+
+Storing key material externally is what enables key rotation:
+:meth:`~.crypto_factory.rotate_master_keys()` re-wraps the data encryption keys of
+a file that uses external key material using new master keys and overwrites the external
Review Comment:
The method cross-reference `:meth:`~.crypto_factory.rotate_master_keys()`` points to a local variable name and is unlikely to resolve in Sphinx. It should reference the `CryptoFactory.rotate_master_keys` method instead.
##########
docs/source/python/parquet/parquet_type_handling.rst:
##########
@@ -0,0 +1,156 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. currentmodule:: pyarrow
+.. _parquet_type_handling:
+
+
+Data Type Handling
+==================
+
+Reading types as DictionaryArray
+--------------------------------
+
+The ``read_dictionary`` option in ``read_table`` and ``ParquetDataset`` will
+cause columns to be read as ``DictionaryArray``, which will become
+``pandas.Categorical`` when converted to pandas. This option is only valid for
+string and binary column types, and it can yield significantly lower memory use
+and improved performance for columns with many repeated string values.
+
+.. code-block:: python
+
+ >>> import pyarrow as pa
+ >>> import pyarrow.parquet as pq
+
+ >>> table = pa.table({'one': [-1, None, 2.5],
+ ... 'two': ['foo', 'bar', 'baz'],
+ ... 'three': [True, False, True]})
+ ...
+ >>> pq.write_table(table, 'example.parquet')
+
+ >>> pq.read_table('example.parquet', read_dictionary=['two'])
+ pyarrow.Table
+ one: double
+ two: dictionary<values=string, indices=int32, ordered=0>
+ three: bool
+ ----
+ one: [[-1,null,2.5]]
+ two: [ -- dictionary:
+ ["foo","bar","baz"] -- indices:
+ [0,1,2]]
+ three: [[true,false,true]]
+
+Reading binary and list columns
+-------------------------------
+
+By default, Parquet ``BYTE_ARRAY`` columns are read as Arrow ``binary`` type
+and Parquet ``LIST`` columns are read as Arrow ``list`` type, both of which use
+32-bit offsets. For very large datasets this can overflow. The ``binary_type``
+and ``list_type`` parameters let you choose a different Arrow type on read.
+
+``binary_type`` accepts ``pa.binary()`` (default), ``pa.large_binary()``, or
+``pa.binary_view()``:
+
+.. code-block:: python
+
+ >>> import pyarrow as pa
+ >>> table = pa.table({'data': pa.array([b'hello', b'world'], pa.binary())})
+ >>> pq.write_table(table, 'binary.parquet', store_schema=False)
+ >>> pq.read_table('binary.parquet', binary_type=pa.large_binary()).schema
Review Comment:
This code block uses `pq.write_table` / `pq.read_table` but does not import `pyarrow.parquet as pq`, so the example won’t run if copied independently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47686: [Docs][Python] Split the Python Parquet docs into separate items [arrow]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #50419:
URL: https://github.com/apache/arrow/pull/50419#discussion_r3615162273
##########
docs/source/python/parquet/parquet_type_handling.rst:
##########
@@ -0,0 +1,156 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. currentmodule:: pyarrow
+.. _parquet_type_handling:
+
+
+Data Type Handling
+==================
+
+Reading types as DictionaryArray
+--------------------------------
+
+The ``read_dictionary`` option in ``read_table`` and ``ParquetDataset`` will
+cause columns to be read as ``DictionaryArray``, which will become
+``pandas.Categorical`` when converted to pandas. This option is only valid for
+string and binary column types, and it can yield significantly lower memory use
+and improved performance for columns with many repeated string values.
+
+.. code-block:: python
+
+ >>> import pyarrow as pa
+ >>> import pyarrow.parquet as pq
+
+ >>> table = pa.table({'one': [-1, None, 2.5],
+ ... 'two': ['foo', 'bar', 'baz'],
+ ... 'three': [True, False, True]})
+ ...
+ >>> pq.write_table(table, 'example.parquet')
+
+ >>> pq.read_table('example.parquet', read_dictionary=['two'])
+ pyarrow.Table
+ one: double
+ two: dictionary<values=string, indices=int32, ordered=0>
+ three: bool
+ ----
+ one: [[-1,null,2.5]]
+ two: [ -- dictionary:
+ ["foo","bar","baz"] -- indices:
+ [0,1,2]]
+ three: [[true,false,true]]
+
+Reading binary and list columns
+-------------------------------
+
+By default, Parquet ``BYTE_ARRAY`` columns are read as Arrow ``binary`` type
+and Parquet ``LIST`` columns are read as Arrow ``list`` type, both of which use
+32-bit offsets. For very large datasets this can overflow. The ``binary_type``
+and ``list_type`` parameters let you choose a different Arrow type on read.
+
+``binary_type`` accepts ``pa.binary()`` (default), ``pa.large_binary()``, or
+``pa.binary_view()``:
+
+.. code-block:: python
+
+ >>> import pyarrow as pa
+ >>> table = pa.table({'data': pa.array([b'hello', b'world'], pa.binary())})
+ >>> pq.write_table(table, 'binary.parquet', store_schema=False)
+ >>> pq.read_table('binary.parquet', binary_type=pa.large_binary()).schema
+ data: large_binary
+
+``list_type`` accepts ``pa.ListType`` or ``pa.LargeListType``:
+
+.. code-block:: python
+
+ >>> table = pa.table({'lists': pa.array([[1, 2], [3]], pa.list_(pa.int32()))})
+ >>> pq.write_table(table, 'lists.parquet', store_schema=False)
+ >>> pq.read_table('lists.parquet', list_type=pa.LargeListType).schema
+ lists: large_list<element: int32>
Review Comment:
This list example also uses `pq.*` without importing `pyarrow.parquet as pq` (and doesn't import `pyarrow as pa` within the snippet), so it's not self-contained.
##########
docs/source/python/parquet/parquet.rst:
##########
@@ -0,0 +1,605 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. currentmodule:: pyarrow
+.. _parquet_single_file:
+
+
+Reading and Writing Single Files
+================================
+
+The functions :func:`~.parquet.read_table` and :func:`~.parquet.write_table`
+read and write the :ref:`pyarrow.Table <data.table>` object, respectively.
+
+Let's look at a simple table:
+
+.. code-block:: python
+
+ >>> import numpy as np
+ >>> import pandas as pd
+ >>> import pyarrow as pa
+ >>> df = pd.DataFrame({'one': [-1, np.nan, 2.5],
+ ... 'two': ['foo', 'bar', 'baz'],
+ ... 'three': [True, False, True]},
+ ... index=list('abc'))
+ >>> table = pa.Table.from_pandas(df)
+
+We write this to Parquet format with ``write_table``:
+
+.. code-block:: python
+
+ >>> import pyarrow.parquet as pq
+ >>> pq.write_table(table, 'example.parquet')
+
+This creates a single Parquet file. In practice, a Parquet dataset may consist
+of many files in many directories. We can read a single file back with
+``read_table``:
+
+.. code-block:: python
+
+ >>> table2 = pq.read_table('example.parquet')
+ >>> table2.to_pandas()
+ one two three
+ a -1.0 foo True
+ b NaN bar False
+ c 2.5 baz True
+
+You can pass a subset of columns to read, which can be much faster than reading
+the whole file (due to the columnar layout):
+
+.. code-block:: python
+
+ >>> pq.read_table('example.parquet', columns=['one', 'three'])
+ pyarrow.Table
+ one: double
+ three: bool
+ ----
+ one: [[-1,null,2.5]]
+ three: [[true,false,true]]
+
+When reading a subset of columns from a file that used a Pandas dataframe as the
+source, we use ``read_pandas`` to maintain any additional index column data:
+
+.. code-block:: python
+
+ >>> pq.read_pandas('example.parquet', columns=['two']).to_pandas()
+ two
+ a foo
+ b bar
+ c baz
+
+We do not need to use a string to specify the origin of the file. It can be any of:
+
+* A file path as a string
+* A :ref:`NativeFile <io.native_file>` from PyArrow
+* A Python file object
+
+In general, a Python file object will have the worst read performance, while a
+string file path or an instance of :class:`~.NativeFile` (especially memory
+maps) will perform the best.
+
+.. _parquet_mmap:
+
+Reading Parquet and Memory Mapping
+----------------------------------
+
+Because Parquet data needs to be decoded from the Parquet format
+and compression, it can't be directly mapped from disk.
+Thus the ``memory_map`` option might perform better on some systems
+but won't help much with resident memory consumption.
+
+.. code-block:: python
+
+ >>> pq_array = pq.read_table(path, memory_map=True) # doctest: +SKIP
+ >>> print("RSS: {}MB".format(pa.total_allocated_bytes() >> 20)) # doctest: +SKIP
+ RSS: 4299MB
Review Comment:
The snippet labels `pa.total_allocated_bytes()` as "RSS", but that API reports Arrow-allocated memory, not process resident set size. This is misleading for readers comparing `memory_map=True/False`.
##########
docs/source/python/parquet/parquet_type_handling.rst:
##########
@@ -0,0 +1,156 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. currentmodule:: pyarrow
+.. _parquet_type_handling:
+
+
+Data Type Handling
+==================
+
+Reading types as DictionaryArray
+--------------------------------
+
+The ``read_dictionary`` option in ``read_table`` and ``ParquetDataset`` will
+cause columns to be read as ``DictionaryArray``, which will become
+``pandas.Categorical`` when converted to pandas. This option is only valid for
+string and binary column types, and it can yield significantly lower memory use
+and improved performance for columns with many repeated string values.
+
+.. code-block:: python
+
+ >>> import pyarrow as pa
+ >>> import pyarrow.parquet as pq
+
+ >>> table = pa.table({'one': [-1, None, 2.5],
+ ... 'two': ['foo', 'bar', 'baz'],
+ ... 'three': [True, False, True]})
+ ...
+ >>> pq.write_table(table, 'example.parquet')
+
+ >>> pq.read_table('example.parquet', read_dictionary=['two'])
+ pyarrow.Table
+ one: double
+ two: dictionary<values=string, indices=int32, ordered=0>
+ three: bool
+ ----
+ one: [[-1,null,2.5]]
+ two: [ -- dictionary:
+ ["foo","bar","baz"] -- indices:
+ [0,1,2]]
+ three: [[true,false,true]]
+
+Reading binary and list columns
+-------------------------------
+
+By default, Parquet ``BYTE_ARRAY`` columns are read as Arrow ``binary`` type
+and Parquet ``LIST`` columns are read as Arrow ``list`` type, both of which use
+32-bit offsets. For very large datasets this can overflow. The ``binary_type``
+and ``list_type`` parameters let you choose a different Arrow type on read.
+
+``binary_type`` accepts ``pa.binary()`` (default), ``pa.large_binary()``, or
+``pa.binary_view()``:
+
+.. code-block:: python
+
+ >>> import pyarrow as pa
+ >>> table = pa.table({'data': pa.array([b'hello', b'world'], pa.binary())})
+ >>> pq.write_table(table, 'binary.parquet', store_schema=False)
+ >>> pq.read_table('binary.parquet', binary_type=pa.large_binary()).schema
+ data: large_binary
+
+``list_type`` accepts ``pa.ListType`` or ``pa.LargeListType``:
+
+.. code-block:: python
+
+ >>> table = pa.table({'lists': pa.array([[1, 2], [3]], pa.list_(pa.int32()))})
+ >>> pq.write_table(table, 'lists.parquet', store_schema=False)
+ >>> pq.read_table('lists.parquet', list_type=pa.LargeListType).schema
+ lists: large_list<element: int32>
+ child 0, element: int32
+
+.. note::
+ Both settings are ignored when a serialized Arrow schema is present in the
+ Parquet file metadata (i.e. when the parquet file was written with
+ ``store_schema=True``).
+
+Read in Arrow Extension Types
+-----------------------------
+
+Certain Parquet logical types (JSON, UUID, Geometry, Geography) are supported
+and read as Arrow extension types by default (``arrow.json``, ``arrow.uuid``
+and ``geoarrow.wkb`` respectively). This support is enabled via the
+``arrow_extensions_enabled`` parameter and is used in :func:`~pyarrow.parquet.read_table`,
+:class:`~pyarrow.parquet.ParquetFile`, :class:`~pyarrow.parquet.ParquetDataset`, and
+:func:`~pyarrow.parquet.read_schema`.
+
+To read these Parquet logical types as storage types (default behavior
+until PyArrow version ``21.0.0``), set ``arrow_extensions_enabled=False``.
+
+.. note::
+ Reading GEOMETRY/GEOGRAPHY columns as ``geoarrow.wkb`` additionally
+ requires the ``geoarrow.wkb`` extension type to be registered. For that
+ you can install Python bindings for GeoArrow
+ (`geoarrow-pyarrow <https://geoarrow.org/geoarrow-python/main/index.html#installation>`_)
+ and import ``geoarrow.pyarrow`` module.
+
+Storing timestamps
+------------------
+
+Some Parquet readers may only support timestamps stored in millisecond
+(``'ms'``) or microsecond (``'us'``) resolution. Since pandas uses nanoseconds
+to represent timestamps, this can occasionally be a nuisance. By default
+(when writing version 1.0 Parquet files), the nanoseconds will be cast to
+microseconds ('us').
Review Comment:
Minor formatting consistency: use literal markup for the `'us'` unit like the surrounding `'ms'` examples (currently it's plain text in parentheses).
##########
docs/source/python/parquet/parquet.rst:
##########
@@ -0,0 +1,605 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. currentmodule:: pyarrow
+.. _parquet_single_file:
+
+
+Reading and Writing Single Files
+================================
+
+The functions :func:`~.parquet.read_table` and :func:`~.parquet.write_table`
+read and write the :ref:`pyarrow.Table <data.table>` object, respectively.
+
+Let's look at a simple table:
+
+.. code-block:: python
+
+ >>> import numpy as np
+ >>> import pandas as pd
+ >>> import pyarrow as pa
+ >>> df = pd.DataFrame({'one': [-1, np.nan, 2.5],
+ ... 'two': ['foo', 'bar', 'baz'],
+ ... 'three': [True, False, True]},
+ ... index=list('abc'))
+ >>> table = pa.Table.from_pandas(df)
+
+We write this to Parquet format with ``write_table``:
+
+.. code-block:: python
+
+ >>> import pyarrow.parquet as pq
+ >>> pq.write_table(table, 'example.parquet')
+
+This creates a single Parquet file. In practice, a Parquet dataset may consist
+of many files in many directories. We can read a single file back with
+``read_table``:
+
+.. code-block:: python
+
+ >>> table2 = pq.read_table('example.parquet')
+ >>> table2.to_pandas()
+ one two three
+ a -1.0 foo True
+ b NaN bar False
+ c 2.5 baz True
+
+You can pass a subset of columns to read, which can be much faster than reading
+the whole file (due to the columnar layout):
+
+.. code-block:: python
+
+ >>> pq.read_table('example.parquet', columns=['one', 'three'])
+ pyarrow.Table
+ one: double
+ three: bool
+ ----
+ one: [[-1,null,2.5]]
+ three: [[true,false,true]]
+
+When reading a subset of columns from a file that used a Pandas dataframe as the
+source, we use ``read_pandas`` to maintain any additional index column data:
+
+.. code-block:: python
+
+ >>> pq.read_pandas('example.parquet', columns=['two']).to_pandas()
+ two
+ a foo
+ b bar
+ c baz
+
+We do not need to use a string to specify the origin of the file. It can be any of:
+
+* A file path as a string
+* A :ref:`NativeFile <io.native_file>` from PyArrow
+* A Python file object
+
+In general, a Python file object will have the worst read performance, while a
+string file path or an instance of :class:`~.NativeFile` (especially memory
+maps) will perform the best.
+
+.. _parquet_mmap:
+
+Reading Parquet and Memory Mapping
+----------------------------------
+
+Because Parquet data needs to be decoded from the Parquet format
+and compression, it can't be directly mapped from disk.
+Thus the ``memory_map`` option might perform better on some systems
+but won't help much with resident memory consumption.
+
+.. code-block:: python
+
+ >>> pq_array = pq.read_table(path, memory_map=True) # doctest: +SKIP
+ >>> print("RSS: {}MB".format(pa.total_allocated_bytes() >> 20)) # doctest: +SKIP
+ RSS: 4299MB
+
+ >>> pq_array = pq.read_table(path, memory_map=False) # doctest: +SKIP
+ >>> print("RSS: {}MB".format(pa.total_allocated_bytes() >> 20)) # doctest: +SKIP
+ RSS: 4299MB
Review Comment:
Same as above: the second print statement also labels Arrow-allocated bytes as "RSS".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47686: [Docs][Python] Split the Python Parquet docs into separate items [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #50419:
URL: https://github.com/apache/arrow/pull/50419#issuecomment-5025149348
@github-actions crossbow submit preview-docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47686: [Docs][Python] Split the Python Parquet docs into separate items [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #50419:
URL: https://github.com/apache/arrow/pull/50419#issuecomment-5265750622
Rendered docs available here: https://s3.amazonaws.com/arrow-data/pr_docs/50419/python/parquet/index.html
I will keep this PR open for a week before merging, in case there are any more comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47686: [Docs][Python] Split the Python Parquet docs into separate items [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #50419:
URL: https://github.com/apache/arrow/pull/50419#issuecomment-4924022948
Revision: 156850c96915e18aa0a926cedd02194c4d99c245
Submitted crossbow builds: [ursacomputing/crossbow @ actions-937dad9db2](https://github.com/ursacomputing/crossbow/branches/all?query=actions-937dad9db2)
|Task|Status|
|----|------|
|preview-docs|[](https://github.com/ursacomputing/crossbow/actions/runs/29011210680/job/86095025094)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47686: [Docs][Python] Split the Python Parquet docs into separate items [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #50419:
URL: https://github.com/apache/arrow/pull/50419#issuecomment-5342451443
@thisisnic would you have some time to look through the general splitting of the Parquet section in Python User Guide? This change has been done in [commit 1](https://github.com/apache/arrow/pull/50419/changes/774f0a95014c4a0dc1b1f46ccb178c9d3665bd82). The preview can be found on the following link: https://s3.amazonaws.com/arrow-data/pr_docs/50419/python/parquet/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47686: [Docs][Python] Split the Python Parquet docs into separate items [arrow]
Posted by "adamreeve (via GitHub)" <gi...@apache.org>.
adamreeve commented on code in PR #50419:
URL: https://github.com/apache/arrow/pull/50419#discussion_r3575171603
##########
docs/source/python/parquet/parquet.rst:
##########
@@ -0,0 +1,605 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. currentmodule:: pyarrow
+.. _parquet_single_file:
+
+
+Reading and Writing Single Files
+================================
+
+The functions :func:`~.parquet.read_table` and :func:`~.parquet.write_table`
+read and write the :ref:`pyarrow.Table <data.table>` object, respectively.
+
+Let's look at a simple table:
+
+.. code-block:: python
+
+ >>> import numpy as np
+ >>> import pandas as pd
Review Comment:
These examples use Pandas where plain PyArrow could be easily used, which might give the impression to users that they need to use Pandas when reading or writing Parquet. I can see these are existing examples that have been moved though, so this isn't really related to this PR. Maybe as a follow up we should look at using pure PyArrow in more places, except where we are explicitly documenting Pandas integration?
##########
docs/source/python/parquet/parquet_encryption.rst:
##########
@@ -0,0 +1,280 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. currentmodule:: pyarrow
+.. _parquet_encryption:
+
+
+Parquet Modular Encryption (Columnar Encryption)
+------------------------------------------------
+
+Columnar encryption is supported for Parquet files in C++ starting from
+Apache Arrow 4.0.0 and in PyArrow starting from Apache Arrow 6.0.0.
+
+Parquet uses the envelope encryption practice, where file parts are encrypted
+with "data encryption keys" (DEKs), and the DEKs are encrypted with "master
+encryption keys" (MEKs). The DEKs are randomly generated by Parquet for each
+encrypted file/column. The MEKs are generated, stored and managed in a Key
+Management Service (KMS) of user’s choice.
+
+Reading and writing encrypted Parquet files involves passing file encryption
+and decryption properties to :class:`~pyarrow.parquet.ParquetWriter` and to
+:class:`~.ParquetFile`, respectively.
+
+Writing an encrypted Parquet file:
+
+.. code-block:: python
+
+ >>> import pyarrow.parquet as pq
+ >>> encryption_properties = crypto_factory.file_encryption_properties( # doctest: +SKIP
+ ... kms_connection_config, encryption_config)
+ >>> with pq.ParquetWriter(filename, schema, # doctest: +SKIP
+ ... encryption_properties=encryption_properties) as writer:
+ ... writer.write_table(table)
+
+Reading an encrypted Parquet file:
+
+.. code-block:: python
+
+ >>> decryption_properties = crypto_factory.file_decryption_properties( # doctest: +SKIP
+ ... kms_connection_config)
+ >>> parquet_file = pq.ParquetFile(filename, # doctest: +SKIP
+ ... decryption_properties=decryption_properties)
+
+
+In order to create the encryption and decryption properties, a
+:class:`pyarrow.parquet.encryption.CryptoFactory` should be created and
+initialized with KMS Client details, as described below.
+
+
+KMS Client
+~~~~~~~~~~
+
+The master encryption keys should be kept and managed in a production-grade
+Key Management System (KMS), deployed in the user's organization. Using Parquet
+encryption requires implementation of a client class for the KMS server.
+Any KmsClient implementation should implement the informal interface
+defined by :class:`pyarrow.parquet.encryption.KmsClient` as following:
+
+.. code-block:: python
+
+ >>> import pyarrow.parquet.encryption as pe
+ >>> class MyKmsClient(pe.KmsClient):
+ ...
+ ... """An example KmsClient implementation skeleton"""
+ ... def __init__(self, kms_connection_configuration):
+ ... pe.KmsClient.__init__(self)
+ ... # Any KMS-specific initialization based on
+ ... # kms_connection_configuration comes here
+ ...
+ ... def wrap_key(self, key_bytes, master_key_identifier):
+ ... wrapped_key = ... # call KMS to wrap key_bytes with key specified by
+ ... # master_key_identifier
+ ... return wrapped_key
+ ...
+ ... def unwrap_key(self, wrapped_key, master_key_identifier):
+ ... key_bytes = ... # call KMS to unwrap wrapped_key with key specified by
+ ... # master_key_identifier
+ ... return key_bytes
+
+The concrete implementation will be loaded at runtime by a factory function
+provided by the user. This factory function will be used to initialize the
+:class:`pyarrow.parquet.encryption.CryptoFactory` for creating file encryption
+and decryption properties.
+
+For example, in order to use the ``MyKmsClient`` defined above:
+
+.. code-block:: python
+
+ >>> def kms_client_factory(kms_connection_configuration):
+ ... return MyKmsClient(kms_connection_configuration)
+
+ >>> crypto_factory = pe.CryptoFactory(kms_client_factory)
+
+An :download:`example <../../../python/examples/parquet_encryption/sample_vault_kms_client.py>`
+of such a class for an open source
+`KMS <https://www.vaultproject.io/api/secret/transit>`_ can be found in the Apache
+Arrow GitHub repository. The production KMS client should be designed in
+cooperation with an organization's security administrators, and built by
+developers with experience in access control management. Once such a class is
+created, it can be passed to applications via a factory method and leveraged
+by general PyArrow users as shown in the encrypted parquet write/read sample
+above.
+
+KMS connection configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Configuration of connection to KMS (:class:`pyarrow.parquet.encryption.KmsConnectionConfig`
+used when creating file encryption and decryption properties) includes the
+following options:
+
+* ``kms_instance_url``, URL of the KMS instance.
+* ``kms_instance_id``, ID of the KMS instance that will be used for encryption
+ (if multiple KMS instances are available).
+* ``key_access_token``, authorization token that will be passed to KMS.
+* ``custom_kms_conf``, a string dictionary with KMS-type-specific configuration.
+
+Encryption configuration
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+:class:`pyarrow.parquet.encryption.EncryptionConfiguration` (used when
+creating file encryption properties) includes the following options:
+
+* ``footer_key``, the ID of the master key for footer encryption/signing.
+* ``column_keys``, which columns to encrypt with which key. Dictionary with
+ master key IDs as the keys, and column name lists as the values,
+ e.g. ``{key1: [col1, col2], key2: [col3]}``. See notes on nested fields below.
+* ``uniform_encryption``, whether to encrypt the footer and all columns with
+ the same ``footer_key``, instead of specifying ``column_keys``
+ individually. Cannot be used together with ``column_keys``.
+* ``encryption_algorithm``, the Parquet encryption algorithm.
+ Can be ``AES_GCM_V1`` (default) or ``AES_GCM_CTR_V1``.
+* ``plaintext_footer``, whether to write the file footer in plain text (otherwise it is encrypted).
+* ``double_wrapping``, whether to use double wrapping - where data encryption keys (DEKs)
+ are encrypted with key encryption keys (KEKs), which in turn are encrypted
+ with master encryption keys (MEKs). If set to ``false``, single wrapping is
+ used - where DEKs are encrypted directly with MEKs.
+* ``cache_lifetime``, the lifetime of cached entities (key encryption keys,
+ local wrapping keys, KMS client objects) represented as a ``datetime.timedelta``.
+* ``internal_key_material``, whether to store key material inside Parquet file footers;
+ this mode doesn’t produce additional files. If set to ``false``, key material is
+ stored in separate files in the same folder, which enables key rotation for
+ immutable Parquet files.
+* ``data_key_length_bits``, the length of data encryption keys (DEKs), randomly
+ generated by Parquet key management tools. Can be 128, 192 or 256 bits.
+
+.. note::
+ When ``double_wrapping`` is true, Parquet implements a "double envelope
+ encryption" mode that minimizes the interaction of the program with a KMS
+ server. In this mode, the DEKs are encrypted with "key encryption keys"
+ (KEKs, randomly generated by Parquet). The KEKs are encrypted with "master
+ encryption keys" (MEKs) in the KMS; the result and the KEK itself are
+ cached in the process memory.
+
+An example encryption configuration:
+
+.. code-block:: python
+
+ >>> encryption_config = pe.EncryptionConfiguration(
+ ... footer_key="footer_key_name",
+ ... column_keys={
+ ... "column_key_name": ["Column1", "Column2"],
+ ... },
+ ... )
+
+.. note::
+
+ Columns with nested fields (struct or map data types) can be encrypted as a whole, or only
+ individual fields. Configure an encryption key for the root column name to encrypt all nested
+ fields with this key, or configure a key for individual leaf nested fields.
+
+ Conventionally, the key and value fields of a map column ``m`` have the names
+ ``m.key_value.key`` and ``m.key_value.value``, respectively.
+ An inner field ``f`` of a struct column ``s`` has the name ``s.f``.
+
+ With above example, *all* inner fields are encrypted with the same key by configuring that key
+ for column ``m`` and ``s``, respectively.
+
+An example encryption configuration for columns with nested fields, where
+all columns are encrypted with the same key identified by ``column_key_id``:
+
+.. code-block:: python
+
+ >>> import pyarrow as pa
+ >>> schema = pa.schema([
+ ... ("MapColumn", pa.map_(pa.string(), pa.int32())),
+ ... ("StructColumn", pa.struct([("f1", pa.int32()), ("f2", pa.string())])),
+ ... ])
+
+ >>> encryption_config = pe.EncryptionConfiguration(
+ ... footer_key="footer_key_name",
+ ... column_keys={
+ ... "column_key_id": [ "MapColumn", "StructColumn" ],
+ ... },
+ ... )
+
+An example encryption configuration for columns with nested fields, where
+some inner fields are encrypted with the same key identified by ``column_key_id``:
+
+.. code-block:: python
+
+ >>> schema = pa.schema([
+ ... ("MapColumn", pa.map_(pa.string(), pa.int32())),
+ ... ("StructColumn", pa.struct([("f1", pa.int32()), ("f2", pa.string())])),
+ ... ])
+
+ >>> encryption_config = pe.EncryptionConfiguration(
+ ... footer_key="footer_key_name",
+ ... column_keys={
+ ... "column_key_id": [ "MapColumn.key_value.value", "StructColumn.f1" ],
+ ... },
+ ... )
+
+Decryption configuration
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+:class:`pyarrow.parquet.encryption.DecryptionConfiguration` (used when creating
+file decryption properties) is optional and it includes the following options:
+
+* ``cache_lifetime``, the lifetime of cached entities (key encryption keys, local
+ wrapping keys, KMS client objects) represented as a ``datetime.timedelta``.
+
+External key material and key rotation
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+When ``internal_key_material=False`` is set on ``EncryptionConfiguration``,
+key material is stored in a separate file next to the Parquet file instead of
+in its footer.
+
+Storing key material externally is what enables key rotation:
+``crypto_factory.rotate_master_keys()`` re-wraps the data encryption keys of
Review Comment:
This should be a cross-reference to the method documentation:
```suggestion
:meth:`~pyarrow.parquet.encryption.CryptoFactory.rotate_master_keys` re-wraps the data encryption keys of
```
##########
docs/source/python/parquet/parquet_encryption.rst:
##########
@@ -0,0 +1,280 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. currentmodule:: pyarrow
+.. _parquet_encryption:
+
+
+Parquet Modular Encryption (Columnar Encryption)
+------------------------------------------------
+
+Columnar encryption is supported for Parquet files in C++ starting from
+Apache Arrow 4.0.0 and in PyArrow starting from Apache Arrow 6.0.0.
+
+Parquet uses the envelope encryption practice, where file parts are encrypted
+with "data encryption keys" (DEKs), and the DEKs are encrypted with "master
+encryption keys" (MEKs). The DEKs are randomly generated by Parquet for each
+encrypted file/column. The MEKs are generated, stored and managed in a Key
+Management Service (KMS) of user’s choice.
+
+Reading and writing encrypted Parquet files involves passing file encryption
+and decryption properties to :class:`~pyarrow.parquet.ParquetWriter` and to
+:class:`~.ParquetFile`, respectively.
+
+Writing an encrypted Parquet file:
+
+.. code-block:: python
+
+ >>> import pyarrow.parquet as pq
+ >>> encryption_properties = crypto_factory.file_encryption_properties( # doctest: +SKIP
+ ... kms_connection_config, encryption_config)
+ >>> with pq.ParquetWriter(filename, schema, # doctest: +SKIP
+ ... encryption_properties=encryption_properties) as writer:
+ ... writer.write_table(table)
+
+Reading an encrypted Parquet file:
+
+.. code-block:: python
+
+ >>> decryption_properties = crypto_factory.file_decryption_properties( # doctest: +SKIP
+ ... kms_connection_config)
+ >>> parquet_file = pq.ParquetFile(filename, # doctest: +SKIP
+ ... decryption_properties=decryption_properties)
+
+
+In order to create the encryption and decryption properties, a
+:class:`pyarrow.parquet.encryption.CryptoFactory` should be created and
+initialized with KMS Client details, as described below.
+
+
+KMS Client
+~~~~~~~~~~
+
+The master encryption keys should be kept and managed in a production-grade
+Key Management System (KMS), deployed in the user's organization. Using Parquet
+encryption requires implementation of a client class for the KMS server.
+Any KmsClient implementation should implement the informal interface
+defined by :class:`pyarrow.parquet.encryption.KmsClient` as following:
+
+.. code-block:: python
+
+ >>> import pyarrow.parquet.encryption as pe
+ >>> class MyKmsClient(pe.KmsClient):
+ ...
+ ... """An example KmsClient implementation skeleton"""
+ ... def __init__(self, kms_connection_configuration):
+ ... pe.KmsClient.__init__(self)
+ ... # Any KMS-specific initialization based on
+ ... # kms_connection_configuration comes here
+ ...
+ ... def wrap_key(self, key_bytes, master_key_identifier):
+ ... wrapped_key = ... # call KMS to wrap key_bytes with key specified by
+ ... # master_key_identifier
+ ... return wrapped_key
+ ...
+ ... def unwrap_key(self, wrapped_key, master_key_identifier):
+ ... key_bytes = ... # call KMS to unwrap wrapped_key with key specified by
+ ... # master_key_identifier
+ ... return key_bytes
+
+The concrete implementation will be loaded at runtime by a factory function
+provided by the user. This factory function will be used to initialize the
+:class:`pyarrow.parquet.encryption.CryptoFactory` for creating file encryption
+and decryption properties.
+
+For example, in order to use the ``MyKmsClient`` defined above:
+
+.. code-block:: python
+
+ >>> def kms_client_factory(kms_connection_configuration):
+ ... return MyKmsClient(kms_connection_configuration)
+
+ >>> crypto_factory = pe.CryptoFactory(kms_client_factory)
+
+An :download:`example <../../../python/examples/parquet_encryption/sample_vault_kms_client.py>`
+of such a class for an open source
+`KMS <https://www.vaultproject.io/api/secret/transit>`_ can be found in the Apache
+Arrow GitHub repository. The production KMS client should be designed in
+cooperation with an organization's security administrators, and built by
+developers with experience in access control management. Once such a class is
+created, it can be passed to applications via a factory method and leveraged
+by general PyArrow users as shown in the encrypted parquet write/read sample
+above.
+
+KMS connection configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Configuration of connection to KMS (:class:`pyarrow.parquet.encryption.KmsConnectionConfig`
+used when creating file encryption and decryption properties) includes the
+following options:
+
+* ``kms_instance_url``, URL of the KMS instance.
+* ``kms_instance_id``, ID of the KMS instance that will be used for encryption
+ (if multiple KMS instances are available).
+* ``key_access_token``, authorization token that will be passed to KMS.
+* ``custom_kms_conf``, a string dictionary with KMS-type-specific configuration.
+
+Encryption configuration
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+:class:`pyarrow.parquet.encryption.EncryptionConfiguration` (used when
+creating file encryption properties) includes the following options:
+
+* ``footer_key``, the ID of the master key for footer encryption/signing.
+* ``column_keys``, which columns to encrypt with which key. Dictionary with
+ master key IDs as the keys, and column name lists as the values,
+ e.g. ``{key1: [col1, col2], key2: [col3]}``. See notes on nested fields below.
+* ``uniform_encryption``, whether to encrypt the footer and all columns with
+ the same ``footer_key``, instead of specifying ``column_keys``
+ individually. Cannot be used together with ``column_keys``.
+* ``encryption_algorithm``, the Parquet encryption algorithm.
+ Can be ``AES_GCM_V1`` (default) or ``AES_GCM_CTR_V1``.
+* ``plaintext_footer``, whether to write the file footer in plain text (otherwise it is encrypted).
+* ``double_wrapping``, whether to use double wrapping - where data encryption keys (DEKs)
+ are encrypted with key encryption keys (KEKs), which in turn are encrypted
+ with master encryption keys (MEKs). If set to ``false``, single wrapping is
+ used - where DEKs are encrypted directly with MEKs.
+* ``cache_lifetime``, the lifetime of cached entities (key encryption keys,
+ local wrapping keys, KMS client objects) represented as a ``datetime.timedelta``.
+* ``internal_key_material``, whether to store key material inside Parquet file footers;
+ this mode doesn’t produce additional files. If set to ``false``, key material is
+ stored in separate files in the same folder, which enables key rotation for
+ immutable Parquet files.
+* ``data_key_length_bits``, the length of data encryption keys (DEKs), randomly
+ generated by Parquet key management tools. Can be 128, 192 or 256 bits.
+
+.. note::
+ When ``double_wrapping`` is true, Parquet implements a "double envelope
+ encryption" mode that minimizes the interaction of the program with a KMS
+ server. In this mode, the DEKs are encrypted with "key encryption keys"
+ (KEKs, randomly generated by Parquet). The KEKs are encrypted with "master
+ encryption keys" (MEKs) in the KMS; the result and the KEK itself are
+ cached in the process memory.
+
+An example encryption configuration:
+
+.. code-block:: python
+
+ >>> encryption_config = pe.EncryptionConfiguration(
+ ... footer_key="footer_key_name",
+ ... column_keys={
+ ... "column_key_name": ["Column1", "Column2"],
+ ... },
+ ... )
+
+.. note::
+
+ Columns with nested fields (struct or map data types) can be encrypted as a whole, or only
+ individual fields. Configure an encryption key for the root column name to encrypt all nested
+ fields with this key, or configure a key for individual leaf nested fields.
+
+ Conventionally, the key and value fields of a map column ``m`` have the names
+ ``m.key_value.key`` and ``m.key_value.value``, respectively.
+ An inner field ``f`` of a struct column ``s`` has the name ``s.f``.
+
+ With above example, *all* inner fields are encrypted with the same key by configuring that key
+ for column ``m`` and ``s``, respectively.
+
+An example encryption configuration for columns with nested fields, where
+all columns are encrypted with the same key identified by ``column_key_id``:
+
+.. code-block:: python
+
+ >>> import pyarrow as pa
+ >>> schema = pa.schema([
+ ... ("MapColumn", pa.map_(pa.string(), pa.int32())),
+ ... ("StructColumn", pa.struct([("f1", pa.int32()), ("f2", pa.string())])),
+ ... ])
+
+ >>> encryption_config = pe.EncryptionConfiguration(
+ ... footer_key="footer_key_name",
+ ... column_keys={
+ ... "column_key_id": [ "MapColumn", "StructColumn" ],
+ ... },
+ ... )
+
+An example encryption configuration for columns with nested fields, where
+some inner fields are encrypted with the same key identified by ``column_key_id``:
+
+.. code-block:: python
+
+ >>> schema = pa.schema([
+ ... ("MapColumn", pa.map_(pa.string(), pa.int32())),
+ ... ("StructColumn", pa.struct([("f1", pa.int32()), ("f2", pa.string())])),
+ ... ])
+
+ >>> encryption_config = pe.EncryptionConfiguration(
+ ... footer_key="footer_key_name",
+ ... column_keys={
+ ... "column_key_id": [ "MapColumn.key_value.value", "StructColumn.f1" ],
+ ... },
+ ... )
+
+Decryption configuration
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+:class:`pyarrow.parquet.encryption.DecryptionConfiguration` (used when creating
+file decryption properties) is optional and it includes the following options:
+
+* ``cache_lifetime``, the lifetime of cached entities (key encryption keys, local
+ wrapping keys, KMS client objects) represented as a ``datetime.timedelta``.
+
+External key material and key rotation
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+When ``internal_key_material=False`` is set on ``EncryptionConfiguration``,
+key material is stored in a separate file next to the Parquet file instead of
+in its footer.
+
+Storing key material externally is what enables key rotation:
+``crypto_factory.rotate_master_keys()`` re-wraps the data encryption keys of
+a file that uses external key material under new master keys, without rewriting
+the file itself:
Review Comment:
I think this could be clarified a bit:
```suggestion
a file that uses external key material using new master keys and overwrites the external
key material file, without changing the Parquet file itself:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47686: [Docs][Python] Split the Python Parquet docs into separate items [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #50419:
URL: https://github.com/apache/arrow/pull/50419#issuecomment-5263524140
@github-actions crossbow submit preview-docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47686: [Docs][Python] Split the Python Parquet docs into separate items [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #50419:
URL: https://github.com/apache/arrow/pull/50419#issuecomment-5023355722
Thank you all for the reviews!
I still need to go through the Copilot review comments, but I have applied other review comment suggestions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47686: [Docs][Python] Split the Python Parquet docs into separate items [arrow]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #50419:
URL: https://github.com/apache/arrow/pull/50419#discussion_r3615286194
##########
docs/source/python/parquet/parquet_encryption.rst:
##########
@@ -0,0 +1,280 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. currentmodule:: pyarrow
+.. _parquet_encryption:
+
+
+Parquet Modular Encryption (Columnar Encryption)
+================================================
+
+Columnar encryption is supported for Parquet files in C++ starting from
+Apache Arrow 4.0.0 and in PyArrow starting from Apache Arrow 6.0.0.
+
+Parquet uses the envelope encryption practice, where file parts are encrypted
+with "data encryption keys" (DEKs), and the DEKs are encrypted with "master
+encryption keys" (MEKs). The DEKs are randomly generated by Parquet for each
+encrypted file/column. The MEKs are generated, stored and managed in a Key
+Management Service (KMS) of user’s choice.
+
+Reading and writing encrypted Parquet files involves passing file encryption
+and decryption properties to :class:`~pyarrow.parquet.ParquetWriter` and to
+:class:`~.ParquetFile`, respectively.
Review Comment:
`:class:`~.ParquetFile`` resolves to `pyarrow.ParquetFile` because of `.. currentmodule:: pyarrow`, but `ParquetFile` lives in `pyarrow.parquet`. This will produce a broken API link in the rendered docs; reference the fully-qualified class instead.
##########
docs/source/python/parquet/parquet_datasets.rst:
##########
@@ -0,0 +1,251 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. currentmodule:: pyarrow
+.. _parquet_datasets:
+
+
+Partitioned Datasets (Multiple Files)
+=====================================
+
+Multiple Parquet files constitute a Parquet *dataset*. These may be present in a
+number of ways:
+
+* A list of Parquet absolute file paths
+* A directory name containing nested directories defining a partitioned dataset
+
+A dataset partitioned by year and month may look like on disk:
+
+.. code-block:: text
+
+ dataset_name/
+ year=2007/
+ month=01/
+ 0.parq
+ 1.parq
+ ...
+ month=02/
+ 0.parq
+ 1.parq
+ ...
+ month=03/
+ ...
+ year=2008/
+ month=01/
+ ...
+ ...
+
+Writing to Partitioned Datasets
+-------------------------------
+
+You can write a partitioned dataset for any ``pyarrow`` file system that is a
+file-store (e.g. local, HDFS, S3). The default behaviour when no filesystem is
+added is to use the local filesystem.
+
+.. code-block:: python
+
+ >>> import pyarrow as pa
+ >>> import pyarrow.parquet as pq
+
+ >>> table = pa.table({'one': [-1, None, 2.5],
+ ... 'two': ['foo', 'bar', 'baz'],
+ ... 'three': [True, False, True]})
+ ...
+
+ >>> # Local dataset write
+ >>> pq.write_to_dataset(table, root_path='dataset_name',
+ ... partition_cols=['one', 'two'])
+
+The root path in this case specifies the parent directory to which data will be
+saved. The partition columns are the column names by which to partition the
+dataset. Columns are partitioned in the order they are given. The partition
+splits are determined by the unique values in the partition columns.
+
+To use another filesystem you only need to add the filesystem parameter, the
+individual table writes are wrapped using ``with`` statements so the
+``pq.write_to_dataset`` function does not need to be.
+
+.. code-block:: python
+
+ >>> # Remote file-system example
+ >>> from pyarrow.fs import HadoopFileSystem # doctest: +SKIP
+ >>> fs = HadoopFileSystem(host, port, user=user, kerb_ticket=ticket_cache_path) # doctest: +SKIP
+ >>> pq.write_to_dataset(table, root_path='dataset_name', # doctest: +SKIP
+ ... partition_cols=['one', 'two'], filesystem=fs)
+
+Compatibility Note: if using ``pq.write_to_dataset`` to create a table that
+will then be used by HIVE then partition column values must be compatible with
+the allowed character set of the HIVE version you are running.
+
+Writing ``_metadata`` and ``_common_metadata`` files
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Some processing frameworks such as Spark or Dask (optionally) use ``_metadata``
+and ``_common_metadata`` files with partitioned datasets.
+
+Those files include information about the schema of the full dataset (for
+``_common_metadata``) and potentially all row group metadata of all files in the
+partitioned dataset as well (for ``_metadata``). The actual files are
+metadata-only Parquet files. Note this is not a Parquet standard, but a
+convention set in practice by those frameworks.
+
+Using those files can give a more efficient creation of a parquet Dataset,
+since it can use the stored schema and file paths of all row groups,
+instead of inferring the schema and crawling the directories for all Parquet
+files (this is especially the case for filesystems where accessing files
+is expensive).
+
+The :func:`~pyarrow.parquet.write_to_dataset` function does not automatically
+write such metadata files, but you can use it to gather the metadata and
+combine and write them manually:
+
+.. code-block:: python
+
+ >>> # Write a dataset and collect metadata information of all written files
+ >>> metadata_collector = []
+ >>> root_path = "dataset_name_1"
+ >>> pq.write_to_dataset(table, root_path, metadata_collector=metadata_collector)
+
+ >>> # Write the ``_common_metadata`` parquet file without row groups statistics
+ >>> pq.write_metadata(table.schema, root_path + '/_common_metadata')
+
+ >>> # Write the ``_metadata`` parquet file with row groups statistics of all files
+ >>> pq.write_metadata(
+ ... table.schema, root_path + '/_metadata',
+ ... metadata_collector=metadata_collector
+ ... )
+
+When not using the :func:`~pyarrow.parquet.write_to_dataset` function, but
+writing the individual files of the partitioned dataset using
+:func:`~pyarrow.parquet.write_table` or :class:`~pyarrow.parquet.ParquetWriter`,
+the ``metadata_collector`` keyword can also be used to collect the FileMetaData
+of the written files. In this case, you need to ensure to set the file path
+contained in the row group metadata yourself before combining the metadata, and
+the schemas of all different files and collected FileMetaData objects should be
+the same:
+
+.. code-block:: python
+
+ >>> import os
+ >>> os.mkdir("year=2017")
+
+ >>> metadata_collector = []
+ >>> pq.write_table(
+ ... table, "year=2017/data1.parquet",
+ ... metadata_collector=metadata_collector
+ ... )
+
+ >>> # set the file path relative to the root of the partitioned dataset
+ >>> metadata_collector[-1].set_file_path("year=2017/data1.parquet")
+
+ >>> # combine and write the metadata
+ >>> metadata = metadata_collector[0]
+ >>> for _meta in metadata_collector[1:]:
+ ... metadata.append_row_groups(_meta)
+ >>> metadata.write_metadata_file("_metadata")
+
+ >>> # or use pq.write_metadata to combine and write in a single step
+ >>> pq.write_metadata(
+ ... table.schema, "_metadata",
+ ... metadata_collector=metadata_collector
+ ... )
+
+ >>> pq.read_metadata("_metadata")
+ <pyarrow._parquet.FileMetaData object at ...>
+ created_by: parquet-cpp-arrow version ...
+ num_columns: 3
+ num_rows: 3
+ num_row_groups: 1
+ format_version: 2.6
+ serialized_size: ...
+
+Reading from Partitioned Datasets
+---------------------------------
+
+The :class:`~.ParquetDataset` class accepts either a directory name or a list
Review Comment:
`:class:`~.ParquetDataset`` resolves to `pyarrow.ParquetDataset` under `.. currentmodule:: pyarrow`, but `ParquetDataset` is in `pyarrow.parquet`. This will render as an unresolved/broken cross-reference.
##########
docs/source/python/parquet/index.rst:
##########
@@ -0,0 +1,89 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. currentmodule:: pyarrow
+.. _parquet:
+
+Reading and Writing the Apache Parquet Format
+=============================================
+
+The `Apache Parquet <http://parquet.apache.org/>`_ project provides a
+standardized open-source columnar storage format for use in data analysis
+systems. It was created originally for use in `Apache Hadoop
+<http://hadoop.apache.org/>`_ with systems like `Apache Drill
+<http://drill.apache.org>`_, `Apache Hive <http://hive.apache.org>`_, `Apache
+Impala <http://impala.apache.org>`_, and `Apache Spark
+<http://spark.apache.org>`_ adopting it as a shared standard for high
+performance data IO.
+
+Apache Arrow is an ideal in-memory transport layer for data that is being read
+or written with Parquet files. We have been concurrently developing the `C++
+implementation of
+Apache Parquet <https://github.com/apache/arrow/tree/main/cpp/tools/parquet>`_,
+which includes a native, multithreaded C++ adapter to and from in-memory Arrow
+data. PyArrow includes Python bindings to this code, which thus enables reading
+and writing Parquet files with pandas as well.
+
+Obtaining pyarrow with Parquet Support
+--------------------------------------
+
+If you installed ``pyarrow`` with pip or conda, it should be built with Parquet
+support bundled:
+
+.. code-block:: python
+
+ >>> import pyarrow.parquet as pq
+
+If you are building ``pyarrow`` from source, you must use ``-DARROW_PARQUET=ON``
+when compiling the C++ libraries and enable the Parquet extensions when
+building ``pyarrow``. If you want to use Parquet Encryption, then you must
+use ``-DPARQUET_REQUIRE_ENCRYPTION=ON`` too when compiling the C++ libraries.
+See the :ref:`Python Development <python-development>` page for more details.
+
+
+Reading and Writing Single Files
+--------------------------------
+
+.. toctree::
+ :maxdepth: 2
+
+ parquet
+
+Data Type Handling
+------------------
+
+.. toctree::
+ :maxdepth: 2
+
+ parquet_type_handling
+
Review Comment:
PR description says the Parquet docs are split into 4 subpages, but this toctree adds an additional dedicated page for type handling (`parquet_type_handling`). Please update the PR description (or adjust the split) so the described scope matches what will be published.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47686: [Docs][Python] Split the Python Parquet docs into separate items [arrow]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #50419:
URL: https://github.com/apache/arrow/pull/50419#discussion_r3615192219
##########
docs/source/python/parquet/parquet.rst:
##########
@@ -0,0 +1,605 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. currentmodule:: pyarrow
+.. _parquet_single_file:
+
+
+Reading and Writing Single Files
+================================
+
+The functions :func:`~.parquet.read_table` and :func:`~.parquet.write_table`
+read and write the :ref:`pyarrow.Table <data.table>` object, respectively.
+
+Let's look at a simple table:
+
+.. code-block:: python
+
+ >>> import numpy as np
+ >>> import pandas as pd
+ >>> import pyarrow as pa
+ >>> df = pd.DataFrame({'one': [-1, np.nan, 2.5],
+ ... 'two': ['foo', 'bar', 'baz'],
+ ... 'three': [True, False, True]},
+ ... index=list('abc'))
+ >>> table = pa.Table.from_pandas(df)
+
+We write this to Parquet format with ``write_table``:
+
+.. code-block:: python
+
+ >>> import pyarrow.parquet as pq
+ >>> pq.write_table(table, 'example.parquet')
+
+This creates a single Parquet file. In practice, a Parquet dataset may consist
+of many files in many directories. We can read a single file back with
+``read_table``:
+
+.. code-block:: python
+
+ >>> table2 = pq.read_table('example.parquet')
+ >>> table2.to_pandas()
+ one two three
+ a -1.0 foo True
+ b NaN bar False
+ c 2.5 baz True
+
+You can pass a subset of columns to read, which can be much faster than reading
+the whole file (due to the columnar layout):
+
+.. code-block:: python
+
+ >>> pq.read_table('example.parquet', columns=['one', 'three'])
+ pyarrow.Table
+ one: double
+ three: bool
+ ----
+ one: [[-1,null,2.5]]
+ three: [[true,false,true]]
+
+When reading a subset of columns from a file that used a Pandas dataframe as the
+source, we use ``read_pandas`` to maintain any additional index column data:
+
+.. code-block:: python
+
+ >>> pq.read_pandas('example.parquet', columns=['two']).to_pandas()
+ two
+ a foo
+ b bar
+ c baz
+
+We do not need to use a string to specify the origin of the file. It can be any of:
+
+* A file path as a string
+* A :ref:`NativeFile <io.native_file>` from PyArrow
+* A Python file object
+
+In general, a Python file object will have the worst read performance, while a
+string file path or an instance of :class:`~.NativeFile` (especially memory
+maps) will perform the best.
+
+.. _parquet_mmap:
+
+Reading Parquet and Memory Mapping
+----------------------------------
+
+Because Parquet data needs to be decoded from the Parquet format
+and compression, it can't be directly mapped from disk.
+Thus the ``memory_map`` option might perform better on some systems
+but won't help much with resident memory consumption.
+
+.. code-block:: python
+
+ >>> pq_array = pq.read_table(path, memory_map=True) # doctest: +SKIP
+ >>> print("RSS: {}MB".format(pa.total_allocated_bytes() >> 20)) # doctest: +SKIP
+ RSS: 4299MB
+
+ >>> pq_array = pq.read_table(path, memory_map=False) # doctest: +SKIP
+ >>> print("RSS: {}MB".format(pa.total_allocated_bytes() >> 20)) # doctest: +SKIP
+ RSS: 4299MB
+
+If you need to deal with Parquet data bigger than memory,
+the :ref:`dataset` and partitioning is probably what you are looking for.
Review Comment:
Grammar: the subject is plural ("dataset and partitioning"), so the verb should be plural as well.
##########
docs/source/python/parquet/parquet_encryption.rst:
##########
@@ -0,0 +1,280 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. currentmodule:: pyarrow
+.. _parquet_encryption:
+
+
+Parquet Modular Encryption (Columnar Encryption)
+------------------------------------------------
Review Comment:
This page title is marked as a lower-level heading ("-") instead of the top-level document title adornment used elsewhere in the Python docs ("="). Using a top-level title improves consistency and avoids odd TOC nesting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47686: [Docs][Python] Split the Python Parquet docs into separate items [arrow]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #50419:
URL: https://github.com/apache/arrow/pull/50419#discussion_r3551230211
##########
docs/source/python/parquet/parquet_datasets.rst:
##########
@@ -0,0 +1,251 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. currentmodule:: pyarrow
+.. _parquet_datasets:
+
+
+Partitioned Datasets (Multiple Files)
+=====================================
+
+Multiple Parquet files constitute a Parquet *dataset*. These may present in a
+number of ways:
Review Comment:
Grammar: "These may present" is missing "be" (should be "may be present").
##########
docs/source/python/parquet/parquet_type_handling.rst:
##########
@@ -0,0 +1,156 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. currentmodule:: pyarrow
+.. _parquet_type_handling:
+
+
+Data Type Handling
+==================
+
+Reading types as DictionaryArray
+--------------------------------
+
+The ``read_dictionary`` option in ``read_table`` and ``ParquetDataset`` will
+cause columns to be read as ``DictionaryArray``, which will become
+``pandas.Categorical`` when converted to pandas. This option is only valid for
+string and binary column types, and it can yield significantly lower memory use
+and improved performance for columns with many repeated string values.
+
+.. code-block:: python
+
+ >>> import pyarrow as pa
+ >>> import pyarrow.parquet as pq
+
+ >>> table = pa.table({'one': [-1, None, 2.5],
+ ... 'two': ['foo', 'bar', 'baz'],
+ ... 'three': [True, False, True]})
+ ...
+ >>> pq.write_table(table, 'example.parquet')
+
+ >>> pq.read_table('example.parquet', read_dictionary=['two'])
+ pyarrow.Table
+ one: double
+ two: dictionary<values=string, indices=int32, ordered=0>
+ three: bool
+ ----
+ one: [[-1,null,2.5]]
+ two: [ -- dictionary:
+ ["foo","bar","baz"] -- indices:
+ [0,1,2]]
+ three: [[true,false,true]]
+
+Reading binary and list columns
+-------------------------------
+
+By default, Parquet ``BYTE_ARRAY`` columns are read as Arrow ``binary`` type
+and Parquet ``LIST`` columns are read as Arrow ``list`` type, both of which use
+32-bit offsets. For very large datasets this can overflow. The ``binary_type``
+and ``list_type`` parameters let you choose a different Arrow type on read.
+
+``binary_type`` accepts ``pa.binary()`` (default), ``pa.large_binary()``, or
+``pa.binary_view()``:
+
+.. code-block:: python
+
+ >>> import pyarrow as pa
+ >>> table = pa.table({'data': pa.array([b'hello', b'world'], pa.binary())})
+ >>> pq.write_table(table, 'binary.parquet', store_schema=False)
+ >>> pq.read_table('binary.parquet', binary_type=pa.large_binary()).schema
+ data: large_binary
+
+``list_type`` accepts ``pa.ListType`` or ``pa.LargeListType``:
+
+.. code-block:: python
+
+ >>> table = pa.table({'lists': pa.array([[1, 2], [3]], pa.list_(pa.int32()))})
+ >>> pq.write_table(table, 'lists.parquet', store_schema=False)
+ >>> pq.read_table('lists.parquet', list_type=pa.LargeListType).schema
+ lists: large_list<element: int32>
+ child 0, element: int32
+
+.. note::
+ Both settings are ignored when a serialized Arrow schema is present in the
+ Parquet file metadata (i.e. when the parquet file was written with
+ ``store_schema=True``).
+
+Read in Arrow Extension Types
+-----------------------------
+
+Certain Parquet logical types (JSON, UUID, Geometry, Geography) are supported
+and read as Arrow extension types by default (``arrow.json``, ``arrow.uuid``
+and ``geoarrow.wkb`` respectively). This support is enabled via the
+``arrow_extensions_enabled`` parameter and is used in :func:`~pyarrow.parquet.read_table`,
+:class:`~pyarrow.parquet.ParquetFile`, :class:`~pyarrow.parquet.ParquetDataset`, and
+:func:`~pyarrow.parquet.read_schema`.
+
+To read these Parquet logical types as storage types (default behavior
+until PyArrow version ``21.0.0``), set ``arrow_extensions_enabled=False``.
+
+.. note::
+ Reading GEOMETRY/GEOGRAPHY columns as ``geoarrow.wkb`` additionally
+ requires the ``geoarrow.wkb`` extension type to be registered. For that
+ you can install Python bindings for GeoArrow
+ (`geoarrow-pyarrow <https://geoarrow.org/geoarrow-python/main/index.html#installation>`_)
+ and import ``geoarrow.pyarrow`` module.
+
+Storing timestamps
+------------------
+
+Some Parquet readers may only support timestamps stored in millisecond
+(``'ms'``) or microsecond (``'us'``) resolution. Since pandas uses nanoseconds
+to represent timestamps, this can occasionally be a nuisance. By default
+(when writing version 1.0 Parquet files), the nanoseconds will be cast to
+microseconds ('us').
+
+In addition, We provide the ``coerce_timestamps`` option to allow you to select
Review Comment:
Mid-sentence capitalization: "We" should be lowercase here.
##########
docs/source/python/parquet/parquet_encryption.rst:
##########
@@ -0,0 +1,280 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. currentmodule:: pyarrow
+.. _parquet_encryption:
+
+
+Parquet Modular Encryption (Columnar Encryption)
+------------------------------------------------
+
+Columnar encryption is supported for Parquet files in C++ starting from
+Apache Arrow 4.0.0 and in PyArrow starting from Apache Arrow 6.0.0.
+
+Parquet uses the envelope encryption practice, where file parts are encrypted
+with "data encryption keys" (DEKs), and the DEKs are encrypted with "master
+encryption keys" (MEKs). The DEKs are randomly generated by Parquet for each
+encrypted file/column. The MEKs are generated, stored and managed in a Key
+Management Service (KMS) of user’s choice.
+
+Reading and writing encrypted Parquet files involves passing file encryption
+and decryption properties to :class:`~pyarrow.parquet.ParquetWriter` and to
+:class:`~.ParquetFile`, respectively.
+
+Writing an encrypted Parquet file:
+
+.. code-block:: python
+
+ >>> import pyarrow.parquet as pq
+ >>> encryption_properties = crypto_factory.file_encryption_properties( # doctest: +SKIP
+ ... kms_connection_config, encryption_config)
+ >>> with pq.ParquetWriter(filename, schema, # doctest: +SKIP
+ ... encryption_properties=encryption_properties) as writer:
+ ... writer.write_table(table)
+
+Reading an encrypted Parquet file:
+
+.. code-block:: python
+
+ >>> decryption_properties = crypto_factory.file_decryption_properties( # doctest: +SKIP
+ ... kms_connection_config)
+ >>> parquet_file = pq.ParquetFile(filename, # doctest: +SKIP
+ ... decryption_properties=decryption_properties)
+
+
+In order to create the encryption and decryption properties, a
+:class:`pyarrow.parquet.encryption.CryptoFactory` should be created and
+initialized with KMS Client details, as described below.
+
+
+KMS Client
+~~~~~~~~~~
+
+The master encryption keys should be kept and managed in a production-grade
+Key Management System (KMS), deployed in the user's organization. Using Parquet
+encryption requires implementation of a client class for the KMS server.
+Any KmsClient implementation should implement the informal interface
+defined by :class:`pyarrow.parquet.encryption.KmsClient` as following:
+
+.. code-block:: python
+
+ >>> import pyarrow.parquet.encryption as pe
+ >>> class MyKmsClient(pe.KmsClient):
+ ...
+ ... """An example KmsClient implementation skeleton"""
+ ... def __init__(self, kms_connection_configuration):
+ ... pe.KmsClient.__init__(self)
+ ... # Any KMS-specific initialization based on
+ ... # kms_connection_configuration comes here
+ ...
+ ... def wrap_key(self, key_bytes, master_key_identifier):
+ ... wrapped_key = ... # call KMS to wrap key_bytes with key specified by
+ ... # master_key_identifier
+ ... return wrapped_key
+ ...
+ ... def unwrap_key(self, wrapped_key, master_key_identifier):
+ ... key_bytes = ... # call KMS to unwrap wrapped_key with key specified by
+ ... # master_key_identifier
+ ... return key_bytes
+
+The concrete implementation will be loaded at runtime by a factory function
+provided by the user. This factory function will be used to initialize the
+:class:`pyarrow.parquet.encryption.CryptoFactory` for creating file encryption
+and decryption properties.
+
+For example, in order to use the ``MyKmsClient`` defined above:
+
+.. code-block:: python
+
+ >>> def kms_client_factory(kms_connection_configuration):
+ ... return MyKmsClient(kms_connection_configuration)
+
+ >>> crypto_factory = pe.CryptoFactory(kms_client_factory)
+
+An :download:`example <../../../python/examples/parquet_encryption/sample_vault_kms_client.py>`
Review Comment:
The :download: link path was correct when this page lived at docs/source/python/parquet.rst, but after moving into docs/source/python/parquet/ it now points to a non-existent location. This will break the downloadable example link (and may error at doc build time).
##########
docs/source/python/parquet/parquet_encryption.rst:
##########
@@ -0,0 +1,280 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. currentmodule:: pyarrow
+.. _parquet_encryption:
+
+
+Parquet Modular Encryption (Columnar Encryption)
+------------------------------------------------
+
+Columnar encryption is supported for Parquet files in C++ starting from
+Apache Arrow 4.0.0 and in PyArrow starting from Apache Arrow 6.0.0.
+
+Parquet uses the envelope encryption practice, where file parts are encrypted
+with "data encryption keys" (DEKs), and the DEKs are encrypted with "master
+encryption keys" (MEKs). The DEKs are randomly generated by Parquet for each
+encrypted file/column. The MEKs are generated, stored and managed in a Key
+Management Service (KMS) of user’s choice.
+
+Reading and writing encrypted Parquet files involves passing file encryption
+and decryption properties to :class:`~pyarrow.parquet.ParquetWriter` and to
+:class:`~.ParquetFile`, respectively.
+
+Writing an encrypted Parquet file:
+
+.. code-block:: python
+
+ >>> import pyarrow.parquet as pq
+ >>> encryption_properties = crypto_factory.file_encryption_properties( # doctest: +SKIP
+ ... kms_connection_config, encryption_config)
+ >>> with pq.ParquetWriter(filename, schema, # doctest: +SKIP
+ ... encryption_properties=encryption_properties) as writer:
+ ... writer.write_table(table)
+
+Reading an encrypted Parquet file:
+
+.. code-block:: python
+
+ >>> decryption_properties = crypto_factory.file_decryption_properties( # doctest: +SKIP
+ ... kms_connection_config)
+ >>> parquet_file = pq.ParquetFile(filename, # doctest: +SKIP
+ ... decryption_properties=decryption_properties)
+
+
+In order to create the encryption and decryption properties, a
+:class:`pyarrow.parquet.encryption.CryptoFactory` should be created and
+initialized with KMS Client details, as described below.
+
+
+KMS Client
+~~~~~~~~~~
+
+The master encryption keys should be kept and managed in a production-grade
+Key Management System (KMS), deployed in the user's organization. Using Parquet
+encryption requires implementation of a client class for the KMS server.
+Any KmsClient implementation should implement the informal interface
+defined by :class:`pyarrow.parquet.encryption.KmsClient` as following:
Review Comment:
Grammar: "as following" should be "as follows".
##########
docs/source/python/index.rst:
##########
@@ -52,17 +52,17 @@ files into Arrow structures.
filesystems
numpy
pandas
- interchange_protocol
- dlpack
timestamps
- orc
+ parquet/index
+ dataset
Review Comment:
The PR description says "No user-facing changes", but switching the top-level Parquet page from python/parquet.rst to python/parquet/index.rst typically changes the built URL (Sphinx defaults to directory-style URLs unless configured otherwise) and can break existing inbound links/bookmarks. Consider preserving the old doc name/URL (e.g., avoid introducing a parquet/ subdir), or add an explicit redirect from the old location to the new one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47686: [Docs][Python] Split the Python Parquet docs into separate items [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #50419:
URL: https://github.com/apache/arrow/pull/50419#discussion_r3560232801
##########
docs/source/python/parquet/parquet_type_handling.rst:
##########
@@ -0,0 +1,156 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. currentmodule:: pyarrow
+.. _parquet_type_handling:
+
+
+Data Type Handling
+==================
+
+Reading types as DictionaryArray
+--------------------------------
+
+The ``read_dictionary`` option in ``read_table`` and ``ParquetDataset`` will
+cause columns to be read as ``DictionaryArray``, which will become
+``pandas.Categorical`` when converted to pandas. This option is only valid for
+string and binary column types, and it can yield significantly lower memory use
+and improved performance for columns with many repeated string values.
+
+.. code-block:: python
+
+ >>> import pyarrow as pa
+ >>> import pyarrow.parquet as pq
+
+ >>> table = pa.table({'one': [-1, None, 2.5],
+ ... 'two': ['foo', 'bar', 'baz'],
+ ... 'three': [True, False, True]})
+ ...
+ >>> pq.write_table(table, 'example.parquet')
+
+ >>> pq.read_table('example.parquet', read_dictionary=['two'])
+ pyarrow.Table
+ one: double
+ two: dictionary<values=string, indices=int32, ordered=0>
+ three: bool
+ ----
+ one: [[-1,null,2.5]]
+ two: [ -- dictionary:
+ ["foo","bar","baz"] -- indices:
+ [0,1,2]]
+ three: [[true,false,true]]
+
+Reading binary and list columns
+-------------------------------
+
+By default, Parquet ``BYTE_ARRAY`` columns are read as Arrow ``binary`` type
+and Parquet ``LIST`` columns are read as Arrow ``list`` type, both of which use
+32-bit offsets. For very large datasets this can overflow. The ``binary_type``
+and ``list_type`` parameters let you choose a different Arrow type on read.
+
+``binary_type`` accepts ``pa.binary()`` (default), ``pa.large_binary()``, or
+``pa.binary_view()``:
+
+.. code-block:: python
+
+ >>> import pyarrow as pa
+ >>> table = pa.table({'data': pa.array([b'hello', b'world'], pa.binary())})
+ >>> pq.write_table(table, 'binary.parquet', store_schema=False)
+ >>> pq.read_table('binary.parquet', binary_type=pa.large_binary()).schema
+ data: large_binary
+
+``list_type`` accepts ``pa.ListType`` or ``pa.LargeListType``:
+
+.. code-block:: python
+
+ >>> table = pa.table({'lists': pa.array([[1, 2], [3]], pa.list_(pa.int32()))})
+ >>> pq.write_table(table, 'lists.parquet', store_schema=False)
+ >>> pq.read_table('lists.parquet', list_type=pa.LargeListType).schema
+ lists: large_list<element: int32>
+ child 0, element: int32
+
+.. note::
+ Both settings are ignored when a serialized Arrow schema is present in the
+ Parquet file metadata (i.e. when the parquet file was written with
+ ``store_schema=True``).
+
+Read in Arrow Extension Types
+-----------------------------
+
+Certain Parquet logical types (JSON, UUID, Geometry, Geography) are supported
+and read as Arrow extension types by default (``arrow.json``, ``arrow.uuid``
+and ``geoarrow.wkb`` respectively). This support is enabled via the
+``arrow_extensions_enabled`` parameter and is used in :func:`~pyarrow.parquet.read_table`,
+:class:`~pyarrow.parquet.ParquetFile`, :class:`~pyarrow.parquet.ParquetDataset`, and
+:func:`~pyarrow.parquet.read_schema`.
+
+To read these Parquet logical types as storage types (default behavior
+until PyArrow version ``21.0.0``), set ``arrow_extensions_enabled=False``.
+
+.. note::
+ Reading GEOMETRY/GEOGRAPHY columns as ``geoarrow.wkb`` additionally
+ requires the ``geoarrow.wkb`` extension type to be registered. For that
+ you can install Python bindings for GeoArrow
+ (`geoarrow-pyarrow <https://geoarrow.org/geoarrow-python/main/index.html#installation>`_)
+ and import ``geoarrow.pyarrow`` module.
Review Comment:
🎉 !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47686: [Docs][Python] Split the Python Parquet docs into separate items [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #50419:
URL: https://github.com/apache/arrow/pull/50419#discussion_r3614975336
##########
docs/source/python/parquet/parquet.rst:
##########
@@ -0,0 +1,605 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. currentmodule:: pyarrow
+.. _parquet_single_file:
+
+
+Reading and Writing Single Files
+================================
+
+The functions :func:`~.parquet.read_table` and :func:`~.parquet.write_table`
+read and write the :ref:`pyarrow.Table <data.table>` object, respectively.
+
+Let's look at a simple table:
+
+.. code-block:: python
+
+ >>> import numpy as np
+ >>> import pandas as pd
Review Comment:
Yes, I totally agree. We need to make sure pandas is not used for creation of tables only (added that to the docs umbrella issue [here](https://github.com/apache/arrow/issues/46601)).
Why I have kept the pandas df here was due to the content further on connected to maintaining pandas index column, see line 74 and 75.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47686: [Docs][Python] Split the Python Parquet docs into separate items [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #50419:
URL: https://github.com/apache/arrow/pull/50419#issuecomment-5024112619
Revision: 58575d9566e8fddd4b60e6fe63b8201bce80e44a
Submitted crossbow builds: [ursacomputing/crossbow @ actions-c31e44a7c8](https://github.com/ursacomputing/crossbow/branches/all?query=actions-c31e44a7c8)
|Task|Status|
|----|------|
|preview-docs|[](https://github.com/ursacomputing/crossbow/actions/runs/29756055009/job/88398546283)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47686: [Docs][Python] Split the Python Parquet docs into separate items [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #50419:
URL: https://github.com/apache/arrow/pull/50419#issuecomment-5263541872
Revision: 706fa2966919845969a1f603f5f209cb6b0c7223
Submitted crossbow builds: [ursacomputing/crossbow @ actions-20af5b3716](https://github.com/ursacomputing/crossbow/branches/all?query=actions-20af5b3716)
|Task|Status|
|----|------|
|preview-docs|[](https://github.com/ursacomputing/crossbow/actions/runs/31573080708/job/94039053088)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47686: [Docs][Python] Split the Python Parquet docs into separate items [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #50419:
URL: https://github.com/apache/arrow/pull/50419#issuecomment-5426436031
Will merge. Any comments can be included as a follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47686: [Docs][Python] Split the Python Parquet docs into separate items [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #50419:
URL: https://github.com/apache/arrow/pull/50419#issuecomment-5024087334
@github-actions crossbow submit preview-docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47686: [Docs][Python] Split the Python Parquet docs into separate items [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #50419:
URL: https://github.com/apache/arrow/pull/50419#issuecomment-5034435523
@github-actions crossbow submit preview-docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47686: [Docs][Python] Split the Python Parquet docs into separate items [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #50419:
URL: https://github.com/apache/arrow/pull/50419#issuecomment-4924574025
I have used Claude for suggesting the split of pyarrow parquet page and to look through the code and the git log history for any features/changes that might be missing from the docs. From what was suggested I continued with the work on my own.
I have split the changes in different commits so the change will be easier to review (due to the docs split, the change is bigger than it really is).
[commit 1](https://github.com/apache/arrow/pull/50419/changes/774f0a95014c4a0dc1b1f46ccb178c9d3665bd82): restructure of the content, no content changes @thisisnic, do you mind having a look?
[commit 2](https://github.com/apache/arrow/pull/50419/changes/0baa52b0c3e3c9a423a2a5761c833f9faabd4279): add content about filtering and ability to write bloom filters @mapleFU, do you mind having a look?
[commit 3](https://github.com/apache/arrow/pull/50419/changes/5542e8415c1dc86c685ce0be15570941f264e908): add section on iter_batches @raulcd, do you mind having a look?
[commit 4](https://github.com/apache/arrow/pull/50419/changes/8da1e4ecc5ac6586a4978447914b622250325458): add section on binary/list_type option on read, reading extension types @pitrou @paleolimbot, do you mind having a look?
[commit 5](https://github.com/apache/arrow/pull/50419/changes/d0f4c4a0e8cf3d8793f887b1b0be43657e5bc58f): datasets note on single file partitions not being included
[commit 6](https://github.com/apache/arrow/pull/50419/changes/32c29e02a78d1e56a5145fe4e2e56eeaffdccab9): update parquet writer's parameter list @raulcd, do you mind having a look?
[commit 7](https://github.com/apache/arrow/pull/50419/changes/f38e1cad230fa834f125153a9bb264275c552298): update encryption section adding info on key rotation and direct encryption @adamreeve, do you mind having a look?
Rendered docs are available here: https://s3.amazonaws.com/arrow-data/pr_docs/50419/python/parquet/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47686: [Docs][Python] Split the Python Parquet docs into separate items [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #50419:
URL: https://github.com/apache/arrow/pull/50419#issuecomment-4924005697
@github-actions crossbow submit preview-docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47686: [Docs][Python] Split the Python Parquet docs into separate items [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF merged PR #50419:
URL: https://github.com/apache/arrow/pull/50419
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org