You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "wjones127 (via GitHub)" <gi...@apache.org> on 2023/05/05 21:52:17 UTC
[GitHub] [arrow] wjones127 opened a new pull request, #35453: GH-35331: [Python] Expose Parquet sorting metadata
wjones127 opened a new pull request, #35453:
URL: https://github.com/apache/arrow/pull/35453
### Rationale for this change
Parquet supports metadata to indicate sort order of the data. This exposes that in a low-level API through row group metadata as well as a high-level API. There isn't anything built into PyArrow for this to hook into yet, but one could imagine in the future RecordBatchReaders or ExecPlans could be tagged with ordering information based on this.
### What changes are included in this PR?
TODO
### Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
### Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please uncomment the line below and explain which changes are breaking.
-->
<!-- **This PR includes breaking changes to public APIs.** -->
<!--
Please uncomment the line below (and provide explanation) if the changes fix either (a) a security vulnerability, (b) a bug that caused incorrect or invalid data to be produced, or (c) a bug that causes a crash (even when the API contract is upheld). We use this to highlight fixes to issues that may affect users without their knowledge. For this reason, fixing bugs that cause errors don't count, since those are usually obvious.
-->
<!-- **This PR contains a "Critical Fix".** -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #35453: GH-35331: [Python] Expose Parquet sorting metadata
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #35453:
URL: https://github.com/apache/arrow/pull/35453#discussion_r1252391999
##########
python/pyarrow/_parquet.pyx:
##########
@@ -494,6 +495,212 @@ cdef class ColumnChunkMetaData(_Weakrefable):
return self.metadata.total_uncompressed_size()
+cdef class SortingColumn:
+ """
+ Sorting specification for a single column.
+
+ Returned by :meth:`RowGroupMetaData.sorting_columns` and used in
+ :class:`ParquetWriter` to specify the sort order of the data.
+
+ Parameters
+ ----------
+ column_index : int
+ Index of column data is sorted by.
+ descending : bool, default False
+ Whether column is sorted in descending order.
+ nulls_first : bool, default False
+ Whether null values appear before valid values.
+
+ Notes
+ -----
+
+ Column indices are zero-based, refer only to leaf fields, and are in
+ depth-first order. This may make the column indices for nested schemas
+ different from what you expect. In most cases, it will be easier to
+ specify the sort order using column names instead of column indices
+ and converting using the ``from_sort_order`` method.
+
+ Examples
+ --------
+
+ In other APIs, sort order is specified by names, such as:
+
+ >>> sort_order = [('id', 'ascending'), ('timestamp', 'descending')]
+
+ For Parquet, the column index must be used instead:
+
+ >>> import pyarrow.parquet as pq
+ >>> [pq.SortingColumn(0), pq.SortingColumn(1, descending=True)]
+ [SortingColumn(column_index=0, descending=False, nulls_first=False), SortingColumn(column_index=1, descending=True, nulls_first=False)]
+
+ Convert the sort_order into the list of sorting columns with
+ ``from_sort_order`` (note that the schema must be provided as well):
+
+ >>> import pyarrow as pa
+ >>> schema = pa.schema([('id', pa.int64()), ('timestamp', pa.timestamp('ms'))])
+ >>> sorting_columns = pq.SortingColumn.from_sort_order(schema, sort_order)
+ >>> sorting_columns
+ (SortingColumn(column_index=0, descending=False, nulls_first=False), SortingColumn(column_index=1, descending=True, nulls_first=False))
+
+ Convert back to the sort order with ``to_sort_order``:
+
+ >>> pq.SortingColumn.to_sort_order(schema, sorting_columns)
+ ((('id', 'ascending'), ('timestamp', 'descending')), 'at_end')
+
+ See Also
+ --------
+ RowGroupMetaData.sorting_columns
+ """
+ cdef int column_index
+ cdef c_bool descending
+ cdef c_bool nulls_first
+
+ def __init__(self, int column_index, c_bool descending=False, c_bool nulls_first=False):
+ self.column_index = column_index
+ self.descending = descending
+ self.nulls_first = nulls_first
+
+ @classmethod
+ def from_sort_order(cls, Schema schema, sort_keys, null_placement='at_end'):
+ """
+ Create a tuple of SortingColumn objects from the same arguments as
+ :class:`pyarrow.compute.SortOptions`.
+
+ Parameters
+ ----------
+ schema : Schema
+ Schema of the input data.
+ sort_keys : Sequence of (name, order) tuples
+ Names of field/column keys (str) to sort the input on,
+ along with the order each field/column is sorted in.
+ Accepted values for `order` are "ascending", "descending".
+ null_placement : {'at_start', 'at_end'}, default 'at_end'
+ Where null values should appear in the sort order.
+
+ Returns
+ -------
+ sorting_columns : tuple of SortingColumn
+ """
+ if null_placement == 'at_start':
+ nulls_first = True
+ elif null_placement == 'at_end':
+ nulls_first = False
+ else:
+ raise ValueError('null_placement must be "at_start" or "at_end"')
+
+ col_map = _name_to_index_map(schema)
+
+ sorting_columns = []
+
+ for sort_key in sort_keys:
+ if isinstance(sort_key, str):
+ name = sort_key
+ descending = False
+ elif (isinstance(sort_key, tuple) and len(sort_key) == 2 and
+ isinstance(sort_key[0], str) and
+ isinstance(sort_key[1], str)):
+ name, descending = sort_key
+ if descending == "descending":
+ descending = True
+ elif descending == "ascending":
+ descending = False
+ else:
+ raise ValueError("Invalid sort key direction: {0}"
+ .format(descending))
+ else:
+ raise ValueError("Invalid sort key: {0}".format(sort_key))
+
+ try:
+ column_index = col_map[name]
+ except KeyError:
+ raise ValueError("Sort key name '{0}' not found in schema:\n{1}"
+ .format(name, schema))
+
+ sorting_columns.append(
+ cls(column_index, descending=descending, nulls_first=nulls_first)
+ )
+
+ return tuple(sorting_columns)
+
+ @staticmethod
+ def to_sort_order(Schema schema, sorting_columns):
+ """
+ Convert a tuple of SortingColumn objects to the same format as
+ :class:`pyarrow.compute.SortOptions`.
+
+ Parameters
+ ----------
+ schema : Schema
+ Schema of the input data.
+ sorting_columns : tuple of SortingColumn
+ Columns to sort the input on.
+
+ Returns
+ -------
+ sort_keys : tuple of (name, order) tuples
+ null_placement : {'at_start', 'at_end'}
Review Comment:
Yeah it's part of numpydoc style: https://numpydoc.readthedocs.io/en/latest/format.html#parameters
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] mapleFU commented on a diff in pull request #35453: GH-35331: [Python] Expose Parquet sorting metadata
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #35453:
URL: https://github.com/apache/arrow/pull/35453#discussion_r1249246627
##########
python/pyarrow/_parquet.pyx:
##########
@@ -1665,6 +1919,82 @@ cdef shared_ptr[ArrowWriterProperties] _create_arrow_writer_properties(
return arrow_properties
+cdef _name_to_index_map(Schema arrow_schema):
+ cdef:
+ shared_ptr[CSchema] sp_arrow_schema
+ shared_ptr[SchemaDescriptor] sp_parquet_schema
+ shared_ptr[WriterProperties] props = _create_writer_properties()
+ shared_ptr[ArrowWriterProperties] arrow_props = _create_arrow_writer_properties(
+ use_deprecated_int96_timestamps=False,
+ coerce_timestamps=None,
+ allow_truncated_timestamps=False,
+ writer_engine_version="V2"
+ )
+
+ sp_arrow_schema = pyarrow_unwrap_schema(arrow_schema)
Review Comment:
so a nested path could be like `a.b.c`?
##########
python/pyarrow/_parquet.pyx:
##########
@@ -1665,6 +1919,82 @@ cdef shared_ptr[ArrowWriterProperties] _create_arrow_writer_properties(
return arrow_properties
+cdef _name_to_index_map(Schema arrow_schema):
+ cdef:
+ shared_ptr[CSchema] sp_arrow_schema
+ shared_ptr[SchemaDescriptor] sp_parquet_schema
+ shared_ptr[WriterProperties] props = _create_writer_properties()
+ shared_ptr[ArrowWriterProperties] arrow_props = _create_arrow_writer_properties(
+ use_deprecated_int96_timestamps=False,
Review Comment:
Is these default schema? or these doesn't matter?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on pull request #35453: GH-35331: [Python] Expose Parquet sorting metadata
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on PR #35453:
URL: https://github.com/apache/arrow/pull/35453#issuecomment-1536823450
Intended usage:
```python
import pyarrow as pa
import pyarrow.parquet as pq
import pyarrow.compute as pc
tab = pa.table({
'a': pa.array([1, 3, 2, 2]),
'b': pa.array(['a', 'b', 'c', 'd'])
})
sort_keys = [('a', 'ascending'), ('b', 'descending')]
sorted_tab = tab.sort_by(sort_keys)
pq.write_table(tab, 'data.parquet', sorting_columns=sort_keys)
metadata = pq.read_metadata('data.parquet')
metadata.row_group(0).sorting_columns
pq_file = pq.ParquetFile('data.parquet')
sort_order, null_placement = pq_file.sort_order
assert sort_order == sort_keys
assert null_placement == "at_end"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on pull request #35453: GH-35331: [Python] Expose Parquet sorting metadata
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on PR #35453:
URL: https://github.com/apache/arrow/pull/35453#issuecomment-1538756938
> @westonpace This PR might interest you. It occurred to me while working with this we can persist the sort order of data in Parquet and retrieve it. Makes me wonder where we could integrate it into the rest of the code base.
It is interesting. Though one challenge would be that a single file doesn't neccesarily tell you about an entire dataset. For example, if the file foo.parquet is sorted by "date" then are all files in that dataset sorted by date? Does foo.parquet come before bar.parquet?
That being said, we could probably hook it into exec plans pretty easily for the cases where the source dataset is a single file.
There may also be some benefit in knowing that individual files are sorted (even if the dataset is not) but I'm not quite sure how exactly we'd exploit that yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wgtmac commented on pull request #35453: GH-35331: [Python] Expose Parquet sorting metadata
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #35453:
URL: https://github.com/apache/arrow/pull/35453#issuecomment-1661535053
@wjones127 Do you want to revive this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on pull request #35453: GH-35331: [Python] Expose Parquet sorting metadata
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on PR #35453:
URL: https://github.com/apache/arrow/pull/35453#issuecomment-1539066125
> Though one challenge would be that a single file doesn't necessarily tell you about an entire dataset. For example, if the file foo.parquet is sorted by "date" then are all files in that dataset sorted by date? Does foo.parquet come before bar.parquet?
Yeah, if you wanted to do that, you'd have to combine it with other metadata to get the sorting relationship between partitions and within partitions. Between partitions, you could sort the partition values and prepend that to the sorting order. For within partitions, you could aggregate the row group stats to file-level stats, and use those to see if they combine to an overall ordering.
So, for example, a dataset like:
```
part=A/file1.parquet (min x: 1, max x: 10)
part=A/file1.parquet (min x: 10, max x: 20)
part=B/file1.parquet (min x: 5, max x: 20)
```
If each of the files returned a sort order of `x, ascending`, then you could construct the dataset to have an overall ordering of `[(part, ascending), (x, ascending)]`.
Honestly, it would be nice just to be able to get a dataset to be considered "ordered" by its partition values. For example, if I had two tables partitioned by date, it would be nice to be able to join them via a sort merge join on that date column while using the inherent sorting of the partition values. (Having flashbacks to some frustrations in my days working with PySpark.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #35453: GH-35331: [Python] Expose Parquet sorting metadata
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35453:
URL: https://github.com/apache/arrow/pull/35453#issuecomment-1536822516
* Closes: #35331
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #35453: GH-35331: [Python] Expose Parquet sorting metadata
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #35453:
URL: https://github.com/apache/arrow/pull/35453#discussion_r1252391253
##########
python/pyarrow/_parquet.pyx:
##########
@@ -494,6 +495,212 @@ cdef class ColumnChunkMetaData(_Weakrefable):
return self.metadata.total_uncompressed_size()
+cdef class SortingColumn:
+ """
+ Sorting specification for a single column.
+
+ Returned by :meth:`RowGroupMetaData.sorting_columns` and used in
+ :class:`ParquetWriter` to specify the sort order of the data.
+
+ Parameters
+ ----------
+ column_index : int
+ Index of column data is sorted by.
+ descending : bool, default False
+ Whether column is sorted in descending order.
+ nulls_first : bool, default False
+ Whether null values appear before valid values.
+
+ Notes
+ -----
+
+ Column indices are zero-based, refer only to leaf fields, and are in
+ depth-first order. This may make the column indices for nested schemas
+ different from what you expect. In most cases, it will be easier to
+ specify the sort order using column names instead of column indices
+ and converting using the ``from_sort_order`` method.
+
+ Examples
+ --------
+
+ In other APIs, sort order is specified by names, such as:
+
+ >>> sort_order = [('id', 'ascending'), ('timestamp', 'descending')]
+
+ For Parquet, the column index must be used instead:
+
+ >>> import pyarrow.parquet as pq
+ >>> [pq.SortingColumn(0), pq.SortingColumn(1, descending=True)]
+ [SortingColumn(column_index=0, descending=False, nulls_first=False), SortingColumn(column_index=1, descending=True, nulls_first=False)]
+
+ Convert the sort_order into the list of sorting columns with
+ ``from_sort_order`` (note that the schema must be provided as well):
+
+ >>> import pyarrow as pa
+ >>> schema = pa.schema([('id', pa.int64()), ('timestamp', pa.timestamp('ms'))])
+ >>> sorting_columns = pq.SortingColumn.from_sort_order(schema, sort_order)
+ >>> sorting_columns
+ (SortingColumn(column_index=0, descending=False, nulls_first=False), SortingColumn(column_index=1, descending=True, nulls_first=False))
+
+ Convert back to the sort order with ``to_sort_order``:
+
+ >>> pq.SortingColumn.to_sort_order(schema, sorting_columns)
+ ((('id', 'ascending'), ('timestamp', 'descending')), 'at_end')
+
+ See Also
+ --------
+ RowGroupMetaData.sorting_columns
+ """
+ cdef int column_index
+ cdef c_bool descending
+ cdef c_bool nulls_first
+
+ def __init__(self, int column_index, c_bool descending=False, c_bool nulls_first=False):
+ self.column_index = column_index
+ self.descending = descending
+ self.nulls_first = nulls_first
+
+ @classmethod
+ def from_sort_order(cls, Schema schema, sort_keys, null_placement='at_end'):
Review Comment:
Yeah, was aiming to make familiar to users of that API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on pull request #35453: GH-35331: [Python] Expose Parquet sorting metadata
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on PR #35453:
URL: https://github.com/apache/arrow/pull/35453#issuecomment-1536846260
@westonpace This PR might interest you. It occurred to me while working with this we can persist the sort order of data in Parquet and retrieve it. Makes me wonder where we could integrate it into the rest of the code base.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #35453: GH-35331: [Python] Expose Parquet sorting metadata
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #35453:
URL: https://github.com/apache/arrow/pull/35453#discussion_r1252392933
##########
python/pyarrow/_parquet.pyx:
##########
@@ -1665,6 +1919,82 @@ cdef shared_ptr[ArrowWriterProperties] _create_arrow_writer_properties(
return arrow_properties
+cdef _name_to_index_map(Schema arrow_schema):
+ cdef:
+ shared_ptr[CSchema] sp_arrow_schema
+ shared_ptr[SchemaDescriptor] sp_parquet_schema
+ shared_ptr[WriterProperties] props = _create_writer_properties()
+ shared_ptr[ArrowWriterProperties] arrow_props = _create_arrow_writer_properties(
+ use_deprecated_int96_timestamps=False,
Review Comment:
Yeah these don't matter much. We just create them because the `ToParquetSchema` method asks for it.
##########
python/pyarrow/_parquet.pyx:
##########
@@ -1665,6 +1919,82 @@ cdef shared_ptr[ArrowWriterProperties] _create_arrow_writer_properties(
return arrow_properties
+cdef _name_to_index_map(Schema arrow_schema):
+ cdef:
+ shared_ptr[CSchema] sp_arrow_schema
+ shared_ptr[SchemaDescriptor] sp_parquet_schema
+ shared_ptr[WriterProperties] props = _create_writer_properties()
+ shared_ptr[ArrowWriterProperties] arrow_props = _create_arrow_writer_properties(
+ use_deprecated_int96_timestamps=False,
+ coerce_timestamps=None,
+ allow_truncated_timestamps=False,
+ writer_engine_version="V2"
+ )
+
+ sp_arrow_schema = pyarrow_unwrap_schema(arrow_schema)
Review Comment:
Yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on a diff in pull request #35453: GH-35331: [Python] Expose Parquet sorting metadata
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on code in PR #35453:
URL: https://github.com/apache/arrow/pull/35453#discussion_r1251304524
##########
python/pyarrow/_parquet.pyx:
##########
@@ -494,6 +495,212 @@ cdef class ColumnChunkMetaData(_Weakrefable):
return self.metadata.total_uncompressed_size()
+cdef class SortingColumn:
+ """
+ Sorting specification for a single column.
+
+ Returned by :meth:`RowGroupMetaData.sorting_columns` and used in
+ :class:`ParquetWriter` to specify the sort order of the data.
+
+ Parameters
+ ----------
+ column_index : int
+ Index of column data is sorted by.
+ descending : bool, default False
+ Whether column is sorted in descending order.
+ nulls_first : bool, default False
+ Whether null values appear before valid values.
+
+ Notes
+ -----
+
+ Column indices are zero-based, refer only to leaf fields, and are in
+ depth-first order. This may make the column indices for nested schemas
+ different from what you expect. In most cases, it will be easier to
+ specify the sort order using column names instead of column indices
+ and converting using the ``from_sort_order`` method.
+
+ Examples
+ --------
+
+ In other APIs, sort order is specified by names, such as:
+
+ >>> sort_order = [('id', 'ascending'), ('timestamp', 'descending')]
+
+ For Parquet, the column index must be used instead:
+
+ >>> import pyarrow.parquet as pq
+ >>> [pq.SortingColumn(0), pq.SortingColumn(1, descending=True)]
+ [SortingColumn(column_index=0, descending=False, nulls_first=False), SortingColumn(column_index=1, descending=True, nulls_first=False)]
+
+ Convert the sort_order into the list of sorting columns with
+ ``from_sort_order`` (note that the schema must be provided as well):
+
+ >>> import pyarrow as pa
+ >>> schema = pa.schema([('id', pa.int64()), ('timestamp', pa.timestamp('ms'))])
+ >>> sorting_columns = pq.SortingColumn.from_sort_order(schema, sort_order)
+ >>> sorting_columns
+ (SortingColumn(column_index=0, descending=False, nulls_first=False), SortingColumn(column_index=1, descending=True, nulls_first=False))
+
+ Convert back to the sort order with ``to_sort_order``:
+
+ >>> pq.SortingColumn.to_sort_order(schema, sorting_columns)
+ ((('id', 'ascending'), ('timestamp', 'descending')), 'at_end')
+
+ See Also
+ --------
+ RowGroupMetaData.sorting_columns
+ """
+ cdef int column_index
+ cdef c_bool descending
+ cdef c_bool nulls_first
+
+ def __init__(self, int column_index, c_bool descending=False, c_bool nulls_first=False):
+ self.column_index = column_index
+ self.descending = descending
+ self.nulls_first = nulls_first
+
+ @classmethod
+ def from_sort_order(cls, Schema schema, sort_keys, null_placement='at_end'):
Review Comment:
Why `null_placement` here and `nulls_first` elsewhere?
##########
python/pyarrow/_parquet.pyx:
##########
@@ -494,6 +495,212 @@ cdef class ColumnChunkMetaData(_Weakrefable):
return self.metadata.total_uncompressed_size()
+cdef class SortingColumn:
+ """
+ Sorting specification for a single column.
+
+ Returned by :meth:`RowGroupMetaData.sorting_columns` and used in
+ :class:`ParquetWriter` to specify the sort order of the data.
+
+ Parameters
+ ----------
+ column_index : int
+ Index of column data is sorted by.
+ descending : bool, default False
+ Whether column is sorted in descending order.
+ nulls_first : bool, default False
+ Whether null values appear before valid values.
+
+ Notes
+ -----
+
+ Column indices are zero-based, refer only to leaf fields, and are in
+ depth-first order. This may make the column indices for nested schemas
+ different from what you expect. In most cases, it will be easier to
+ specify the sort order using column names instead of column indices
+ and converting using the ``from_sort_order`` method.
+
+ Examples
+ --------
+
+ In other APIs, sort order is specified by names, such as:
+
+ >>> sort_order = [('id', 'ascending'), ('timestamp', 'descending')]
+
+ For Parquet, the column index must be used instead:
+
+ >>> import pyarrow.parquet as pq
+ >>> [pq.SortingColumn(0), pq.SortingColumn(1, descending=True)]
+ [SortingColumn(column_index=0, descending=False, nulls_first=False), SortingColumn(column_index=1, descending=True, nulls_first=False)]
+
+ Convert the sort_order into the list of sorting columns with
+ ``from_sort_order`` (note that the schema must be provided as well):
+
+ >>> import pyarrow as pa
+ >>> schema = pa.schema([('id', pa.int64()), ('timestamp', pa.timestamp('ms'))])
+ >>> sorting_columns = pq.SortingColumn.from_sort_order(schema, sort_order)
+ >>> sorting_columns
+ (SortingColumn(column_index=0, descending=False, nulls_first=False), SortingColumn(column_index=1, descending=True, nulls_first=False))
+
+ Convert back to the sort order with ``to_sort_order``:
+
+ >>> pq.SortingColumn.to_sort_order(schema, sorting_columns)
+ ((('id', 'ascending'), ('timestamp', 'descending')), 'at_end')
+
+ See Also
+ --------
+ RowGroupMetaData.sorting_columns
+ """
+ cdef int column_index
+ cdef c_bool descending
+ cdef c_bool nulls_first
+
+ def __init__(self, int column_index, c_bool descending=False, c_bool nulls_first=False):
+ self.column_index = column_index
+ self.descending = descending
+ self.nulls_first = nulls_first
+
+ @classmethod
+ def from_sort_order(cls, Schema schema, sort_keys, null_placement='at_end'):
+ """
+ Create a tuple of SortingColumn objects from the same arguments as
+ :class:`pyarrow.compute.SortOptions`.
+
+ Parameters
+ ----------
+ schema : Schema
+ Schema of the input data.
+ sort_keys : Sequence of (name, order) tuples
+ Names of field/column keys (str) to sort the input on,
+ along with the order each field/column is sorted in.
+ Accepted values for `order` are "ascending", "descending".
+ null_placement : {'at_start', 'at_end'}, default 'at_end'
+ Where null values should appear in the sort order.
+
+ Returns
+ -------
+ sorting_columns : tuple of SortingColumn
+ """
+ if null_placement == 'at_start':
+ nulls_first = True
+ elif null_placement == 'at_end':
+ nulls_first = False
+ else:
+ raise ValueError('null_placement must be "at_start" or "at_end"')
+
+ col_map = _name_to_index_map(schema)
+
+ sorting_columns = []
+
+ for sort_key in sort_keys:
+ if isinstance(sort_key, str):
+ name = sort_key
+ descending = False
+ elif (isinstance(sort_key, tuple) and len(sort_key) == 2 and
+ isinstance(sort_key[0], str) and
+ isinstance(sort_key[1], str)):
+ name, descending = sort_key
+ if descending == "descending":
+ descending = True
+ elif descending == "ascending":
+ descending = False
+ else:
+ raise ValueError("Invalid sort key direction: {0}"
+ .format(descending))
+ else:
+ raise ValueError("Invalid sort key: {0}".format(sort_key))
+
+ try:
+ column_index = col_map[name]
+ except KeyError:
+ raise ValueError("Sort key name '{0}' not found in schema:\n{1}"
+ .format(name, schema))
+
+ sorting_columns.append(
+ cls(column_index, descending=descending, nulls_first=nulls_first)
+ )
+
+ return tuple(sorting_columns)
+
+ @staticmethod
+ def to_sort_order(Schema schema, sorting_columns):
+ """
+ Convert a tuple of SortingColumn objects to the same format as
+ :class:`pyarrow.compute.SortOptions`.
+
+ Parameters
+ ----------
+ schema : Schema
+ Schema of the input data.
+ sorting_columns : tuple of SortingColumn
+ Columns to sort the input on.
+
+ Returns
+ -------
+ sort_keys : tuple of (name, order) tuples
+ null_placement : {'at_start', 'at_end'}
Review Comment:
I assume this means "one of `at_start` or `at_end`"? Is using curly braces here a standard way to do this? The only way I'm familiar with is python typings which would be `Literal['at_start', 'at_end']` but I'm not really an expert in docstrings so I wouldn't be surprised if this is some pattern I'm just unfamiliar with.
##########
python/pyarrow/parquet/core.py:
##########
@@ -867,6 +915,10 @@ def _sanitize_table(table, new_schema, flavor):
it will restore the timezone (Parquet only stores the UTC values without
timezone), or columns with duration type will be restored from the int64
Parquet column.
+sorting_columns : Sequence of SortingColumn, default None
+ Specify the sort order of the data being written. The writer does not sort
+ the data nor does it verify that the data is sorted. The sort order is
Review Comment:
There is a danger here but I think you explain it here well enough. I can't think of any way to be more cautionary without just being overly verbose.
##########
python/pyarrow/_parquet.pyx:
##########
@@ -494,6 +495,212 @@ cdef class ColumnChunkMetaData(_Weakrefable):
return self.metadata.total_uncompressed_size()
+cdef class SortingColumn:
+ """
+ Sorting specification for a single column.
+
+ Returned by :meth:`RowGroupMetaData.sorting_columns` and used in
+ :class:`ParquetWriter` to specify the sort order of the data.
+
+ Parameters
+ ----------
+ column_index : int
+ Index of column data is sorted by.
Review Comment:
```suggestion
Index that column data is sorted by.
```
##########
python/pyarrow/_parquet.pyx:
##########
@@ -494,6 +495,212 @@ cdef class ColumnChunkMetaData(_Weakrefable):
return self.metadata.total_uncompressed_size()
+cdef class SortingColumn:
+ """
+ Sorting specification for a single column.
+
+ Returned by :meth:`RowGroupMetaData.sorting_columns` and used in
+ :class:`ParquetWriter` to specify the sort order of the data.
+
+ Parameters
+ ----------
+ column_index : int
+ Index of column data is sorted by.
+ descending : bool, default False
+ Whether column is sorted in descending order.
+ nulls_first : bool, default False
+ Whether null values appear before valid values.
+
+ Notes
+ -----
+
+ Column indices are zero-based, refer only to leaf fields, and are in
+ depth-first order. This may make the column indices for nested schemas
+ different from what you expect. In most cases, it will be easier to
+ specify the sort order using column names instead of column indices
+ and converting using the ``from_sort_order`` method.
+
+ Examples
+ --------
+
+ In other APIs, sort order is specified by names, such as:
+
+ >>> sort_order = [('id', 'ascending'), ('timestamp', 'descending')]
+
+ For Parquet, the column index must be used instead:
+
+ >>> import pyarrow.parquet as pq
+ >>> [pq.SortingColumn(0), pq.SortingColumn(1, descending=True)]
+ [SortingColumn(column_index=0, descending=False, nulls_first=False), SortingColumn(column_index=1, descending=True, nulls_first=False)]
+
+ Convert the sort_order into the list of sorting columns with
+ ``from_sort_order`` (note that the schema must be provided as well):
+
+ >>> import pyarrow as pa
+ >>> schema = pa.schema([('id', pa.int64()), ('timestamp', pa.timestamp('ms'))])
+ >>> sorting_columns = pq.SortingColumn.from_sort_order(schema, sort_order)
+ >>> sorting_columns
+ (SortingColumn(column_index=0, descending=False, nulls_first=False), SortingColumn(column_index=1, descending=True, nulls_first=False))
+
+ Convert back to the sort order with ``to_sort_order``:
+
+ >>> pq.SortingColumn.to_sort_order(schema, sorting_columns)
+ ((('id', 'ascending'), ('timestamp', 'descending')), 'at_end')
+
+ See Also
+ --------
+ RowGroupMetaData.sorting_columns
+ """
+ cdef int column_index
+ cdef c_bool descending
+ cdef c_bool nulls_first
+
+ def __init__(self, int column_index, c_bool descending=False, c_bool nulls_first=False):
+ self.column_index = column_index
+ self.descending = descending
+ self.nulls_first = nulls_first
+
+ @classmethod
+ def from_sort_order(cls, Schema schema, sort_keys, null_placement='at_end'):
Review Comment:
ah, I see, perhaps to align `pc.sort_indices`?
##########
python/pyarrow/tests/parquet/test_metadata.py:
##########
@@ -301,6 +301,91 @@ def test_parquet_write_disable_statistics(tempdir):
assert cc_b.statistics is None
+def test_parquet_sorting_column():
+ sorting_col = pq.SortingColumn(10)
+ assert sorting_col.column_index == 10
+ assert sorting_col.descending is False
+ assert sorting_col.nulls_first is False
+
+ sorting_col = pq.SortingColumn(0, descending=True, nulls_first=True)
+ assert sorting_col.column_index == 0
+ assert sorting_col.descending is True
+ assert sorting_col.nulls_first is True
+
+ schema = pa.schema([('a', pa.int64()), ('b', pa.int64())])
+ sorting_cols = (
+ pq.SortingColumn(1, descending=True),
+ pq.SortingColumn(0, descending=False),
+ )
+ sort_order, null_placement = pq.SortingColumn.to_sort_order(schema, sorting_cols)
+ assert sort_order == (('b', "descending"), ('a', "ascending"))
+ assert null_placement == "at_end"
+
+ sorting_cols_roundtripped = pq.SortingColumn.from_sort_order(
+ schema, sort_order, null_placement)
+ assert sorting_cols_roundtripped == sorting_cols
+
+ sorting_cols = pq.SortingColumn.from_sort_order(
+ schema, ('a', ('b', "descending")), null_placement="at_start")
+ expected = (
+ pq.SortingColumn(0, descending=False, nulls_first=True),
+ pq.SortingColumn(1, descending=True, nulls_first=True),
+ )
+ assert sorting_cols == expected
+
+ # Conversions handle empty tuples
+ empty_sorting_cols = pq.SortingColumn.from_sort_order(schema, ())
+ assert empty_sorting_cols == ()
+
+ assert pq.SortingColumn.to_sort_order(schema, ()) == ((), "at_end")
+
+ with pytest.raises(ValueError):
+ pq.SortingColumn.from_sort_order(schema, (("a", "not a valid sort order")))
+
+ with pytest.raises(ValueError, match="inconsistent null placement"):
+ sorting_cols = (
+ pq.SortingColumn(1, nulls_first=True),
+ pq.SortingColumn(0, nulls_first=False),
+ )
+ pq.SortingColumn.to_sort_order(schema, sorting_cols)
+
+
+def test_parquet_sorting_column_nested():
+ schema = pa.schema({
+ 'a': pa.struct([('x', pa.int64()), ('y', pa.int64())]),
+ 'b': pa.int64()
+ })
+
+ sorting_columns = [
+ pq.SortingColumn(0, descending=True), # a.x
+ pq.SortingColumn(2, descending=False) # b
+ ]
+
+ sort_order, null_placement = pq.SortingColumn.to_sort_order(schema, sorting_columns)
+ assert null_placement == "at_end"
+ assert len(sort_order) == 2
+ assert sort_order[0] == ("a.x", "descending")
+ assert sort_order[1] == ("b", "ascending")
+
+
+def test_parquet_file_sorting_columns():
+ table = pa.table({'a': [1, 2, 3], 'b': ['a', 'b', 'c']})
+
+ sorting_columns = (
+ pq.SortingColumn(column_index=0, descending=True, nulls_first=True),
+ pq.SortingColumn(column_index=1, descending=False),
+ )
+ writer = pa.BufferOutputStream()
+ _write_table(table, writer, sorting_columns=sorting_columns)
+ reader = pa.BufferReader(writer.getvalue())
+
+ # Can retrieve sorting columns from metadata
+ metadata = pq.read_metadata(reader)
+ set_sorting_columns = {metadata.row_group(i).sorting_columns
+ for i in range(metadata.num_row_groups)}
+ assert set_sorting_columns == set([sorting_columns])
Review Comment:
Why a set here? Can't we assert that there will only be one row group?
##########
python/pyarrow/_parquet.pyx:
##########
@@ -494,6 +495,212 @@ cdef class ColumnChunkMetaData(_Weakrefable):
return self.metadata.total_uncompressed_size()
+cdef class SortingColumn:
+ """
+ Sorting specification for a single column.
+
+ Returned by :meth:`RowGroupMetaData.sorting_columns` and used in
+ :class:`ParquetWriter` to specify the sort order of the data.
+
+ Parameters
+ ----------
+ column_index : int
+ Index of column data is sorted by.
+ descending : bool, default False
+ Whether column is sorted in descending order.
+ nulls_first : bool, default False
+ Whether null values appear before valid values.
+
+ Notes
+ -----
+
+ Column indices are zero-based, refer only to leaf fields, and are in
+ depth-first order. This may make the column indices for nested schemas
+ different from what you expect. In most cases, it will be easier to
+ specify the sort order using column names instead of column indices
+ and converting using the ``from_sort_order`` method.
+
+ Examples
+ --------
+
+ In other APIs, sort order is specified by names, such as:
+
+ >>> sort_order = [('id', 'ascending'), ('timestamp', 'descending')]
+
+ For Parquet, the column index must be used instead:
+
+ >>> import pyarrow.parquet as pq
+ >>> [pq.SortingColumn(0), pq.SortingColumn(1, descending=True)]
+ [SortingColumn(column_index=0, descending=False, nulls_first=False), SortingColumn(column_index=1, descending=True, nulls_first=False)]
+
+ Convert the sort_order into the list of sorting columns with
+ ``from_sort_order`` (note that the schema must be provided as well):
+
+ >>> import pyarrow as pa
+ >>> schema = pa.schema([('id', pa.int64()), ('timestamp', pa.timestamp('ms'))])
+ >>> sorting_columns = pq.SortingColumn.from_sort_order(schema, sort_order)
+ >>> sorting_columns
+ (SortingColumn(column_index=0, descending=False, nulls_first=False), SortingColumn(column_index=1, descending=True, nulls_first=False))
+
+ Convert back to the sort order with ``to_sort_order``:
+
+ >>> pq.SortingColumn.to_sort_order(schema, sorting_columns)
+ ((('id', 'ascending'), ('timestamp', 'descending')), 'at_end')
+
+ See Also
+ --------
+ RowGroupMetaData.sorting_columns
+ """
+ cdef int column_index
+ cdef c_bool descending
+ cdef c_bool nulls_first
+
+ def __init__(self, int column_index, c_bool descending=False, c_bool nulls_first=False):
+ self.column_index = column_index
+ self.descending = descending
+ self.nulls_first = nulls_first
+
+ @classmethod
+ def from_sort_order(cls, Schema schema, sort_keys, null_placement='at_end'):
+ """
+ Create a tuple of SortingColumn objects from the same arguments as
+ :class:`pyarrow.compute.SortOptions`.
+
+ Parameters
+ ----------
+ schema : Schema
+ Schema of the input data.
+ sort_keys : Sequence of (name, order) tuples
+ Names of field/column keys (str) to sort the input on,
+ along with the order each field/column is sorted in.
+ Accepted values for `order` are "ascending", "descending".
+ null_placement : {'at_start', 'at_end'}, default 'at_end'
+ Where null values should appear in the sort order.
+
+ Returns
+ -------
+ sorting_columns : tuple of SortingColumn
+ """
+ if null_placement == 'at_start':
+ nulls_first = True
+ elif null_placement == 'at_end':
+ nulls_first = False
+ else:
+ raise ValueError('null_placement must be "at_start" or "at_end"')
+
+ col_map = _name_to_index_map(schema)
+
+ sorting_columns = []
+
+ for sort_key in sort_keys:
+ if isinstance(sort_key, str):
+ name = sort_key
+ descending = False
+ elif (isinstance(sort_key, tuple) and len(sort_key) == 2 and
+ isinstance(sort_key[0], str) and
+ isinstance(sort_key[1], str)):
+ name, descending = sort_key
+ if descending == "descending":
+ descending = True
+ elif descending == "ascending":
+ descending = False
+ else:
+ raise ValueError("Invalid sort key direction: {0}"
+ .format(descending))
+ else:
+ raise ValueError("Invalid sort key: {0}".format(sort_key))
+
+ try:
+ column_index = col_map[name]
+ except KeyError:
+ raise ValueError("Sort key name '{0}' not found in schema:\n{1}"
+ .format(name, schema))
+
+ sorting_columns.append(
+ cls(column_index, descending=descending, nulls_first=nulls_first)
+ )
+
+ return tuple(sorting_columns)
+
+ @staticmethod
+ def to_sort_order(Schema schema, sorting_columns):
+ """
+ Convert a tuple of SortingColumn objects to the same format as
+ :class:`pyarrow.compute.SortOptions`.
+
+ Parameters
+ ----------
+ schema : Schema
+ Schema of the input data.
+ sorting_columns : tuple of SortingColumn
+ Columns to sort the input on.
+
+ Returns
+ -------
+ sort_keys : tuple of (name, order) tuples
+ null_placement : {'at_start', 'at_end'}
+ """
+ col_map = {i: name for name, i in _name_to_index_map(schema).items()}
+
+ sort_keys = []
+ nulls_first = None
+
+ for sorting_column in sorting_columns:
+ name = col_map[sorting_column.column_index]
+ if sorting_column.descending:
+ order = "descending"
+ else:
+ order = "ascending"
+ sort_keys.append((name, order))
+ if nulls_first is None:
+ nulls_first = sorting_column.nulls_first
+ elif nulls_first != sorting_column.nulls_first:
+ raise ValueError("Sorting columns have inconsistent null placement")
+
+ if nulls_first:
+ null_placement = "at_start"
+ else:
+ null_placement = "at_end"
+
+ return tuple(sort_keys), null_placement
+
+ def __repr__(self):
+ return """{}(column_index={}, descending={}, nulls_first={})""".format(
+ self.__class__.__name__,
+ self.column_index, self.descending, self.nulls_first)
+
+ def __eq__(self, SortingColumn other):
+ return (self.column_index == other.column_index and
+ self.descending == other.descending and
+ self.nulls_first == other.nulls_first)
+
+ def __hash__(self):
+ return hash((self.column_index, self.descending, self.nulls_first))
+
+ @property
+ def column_index(self):
+ """"Index of column data is sorted by (int)."""
+ return self.column_index
+
+ @property
+ def descending(self):
+ """Whether column is sorted in descending order (bool)."""
+ return self.descending
+
+ @property
+ def nulls_first(self):
+ """Whether null values appear before valid values (bool)."""
+ return self.nulls_first
+
+ def to_dict(self):
+ """Convert to dictionary representation."""
+ return {
+ 'column_index': self.column_index,
+ 'descending': self.descending,
+ 'nulls_first': self.nulls_first
+ }
Review Comment:
Is this needed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] mapleFU commented on a diff in pull request #35453: GH-35331: [Python] Expose Parquet sorting metadata
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #35453:
URL: https://github.com/apache/arrow/pull/35453#discussion_r1186600013
##########
python/pyarrow/parquet/core.py:
##########
@@ -867,6 +915,10 @@ def _sanitize_table(table, new_schema, flavor):
it will restore the timezone (Parquet only stores the UTC values without
timezone), or columns with duration type will be restored from the int64
Parquet column.
+sorting_columns : Sequence of SortingColumn, default None
+ Specify the sort order of the data being written. The writer does not sort
+ the data nor does it verify that the data is sorted. The sort order is
Review Comment:
So, user him/her self should ensure the correctness of sorting_columns?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on pull request #35453: GH-35331: [Python] Expose Parquet sorting metadata
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on PR #35453:
URL: https://github.com/apache/arrow/pull/35453#issuecomment-1616066805
Yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-35331: [Python] Expose Parquet sorting metadata [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF closed pull request #35453: GH-35331: [Python] Expose Parquet sorting metadata
URL: https://github.com/apache/arrow/pull/35453
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #35453: GH-35331: [Python] Expose Parquet sorting metadata
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #35453:
URL: https://github.com/apache/arrow/pull/35453#discussion_r1190491683
##########
python/pyarrow/_parquet.pxd:
##########
@@ -502,8 +509,8 @@ cdef extern from "parquet/arrow/schema.h" namespace "parquet::arrow" nogil:
CStatus ToParquetSchema(
const CSchema* arrow_schema,
- const ArrowReaderProperties& properties,
- const shared_ptr[const CKeyValueMetadata]& key_value_metadata,
+ const WriterProperties& properties,
+ const ArrowWriterProperties& arrow_properties,
Review Comment:
This declaration was never used and was out of date.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on pull request #35453: GH-35331: [Python] Expose Parquet sorting metadata
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on PR #35453:
URL: https://github.com/apache/arrow/pull/35453#issuecomment-1616066382
@westonpace would you have time to review next week?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] judahrand commented on pull request #35453: GH-35331: [Python] Expose Parquet sorting metadata
Posted by "judahrand (via GitHub)" <gi...@apache.org>.
judahrand commented on PR #35453:
URL: https://github.com/apache/arrow/pull/35453#issuecomment-1714232542
> Seems I and weston all approve this pull request, what is the remaining blocker for this? Can we just rebase and move forward?
Is there more work which needs doing here? I'm happy to pick this up and rebase?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #35453: GH-35331: [Python] Expose Parquet sorting metadata
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #35453:
URL: https://github.com/apache/arrow/pull/35453#discussion_r1252392206
##########
python/pyarrow/_parquet.pyx:
##########
@@ -494,6 +495,212 @@ cdef class ColumnChunkMetaData(_Weakrefable):
return self.metadata.total_uncompressed_size()
+cdef class SortingColumn:
+ """
+ Sorting specification for a single column.
+
+ Returned by :meth:`RowGroupMetaData.sorting_columns` and used in
+ :class:`ParquetWriter` to specify the sort order of the data.
+
+ Parameters
+ ----------
+ column_index : int
+ Index of column data is sorted by.
+ descending : bool, default False
+ Whether column is sorted in descending order.
+ nulls_first : bool, default False
+ Whether null values appear before valid values.
+
+ Notes
+ -----
+
+ Column indices are zero-based, refer only to leaf fields, and are in
+ depth-first order. This may make the column indices for nested schemas
+ different from what you expect. In most cases, it will be easier to
+ specify the sort order using column names instead of column indices
+ and converting using the ``from_sort_order`` method.
+
+ Examples
+ --------
+
+ In other APIs, sort order is specified by names, such as:
+
+ >>> sort_order = [('id', 'ascending'), ('timestamp', 'descending')]
+
+ For Parquet, the column index must be used instead:
+
+ >>> import pyarrow.parquet as pq
+ >>> [pq.SortingColumn(0), pq.SortingColumn(1, descending=True)]
+ [SortingColumn(column_index=0, descending=False, nulls_first=False), SortingColumn(column_index=1, descending=True, nulls_first=False)]
+
+ Convert the sort_order into the list of sorting columns with
+ ``from_sort_order`` (note that the schema must be provided as well):
+
+ >>> import pyarrow as pa
+ >>> schema = pa.schema([('id', pa.int64()), ('timestamp', pa.timestamp('ms'))])
+ >>> sorting_columns = pq.SortingColumn.from_sort_order(schema, sort_order)
+ >>> sorting_columns
+ (SortingColumn(column_index=0, descending=False, nulls_first=False), SortingColumn(column_index=1, descending=True, nulls_first=False))
+
+ Convert back to the sort order with ``to_sort_order``:
+
+ >>> pq.SortingColumn.to_sort_order(schema, sorting_columns)
+ ((('id', 'ascending'), ('timestamp', 'descending')), 'at_end')
+
+ See Also
+ --------
+ RowGroupMetaData.sorting_columns
+ """
+ cdef int column_index
+ cdef c_bool descending
+ cdef c_bool nulls_first
+
+ def __init__(self, int column_index, c_bool descending=False, c_bool nulls_first=False):
+ self.column_index = column_index
+ self.descending = descending
+ self.nulls_first = nulls_first
+
+ @classmethod
+ def from_sort_order(cls, Schema schema, sort_keys, null_placement='at_end'):
+ """
+ Create a tuple of SortingColumn objects from the same arguments as
+ :class:`pyarrow.compute.SortOptions`.
+
+ Parameters
+ ----------
+ schema : Schema
+ Schema of the input data.
+ sort_keys : Sequence of (name, order) tuples
+ Names of field/column keys (str) to sort the input on,
+ along with the order each field/column is sorted in.
+ Accepted values for `order` are "ascending", "descending".
+ null_placement : {'at_start', 'at_end'}, default 'at_end'
+ Where null values should appear in the sort order.
+
+ Returns
+ -------
+ sorting_columns : tuple of SortingColumn
+ """
+ if null_placement == 'at_start':
+ nulls_first = True
+ elif null_placement == 'at_end':
+ nulls_first = False
+ else:
+ raise ValueError('null_placement must be "at_start" or "at_end"')
+
+ col_map = _name_to_index_map(schema)
+
+ sorting_columns = []
+
+ for sort_key in sort_keys:
+ if isinstance(sort_key, str):
+ name = sort_key
+ descending = False
+ elif (isinstance(sort_key, tuple) and len(sort_key) == 2 and
+ isinstance(sort_key[0], str) and
+ isinstance(sort_key[1], str)):
+ name, descending = sort_key
+ if descending == "descending":
+ descending = True
+ elif descending == "ascending":
+ descending = False
+ else:
+ raise ValueError("Invalid sort key direction: {0}"
+ .format(descending))
+ else:
+ raise ValueError("Invalid sort key: {0}".format(sort_key))
+
+ try:
+ column_index = col_map[name]
+ except KeyError:
+ raise ValueError("Sort key name '{0}' not found in schema:\n{1}"
+ .format(name, schema))
+
+ sorting_columns.append(
+ cls(column_index, descending=descending, nulls_first=nulls_first)
+ )
+
+ return tuple(sorting_columns)
+
+ @staticmethod
+ def to_sort_order(Schema schema, sorting_columns):
+ """
+ Convert a tuple of SortingColumn objects to the same format as
+ :class:`pyarrow.compute.SortOptions`.
+
+ Parameters
+ ----------
+ schema : Schema
+ Schema of the input data.
+ sorting_columns : tuple of SortingColumn
+ Columns to sort the input on.
+
+ Returns
+ -------
+ sort_keys : tuple of (name, order) tuples
+ null_placement : {'at_start', 'at_end'}
+ """
+ col_map = {i: name for name, i in _name_to_index_map(schema).items()}
+
+ sort_keys = []
+ nulls_first = None
+
+ for sorting_column in sorting_columns:
+ name = col_map[sorting_column.column_index]
+ if sorting_column.descending:
+ order = "descending"
+ else:
+ order = "ascending"
+ sort_keys.append((name, order))
+ if nulls_first is None:
+ nulls_first = sorting_column.nulls_first
+ elif nulls_first != sorting_column.nulls_first:
+ raise ValueError("Sorting columns have inconsistent null placement")
+
+ if nulls_first:
+ null_placement = "at_start"
+ else:
+ null_placement = "at_end"
+
+ return tuple(sort_keys), null_placement
+
+ def __repr__(self):
+ return """{}(column_index={}, descending={}, nulls_first={})""".format(
+ self.__class__.__name__,
+ self.column_index, self.descending, self.nulls_first)
+
+ def __eq__(self, SortingColumn other):
+ return (self.column_index == other.column_index and
+ self.descending == other.descending and
+ self.nulls_first == other.nulls_first)
+
+ def __hash__(self):
+ return hash((self.column_index, self.descending, self.nulls_first))
+
+ @property
+ def column_index(self):
+ """"Index of column data is sorted by (int)."""
+ return self.column_index
+
+ @property
+ def descending(self):
+ """Whether column is sorted in descending order (bool)."""
+ return self.descending
+
+ @property
+ def nulls_first(self):
+ """Whether null values appear before valid values (bool)."""
+ return self.nulls_first
+
+ def to_dict(self):
+ """Convert to dictionary representation."""
+ return {
+ 'column_index': self.column_index,
+ 'descending': self.descending,
+ 'nulls_first': self.nulls_first
+ }
Review Comment:
I don't think so anymore. I'll remove.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on pull request #35453: GH-35331: [Python] Expose Parquet sorting metadata
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on PR #35453:
URL: https://github.com/apache/arrow/pull/35453#issuecomment-1540452046
> Honestly, it would be nice just to be able to get a dataset to be considered "ordered" by its partition values. For example, if I had two tables partitioned by date, it would be nice to be able to join them via a sort merge join (or maybe asof join?) on that date column while using the inherent sorting of the partition values. (Having flashbacks to some frustrations in my days working with PySpark.)
We are quite close to being able to do this. In particular, I think we are very close to being able to take a query like "SELECT MEAN(x) FROM dataset GROUP BY part" and, instead of having aggregation be a pipeline breaker, have it scan values in partition order and emit values as soon as a partition is exhausted. Here is what we have, and what is missing:
* DONE: ExecBatch has an `index` property to determine its sorting.
* DONE: ExecNode has an `ordering` property which says which columns its output are ordered by
* DONE: The scan node has the ability to scan a dataset in order, this means the AsyncGenerator<ExecBatch> will emit exec batches in a deterministic order (filename, batch_in_file)
* NEEDED: The scan node needs to detect if the dataset is partitioned, ask the dataset for fragments ordered by partition, and then assign its `ordering` property based on the partition columns (this sounds harder than it is)
* DONE: The aggregate node is able to define "segment keys" which, if given, it will use to break up the output
* NEEDED: The aggregate node's segment keys currently only work if run in single threaded mode. We need to support parallelism. This should be easily obtainable by using a sequencing queue
* NOTE: The above assumes that the user intelligently crafts the plan (e.g. they know to assign "part" as a segment key when creating the aggregate node). In reality, this is something that would happen in a planner (it would notice that "part" is a partition key, figure out that it can configure the scan to emit in sequence, and then use part when creating the aggregate node). This means this feature will need to be exploited by reasonable intelligent users. However, this is in line with Acero being a dumb execution plan without a planner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #35453: GH-35331: [Python] Expose Parquet sorting metadata
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #35453:
URL: https://github.com/apache/arrow/pull/35453#discussion_r1191692072
##########
python/pyarrow/parquet/core.py:
##########
@@ -867,6 +915,10 @@ def _sanitize_table(table, new_schema, flavor):
it will restore the timezone (Parquet only stores the UTC values without
timezone), or columns with duration type will be restored from the int64
Parquet column.
+sorting_columns : Sequence of SortingColumn, default None
+ Specify the sort order of the data being written. The writer does not sort
+ the data nor does it verify that the data is sorted. The sort order is
Review Comment:
Yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] mapleFU commented on pull request #35453: GH-35331: [Python] Expose Parquet sorting metadata
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #35453:
URL: https://github.com/apache/arrow/pull/35453#issuecomment-1661542196
Seems I and weston all approve this pull request, what is the remaining blocker for this? Can we just rebase and move forward?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org