You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@parquet.apache.org by "mapleFU (via GitHub)" <gi...@apache.org> on 2024/08/07 07:25:52 UTC
[PR] Add test file with sorting columns [parquet-testing]
mapleFU opened a new pull request, #56:
URL: https://github.com/apache/parquet-testing/pull/56
Generate script: (with pyarrow 16.1.0 )
```python
>>> import pyarrow as pa
>>> import pyarrow.parquet as pq
>>> sorting_columns = (pq.SortingColumn(column_index=0, descending=True, nulls_first=True), pq.SortingColumn(column_index=1, descending=False))
>>> table = pa.table({'a': [null, 2, 1], 'b': ['a', 'b', 'c']})
>>> pq.write_table(table, "sorting_columns.parquet", sorting_columns=sorting_columns)
```
File Metadata:
```
{
"Version": "2.6",
"CreatedBy": "parquet-cpp-arrow version 16.1.0",
"TotalRows": "3",
"NumberOfRowGroups": "1",
"NumberOfRealColumns": "2",
"NumberOfColumns": "2",
"Columns": [
{ "Id": "0", "Name": "a", "PhysicalType": "INT64", "ConvertedType": "NONE", "LogicalType": {"Type": "None"} },
{ "Id": "1", "Name": "b", "PhysicalType": "BYTE_ARRAY", "ConvertedType": "UTF8", "LogicalType": {"Type": "String"} }
],
"RowGroups": [
{
"Id": "0", "TotalBytes": "166", "TotalCompressedBytes": "174", "SortColumns": ["{"column_idx":0, "descending":1, "nulls_first": 1}, "{"column_idx":1, "descending":0, "nulls_first": 0}], "Rows": "3",
"ColumnChunks": [
{"Id": "0", "Values": "3", "StatsSet": "True", "Stats": {"NumNulls": "1", "Max": "2", "Min": "1" },
"Compression": "SNAPPY", "Encodings": "PLAIN(DICT_PAGE) RLE_DICTIONARY", "UncompressedSize": "100", "CompressedSize": "104" },
{"Id": "1", "Values": "3", "StatsSet": "True", "Stats": {"NumNulls": "0", "Max": "c", "Min": "a" },
"Compression": "SNAPPY", "Encodings": "PLAIN(DICT_PAGE) RLE_DICTIONARY", "UncompressedSize": "66", "CompressedSize": "70" }
]
}
]
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Add test file with sorting columns [parquet-testing]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #56:
URL: https://github.com/apache/parquet-testing/pull/56#issuecomment-2275046710
I think it is fine to use the same properties across all row groups.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Add test file with sorting columns [parquet-testing]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #56:
URL: https://github.com/apache/parquet-testing/pull/56#issuecomment-2273841874
If might be useful to have multiple row groups.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Add test file with sorting columns [parquet-testing]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #56:
URL: https://github.com/apache/parquet-testing/pull/56#issuecomment-2277263369
@wgtmac @pitrou
I've change to use 2 row-groups with same data. Pr description is changed for this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Add test file with sorting columns [parquet-testing]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #56:
URL: https://github.com/apache/parquet-testing/pull/56#discussion_r1712948175
##########
data/README.md:
##########
@@ -52,6 +52,7 @@
| byte_stream_split.zstd.parquet | Standard normals with `BYTE_STREAM_SPLIT` encoding. See [note](#byte-stream-split) below |
| incorrect_map_schema.parquet | Contains a Map schema without explicitly required keys, produced by Presto. See [note](#incorrect-map-schema) |
| column_chunk_key_value_metadata.parquet | two INT32 columns, one with column chunk key-value metadata {"foo": "bar", "thisiskeywithoutvalue": null} note that the second key "thisiskeywithoutvalue", does not have a value, but the value can be mapped to an empty string "" when read depending on the client |
+| sorting_columns.parquet | INT64 and BYTE_ARRAY columns with first column with nulls_first and descending, second column with not nulls first and ascending. This file contains two row-groups with same data and sorting_columns. |
Review Comment:
```suggestion
| sorting_columns.parquet | INT64 and BYTE_ARRAY columns with first column with nulls first and descending, second column with nulls last and ascending. This file contains two row groups with same data and sorting columns. |
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Add test file with sorting columns [parquet-testing]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #56:
URL: https://github.com/apache/parquet-testing/pull/56#issuecomment-2286704744
I would merge this first. Feel free to edit or overwrite this if has any issue on this file or description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Add test file with sorting columns [parquet-testing]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU merged PR #56:
URL: https://github.com/apache/parquet-testing/pull/56
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Add test file with sorting columns [parquet-testing]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #56:
URL: https://github.com/apache/parquet-testing/pull/56#issuecomment-2272801873
cc @wgtmac @pitrou
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Add test file with sorting columns [parquet-testing]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #56:
URL: https://github.com/apache/parquet-testing/pull/56#issuecomment-2274944578
> If might be useful to have multiple row groups.
Currently, we're not able to create file with different sort-props in multiple row-groups. Should I first implement that or just same sort-properties in two row-groups?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org