You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@parquet.apache.org by "alkis (via GitHub)" <gi...@apache.org> on 2024/05/21 15:25:44 UTC
[PR] DRAFT: Incremental improvements to parquet metadata [parquet-format]
alkis opened a new pull request, #248:
URL: https://github.com/apache/parquet-format/pull/248
### Incremental parquet metadata improvements
This is an alternative proposal to https://github.com/apache/parquet-format/pull/242 which can be executed with minimal changes to parquet readers/writers.
Wide schemata (large number of columns) make `FileMetadata` very slow to parse. The majority of the time is spent in parsing thrift `list<>` and `binary` fields. These are notoriously slow to decode because they are variable sized. In this proposal we avoid such fields as much as possible to improve decoding. In addition we allow columns that do not participate in a row group to have their column chunk metadata skipped.
### Jira
- [ ] My PR addresses the following [Parquet Jira](https://issues.apache.org/jira/browse/PARQUET/) issues and references them in the PR title. For example, "PARQUET-1234: My Parquet PR"
- https://issues.apache.org/jira/browse/PARQUET-XXX
- In case you are adding a dependency, check if the license complies with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
### Commits
- [ ] My commits all reference Jira issues in their subject lines. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes how to use it.
- All the public functions and the classes in the PR contain Javadoc that explain what it does
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] DRAFT: Incremental improvements to parquet metadata [parquet-format]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #248:
URL: https://github.com/apache/parquet-format/pull/248#discussion_r1617768365
##########
src/main/thrift/parquet.thrift:
##########
@@ -242,43 +242,42 @@ struct SizeStatistics {
* All fields are optional.
*/
struct Statistics {
- /**
- * DEPRECATED: min and max value of the column. Use min_value and max_value.
- *
- * Values are encoded using PLAIN encoding, except that variable-length byte
- * arrays do not include a length prefix.
- *
- * These fields encode min and max values determined by signed comparison
- * only. New files should use the correct order for a column's logical type
- * and store the values in the min_value and max_value fields.
- *
- * To support older readers, these may be set when the column order is
- * signed.
- */
+ /* DEPRECATED: do not use */
1: optional binary max;
2: optional binary min;
/** count of null value in the column */
3: optional i64 null_count;
/** count of distinct values occurring */
4: optional i64 distinct_count;
/**
- * Lower and upper bound values for the column, determined by its ColumnOrder.
+ * Only one pair of max_value/min_value, max1/min1, max2/min2, max4/min4,
+ * max8/min8 can be set. The pair is determined by the physical type of the
+ * column. Floating point values are bitcasted to integers. Variable length
+ * values are set in min_value/max_value.
+ *
+ * Min and Max are the lower and upper bound values for the column,
Review Comment:
I would think not. I can definitely see a use for fixed length byte array as well. Also a lazy implementation could set min to 0 and max to INT_MAX to indicate only positive values are present.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] DRAFT: Incremental improvements to parquet metadata [parquet-format]
Posted by "alkis (via GitHub)" <gi...@apache.org>.
alkis closed pull request #248: DRAFT: Incremental improvements to parquet metadata
URL: https://github.com/apache/parquet-format/pull/248
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] DRAFT: Incremental improvements to parquet metadata [parquet-format]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #248:
URL: https://github.com/apache/parquet-format/pull/248#issuecomment-2135316937
> The majority of the time is spent in parsing thrift `list<>` and `binary` fields. These are notoriously slow to decode because they are variable sized.
Do you have actual data to support this? For `list` I could understand (but the problem is really the number of nested elements), but for `binary` this sounds rather unexpected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] DRAFT: Incremental improvements to parquet metadata [parquet-format]
Posted by "alkis (via GitHub)" <gi...@apache.org>.
alkis commented on PR #248:
URL: https://github.com/apache/parquet-format/pull/248#issuecomment-2135669062
> Do you have actual data to support this? For `list` I could understand (but the problem is really the number of nested elements), but for `binary` this sounds rather unexpected.
The slowness is actually `list<>` and particularly `list<StructType>` with nested lists. `binary` is not as large of a problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] DRAFT: Incremental improvements to parquet metadata [parquet-format]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #248:
URL: https://github.com/apache/parquet-format/pull/248#discussion_r1610425521
##########
src/main/thrift/parquet.thrift:
##########
@@ -242,43 +242,42 @@ struct SizeStatistics {
* All fields are optional.
*/
struct Statistics {
- /**
- * DEPRECATED: min and max value of the column. Use min_value and max_value.
- *
- * Values are encoded using PLAIN encoding, except that variable-length byte
- * arrays do not include a length prefix.
- *
- * These fields encode min and max values determined by signed comparison
- * only. New files should use the correct order for a column's logical type
- * and store the values in the min_value and max_value fields.
- *
- * To support older readers, these may be set when the column order is
- * signed.
- */
+ /* DEPRECATED: do not use */
1: optional binary max;
2: optional binary min;
/** count of null value in the column */
3: optional i64 null_count;
/** count of distinct values occurring */
4: optional i64 distinct_count;
/**
- * Lower and upper bound values for the column, determined by its ColumnOrder.
+ * Only one pair of max_value/min_value, max1/min1, max2/min2, max4/min4,
+ * max8/min8 can be set. The pair is determined by the physical type of the
+ * column. Floating point values are bitcasted to integers. Variable length
+ * values are set in min_value/max_value.
+ *
+ * Min and Max are the lower and upper bound values for the column,
+ * respectively, as determined by its ColumnOrder.
*
* These may be the actual minimum and maximum values found on a page or column
* chunk, but can also be (more compact) values that do not exist on a page or
* column chunk. For example, instead of storing "Blart Versenwald III", a writer
* may set min_value="B", max_value="C". Such more compact values must still be
* valid values within the column's logical type.
- *
- * Values are encoded using PLAIN encoding, except that variable-length byte
- * arrays do not include a length prefix.
*/
5: optional binary max_value;
6: optional binary min_value;
/** If true, max_value is the actual maximum value for a column */
7: optional bool is_max_value_exact;
/** If true, min_value is the actual minimum value for a column */
8: optional bool is_min_value_exact;
+ 9: optional byte max1;
+ 10: optional byte min1;
+ 11: optional i16 max2;
+ 12: optional i16 min2;
+ 13: optional i32 max4;
+ 14: optional i32 min4;
+ 15: optional i64 max8;
+ 16: optional i64 min8;
Review Comment:
At first I was thinking a union could be used for these, which would reduce the (logical) complexity some (it would also include the binary pair), but would add encoding overhead. But how about a single fixed-width i64 pair. These are zig-zag encoded anyway, so a single byte value won't take any extra space in the file. And then there would be fewer members in the thrift struct as well. We could save a byte in the file for booleans, but then that adds to the struct as well, so probably not worth adding a bool pair.
##########
src/main/thrift/parquet.thrift:
##########
@@ -833,6 +832,9 @@ struct ColumnMetaData {
* filter pushdown.
*/
16: optional SizeStatistics size_statistics;
+
+ /* The index into FileMetadata.schema (list<SchemaElement>) for this column */
+ 17: optional i32 schema_index;
Review Comment:
I would have found this helpful :smile: But I'm sure others would argue that it's easy enough to create an in memory map during schema parsing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] DRAFT: Incremental improvements to parquet metadata [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #248:
URL: https://github.com/apache/parquet-format/pull/248#discussion_r1617758127
##########
src/main/thrift/parquet.thrift:
##########
@@ -242,43 +242,42 @@ struct SizeStatistics {
* All fields are optional.
*/
struct Statistics {
- /**
- * DEPRECATED: min and max value of the column. Use min_value and max_value.
- *
- * Values are encoded using PLAIN encoding, except that variable-length byte
- * arrays do not include a length prefix.
- *
- * These fields encode min and max values determined by signed comparison
- * only. New files should use the correct order for a column's logical type
- * and store the values in the min_value and max_value fields.
- *
- * To support older readers, these may be set when the column order is
- * signed.
- */
+ /* DEPRECATED: do not use */
1: optional binary max;
2: optional binary min;
/** count of null value in the column */
3: optional i64 null_count;
/** count of distinct values occurring */
4: optional i64 distinct_count;
/**
- * Lower and upper bound values for the column, determined by its ColumnOrder.
+ * Only one pair of max_value/min_value, max1/min1, max2/min2, max4/min4,
+ * max8/min8 can be set. The pair is determined by the physical type of the
+ * column. Floating point values are bitcasted to integers. Variable length
+ * values are set in min_value/max_value.
+ *
+ * Min and Max are the lower and upper bound values for the column,
Review Comment:
does this imply is_min_value_exact is only used for variable length values?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] DRAFT: Incremental improvements to parquet metadata [parquet-format]
Posted by "alkis (via GitHub)" <gi...@apache.org>.
alkis commented on code in PR #248:
URL: https://github.com/apache/parquet-format/pull/248#discussion_r1618759771
##########
src/main/thrift/parquet.thrift:
##########
@@ -833,6 +832,9 @@ struct ColumnMetaData {
* filter pushdown.
*/
16: optional SizeStatistics size_statistics;
+
+ /* The index into FileMetadata.schema (list<SchemaElement>) for this column */
+ 17: optional i32 schema_index;
Review Comment:
With this added we can skip metadata for columns that do not participate in a row group. Without this index we won't be able to reconstruct the map.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] DRAFT: Incremental improvements to parquet metadata [parquet-format]
Posted by "alkis (via GitHub)" <gi...@apache.org>.
alkis commented on PR #248:
URL: https://github.com/apache/parquet-format/pull/248#issuecomment-2138133203
I am retracting this PR in favor of:
https://docs.google.com/document/d/1PQpY418LkIDHMFYCY8ne_G-CFpThK15LLpzWYbc7rFU/edit
and:
https://github.com/apache/parquet-format/pull/252
https://github.com/apache/parquet-format/pull/253
https://github.com/apache/parquet-format/pull/254
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] DRAFT: Incremental improvements to parquet metadata [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #248:
URL: https://github.com/apache/parquet-format/pull/248#discussion_r1618916332
##########
src/main/thrift/parquet.thrift:
##########
@@ -833,6 +832,9 @@ struct ColumnMetaData {
* filter pushdown.
*/
16: optional SizeStatistics size_statistics;
+
+ /* The index into FileMetadata.schema (list<SchemaElement>) for this column */
+ 17: optional i32 schema_index;
Review Comment:
Just to be clear this would be a future optimization to drop columns that are completely null?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org