You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by "raunaqmorarka (via GitHub)" <gi...@apache.org> on 2023/09/21 03:34:55 UTC
[GitHub] [parquet-format] raunaqmorarka opened a new pull request, #216: PARQUET-2352: Allow truncation of row group min_values/max_value statistics
raunaqmorarka opened a new pull request, #216:
URL: https://github.com/apache/parquet-format/pull/216
### Jira
- https://issues.apache.org/jira/browse/PARQUET-2352
This updates the spec to allow truncation of row group min_values/max_value statistics so that readers can take advantage of row group pruning for predicates on columns containing long strings.
https://issues.apache.org/jira/browse/PARQUET-1685 already introduced a feature to parquet-mr which allows users to deviate from the current spec and configure truncation of row group statistics.
Since the possibility of truncation exists and is not possible to explicitly detect, attempts to pushdown min/max aggregation to parquet have avoided implementing it for string columns (e.g. https://issues.apache.org/jira/browse/SPARK-36645)
Given the above situation, the spec should be updated to allow truncation of min/max row group stats. This would align the spec with current reality that string column min/max row group stats could be truncated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PARQUET-2352: Allow truncation of row group min_values/max_value statistics [parquet-format]
Posted by "raunaqmorarka (via GitHub)" <gi...@apache.org>.
raunaqmorarka commented on code in PR #216:
URL: https://github.com/apache/parquet-format/pull/216#discussion_r1343751395
##########
src/main/thrift/parquet.thrift:
##########
@@ -216,13 +216,22 @@ struct Statistics {
/** count of distinct values occurring */
4: optional i64 distinct_count;
/**
- * Min and max values for the column, determined by its ColumnOrder.
+ * lower and upper bound values for the column, determined by its ColumnOrder.
+ * These may be the actual minimum and maximum values found on a column chunk,
+ * but can also be (more compact) values that do not exist on a column chunk.
+ * For example, instead of storing "Blart Versenwald III", a writer may set
+ * min_value="B", max_value="C". Such more compact values must still be valid
+ * values within the column's logical type.
*
* Values are encoded using PLAIN encoding, except that variable-length byte
* arrays do not include a length prefix.
*/
5: optional binary max_value;
6: optional binary min_value;
+ /** If true, max_value is the actual maximum value found on a column chunk **/
+ 7: optional bool is_max_value_exact;
+ /** If true, min_value is the actual minimum value found on a column chunk **/
+ 8: optional bool is_min_value_exact;
Review Comment:
I think these fields should be empty whenever max_value/min_value are themselves empty. Some writer implementations may choose to leave this empty even after populating min/max, in that case the readers should assume that the value is not exact for safety.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-format] raunaqmorarka commented on a diff in pull request #216: PARQUET-2352: Allow truncation of row group min_values/max_value statistics
Posted by "raunaqmorarka (via GitHub)" <gi...@apache.org>.
raunaqmorarka commented on code in PR #216:
URL: https://github.com/apache/parquet-format/pull/216#discussion_r1335484908
##########
src/main/thrift/parquet.thrift:
##########
@@ -216,7 +216,12 @@ struct Statistics {
/** count of distinct values occurring */
4: optional i64 distinct_count;
/**
- * Min and max values for the column, determined by its ColumnOrder.
+ * lower and upper bound values for the column, determined by its ColumnOrder.
Review Comment:
@wgtmac @gszadovszky could you please explain why the addition of a flag for truncation needs to happen together with the proposed change here ?
I believe https://issues.apache.org/jira/browse/PARQUET-1685 already broke the current spec by allowing for truncation of min/max for strings here. Even after we add a flag to indicate truncation, no application can safely assume that the min/max stats in existing parquet files are not truncated for strings. So I don't get why the addition of that flag shouldn't go as a separate change to the spec.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PARQUET-2352: Allow truncation of row group min_values/max_value statistics [parquet-format]
Posted by "raunaqmorarka (via GitHub)" <gi...@apache.org>.
raunaqmorarka commented on code in PR #216:
URL: https://github.com/apache/parquet-format/pull/216#discussion_r1363572486
##########
src/main/thrift/parquet.thrift:
##########
@@ -216,7 +216,12 @@ struct Statistics {
/** count of distinct values occurring */
4: optional i64 distinct_count;
/**
- * Min and max values for the column, determined by its ColumnOrder.
+ * lower and upper bound values for the column, determined by its ColumnOrder.
Review Comment:
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-format] raunaqmorarka commented on a diff in pull request #216: PARQUET-2352: Allow truncation of row group min_values/max_value statistics
Posted by "raunaqmorarka (via GitHub)" <gi...@apache.org>.
raunaqmorarka commented on code in PR #216:
URL: https://github.com/apache/parquet-format/pull/216#discussion_r1333328796
##########
src/main/thrift/parquet.thrift:
##########
@@ -216,7 +216,12 @@ struct Statistics {
/** count of distinct values occurring */
4: optional i64 distinct_count;
/**
- * Min and max values for the column, determined by its ColumnOrder.
+ * lower and upper bound values for the column, determined by its ColumnOrder.
Review Comment:
Given the feature in https://issues.apache.org/jira/browse/PARQUET-1685, I want to assume that all existing stats are truncated. Going forward we should have a flag to explicitly indicate whether or not truncation took place and applications should perform aggregation pushdown only if that flag is found to indicate no truncation. But I think adding that flag can be tackled separately as a follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PARQUET-2352: Allow truncation of row group min_values/max_value statistics [parquet-format]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #216:
URL: https://github.com/apache/parquet-format/pull/216#discussion_r1349498483
##########
src/main/thrift/parquet.thrift:
##########
@@ -216,13 +216,22 @@ struct Statistics {
/** count of distinct values occurring */
4: optional i64 distinct_count;
/**
- * Min and max values for the column, determined by its ColumnOrder.
+ * lower and upper bound values for the column, determined by its ColumnOrder.
+ * These may be the actual minimum and maximum values found on a column chunk,
+ * but can also be (more compact) values that do not exist on a column chunk.
+ * For example, instead of storing "Blart Versenwald III", a writer may set
+ * min_value="B", max_value="C". Such more compact values must still be valid
+ * values within the column's logical type.
*
* Values are encoded using PLAIN encoding, except that variable-length byte
* arrays do not include a length prefix.
*/
5: optional binary max_value;
6: optional binary min_value;
+ /** If true, max_value is the actual maximum value found on a column chunk **/
+ 7: optional bool is_max_value_exact;
+ /** If true, min_value is the actual minimum value found on a column chunk **/
+ 8: optional bool is_min_value_exact;
Review Comment:
The statistics object is also used in the page header, so the `column chunk` in the comment may not be correct in that case. Otherwise LGTM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-format] raunaqmorarka commented on a diff in pull request #216: PARQUET-2352: Allow truncation of row group min_values/max_value statistics
Posted by "raunaqmorarka (via GitHub)" <gi...@apache.org>.
raunaqmorarka commented on code in PR #216:
URL: https://github.com/apache/parquet-format/pull/216#discussion_r1341907887
##########
src/main/thrift/parquet.thrift:
##########
@@ -216,7 +216,12 @@ struct Statistics {
/** count of distinct values occurring */
4: optional i64 distinct_count;
/**
- * Min and max values for the column, determined by its ColumnOrder.
+ * lower and upper bound values for the column, determined by its ColumnOrder.
Review Comment:
I've updated this PR to now include is_max_value_exact/is_min_value_exact for specifying when the max_value/min_value are the actual max and min values
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-format] findepi commented on pull request #216: PARQUET-2352: Allow truncation of row group min_values/max_value statistics
Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi commented on PR #216:
URL: https://github.com/apache/parquet-format/pull/216#issuecomment-1730285274
Thanks @raunaqmorarka, this lgtm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-format] wgtmac commented on a diff in pull request #216: PARQUET-2352: Allow truncation of row group min_values/max_value statistics
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #216:
URL: https://github.com/apache/parquet-format/pull/216#discussion_r1333778393
##########
src/main/thrift/parquet.thrift:
##########
@@ -216,7 +216,12 @@ struct Statistics {
/** count of distinct values occurring */
4: optional i64 distinct_count;
/**
- * Min and max values for the column, determined by its ColumnOrder.
+ * lower and upper bound values for the column, determined by its ColumnOrder.
Review Comment:
I think they are relevant and can be discussed together.
What do you think? @gszadovszky @shangxinli
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-format] wgtmac commented on a diff in pull request #216: PARQUET-2352: Allow truncation of row group min_values/max_value statistics
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #216:
URL: https://github.com/apache/parquet-format/pull/216#discussion_r1333286445
##########
src/main/thrift/parquet.thrift:
##########
@@ -216,7 +216,12 @@ struct Statistics {
/** count of distinct values occurring */
4: optional i64 distinct_count;
/**
- * Min and max values for the column, determined by its ColumnOrder.
+ * lower and upper bound values for the column, determined by its ColumnOrder.
Review Comment:
IIRC, you want to allow truncation by adding a flag to indicate whether the min_value/max_value is truncated or not, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-format] gszadovszky commented on a diff in pull request #216: PARQUET-2352: Allow truncation of row group min_values/max_value statistics
Posted by "gszadovszky (via GitHub)" <gi...@apache.org>.
gszadovszky commented on code in PR #216:
URL: https://github.com/apache/parquet-format/pull/216#discussion_r1335662614
##########
src/main/thrift/parquet.thrift:
##########
@@ -216,7 +216,12 @@ struct Statistics {
/** count of distinct values occurring */
4: optional i64 distinct_count;
/**
- * Min and max values for the column, determined by its ColumnOrder.
+ * lower and upper bound values for the column, determined by its ColumnOrder.
Review Comment:
I am fine going in separately but it needs to be released together. I think, the additional flag would complete fixing this situation. If an implementation uses min/max values as existing values and realize this spec change we would still give something to them to continue their approach. I agree if the flag is not there we cannot assume the min/max values are actual values.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PARQUET-2352: Allow truncation of row group min_values/max_value statistics [parquet-format]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #216:
URL: https://github.com/apache/parquet-format/pull/216#discussion_r1349706308
##########
src/main/thrift/parquet.thrift:
##########
@@ -216,13 +216,22 @@ struct Statistics {
/** count of distinct values occurring */
4: optional i64 distinct_count;
/**
- * Min and max values for the column, determined by its ColumnOrder.
+ * lower and upper bound values for the column, determined by its ColumnOrder.
+ * These may be the actual minimum and maximum values found on a column chunk,
+ * but can also be (more compact) values that do not exist on a column chunk.
+ * For example, instead of storing "Blart Versenwald III", a writer may set
+ * min_value="B", max_value="C". Such more compact values must still be valid
+ * values within the column's logical type.
*
* Values are encoded using PLAIN encoding, except that variable-length byte
* arrays do not include a length prefix.
*/
5: optional binary max_value;
6: optional binary min_value;
+ /** If true, max_value is the actual maximum value found on a column chunk **/
+ 7: optional bool is_max_value_exact;
+ /** If true, min_value is the actual minimum value found on a column chunk **/
Review Comment:
```suggestion
/** If true, min_value is the actual minimum value for a column */
```
##########
src/main/thrift/parquet.thrift:
##########
@@ -216,13 +216,22 @@ struct Statistics {
/** count of distinct values occurring */
4: optional i64 distinct_count;
/**
- * Min and max values for the column, determined by its ColumnOrder.
+ * lower and upper bound values for the column, determined by its ColumnOrder.
+ * These may be the actual minimum and maximum values found on a column chunk,
+ * but can also be (more compact) values that do not exist on a column chunk.
+ * For example, instead of storing "Blart Versenwald III", a writer may set
+ * min_value="B", max_value="C". Such more compact values must still be valid
+ * values within the column's logical type.
Review Comment:
```suggestion
* Lower and upper bound values for the column, determined by its ColumnOrder.
*
* These may be the actual minimum and maximum values found on a page or column
* chunk, but can also be (more compact) values that do not exist on a page or
* column chunk. For example, instead of storing "Blart Versenwald III", a writer
* may set min_value="B", max_value="C". Such more compact values must still be
* valid values within the column's logical type.
```
##########
src/main/thrift/parquet.thrift:
##########
@@ -216,13 +216,22 @@ struct Statistics {
/** count of distinct values occurring */
4: optional i64 distinct_count;
/**
- * Min and max values for the column, determined by its ColumnOrder.
+ * lower and upper bound values for the column, determined by its ColumnOrder.
+ * These may be the actual minimum and maximum values found on a column chunk,
+ * but can also be (more compact) values that do not exist on a column chunk.
+ * For example, instead of storing "Blart Versenwald III", a writer may set
+ * min_value="B", max_value="C". Such more compact values must still be valid
+ * values within the column's logical type.
*
* Values are encoded using PLAIN encoding, except that variable-length byte
* arrays do not include a length prefix.
*/
5: optional binary max_value;
6: optional binary min_value;
+ /** If true, max_value is the actual maximum value found on a column chunk **/
Review Comment:
```suggestion
/** If true, max_value is the actual maximum value for a column */
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PARQUET-2352: Allow truncation of row group min_values/max_value statistics [parquet-format]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #216:
URL: https://github.com/apache/parquet-format/pull/216#discussion_r1343638311
##########
src/main/thrift/parquet.thrift:
##########
@@ -216,13 +216,22 @@ struct Statistics {
/** count of distinct values occurring */
4: optional i64 distinct_count;
/**
- * Min and max values for the column, determined by its ColumnOrder.
+ * lower and upper bound values for the column, determined by its ColumnOrder.
+ * These may be the actual minimum and maximum values found on a column chunk,
+ * but can also be (more compact) values that do not exist on a column chunk.
+ * For example, instead of storing "Blart Versenwald III", a writer may set
+ * min_value="B", max_value="C". Such more compact values must still be valid
+ * values within the column's logical type.
*
* Values are encoded using PLAIN encoding, except that variable-length byte
* arrays do not include a length prefix.
*/
5: optional binary max_value;
6: optional binary min_value;
+ /** If true, max_value is the actual maximum value found on a column chunk **/
+ 7: optional bool is_max_value_exact;
+ /** If true, min_value is the actual minimum value found on a column chunk **/
+ 8: optional bool is_min_value_exact;
Review Comment:
if this is not exists, we don't know that if the min-max is exact?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PARQUET-2352: Allow truncation of row group min_values/max_value statistics [parquet-format]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #216:
URL: https://github.com/apache/parquet-format/pull/216#discussion_r1363545890
##########
src/main/thrift/parquet.thrift:
##########
@@ -216,7 +216,12 @@ struct Statistics {
/** count of distinct values occurring */
4: optional i64 distinct_count;
/**
- * Min and max values for the column, determined by its ColumnOrder.
+ * lower and upper bound values for the column, determined by its ColumnOrder.
Review Comment:
I just merged it. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PARQUET-2352: Allow truncation of row group min_values/max_value statistics [parquet-format]
Posted by "raunaqmorarka (via GitHub)" <gi...@apache.org>.
raunaqmorarka commented on code in PR #216:
URL: https://github.com/apache/parquet-format/pull/216#discussion_r1347903866
##########
src/main/thrift/parquet.thrift:
##########
@@ -216,7 +216,12 @@ struct Statistics {
/** count of distinct values occurring */
4: optional i64 distinct_count;
/**
- * Min and max values for the column, determined by its ColumnOrder.
+ * lower and upper bound values for the column, determined by its ColumnOrder.
Review Comment:
@wgtmac @gszadovszky can you please take a look at current changes ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-format] findepi commented on a diff in pull request #216: PARQUET-2352: Allow truncation of row group min_values/max_value statistics
Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi commented on code in PR #216:
URL: https://github.com/apache/parquet-format/pull/216#discussion_r1333952085
##########
src/main/thrift/parquet.thrift:
##########
@@ -216,7 +216,12 @@ struct Statistics {
/** count of distinct values occurring */
4: optional i64 distinct_count;
/**
- * Min and max values for the column, determined by its ColumnOrder.
+ * lower and upper bound values for the column, determined by its ColumnOrder.
Review Comment:
@mapleFU very good point, we should not allow truncation to produce a value that is invalid for given logical type
i think this is what @raunaqmorarka meant with `Such more compact values must still be valid values within the column's logical type.` in the code comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-format] mapleFU commented on a diff in pull request #216: PARQUET-2352: Allow truncation of row group min_values/max_value statistics
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #216:
URL: https://github.com/apache/parquet-format/pull/216#discussion_r1333823691
##########
src/main/thrift/parquet.thrift:
##########
@@ -216,7 +216,12 @@ struct Statistics {
/** count of distinct values occurring */
4: optional i64 distinct_count;
/**
- * Min and max values for the column, determined by its ColumnOrder.
+ * lower and upper bound values for the column, determined by its ColumnOrder.
Review Comment:
Might not directly related to this question. If the column is a utf-8 column ( with LogicalType `UTF8` ), can the binary here be a non-utf8 bytes?
e.g. `s[...100]` is a valid utf-8, however, `s[...50]` is not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PARQUET-2352: Allow truncation of row group min_values/max_value statistics [parquet-format]
Posted by "raunaqmorarka (via GitHub)" <gi...@apache.org>.
raunaqmorarka commented on code in PR #216:
URL: https://github.com/apache/parquet-format/pull/216#discussion_r1363143224
##########
src/main/thrift/parquet.thrift:
##########
@@ -216,7 +216,12 @@ struct Statistics {
/** count of distinct values occurring */
4: optional i64 distinct_count;
/**
- * Min and max values for the column, determined by its ColumnOrder.
+ * lower and upper bound values for the column, determined by its ColumnOrder.
Review Comment:
@wgtmac @gszadovszky is anything more needed to merge this ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PARQUET-2352: Allow truncation of row group min_values/max_value statistics [parquet-format]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac merged PR #216:
URL: https://github.com/apache/parquet-format/pull/216
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PARQUET-2352: Allow truncation of row group min_values/max_value statistics [parquet-format]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #216:
URL: https://github.com/apache/parquet-format/pull/216#issuecomment-1824221453
So seems that for `PageHeader` (though if PageIndex enabled, we might not write page statistics) might also have to write these two statistics?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PARQUET-2352: Allow truncation of row group min_values/max_value statistics [parquet-format]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #216:
URL: https://github.com/apache/parquet-format/pull/216#issuecomment-1762928233
cc @pitrou @emkornfield @shangxinli
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-format] gszadovszky commented on a diff in pull request #216: PARQUET-2352: Allow truncation of row group min_values/max_value statistics
Posted by "gszadovszky (via GitHub)" <gi...@apache.org>.
gszadovszky commented on code in PR #216:
URL: https://github.com/apache/parquet-format/pull/216#discussion_r1335458337
##########
src/main/thrift/parquet.thrift:
##########
@@ -216,7 +216,12 @@ struct Statistics {
/** count of distinct values occurring */
4: optional i64 distinct_count;
/**
- * Min and max values for the column, determined by its ColumnOrder.
+ * lower and upper bound values for the column, determined by its ColumnOrder.
Review Comment:
I agree, @wgtmac. I think, we should add the flag (if everyone agrees) along with the update of the min/max value spec.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PARQUET-2352: Allow truncation of row group min_values/max_value statistics [parquet-format]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #216:
URL: https://github.com/apache/parquet-format/pull/216#issuecomment-1853407048
Also, seems that this property only works for ByteArray, right (maybe FLBA also included)
https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L965-L979
The syntax like this is not included in `exact`, right( at least in current arrow-rs impl)? @raunaqmorarka @tustvold
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org