You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "sopel39 (via GitHub)" <gi...@apache.org> on 2024/08/13 10:48:31 UTC
[I] Specify in lower/upper bounds in data_file struct are exact [iceberg]
sopel39 opened a new issue, #10930:
URL: https://github.com/apache/iceberg/issues/10930
### Proposed Change
There is a need to perform exploratory aggregation queries on tables with `min/max` aggregations on data columns. Currently spec for `data_file` struct specifies the following meaning for `lower_bounds` and `upper_bounds`:
_Map from column id to lower bound in the column serialized as binary [1].
Each value **MUST BE LESS OR EQUAL** to all non-null, non-NaN values in the column for the file [2]_
Hence the values stored in `lower_bounds` and `upper_bounds` cannot be reliably used in aggregation pushdowns. Additionally, these stats can be truncated per https://iceberg.apache.org/docs/1.5.2/configuration/#write-properties
Even though these bound are unreliable, they are still used in Iceberg's Spark connector itself: https://github.com/apache/iceberg/blob/main/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java#L309 which most likely has silent correctness issues.
I propose to add two new, optional fields along with `lower_bounds` and `upper_bounds`:
```
map<int, boolean> lower_bound_exact
map<int, boolean> upper_bound_exact
```
which would specify if bounds are exact for a particular data file. This would allow to reliably perform aggregation pushdown to Iceberg by engines. By default `lower_bounds` and `upper_bounds` should be treated as non-exact by engines.
### Proposal document
_No response_
### Specifications
- [X] Table
- [ ] View
- [ ] REST
- [ ] Puffin
- [ ] Encryption
- [X] Other
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Specify in lower/upper bounds in data_file struct are exact [iceberg]
Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi commented on issue #10930:
URL: https://github.com/apache/iceberg/issues/10930#issuecomment-2295721068
i do you mean min/max and count from the Iceberg metadata are already used for query planning? though none of them used for the agg pushdown?
@osscm Correct.
Basic idea of data pruning for aggregation queries:
during planning of a `max(v)` query, do not create splits for files which have `v`'s _higher bound_ that is lower than some other split's `v`'s _lower bound_.
Obviously this wouldn't be as efficient when splits have greatly overlapping `v` values (e.g. a hash value), as the pruning condition would rarely trigger. It's unclear to me how often such "random" values are used for dashboarding though.
So it is possible that this strategy is as efficient as we can do for the workloads we care about, yet doesn't require spec changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Specify in lower/upper bounds in data_file struct are exact [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #10930:
URL: https://github.com/apache/iceberg/issues/10930#issuecomment-2692483070
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Specify in lower/upper bounds in data_file struct are exact [iceberg]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on issue #10930:
URL: https://github.com/apache/iceberg/issues/10930#issuecomment-2295114217
> which would specify if bounds are exact for a particular data file. This would allow to reliably perform aggregation pushdown to Iceberg by engines. By default lower_bounds and upper_bounds should be treated as non-exact by engines.
+1, IMHO its not possible to track with what metric configuration a particular data file was created, having a dedicated field would be great though optional drops a ball when different writers are concerned, but I think lower_bound and upper_bound are optional in first place :) so we should be good in that aspect !
> how much real-world improvement it would bring, cause it's not like the current file-level metadata is useless for aggregation queries speedup
Looking forward to it in spark atleast we replace the scan to Local Scan so actually don't read any data file, not sure how trino implements it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Specify in lower/upper bounds in data_file struct are exact [iceberg]
Posted by "sopel39 (via GitHub)" <gi...@apache.org>.
sopel39 commented on issue #10930:
URL: https://github.com/apache/iceberg/issues/10930#issuecomment-2285948879
cc @findepi @osscm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Specify in lower/upper bounds in data_file struct are exact [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #10930: Specify in lower/upper bounds in data_file struct are exact
URL: https://github.com/apache/iceberg/issues/10930
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Specify in lower/upper bounds in data_file struct are exact [iceberg]
Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi commented on issue #10930:
URL: https://github.com/apache/iceberg/issues/10930#issuecomment-2291761633
I like the idea. It would help process aggregation queries eg in Trino (but only when there are no deletions).
Before committing to such spec change, it would be great to have good understanding on how much real-world improvement it would bring, cause it's not like the current file-level metadata is useless for aggregation queries speedup. It's just that eg in Trino it's not utilized to the maximum yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Specify in lower/upper bounds in data_file struct are exact [iceberg]
Posted by "osscm (via GitHub)" <gi...@apache.org>.
osscm commented on issue #10930:
URL: https://github.com/apache/iceberg/issues/10930#issuecomment-2295688832
> Before committing to such spec change, it would be great to have good understanding on how much real-world improvement it would bring,
Based on my observations, in addition to standard queries for counting or determining minimum/maximum values in ad-hoc analysis, count/min/max queries are frequently used on dashboards. These queries are either run dynamically or help users create their own queries. Additionally, Agg pushdown can handle a broader range of queries by supporting predicates and groupings by partition columns.
> cause it's not like the current file-level metadata is useless for aggregation queries speedup. It's just that eg in Trino it's not utilized to the maximum yet.
@findepi do you mean min/max and count from the Iceberg metadata are already used for query planning?
though none of them used for the agg pushdown?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Specify in lower/upper bounds in data_file struct are exact [iceberg]
Posted by "osscm (via GitHub)" <gi...@apache.org>.
osscm commented on issue #10930:
URL: https://github.com/apache/iceberg/issues/10930#issuecomment-2286453094
Thanks @sopel39
As we were discussing this will help to extend aggregate pushdown for min/max as well.
This will help to further improve the Iceberg queries, that is already much better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Specify in lower/upper bounds in data_file struct are exact [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #10930:
URL: https://github.com/apache/iceberg/issues/10930#issuecomment-2661159019
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org