You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "qlong (via GitHub)" <gi...@apache.org> on 2026/03/04 15:22:43 UTC
[I] Support row group skipping for shredded variant columns [iceberg]
qlong opened a new issue, #15510:
URL: https://github.com/apache/iceberg/issues/15510
### Feature Request / Improvement
Iceberg supports writing shredded variant columns to Parquet (PR #14297).
For a query like `WHERE variant_get(v, '$.price', 'int') > 100`, Iceberg can already skip Parquet files using manifest-level bounds, where the lower/upper bounds for $.price are stored as a Variant-encoded object on the v column's manifest entry (PR #15385).
For better performance, Iceberg should also skip row groups within a file: when $.price is shredded, the physical Parquet column `v.typed_value.price.typed_value` has standard Parquet min/max statistics. Iceberg should use those stats to skip row groups where max < 100, the same way ParquetMetricsRowGroupFilter handles regular column predicates.
The change is in parquet module, and should work with all engines.
**Scope**
- Add BoundExtract handling to ParquetMetricsRowGroupFilter
- Map variant path → physical shredded Parquet column
- Unshredded variant columns are not in scope (binary stats are not useful for path-based filtering)
- Test with Spark
### Query engine
None
### Willingness to contribute
- [x] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support row group skipping for shredded variant columns [iceberg]
Posted by "soumilshah199500 (via GitHub)" <gi...@apache.org>.
soumilshah199500 commented on issue #15510:
URL: https://github.com/apache/iceberg/issues/15510#issuecomment-4489024055
Hi there, just wanted to inquire when we plan to merge this. I wanted to rerun the tests again. as filters on shredding variant table are very slow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support row group skipping for shredded variant columns [iceberg]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on issue #15510:
URL: https://github.com/apache/iceberg/issues/15510#issuecomment-4546167901
@soumilshah199500 yes they are. As for schedule, lots to work on. See : https://steveloughran.github.io/benchmarking-variants/
Some things we would love to have here are
1. real world datasets (sanitized, obviously)
2. example queries you want to run over them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support row group skipping for shredded variant columns [iceberg]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on issue #15510:
URL: https://github.com/apache/iceberg/issues/15510#issuecomment-4306665567
i will have a go at this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org