You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "gaborkaszab (via GitHub)" <gi...@apache.org> on 2025/11/19 14:14:45 UTC
[PR] Core, Parquet: Add filter to InternalData [iceberg]
gaborkaszab opened a new pull request, #14630:
URL: https://github.com/apache/iceberg/pull/14630
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Parquet: Add filter to InternalData [iceberg]
Posted by "gaborkaszab (via GitHub)" <gi...@apache.org>.
gaborkaszab commented on PR #14630:
URL: https://github.com/apache/iceberg/pull/14630#issuecomment-3552960015
This is needed for a follow-up step for the [partition stat Scan API](https://github.com/apache/iceberg/pull/14508) to filter the stats. InternalData is used to read the partition stats and as a next step filtering is to be introduced, however, InternalData doesn't have a support for this now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Parquet: Add filter to InternalData [iceberg]
Posted by "gaborkaszab (via GitHub)" <gi...@apache.org>.
gaborkaszab closed pull request #14630: Core, Parquet: Add filter to InternalData
URL: https://github.com/apache/iceberg/pull/14630
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Parquet: Add filter to InternalData [iceberg]
Posted by "gaborkaszab (via GitHub)" <gi...@apache.org>.
gaborkaszab commented on PR #14630:
URL: https://github.com/apache/iceberg/pull/14630#issuecomment-3557186543
Took another look, and apparently our internal Parquet reader can only do row group filtering, so for InternalData that reads metadata (including partitions stats) probably such filtering won't bring any additional value, because such data rarely spans multiple row groups.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Parquet: Add filter to InternalData [iceberg]
Posted by "gaborkaszab (via GitHub)" <gi...@apache.org>.
gaborkaszab commented on PR #14630:
URL: https://github.com/apache/iceberg/pull/14630#issuecomment-3553359569
Anther way to implement this is to introduce a SupportsFiltering interface and parquet reader can derive from that. At the creation site we have to check if the reader is an instance of SupportsFiltering and add filter if yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Parquet: Add filter to InternalData [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #14630:
URL: https://github.com/apache/iceberg/pull/14630#discussion_r2544991507
##########
core/src/main/java/org/apache/iceberg/InternalData.java:
##########
@@ -163,6 +164,11 @@ public interface ReadBuilder {
/** Set a custom class for in-memory objects at the given field ID. */
ReadBuilder setCustomType(int fieldId, Class<? extends StructLike> structClass);
+ /** Set a filter to apply on result rows if applicable. */
+ default ReadBuilder filter(Expression newFilter) {
Review Comment:
nit: maybe call the parameter `filter`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Parquet: Add filter to InternalData [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #14630:
URL: https://github.com/apache/iceberg/pull/14630#discussion_r2544989615
##########
core/src/main/java/org/apache/iceberg/InternalData.java:
##########
@@ -163,6 +164,11 @@ public interface ReadBuilder {
/** Set a custom class for in-memory objects at the given field ID. */
ReadBuilder setCustomType(int fieldId, Class<? extends StructLike> structClass);
+ /** Set a filter to apply on result rows if applicable. */
Review Comment:
It is very important to mention that the caller still needs to do residual filtering, because some filters might not be supported, and some formats might not support filtering at all.
Something like this:
```
/**
* Pushes down the {@link Expression} filter for the reader to prevent reading unnecessary
* records. Some readers may not support filtering, or may only support filtering for certain expressions.
* In this case the reader might return unfiltered or partially filtered rows. It is the caller's responsibility to
* apply the filter again.
*
* @param filter the filter to set
*/
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org