You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "dramaticlly (via GitHub)" <gi...@apache.org> on 2026/07/06 22:51:46 UTC
[PR] API, Core: Add CloseableIterable.filter with keep/skip callback for hetergeneous element [iceberg]
dramaticlly opened a new pull request, #17118:
URL: https://github.com/apache/iceberg/pull/17118
## What
Add a new filter overload that invokes a Consumer on each accepted and rejected element, so callers can attribute both kept and skipped entries to metrics in a single pass.
## Why
This help provide a path for hetergeneous entry of `TrackedFile` in v4, where a given entry can carry contentType of DATA, EQUALITY_DELETES, DATA_MANIFEST, or DELETE_MANIFEST within a single root manifest. Callback is better suited than a dedicated counter to allow more flexibility. Plus, import `org.apache.iceberg.metrics.Counter` in `org.apache.iceberg.io. CloseableIterable` is not ideal for a general purpose iteration utility.
## Existing caller
Migrate the manifest-level scan pipelines in `DeleteFileIndex`, `PositionDeletesTable`, and `ManifestGroup` where each previously chained a skip-counting filter with a scan-counting count call. Now replaced with a single filter call with both onKeep and onSkip callback. File-level filters in ManifestReader and ManifestGroup remain on the counter-based overload since ScanMetrics only exposes a skipped counter at that layer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Support onKeep/onSkip callback for FilterIterator [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #17118:
URL: https://github.com/apache/iceberg/pull/17118#issuecomment-5223458448
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Support onKeep/onSkip callback for FilterIterator [iceberg]
Posted by "dramaticlly (via GitHub)" <gi...@apache.org>.
dramaticlly commented on PR #17118:
URL: https://github.com/apache/iceberg/pull/17118#issuecomment-5297056969
not stale
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Support onKeep/onSkip callback for FilterIterator [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #17118:
URL: https://github.com/apache/iceberg/pull/17118#issuecomment-6227289242
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Add CloseableIterable.filter with keep/skip callback for hetergeneous element [iceberg]
Posted by "dramaticlly (via GitHub)" <gi...@apache.org>.
dramaticlly commented on code in PR #17118:
URL: https://github.com/apache/iceberg/pull/17118#discussion_r3538408332
##########
api/src/main/java/org/apache/iceberg/io/FilterIterator.java:
##########
@@ -67,6 +74,7 @@ private boolean advance() {
this.nextReady = true;
return true;
}
+ onSkip(next);
Review Comment:
@rdblue I think this align with your previous suggestion in https://github.com/apache/iceberg/pull/5268#discussion_r929192438. A custom callback enable different counter for v4 heterogenous entry of trackedFile
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Support onKeep/onSkip callback for FilterIterator [iceberg]
Posted by "dramaticlly (via GitHub)" <gi...@apache.org>.
dramaticlly commented on PR #17118:
URL: https://github.com/apache/iceberg/pull/17118#issuecomment-5689554338
try one more time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Support onKeep/onSkip callback for FilterIterator [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #17118:
URL: https://github.com/apache/iceberg/pull/17118#issuecomment-6377475754
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Add CloseableIterable.filter with keep/skip callback for hetergeneous element [iceberg]
Posted by "dramaticlly (via GitHub)" <gi...@apache.org>.
dramaticlly commented on PR #17118:
URL: https://github.com/apache/iceberg/pull/17118#issuecomment-4898700468
FYI @stevenzwu @anoopj @nastra if you want to take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Add CloseableIterable.filter with keep/skip callback for hetergeneous element [iceberg]
Posted by "wombatu-kun (via GitHub)" <gi...@apache.org>.
wombatu-kun commented on code in PR #17118:
URL: https://github.com/apache/iceberg/pull/17118#discussion_r3533110207
##########
api/src/main/java/org/apache/iceberg/io/CloseableIterable.java:
##########
@@ -175,18 +199,7 @@ protected boolean shouldKeep(E item) {
*/
static <T> CloseableIterable<T> count(Counter counter, CloseableIterable<T> iterable) {
Preconditions.checkArgument(null != counter, "Invalid counter: null");
- Preconditions.checkArgument(null != iterable, "Invalid iterable: null");
- return new CloseableIterable<T>() {
- @Override
- public CloseableIterator<T> iterator() {
- return CloseableIterator.count(counter, iterable.iterator());
- }
-
- @Override
- public void close() throws IOException {
- iterable.close();
- }
- };
+ return filter(iterable, item -> true, item -> counter.increment(), item -> {});
Review Comment:
count now delegates to the FilterIterator-based filter, which increments during hasNext()/advance() rather than on next() like CloseableIterator.count. The javadoc still says the counter is incremented on each Iterator#next() call, so peeking via hasNext() without a following next() over-counts. Update the javadoc, or keep count on CloseableIterator.count.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Add CloseableIterable.filter with keep/skip callback for hetergeneous element [iceberg]
Posted by "dramaticlly (via GitHub)" <gi...@apache.org>.
dramaticlly commented on code in PR #17118:
URL: https://github.com/apache/iceberg/pull/17118#discussion_r3538420251
##########
api/src/main/java/org/apache/iceberg/io/CloseableIterable.java:
##########
@@ -175,18 +199,7 @@ protected boolean shouldKeep(E item) {
*/
static <T> CloseableIterable<T> count(Counter counter, CloseableIterable<T> iterable) {
Preconditions.checkArgument(null != counter, "Invalid counter: null");
- Preconditions.checkArgument(null != iterable, "Invalid iterable: null");
- return new CloseableIterable<T>() {
- @Override
- public CloseableIterator<T> iterator() {
- return CloseableIterator.count(counter, iterable.iterator());
- }
-
- @Override
- public void close() throws IOException {
- iterable.close();
- }
- };
+ return filter(iterable, item -> true, item -> counter.increment(), item -> {});
Review Comment:
Thanks @wombatu-kun for the review. I updated to change FilterIterator directly so that we keep exactly once semantics for skip on hasNext() and kept on next(), same as before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Support onKeep/onSkip callback for FilterIterator [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on PR #17118:
URL: https://github.com/apache/iceberg/pull/17118#issuecomment-4919791346
If this is mainly for v4, maybe we should wait until the basic plumbing of write and read get in first. Then we will have a better idea on how other pieces (like this) fit in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Support onKeep/onSkip callback for FilterIterator [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #17118:
URL: https://github.com/apache/iceberg/pull/17118#issuecomment-5657392187
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API, Core: Support onKeep/onSkip callback for FilterIterator [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #17118: API, Core: Support onKeep/onSkip callback for FilterIterator
URL: https://github.com/apache/iceberg/pull/17118
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org