You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "jonah-rpo (via GitHub)" <gi...@apache.org> on 2026/10/14 15:42:27 UTC
[PR] [flink] Skip empty lake snapshot splits [fluss]
jonah-rpo opened a new pull request, #4649:
URL: https://github.com/apache/fluss/pull/4649
## Summary
The streaming lake snapshot reader can silently truncate a bucket's bootstrap when an intermediate lake split has no remaining rows. Empty splits are valid after deletes. `pollBatch()` advances only once and returns null if the next split is empty; the bounded source reader interprets null as snapshot completion and never reads later populated splits.
Continue advancing through exhausted or empty splits until a populated iterator is found or the complete split list is exhausted. Close exhausted iterators before replacing them. Keep split indices and the constructor's restored starting position unchanged.
The change is confined to the common Flink snapshot scanner and its regression test. It adds no lake-format dependency, configuration, or serialization change.
## Test Plan
- Reproduced against Fluss v1.0.0: five of six parameterized regression cases fail before the patch.
- Patched v1.0.0: focused scanner, lake split generator and lake split serializer verification passes (14 tests). Current main passes 15 focused tests.
- Regression cases cover leading, intermediate and trailing empty splits, multiple consecutive empty splits, all-empty/no-split input, every restored starting index, iterator closure and repeated terminal polling.
- Formatting, licensing and checkstyle are included in Maven verification.
- The full common Flink module run exercised 686 tests: 684 passed, with one failure in `FlinkSourceEnumeratorTest.testNewPartitionsUseEarliestOffset` and one error in `testBatchPartitionFilterOnlyGeneratesMatchingPartitionSplits`. The latter passed on isolated rerun. The former also fails with the scanner patch removed on the same upstream main revision (`2d6d11419aaf9dad71c1f78cc472e5d7b1264ce3`), establishing an existing baseline failure in this local environment. No enumerator code or tests are changed by this PR.
AI-assisted contribution; this draft is available for human review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Skip empty lake snapshot splits [fluss]
Posted by "jonah-rpo (via GitHub)" <gi...@apache.org>.
jonah-rpo commented on code in PR #4649:
URL: https://github.com/apache/fluss/pull/4649#discussion_r4285428036
##########
fluss-flink/fluss-flink-common/src/main/java/org/apache/fluss/flink/lake/reader/SeekableLakeSnapshotSplitScanner.java:
##########
@@ -71,8 +71,8 @@ public CloseableIterator<InternalRow> pollBatch(Duration timeout) throws IOExcep
updateCurrentIterator();
Review Comment:
I’d avoid adding an unconditional close here alone: returned iterators are already closed through batch recycling, so this could close them again. I’ve opened draft #4661 to address cleanup separately, with a close-state guard and lifecycle tests covering returned and internally skipped iterators.
##########
fluss-flink/fluss-flink-common/src/main/java/org/apache/fluss/flink/lake/reader/SeekableLakeSnapshotSplitScanner.java:
##########
@@ -71,8 +71,8 @@ public CloseableIterator<InternalRow> pollBatch(Duration timeout) throws IOExcep
updateCurrentIterator();
Review Comment:
I’d avoid adding an unconditional close here alone: returned iterators are already closed through batch recycling, so this could close them again. I’ve opened #4661 to address cleanup separately, with a close-state guard and lifecycle tests covering returned and internally skipped iterators.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Skip empty lake snapshot splits [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia merged PR #4649:
URL: https://github.com/apache/fluss/pull/4649
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Skip empty lake snapshot splits [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #4649:
URL: https://github.com/apache/fluss/pull/4649#discussion_r4282582443
##########
fluss-flink/fluss-flink-common/src/main/java/org/apache/fluss/flink/lake/reader/SeekableLakeSnapshotSplitScanner.java:
##########
@@ -71,8 +71,8 @@ public CloseableIterator<InternalRow> pollBatch(Duration timeout) throws IOExcep
updateCurrentIterator();
Review Comment:
For updateCurrentIterator, can we also close the previous iterator?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org