[PR] Spark: Short-circuit SyncSparkMicroBatchPlanner.latestOffset for unbounded limits [iceberg]

["binarybrain-009 (via GitHub)" <gi...@apache.org>]

binarybrain-009 opened a new pull request, #18494:
URL: https://github.com/apache/iceberg/pull/18494

   When running a structured streaming read with an unbounded read limit such as `Trigger.AvailableNow` or `ReadLimit.allAvailable()` and the default synchronous micro-batch planner (`streaming-async-micro-batch-planning-enabled=false`), `SyncSparkMicroBatchPlanner.latestOffset` previously unpacked `maxFiles` and `maxRows` to `Integer.MAX_VALUE` and walked every manifest and file scan task from the starting offset up to the latest snapshot. Because neither limit can trip during an unbounded run, this single-threaded driver walk performs redundant manifest I/O across the entire snapshot gap just to return the latest valid snapshot's added-file count, and `planFiles` immediately re-plans the same range afterwards.
   
   This change updates `SyncSparkMicroBatchPlanner.latestOffset` across Spark 3.5, 4.0, 4.1, and 4.2 to short-circuit when `limit instanceof ReadAllAvailable`, mirroring the existing behavior in `AsyncSparkMicroBatchPlanner.latestOffset`. When a `Trigger.AvailableNow` cap has already been computed, `latestOffsetForUnbounded` uses that pre-computed offset directly; otherwise it walks only the snapshot metadata chain via `nextValidSnapshot` (skipping rewrite and configured delete/overwrite snapshots) and constructs the end offset from `MicroBatchUtils.addedFilesCount(table(), lastValidSnapshot)`, returning `null` if the offset did not advance from `startingOffset`.
   
   This refreshes and extends the changes from stale PR #16941 to include Spark 4.2, and adds `TestStructuredStreamingRead3#testLatestOffsetForUnboundedLimitAdvancesToHeadSnapshot` across all supported Spark versions to verify that both the synchronous and asynchronous planners advance directly to the head snapshot and its added-file count under `ReadLimit.allAvailable()`. Fixes #16940.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Spark: Short-circuit SyncSparkMicroBatchPlanner.latestOffset for unbounded limits [iceberg]

["binarybrain-009 (via GitHub)" <gi...@apache.org>]

binarybrain-009 commented on PR #18494:
URL: https://github.com/apache/iceberg/pull/18494#issuecomment-6388765225

   @manuzhang could you take a look when you have a chance? You've worked on `SyncSparkMicroBatchPlanner` recently, so you probably have the most context here. Thanks!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org