You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "mxm (via GitHub)" <gi...@apache.org> on 2026/10/02 14:48:49 UTC

[PR] Spark: Don't reapply stream-from-timestamp when a stream resumes [iceberg]

mxm opened a new pull request, #18348:
URL: https://github.com/apache/iceberg/pull/18348

   `stream-from-timestamp` tells a new streaming query which snapshot to start from. Until that snapshot exists, the query uses the offset -1, which means "no position yet".
   
   Problem 1: The option was applied again on every restart. For example, a job may pass the current time every time it starts. After a restart, the newest snapshot was usually older than that time, so the query fell back to -1 instead of continuing where it stopped. Everything committed while the job was down was skipped (see #10156), and -1 was saved in the checkpoint.
   
   Problem 2: When Spark replays a batch that starts at -1, Iceberg looks up the start again with the option's current value. If the job stopped in the middle of such a batch, the restart used a later time, found no snapshot after it, and failed with "Cannot load current offset at snapshot -1". It failed the same way on every further restart. A new checkpoint could hit this as well, because it saved -1 as its starting point.
   
   Fix: A query that already has a position continues from it. The option is only used to find the first snapshot. -1 is never saved: the query waits until there is a snapshot to start from, and saves it as its starting point before the first batch is recorded in the checkpoint.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Don't reapply stream-from-timestamp when a stream resumes [iceberg]

Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #18348:
URL: https://github.com/apache/iceberg/pull/18348#discussion_r4217759520


##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/SyncSparkMicroBatchPlanner.java:
##########
@@ -118,7 +118,9 @@ public StreamingOffset latestOffset(StreamingOffset startOffset, ReadLimit limit
       return StreamingOffset.START_OFFSET;
     }
 
-    if (table().currentSnapshot().timestampMillis() < fromTimestamp) {
+    // Only a new stream starts from the timestamp. A resumed stream continues from its offset.
+    if (startOffset.equals(StreamingOffset.START_OFFSET)
+        && table().currentSnapshot().timestampMillis() < fromTimestamp) {

Review Comment:
   Good idea. I've moved the check into the `START_OFFSET` branch, so `determineStartingOffset` handles the timestamp. The async planner keeps its check because it doesn't look up the start in `latestOffset`.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Don't reapply stream-from-timestamp when a stream resumes [iceberg]

Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on PR #18348:
URL: https://github.com/apache/iceberg/pull/18348#issuecomment-6057871550

   @huaxingao I've added a note to `spark-configuration.md`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Don't reapply stream-from-timestamp when a stream resumes [iceberg]

Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #18348:
URL: https://github.com/apache/iceberg/pull/18348#discussion_r4212023701


##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/SparkMicroBatchStream.java:
##########
@@ -244,15 +253,32 @@ public ReadLimit getDefaultReadLimit() {
   public void prepareForTriggerAvailableNow() {
     LOG.info("The streaming query reports to use Trigger.AvailableNow");
 
-    lastOffsetForTriggerAvailableNow =
+    StreamingOffset lastOffset =
         (StreamingOffset) latestOffset(initialOffset, ReadLimit.allAvailable());
+    // START_OFFSET means that no snapshot matched stream-from-timestamp. A new stream has nothing
+    // to read in this run, but a resumed stream continues from its offset and needs an actual cap.
+    this.noSnapshotMatchedForTriggerAvailableNow = StreamingOffset.START_OFFSET.equals(lastOffset);
+    this.lastOffsetForTriggerAvailableNow =
+        noSnapshotMatchedForTriggerAvailableNow ? latestAppendOffset() : lastOffset;
 
-    LOG.info("lastOffset for Trigger.AvailableNow is {}", lastOffsetForTriggerAvailableNow.json());
+    LOG.info("lastOffset for Trigger.AvailableNow is {}", lastOffsetForTriggerAvailableNow);
 
     // Reset planner so it gets recreated with the cap on next call
     if (planner != null) {
       planner.stop();
       planner = null;
     }
   }
+
+  // The planners only read append snapshots. A cap at any other snapshot is never reached.
+  private StreamingOffset latestAppendOffset() {
+    for (Snapshot snapshot : SnapshotUtil.currentAncestors(table)) {
+      if (DataOperations.APPEND.equals(snapshot.operation())) {
+        return new StreamingOffset(
+            snapshot.snapshotId(), MicroBatchUtils.addedFilesCount(table, snapshot), false);
+      }
+    }
+
+    return null;
+  }

Review Comment:
   +1. This method has its own rule for which snapshots count ("only APPEND"), but the planners use `shouldProcess`. The two rules can disagree today: with `streaming-skip-delete-snapshots=false`, `shouldProcess` throws on a DELETE snapshot, but this method skips it. Could this method reuse the `shouldProcess` logic? Then the two can't get out of sync.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Don't reapply stream-from-timestamp when a stream resumes [iceberg]

Posted by "dramaticlly (via GitHub)" <gi...@apache.org>.
dramaticlly commented on code in PR #18348:
URL: https://github.com/apache/iceberg/pull/18348#discussion_r4210765678


##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/SparkMicroBatchStream.java:
##########


Review Comment:
   do you need to update this?



##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/SyncSparkMicroBatchPlanner.java:
##########
@@ -118,7 +118,9 @@ public StreamingOffset latestOffset(StreamingOffset startOffset, ReadLimit limit
       return StreamingOffset.START_OFFSET;
     }
 
-    if (table().currentSnapshot().timestampMillis() < fromTimestamp) {
+    // Only a new stream starts from the timestamp. A resumed stream continues from its offset.
+    if (startOffset.equals(StreamingOffset.START_OFFSET)
+        && table().currentSnapshot().timestampMillis() < fromTimestamp) {

Review Comment:
   not sure if this worth reusing MicroBatchUtils.determineStartingOffset ?



##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/SparkMicroBatchStream.java:
##########
@@ -244,15 +253,32 @@ public ReadLimit getDefaultReadLimit() {
   public void prepareForTriggerAvailableNow() {
     LOG.info("The streaming query reports to use Trigger.AvailableNow");
 
-    lastOffsetForTriggerAvailableNow =
+    StreamingOffset lastOffset =
         (StreamingOffset) latestOffset(initialOffset, ReadLimit.allAvailable());
+    // START_OFFSET means that no snapshot matched stream-from-timestamp. A new stream has nothing
+    // to read in this run, but a resumed stream continues from its offset and needs an actual cap.
+    this.noSnapshotMatchedForTriggerAvailableNow = StreamingOffset.START_OFFSET.equals(lastOffset);
+    this.lastOffsetForTriggerAvailableNow =
+        noSnapshotMatchedForTriggerAvailableNow ? latestAppendOffset() : lastOffset;
 
-    LOG.info("lastOffset for Trigger.AvailableNow is {}", lastOffsetForTriggerAvailableNow.json());
+    LOG.info("lastOffset for Trigger.AvailableNow is {}", lastOffsetForTriggerAvailableNow);

Review Comment:
   I think now lastOffsetForTriggerAvailableNow is nullable from return of latestAppendOffset()



##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/SparkMicroBatchStream.java:
##########
@@ -244,15 +253,32 @@ public ReadLimit getDefaultReadLimit() {
   public void prepareForTriggerAvailableNow() {
     LOG.info("The streaming query reports to use Trigger.AvailableNow");
 
-    lastOffsetForTriggerAvailableNow =
+    StreamingOffset lastOffset =
         (StreamingOffset) latestOffset(initialOffset, ReadLimit.allAvailable());
+    // START_OFFSET means that no snapshot matched stream-from-timestamp. A new stream has nothing
+    // to read in this run, but a resumed stream continues from its offset and needs an actual cap.
+    this.noSnapshotMatchedForTriggerAvailableNow = StreamingOffset.START_OFFSET.equals(lastOffset);
+    this.lastOffsetForTriggerAvailableNow =
+        noSnapshotMatchedForTriggerAvailableNow ? latestAppendOffset() : lastOffset;
 
-    LOG.info("lastOffset for Trigger.AvailableNow is {}", lastOffsetForTriggerAvailableNow.json());
+    LOG.info("lastOffset for Trigger.AvailableNow is {}", lastOffsetForTriggerAvailableNow);
 
     // Reset planner so it gets recreated with the cap on next call
     if (planner != null) {
       planner.stop();
       planner = null;
     }
   }
+
+  // The planners only read append snapshots. A cap at any other snapshot is never reached.
+  private StreamingOffset latestAppendOffset() {
+    for (Snapshot snapshot : SnapshotUtil.currentAncestors(table)) {
+      if (DataOperations.APPEND.equals(snapshot.operation())) {
+        return new StreamingOffset(
+            snapshot.snapshotId(), MicroBatchUtils.addedFilesCount(table, snapshot), false);
+      }
+    }
+
+    return null;

Review Comment:
   wondering if we need UT to cover the OVERWRITE data operations which this return null and append is expired ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Don't reapply stream-from-timestamp when a stream resumes [iceberg]

Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on PR #18348:
URL: https://github.com/apache/iceberg/pull/18348#issuecomment-6121136741

   Merging, please let me know if you have any further comments. I'd be happy to address them in a follow-up.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Don't reapply stream-from-timestamp when a stream resumes [iceberg]

Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on PR #18348:
URL: https://github.com/apache/iceberg/pull/18348#issuecomment-6046976343

   Thanks @mxm for the PR! Overall LGTM. `docs/docs/spark-configuration.md` describes `stream-from-timestamp` without mentioning restarts. Since this PR changes what happens on restart, maybe add something like: "Only applies when a query starts without a checkpoint; a query restarted from a checkpoint continues from its saved position."


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Don't reapply stream-from-timestamp when a stream resumes [iceberg]

Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm merged PR #18348:
URL: https://github.com/apache/iceberg/pull/18348


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Don't reapply stream-from-timestamp when a stream resumes [iceberg]

Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #18348:
URL: https://github.com/apache/iceberg/pull/18348#discussion_r4185146224


##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/SparkMicroBatchStream.java:
##########
@@ -247,7 +267,7 @@ public void prepareForTriggerAvailableNow() {
     lastOffsetForTriggerAvailableNow =

Review Comment:
   This part is gone now that the PR is smaller. For a stream without an offset the cap behaves as before, so the async planner still stops in your scenario. The only change left is that it no longer returns a -1 cap as the end offset of a resumed stream.



##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/SparkMicroBatchStream.java:
##########
@@ -215,12 +213,34 @@ public Offset latestOffset(Offset startOffset, ReadLimit limit) {
         "Invalid start offset: %s is not a StreamingOffset",
         startOffset);
 
+    StreamingOffset start = (StreamingOffset) startOffset;
+    // Resolve start offset once, if it hasn't been resolved already.
+    if (StreamingOffset.START_OFFSET.equals(start)) {
+      start = startingOffsetFromTimestamp();
+      if (StreamingOffset.START_OFFSET.equals(start)) {
+        return null;
+      }
+
+      // Spark starts the first batch at initialOffset(), also when it replays the batch after a
+      // restart. Store the resolved offset before Spark records the first batch's end offset in the
+      // checkpoint, so the replay starts from the same offset.
+      if (StreamingOffset.START_OFFSET.equals(initialOffset)) {
+        initialOffsetStore.storeInitialOffset(start);
+        this.initialOffset = start;
+      }
+    }
+
     // Initialize planner if not already done
     if (planner == null) {
-      initializePlanner((StreamingOffset) startOffset, null);
+      initializePlanner(start, null);
     }
 
-    return planner.latestOffset((StreamingOffset) startOffset, limit);
+    return planner.latestOffset(start, limit);
+  }
+
+  private StreamingOffset startingOffsetFromTimestamp() {
+    table.refresh();

Review Comment:
   Gone as well. The start offset lookup is back in the planners, so the refresh rate is unchanged.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Don't reapply stream-from-timestamp when a stream resumes [iceberg]

Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #18348:
URL: https://github.com/apache/iceberg/pull/18348#discussion_r4217793805


##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/SparkMicroBatchStream.java:
##########
@@ -244,15 +253,32 @@ public ReadLimit getDefaultReadLimit() {
   public void prepareForTriggerAvailableNow() {
     LOG.info("The streaming query reports to use Trigger.AvailableNow");
 
-    lastOffsetForTriggerAvailableNow =
+    StreamingOffset lastOffset =
         (StreamingOffset) latestOffset(initialOffset, ReadLimit.allAvailable());
+    // START_OFFSET means that no snapshot matched stream-from-timestamp. A new stream has nothing
+    // to read in this run, but a resumed stream continues from its offset and needs an actual cap.
+    this.noSnapshotMatchedForTriggerAvailableNow = StreamingOffset.START_OFFSET.equals(lastOffset);
+    this.lastOffsetForTriggerAvailableNow =
+        noSnapshotMatchedForTriggerAvailableNow ? latestAppendOffset() : lastOffset;
 
-    LOG.info("lastOffset for Trigger.AvailableNow is {}", lastOffsetForTriggerAvailableNow.json());
+    LOG.info("lastOffset for Trigger.AvailableNow is {}", lastOffsetForTriggerAvailableNow);
 
     // Reset planner so it gets recreated with the cap on next call
     if (planner != null) {
       planner.stop();
       planner = null;
     }
   }
+
+  // The planners only read append snapshots. A cap at any other snapshot is never reached.
+  private StreamingOffset latestAppendOffset() {
+    for (Snapshot snapshot : SnapshotUtil.currentAncestors(table)) {
+      if (DataOperations.APPEND.equals(snapshot.operation())) {
+        return new StreamingOffset(
+            snapshot.snapshotId(), MicroBatchUtils.addedFilesCount(table, snapshot), false);
+      }
+    }
+
+    return null;

Review Comment:
   I've added a test for the null case: `availableNowReadsNothingFromEmptyTable`. For a resumed stream, the snapshot of its own offset is always found, unless it was expired. In that case the stream fails before this, which is pre-existing behavior (see #18000).
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Don't reapply stream-from-timestamp when a stream resumes [iceberg]

Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #18348:
URL: https://github.com/apache/iceberg/pull/18348#discussion_r4195788365


##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/AsyncSparkMicroBatchPlanner.java:
##########
@@ -267,10 +269,16 @@ public synchronized StreamingOffset latestOffset(StreamingOffset startOffset, Re
 
     // if we want to read all available we don't need to scan files, just snapshots
     if (limit instanceof ReadAllAvailable) {
+      // A START_OFFSET cap only means that no snapshot matched stream-from-timestamp when the run
+      // started. Returned as the end offset of a resumed stream, it would reset the stream.
+      boolean capResetsStream =

Review Comment:
   Thanks for the suggestion! I've changed `prepareForTriggerAvailableNow` to cap a resumed stream at the newest append snapshot when no snapshot matched the timestamp. I didn't use the current snapshot because it can be a replace from compaction, which the async planner never processes. A new stream still reads nothing in that run, now with both planners. I've added your test plus one for the replace case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Don't reapply stream-from-timestamp when a stream resumes [iceberg]

Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #18348:
URL: https://github.com/apache/iceberg/pull/18348#discussion_r4217750149


##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/SparkMicroBatchStream.java:
##########
@@ -244,15 +253,32 @@ public ReadLimit getDefaultReadLimit() {
   public void prepareForTriggerAvailableNow() {
     LOG.info("The streaming query reports to use Trigger.AvailableNow");
 
-    lastOffsetForTriggerAvailableNow =
+    StreamingOffset lastOffset =
         (StreamingOffset) latestOffset(initialOffset, ReadLimit.allAvailable());
+    // START_OFFSET means that no snapshot matched stream-from-timestamp. A new stream has nothing
+    // to read in this run, but a resumed stream continues from its offset and needs an actual cap.
+    this.noSnapshotMatchedForTriggerAvailableNow = StreamingOffset.START_OFFSET.equals(lastOffset);
+    this.lastOffsetForTriggerAvailableNow =
+        noSnapshotMatchedForTriggerAvailableNow ? latestAppendOffset() : lastOffset;
 
-    LOG.info("lastOffset for Trigger.AvailableNow is {}", lastOffsetForTriggerAvailableNow.json());
+    LOG.info("lastOffset for Trigger.AvailableNow is {}", lastOffsetForTriggerAvailableNow);

Review Comment:
   Exactly, the toString() method is also better suited for log output than the json and contains the same info.



##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/SparkMicroBatchStream.java:
##########
@@ -244,15 +253,32 @@ public ReadLimit getDefaultReadLimit() {
   public void prepareForTriggerAvailableNow() {
     LOG.info("The streaming query reports to use Trigger.AvailableNow");
 
-    lastOffsetForTriggerAvailableNow =
+    StreamingOffset lastOffset =
         (StreamingOffset) latestOffset(initialOffset, ReadLimit.allAvailable());
+    // START_OFFSET means that no snapshot matched stream-from-timestamp. A new stream has nothing
+    // to read in this run, but a resumed stream continues from its offset and needs an actual cap.
+    this.noSnapshotMatchedForTriggerAvailableNow = StreamingOffset.START_OFFSET.equals(lastOffset);
+    this.lastOffsetForTriggerAvailableNow =
+        noSnapshotMatchedForTriggerAvailableNow ? latestAppendOffset() : lastOffset;
 
-    LOG.info("lastOffset for Trigger.AvailableNow is {}", lastOffsetForTriggerAvailableNow.json());
+    LOG.info("lastOffset for Trigger.AvailableNow is {}", lastOffsetForTriggerAvailableNow);
 
     // Reset planner so it gets recreated with the cap on next call
     if (planner != null) {
       planner.stop();
       planner = null;
     }
   }
+
+  // The planners only read append snapshots. A cap at any other snapshot is never reached.
+  private StreamingOffset latestAppendOffset() {
+    for (Snapshot snapshot : SnapshotUtil.currentAncestors(table)) {
+      if (DataOperations.APPEND.equals(snapshot.operation())) {
+        return new StreamingOffset(
+            snapshot.snapshotId(), MicroBatchUtils.addedFilesCount(table, snapshot), false);
+      }
+    }
+
+    return null;
+  }

Review Comment:
   I thought about this, but it wasn't straightforward to reuse `shouldProcess`, so I added a test instead. However, I agree with the concern.
   
   I've moved the lookup into the planner, so it uses `shouldProcess`. As Huaxin mentioned, with `streaming-skip-delete-snapshots=false`, an AvailableNow run in which no snapshot matches the timestamp now fails at start if the newest snapshot is a delete, also for a new stream that wouldn't read it. I'm leaning towards accepting that to keep the code consistent.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Don't reapply stream-from-timestamp when a stream resumes [iceberg]

Posted by "anuragmantri (via GitHub)" <gi...@apache.org>.
anuragmantri commented on code in PR #18348:
URL: https://github.com/apache/iceberg/pull/18348#discussion_r4169463566


##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/SparkMicroBatchStream.java:
##########
@@ -215,12 +213,34 @@ public Offset latestOffset(Offset startOffset, ReadLimit limit) {
         "Invalid start offset: %s is not a StreamingOffset",
         startOffset);
 
+    StreamingOffset start = (StreamingOffset) startOffset;
+    // Resolve start offset once, if it hasn't been resolved already.
+    if (StreamingOffset.START_OFFSET.equals(start)) {
+      start = startingOffsetFromTimestamp();
+      if (StreamingOffset.START_OFFSET.equals(start)) {
+        return null;
+      }
+
+      // Spark starts the first batch at initialOffset(), also when it replays the batch after a
+      // restart. Store the resolved offset before Spark records the first batch's end offset in the
+      // checkpoint, so the replay starts from the same offset.
+      if (StreamingOffset.START_OFFSET.equals(initialOffset)) {
+        initialOffsetStore.storeInitialOffset(start);
+        this.initialOffset = start;
+      }
+    }
+
     // Initialize planner if not already done
     if (planner == null) {
-      initializePlanner((StreamingOffset) startOffset, null);
+      initializePlanner(start, null);
     }
 
-    return planner.latestOffset((StreamingOffset) startOffset, limit);
+    return planner.latestOffset(start, limit);
+  }
+
+  private StreamingOffset startingOffsetFromTimestamp() {
+    table.refresh();

Review Comment:
   In async mode, this adds a synchronous `table.refresh()` to every `latestOffset` call until a snapshot after the timestamp appears. With the default trigger, Spark waits only `spark.sql.streaming.pollingDelay` (10 ms) between empty triggers. A job started with the timestamp set to "now" can then load table metadata many times a second until the next commit, and again after each restart. Before this change, the async planner's background thread did this every `streaming-snapshot-polling-interval-ms`, which defaults to 30 s. Could this lookup follow the polling interval in async mode, or stay with the async planner?



##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/StreamingInitialOffsetStore.java:
##########
@@ -50,16 +50,28 @@ class StreamingInitialOffsetStore {
   StreamingOffset initialOffset() {
     InputFile inputFile = io.newInputFile(initialOffsetLocation);
     if (inputFile.exists()) {
-      return readOffset(inputFile);
+      StreamingOffset offset = readOffset(inputFile);
+      if (!StreamingOffset.START_OFFSET.equals(offset)) {
+        return offset;
+      }
     }
 
+    // START_OFFSET has no position yet, so it is derived again instead of stored
     StreamingOffset offset = offsetSupplier.get();

Review Comment:
   Thanks for handling checkpoints that already have `-1` stored. I'm curious about the stuck case from the description. The old code stored `-1`, Spark recorded batch 0 with end `E`, and the job stopped before committing it. After upgrading, Spark replays `planInputPartitions(initialOffset(), E)`:
   
   - If nothing exists after the new timestamp, `initialOffset()` is still `START_OFFSET`. `SyncSparkMicroBatchPlanner.planFiles` then derives the start from the timestamp again and fails with the same "Cannot load current offset at snapshot -1".
   - If a snapshot does exist after the new timestamp, this overwrites `offsets/0` with a start that comes after `E`. The replay can't reach `E`. And because the new start is now persisted, restarting with the original timestamp no longer recovers the query.
   
   What do you think about returning a stored `START_OFFSET` unchanged here, and letting `latestOffset` resolve and store it? A batch 0 that Spark hasn't recorded yet already takes that path, and it never replaces the start under a batch Spark has recorded.



##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/SparkMicroBatchStream.java:
##########
@@ -247,7 +267,7 @@ public void prepareForTriggerAvailableNow() {
     lastOffsetForTriggerAvailableNow =

Review Comment:
   When no snapshot at or after `stream-from-timestamp` exists yet, `latestOffset` now returns `null` here, so `lastOffsetForTriggerAvailableNow` ends up `null`. Both planners treat a `null` cap as no cap. If a snapshot is committed between `prepareForTriggerAvailableNow` and the first `latestOffset` call, the run reads it and keeps picking up new commits until a batch comes back empty. Before this change, the async planner got `START_OFFSET` as its cap and stopped. The sync planner was already uncapped in this case.
   
   I checked this locally with the test below. On this branch it fails for both `async=true` and `async=false`, because `latestOffset` returns the new snapshot at position 1.
   
   ```java
     @TestTemplate
     void availableNowIgnoresSnapshotsCommittedAfterStart() {
       appendData(List.of(new SimpleRecord(1, "one")));
       SparkMicroBatchStream stream =
           newMicroBatchStream(
               Map.of(
                   SparkReadOptions.STREAM_FROM_TIMESTAMP,
                   Long.toString(timestampAfterCurrentSnapshot())),
               "available-now-unresolved-start-checkpoint");
   
       try {
         stream.prepareForTriggerAvailableNow();
         appendData(List.of(new SimpleRecord(2, "two")));
   
         assertThat(stream.latestOffset(stream.initialOffset(), stream.getDefaultReadLimit()))
             .isNull();
       } finally {
         stream.stop();
       }
     }
   ```
   
   Would it work to have `prepareForTriggerAvailableNow` remember that nothing was available, and then return `null` from `latestOffset` for the rest of the run?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Don't reapply stream-from-timestamp when a stream resumes [iceberg]

Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #18348:
URL: https://github.com/apache/iceberg/pull/18348#discussion_r4207731227


##########
spark/v4.2/spark/src/test/java/org/apache/iceberg/spark/source/TestStructuredStreamingRead3.java:
##########
@@ -572,6 +575,125 @@ public void testPlanInputPartitionsIsIdempotentForSameOffsets() {
     }
   }
 
+  @TestTemplate
+  void latestOffsetIgnoresStreamFromTimestampWhenResuming() {
+    appendData(List.of(new SimpleRecord(1, "one")));
+    table.refresh();
+    Snapshot resumedSnapshot = table.currentSnapshot();
+    appendData(List.of(new SimpleRecord(2, "two")));
+    table.refresh();
+    long latestSnapshotId = table.currentSnapshot().snapshotId();
+
+    SparkMicroBatchStream stream =
+        newMicroBatchStream(
+            ImmutableMap.of(
+                SparkReadOptions.STREAM_FROM_TIMESTAMP,
+                Long.toString(timestampAfterCurrentSnapshot())),
+            "resumed-stream-checkpoint");
+
+    try {
+      StreamingOffset resumedOffset =
+          new StreamingOffset(
+              resumedSnapshot.snapshotId(),
+              MicroBatchUtils.addedFilesCount(table, resumedSnapshot),
+              false);
+      StreamingOffset endOffset =
+          (StreamingOffset) stream.latestOffset(resumedOffset, stream.getDefaultReadLimit());
+
+      assertThat(endOffset.snapshotId()).isEqualTo(latestSnapshotId);
+    } finally {
+      stream.stop();
+    }
+  }
+
+  @TestTemplate
+  void availableNowResumeStopsAtSnapshotsAvailableAtStart() {
+    appendData(List.of(new SimpleRecord(1, "one")));
+    table.refresh();
+    Snapshot committed = table.currentSnapshot();
+    appendData(List.of(new SimpleRecord(2, "two")));
+    table.refresh();
+    Snapshot latest = table.currentSnapshot();
+    // A timestamp after the newest snapshot stores START_OFFSET as the initial offset
+    SparkMicroBatchStream stream =
+        newMicroBatchStream(
+            ImmutableMap.of(
+                SparkReadOptions.STREAM_FROM_TIMESTAMP,
+                Long.toString(timestampAfterCurrentSnapshot())),
+            "available-now-resume-checkpoint");
+
+    try {
+      stream.prepareForTriggerAvailableNow();
+      appendData(List.of(new SimpleRecord(3, "three")));
+      StreamingOffset committedOffset =
+          new StreamingOffset(
+              committed.snapshotId(), MicroBatchUtils.addedFilesCount(table, committed), false);
+
+      assertThat(stream.latestOffset(committedOffset, stream.getDefaultReadLimit()))
+          .isEqualTo(
+              new StreamingOffset(
+                  latest.snapshotId(), MicroBatchUtils.addedFilesCount(table, latest), false));
+    } finally {
+      stream.stop();
+    }
+  }
+
+  @TestTemplate
+  void availableNowResumeStopsAtLatestAppendSnapshot() {
+    appendData(List.of(new SimpleRecord(1, "one")));
+    table.refresh();
+    Snapshot committed = table.currentSnapshot();
+    appendData(List.of(new SimpleRecord(2, "two")));
+    table.refresh();
+    Snapshot latestAppend = table.currentSnapshot();
+    table.rewriteManifests().clusterBy(file -> 1).commit();
+    table.refresh();
+    assertThat(table.currentSnapshot().operation()).isEqualTo(DataOperations.REPLACE);
+    SparkMicroBatchStream stream =
+        newMicroBatchStream(
+            ImmutableMap.of(
+                SparkReadOptions.STREAM_FROM_TIMESTAMP,
+                Long.toString(timestampAfterCurrentSnapshot())),
+            "available-now-resume-replace-checkpoint");
+
+    try {
+      stream.prepareForTriggerAvailableNow();

Review Comment:
   Thanks. I've added the extra snapshot after the `prepareForTriggerAvailableNow()`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Don't reapply stream-from-timestamp when a stream resumes [iceberg]

Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on PR #18348:
URL: https://github.com/apache/iceberg/pull/18348#issuecomment-6081051244

   Please let me know @singhpk234 if you have any further comments.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Don't reapply stream-from-timestamp when a stream resumes [iceberg]

Posted by "anuragmantri (via GitHub)" <gi...@apache.org>.
anuragmantri commented on code in PR #18348:
URL: https://github.com/apache/iceberg/pull/18348#discussion_r4199245871


##########
spark/v4.2/spark/src/test/java/org/apache/iceberg/spark/source/TestStructuredStreamingRead3.java:
##########
@@ -572,6 +575,125 @@ public void testPlanInputPartitionsIsIdempotentForSameOffsets() {
     }
   }
 
+  @TestTemplate
+  void latestOffsetIgnoresStreamFromTimestampWhenResuming() {
+    appendData(List.of(new SimpleRecord(1, "one")));
+    table.refresh();
+    Snapshot resumedSnapshot = table.currentSnapshot();
+    appendData(List.of(new SimpleRecord(2, "two")));
+    table.refresh();
+    long latestSnapshotId = table.currentSnapshot().snapshotId();
+
+    SparkMicroBatchStream stream =
+        newMicroBatchStream(
+            ImmutableMap.of(
+                SparkReadOptions.STREAM_FROM_TIMESTAMP,
+                Long.toString(timestampAfterCurrentSnapshot())),
+            "resumed-stream-checkpoint");
+
+    try {
+      StreamingOffset resumedOffset =
+          new StreamingOffset(
+              resumedSnapshot.snapshotId(),
+              MicroBatchUtils.addedFilesCount(table, resumedSnapshot),
+              false);
+      StreamingOffset endOffset =
+          (StreamingOffset) stream.latestOffset(resumedOffset, stream.getDefaultReadLimit());
+
+      assertThat(endOffset.snapshotId()).isEqualTo(latestSnapshotId);
+    } finally {
+      stream.stop();
+    }
+  }
+
+  @TestTemplate
+  void availableNowResumeStopsAtSnapshotsAvailableAtStart() {
+    appendData(List.of(new SimpleRecord(1, "one")));
+    table.refresh();
+    Snapshot committed = table.currentSnapshot();
+    appendData(List.of(new SimpleRecord(2, "two")));
+    table.refresh();
+    Snapshot latest = table.currentSnapshot();
+    // A timestamp after the newest snapshot stores START_OFFSET as the initial offset
+    SparkMicroBatchStream stream =
+        newMicroBatchStream(
+            ImmutableMap.of(
+                SparkReadOptions.STREAM_FROM_TIMESTAMP,
+                Long.toString(timestampAfterCurrentSnapshot())),
+            "available-now-resume-checkpoint");
+
+    try {
+      stream.prepareForTriggerAvailableNow();
+      appendData(List.of(new SimpleRecord(3, "three")));
+      StreamingOffset committedOffset =
+          new StreamingOffset(
+              committed.snapshotId(), MicroBatchUtils.addedFilesCount(table, committed), false);
+
+      assertThat(stream.latestOffset(committedOffset, stream.getDefaultReadLimit()))
+          .isEqualTo(
+              new StreamingOffset(
+                  latest.snapshotId(), MicroBatchUtils.addedFilesCount(table, latest), false));
+    } finally {
+      stream.stop();
+    }
+  }
+
+  @TestTemplate
+  void availableNowResumeStopsAtLatestAppendSnapshot() {
+    appendData(List.of(new SimpleRecord(1, "one")));
+    table.refresh();
+    Snapshot committed = table.currentSnapshot();
+    appendData(List.of(new SimpleRecord(2, "two")));
+    table.refresh();
+    Snapshot latestAppend = table.currentSnapshot();
+    table.rewriteManifests().clusterBy(file -> 1).commit();
+    table.refresh();
+    assertThat(table.currentSnapshot().operation()).isEqualTo(DataOperations.REPLACE);
+    SparkMicroBatchStream stream =
+        newMicroBatchStream(
+            ImmutableMap.of(
+                SparkReadOptions.STREAM_FROM_TIMESTAMP,
+                Long.toString(timestampAfterCurrentSnapshot())),
+            "available-now-resume-replace-checkpoint");
+
+    try {
+      stream.prepareForTriggerAvailableNow();

Review Comment:
   Good call on compaction. Capping at the newest append makes sense. I tried changing `latestAppendOffset()` to return the current snapshot, which here is the `REPLACE` from `rewriteManifests()`. This test still passes with `async=false`. The sync planner skips the `REPLACE` and stops at the current snapshot anyway, because nothing is committed after prepare. Committing one more snapshot after `prepareForTriggerAvailableNow` makes the test fail for both planners under that change, and it passes on this branch.
   
   ```suggestion
         stream.prepareForTriggerAvailableNow();
         appendData(List.of(new SimpleRecord(3, "three")));
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Don't reapply stream-from-timestamp when a stream resumes [iceberg]

Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on PR #18348:
URL: https://github.com/apache/iceberg/pull/18348#issuecomment-6017224165

   Fair. I've added to the description that only the restart case is fixed. The first batch of a new checkpoint can still start at -1, which we can address in a follow-up.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Don't reapply stream-from-timestamp when a stream resumes [iceberg]

Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #18348:
URL: https://github.com/apache/iceberg/pull/18348#discussion_r4217756050


##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/AsyncSparkMicroBatchPlanner.java:
##########
@@ -252,7 +252,9 @@ public synchronized StreamingOffset latestOffset(StreamingOffset startOffset, Re
       return StreamingOffset.START_OFFSET;
     }
 
-    if (table().currentSnapshot().timestampMillis() < readConf().streamFromTimestamp()) {
+    // Only a new stream starts from the timestamp. A resumed stream continues from its offset.
+    if (startOffset.equals(StreamingOffset.START_OFFSET)
+        && table().currentSnapshot().timestampMillis() < readConf().streamFromTimestamp()) {

Review Comment:
   Yes. Jobs often pass the current time as `stream-from-timestamp` on every launch. After a restart, the newest snapshot was then usually older than that timestamp, so this check returned -1 for a resumed stream. Spark used -1 as the end offset, which skipped everything committed while the job was down and wrote -1 into the offset log. Now a resumed stream continues from its offset. `resumeIgnoresLaterStreamFromTimestamp` tests this.
   



##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/SparkMicroBatchStream.java:
##########


Review Comment:
   Spark doesn't call it for this source. Since it implements `SupportsAdmissionControl`, `MicroBatchExecution` calls `latestOffset(startOffset, limit)` instead. `MicroBatchStream` still requires the no-arg version and we should probably fail if the no-arg version is called, but not sure this should be handled in this PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Don't reapply stream-from-timestamp when a stream resumes [iceberg]

Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #18348:
URL: https://github.com/apache/iceberg/pull/18348#discussion_r4185150625


##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/StreamingInitialOffsetStore.java:
##########
@@ -50,16 +50,28 @@ class StreamingInitialOffsetStore {
   StreamingOffset initialOffset() {
     InputFile inputFile = io.newInputFile(initialOffsetLocation);
     if (inputFile.exists()) {
-      return readOffset(inputFile);
+      StreamingOffset offset = readOffset(inputFile);
+      if (!StreamingOffset.START_OFFSET.equals(offset)) {
+        return offset;
+      }
     }
 
+    // START_OFFSET has no position yet, so it is derived again instead of stored
     StreamingOffset offset = offsetSupplier.get();

Review Comment:
   Agreed. The PR doesn't touch the initial offset store anymore, so a stored -1 is never replaced under a recorded batch. An old checkpoint with -1 under an uncommitted batch still needs the original timestamp to replay it, same as before.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Don't reapply stream-from-timestamp when a stream resumes [iceberg]

Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on PR #18348:
URL: https://github.com/apache/iceberg/pull/18348#issuecomment-5996588456

   I've reduced the scope of this PR. The first version also stopped saving -1 as the initial offset, so that the first batch of a new checkpoint could be replayed after a restart with a different timestamp. That turned out to be a much bigger change with implications on other parts of the code. Please check the update code.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Don't reapply stream-from-timestamp when a stream resumes [iceberg]

Posted by "anuragmantri (via GitHub)" <gi...@apache.org>.
anuragmantri commented on code in PR #18348:
URL: https://github.com/apache/iceberg/pull/18348#discussion_r4186848615


##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/AsyncSparkMicroBatchPlanner.java:
##########
@@ -267,10 +269,16 @@ public synchronized StreamingOffset latestOffset(StreamingOffset startOffset, Re
 
     // if we want to read all available we don't need to scan files, just snapshots
     if (limit instanceof ReadAllAvailable) {
+      // A START_OFFSET cap only means that no snapshot matched stream-from-timestamp when the run
+      // started. Returned as the end offset of a resumed stream, it would reset the stream.
+      boolean capResetsStream =

Review Comment:
   Thanks for slimming this down. The new condition is much easier to follow. I tried one more case on top of `availableNowResumesCheckpointWithUnresolvedInitialOffset`, where a snapshot is committed after `prepareForTriggerAvailableNow`. For a resumed stream with a `START_OFFSET` cap, this branch walks to whatever snapshot is latest when it's called, so the run also reads snapshots committed after it started. The sync planner does the same, because `latestSnapshotId` becomes `-1` and never matches. On a table with steady commits, the AvailableNow run keeps going for as long as new commits arrive.
   
   This variant fails on this branch for both `async=true` and `async=false`. `latestOffset` returns the snapshot committed after prepare.
   
   ```java
     @TestTemplate
     void availableNowResumeStopsAtSnapshotsAvailableAtStart() {
       appendData(List.of(new SimpleRecord(1, "one")));
       table.refresh();
       Snapshot committed = table.currentSnapshot();
       appendData(List.of(new SimpleRecord(2, "two")));
       table.refresh();
       Snapshot latest = table.currentSnapshot();
       SparkMicroBatchStream stream =
           newMicroBatchStream(
               ImmutableMap.of(
                   SparkReadOptions.STREAM_FROM_TIMESTAMP,
                   Long.toString(timestampAfterCurrentSnapshot())),
               "available-now-resume-cap-checkpoint");
   
       try {
         stream.prepareForTriggerAvailableNow();
         appendData(List.of(new SimpleRecord(3, "three")));
         StreamingOffset committedOffset =
             new StreamingOffset(
                 committed.snapshotId(), MicroBatchUtils.addedFilesCount(table, committed), false);
   
         assertThat(stream.latestOffset(committedOffset, stream.getDefaultReadLimit()))
             .isEqualTo(
                 new StreamingOffset(
                     latest.snapshotId(), MicroBatchUtils.addedFilesCount(table, latest), false));
       } finally {
         stream.stop();
       }
     }
   ```
   
   Could `prepareForTriggerAvailableNow` also record the snapshot that is current at that point, and both planners use it as the cap for a resumed stream? A new stream would keep the `START_OFFSET` cap it has today. That would also give the async planner a real end for its initial preload. Right now `initialPreloadEndSnapshot()` returns `table().snapshot(-1)`, which is `null`, so `fillQueueInitialBuffer` skips the preload as if the table were empty.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Don't reapply stream-from-timestamp when a stream resumes [iceberg]

Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on code in PR #18348:
URL: https://github.com/apache/iceberg/pull/18348#discussion_r4209233407


##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/SparkMicroBatchStream.java:
##########
@@ -244,15 +253,32 @@ public ReadLimit getDefaultReadLimit() {
   public void prepareForTriggerAvailableNow() {
     LOG.info("The streaming query reports to use Trigger.AvailableNow");
 
-    lastOffsetForTriggerAvailableNow =
+    StreamingOffset lastOffset =
         (StreamingOffset) latestOffset(initialOffset, ReadLimit.allAvailable());
+    // START_OFFSET means that no snapshot matched stream-from-timestamp. A new stream has nothing
+    // to read in this run, but a resumed stream continues from its offset and needs an actual cap.
+    this.noSnapshotMatchedForTriggerAvailableNow = StreamingOffset.START_OFFSET.equals(lastOffset);
+    this.lastOffsetForTriggerAvailableNow =
+        noSnapshotMatchedForTriggerAvailableNow ? latestAppendOffset() : lastOffset;
 
-    LOG.info("lastOffset for Trigger.AvailableNow is {}", lastOffsetForTriggerAvailableNow.json());
+    LOG.info("lastOffset for Trigger.AvailableNow is {}", lastOffsetForTriggerAvailableNow);

Review Comment:
   why did we remove `.json()` here ? 



##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/SparkMicroBatchStream.java:
##########
@@ -244,15 +253,32 @@ public ReadLimit getDefaultReadLimit() {
   public void prepareForTriggerAvailableNow() {
     LOG.info("The streaming query reports to use Trigger.AvailableNow");
 
-    lastOffsetForTriggerAvailableNow =
+    StreamingOffset lastOffset =
         (StreamingOffset) latestOffset(initialOffset, ReadLimit.allAvailable());
+    // START_OFFSET means that no snapshot matched stream-from-timestamp. A new stream has nothing
+    // to read in this run, but a resumed stream continues from its offset and needs an actual cap.
+    this.noSnapshotMatchedForTriggerAvailableNow = StreamingOffset.START_OFFSET.equals(lastOffset);
+    this.lastOffsetForTriggerAvailableNow =
+        noSnapshotMatchedForTriggerAvailableNow ? latestAppendOffset() : lastOffset;
 
-    LOG.info("lastOffset for Trigger.AvailableNow is {}", lastOffsetForTriggerAvailableNow.json());
+    LOG.info("lastOffset for Trigger.AvailableNow is {}", lastOffsetForTriggerAvailableNow);
 
     // Reset planner so it gets recreated with the cap on next call
     if (planner != null) {
       planner.stop();
       planner = null;
     }
   }
+
+  // The planners only read append snapshots. A cap at any other snapshot is never reached.
+  private StreamingOffset latestAppendOffset() {
+    for (Snapshot snapshot : SnapshotUtil.currentAncestors(table)) {
+      if (DataOperations.APPEND.equals(snapshot.operation())) {
+        return new StreamingOffset(
+            snapshot.snapshotId(), MicroBatchUtils.addedFilesCount(table, snapshot), false);
+      }
+    }
+
+    return null;
+  }

Review Comment:
   I thought we just check the addedDataFiles (check MicrobatchUtils#), as there were discussion on mode to allow operations on non append snapshots ... 



##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/AsyncSparkMicroBatchPlanner.java:
##########
@@ -252,7 +252,9 @@ public synchronized StreamingOffset latestOffset(StreamingOffset startOffset, Re
       return StreamingOffset.START_OFFSET;
     }
 
-    if (table().currentSnapshot().timestampMillis() < readConf().streamFromTimestamp()) {
+    // Only a new stream starts from the timestamp. A resumed stream continues from its offset.
+    if (startOffset.equals(StreamingOffset.START_OFFSET)
+        && table().currentSnapshot().timestampMillis() < readConf().streamFromTimestamp()) {

Review Comment:
   can you please elaborate this case more ... is this protecting the stream resume but the stream from timestamp is leading to skip the snapshot / missing data ? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: Don't reapply stream-from-timestamp when a stream resumes [iceberg]

Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on PR #18348:
URL: https://github.com/apache/iceberg/pull/18348#issuecomment-6057890384

   Thank you @singhpk234 @dramaticlly @huaxingao @anuragmantri  for the reviews! 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org