[GitHub] [iceberg] cccs-jc commented on a diff in pull request #6480: Spark: Fail streaming planning when snapshot not found

["cccs-jc (via GitHub)" <gi...@apache.org>]

cccs-jc commented on code in PR #6480:
URL: https://github.com/apache/iceberg/pull/6480#discussion_r1166043802


##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkMicroBatchStream.java:
##########
@@ -207,7 +207,14 @@ private List<FileScanTask> planFiles(StreamingOffset startOffset, StreamingOffse
         currentOffset = new StreamingOffset(snapshotAfter.snapshotId(), 0L, false);
       }
 
-      if (!shouldProcess(table.snapshot(currentOffset.snapshotId()))) {
+      Snapshot snapshot = table.snapshot(currentOffset.snapshotId());
+
+      if (snapshot == null) {
+        throw new IllegalStateException(
+            String.format("Failed to find expected snapshot %d", snapshot.snapshotId()));
+      }
+

Review Comment:
   agreed, if a snapshot is not found in the table, it's bad. It indicates the reader's checkpoint is too old. There is no nice way to recover from this, except clearing your checkpoints and restarting.
   
   this happen to me because I was clearing my snapshots every hour. I realized then that this made no sense and that I should keep a week of snapshot. Readers have up to a week to fix any streaming job which went down.
   
   So I'm all good with throwing an exception that points the user to the fact that the checkpointed snapshot ID is no longer available in the table.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org