[GitHub] [iceberg] singhpk234 commented on a diff in pull request #6480: Spark: Fail streaming planning when snapshot not found

[GitBox <gi...@apache.org>]

singhpk234 commented on code in PR #6480:
URL: https://github.com/apache/iceberg/pull/6480#discussion_r1059091796


##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkMicroBatchStream.java:
##########
@@ -207,7 +207,14 @@ private List<FileScanTask> planFiles(StreamingOffset startOffset, StreamingOffse
         currentOffset = new StreamingOffset(snapshotAfter.snapshotId(), 0L, false);
       }
 
-      if (!shouldProcess(table.snapshot(currentOffset.snapshotId()))) {
+      Snapshot snapshot = table.snapshot(currentOffset.snapshotId());
+
+      if (snapshot == null) {
+        throw new IllegalStateException(
+            String.format("Failed to find expected snapshot %d", snapshot.snapshotId()));
+      }
+

Review Comment:
   as per my understanding returning `false` from shouldProcess will make it skip the expired snapshot, which might not be expected if the expired snapshot was of type `APPEND`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org