Re: [PR] [SPARK-56720][SS] Fail subsequent async log writes after a prior failure in async progress tracking [spark]

["jerrypeng (via GitHub)" <gi...@apache.org>]

jerrypeng commented on code in PR #55676:
URL: https://github.com/apache/spark/pull/55676#discussion_r3192305578


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/AsyncProgressTrackingMicroBatchExecution.scala:
##########
@@ -57,6 +57,13 @@ class AsyncProgressTrackingMicroBatchExecution(
   // used to check during the first batch if the pipeline is stateful
   private var isFirstBatch: Boolean = true
 
+  // Records the first error seen by any async log write task. Subsequent async
+  // log writes short-circuit by failing with this error before touching storage.
+  // This prevents creating gaps on durable storage (e.g. offset N missing while
+  // offset N+1 is present, or commit N+1 written without offset N) when an

Review Comment:
   > This prevents creating gaps on durable storage (e.g. offset N missing while
     // offset N+1 is present, or commit N+1 written without offset N
   
   There can be gaps in terms of batches when the APT interval is set but what cannot happen is commit log for batch n is written but not offset log for batch n.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org