[GitHub] [spark] martin-g commented on a change in pull request #35458: [SPARK-38033] The structured streaming processing cannot be started b…

[GitBox <gi...@apache.org>]

martin-g commented on a change in pull request #35458:
URL: https://github.com/apache/spark/pull/35458#discussion_r802331141



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala
##########
@@ -282,7 +282,27 @@ class MicroBatchExecution(
         /* Initialize committed offsets to a committed batch, which at this
          * is the second latest batch id in the offset log. */
         if (latestBatchId != 0) {
+          /* SPARK-38033: In some unexpected cases, commit and offset are inconsistent,
+            * and offset is not written into HDFS continuously, as follows:
+            *
+            * commits
+            * /tmp/streaming_xxxxxxxx/commits/113256
+            * /tmp/streaming_xxxxxxxx/commits/113257
+            * offsets
+            * /tmp/streaming_xxxxxxxx/offsets/113257
+            * /tmp/streaming_xxxxxxxx/offsets/113259
+            *
+            * When we start the streaming program, batch ${latestBatchId - 1} is 113258,
+            * but offsets 113258 doesn't exist, an exception will be thrown,resulting in
+            * the program cannot be started. As an improvement, we could probably do some
+            * simply analysis and give better error message. */
           val secondLatestOffsets = offsetLog.get(latestBatchId - 1).getOrElse {
+            logError(s"Please check the checkpoint, batch ${latestBatchId - 1} doesn't exist. " +
+              s"If the latest offset is $latestBatchId, the latest commit is ${latestBatchId - 2}" +
+              s" and offset ${latestBatchId - 1} doesn't exist. You can try to remove the offset" +
+              s" $latestBatchId and start over. If your query aims end-to-end exactly once" +
+              s" semantic, and you can may want to also remove the output from the batch" +

Review comment:
       `... you can may want to also remove ...` 
   I am not native English speaker but this part of the sentence sounds wrong to me.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org