[GitHub] [spark] uncleGen opened a new pull request #24283: [SPARK-27355][SS] Make query execution more sensitive to epoch message late or lost

[GitBox <gi...@apache.org>]

uncleGen opened a new pull request #24283: [SPARK-27355][SS] Make query execution more sensitive to epoch message late or lost
URL: https://github.com/apache/spark/pull/24283
 
 
   ## What changes were proposed in this pull request?
   
   In SPARK-23503, we enforce sequencing of committed epochs for Continuous Execution. In case a message for epoch n is lost and epoch (n + 1) is ready for commit before epoch n is, epoch (n + 1) will wait for epoch n to be committed first. With extreme condition, we will wait for `epochBacklogQueueSize` (10000 in default) epochs and then failed. There is no need to wait for such a long time before query fail, and we can make the condition more sensitive.
   
   ## How was this patch tested?
   
   update existing unit tests.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org