[GitHub] [spark] uncleGen edited a comment on issue #24283: [SPARK-27355][SS] Make query execution more sensitive to epoch message late or lost

[GitBox <gi...@apache.org>]

uncleGen edited a comment on issue #24283: [SPARK-27355][SS] Make query execution more sensitive to epoch message late or lost
URL: https://github.com/apache/spark/pull/24283#issuecomment-480701061
 
 
   @gaborgsomogyi Thanks for your reply.  [#23156](https://github.com/apache/spark/pull/20936) introduced a maximum queue threshold before stop the stream with a error. In [#23156](https://github.com/apache/spark/pull/20936) , we used the same threshold for different queue, i.e. `partitionCommits`, `partitionOffsets` and `epochsWaitingToBeCommitted`. Generally, the size of `partitionCommits` and `partitionOffsets` grow much faster than `epochsWaitingToBeCommitted`. The stream may fail with 10 epochs if partition number is 100. However, we may wait for 10000 epochs before failure if partition number is 1 (if i understand correctly). It is such a long time before query fail. Well, this may be just a harsh boundary condition. The main concern of PR is to split there two thresholds to make query execution more sensitive to epoch message late or lost. If you feel like `10 epoch` is too sensitive in some  intermittent problem, we can relax this condition to 100 or other.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org