[GitHub] [spark] yaooqinn edited a comment on pull request #27943: [SPARK-31179] Fast fail the connection while last connection failed in fast fail time window

[GitBox <gi...@apache.org>]

yaooqinn edited a comment on pull request #27943:
URL: https://github.com/apache/spark/pull/27943#issuecomment-636594123


   I have applied this patch to our in-house spark distribution and run for more than one week. Sometimes, making things worse, especially, when the shuffle server JUST in GC STW periods but not eventually become unreachable. Tasks become more sensitive to fail and stage retries occur which may be unnecessary at all and heavy.  Also, the upper limit of `spark.stage.maxConsecutiveAttempts` is much easier to hit and fail the job rather than to get more chances to recover.
   
   ```scala
   org.apache.spark.shuffle.FetchFailedException:
    Connecting to xxxx failed in the last 5700 ms, fail this connection directly
   ```
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org