[GitHub] [spark] HeartSaVioR commented on pull request #35994: [SPARK-38605][SS] Retrying on file manager open operation in HDFSMetadataLog

[GitBox <gi...@apache.org>]

HeartSaVioR commented on pull request #35994:
URL: https://github.com/apache/spark/pull/35994#issuecomment-1082539764


   Probably there is another good point the FS API is the better layer to fix.
   
   In Spark's point of view, there is no notion of distinguish whether the failure is able to retry or not (we have to deal with NonFatal since we have no context, which IOException may be even too general), whereas the FS client layer it should have more context to decide. Once the client layer decides that the failure is able to retry without changing semantic (e.g. the operation is idempotent), the client layer can do the retry and it can be transparent to Spark.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org