[GitHub] [spark] zsxwing commented on pull request #35994: [SPARK-38605][SS] Retrying on file manager open operation in HDFSMetadataLog

[GitBox <gi...@apache.org>]

zsxwing commented on pull request #35994:
URL: https://github.com/apache/spark/pull/35994#issuecomment-1082453714


   > It is very easy to be affected by network blips and causes the streaming query failed.
   
   Is it a unique issue of HDFS? If so, I’m surprised that HDFS client cannot survive from transient errors. Is Spark the right layer to fix this issue?
   
   In addition, will the same issue happen after `open`? For example, when reading the file content? Do we need to worry about other places as well? This is not the only place that touches FileSystem in the driver.
   
   Could you also add a unit test to verify the retry code? For example, you can use a fake file system to simulate the errors from `open`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org