You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "L. C. Hsieh (Jira)" <ji...@apache.org> on 2022/03/19 20:17:00 UTC

[jira] [Commented] (SPARK-38605) Retrying on file manager operation in HDFSMetadataLog

    [ https://issues.apache.org/jira/browse/SPARK-38605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17509337#comment-17509337 ] 

L. C. Hsieh commented on SPARK-38605:
-------------------------------------

cc [~kabhwan] Please let me know if you have some concerns or ideas, before I work on this. I'm fine too if you're interested to work on this.

> Retrying on file manager operation in HDFSMetadataLog
> -----------------------------------------------------
>
>                 Key: SPARK-38605
>                 URL: https://issues.apache.org/jira/browse/SPARK-38605
>             Project: Spark
>          Issue Type: Improvement
>          Components: Structured Streaming
>    Affects Versions: 3.4.0
>            Reporter: L. C. Hsieh
>            Priority: Major
>
> Currently HDFSMetadataLog uses CheckpointFileManager to do some file operation like opening metadata file. It is very easy to be affected by network blips and causes the streaming query failed. Although we can restart the streaming query, but it takes more time to recover.
> Such file operations should be resilient with such situation by retrying.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org