You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "L. C. Hsieh (Jira)" <ji...@apache.org> on 2022/10/18 03:15:00 UTC

[jira] [Created] (SPARK-40826) Add additional checkpoint rename file check

L. C. Hsieh created SPARK-40826:
-----------------------------------

             Summary: Add additional checkpoint rename file check
                 Key: SPARK-40826
                 URL: https://issues.apache.org/jira/browse/SPARK-40826
             Project: Spark
          Issue Type: Improvement
          Components: Structured Streaming
    Affects Versions: 3.4.0
            Reporter: L. C. Hsieh


We encountered an issue recently that one customer's structured streaming job failed to read delta file. 

The temporary file exists but it was not successfully renamed to final delta file path. 

We currently don't check if renamed file exists but assume it successful. As the result, failing to read delta file assumed to be committed in last batch makes re-triggering the job impossible. 

We should be able to do a check against checkpoint renamed file to prevent such difficulty in advance.






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org