[jira] [Created] (SPARK-55892) Unable to load state store because reused SST file was deleted by maintenance

["B. Micheal Okutubo (Jira)" <ji...@apache.org>]

B. Micheal Okutubo created SPARK-55892:
------------------------------------------

             Summary: Unable to load state store because reused SST file was deleted by maintenance
                 Key: SPARK-55892
                 URL: https://issues.apache.org/jira/browse/SPARK-55892
             Project: Spark
          Issue Type: Task
          Components: Structured Streaming
    Affects Versions: 4.2.0
            Reporter: B. Micheal Okutubo


When there is a large delay between when a snapshot was created and when it was uploaded by maintenance, it is possible for the files that were reused from old versions, which are now eligible for deletion by another executor's maintenance to be deleted.
 
Hence, when trying to load the state store using the uploaded snapshot, it will fail with FileNotFoundException for the reused SST file.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org