[jira] [Resolved] (SPARK-55892) Unable to load state store because reused SST file was deleted by maintenance

["Anish Shrigondekar (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-55892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Anish Shrigondekar resolved SPARK-55892.
----------------------------------------
    Fix Version/s: 4.2.0
       Resolution: Fixed

Issue resolved by pull request 54697
[https://github.com/apache/spark/pull/54697]

> Unable to load state store because reused SST file was deleted by maintenance
> -----------------------------------------------------------------------------
>
>                 Key: SPARK-55892
>                 URL: https://issues.apache.org/jira/browse/SPARK-55892
>             Project: Spark
>          Issue Type: Task
>          Components: Structured Streaming
>    Affects Versions: 4.2.0
>            Reporter: B. Micheal Okutubo
>            Assignee: B. Micheal Okutubo
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.2.0
>
>
> When there is a large delay between when a snapshot was created and when it was uploaded by maintenance, it is possible for the files that were reused from old versions, which are now eligible for deletion by another executor's maintenance to be deleted.
>  
> Hence, when trying to load the state store using the uploaded snapshot, it will fail with FileNotFoundException for the reused SST file.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org