[jira] [Created] (FLINK-25436) Allow BlobServer/BlobCache to clean up unused blobs after recovering from working directory

["Till Rohrmann (Jira)" <ji...@apache.org>]

Till Rohrmann created FLINK-25436:
-------------------------------------

             Summary: Allow BlobServer/BlobCache to clean up unused blobs after recovering from working directory
                 Key: FLINK-25436
                 URL: https://issues.apache.org/jira/browse/FLINK-25436
             Project: Flink
          Issue Type: Sub-task
          Components: Runtime / Coordination
    Affects Versions: 1.15.0
            Reporter: Till Rohrmann
            Assignee: Till Rohrmann
             Fix For: 1.15.0


In order to let the {{BlobServer}} and the {{BlobCache}} properly clean up unused blobs that are recovered from the working directory, we have to register them for clean up and offer hooks to delete irrelevant job artifacts.

I propose to scan the blobStorage directory at startup and to register for transient blobs the expiry timeouts. Moreover, for the {{BlobServer}} we need to add a {{retainJobs}} method that deletes all jobs that are not in the given list of {{JobIDs}}. Last but not least we also need to register the permanent blobs in the {{PermanentBlobCacheService}} so that they get timed out if not used anymore.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)