You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "francis-a (via GitHub)" <gi...@apache.org> on 2026/07/22 11:06:38 UTC

[PR] FLINK-37686 - Do not fail on NOT_OWNED deleted files [flink]

francis-a opened a new pull request, #28801:
URL: https://github.com/apache/flink/pull/28801

   ## What is the purpose of the change
   
   This PR aims to fix the issue documented in https://issues.apache.org/jira/browse/FLINK-37686 where remote objects cleaned up by the JM cause checkpoint failures even when the object is marked as `NOT_OWNED`.
   
   
   ## Brief change log
   
     - Added a `FileNotFoundException` catch block to `ForStFlinkFileSystem::listStatus`
     - In this catch block we verify the object is `NOT_OWNED`
     - If true, we assume the JM removed this object and delete it using the `FileMappingManager`
     - Tests have been added to cover this case as well as ensuring the exception still bubbles up if the object is **not** `NOT_OWNED`
   
   
   ## Verifying this change
   
   This change added tests and can be verified as follows:
   - Added two new tests to `ForStFlinkFileSystemTest`,  `testListStatusDropsDeletedNotOwnedEntryAfterGiveUpOwnership` and `testListStatusStillFailsForDeletedDbOwnedFile`
   - These tests cover the `NOT_OWNED` and verify the removed object is also removed from the `FileMappingManager`, the other test verifies the baseline hasn't changed for other object types
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): (no)
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
     - The serializers: (yes / no / don't know)
     - The runtime per-record code paths (performance sensitive): (don't know)
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
     - The S3 file system connector: (no)
   
   ## Documentation
   
     - Does this pull request introduce a new feature? (no)
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   
   - [X] Yes
   
   Generated-by: [Claude Opus]
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] FLINK-37686 - Do not fail on NOT_OWNED deleted files [flink]

Posted by "AlexYinHan (via GitHub)" <gi...@apache.org>.
AlexYinHan commented on PR #28801:
URL: https://github.com/apache/flink/pull/28801#issuecomment-5053943353

   @francis-a Thanks for the PR. LGTM. The orphan files deleted by the JM are abandoned by ForSt. They should be removed and not cause failures during a full scan. @Zakelly Could you take a look?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] FLINK-37686 - Do not fail on NOT_OWNED deleted files [flink]

Posted by "francis-a (via GitHub)" <gi...@apache.org>.
francis-a commented on PR #28801:
URL: https://github.com/apache/flink/pull/28801#issuecomment-5107192323

   @Zakelly done


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] FLINK-37686 - Do not fail on NOT_OWNED deleted files [flink]

Posted by "Zakelly (via GitHub)" <gi...@apache.org>.
Zakelly commented on PR #28801:
URL: https://github.com/apache/flink/pull/28801#issuecomment-5107111017

   > thanks @Zakelly, do you plan on handling merging? Is there anything else I should do?
   
   @francis-a yeah, I was about to merge but the CI failed. I think it's irrelevant but the CI bot is not responding to the re-run request. It would be nice if you could amend the commit and force push once, to re-trigger the CI.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] FLINK-37686 - Do not fail on NOT_OWNED deleted files [flink]

Posted by "francis-a (via GitHub)" <gi...@apache.org>.
francis-a commented on PR #28801:
URL: https://github.com/apache/flink/pull/28801#issuecomment-5107233620

   It doesn't look like checks are running, I'll push a minor change


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] FLINK-37686 - Do not fail on NOT_OWNED deleted files [flink]

Posted by "Zakelly (via GitHub)" <gi...@apache.org>.
Zakelly merged PR #28801:
URL: https://github.com/apache/flink/pull/28801


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] FLINK-37686 - Do not fail on NOT_OWNED deleted files [flink]

Posted by "francis-a (via GitHub)" <gi...@apache.org>.
francis-a commented on PR #28801:
URL: https://github.com/apache/flink/pull/28801#issuecomment-5089096546

   thanks @Zakelly, do you plan on handling merging? Is there anything else I should do?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] FLINK-37686 - Do not fail on NOT_OWNED deleted files [flink]

Posted by "Zakelly (via GitHub)" <gi...@apache.org>.
Zakelly commented on PR #28801:
URL: https://github.com/apache/flink/pull/28801#issuecomment-5070803049

   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] FLINK-37686 - Do not fail on NOT_OWNED deleted files [flink]

Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #28801:
URL: https://github.com/apache/flink/pull/28801#issuecomment-5045089652

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f71b66eb658be383ae3e3abc14a098e91d564e48",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f71b66eb658be383ae3e3abc14a098e91d564e48",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f71b66eb658be383ae3e3abc14a098e91d564e48 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org