You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Zakelly (via GitHub)" <gi...@apache.org> on 2025/03/07 19:15:57 UTC
[PR] [FLINK-37437][state/forst] Fix #exist and #getFileStatus to give the information of writing files [flink]
Zakelly opened a new pull request, #26271:
URL: https://github.com/apache/flink/pull/26271
## What is the purpose of the change
We leverage `disableFileDeletions` and `enableFileDeletions` of ForSt to keep files alive during checkpoint. When checkpoint finished, the `enableFileDeletions` will be invoked, while ForSt will list all the files and check status. These files, however, include the writing ones, which is not visible in some DFS (e.g. oss or s3) yet. Although the checkpoints are actually not missing, the thrown exception make the finalization of checkpoint failed, resulting in consecutive cp failures.
The root cause would be the `ForStFlinkFileSystem` does not give correct information of files which are being written. This PR fixes this.
## Brief change log
*(for example:)*
- Fixes `#exist` and `getFileStatus` to give information of writing files.
- Enrich `MappingEntry` and introduce `FSDataOutputStreamWithEntry` to maintain the writing flag of file.
## Verifying this change
New UT added.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37437][state/forst] Fix #exist and #getFileStatus to give the information of writing files [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #26271:
URL: https://github.com/apache/flink/pull/26271#issuecomment-2707315746
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ffed00e49bd1da0533caacd086d078d00f279d02",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ffed00e49bd1da0533caacd086d078d00f279d02",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ffed00e49bd1da0533caacd086d078d00f279d02 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37437][state/forst] Fix #exist and #getFileStatus to give the information of writing files [flink]
Posted by "Zakelly (via GitHub)" <gi...@apache.org>.
Zakelly commented on code in PR #26271:
URL: https://github.com/apache/flink/pull/26271#discussion_r1986038886
##########
flink-state-backends/flink-statebackend-forst/src/main/java/org/apache/flink/state/forst/fs/ForStFlinkFileSystem.java:
##########
@@ -273,18 +277,38 @@ public synchronized boolean exists(final Path f) throws IOException {
}
if (FileOwnershipDecider.shouldAlwaysBeLocal(f)) {
- return localFS.exists(mappingEntry.getSourcePath())
- || delegateFS.exists(mappingEntry.getSourcePath());
+ return localFS.exists(mappingEntry.getSourcePath());
} else {
+ // Should be protected with synchronized, since the file closing is not an atomic
+ // operation, see FSDataOutputStreamWithEntry.close()
+ synchronized (mappingEntry) {
+ if (mappingEntry.isWriting()) {
+ return true;
+ }
+ }
return delegateFS.exists(mappingEntry.getSourcePath());
}
}
@Override
public synchronized FileStatus getFileStatus(Path path) throws IOException {
Review Comment:
No need as it never be called for now. But I still change the `getFileBlockLocations`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37437][state/forst] Fix #exist and #getFileStatus to give the information of writing files [flink]
Posted by "fredia (via GitHub)" <gi...@apache.org>.
fredia commented on code in PR #26271:
URL: https://github.com/apache/flink/pull/26271#discussion_r1985936788
##########
flink-state-backends/flink-statebackend-forst/src/main/java/org/apache/flink/state/forst/fs/ForStFlinkFileSystem.java:
##########
@@ -273,18 +277,38 @@ public synchronized boolean exists(final Path f) throws IOException {
}
if (FileOwnershipDecider.shouldAlwaysBeLocal(f)) {
- return localFS.exists(mappingEntry.getSourcePath())
- || delegateFS.exists(mappingEntry.getSourcePath());
+ return localFS.exists(mappingEntry.getSourcePath());
} else {
+ // Should be protected with synchronized, since the file closing is not an atomic
+ // operation, see FSDataOutputStreamWithEntry.close()
+ synchronized (mappingEntry) {
+ if (mappingEntry.isWriting()) {
+ return true;
+ }
+ }
return delegateFS.exists(mappingEntry.getSourcePath());
}
}
@Override
public synchronized FileStatus getFileStatus(Path path) throws IOException {
Review Comment:
Does `getFileBlockLocations` need to be modified like `getFileStatus`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37437][state/forst] Fix #exist and #getFileStatus to give the information of writing files [flink]
Posted by "Zakelly (via GitHub)" <gi...@apache.org>.
Zakelly merged PR #26271:
URL: https://github.com/apache/flink/pull/26271
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org