[jira] [Created] (FLINK-23725) HadoopFsCommitter， Prompt for file rename failure

["todd (Jira)" <ji...@apache.org>]

todd created FLINK-23725:
----------------------------

             Summary: HadoopFsCommitter， Prompt for file rename failure
                 Key: FLINK-23725
                 URL: https://issues.apache.org/jira/browse/FLINK-23725
             Project: Flink
          Issue Type: Bug
          Components: Connectors / FileSystem, Connectors / Hadoop Compatibility
    Affects Versions: 1.12.1, 1.11.1
            Reporter: todd


When the HDFS file is written, if the part file exists, only false will be returned if the duplicate name fails.Whether to throw an exception that already exists in the part, or print related logs.

 

```

org.apache.flink.runtime.fs.hdfs.HadoopRecoverableFsDataOutputStream.HadoopFsCommitter#commit

 

public void commit() throws IOException {
 final Path src = recoverable.tempFile();
 final Path dest = recoverable.targetFile();
 final long expectedLength = recoverable.offset();

 final FileStatus srcStatus;
 try {
 srcStatus = fs.getFileStatus(src);
 } catch (IOException e) {
 throw new IOException("Cannot clean commit: Staging file does not exist.");
 }

 if (srcStatus.getLen() != expectedLength) {
 // something was done to this file since the committer was created.
 // this is not the "clean" case
 throw new IOException("Cannot clean commit: File has trailing junk data.");
 }

 try {
 // return false or ture
 fs.rename(src, dest);
 } catch (IOException e) {
 throw new IOException(
 "Committing file by rename failed: " + src + " to " + dest, e);
 }
}

 

 

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)