[jira] [Created] (FLINK-28548) The commit partition base path is not created when no data is sent which may cause FileNotFoundException

["Liu (Jira)" <ji...@apache.org>]

Liu created FLINK-28548:
---------------------------

             Summary: The commit partition base path is not created when no data is sent which may cause FileNotFoundException
                 Key: FLINK-28548
                 URL: https://issues.apache.org/jira/browse/FLINK-28548
             Project: Flink
          Issue Type: Bug
          Components: Connectors / FileSystem
    Affects Versions: 1.15.1, 1.14.5, 1.16.0
            Reporter: Liu


The commit partition base path is not created when no data is sent which may cause FileNotFoundException.  The exception is as following:
{code:java}
Caused by: java.io.FileNotFoundException: File /home/ljg/test_sql.db/flink_batch_test/.staging_1657697612169 does not exist.
	at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:771) ~[hadoop-hdfs-2.6.0U203-cdh5.4.4.jar:?]
	at org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:120) ~[hadoop-hdfs-2.6.0U203-cdh5.4.4.jar:?]
	at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:828) ~[hadoop-hdfs-2.6.0U203-cdh5.4.4.jar:?]
	at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:824) ~[hadoop-hdfs-2.6.0U203-cdh5.4.4.jar:?]
	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) ~[hadoop-common-2.6.0U203-cdh5.4.4.jar:?]
	at org.apache.hadoop.hdfs.perflog.FileSystemLinkResolverWithStatistics$1.doCall(FileSystemLinkResolverWithStatistics.java:37) ~[hadoop-hdfs-2.6.0U203-cdh5.4.4.jar:?]
	at org.apache.hadoop.hdfs.perflog.PerfProxy.call(PerfProxy.java:49) ~[hadoop-hdfs-2.6.0U203-cdh5.4.4.jar:?]
	at org.apache.hadoop.hdfs.perflog.FileSystemLinkResolverWithStatistics.resolve(FileSystemLinkResolverWithStatistics.java:39) ~[hadoop-hdfs-2.6.0U203-cdh5.4.4.jar:?]
	at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:835) ~[hadoop-hdfs-2.6.0U203-cdh5.4.4.jar:?]
	at org.apache.hadoop.fs.FilterFileSystem.listStatus(FilterFileSystem.java:238) ~[hadoop-common-2.6.0U203-cdh5.4.4.jar:?]
	at org.apache.hadoop.fs.FilterFileSystem.listStatus(FilterFileSystem.java:238) ~[hadoop-common-2.6.0U203-cdh5.4.4.jar:?]
	at org.apache.hadoop.fs.viewfs.ChRootedFileSystem.listStatus(ChRootedFileSystem.java:241) ~[hadoop-common-2.6.0U203-cdh5.4.4.jar:?]
	at org.apache.hadoop.fs.viewfs.ViewFileSystem.listStatus(ViewFileSystem.java:376) ~[hadoop-common-2.6.0U203-cdh5.4.4.jar:?]
	at org.apache.flink.hive.shaded.fs.hdfs.HadoopFileSystem.listStatus(HadoopFileSystem.java:170) ~[flink-sql-connector-hive-2.3.6_2.12-1.15.0.jar:1.15.0]
	at org.apache.flink.connector.file.table.PartitionTempFileManager.listTaskTemporaryPaths(PartitionTempFileManager.java:87) ~[flink-connector-files-1.15.0.jar:1.15.0]
	at org.apache.flink.connector.file.table.FileSystemCommitter.commitPartitions(FileSystemCommitter.java:78) ~[flink-connector-files-1.15.0.jar:1.15.0]
	at org.apache.flink.connector.file.table.FileSystemOutputFormat.finalizeGlobal(FileSystemOutputFormat.java:89) ~[flink-connector-files-1.15.0.jar:1.15.0]
	at org.apache.flink.runtime.jobgraph.InputOutputFormatVertex.finalizeOnMaster(InputOutputFormatVertex.java:153) ~[flink-dist-1.15.0.jar:1.15.0]
	at org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.jobFinished(DefaultExecutionGraph.java:1190) ~[flink-dist-1.15.0.jar:1.15.0]
	... 43 more {code}
We should check whether the base path exists before listStatus for the path.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)