You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2022/07/14 04:23:00 UTC
[jira] [Updated] (FLINK-28548) The commit partition base path is not created when no data is sent which may cause FileNotFoundException
[ https://issues.apache.org/jira/browse/FLINK-28548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated FLINK-28548:
-----------------------------------
Labels: pull-request-available (was: )
> The commit partition base path is not created when no data is sent which may cause FileNotFoundException
> --------------------------------------------------------------------------------------------------------
>
> Key: FLINK-28548
> URL: https://issues.apache.org/jira/browse/FLINK-28548
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
> Affects Versions: 1.14.5, 1.15.1, 1.16.0
> Reporter: Liu
> Priority: Major
> Labels: pull-request-available
>
> The commit partition base path is not created when no data is sent which may cause FileNotFoundException. The exception is as following:
> {code:java}
> Caused by: java.io.FileNotFoundException: File /home/ljg/test_sql.db/flink_batch_test/.staging_1657697612169 does not exist.
> at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:771) ~[hadoop-hdfs-2.6.0U203-cdh5.4.4.jar:?]
> at org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:120) ~[hadoop-hdfs-2.6.0U203-cdh5.4.4.jar:?]
> at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:828) ~[hadoop-hdfs-2.6.0U203-cdh5.4.4.jar:?]
> at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:824) ~[hadoop-hdfs-2.6.0U203-cdh5.4.4.jar:?]
> at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) ~[hadoop-common-2.6.0U203-cdh5.4.4.jar:?]
> at org.apache.hadoop.hdfs.perflog.FileSystemLinkResolverWithStatistics$1.doCall(FileSystemLinkResolverWithStatistics.java:37) ~[hadoop-hdfs-2.6.0U203-cdh5.4.4.jar:?]
> at org.apache.hadoop.hdfs.perflog.PerfProxy.call(PerfProxy.java:49) ~[hadoop-hdfs-2.6.0U203-cdh5.4.4.jar:?]
> at org.apache.hadoop.hdfs.perflog.FileSystemLinkResolverWithStatistics.resolve(FileSystemLinkResolverWithStatistics.java:39) ~[hadoop-hdfs-2.6.0U203-cdh5.4.4.jar:?]
> at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:835) ~[hadoop-hdfs-2.6.0U203-cdh5.4.4.jar:?]
> at org.apache.hadoop.fs.FilterFileSystem.listStatus(FilterFileSystem.java:238) ~[hadoop-common-2.6.0U203-cdh5.4.4.jar:?]
> at org.apache.hadoop.fs.FilterFileSystem.listStatus(FilterFileSystem.java:238) ~[hadoop-common-2.6.0U203-cdh5.4.4.jar:?]
> at org.apache.hadoop.fs.viewfs.ChRootedFileSystem.listStatus(ChRootedFileSystem.java:241) ~[hadoop-common-2.6.0U203-cdh5.4.4.jar:?]
> at org.apache.hadoop.fs.viewfs.ViewFileSystem.listStatus(ViewFileSystem.java:376) ~[hadoop-common-2.6.0U203-cdh5.4.4.jar:?]
> at org.apache.flink.hive.shaded.fs.hdfs.HadoopFileSystem.listStatus(HadoopFileSystem.java:170) ~[flink-sql-connector-hive-2.3.6_2.12-1.15.0.jar:1.15.0]
> at org.apache.flink.connector.file.table.PartitionTempFileManager.listTaskTemporaryPaths(PartitionTempFileManager.java:87) ~[flink-connector-files-1.15.0.jar:1.15.0]
> at org.apache.flink.connector.file.table.FileSystemCommitter.commitPartitions(FileSystemCommitter.java:78) ~[flink-connector-files-1.15.0.jar:1.15.0]
> at org.apache.flink.connector.file.table.FileSystemOutputFormat.finalizeGlobal(FileSystemOutputFormat.java:89) ~[flink-connector-files-1.15.0.jar:1.15.0]
> at org.apache.flink.runtime.jobgraph.InputOutputFormatVertex.finalizeOnMaster(InputOutputFormatVertex.java:153) ~[flink-dist-1.15.0.jar:1.15.0]
> at org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.jobFinished(DefaultExecutionGraph.java:1190) ~[flink-dist-1.15.0.jar:1.15.0]
> ... 43 more {code}
> We should check whether the base path exists before listStatus for the path.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)