[jira] [Updated] (FLINK-20935) can't write flink configuration to tmp file and add it to local resource in yarn session mode

["Flink Jira Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-20935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Flink Jira Bot updated FLINK-20935:
-----------------------------------
    Labels: auto-deprioritized-major auto-unassigned pull-request-available stale-major  (was: auto-deprioritized-major auto-unassigned pull-request-available)

I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 60 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized.


> can't write flink configuration to tmp file and add it to local resource in yarn session mode
> ---------------------------------------------------------------------------------------------
>
>                 Key: FLINK-20935
>                 URL: https://issues.apache.org/jira/browse/FLINK-20935
>             Project: Flink
>          Issue Type: Bug
>          Components: Deployment / YARN
>    Affects Versions: 1.12.0, 1.13.0
>            Reporter: yuemeng
>            Priority: Major
>              Labels: auto-deprioritized-major, auto-unassigned, pull-request-available, stale-major
>
> In flink 1.12.0 or lastest version,when we execute command such as bin/yarn-session.sh -n 20 -jm 9096 -nm 4096 -st,the depoy will be failed with follow errors:
> {code}
> org.apache.flink.client.deployment.ClusterDeploymentException: Couldn't deploy Yarn session cluster
> 	at org.apache.flink.yarn.YarnClusterDescriptor.deploySessionCluster(YarnClusterDescriptor.java:411)
> 	at org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:498)
> 	at org.apache.flink.yarn.cli.FlinkYarnSessionCli.lambda$main$4(FlinkYarnSessionCli.java:730)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:422)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754)
> 	at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
> 	at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:730)
> Caused by: java.io.FileNotFoundException: File does not exist: /tmp/application_1573723355201_0036-flink-conf.yaml688141408443326132.tmp
> 	at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1309)
> {code}
> when we called startAppMaster method in YarnClusterDescriptor,it will be try to write flink configuration to tmp file and add it to local resource. but the follow code will make the tmp file system as a distribute file system
> {code}
> // Upload the flink configuration
> 		// write out configuration file
> 		File tmpConfigurationFile = null;
> 		try {
> 			tmpConfigurationFile = File.createTempFile(appId + "-flink-conf.yaml", null);
> 			BootstrapTools.writeConfiguration(configuration, tmpConfigurationFile);
> 			String flinkConfigKey = "flink-conf.yaml";
> 			fileUploader.registerSingleLocalResource(
> 				flinkConfigKey,
> 				new Path(tmpConfigurationFile.getAbsolutePath()),
> 				"",
> 				LocalResourceType.FILE,
> 				true,
> 				true);
> 			classPathBuilder.append("flink-conf.yaml").append(File.pathSeparator);
> 		} finally {
> 			if (tmpConfigurationFile != null && !tmpConfigurationFile.delete()) {
> 				LOG.warn("Fail to delete temporary file {}.", tmpConfigurationFile.toPath());
> 			}
> 		}
> {code}
> {code} tmpConfigurationFile.getAbsolutePath() {code} method will be return a path without file schema and the file system will be considered as a distribute file system



--
This message was sent by Atlassian Jira
(v8.3.4#803005)