You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Victor Wong (Jira)" <ji...@apache.org> on 2020/03/02 10:55:00 UTC
[jira] [Updated] (FLINK-16376) Using consistent method to get Yarn
application directory
[ https://issues.apache.org/jira/browse/FLINK-16376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Victor Wong updated FLINK-16376:
--------------------------------
Description:
The Yarn application directory of Flink is "/user/{user.name}/.flink", but this logic is separated in different places.
1. org.apache.flink.yarn.YarnClusterDescriptor#getYarnFilesDir
{code:java}
private Path getYarnFilesDir(final ApplicationId appId) throws IOException {
final FileSystem fileSystem = FileSystem.get(yarnConfiguration);
final Path homeDir = fileSystem.getHomeDirectory();
return new Path(homeDir, ".flink/" + appId + '/');
}
{code}
2. org.apache.flink.yarn.Utils#uploadLocalFileToRemote
{code:java}
// copy resource to HDFS
String suffix =
".flink/"
+ appId
+ (relativeTargetPath.isEmpty() ? "" : "/" + relativeTargetPath)
+ "/" + localSrcPath.getName();
Path dst = new Path(homedir, suffix);
{code}
We can extract `getYarnFilesDir` method to `org.apache.flink.yarn.Utils`, and use this method to get Yarn application directory in all the other places.
was:
The Yarn application directory of Flink is "/user/{user.name}/.flink", but this logic is separated in different places.
1. org.apache.flink.yarn.YarnClusterDescriptor#getYarnFilesDir
{code:java}
private Path getYarnFilesDir(final ApplicationId appId) throws IOException {
final FileSystem fileSystem = FileSystem.get(yarnConfiguration);
final Path homeDir = fileSystem.getHomeDirectory();
return *new Path(homeDir, ".flink/" + appId + '/');*
}
{code}
2. org.apache.flink.yarn.Utils#uploadLocalFileToRemote
{code:java}
// copy resource to HDFS
String suffix =
*".flink/"*
*+ appId*
+ (relativeTargetPath.isEmpty() ? "" : "/" + relativeTargetPath)
+ "/" + localSrcPath.getName();
Path dst = new Path(homedir, suffix);
{code}
We can extract `getYarnFilesDir` method to `org.apache.flink.yarn.Utils`, and use this method to get Yarn application directory in all the other places.
> Using consistent method to get Yarn application directory
> ---------------------------------------------------------
>
> Key: FLINK-16376
> URL: https://issues.apache.org/jira/browse/FLINK-16376
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / YARN
> Affects Versions: 1.10.0
> Reporter: Victor Wong
> Priority: Major
>
> The Yarn application directory of Flink is "/user/{user.name}/.flink", but this logic is separated in different places.
> 1. org.apache.flink.yarn.YarnClusterDescriptor#getYarnFilesDir
> {code:java}
> private Path getYarnFilesDir(final ApplicationId appId) throws IOException {
> final FileSystem fileSystem = FileSystem.get(yarnConfiguration);
> final Path homeDir = fileSystem.getHomeDirectory();
> return new Path(homeDir, ".flink/" + appId + '/');
> }
> {code}
> 2. org.apache.flink.yarn.Utils#uploadLocalFileToRemote
> {code:java}
> // copy resource to HDFS
> String suffix =
> ".flink/"
> + appId
> + (relativeTargetPath.isEmpty() ? "" : "/" + relativeTargetPath)
> + "/" + localSrcPath.getName();
> Path dst = new Path(homedir, suffix);
> {code}
> We can extract `getYarnFilesDir` method to `org.apache.flink.yarn.Utils`, and use this method to get Yarn application directory in all the other places.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)