You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Yang Wang (Jira)" <ji...@apache.org> on 2019/11/05 10:11:00 UTC
[jira] [Commented] (FLINK-14382) Incorrect handling of
FLINK_PLUGINS_DIR on Yarn
[ https://issues.apache.org/jira/browse/FLINK-14382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16967410#comment-16967410 ]
Yang Wang commented on FLINK-14382:
-----------------------------------
[~pnowojski] [~trohrmann]
I have attached a PR to fix this issue. Please help to review.
[https://github.com/apache/flink/pull/10084]
> Incorrect handling of FLINK_PLUGINS_DIR on Yarn
> -----------------------------------------------
>
> Key: FLINK-14382
> URL: https://issues.apache.org/jira/browse/FLINK-14382
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
> Affects Versions: 1.9.0, 1.10.0
> Reporter: Yang Wang
> Assignee: Yang Wang
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.10.0, 1.9.2
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> *(This ticket is a blocker for 1.10 release while critical for 1.9.x)*
> When creating and starting up the yarn containers there are two issues with how the {{FLINK_PLUGINS_DIR}} is being handled.
> # Content of the {{plugins}} directory is currently added to the class path, braking the encapsulation of the plugins from one another
> # {{FLINK_PLUGINS_DIR}} is passed to the container as an absolute path as seen by the client. Because of that TaskManager or JobManager can not use it.
> Both bugs are probably contained to {{YarnClusterDescriptor#startAppMaster}} method (which calls relevant {{addEnvironmentFoldersToShipFiles}} and {{uploadAndRegisterFiles}} methods)
> (original description)
> If we do not set FLINK_PLUGINS_DIR in flink-conf.yaml, it will be set to [flink configuration|https://github.com/apache/flink/blob/9e6ff81e22d6f5f04abb50ca1aea84fd2542bf9d/flink-core/src/main/java/org/apache/flink/configuration/GlobalConfiguration.java#L158] according to the environment.
> In yarn mode, the local path will be set in flink-conf.yaml and used by jobmanager and taskmanager. We will find the warning log like below.
> {code:java}
> 2019-10-12 19:24:58,165 WARN org.apache.flink.core.plugin.PluginConfig - Environment variable [FLINK_PLUGINS_DIR] is set to [/Users/wangy/IdeaProjects/apache-flink/flink-dist/target/flink-1.10-SNAPSHOT-bin/flink-1.10-SNAPSHOT/plugins] but the directory doesn't exist
> {code}
> It was in introduced by FLINK-12143.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)