You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@zeppelin.apache.org by zj...@apache.org on 2020/06/25 11:14:35 UTC
[zeppelin] branch branch-0.9 updated: [ZEPPELIN-4906].
FLINK_CONF_DIR doesn't work in yarn interpreter mode
This is an automated email from the ASF dual-hosted git repository.
zjffdu pushed a commit to branch branch-0.9
in repository https://gitbox.apache.org/repos/asf/zeppelin.git
The following commit(s) were added to refs/heads/branch-0.9 by this push:
new c72eaf2 [ZEPPELIN-4906]. FLINK_CONF_DIR doesn't work in yarn interpreter mode
c72eaf2 is described below
commit c72eaf217384de93d9405055be90f03bb8be7249
Author: Jeff Zhang <zj...@apache.org>
AuthorDate: Wed Jun 24 10:48:10 2020 +0800
[ZEPPELIN-4906]. FLINK_CONF_DIR doesn't work in yarn interpreter mode
### What is this PR for?
This use env instead of properties for these flink related config. Because the env will be reset in yarn interpreter mode (https://github.com/apache/zeppelin/blob/master/zeppelin-plugins/launcher/yarn/src/main/java/org/apache/zeppelin/interpreter/launcher/YarnRemoteInterpreterProcess.java#L295)
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-4906
### How should this be tested?
* Manually tested,
Here's what I see after this PR.
```
INFO [2020-06-24 10:46:06,461] ({FIFOScheduler-interpreter_583675831-Worker-1} FlinkScalaInterpreter.scala[initFlinkConfig]:153) - FLINK_HOME: /Users/jzhang/Temp/hadoop/yarn/usercache/jzhang/appcache/application_1591195707498_0116/container_1591195707498_0116_01_000001/flink
INFO [2020-06-24 10:46:06,462] ({FIFOScheduler-interpreter_583675831-Worker-1} FlinkScalaInterpreter.scala[initFlinkConfig]:154) - FLINK_CONF_DIR: /Users/jzhang/Temp/hadoop/yarn/usercache/jzhang/appcache/application_1591195707498_0116/container_1591195707498_0116_01_000001/flink/conf
INFO [2020-06-24 10:46:06,462] ({FIFOScheduler-interpreter_583675831-Worker-1} FlinkScalaInterpreter.scala[initFlinkConfig]:155) - HADOOP_CONF_DIR: /Users/jzhang/Java/lib/hadoop-2.7.7/etc/hadoop
INFO [2020-06-24 10:46:06,462] ({FIFOScheduler-interpreter_583675831-Worker-1} FlinkScalaInterpreter.scala[initFlinkConfig]:156) - YARN_CONF_DIR: /Users/jzhang/Java/lib/hadoop-2.7.7/etc/hadoop
INFO [2020-06-24 10:46:06,462] ({FIFOScheduler-interpreter_583675831-Worker-1} FlinkScalaInterpreter.scala[initFlinkConfig]:157) - HIVE_CONF_DIR:
INFO [2020-06-24 10:46:06,464] ({FIFOScheduler-interpreter_583675831-Worker-1} FlinkScalaInterpreter.scala[initFlinkConfig]:160) - Using flink: 1.10.0
INFO [2020-06-24 10:46:06,465] ({FIFOScheduler-interpreter_583675831-Worker-1} FlinkShims.java[loadShims]:51) - Initializing shims for Flink 1.10
```
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <zj...@apache.org>
Closes #3820 from zjffdu/ZEPPELIN-4906 and squashes the following commits:
e38cc301b [Jeff Zhang] [ZEPPELIN-4906]. FLINK_CONF_DIR doesn't work in yarn interpreter mode
(cherry picked from commit 5dedab9e507257fff428abf75a32b45536cb04ba)
Signed-off-by: Jeff Zhang <zj...@apache.org>
---
.../org/apache/zeppelin/flink/FlinkScalaInterpreter.scala | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/flink/interpreter/src/main/scala/org/apache/zeppelin/flink/FlinkScalaInterpreter.scala b/flink/interpreter/src/main/scala/org/apache/zeppelin/flink/FlinkScalaInterpreter.scala
index 027838e..1175b7b 100644
--- a/flink/interpreter/src/main/scala/org/apache/zeppelin/flink/FlinkScalaInterpreter.scala
+++ b/flink/interpreter/src/main/scala/org/apache/zeppelin/flink/FlinkScalaInterpreter.scala
@@ -145,11 +145,11 @@ class FlinkScalaInterpreter(val properties: Properties) {
}
private def initFlinkConfig(): Config = {
- val flinkHome = properties.getProperty("FLINK_HOME", sys.env.getOrElse("FLINK_HOME", ""))
- val flinkConfDir = properties.getProperty("FLINK_CONF_DIR", sys.env.getOrElse("FLINK_CONF_DIR", ""))
- val hadoopConfDir = properties.getProperty("HADOOP_CONF_DIR", sys.env.getOrElse("HADOOP_CONF_DIR", ""))
- val yarnConfDir = properties.getProperty("YARN_CONF_DIR", sys.env.getOrElse("YARN_CONF_DIR", ""))
- val hiveConfDir = properties.getProperty("HIVE_CONF_DIR", sys.env.getOrElse("HIVE_CONF_DIR", ""))
+ val flinkHome = sys.env.getOrElse("FLINK_HOME", "")
+ val flinkConfDir = sys.env.getOrElse("FLINK_CONF_DIR", "")
+ val hadoopConfDir = sys.env.getOrElse("HADOOP_CONF_DIR", "")
+ val yarnConfDir = sys.env.getOrElse("YARN_CONF_DIR", "")
+ val hiveConfDir = sys.env.getOrElse("HIVE_CONF_DIR", "")
LOGGER.info("FLINK_HOME: " + flinkHome)
LOGGER.info("FLINK_CONF_DIR: " + flinkConfDir)
LOGGER.info("HADOOP_CONF_DIR: " + hadoopConfDir)