You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by GitBox <gi...@apache.org> on 2022/09/05 07:23:51 UTC
[GitHub] [zeppelin] zlosim commented on a diff in pull request #4398: [ZEPPELIN-5760] fix passing configs to spark in k8s
zlosim commented on code in PR #4398:
URL: https://github.com/apache/zeppelin/pull/4398#discussion_r962569551
##########
zeppelin-plugins/launcher/k8s-standard/src/main/java/org/apache/zeppelin/interpreter/launcher/K8sRemoteInterpreterProcess.java:
##########
@@ -403,28 +420,35 @@ boolean isSparkOnKubernetes(Properties interpreterProperties) {
}
@VisibleForTesting
- String buildSparkSubmitOptions(String userName) {
- StringBuilder options = new StringBuilder();
-
- options.append(" --master k8s://https://kubernetes.default.svc");
- options.append(" --deploy-mode client");
- if (properties.containsKey(SPARK_DRIVER_MEMORY)) {
- options.append(" --driver-memory ").append(properties.get(SPARK_DRIVER_MEMORY));
- }
+ String prepareZeppelinSparkConf(String userName) {
+ StringJoiner sparkConfSJ = new StringJoiner("|");
if (isUserImpersonated() && !StringUtils.containsIgnoreCase(userName, "anonymous")) {
- options.append(" --proxy-user ").append(userName);
+ sparkConfSJ.add("--proxy-user");
+ sparkConfSJ.add(userName);
Review Comment:
Hi @Reamer I`m not sure I can see the bug here, right now the output of this snippet is `--proxy-user|user` and the pipe would be replaced in interpreter.sh with space
as far as i can see same code snippet can be found in [SparkInterpreterLauncher](https://github.com/apache/zeppelin/blob/c4c580a37fde649553d336984a94bcb1b2821201/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/launcher/SparkInterpreterLauncher.java#L196)
##########
zeppelin-plugins/launcher/k8s-standard/src/main/java/org/apache/zeppelin/interpreter/launcher/K8sRemoteInterpreterProcess.java:
##########
@@ -403,28 +420,35 @@ boolean isSparkOnKubernetes(Properties interpreterProperties) {
}
@VisibleForTesting
- String buildSparkSubmitOptions(String userName) {
- StringBuilder options = new StringBuilder();
-
- options.append(" --master k8s://https://kubernetes.default.svc");
- options.append(" --deploy-mode client");
- if (properties.containsKey(SPARK_DRIVER_MEMORY)) {
- options.append(" --driver-memory ").append(properties.get(SPARK_DRIVER_MEMORY));
- }
+ String prepareZeppelinSparkConf(String userName) {
+ StringJoiner sparkConfSJ = new StringJoiner("|");
if (isUserImpersonated() && !StringUtils.containsIgnoreCase(userName, "anonymous")) {
- options.append(" --proxy-user ").append(userName);
+ sparkConfSJ.add("--proxy-user");
+ sparkConfSJ.add(userName);
Review Comment:
Hi @Reamer I'm not sure I can see the bug here, right now the output of this snippet is `--proxy-user|user` and the pipe would be replaced in interpreter.sh with space
as far as i can see same code snippet can be found in [SparkInterpreterLauncher](https://github.com/apache/zeppelin/blob/c4c580a37fde649553d336984a94bcb1b2821201/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/launcher/SparkInterpreterLauncher.java#L196)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org