You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by yi...@apache.org on 2023/05/25 11:05:37 UTC
[spark-docker] branch master updated: [SPARK-43793] Fix SPARK_EXECUTOR_JAVA_OPTS assignment bug
This is an automated email from the ASF dual-hosted git repository.
yikun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-docker.git
The following commit(s) were added to refs/heads/master by this push:
new 006e8fa [SPARK-43793] Fix SPARK_EXECUTOR_JAVA_OPTS assignment bug
006e8fa is described below
commit 006e8fade69f148a05fc73f591f52c7678e48f04
Author: Yikun Jiang <yi...@gmail.com>
AuthorDate: Thu May 25 19:05:26 2023 +0800
[SPARK-43793] Fix SPARK_EXECUTOR_JAVA_OPTS assignment bug
### What changes were proposed in this pull request?
In previous code, this is susceptible to a few bugs particularly around newlines in values.
```
env | grep SPARK_JAVA_OPT_ | sort -t_ -k4 -n | sed 's/[^=]*=\(.*\)/\1/g' > /tmp/java_opts.txt
readarray -t SPARK_EXECUTOR_JAVA_OPTS < /tmp/java_opts.txt
```
### Why are the changes needed?
To address DOI comments: https://github.com/docker-library/official-images/pull/13089#issuecomment-1533540388 , question 6.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
1. Test mannully
```
export SPARK_JAVA_OPT_0="foo=bar"
export SPARK_JAVA_OPT_1="foo1=bar1"
for v in "${!SPARK_JAVA_OPT_}"; do
SPARK_EXECUTOR_JAVA_OPTS+=( "${!v}" )
done
for v in ${SPARK_EXECUTOR_JAVA_OPTS[]}; do
echo $v
done
# foo=bar
# foo1=bar1
```
2. CI passed
Closes #42 from Yikun/SPARK-43793.
Authored-by: Yikun Jiang <yi...@gmail.com>
Signed-off-by: Yikun Jiang <yi...@gmail.com>
---
3.4.0/scala2.12-java11-ubuntu/entrypoint.sh | 5 +++--
entrypoint.sh.template | 5 +++--
2 files changed, 6 insertions(+), 4 deletions(-)
diff --git a/3.4.0/scala2.12-java11-ubuntu/entrypoint.sh b/3.4.0/scala2.12-java11-ubuntu/entrypoint.sh
index 4bb1557..716f1af 100755
--- a/3.4.0/scala2.12-java11-ubuntu/entrypoint.sh
+++ b/3.4.0/scala2.12-java11-ubuntu/entrypoint.sh
@@ -38,8 +38,9 @@ if [ -z "$JAVA_HOME" ]; then
fi
SPARK_CLASSPATH="$SPARK_CLASSPATH:${SPARK_HOME}/jars/*"
-env | grep SPARK_JAVA_OPT_ | sort -t_ -k4 -n | sed 's/[^=]*=\(.*\)/\1/g' > /tmp/java_opts.txt
-readarray -t SPARK_EXECUTOR_JAVA_OPTS < /tmp/java_opts.txt
+for v in "${!SPARK_JAVA_OPT_@}"; do
+ SPARK_EXECUTOR_JAVA_OPTS+=( "${!v}" )
+done
if [ -n "$SPARK_EXTRA_CLASSPATH" ]; then
SPARK_CLASSPATH="$SPARK_CLASSPATH:$SPARK_EXTRA_CLASSPATH"
diff --git a/entrypoint.sh.template b/entrypoint.sh.template
index 4bb1557..716f1af 100644
--- a/entrypoint.sh.template
+++ b/entrypoint.sh.template
@@ -38,8 +38,9 @@ if [ -z "$JAVA_HOME" ]; then
fi
SPARK_CLASSPATH="$SPARK_CLASSPATH:${SPARK_HOME}/jars/*"
-env | grep SPARK_JAVA_OPT_ | sort -t_ -k4 -n | sed 's/[^=]*=\(.*\)/\1/g' > /tmp/java_opts.txt
-readarray -t SPARK_EXECUTOR_JAVA_OPTS < /tmp/java_opts.txt
+for v in "${!SPARK_JAVA_OPT_@}"; do
+ SPARK_EXECUTOR_JAVA_OPTS+=( "${!v}" )
+done
if [ -n "$SPARK_EXTRA_CLASSPATH" ]; then
SPARK_CLASSPATH="$SPARK_CLASSPATH:$SPARK_EXTRA_CLASSPATH"
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org