You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kyuubi.apache.org by ul...@apache.org on 2021/08/27 08:37:27 UTC
[incubator-kyuubi] branch master updated: [KYUUBI #991][FOLLOWUP]
Improve error msg and add SPARK_HOME in Examples
This is an automated email from the ASF dual-hosted git repository.
ulyssesyou pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-kyuubi.git
The following commit(s) were added to refs/heads/master by this push:
new cd42679 [KYUUBI #991][FOLLOWUP] Improve error msg and add SPARK_HOME in Examples
cd42679 is described below
commit cd426798bf5c97efc75571a618960fce93f6af2f
Author: Fu Chen <cf...@gmail.com>
AuthorDate: Fri Aug 27 16:37:16 2021 +0800
[KYUUBI #991][FOLLOWUP] Improve error msg and add SPARK_HOME in Examples
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
According [comments](https://github.com/apache/incubator-kyuubi/pull/991#issuecomment-906914618)
1. Improve error message.
2. add SPARK_HOME in Examples.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes #993 from cfmcgrady/ky-991-followup.
Closes #991
e7c25be6 [Fu Chen] update docs
945eeb73 [Fu Chen] review
98555196 [Fu Chen] improve error msg and add SPARK_HOME in Examples
Authored-by: Fu Chen <cf...@gmail.com>
Signed-off-by: ulysses-you <ul...@gmail.com>
---
conf/kyuubi-env.sh.template | 1 +
docs/deployment/settings.md | 1 +
.../scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala | 4 +++-
3 files changed, 5 insertions(+), 1 deletion(-)
diff --git a/conf/kyuubi-env.sh.template b/conf/kyuubi-env.sh.template
index 90fe102..05907da 100755
--- a/conf/kyuubi-env.sh.template
+++ b/conf/kyuubi-env.sh.template
@@ -44,5 +44,6 @@
## Examples ##
# export JAVA_HOME=/usr/jdk64/jdk1.8.0_152
+# export SPARK_HOME=/opt/spark
# export HADOOP_CONF_DIR=/usr/ndp/current/mapreduce_client/conf
# export KYUUBI_JAVA_OPTS="-Xmx10g -XX:+UnlockDiagnosticVMOptions -XX:ParGCCardsPerStrideChunk=4096 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:+UseCondCardMark -XX:MaxDirectMemorySize=1024m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=./logs -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribut [...]
diff --git a/docs/deployment/settings.md b/docs/deployment/settings.md
index 35b8a28..934a151 100644
--- a/docs/deployment/settings.md
+++ b/docs/deployment/settings.md
@@ -79,6 +79,7 @@ You can configure the environment variables in `$KYUUBI_HOME/conf/kyuubi-env.sh`
## Examples ##
# export JAVA_HOME=/usr/jdk64/jdk1.8.0_152
+# export SPARK_HOME=/opt/spark
# export HADOOP_CONF_DIR=/usr/ndp/current/mapreduce_client/conf
# export KYUUBI_JAVA_OPTS="-Xmx10g -XX:+UnlockDiagnosticVMOptions -XX:ParGCCardsPerStrideChunk=4096 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:+UseCondCardMark -XX:MaxDirectMemorySize=1024m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=./logs -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribut [...]
```
diff --git a/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala b/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala
index 47077fb..4d4a454 100644
--- a/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala
+++ b/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala
@@ -58,7 +58,9 @@ class SparkProcessBuilder(
sparkHomeOpt.map{ dir =>
Paths.get(dir, "bin", SPARK_SUBMIT_FILE).toAbsolutePath.toFile.getCanonicalPath
}.getOrElse {
- throw KyuubiSQLException("SPARK_HOME is not set!")
+ throw KyuubiSQLException("SPARK_HOME is not set! " +
+ "For more detail information on installing and configuring Spark, please visit " +
+ "https://kyuubi.apache.org/docs/stable/deployment/settings.html#environments")
}
}