You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by "Lee moon soo (JIRA)" <ji...@apache.org> on 2016/09/19 20:06:24 UTC
[jira] [Created] (ZEPPELIN-1460) Make spark configuration less
confusing and ambiguous, more intuitive
Lee moon soo created ZEPPELIN-1460:
--------------------------------------
Summary: Make spark configuration less confusing and ambiguous, more intuitive
Key: ZEPPELIN-1460
URL: https://issues.apache.org/jira/browse/ZEPPELIN-1460
Project: Zeppelin
Issue Type: Improvement
Affects Versions: 0.6.2
Reporter: Lee moon soo
Fix For: 0.7.0
Currently, Spark context in Spark interpreter takes properties from many different places.
- Interpreter setting page on Zeppelin GUI
- Environment variable in conf/zeppelin-env.sh
- Through --conf of SPARK_SUBMIT_OPTIONS in conf/zeppelin-env.sh
- SPARK_HOME/conf/spark-defaults.conf
There're some problems
a) It's not clearly documented which way overrides the other
b) Difficult to configure all properties of Spark Interpreter programmatically
c) Difficult to see effective configuration value which combines all configurations in 4 different places above.
I think a) and b) can be resolved by writing document that describes how configurations are applied and guide how to configure spark interpreter without accessing GUI.
c) Many users expect to show interpreter setting GUI shows all effective configuration values. not only the values that interpreter.json overrides. But it can not be done without changing interpreter architecture, i think.
We need some good ideas for c).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)