You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by Polina Marasanova <Po...@quantium.com.au> on 2016/09/08 03:46:29 UTC
RE: Config for spark interpreter
Hi Mina,
Thank you for your response.
I double checked approach 1 and 3, still no luck.
Probably the point is that I'm not using default Zeppelin binary zeppelin-0.6.0-bin-netinst.tgz because it doesn't fit our cluster architecture (we use MapR 5.1). I built it from source code for version 0.6.0 you provide here https://zeppelin.apache.org/download.html using parameters:
mvn clean package -Pspark-1.6 -Pmapr51 -Pyarn -DskipTests -DproxySet=true -DproxyHost=192.168.16.1 -DproxyPort=8080
And could you please clarify for me this point: "Here is one suspicious part you might missed, did you create new Spark Interpreter after restarting zeppelin?"
I dont create any new interpreters. I just want to use default one with my custom settings.
Regards,
Polina
________________________________________
From: mina lee [minalee@apache.org]
Sent: Friday, 12 August 2016 2:08 PM
To: users@zeppelin.apache.org
Subject: Re: Config for spark interpreter
Hi Polina,
I tried first/third approach with zeppelin-0.6.0-bin-netinst.tgz and both seems to work for me.
> 3. restart zeppelin, check interpreter tab.
Here is one suspicious part you might missed, did you create new Spark Interpreter after restarting zeppelin?
One thing you need to keep in mind is that properties that is defined in interpreter/spark/interpreter-setting.json will take priority over spark-default.conf.
To be more specific, you will need to change interpreter/spark/interpreter-setting.json file if you want to change one of below properties:
* spark.app.name<http://spark.app.name>
* spark.executor.memory
* spark.cores.max
* spark.master
The properties other than above, you can configure them in spark-default.conf.
Regards,
Mina
On Tue, Aug 9, 2016 at 8:12 AM Polina Marasanova <Po...@quantium.com.au>> wrote:
Hi,
I think I faced a bug of a Zeppelin 0.6.0.
What happened: I'm not able to overwrite spark interpreter properties from config, only via GUI.
What I've tried:
first approach, worked on previous version
1. export SPARK_CONF_DIR=/zeppelin/conf/spark
2. add my "spark-defaults.conf" file to this directory
3. restart zeppelin, check interpreter tab.
doesn't work
second approach:
1. add "spark-defaults.conf" to ZEPPELIN_CONF_DIR
2. pass this directory to starting script /bin/zeppelin.sh --config ZEPPELIN_CONF_DIR
doesn't work
third approach:
1. Create a file interpreter-setting.json with my configs similar to this one https://github.com/apache/zeppelin/blob/master/spark/src/main/resources/interpreter-setting.json
2. add to directory /zeppelin/interpreter/spark
3. restart zeppelin
according to logs looks like zeppelin actually fetching this file, but still no changes in GUI
if i add all properties via GUI it works ok, but it means that users should do it by themselves which is a bit of a pain
Have anyone seen something similar to my issue?
Thank you.
Regards,
Polina
________________________________________
From: ndjido@gmail.com<ma...@gmail.com> [ndjido@gmail.com<ma...@gmail.com>]
Sent: Monday, 8 August 2016 3:41 PM
To: users@zeppelin.apache.org<ma...@zeppelin.apache.org>
Subject: Re: Config for spark interpreter
Hi Polina,
You can just define the SPARK_HOME one the conf/zeppelin-Envoyez.sh and get rid of any other Spark configuration from that file, otherwise Zeppelin will just overwrite them. Once this is done, you can define the Spark default configurations in its config file living in conf/spark.default.conf.
Cheers,
Ndjido.
> On 08 Aug 2016, at 07:31, Polina Marasanova <Po...@quantium.com.au>> wrote:
>
> Hi everyone,
>
> I have a question: in previous versions of Zeppelin all settings for interpreters were stored in a file called "interpreter.json". It was very convenient to provide there some default spark settings such as spark.master, default driver memory and etc.
> What is the best way for version 0.6.0 to provide a bunch of defaults specific to a project?
>
> Thanks
> Cheers
> Polina Marasanova
RE: Config for spark interpreter
Posted by Polina Marasanova <Po...@quantium.com.au>.
Hi,
We found a way how to solve this issue. It's hacky but probably give you some ideas how to fix it.
Steps:
1. Unpack zeppelin-spark-0.6.0.jar inside zeppelin instance
2. Overwrite interpreter-setting.json
3. Repack jar
4. Restart Zeppelin
Regards,
Polina
________________________________________
From: Polina Marasanova [Polina.Marasanova@quantium.com.au]
Sent: Thursday, 8 September 2016 1:46 PM
To: users@zeppelin.apache.org
Subject: RE: Config for spark interpreter
Hi Mina,
Thank you for your response.
I double checked approach 1 and 3, still no luck.
Probably the point is that I'm not using default Zeppelin binary zeppelin-0.6.0-bin-netinst.tgz because it doesn't fit our cluster architecture (we use MapR 5.1). I built it from source code for version 0.6.0 you provide here https://zeppelin.apache.org/download.html using parameters:
mvn clean package -Pspark-1.6 -Pmapr51 -Pyarn -DskipTests -DproxySet=true -DproxyHost=192.168.16.1 -DproxyPort=8080
And could you please clarify for me this point: "Here is one suspicious part you might missed, did you create new Spark Interpreter after restarting zeppelin?"
I dont create any new interpreters. I just want to use default one with my custom settings.
Regards,
Polina
________________________________________
From: mina lee [minalee@apache.org]
Sent: Friday, 12 August 2016 2:08 PM
To: users@zeppelin.apache.org
Subject: Re: Config for spark interpreter
Hi Polina,
I tried first/third approach with zeppelin-0.6.0-bin-netinst.tgz and both seems to work for me.
> 3. restart zeppelin, check interpreter tab.
Here is one suspicious part you might missed, did you create new Spark Interpreter after restarting zeppelin?
One thing you need to keep in mind is that properties that is defined in interpreter/spark/interpreter-setting.json will take priority over spark-default.conf.
To be more specific, you will need to change interpreter/spark/interpreter-setting.json file if you want to change one of below properties:
* spark.app.name<http://spark.app.name>
* spark.executor.memory
* spark.cores.max
* spark.master
The properties other than above, you can configure them in spark-default.conf.
Regards,
Mina
On Tue, Aug 9, 2016 at 8:12 AM Polina Marasanova <Po...@quantium.com.au>> wrote:
Hi,
I think I faced a bug of a Zeppelin 0.6.0.
What happened: I'm not able to overwrite spark interpreter properties from config, only via GUI.
What I've tried:
first approach, worked on previous version
1. export SPARK_CONF_DIR=/zeppelin/conf/spark
2. add my "spark-defaults.conf" file to this directory
3. restart zeppelin, check interpreter tab.
doesn't work
second approach:
1. add "spark-defaults.conf" to ZEPPELIN_CONF_DIR
2. pass this directory to starting script /bin/zeppelin.sh --config ZEPPELIN_CONF_DIR
doesn't work
third approach:
1. Create a file interpreter-setting.json with my configs similar to this one https://github.com/apache/zeppelin/blob/master/spark/src/main/resources/interpreter-setting.json
2. add to directory /zeppelin/interpreter/spark
3. restart zeppelin
according to logs looks like zeppelin actually fetching this file, but still no changes in GUI
if i add all properties via GUI it works ok, but it means that users should do it by themselves which is a bit of a pain
Have anyone seen something similar to my issue?
Thank you.
Regards,
Polina
________________________________________
From: ndjido@gmail.com<ma...@gmail.com> [ndjido@gmail.com<ma...@gmail.com>]
Sent: Monday, 8 August 2016 3:41 PM
To: users@zeppelin.apache.org<ma...@zeppelin.apache.org>
Subject: Re: Config for spark interpreter
Hi Polina,
You can just define the SPARK_HOME one the conf/zeppelin-Envoyez.sh and get rid of any other Spark configuration from that file, otherwise Zeppelin will just overwrite them. Once this is done, you can define the Spark default configurations in its config file living in conf/spark.default.conf.
Cheers,
Ndjido.
> On 08 Aug 2016, at 07:31, Polina Marasanova <Po...@quantium.com.au>> wrote:
>
> Hi everyone,
>
> I have a question: in previous versions of Zeppelin all settings for interpreters were stored in a file called "interpreter.json". It was very convenient to provide there some default spark settings such as spark.master, default driver memory and etc.
> What is the best way for version 0.6.0 to provide a bunch of defaults specific to a project?
>
> Thanks
> Cheers
> Polina Marasanova