You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by "Meier, Alexander" <Al...@t-systems-dmc.com> on 2017/10/18 09:14:30 UTC
Multiple Spark Versions / Interpreter on CDH 5.9 Cluster
Hi there
We have a CDH 5.9 Cluster where zeppelin 0.7.1is running fine with spark 1.6, impala, hive, hbase and apache phoenix. As we have both, Spark 1.6 and Spark 2 distributed and activated on the cluster, we now also want to use a spark2 interpreter in zeppelin.
I created a new interpreter %spark2 in zeppelin belonging to the spark interpreter group.
Also I added an interpreter property “SPARK_HOME” with value /opt/cloudera/parcels/SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904/lib/spark2
(I have also tried to set SPARK_HOME to /opt/cloudera/parcels/SPARK2/lib/spark2 with the same result.)
When testing the interpreter:
%spark2
sc.version
spark.version
we get
res0: String = 1.6.0
<console>:28: error: not found: value spark
spark.version
Why is the interpreter still using Spark 1.6? I thought the interpreter property would override the SPARK_HOME setting in zeppelin-env.sh
Any ideas?
btw: we also use impersonation and run the interpreter instantiated per user in isolated mode, but that as "just in case” info.
Best regards
Alex
Re: Multiple Spark Versions / Interpreter on CDH 5.9 Cluster
Posted by Jongyoul Lee <jo...@gmail.com>.
Hi,
You can set SPARK_HOME in your interpreter tab instead of setting in
zeppelin-env.sh. Remove SPARK_HOME in your zeppelin-env.sh and set it up
interpreter tabs with two different spark interpreters.
On Tue, Oct 24, 2017 at 11:39 AM, Jeff Zhang <zj...@gmail.com> wrote:
>
> Take a look at spark submit command
>
> Refer this link for details
> https://community.hortonworks.com/articles/70658/how-to-
> diagnose-zeppelin.html
>
>
>
> Meier, Alexander <Al...@t-systems-dmc.com>于2017年10月24日周二
> 上午2:54写道：
>
>> Hi there
>>
>> Any ideas why the new interpreter is not using the specified SPARK_HOME?
>>
>> Best regards
>> Alex Meier
>>
>>
>> On 18 Oct 2017, at 11:14, Meier, Alexander <Alexander.Meier@t-systems-
>> dmc.com> wrote:
>>
>> Hi there
>>
>> We have a CDH 5.9 Cluster where zeppelin 0.7.1is running fine with spark
>> 1.6, impala, hive, hbase and apache phoenix. As we have both, Spark 1.6 and
>> Spark 2 distributed and activated on the cluster, we now also want to use a
>> spark2 interpreter in zeppelin.
>>
>> I created a new interpreter %spark2 in zeppelin belonging to the spark
>> interpreter group.
>> Also I added an interpreter property “SPARK_HOME” with
>> value /opt/cloudera/parcels/SPARK2-2.1.0.cloudera1-1.cdh5.
>> 7.0.p0.120904/lib/spark2
>> (I have also tried to set SPARK_HOME to /opt/cloudera/parcels/SPARK2/lib/spark2
>> with the same result.)
>>
>> When testing the interpreter:
>>
>> %spark2
>> sc.version
>> spark.version
>>
>> we get
>>
>> res0: String = 1.6.0
>> <console>:28: error: not found: value spark
>> spark.version
>>
>>
>> Why is the interpreter still using Spark 1.6? I thought the interpreter
>> property would override the SPARK_HOME setting in zeppelin-env.sh
>>
>> Any ideas?
>>
>> btw: we also use impersonation and run the interpreter instantiated per
>> user in isolated mode, but that as "just in case” info.
>>
>> Best regards
>> Alex
>>
>>
>>
>>
--
이종열, Jongyoul Lee, 李宗烈
http://madeng.net
Re: Multiple Spark Versions / Interpreter on CDH 5.9 Cluster
Posted by Jeff Zhang <zj...@gmail.com>.
Take a look at spark submit command
Refer this link for details
https://community.hortonworks.com/articles/70658/how-to-diagnose-zeppelin.html
Meier, Alexander <Al...@t-systems-dmc.com>于2017年10月24日周二 上午2:54写道：
> Hi there
>
> Any ideas why the new interpreter is not using the specified SPARK_HOME?
>
> Best regards
> Alex Meier
>
>
> On 18 Oct 2017, at 11:14, Meier, Alexander <
> Alexander.Meier@t-systems-dmc.com> wrote:
>
> Hi there
>
> We have a CDH 5.9 Cluster where zeppelin 0.7.1is running fine with spark
> 1.6, impala, hive, hbase and apache phoenix. As we have both, Spark 1.6 and
> Spark 2 distributed and activated on the cluster, we now also want to use a
> spark2 interpreter in zeppelin.
>
> I created a new interpreter %spark2 in zeppelin belonging to the spark
> interpreter group.
> Also I added an interpreter property “SPARK_HOME” with
> value /opt/cloudera/parcels/SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904/lib/spark2
> (I have also tried to set SPARK_HOME
> to /opt/cloudera/parcels/SPARK2/lib/spark2 with the same result.)
>
> When testing the interpreter:
>
> %spark2
> sc.version
> spark.version
>
> we get
>
> res0: String = 1.6.0
> <console>:28: error: not found: value spark
> spark.version
>
>
> Why is the interpreter still using Spark 1.6? I thought the interpreter
> property would override the SPARK_HOME setting in zeppelin-env.sh
>
> Any ideas?
>
> btw: we also use impersonation and run the interpreter instantiated per
> user in isolated mode, but that as "just in case” info.
>
> Best regards
> Alex
>
>
>
>
Re: Multiple Spark Versions / Interpreter on CDH 5.9 Cluster
Posted by "Meier, Alexander" <Al...@t-systems-dmc.com>.
Hi there
Any ideas why the new interpreter is not using the specified SPARK_HOME?
Best regards
Alex Meier
On 18 Oct 2017, at 11:14, Meier, Alexander <Al...@t-systems-dmc.com>> wrote:
Hi there
We have a CDH 5.9 Cluster where zeppelin 0.7.1is running fine with spark 1.6, impala, hive, hbase and apache phoenix. As we have both, Spark 1.6 and Spark 2 distributed and activated on the cluster, we now also want to use a spark2 interpreter in zeppelin.
I created a new interpreter %spark2 in zeppelin belonging to the spark interpreter group.
Also I added an interpreter property “SPARK_HOME” with value /opt/cloudera/parcels/SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904/lib/spark2
(I have also tried to set SPARK_HOME to /opt/cloudera/parcels/SPARK2/lib/spark2 with the same result.)
When testing the interpreter:
%spark2
sc.version
spark.version
we get
res0: String = 1.6.0
<console>:28: error: not found: value spark
spark.version
Why is the interpreter still using Spark 1.6? I thought the interpreter property would override the SPARK_HOME setting in zeppelin-env.sh
Any ideas?
btw: we also use impersonation and run the interpreter instantiated per user in isolated mode, but that as "just in case” info.
Best regards
Alex