You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by Michaël Van de Borne <mi...@gmail.com> on 2018/01/23 10:42:32 UTC
Supporting several spark versions simltaneously
Hi list,
I'd like my notebooks to support both spark 1.6 & spark 2.
I managed to get both versions working fine, with the SPARK_HOME variable
in zeppelin-env.sh. But just one at a time.
So I need to change the variable and restart zeppelin when I want to swap
spark versions.
Is it possible to somehow map a spark distribution to an interpreter group
so that I can get both versions supported at the same time?
Thank you,
cheers,
m.
Re: Supporting several spark versions simltaneously
Posted by Jeff Zhang <zj...@gmail.com>.
Zeppelin doesn't need to take care of py4j, it is spark's responsibility.
As long as you set SPARK_HOME, spark will find py4j properly.
Michaël Van de Borne <mi...@gmail.com>于2018年1月23日周二 下午10:37写道：
> As long as SPARK_HOME is concerned, this might work.
> However, other variables, such as PYTHONPATH and SPARK_YARN_USER_ENV
> should be defined in zeppelin-env.sh as well.
> Without those variables, zeppelin cannot access, for instance, py4j or
> pyspark.
> And guess where py4j sits? in SPARK_HOME/python/lib/
> So it's a chicken-and-egg problem: if I define the SPARK_HOME in
> zeppelin-env.sh, I cannot run both spark version. And if I don't, then
> other paths are not defined, and zeppelin doesn't know where to look for
> py4j and pyspark.
>
> 2018-01-23 12:45 GMT+01:00 Jeff Zhang <zj...@gmail.com>:
>
>>
>> Just remove SPARK_HOME in spark-env.sh, and instead define them in
>> spark’s interpreter setting. You can create 2 spark interpreter, one for
>> spark 1.6, another for spark 2. Only difference between them is the
>> SPARK_HOME you defined in interpreter setting.
>>
>>
>>
>> Michaël Van de Borne <mi...@gmail.com>于2018年1月23日周二
>> 下午6:42写道：
>>
>>> Hi list,
>>>
>>> I'd like my notebooks to support both spark 1.6 & spark 2.
>>> I managed to get both versions working fine, with the SPARK_HOME
>>> variable in zeppelin-env.sh. But just one at a time.
>>> So I need to change the variable and restart zeppelin when I want to
>>> swap spark versions.
>>> Is it possible to somehow map a spark distribution to an interpreter
>>> group so that I can get both versions supported at the same time?
>>>
>>>
>>> Thank you,
>>>
>>> cheers,
>>>
>>> m.
>>>
>>
>
Re: Supporting several spark versions simltaneously
Posted by Michaël Van de Borne <mi...@gmail.com>.
As long as SPARK_HOME is concerned, this might work.
However, other variables, such as PYTHONPATH and SPARK_YARN_USER_ENV should
be defined in zeppelin-env.sh as well.
Without those variables, zeppelin cannot access, for instance, py4j or
pyspark.
And guess where py4j sits? in SPARK_HOME/python/lib/
So it's a chicken-and-egg problem: if I define the SPARK_HOME in
zeppelin-env.sh, I cannot run both spark version. And if I don't, then
other paths are not defined, and zeppelin doesn't know where to look for
py4j and pyspark.
2018-01-23 12:45 GMT+01:00 Jeff Zhang <zj...@gmail.com>:
>
> Just remove SPARK_HOME in spark-env.sh, and instead define them in spark’s
> interpreter setting. You can create 2 spark interpreter, one for spark 1.6,
> another for spark 2. Only difference between them is the SPARK_HOME you
> defined in interpreter setting.
>
>
>
> Michaël Van de Borne <mi...@gmail.com>于2018年1月23日周二 下午6:42写道：
>
>> Hi list,
>>
>> I'd like my notebooks to support both spark 1.6 & spark 2.
>> I managed to get both versions working fine, with the SPARK_HOME variable
>> in zeppelin-env.sh. But just one at a time.
>> So I need to change the variable and restart zeppelin when I want to swap
>> spark versions.
>> Is it possible to somehow map a spark distribution to an interpreter
>> group so that I can get both versions supported at the same time?
>>
>>
>> Thank you,
>>
>> cheers,
>>
>> m.
>>
>
Re: Supporting several spark versions simltaneously
Posted by Jeff Zhang <zj...@gmail.com>.
Just remove SPARK_HOME in spark-env.sh, and instead define them in spark’s
interpreter setting. You can create 2 spark interpreter, one for spark 1.6,
another for spark 2. Only difference between them is the SPARK_HOME you
defined in interpreter setting.
Michaël Van de Borne <mi...@gmail.com>于2018年1月23日周二 下午6:42写道：
> Hi list,
>
> I'd like my notebooks to support both spark 1.6 & spark 2.
> I managed to get both versions working fine, with the SPARK_HOME variable
> in zeppelin-env.sh. But just one at a time.
> So I need to change the variable and restart zeppelin when I want to swap
> spark versions.
> Is it possible to somehow map a spark distribution to an interpreter group
> so that I can get both versions supported at the same time?
>
>
> Thank you,
>
> cheers,
>
> m.
>