You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by Jeff Steinmetz <je...@gmail.com> on 2015/12/23 19:13:31 UTC

zeppelin build - downloading spark distribution

I was curious if this build step has been part of the Zeppelin build thus far, is it required, and curious if this step is new?

  Downloading: http://archive.apache.org/dist/spark/spark-1.5.2/spark-1.5.2.tgz

  Expanding: /spark-dependencies/target/spark-dist/spark-1.5.2.tgz into /incubator-zeppelin/spark-dependencies/target/spark-dist



I ask because it slows down the initial build a bit.  I recall in the past Zeppelin pulled in spark as a dependency vs the entire distribution?  ( I may be wrong)
Mainly curious at this point.

Best
Jeff Steinmetz

Re: zeppelin build - downloading spark distribution

Posted by Jeff Steinmetz <je...@gmail.com>.
Ah yes - Now I recall the discussion related to ensuring -Ppyspark will work.
Thank you for the reminder.

J




On 12/23/15, 5:34 PM, "moon soo Lee" <mo...@apache.org> wrote:

>Hi Jeff Steinmetz,
>
>-Ppyspark flag downloads spark distribution. See
>https://github.com/apache/incubator-zeppelin/blob/master/spark-dependencies/pom.xml#L733
>.
>
>It copies python related files (pyspark python module, py4j) from spark
>distribution to interpreter/spark/pyspark for supporting %pyspark in local
>mode (without SPARK_HOME).
>
>Hope this helps.
>
>Thanks,
>moon
>
>On Thu, Dec 24, 2015 at 3:13 AM Jeff Steinmetz <je...@gmail.com>
>wrote:
>
>> I was curious if this build step has been part of the Zeppelin build thus
>> far, is it required, and curious if this step is new?
>>
>>   Downloading:
>> http://archive.apache.org/dist/spark/spark-1.5.2/spark-1.5.2.tgz
>>
>>   Expanding: /spark-dependencies/target/spark-dist/spark-1.5.2.tgz into
>> /incubator-zeppelin/spark-dependencies/target/spark-dist
>>
>>
>>
>> I ask because it slows down the initial build a bit.  I recall in the past
>> Zeppelin pulled in spark as a dependency vs the entire distribution?  ( I
>> may be wrong)
>> Mainly curious at this point.
>>
>> Best
>> Jeff Steinmetz
>>
>>
>>
>>
>>
>>

Re: zeppelin build - downloading spark distribution

Posted by moon soo Lee <mo...@apache.org>.
Hi Jeff Steinmetz,

-Ppyspark flag downloads spark distribution. See
https://github.com/apache/incubator-zeppelin/blob/master/spark-dependencies/pom.xml#L733
.

It copies python related files (pyspark python module, py4j) from spark
distribution to interpreter/spark/pyspark for supporting %pyspark in local
mode (without SPARK_HOME).

Hope this helps.

Thanks,
moon

On Thu, Dec 24, 2015 at 3:13 AM Jeff Steinmetz <je...@gmail.com>
wrote:

> I was curious if this build step has been part of the Zeppelin build thus
> far, is it required, and curious if this step is new?
>
>   Downloading:
> http://archive.apache.org/dist/spark/spark-1.5.2/spark-1.5.2.tgz
>
>   Expanding: /spark-dependencies/target/spark-dist/spark-1.5.2.tgz into
> /incubator-zeppelin/spark-dependencies/target/spark-dist
>
>
>
> I ask because it slows down the initial build a bit.  I recall in the past
> Zeppelin pulled in spark as a dependency vs the entire distribution?  ( I
> may be wrong)
> Mainly curious at this point.
>
> Best
> Jeff Steinmetz
>
>
>
>
>
>