You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by lihu <li...@gmail.com> on 2014/11/26 04:50:52 UTC
do not assemble the spark example jar
Hi,
The spark assembly is time costly. If I only need
the spark-assembly-1.1.0-hadoop2.3.0.jar, do not need
the spark-examples-1.1.0-hadoop2.3.0.jar. How to configure the spark to
avoid assemble the example jar. I know *export
SPARK_PREPEND_CLASSES=**true* method
can reduce the assembly, but I do not
develop locally. Any advice?
--
*Best Wishes!*
Re: do not assemble the spark example jar
Posted by lihu <li...@gmail.com>.
Can this assembly get faster if we do not need the Spark SQL or some other
component in spark ? such as we only need the core of spark.
On Wed, Nov 26, 2014 at 3:37 PM, lihu <li...@gmail.com> wrote:
> Matei, sorry for my last typo error. And the tip can improve about 30s in
> my computer.
>
> On Wed, Nov 26, 2014 at 3:34 PM, lihu <li...@gmail.com> wrote:
>
>> Mater, thank you very much!
>> After take your advice, the time for assembly from about 20min down to
>> 6min in my computer. that's a very big improvement.
>>
>> On Wed, Nov 26, 2014 at 12:32 PM, Matei Zaharia <ma...@gmail.com>
>> wrote:
>>
>>> BTW as another tip, it helps to keep the SBT console open as you make
>>> source changes (by just running sbt/sbt with no args). It's a lot faster
>>> the second time it builds something.
>>>
>>> Matei
>>>
>>> On Nov 25, 2014, at 8:31 PM, Matei Zaharia <ma...@gmail.com>
>>> wrote:
>>>
>>> You can do sbt/sbt assembly/assembly to assemble only the main package.
>>>
>>> Matei
>>>
>>> On Nov 25, 2014, at 7:50 PM, lihu <li...@gmail.com> wrote:
>>>
>>> Hi,
>>> The spark assembly is time costly. If I only need
>>> the spark-assembly-1.1.0-hadoop2.3.0.jar, do not need
>>> the spark-examples-1.1.0-hadoop2.3.0.jar. How to configure the spark to
>>> avoid assemble the example jar. I know *export SPARK_PREPEND_CLASSES=*
>>> *true* method can reduce the assembly, but I do not
>>> develop locally. Any advice?
>>>
>>> --
>>> *Best Wishes!*
>>>
>>>
>>>
>>>
>>>
>>
>>
>> --
>> *Best Wishes!*
>>
>> *Li Hu(李浒) | Graduate Student*
>>
>> *Institute for Interdisciplinary Information Sciences(IIIS
>> <http://iiis.tsinghua.edu.cn/>)*
>> *Tsinghua University, China*
>>
>> *Email: lihu723@gmail.com <li...@gmail.com>*
>> *Homepage: http://iiis.tsinghua.edu.cn/zh/lihu/
>> <http://iiis.tsinghua.edu.cn/zh/lihu/>*
>>
>>
>>
>
>
> --
> *Best Wishes!*
>
> *Li Hu(李浒) | Graduate Student*
>
> *Institute for Interdisciplinary Information Sciences(IIIS
> <http://iiis.tsinghua.edu.cn/>)*
> *Tsinghua University, China*
>
> *Email: lihu723@gmail.com <li...@gmail.com>*
> *Homepage: http://iiis.tsinghua.edu.cn/zh/lihu/
> <http://iiis.tsinghua.edu.cn/zh/lihu/>*
>
>
>
--
*Best Wishes!*
*Li Hu(李浒) | Graduate Student*
*Institute for Interdisciplinary Information Sciences(IIIS
<http://iiis.tsinghua.edu.cn/>)*
*Tsinghua University, China*
*Email: lihu723@gmail.com <li...@gmail.com>*
*Homepage: http://iiis.tsinghua.edu.cn/zh/lihu/
<http://iiis.tsinghua.edu.cn/zh/lihu/>*
Re: do not assemble the spark example jar
Posted by lihu <li...@gmail.com>.
Matei, sorry for my last typo error. And the tip can improve about 30s in
my computer.
On Wed, Nov 26, 2014 at 3:34 PM, lihu <li...@gmail.com> wrote:
> Mater, thank you very much!
> After take your advice, the time for assembly from about 20min down to
> 6min in my computer. that's a very big improvement.
>
> On Wed, Nov 26, 2014 at 12:32 PM, Matei Zaharia <ma...@gmail.com>
> wrote:
>
>> BTW as another tip, it helps to keep the SBT console open as you make
>> source changes (by just running sbt/sbt with no args). It's a lot faster
>> the second time it builds something.
>>
>> Matei
>>
>> On Nov 25, 2014, at 8:31 PM, Matei Zaharia <ma...@gmail.com>
>> wrote:
>>
>> You can do sbt/sbt assembly/assembly to assemble only the main package.
>>
>> Matei
>>
>> On Nov 25, 2014, at 7:50 PM, lihu <li...@gmail.com> wrote:
>>
>> Hi,
>> The spark assembly is time costly. If I only need
>> the spark-assembly-1.1.0-hadoop2.3.0.jar, do not need
>> the spark-examples-1.1.0-hadoop2.3.0.jar. How to configure the spark to
>> avoid assemble the example jar. I know *export SPARK_PREPEND_CLASSES=*
>> *true* method can reduce the assembly, but I do not
>> develop locally. Any advice?
>>
>> --
>> *Best Wishes!*
>>
>>
>>
>>
>>
>
>
> --
> *Best Wishes!*
>
> *Li Hu(李浒) | Graduate Student*
>
> *Institute for Interdisciplinary Information Sciences(IIIS
> <http://iiis.tsinghua.edu.cn/>)*
> *Tsinghua University, China*
>
> *Email: lihu723@gmail.com <li...@gmail.com>*
> *Homepage: http://iiis.tsinghua.edu.cn/zh/lihu/
> <http://iiis.tsinghua.edu.cn/zh/lihu/>*
>
>
>
--
*Best Wishes!*
*Li Hu(李浒) | Graduate Student*
*Institute for Interdisciplinary Information Sciences(IIIS
<http://iiis.tsinghua.edu.cn/>)*
*Tsinghua University, China*
*Email: lihu723@gmail.com <li...@gmail.com>*
*Homepage: http://iiis.tsinghua.edu.cn/zh/lihu/
<http://iiis.tsinghua.edu.cn/zh/lihu/>*
Re: do not assemble the spark example jar
Posted by lihu <li...@gmail.com>.
Mater, thank you very much!
After take your advice, the time for assembly from about 20min down to 6min
in my computer. that's a very big improvement.
On Wed, Nov 26, 2014 at 12:32 PM, Matei Zaharia <ma...@gmail.com>
wrote:
> BTW as another tip, it helps to keep the SBT console open as you make
> source changes (by just running sbt/sbt with no args). It's a lot faster
> the second time it builds something.
>
> Matei
>
> On Nov 25, 2014, at 8:31 PM, Matei Zaharia <ma...@gmail.com>
> wrote:
>
> You can do sbt/sbt assembly/assembly to assemble only the main package.
>
> Matei
>
> On Nov 25, 2014, at 7:50 PM, lihu <li...@gmail.com> wrote:
>
> Hi,
> The spark assembly is time costly. If I only need
> the spark-assembly-1.1.0-hadoop2.3.0.jar, do not need
> the spark-examples-1.1.0-hadoop2.3.0.jar. How to configure the spark to
> avoid assemble the example jar. I know *export SPARK_PREPEND_CLASSES=*
> *true* method can reduce the assembly, but I do not
> develop locally. Any advice?
>
> --
> *Best Wishes!*
>
>
>
>
>
--
*Best Wishes!*
*Li Hu(李浒) | Graduate Student*
*Institute for Interdisciplinary Information Sciences(IIIS
<http://iiis.tsinghua.edu.cn/>)*
*Tsinghua University, China*
*Email: lihu723@gmail.com <li...@gmail.com>*
*Homepage: http://iiis.tsinghua.edu.cn/zh/lihu/
<http://iiis.tsinghua.edu.cn/zh/lihu/>*
Re: do not assemble the spark example jar
Posted by Matei Zaharia <ma...@gmail.com>.
BTW as another tip, it helps to keep the SBT console open as you make source changes (by just running sbt/sbt with no args). It's a lot faster the second time it builds something.
Matei
> On Nov 25, 2014, at 8:31 PM, Matei Zaharia <ma...@gmail.com> wrote:
>
> You can do sbt/sbt assembly/assembly to assemble only the main package.
>
> Matei
>
>> On Nov 25, 2014, at 7:50 PM, lihu <lihu723@gmail.com <ma...@gmail.com>> wrote:
>>
>> Hi,
>> The spark assembly is time costly. If I only need the spark-assembly-1.1.0-hadoop2.3.0.jar, do not need the spark-examples-1.1.0-hadoop2.3.0.jar. How to configure the spark to avoid assemble the example jar. I know export SPARK_PREPEND_CLASSES=true method can reduce the assembly, but I do not
>> develop locally. Any advice?
>>
>> --
>> Best Wishes!
>>
>>
>
Re: do not assemble the spark example jar
Posted by Matei Zaharia <ma...@gmail.com>.
You can do sbt/sbt assembly/assembly to assemble only the main package.
Matei
> On Nov 25, 2014, at 7:50 PM, lihu <li...@gmail.com> wrote:
>
> Hi,
> The spark assembly is time costly. If I only need the spark-assembly-1.1.0-hadoop2.3.0.jar, do not need the spark-examples-1.1.0-hadoop2.3.0.jar. How to configure the spark to avoid assemble the example jar. I know export SPARK_PREPEND_CLASSES=true method can reduce the assembly, but I do not
> develop locally. Any advice?
>
> --
> Best Wishes!
>
>