You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by dachuan <hd...@gmail.com> on 2014/01/22 20:15:23 UTC
why is it so slow to run sbt/sbt assembly in my machine?
Hi, community,
I have run sbt/sbt assembly, but it is stuck at the following place for 30
minutes.
Does anybody know what's the root cause? my machine's memory size is 4GB.
thanks,
dachuan.
[info] Packaging
C:\cygwin\home\dachuan\incubator-spark\assembly\target\scala-2.10\spark-assembly-0.9.0-incubating-SNAPSHOT-hadoop1.0.4.jar
...
[info] Packaging
C:\cygwin\home\dachuan\incubator-spark\examples\target\scala-2.10\spark-examples-assembly-0.9.0-incubating-SNAPSHOT.jar
...
(STUCK HERE)
--
Dachuan Huang
Cellphone: 614-390-7234
2015 Neil Avenue
Ohio State University
Columbus, Ohio
U.S.A.
43210
Re: why is it so slow to run sbt/sbt assembly in my machine?
Posted by Mark Hamstra <ma...@clearstorydata.com>.
In case that wasn't clear, what I am saying is that starting sbt with
`./sbt/sbt` and then running `assembly/assembly`, `tools/assembly` and
`examples/assembly` will get you to the same state as if you had run
`assembly` -- but without the slow assembly bug.
On Wed, Jan 22, 2014 at 11:27 AM, Mark Hamstra <ma...@clearstorydata.com>wrote:
> It's a bug in sbt and/or the sbt-assembly plugin. AFAIK, nobody has
> isolated yet exactly what triggers the problem (only a minority of users
> run into it), but there are workarounds and an eventual fix. The fix
> appears to be PR #266 <https://github.com/apache/incubator-spark/pull/266>.
> Until that is merged, the combination of assembly/assembly, tools/assembly
> and examples/assembly will get you to the same state as running the
> top-level assembly target, but a lot faster if you are running into the
> slow assembly bug. There is also an assemble-deps<https://github.com/apache/incubator-spark/pull/66>that can be used by Spark developers.
>
>
> On Wed, Jan 22, 2014 at 11:15 AM, dachuan <hd...@gmail.com> wrote:
>
>> Hi, community,
>>
>> I have run sbt/sbt assembly, but it is stuck at the following place for
>> 30 minutes.
>>
>> Does anybody know what's the root cause? my machine's memory size is 4GB.
>>
>> thanks,
>> dachuan.
>>
>> [info] Packaging
>> C:\cygwin\home\dachuan\incubator-spark\assembly\target\scala-2.10\spark-assembly-0.9.0-incubating-SNAPSHOT-hadoop1.0.4.jar
>> ...
>> [info] Packaging
>> C:\cygwin\home\dachuan\incubator-spark\examples\target\scala-2.10\spark-examples-assembly-0.9.0-incubating-SNAPSHOT.jar
>> ...
>> (STUCK HERE)
>>
>> --
>> Dachuan Huang
>> Cellphone: 614-390-7234
>> 2015 Neil Avenue
>> Ohio State University
>> Columbus, Ohio
>> U.S.A.
>> 43210
>>
>
>
Re: why is it so slow to run sbt/sbt assembly in my machine?
Posted by Mark Hamstra <ma...@clearstorydata.com>.
It's a bug in sbt and/or the sbt-assembly plugin. AFAIK, nobody has
isolated yet exactly what triggers the problem (only a minority of users
run into it), but there are workarounds and an eventual fix. The fix
appears to be PR #266 <https://github.com/apache/incubator-spark/pull/266>.
Until that is merged, the combination of assembly/assembly, tools/assembly
and examples/assembly will get you to the same state as running the
top-level assembly target, but a lot faster if you are running into the
slow assembly bug. There is also an
assemble-deps<https://github.com/apache/incubator-spark/pull/66>that
can be used by Spark developers.
On Wed, Jan 22, 2014 at 11:15 AM, dachuan <hd...@gmail.com> wrote:
> Hi, community,
>
> I have run sbt/sbt assembly, but it is stuck at the following place for 30
> minutes.
>
> Does anybody know what's the root cause? my machine's memory size is 4GB.
>
> thanks,
> dachuan.
>
> [info] Packaging
> C:\cygwin\home\dachuan\incubator-spark\assembly\target\scala-2.10\spark-assembly-0.9.0-incubating-SNAPSHOT-hadoop1.0.4.jar
> ...
> [info] Packaging
> C:\cygwin\home\dachuan\incubator-spark\examples\target\scala-2.10\spark-examples-assembly-0.9.0-incubating-SNAPSHOT.jar
> ...
> (STUCK HERE)
>
> --
> Dachuan Huang
> Cellphone: 614-390-7234
> 2015 Neil Avenue
> Ohio State University
> Columbus, Ohio
> U.S.A.
> 43210
>
Re: why is it so slow to run sbt/sbt assembly in my machine?
Posted by dachuan <hd...@gmail.com>.
I didn't try too many times. Because I just git cloned the repo.
And I have just realized that it now had passed that stage and finished,
maybe the whole process takes at least 45 minutes.
I am using cygwin on windows 7.
And Mark, thanks for your suggestion, I will definitely try that if I run
into the same problem.
On Wed, Jan 22, 2014 at 2:17 PM, Tathagata Das
<ta...@gmail.com>wrote:
> Does is occur every time? I have notices once in a while sbt gets stuck
> and control-C + restart seemed to work fine.
>
> TD
>
>
> On Wed, Jan 22, 2014 at 11:15 AM, dachuan <hd...@gmail.com> wrote:
>
>> Hi, community,
>>
>> I have run sbt/sbt assembly, but it is stuck at the following place for
>> 30 minutes.
>>
>> Does anybody know what's the root cause? my machine's memory size is 4GB.
>>
>> thanks,
>> dachuan.
>>
>> [info] Packaging
>> C:\cygwin\home\dachuan\incubator-spark\assembly\target\scala-2.10\spark-assembly-0.9.0-incubating-SNAPSHOT-hadoop1.0.4.jar
>> ...
>> [info] Packaging
>> C:\cygwin\home\dachuan\incubator-spark\examples\target\scala-2.10\spark-examples-assembly-0.9.0-incubating-SNAPSHOT.jar
>> ...
>> (STUCK HERE)
>>
>> --
>> Dachuan Huang
>> Cellphone: 614-390-7234
>> 2015 Neil Avenue
>> Ohio State University
>> Columbus, Ohio
>> U.S.A.
>> 43210
>>
>
>
--
Dachuan Huang
Cellphone: 614-390-7234
2015 Neil Avenue
Ohio State University
Columbus, Ohio
U.S.A.
43210
Re: why is it so slow to run sbt/sbt assembly in my machine?
Posted by Tathagata Das <ta...@gmail.com>.
Does is occur every time? I have notices once in a while sbt gets stuck and
control-C + restart seemed to work fine.
TD
On Wed, Jan 22, 2014 at 11:15 AM, dachuan <hd...@gmail.com> wrote:
> Hi, community,
>
> I have run sbt/sbt assembly, but it is stuck at the following place for 30
> minutes.
>
> Does anybody know what's the root cause? my machine's memory size is 4GB.
>
> thanks,
> dachuan.
>
> [info] Packaging
> C:\cygwin\home\dachuan\incubator-spark\assembly\target\scala-2.10\spark-assembly-0.9.0-incubating-SNAPSHOT-hadoop1.0.4.jar
> ...
> [info] Packaging
> C:\cygwin\home\dachuan\incubator-spark\examples\target\scala-2.10\spark-examples-assembly-0.9.0-incubating-SNAPSHOT.jar
> ...
> (STUCK HERE)
>
> --
> Dachuan Huang
> Cellphone: 614-390-7234
> 2015 Neil Avenue
> Ohio State University
> Columbus, Ohio
> U.S.A.
> 43210
>