You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Olivier Girardot <ss...@gmail.com> on 2015/04/21 21:54:07 UTC
Spark build time
Hi everyone,
I was just wandering about the Spark full build time (including tests),
1h48 seems to me quite... spacious. What's taking most of the time ? Is the
build mainly integration tests ? Is there any roadmap or jiras dedicated to
that we can chip in ?
Regards,
Olivier.
RE: Spark build time
Posted by Alex <lx...@gmail.com>.
If you are using MVN there are some parameters (MAVEN_OPTS) which need to be set in order to give the underlying environment enough memory. See the instructions here: https://spark.apache.org/docs/latest/building-spark.html
-----Original Message-----
From: "Reynold Xin" <rx...@databricks.com>
Sent: 4/21/2015 4:21 PM
To: "Olivier Girardot" <ss...@gmail.com>
Cc: "dev@spark.apache.org" <de...@spark.apache.org>
Subject: Re: Spark build time
It runs tons of integration tests. I think most developers just let Jenkins
run the full suite of them.
On Tue, Apr 21, 2015 at 12:54 PM, Olivier Girardot <ss...@gmail.com>
wrote:
> Hi everyone,
> I was just wandering about the Spark full build time (including tests),
> 1h48 seems to me quite... spacious. What's taking most of the time ? Is the
> build mainly integration tests ? Is there any roadmap or jiras dedicated to
> that we can chip in ?
>
> Regards,
>
> Olivier.
>
Re: Spark build time
Posted by Nicholas Chammas <ni...@gmail.com>.
I suggest searching the archives for this list as there were several
previous discussions about this problem. JIRA also has several issues
related to this.
Some pointers:
- SPARK-3431 <https://issues.apache.org/jira/browse/SPARK-3431>:
Parallelize Scala/Java test execution
-
http://apache-spark-developers-list.1001551.n3.nabble.com/Unit-tests-in-lt-5-minutes-td7757.html
- SPARK-4746 <https://issues.apache.org/jira/browse/SPARK-4746>:
integration tests should be separated from faster unit tests
-
http://apache-spark-developers-list.1001551.n3.nabble.com/Building-Spark-with-Pants-td10397.html
Summary is, everyone agrees the long times are a problem and wants the
build and tests to run faster. There are several things that can be done,
but they all require a lot of work.
Nick
On Wed, Apr 22, 2015 at 5:25 AM Olivier Girardot <ss...@gmail.com> wrote:
> I agree, it's what I did :)
> I was just wondering if it was considered a "problem" or something to work
> on, I personally think so because the feedback loop should be as quick as
> possible, and therefore if there was someone I could help.
>
> Le mar. 21 avr. 2015 à 22:20, Reynold Xin <rx...@databricks.com> a écrit :
>
> > It runs tons of integration tests. I think most developers just let
> > Jenkins run the full suite of them.
> >
> > On Tue, Apr 21, 2015 at 12:54 PM, Olivier Girardot <ss...@gmail.com>
> > wrote:
> >
> >> Hi everyone,
> >> I was just wandering about the Spark full build time (including tests),
> >> 1h48 seems to me quite... spacious. What's taking most of the time ? Is
> >> the
> >> build mainly integration tests ? Is there any roadmap or jiras dedicated
> >> to
> >> that we can chip in ?
> >>
> >> Regards,
> >>
> >> Olivier.
> >>
> >
> >
>
Re: Spark build time
Posted by Olivier Girardot <ss...@gmail.com>.
I agree, it's what I did :)
I was just wondering if it was considered a "problem" or something to work
on, I personally think so because the feedback loop should be as quick as
possible, and therefore if there was someone I could help.
Le mar. 21 avr. 2015 à 22:20, Reynold Xin <rx...@databricks.com> a écrit :
> It runs tons of integration tests. I think most developers just let
> Jenkins run the full suite of them.
>
> On Tue, Apr 21, 2015 at 12:54 PM, Olivier Girardot <ss...@gmail.com>
> wrote:
>
>> Hi everyone,
>> I was just wandering about the Spark full build time (including tests),
>> 1h48 seems to me quite... spacious. What's taking most of the time ? Is
>> the
>> build mainly integration tests ? Is there any roadmap or jiras dedicated
>> to
>> that we can chip in ?
>>
>> Regards,
>>
>> Olivier.
>>
>
>
Re: Spark build time
Posted by Reynold Xin <rx...@databricks.com>.
It runs tons of integration tests. I think most developers just let Jenkins
run the full suite of them.
On Tue, Apr 21, 2015 at 12:54 PM, Olivier Girardot <ss...@gmail.com>
wrote:
> Hi everyone,
> I was just wandering about the Spark full build time (including tests),
> 1h48 seems to me quite... spacious. What's taking most of the time ? Is the
> build mainly integration tests ? Is there any roadmap or jiras dedicated to
> that we can chip in ?
>
> Regards,
>
> Olivier.
>