You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Hari Shreedharan <hs...@cloudera.com> on 2014/10/24 22:39:13 UTC
Moving PR Builder to mvn
Over the last few months, it seems like we have selected Maven to be the “official” build system for Spark.
I realize that removing the sbt build may not be easy, but it might be a good idea to start looking into that. We had issues over the past few days where mvn builds were fine, while sbt was failing to resolve dependencies which were test-jars causing compilation of certain tests to fail.
As a first step, I am wondering if it might be a good idea to change the PR builder to mvn and test PRs consistent with the way we test releases. I am not sure how technically feasible it is, but it would be a start to standardizing on one build system.
Thanks,
Hari
Re: Moving PR Builder to mvn
Posted by Mark Hamstra <ma...@clearstorydata.com>.
Your's are in the same ballpark with mine, where maven builds with zinc
take about 1.4x the time to build with SBT.
On Fri, Oct 24, 2014 at 4:24 PM, Sean Owen <so...@cloudera.com> wrote:
> Here's a crude benchmark on a Linux box (GCE n1-standard-4). zinc gets
> the assembly build in range of SBT's time.
>
> mvn -DskipTests clean package
> 15:27
> (start zinc)
> 8:18
> (rebuild)
> 7:08
>
> ./sbt/sbt -DskipTests clean assembly
> 5:10
> (start zinc)
> 5:11
> (rebuild)
> 5:06
>
> The dependencies were already downloaded, and the whole build was
> cleaned in between.
>
> These are smallish in comparison with time to run tests. I admit I
> didn't run them here in the interest of time and because I assumed
> zinc doesn't help that.
>
>
> On Fri, Oct 24, 2014 at 10:06 PM, Patrick Wendell <pw...@gmail.com>
> wrote:
> > Does Zinc still help if you are just running a single totally fresh
> > build? For the pull request builder we purge all state from previous
> > builds.
> >
> > - Patrick
> >
> > On Fri, Oct 24, 2014 at 1:55 PM, Hari Shreedharan
> > <hs...@cloudera.com> wrote:
> >> I have zinc server running on my mac, and I see maven compilation to be
> much
> >> better than before I had it running. Is the sbt build still faster
> (sorry,
> >> long time since I did a build with sbt).
> >>
> >> Thanks,
> >> Hari
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>
Re: Moving PR Builder to mvn
Posted by Sean Owen <so...@cloudera.com>.
Here's a crude benchmark on a Linux box (GCE n1-standard-4). zinc gets
the assembly build in range of SBT's time.
mvn -DskipTests clean package
15:27
(start zinc)
8:18
(rebuild)
7:08
./sbt/sbt -DskipTests clean assembly
5:10
(start zinc)
5:11
(rebuild)
5:06
The dependencies were already downloaded, and the whole build was
cleaned in between.
These are smallish in comparison with time to run tests. I admit I
didn't run them here in the interest of time and because I assumed
zinc doesn't help that.
On Fri, Oct 24, 2014 at 10:06 PM, Patrick Wendell <pw...@gmail.com> wrote:
> Does Zinc still help if you are just running a single totally fresh
> build? For the pull request builder we purge all state from previous
> builds.
>
> - Patrick
>
> On Fri, Oct 24, 2014 at 1:55 PM, Hari Shreedharan
> <hs...@cloudera.com> wrote:
>> I have zinc server running on my mac, and I see maven compilation to be much
>> better than before I had it running. Is the sbt build still faster (sorry,
>> long time since I did a build with sbt).
>>
>> Thanks,
>> Hari
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Moving PR Builder to mvn
Posted by Hari Shreedharan <hs...@cloudera.com>.
+1. From what I can see, it definitely does - though I must say I rarely do full end to end builds though. Maybe worth running as an experiment?
Thanks,
Hari
On Fri, Oct 24, 2014 at 2:34 PM, Stephen Boesch <ja...@gmail.com> wrote:
> Zinc absolutely helps - feels like makes builds more than twice as fast -
> both on Mac and Linux. It helps both on fresh and existing builds.
> 2014-10-24 14:06 GMT-07:00 Patrick Wendell <pw...@gmail.com>:
>> Does Zinc still help if you are just running a single totally fresh
>> build? For the pull request builder we purge all state from previous
>> builds.
>>
>> - Patrick
>>
>> On Fri, Oct 24, 2014 at 1:55 PM, Hari Shreedharan
>> <hs...@cloudera.com> wrote:
>> > I have zinc server running on my mac, and I see maven compilation to be
>> much
>> > better than before I had it running. Is the sbt build still faster
>> (sorry,
>> > long time since I did a build with sbt).
>> >
>> > Thanks,
>> > Hari
>> >
>> >
>> > On Fri, Oct 24, 2014 at 1:46 PM, Patrick Wendell <pw...@gmail.com>
>> wrote:
>> >>
>> >> Overall I think this would be a good idea. The main blocker is just
>> >> that I think the Maven build is much slower right now than the SBT
>> >> build. However, if we were able to e.g. parallelize the test build on
>> >> Jenkins that might make up for it.
>> >>
>> >> I'd actually like to have a trigger where we could tests pull requests
>> >> with either one.
>> >>
>> >> - Patrick
>> >>
>> >> On Fri, Oct 24, 2014 at 1:39 PM, Hari Shreedharan
>> >> <hs...@cloudera.com> wrote:
>> >> > Over the last few months, it seems like we have selected Maven to be
>> the
>> >> > "official" build system for Spark.
>> >> >
>> >> >
>> >> > I realize that removing the sbt build may not be easy, but it might
>> be a
>> >> > good idea to start looking into that. We had issues over the past few
>> days
>> >> > where mvn builds were fine, while sbt was failing to resolve
>> dependencies
>> >> > which were test-jars causing compilation of certain tests to fail.
>> >> >
>> >> >
>> >> > As a first step, I am wondering if it might be a good idea to change
>> the
>> >> > PR builder to mvn and test PRs consistent with the way we test
>> releases. I
>> >> > am not sure how technically feasible it is, but it would be a start to
>> >> > standardizing on one build system.
>> >> >
>> >> > Thanks,
>> >> > Hari
>> >
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> For additional commands, e-mail: dev-help@spark.apache.org
>>
>>
Re: Moving PR Builder to mvn
Posted by Stephen Boesch <ja...@gmail.com>.
Zinc absolutely helps - feels like makes builds more than twice as fast -
both on Mac and Linux. It helps both on fresh and existing builds.
2014-10-24 14:06 GMT-07:00 Patrick Wendell <pw...@gmail.com>:
> Does Zinc still help if you are just running a single totally fresh
> build? For the pull request builder we purge all state from previous
> builds.
>
> - Patrick
>
> On Fri, Oct 24, 2014 at 1:55 PM, Hari Shreedharan
> <hs...@cloudera.com> wrote:
> > I have zinc server running on my mac, and I see maven compilation to be
> much
> > better than before I had it running. Is the sbt build still faster
> (sorry,
> > long time since I did a build with sbt).
> >
> > Thanks,
> > Hari
> >
> >
> > On Fri, Oct 24, 2014 at 1:46 PM, Patrick Wendell <pw...@gmail.com>
> wrote:
> >>
> >> Overall I think this would be a good idea. The main blocker is just
> >> that I think the Maven build is much slower right now than the SBT
> >> build. However, if we were able to e.g. parallelize the test build on
> >> Jenkins that might make up for it.
> >>
> >> I'd actually like to have a trigger where we could tests pull requests
> >> with either one.
> >>
> >> - Patrick
> >>
> >> On Fri, Oct 24, 2014 at 1:39 PM, Hari Shreedharan
> >> <hs...@cloudera.com> wrote:
> >> > Over the last few months, it seems like we have selected Maven to be
> the
> >> > "official" build system for Spark.
> >> >
> >> >
> >> > I realize that removing the sbt build may not be easy, but it might
> be a
> >> > good idea to start looking into that. We had issues over the past few
> days
> >> > where mvn builds were fine, while sbt was failing to resolve
> dependencies
> >> > which were test-jars causing compilation of certain tests to fail.
> >> >
> >> >
> >> > As a first step, I am wondering if it might be a good idea to change
> the
> >> > PR builder to mvn and test PRs consistent with the way we test
> releases. I
> >> > am not sure how technically feasible it is, but it would be a start to
> >> > standardizing on one build system.
> >> >
> >> > Thanks,
> >> > Hari
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>
Re: Moving PR Builder to mvn
Posted by Patrick Wendell <pw...@gmail.com>.
Does Zinc still help if you are just running a single totally fresh
build? For the pull request builder we purge all state from previous
builds.
- Patrick
On Fri, Oct 24, 2014 at 1:55 PM, Hari Shreedharan
<hs...@cloudera.com> wrote:
> I have zinc server running on my mac, and I see maven compilation to be much
> better than before I had it running. Is the sbt build still faster (sorry,
> long time since I did a build with sbt).
>
> Thanks,
> Hari
>
>
> On Fri, Oct 24, 2014 at 1:46 PM, Patrick Wendell <pw...@gmail.com> wrote:
>>
>> Overall I think this would be a good idea. The main blocker is just
>> that I think the Maven build is much slower right now than the SBT
>> build. However, if we were able to e.g. parallelize the test build on
>> Jenkins that might make up for it.
>>
>> I'd actually like to have a trigger where we could tests pull requests
>> with either one.
>>
>> - Patrick
>>
>> On Fri, Oct 24, 2014 at 1:39 PM, Hari Shreedharan
>> <hs...@cloudera.com> wrote:
>> > Over the last few months, it seems like we have selected Maven to be the
>> > "official" build system for Spark.
>> >
>> >
>> > I realize that removing the sbt build may not be easy, but it might be a
>> > good idea to start looking into that. We had issues over the past few days
>> > where mvn builds were fine, while sbt was failing to resolve dependencies
>> > which were test-jars causing compilation of certain tests to fail.
>> >
>> >
>> > As a first step, I am wondering if it might be a good idea to change the
>> > PR builder to mvn and test PRs consistent with the way we test releases. I
>> > am not sure how technically feasible it is, but it would be a start to
>> > standardizing on one build system.
>> >
>> > Thanks,
>> > Hari
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Moving PR Builder to mvn
Posted by Hari Shreedharan <hs...@cloudera.com>.
I have zinc server running on my mac, and I see maven compilation to be much better than before I had it running. Is the sbt build still faster (sorry, long time since I did a build with sbt).
Thanks,
Hari
On Fri, Oct 24, 2014 at 1:46 PM, Patrick Wendell <pw...@gmail.com>
wrote:
> Overall I think this would be a good idea. The main blocker is just
> that I think the Maven build is much slower right now than the SBT
> build. However, if we were able to e.g. parallelize the test build on
> Jenkins that might make up for it.
> I'd actually like to have a trigger where we could tests pull requests
> with either one.
> - Patrick
> On Fri, Oct 24, 2014 at 1:39 PM, Hari Shreedharan
> <hs...@cloudera.com> wrote:
>> Over the last few months, it seems like we have selected Maven to be the "official" build system for Spark.
>>
>>
>> I realize that removing the sbt build may not be easy, but it might be a good idea to start looking into that. We had issues over the past few days where mvn builds were fine, while sbt was failing to resolve dependencies which were test-jars causing compilation of certain tests to fail.
>>
>>
>> As a first step, I am wondering if it might be a good idea to change the PR builder to mvn and test PRs consistent with the way we test releases. I am not sure how technically feasible it is, but it would be a start to standardizing on one build system.
>>
>> Thanks,
>> Hari
Re: Moving PR Builder to mvn
Posted by Patrick Wendell <pw...@gmail.com>.
Overall I think this would be a good idea. The main blocker is just
that I think the Maven build is much slower right now than the SBT
build. However, if we were able to e.g. parallelize the test build on
Jenkins that might make up for it.
I'd actually like to have a trigger where we could tests pull requests
with either one.
- Patrick
On Fri, Oct 24, 2014 at 1:39 PM, Hari Shreedharan
<hs...@cloudera.com> wrote:
> Over the last few months, it seems like we have selected Maven to be the "official" build system for Spark.
>
>
> I realize that removing the sbt build may not be easy, but it might be a good idea to start looking into that. We had issues over the past few days where mvn builds were fine, while sbt was failing to resolve dependencies which were test-jars causing compilation of certain tests to fail.
>
>
> As a first step, I am wondering if it might be a good idea to change the PR builder to mvn and test PRs consistent with the way we test releases. I am not sure how technically feasible it is, but it would be a start to standardizing on one build system.
>
> Thanks,
> Hari
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org