You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Patrick Wendell <pw...@gmail.com> on 2014/06/22 09:24:50 UTC
Assorted project updates (tests, build, etc)
Hey All,
1. The original test infrastructure hosted by the AMPLab has been
fully restored and also expanded with many more executor slots for
tests. Thanks to Matt Massie at the Amplab for helping with this.
2. We now have a nightly build matrix across different Hadoop
versions. It appears that the Maven build is failing tests with some
of the newer Hadoop versions. If people from the community are
interested, diagnosing and fixing test issues would be welcome patches
(they are all dependency related).
https://issues.apache.org/jira/browse/SPARK-2232
3. Prashant Sharma has spent a lot of time to make it possible for our
sbt build to read dependencies from Maven. This will save us a huge
amount of headache keeping the builds consistent. I just wanted to
give a heads up to users about this - we should retain compatibility
with features of the sbt build, but if you are e.g. hooking into deep
internals of our build it may affect you. I'm hoping this can be
updated and merged in the next week:
https://github.com/apache/spark/pull/77
4. We've moved most of the documentation over to recommending users
build with Maven when creating official packages. This is just to
provide a single "reference build" of Spark since it's the one we test
and package for releases, we make sure all recursive dependencies are
correct, etc. I'd recommend that all downstream packagers use this
build.
For day-to-day development I imagine sbt will remain more popular
(repl, incremental builds, etc). Prashant's work allows us to get the
"best of both worlds" which is great.
- Patrick
Re: Assorted project updates (tests, build, etc)
Posted by Patrick Wendell <pw...@gmail.com>.
Sorry all, I sent the wrong pull request to refer to Prashant's work:
https://github.com/apache/spark/pull/772
On Thu, Jul 3, 2014 at 1:37 PM, Patrick Wendell <pw...@gmail.com> wrote:
> Just a reminder here - we'll soon be merging a patch that changes the
> SBT build internals significantly. We've tried to make this fully
> backwards compatible, but there may be issues (which we'll resolve as
> they arrive).
>
> https://github.com/apache/spark/pull/77
>
> - Patrick
>
> On Sun, Jun 22, 2014 at 10:27 AM, Mark Hamstra <ma...@clearstorydata.com> wrote:
>> Just a couple of FYI notes: With Zinc and the scala-maven-plugin, repl and
>> incremental builds are also available to those doing day-to-day development
>> using Maven. As long as you don't have to delve into the extra boilerplate
>> and verbosity of Maven's POMs relative to an SBT build file, there is
>> little day-to-day functional difference between the two -- if anything, I
>> find that Maven supports faster development cycles.
>>
>>
>> On Sun, Jun 22, 2014 at 12:24 AM, Patrick Wendell <pw...@gmail.com>
>> wrote:
>>
>>> Hey All,
>>>
>>> 1. The original test infrastructure hosted by the AMPLab has been
>>> fully restored and also expanded with many more executor slots for
>>> tests. Thanks to Matt Massie at the Amplab for helping with this.
>>>
>>> 2. We now have a nightly build matrix across different Hadoop
>>> versions. It appears that the Maven build is failing tests with some
>>> of the newer Hadoop versions. If people from the community are
>>> interested, diagnosing and fixing test issues would be welcome patches
>>> (they are all dependency related).
>>>
>>> https://issues.apache.org/jira/browse/SPARK-2232
>>>
>>> 3. Prashant Sharma has spent a lot of time to make it possible for our
>>> sbt build to read dependencies from Maven. This will save us a huge
>>> amount of headache keeping the builds consistent. I just wanted to
>>> give a heads up to users about this - we should retain compatibility
>>> with features of the sbt build, but if you are e.g. hooking into deep
>>> internals of our build it may affect you. I'm hoping this can be
>>> updated and merged in the next week:
>>>
>>> https://github.com/apache/spark/pull/77
>>>
>>> 4. We've moved most of the documentation over to recommending users
>>> build with Maven when creating official packages. This is just to
>>> provide a single "reference build" of Spark since it's the one we test
>>> and package for releases, we make sure all recursive dependencies are
>>> correct, etc. I'd recommend that all downstream packagers use this
>>> build.
>>>
>>> For day-to-day development I imagine sbt will remain more popular
>>> (repl, incremental builds, etc). Prashant's work allows us to get the
>>> "best of both worlds" which is great.
>>>
>>> - Patrick
>>>
Re: Assorted project updates (tests, build, etc)
Posted by Patrick Wendell <pw...@gmail.com>.
Just a reminder here - we'll soon be merging a patch that changes the
SBT build internals significantly. We've tried to make this fully
backwards compatible, but there may be issues (which we'll resolve as
they arrive).
https://github.com/apache/spark/pull/77
- Patrick
On Sun, Jun 22, 2014 at 10:27 AM, Mark Hamstra <ma...@clearstorydata.com> wrote:
> Just a couple of FYI notes: With Zinc and the scala-maven-plugin, repl and
> incremental builds are also available to those doing day-to-day development
> using Maven. As long as you don't have to delve into the extra boilerplate
> and verbosity of Maven's POMs relative to an SBT build file, there is
> little day-to-day functional difference between the two -- if anything, I
> find that Maven supports faster development cycles.
>
>
> On Sun, Jun 22, 2014 at 12:24 AM, Patrick Wendell <pw...@gmail.com>
> wrote:
>
>> Hey All,
>>
>> 1. The original test infrastructure hosted by the AMPLab has been
>> fully restored and also expanded with many more executor slots for
>> tests. Thanks to Matt Massie at the Amplab for helping with this.
>>
>> 2. We now have a nightly build matrix across different Hadoop
>> versions. It appears that the Maven build is failing tests with some
>> of the newer Hadoop versions. If people from the community are
>> interested, diagnosing and fixing test issues would be welcome patches
>> (they are all dependency related).
>>
>> https://issues.apache.org/jira/browse/SPARK-2232
>>
>> 3. Prashant Sharma has spent a lot of time to make it possible for our
>> sbt build to read dependencies from Maven. This will save us a huge
>> amount of headache keeping the builds consistent. I just wanted to
>> give a heads up to users about this - we should retain compatibility
>> with features of the sbt build, but if you are e.g. hooking into deep
>> internals of our build it may affect you. I'm hoping this can be
>> updated and merged in the next week:
>>
>> https://github.com/apache/spark/pull/77
>>
>> 4. We've moved most of the documentation over to recommending users
>> build with Maven when creating official packages. This is just to
>> provide a single "reference build" of Spark since it's the one we test
>> and package for releases, we make sure all recursive dependencies are
>> correct, etc. I'd recommend that all downstream packagers use this
>> build.
>>
>> For day-to-day development I imagine sbt will remain more popular
>> (repl, incremental builds, etc). Prashant's work allows us to get the
>> "best of both worlds" which is great.
>>
>> - Patrick
>>
Re: Assorted project updates (tests, build, etc)
Posted by Mark Hamstra <ma...@clearstorydata.com>.
Just a couple of FYI notes: With Zinc and the scala-maven-plugin, repl and
incremental builds are also available to those doing day-to-day development
using Maven. As long as you don't have to delve into the extra boilerplate
and verbosity of Maven's POMs relative to an SBT build file, there is
little day-to-day functional difference between the two -- if anything, I
find that Maven supports faster development cycles.
On Sun, Jun 22, 2014 at 12:24 AM, Patrick Wendell <pw...@gmail.com>
wrote:
> Hey All,
>
> 1. The original test infrastructure hosted by the AMPLab has been
> fully restored and also expanded with many more executor slots for
> tests. Thanks to Matt Massie at the Amplab for helping with this.
>
> 2. We now have a nightly build matrix across different Hadoop
> versions. It appears that the Maven build is failing tests with some
> of the newer Hadoop versions. If people from the community are
> interested, diagnosing and fixing test issues would be welcome patches
> (they are all dependency related).
>
> https://issues.apache.org/jira/browse/SPARK-2232
>
> 3. Prashant Sharma has spent a lot of time to make it possible for our
> sbt build to read dependencies from Maven. This will save us a huge
> amount of headache keeping the builds consistent. I just wanted to
> give a heads up to users about this - we should retain compatibility
> with features of the sbt build, but if you are e.g. hooking into deep
> internals of our build it may affect you. I'm hoping this can be
> updated and merged in the next week:
>
> https://github.com/apache/spark/pull/77
>
> 4. We've moved most of the documentation over to recommending users
> build with Maven when creating official packages. This is just to
> provide a single "reference build" of Spark since it's the one we test
> and package for releases, we make sure all recursive dependencies are
> correct, etc. I'd recommend that all downstream packagers use this
> build.
>
> For day-to-day development I imagine sbt will remain more popular
> (repl, incremental builds, etc). Prashant's work allows us to get the
> "best of both worlds" which is great.
>
> - Patrick
>