You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Nicholas Chammas <ni...@gmail.com> on 2015/02/03 01:25:47 UTC
Building Spark with Pants
Does anyone here have experience with Pants
<http://pantsbuild.github.io/index.html> or interest in trying to build
Spark with it?
Pants has an interesting story. It was born at Twitter to help them build
their Scala, Java, and Python projects as several independent components in
one monolithic repo. (It was inspired by a similar build tool at Google
called blaze.) The mix of languages and sub-projects at Twitter seems
similar to the breakdown we have in Spark.
Pants has an interesting take on how a build system should work, and
Twitter and Foursquare (who use Pants as their primary build tool) claim it
helps enforce better build hygiene and maintainability.
Some relevant talks:
- Building Scala Hygienically with Pants
<https://www.youtube.com/watch?v=ukqke8iTuH0>
- The Pants Build Tool at Twitter
<https://engineering.twitter.com/university/videos/the-pants-build-tool-at-twitter>
- Getting Started with the Pants Build System: Why Pants?
<https://engineering.twitter.com/university/videos/getting-started-with-the-pants-build-system-why-pants>
At some point I may take a shot at converting Spark to use Pants as an
experiment and just see what it’s like.
Nick
Re: Building Spark with Pants
Posted by Ryan Williams <ry...@gmail.com>.
I worked on Pants at Foursquare for a while and when coming up to speed on
Spark was interested in the possibility of building it with Pants,
particularly because allowing developers to share/reuse each others'
compilation artifacts seems like it would be a boon to productivity; that
was/is Pants' "killer feature" for Foursquare, as mentioned on the
pants-devel thread.
Given the monumental nature of the task of making Spark build with Pants,
most of my enthusiasm was deflected to SPARK-1517
<https://issues.apache.org/jira/browse/SPARK-1517>, which deals with
publishing nightly builds (or better, exposing all assembly JARs built by
Jenkins?) that people could use rather than having to assemble their own.
Anyway, it's an intriguing idea, Nicholas, I'm glad you are pursuing it!
On Sat Feb 14 2015 at 4:21:16 AM Nicholas Chammas <
nicholas.chammas@gmail.com> wrote:
> FYI: Here is the matching discussion over on the Pants dev list.
> <https://groups.google.com/forum/#!topic/pants-devel/rTaU-iIOIFE>
>
> On Mon Feb 02 2015 at 4:50:33 PM Nicholas Chammas
> nicholas.chammas@gmail.com
> <ht...@gmail.com> wrote:
>
> To reiterate, I'm asking from an experimental perspective. I'm not
> > proposing we change Spark to build with Pants or anything like that.
> >
> > I'm interested in trying Pants out and I'm wondering if anyone else
> shares
> > my interest or already has experience with Pants that they can share.
> >
> > On Mon Feb 02 2015 at 4:40:45 PM Nicholas Chammas <
> > nicholas.chammas@gmail.com> wrote:
> >
> >> I'm asking from an experimental standpoint; this is not happening
> anytime
> >> soon.
> >>
> >> Of course, if the experiment turns out very well, Pants would replace
> >> both sbt and Maven (like it has at Twitter, for example). Pants also
> works
> >> with IDEs <http://pantsbuild.github.io/index.html#using-pants-with>.
> >>
> >> On Mon Feb 02 2015 at 4:33:11 PM Stephen Boesch <ja...@gmail.com>
> >> wrote:
> >>
> >>> There is a significant investment in sbt and maven - and they are not
> at
> >>> all likely to be going away. A third build tool? Note that there is
> also
> >>> the perspective of building within an IDE - which actually works
> presently
> >>> for sbt and with a little bit of tweaking with maven as well.
> >>>
> >>> 2015-02-02 16:25 GMT-08:00 Nicholas Chammas <
> nicholas.chammas@gmail.com>
> >>> :
> >>>
> >>>> Does anyone here have experience with Pants
> >>>>
> >>> <http://pantsbuild.github.io/index.html> or interest in trying to
> build
> >>>
> >>>
> >>>> Spark with it?
> >>>>
> >>>> Pants has an interesting story. It was born at Twitter to help them
> >>>> build
> >>>> their Scala, Java, and Python projects as several independent
> >>>> components in
> >>>> one monolithic repo. (It was inspired by a similar build tool at
> Google
> >>>> called blaze.) The mix of languages and sub-projects at Twitter seems
> >>>> similar to the breakdown we have in Spark.
> >>>>
> >>>> Pants has an interesting take on how a build system should work, and
> >>>> Twitter and Foursquare (who use Pants as their primary build tool)
> >>>> claim it
> >>>> helps enforce better build hygiene and maintainability.
> >>>>
> >>>> Some relevant talks:
> >>>>
> >>>> - Building Scala Hygienically with Pants
> >>>> <https://www.youtube.com/watch?v=ukqke8iTuH0>
> >>>> - The Pants Build Tool at Twitter
> >>>> <https://engineering.twitter.com/university/videos/the-pant
> >>>> s-build-tool-at-twitter>
> >>>> - Getting Started with the Pants Build System: Why Pants?
> >>>> <https://engineering.twitter.com/university/videos/getting-
> >>>> started-with-the-pants-build-system-why-pants>
> >>>
> >>>
> >>>>
> >>>> At some point I may take a shot at converting Spark to use Pants as an
> >>>> experiment and just see what it’s like.
> >>>>
> >>>> Nick
> >>>>
> >>>>
> >>>
>
Re: Building Spark with Pants
Posted by Nicholas Chammas <ni...@gmail.com>.
FYI: Here is the matching discussion over on the Pants dev list.
<https://groups.google.com/forum/#!topic/pants-devel/rTaU-iIOIFE>
On Mon Feb 02 2015 at 4:50:33 PM Nicholas Chammas nicholas.chammas@gmail.com
<ht...@gmail.com> wrote:
To reiterate, I'm asking from an experimental perspective. I'm not
> proposing we change Spark to build with Pants or anything like that.
>
> I'm interested in trying Pants out and I'm wondering if anyone else shares
> my interest or already has experience with Pants that they can share.
>
> On Mon Feb 02 2015 at 4:40:45 PM Nicholas Chammas <
> nicholas.chammas@gmail.com> wrote:
>
>> I'm asking from an experimental standpoint; this is not happening anytime
>> soon.
>>
>> Of course, if the experiment turns out very well, Pants would replace
>> both sbt and Maven (like it has at Twitter, for example). Pants also works
>> with IDEs <http://pantsbuild.github.io/index.html#using-pants-with>.
>>
>> On Mon Feb 02 2015 at 4:33:11 PM Stephen Boesch <ja...@gmail.com>
>> wrote:
>>
>>> There is a significant investment in sbt and maven - and they are not at
>>> all likely to be going away. A third build tool? Note that there is also
>>> the perspective of building within an IDE - which actually works presently
>>> for sbt and with a little bit of tweaking with maven as well.
>>>
>>> 2015-02-02 16:25 GMT-08:00 Nicholas Chammas <ni...@gmail.com>
>>> :
>>>
>>>> Does anyone here have experience with Pants
>>>>
>>> <http://pantsbuild.github.io/index.html> or interest in trying to build
>>>
>>>
>>>> Spark with it?
>>>>
>>>> Pants has an interesting story. It was born at Twitter to help them
>>>> build
>>>> their Scala, Java, and Python projects as several independent
>>>> components in
>>>> one monolithic repo. (It was inspired by a similar build tool at Google
>>>> called blaze.) The mix of languages and sub-projects at Twitter seems
>>>> similar to the breakdown we have in Spark.
>>>>
>>>> Pants has an interesting take on how a build system should work, and
>>>> Twitter and Foursquare (who use Pants as their primary build tool)
>>>> claim it
>>>> helps enforce better build hygiene and maintainability.
>>>>
>>>> Some relevant talks:
>>>>
>>>> - Building Scala Hygienically with Pants
>>>> <https://www.youtube.com/watch?v=ukqke8iTuH0>
>>>> - The Pants Build Tool at Twitter
>>>> <https://engineering.twitter.com/university/videos/the-pant
>>>> s-build-tool-at-twitter>
>>>> - Getting Started with the Pants Build System: Why Pants?
>>>> <https://engineering.twitter.com/university/videos/getting-
>>>> started-with-the-pants-build-system-why-pants>
>>>
>>>
>>>>
>>>> At some point I may take a shot at converting Spark to use Pants as an
>>>> experiment and just see what it’s like.
>>>>
>>>> Nick
>>>>
>>>>
>>>
Re: Building Spark with Pants
Posted by Nicholas Chammas <ni...@gmail.com>.
To reiterate, I'm asking from an experimental perspective. I'm not
proposing we change Spark to build with Pants or anything like that.
I'm interested in trying Pants out and I'm wondering if anyone else shares
my interest or already has experience with Pants that they can share.
On Mon Feb 02 2015 at 4:40:45 PM Nicholas Chammas <
nicholas.chammas@gmail.com> wrote:
> I'm asking from an experimental standpoint; this is not happening anytime
> soon.
>
> Of course, if the experiment turns out very well, Pants would replace both
> sbt and Maven (like it has at Twitter, for example). Pants also works
> with IDEs <http://pantsbuild.github.io/index.html#using-pants-with>.
>
> On Mon Feb 02 2015 at 4:33:11 PM Stephen Boesch <ja...@gmail.com> wrote:
>
>> There is a significant investment in sbt and maven - and they are not at
>> all likely to be going away. A third build tool? Note that there is also
>> the perspective of building within an IDE - which actually works presently
>> for sbt and with a little bit of tweaking with maven as well.
>>
>> 2015-02-02 16:25 GMT-08:00 Nicholas Chammas <ni...@gmail.com>:
>>
>>> Does anyone here have experience with Pants
>>>
>> <http://pantsbuild.github.io/index.html> or interest in trying to build
>>
>>
>>> Spark with it?
>>>
>>> Pants has an interesting story. It was born at Twitter to help them build
>>> their Scala, Java, and Python projects as several independent components
>>> in
>>> one monolithic repo. (It was inspired by a similar build tool at Google
>>> called blaze.) The mix of languages and sub-projects at Twitter seems
>>> similar to the breakdown we have in Spark.
>>>
>>> Pants has an interesting take on how a build system should work, and
>>> Twitter and Foursquare (who use Pants as their primary build tool) claim
>>> it
>>> helps enforce better build hygiene and maintainability.
>>>
>>> Some relevant talks:
>>>
>>> - Building Scala Hygienically with Pants
>>> <https://www.youtube.com/watch?v=ukqke8iTuH0>
>>> - The Pants Build Tool at Twitter
>>> <https://engineering.twitter.com/university/videos/the-
>>> pants-build-tool-at-twitter>
>>> - Getting Started with the Pants Build System: Why Pants?
>>> <https://engineering.twitter.com/university/videos/getting-
>>> started-with-the-pants-build-system-why-pants>
>>
>>
>>>
>>> At some point I may take a shot at converting Spark to use Pants as an
>>> experiment and just see what it’s like.
>>>
>>> Nick
>>>
>>>
>>
Re: Building Spark with Pants
Posted by Nicholas Chammas <ni...@gmail.com>.
I'm asking from an experimental standpoint; this is not happening anytime
soon.
Of course, if the experiment turns out very well, Pants would replace both
sbt and Maven (like it has at Twitter, for example). Pants also works with
IDEs <http://pantsbuild.github.io/index.html#using-pants-with>.
On Mon Feb 02 2015 at 4:33:11 PM Stephen Boesch <ja...@gmail.com> wrote:
> There is a significant investment in sbt and maven - and they are not at
> all likely to be going away. A third build tool? Note that there is also
> the perspective of building within an IDE - which actually works presently
> for sbt and with a little bit of tweaking with maven as well.
>
> 2015-02-02 16:25 GMT-08:00 Nicholas Chammas <ni...@gmail.com>:
>
>> Does anyone here have experience with Pants
>>
> <http://pantsbuild.github.io/index.html> or interest in trying to build
>
>
>> Spark with it?
>>
>> Pants has an interesting story. It was born at Twitter to help them build
>> their Scala, Java, and Python projects as several independent components
>> in
>> one monolithic repo. (It was inspired by a similar build tool at Google
>> called blaze.) The mix of languages and sub-projects at Twitter seems
>> similar to the breakdown we have in Spark.
>>
>> Pants has an interesting take on how a build system should work, and
>> Twitter and Foursquare (who use Pants as their primary build tool) claim
>> it
>> helps enforce better build hygiene and maintainability.
>>
>> Some relevant talks:
>>
>> - Building Scala Hygienically with Pants
>> <https://www.youtube.com/watch?v=ukqke8iTuH0>
>> - The Pants Build Tool at Twitter
>> <
>> https://engineering.twitter.com/university/videos/the-pants-build-tool-at-twitter
>> >
>> - Getting Started with the Pants Build System: Why Pants?
>> <
>> https://engineering.twitter.com/university/videos/getting-started-with-the-pants-build-system-why-pants
>> >
>
>
>>
>> At some point I may take a shot at converting Spark to use Pants as an
>> experiment and just see what it’s like.
>>
>> Nick
>>
>>
>
Re: Building Spark with Pants
Posted by Stephen Boesch <ja...@gmail.com>.
There is a significant investment in sbt and maven - and they are not at
all likely to be going away. A third build tool? Note that there is also
the perspective of building within an IDE - which actually works presently
for sbt and with a little bit of tweaking with maven as well.
2015-02-02 16:25 GMT-08:00 Nicholas Chammas <ni...@gmail.com>:
> Does anyone here have experience with Pants
> <http://pantsbuild.github.io/index.html> or interest in trying to build
> Spark with it?
>
> Pants has an interesting story. It was born at Twitter to help them build
> their Scala, Java, and Python projects as several independent components in
> one monolithic repo. (It was inspired by a similar build tool at Google
> called blaze.) The mix of languages and sub-projects at Twitter seems
> similar to the breakdown we have in Spark.
>
> Pants has an interesting take on how a build system should work, and
> Twitter and Foursquare (who use Pants as their primary build tool) claim it
> helps enforce better build hygiene and maintainability.
>
> Some relevant talks:
>
> - Building Scala Hygienically with Pants
> <https://www.youtube.com/watch?v=ukqke8iTuH0>
> - The Pants Build Tool at Twitter
> <
> https://engineering.twitter.com/university/videos/the-pants-build-tool-at-twitter
> >
> - Getting Started with the Pants Build System: Why Pants?
> <
> https://engineering.twitter.com/university/videos/getting-started-with-the-pants-build-system-why-pants
> >
>
> At some point I may take a shot at converting Spark to use Pants as an
> experiment and just see what it’s like.
>
> Nick
>
>