You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Mridul Muralidharan <mr...@gmail.com> on 2014/02/23 07:06:05 UTC
[DISCUSS] Extending public API
Hi,
Over the past few months, I have seen a bunch of pull requests which have
extended spark api ... most commonly RDD itself.
Most of them are either relatively niche case of specialization (which
might not be useful for most cases) or idioms which can be expressed
(sometimes with minor perf penalty) using existing api.
While all of them have non zero value (hence the effort to contribute, and
gladly welcomed !) they are extending the api in nontrivial ways and have a
maintenance cost ... and we already have a pending effort to clean up our
interfaces prior to 1.0
I believe there is a need to keep exposed api succint, expressive and
functional in spark; while at the same time, encouraging extensions and
specialization within spark codebase so that other users can benefit from
the shared contributions.
One approach could be to start something akin to piggybank in pig to
contribute user generated specializations, helper utils, etc : bundled as
part of spark, but not part of core itself.
Thoughts, comments ?
Regards,
Mridul
Re: [DISCUSS] Extending public API
Posted by Cheng Lian <rh...@gmail.com>.
+1
We can maintain a contrib component just like what Akka and many other projects do. With the help of Scala implicits, extensions to the RDD API can be done in a non-intrusive way and leave spark-core untouched.
On Feb 23, 2014, at 2:06 PM, Mridul Muralidharan <mr...@gmail.com> wrote:
> Hi,
>
> Over the past few months, I have seen a bunch of pull requests which have
> extended spark api ... most commonly RDD itself.
>
> Most of them are either relatively niche case of specialization (which
> might not be useful for most cases) or idioms which can be expressed
> (sometimes with minor perf penalty) using existing api.
>
> While all of them have non zero value (hence the effort to contribute, and
> gladly welcomed !) they are extending the api in nontrivial ways and have a
> maintenance cost ... and we already have a pending effort to clean up our
> interfaces prior to 1.0
>
> I believe there is a need to keep exposed api succint, expressive and
> functional in spark; while at the same time, encouraging extensions and
> specialization within spark codebase so that other users can benefit from
> the shared contributions.
>
> One approach could be to start something akin to piggybank in pig to
> contribute user generated specializations, helper utils, etc : bundled as
> part of spark, but not part of core itself.
>
> Thoughts, comments ?
>
> Regards,
> Mridul
Re: [DISCUSS] Extending public API
Posted by Mark Hamstra <ma...@clearstorydata.com>.
I'm also curious what the vetting process will be for this spark-contrib
code? Does inclusion in spark-contrib mean that it has received some sort
of review and official blessing, or is contrib just a dumping ground for
code of questionable quality, utility, maintenance, etc.?
On Sat, Feb 22, 2014 at 10:23 PM, Amandeep Khurana <am...@gmail.com> wrote:
> Mridul,
>
> Can you give examples of APIs that people have contributed (or wanted
> to contribute) but you categorize as something that would go into
> piggybank-like (sparkbank)? Curious to know how you'd decide what
> should go where.
>
> Amandeep
>
> > On Feb 22, 2014, at 10:06 PM, Mridul Muralidharan <mr...@gmail.com>
> wrote:
> >
> > Hi,
> >
> > Over the past few months, I have seen a bunch of pull requests which
> have
> > extended spark api ... most commonly RDD itself.
> >
> > Most of them are either relatively niche case of specialization (which
> > might not be useful for most cases) or idioms which can be expressed
> > (sometimes with minor perf penalty) using existing api.
> >
> > While all of them have non zero value (hence the effort to contribute,
> and
> > gladly welcomed !) they are extending the api in nontrivial ways and
> have a
> > maintenance cost ... and we already have a pending effort to clean up our
> > interfaces prior to 1.0
> >
> > I believe there is a need to keep exposed api succint, expressive and
> > functional in spark; while at the same time, encouraging extensions and
> > specialization within spark codebase so that other users can benefit from
> > the shared contributions.
> >
> > One approach could be to start something akin to piggybank in pig to
> > contribute user generated specializations, helper utils, etc : bundled as
> > part of spark, but not part of core itself.
> >
> > Thoughts, comments ?
> >
> > Regards,
> > Mridul
>
Re: [DISCUSS] Extending public API
Posted by Matei Zaharia <ma...@gmail.com>.
My sense on all this is that it should be done on a case-by-case basis. To add a new API, it needs to be general enough that a lot of users will want to use it. If adding that API confuses users, that’s a problem. However, on the flip side, if it’s not a super-popular function but it’s just 10-20 lines of code, it may still be worth having. The maintenance burden on that is not too high, and users are used to fairly extensive collection libraries.
For the joins in particular, we added them because it’s quite easy to mess up writing joins by hand, even once you have cogroup().
One thing we do want to do is start implementing more specialized functionality, like statistics functions, in separate libraries. Right now there are some functions in the RDD API (e.g. sums, means, histograms, etc) that are fairly specific to this domain.
Matei
On Feb 23, 2014, at 10:18 AM, Amandeep Khurana <am...@gmail.com> wrote:
> This makes sense. Thanks for clarifying, Mridul.
>
> As Sean pointed out - a contrib module quickly turns into a legacy code
> base that becomes hard to maintain. From that perspective, I think the idea
> of a separate sparkbank github that is maintained by Spark contributors
> (along with users who wish to contribute add-ons like you've described) and
> adhere to the code quality and reviews like the main project seems
> appealing. And then not just sparkbank but other things that people might
> want to have as a part of the project but doesn't belong to the core
> codebase can go there? I don't know if things like this have come up in the
> past pull requests.
>
> -Amandeep
>
> PS: I'm not a spark committer/contributor so take my opinion fwiw. :)
>
>
> On Sun, Feb 23, 2014 at 1:40 AM, Mridul Muralidharan <mr...@gmail.com>wrote:
>
>> Good point, and I was purposefully vague on that since that is something
>> which our community should evolve imo : this was just an initial proposal
>> :-)
>>
>> For example: there are multiple ways to do cartesian - and each has its own
>> trade offs.
>>
>> Another candidate could be, as I mentioned, new methods which can be
>> expressed as sequences of existing methods but would be slightly more
>> performent if done in one shot - like the self cartesian pr, various types
>> of join (which can become a contrib of its own btw !), experiments using
>> key indexes, ordering, etc.
>>
>> Addition into sparkbank or contrib (or something bettrr named !) does not
>> preclude future migration into core ... just an initial staging area for us
>> to e olve the api and get user feedback; without necessarily making spark
>> core api unstable.
>>
>> Obviously, it is not a dumping ground for broken code/ideas ... and must
>> follow same level of scrutiny and rigour before committing.
>> Regards
>> Mridul
>> On Feb 23, 2014 11:53 AM, "Amandeep Khurana" <am...@gmail.com> wrote:
>>
>>> Mridul,
>>>
>>> Can you give examples of APIs that people have contributed (or wanted
>>> to contribute) but you categorize as something that would go into
>>> piggybank-like (sparkbank)? Curious to know how you'd decide what
>>> should go where.
>>>
>>> Amandeep
>>>
>>>> On Feb 22, 2014, at 10:06 PM, Mridul Muralidharan <mr...@gmail.com>
>>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> Over the past few months, I have seen a bunch of pull requests which
>>> have
>>>> extended spark api ... most commonly RDD itself.
>>>>
>>>> Most of them are either relatively niche case of specialization (which
>>>> might not be useful for most cases) or idioms which can be expressed
>>>> (sometimes with minor perf penalty) using existing api.
>>>>
>>>> While all of them have non zero value (hence the effort to contribute,
>>> and
>>>> gladly welcomed !) they are extending the api in nontrivial ways and
>>> have a
>>>> maintenance cost ... and we already have a pending effort to clean up
>> our
>>>> interfaces prior to 1.0
>>>>
>>>> I believe there is a need to keep exposed api succint, expressive and
>>>> functional in spark; while at the same time, encouraging extensions and
>>>> specialization within spark codebase so that other users can benefit
>> from
>>>> the shared contributions.
>>>>
>>>> One approach could be to start something akin to piggybank in pig to
>>>> contribute user generated specializations, helper utils, etc : bundled
>> as
>>>> part of spark, but not part of core itself.
>>>>
>>>> Thoughts, comments ?
>>>>
>>>> Regards,
>>>> Mridul
>>>
>>
Re: [DISCUSS] Extending public API
Posted by Amandeep Khurana <am...@gmail.com>.
This makes sense. Thanks for clarifying, Mridul.
As Sean pointed out - a contrib module quickly turns into a legacy code
base that becomes hard to maintain. From that perspective, I think the idea
of a separate sparkbank github that is maintained by Spark contributors
(along with users who wish to contribute add-ons like you've described) and
adhere to the code quality and reviews like the main project seems
appealing. And then not just sparkbank but other things that people might
want to have as a part of the project but doesn't belong to the core
codebase can go there? I don't know if things like this have come up in the
past pull requests.
-Amandeep
PS: I'm not a spark committer/contributor so take my opinion fwiw. :)
On Sun, Feb 23, 2014 at 1:40 AM, Mridul Muralidharan <mr...@gmail.com>wrote:
> Good point, and I was purposefully vague on that since that is something
> which our community should evolve imo : this was just an initial proposal
> :-)
>
> For example: there are multiple ways to do cartesian - and each has its own
> trade offs.
>
> Another candidate could be, as I mentioned, new methods which can be
> expressed as sequences of existing methods but would be slightly more
> performent if done in one shot - like the self cartesian pr, various types
> of join (which can become a contrib of its own btw !), experiments using
> key indexes, ordering, etc.
>
> Addition into sparkbank or contrib (or something bettrr named !) does not
> preclude future migration into core ... just an initial staging area for us
> to e olve the api and get user feedback; without necessarily making spark
> core api unstable.
>
> Obviously, it is not a dumping ground for broken code/ideas ... and must
> follow same level of scrutiny and rigour before committing.
> Regards
> Mridul
> On Feb 23, 2014 11:53 AM, "Amandeep Khurana" <am...@gmail.com> wrote:
>
> > Mridul,
> >
> > Can you give examples of APIs that people have contributed (or wanted
> > to contribute) but you categorize as something that would go into
> > piggybank-like (sparkbank)? Curious to know how you'd decide what
> > should go where.
> >
> > Amandeep
> >
> > > On Feb 22, 2014, at 10:06 PM, Mridul Muralidharan <mr...@gmail.com>
> > wrote:
> > >
> > > Hi,
> > >
> > > Over the past few months, I have seen a bunch of pull requests which
> > have
> > > extended spark api ... most commonly RDD itself.
> > >
> > > Most of them are either relatively niche case of specialization (which
> > > might not be useful for most cases) or idioms which can be expressed
> > > (sometimes with minor perf penalty) using existing api.
> > >
> > > While all of them have non zero value (hence the effort to contribute,
> > and
> > > gladly welcomed !) they are extending the api in nontrivial ways and
> > have a
> > > maintenance cost ... and we already have a pending effort to clean up
> our
> > > interfaces prior to 1.0
> > >
> > > I believe there is a need to keep exposed api succint, expressive and
> > > functional in spark; while at the same time, encouraging extensions and
> > > specialization within spark codebase so that other users can benefit
> from
> > > the shared contributions.
> > >
> > > One approach could be to start something akin to piggybank in pig to
> > > contribute user generated specializations, helper utils, etc : bundled
> as
> > > part of spark, but not part of core itself.
> > >
> > > Thoughts, comments ?
> > >
> > > Regards,
> > > Mridul
> >
>
Re: [DISCUSS] Extending public API
Posted by Mridul Muralidharan <mr...@gmail.com>.
Good point, and I was purposefully vague on that since that is something
which our community should evolve imo : this was just an initial proposal
:-)
For example: there are multiple ways to do cartesian - and each has its own
trade offs.
Another candidate could be, as I mentioned, new methods which can be
expressed as sequences of existing methods but would be slightly more
performent if done in one shot - like the self cartesian pr, various types
of join (which can become a contrib of its own btw !), experiments using
key indexes, ordering, etc.
Addition into sparkbank or contrib (or something bettrr named !) does not
preclude future migration into core ... just an initial staging area for us
to e olve the api and get user feedback; without necessarily making spark
core api unstable.
Obviously, it is not a dumping ground for broken code/ideas ... and must
follow same level of scrutiny and rigour before committing.
Regards
Mridul
On Feb 23, 2014 11:53 AM, "Amandeep Khurana" <am...@gmail.com> wrote:
> Mridul,
>
> Can you give examples of APIs that people have contributed (or wanted
> to contribute) but you categorize as something that would go into
> piggybank-like (sparkbank)? Curious to know how you'd decide what
> should go where.
>
> Amandeep
>
> > On Feb 22, 2014, at 10:06 PM, Mridul Muralidharan <mr...@gmail.com>
> wrote:
> >
> > Hi,
> >
> > Over the past few months, I have seen a bunch of pull requests which
> have
> > extended spark api ... most commonly RDD itself.
> >
> > Most of them are either relatively niche case of specialization (which
> > might not be useful for most cases) or idioms which can be expressed
> > (sometimes with minor perf penalty) using existing api.
> >
> > While all of them have non zero value (hence the effort to contribute,
> and
> > gladly welcomed !) they are extending the api in nontrivial ways and
> have a
> > maintenance cost ... and we already have a pending effort to clean up our
> > interfaces prior to 1.0
> >
> > I believe there is a need to keep exposed api succint, expressive and
> > functional in spark; while at the same time, encouraging extensions and
> > specialization within spark codebase so that other users can benefit from
> > the shared contributions.
> >
> > One approach could be to start something akin to piggybank in pig to
> > contribute user generated specializations, helper utils, etc : bundled as
> > part of spark, but not part of core itself.
> >
> > Thoughts, comments ?
> >
> > Regards,
> > Mridul
>
Re: [DISCUSS] Extending public API
Posted by Cheng Lian <rh...@gmail.com>.
I think SPARK-1063 (PR-503) “Add .sortBy(f) method on RDD” would be a good example. Note that I’m not saying that this PR is already qualified to be accepted, just take it as an example:
JIRA issue: https://spark-project.atlassian.net/browse/SPARK-1063
GitHub PR: https://github.com/apache/incubator-spark/pull/508
On Feb 23, 2014, at 2:23 PM, Amandeep Khurana <am...@gmail.com> wrote:
> Mridul,
>
> Can you give examples of APIs that people have contributed (or wanted
> to contribute) but you categorize as something that would go into
> piggybank-like (sparkbank)? Curious to know how you'd decide what
> should go where.
>
> Amandeep
>
>> On Feb 22, 2014, at 10:06 PM, Mridul Muralidharan <mr...@gmail.com> wrote:
>>
>> Hi,
>>
>> Over the past few months, I have seen a bunch of pull requests which have
>> extended spark api ... most commonly RDD itself.
>>
>> Most of them are either relatively niche case of specialization (which
>> might not be useful for most cases) or idioms which can be expressed
>> (sometimes with minor perf penalty) using existing api.
>>
>> While all of them have non zero value (hence the effort to contribute, and
>> gladly welcomed !) they are extending the api in nontrivial ways and have a
>> maintenance cost ... and we already have a pending effort to clean up our
>> interfaces prior to 1.0
>>
>> I believe there is a need to keep exposed api succint, expressive and
>> functional in spark; while at the same time, encouraging extensions and
>> specialization within spark codebase so that other users can benefit from
>> the shared contributions.
>>
>> One approach could be to start something akin to piggybank in pig to
>> contribute user generated specializations, helper utils, etc : bundled as
>> part of spark, but not part of core itself.
>>
>> Thoughts, comments ?
>>
>> Regards,
>> Mridul
Re: [DISCUSS] Extending public API
Posted by Amandeep Khurana <am...@gmail.com>.
Mridul,
Can you give examples of APIs that people have contributed (or wanted
to contribute) but you categorize as something that would go into
piggybank-like (sparkbank)? Curious to know how you'd decide what
should go where.
Amandeep
> On Feb 22, 2014, at 10:06 PM, Mridul Muralidharan <mr...@gmail.com> wrote:
>
> Hi,
>
> Over the past few months, I have seen a bunch of pull requests which have
> extended spark api ... most commonly RDD itself.
>
> Most of them are either relatively niche case of specialization (which
> might not be useful for most cases) or idioms which can be expressed
> (sometimes with minor perf penalty) using existing api.
>
> While all of them have non zero value (hence the effort to contribute, and
> gladly welcomed !) they are extending the api in nontrivial ways and have a
> maintenance cost ... and we already have a pending effort to clean up our
> interfaces prior to 1.0
>
> I believe there is a need to keep exposed api succint, expressive and
> functional in spark; while at the same time, encouraging extensions and
> specialization within spark codebase so that other users can benefit from
> the shared contributions.
>
> One approach could be to start something akin to piggybank in pig to
> contribute user generated specializations, helper utils, etc : bundled as
> part of spark, but not part of core itself.
>
> Thoughts, comments ?
>
> Regards,
> Mridul
Re: [DISCUSS] Extending public API
Posted by Sean Owen <so...@cloudera.com>.
Thank you for bringing this up. I think the current committers are
bravely facing down a flood of PRs, and this (among other things) is a
step that needs to be taken to scale up and keep this fun. I'd love to
have a separate discussion about more steps, but for here I offer two
bits of advice from experience:
First, you guys most certainly can and should say 'no' to some
changes. It's part of keeping the project coherent. It's always good
to try to include all contributions, but, appreciating contributions
does not always mean accepting them. I have seen projects turned to
mush by the 'anything's welcome' mentality. Push back on contributors
to contribute the thing you think is right. Please keep the API
succinct, yes.
Second, contrib/ modules are problematic. It becomes a ball of legacy
code that you still have to keep maintaining to compile and run. In a
world of Github, I think 'contrib' stuff just belongs in other repos.
I know it sounds harmless to have a contrib, but I think you'd find
the consensus here is that contrib is a mistake.
$0.02 --
--
Sean Owen | Director, Data Science | London
On Sun, Feb 23, 2014 at 6:06 AM, Mridul Muralidharan <mr...@gmail.com> wrote:
> Hi,
>
> Over the past few months, I have seen a bunch of pull requests which have
> extended spark api ... most commonly RDD itself.
>
> Most of them are either relatively niche case of specialization (which
> might not be useful for most cases) or idioms which can be expressed
> (sometimes with minor perf penalty) using existing api.
>
> While all of them have non zero value (hence the effort to contribute, and
> gladly welcomed !) they are extending the api in nontrivial ways and have a
> maintenance cost ... and we already have a pending effort to clean up our
> interfaces prior to 1.0
>
> I believe there is a need to keep exposed api succint, expressive and
> functional in spark; while at the same time, encouraging extensions and
> specialization within spark codebase so that other users can benefit from
> the shared contributions.
>
> One approach could be to start something akin to piggybank in pig to
> contribute user generated specializations, helper utils, etc : bundled as
> part of spark, but not part of core itself.
>
> Thoughts, comments ?
>
> Regards,
> Mridul
Re: [DISCUSS] Extending public API
Posted by Ryan LeCompte <le...@gmail.com>.
Perhaps something similar to akka-contrib? Maybe spark-contrib?
https://github.com/akka/akka/tree/master/akka-contrib
On Sat, Feb 22, 2014 at 10:11 PM, Surendranauth Hiraman
<su...@velos.io> wrote:
> +1
>
>
>
> On Sun, Feb 23, 2014 at 1:14 AM, Nan Zhu <zh...@gmail.com> wrote:
>
>> piggybank-like +1
>>
>> --
>> Nan Zhu
>>
>>
>>
>> On Sunday, February 23, 2014 at 1:06 AM, Mridul Muralidharan wrote:
>>
>> > piggybank
>>
>>
>
>
> --
>
> SUREN HIRAMAN, VP TECHNOLOGY
> Velos
> Simple. Powerful. Predictions.
>
> 440 NINTH AVENUE, 11TH FLOOR
> NEW YORK, NY 10001
> O: (917) 525-2466 ext. 105
> F: 646.349.4063
> E: suren.hiraman@v <su...@sociocast.com>elos.io
> W: www.velos.io
Re: [DISCUSS] Extending public API
Posted by Surendranauth Hiraman <su...@velos.io>.
+1
On Sun, Feb 23, 2014 at 1:14 AM, Nan Zhu <zh...@gmail.com> wrote:
> piggybank-like +1
>
> --
> Nan Zhu
>
>
>
> On Sunday, February 23, 2014 at 1:06 AM, Mridul Muralidharan wrote:
>
> > piggybank
>
>
--
SUREN HIRAMAN, VP TECHNOLOGY
Velos
Simple. Powerful. Predictions.
440 NINTH AVENUE, 11TH FLOOR
NEW YORK, NY 10001
O: (917) 525-2466 ext. 105
F: 646.349.4063
E: suren.hiraman@v <su...@sociocast.com>elos.io
W: www.velos.io
Re: [DISCUSS] Extending public API
Posted by Nan Zhu <zh...@gmail.com>.
piggybank-like +1
--
Nan Zhu
On Sunday, February 23, 2014 at 1:06 AM, Mridul Muralidharan wrote:
> piggybank