Guideline for contributing new extensions

[Luciano Resende <lu...@gmail.com>]

I have drafted a very basic guideline for contributing new extensions for
Apache Bahir. My main goal is to use that as the start point to a community
discussion that can flush out some details and make the process for
proposing, accepting and retiring a extension very transparent.

Please review the initial draft at :
http://bahir.apache.org/contributing-extensions/

And let's use this thread for any question, comments, feedback, etc.

-- 
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/

Re: Guideline for contributing new extensions

[Marcelo Vanzin <va...@apache.org>]

On Wed, Jun 1, 2016 at 10:34 AM, Luciano Resende <lu...@gmail.com> wrote:
> This brings a good point, but still, i think for management purposes, one
> repo might make things easier, and we could easily enable this by providing
> minor releases between runtime releases (e.g. bahir 2.0.0.1 when kafka 0.11
> comes up but spark hasn't released 2.0.1 yet)

I was envisioning something closer to Apache Commons (with a
commons-parent and each component with its own repo), but I'm fine
with starting up with your suggestion and adapting as needed.

Re: Guideline for contributing new extensions

[Luciano Resende <lu...@gmail.com>]

On Tue, May 31, 2016 at 6:51 PM, Marcelo Vanzin <va...@apache.org> wrote:

> The document seems to imply that all source will live in the same
> repository. Is that the goal?
>
>
Because the main goal is to make these extensions to follow their "runtime"
release and because we might end up having several extensions for a given
"runtime", I was thinking on having one repository per "runtime" (e.g.
Spark) which would make it easier to manage and release at the same time
Spark releases for example.


> My preference would be to have separate extensions live in separate
> repos; that way, they can have their different release cycles that
> might align better with the systems they're actually dependent on
> (e.g. a Kafka extension releasing right after a new major Kafka
> version). And one extension being unstable doesn't affect development
> for other extensions.
>
>
This brings a good point, but still, i think for management purposes, one
repo might make things easier, and we could easily enable this by providing
minor releases between runtime releases (e.g. bahir 2.0.0.1 when kafka 0.11
comes up but spark hasn't released 2.0.1 yet)

Would that be ok for starting ? And I am sure we can re-evaluate moving to
a repo per extension in the future.

Thoughts ?

-- 
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/

Re: Guideline for contributing new extensions

[Marcelo Vanzin <va...@apache.org>]

The document seems to imply that all source will live in the same
repository. Is that the goal?

My preference would be to have separate extensions live in separate
repos; that way, they can have their different release cycles that
might align better with the systems they're actually dependent on
(e.g. a Kafka extension releasing right after a new major Kafka
version). And one extension being unstable doesn't affect development
for other extensions.


On Tue, May 31, 2016 at 4:40 PM, Luciano Resende <lu...@gmail.com> wrote:
> I have drafted a very basic guideline for contributing new extensions for
> Apache Bahir. My main goal is to use that as the start point to a community
> discussion that can flush out some details and make the process for
> proposing, accepting and retiring a extension very transparent.
>
> Please review the initial draft at :
> http://bahir.apache.org/contributing-extensions/
>
> And let's use this thread for any question, comments, feedback, etc.
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/

Re: Guideline for contributing new extensions

[Luciano Resende <lu...@gmail.com>]

On Tue, May 31, 2016 at 10:43 PM, Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:

> Hi Luciano,
>
> it looks good to me. The challenge could be when accepting extensions for
> different projects (Spark, Flink, Beam for instance), not necessary sharing
> the same dependencies/technologies. We will have to provide kind of BoM for
> the dependencies.
>
> Regards
> JB


Hopefully this will be a good problem to have, it will be definetlly good
to have more "runtime" support from Flink, Beam, etc


-- 
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/

Re: Guideline for contributing new extensions

[Luciano Resende <lu...@gmail.com>]

On Wed, Jun 1, 2016 at 3:16 AM, Steve Loughran <st...@apache.org> wrote:

> On 1 June 2016 at 06:43, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>
> > Hi Luciano,
> >
> > it looks good to me. The challenge could be when accepting extensions for
> > different projects (Spark, Flink, Beam for instance), not necessary
> sharing
> > the same dependencies/technologies. We will have to provide kind of BoM
> for
> > the dependencies.
> >
>
> ooh, have we got that in scope. nice. I'll be at berlin buzzwords next week
> and talk to the flink team, though I think their stage of project is likely
> to have this plugin problem.
>
>
+1


>
> Something else to consider: use of package names from other projects in
> plugins. I'm thinking generally of spark here, where you end up having to
> put things into org.apache.spark or packages below in order to get at
> private[spark] stuff. Simiarly, spark itself puts some stuff into
> org.apache.hive
>
>
Good point, what do we want to do for java package naming, I will bring
this up on the code import thread.


> While this is currently necessary, it's dangerous as it creates brittle
> code —and when it does break, you can't really expect the upstream project
> to care that much. Really we should be advocating making stuff (spark:
> management, history) public in some form, even if there is an explicit
> compatibility statement around plugins saying "you are expected to rebuild
> this even for point releases". That's something to discuss with the
> projects, with the arguments in favour being: things are less likely to
> break, and when they do break, the stack traces won't have your package
> names at the top.
>
>
Yes, and hopefully the spark committers that are here would help.


> The cost: more obligation to expose code with stability guarantees.
>
> FWIW, I've started a discussion on yarn-dev about the semantics of @Stable
> classes there, as SLIDER-1130 showed that it didn't meet my expectations.
> We may end up with some attribute to indicate "stable to use, evolving to
> extend", and one for @Experimental —may go away without any warning.
>
>
> >
> > Regards
> > JB
> >
> >
> > On 06/01/2016 01:40 AM, Luciano Resende wrote:
> >
> >> I have drafted a very basic guideline for contributing new extensions
> for
> >> Apache Bahir. My main goal is to use that as the start point to a
> >> community
> >> discussion that can flush out some details and make the process for
> >> proposing, accepting and retiring a extension very transparent.
> >>
> >> Please review the initial draft at :
> >> http://bahir.apache.org/contributing-extensions/
> >>
> >> And let's use this thread for any question, comments, feedback, etc.
> >>
> >>
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
>



-- 
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/

Re: Guideline for contributing new extensions

[Steve Loughran <st...@apache.org>]

On 1 June 2016 at 06:43, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:

> Hi Luciano,
>
> it looks good to me. The challenge could be when accepting extensions for
> different projects (Spark, Flink, Beam for instance), not necessary sharing
> the same dependencies/technologies. We will have to provide kind of BoM for
> the dependencies.
>

ooh, have we got that in scope. nice. I'll be at berlin buzzwords next week
and talk to the flink team, though I think their stage of project is likely
to have this plugin problem.


Something else to consider: use of package names from other projects in
plugins. I'm thinking generally of spark here, where you end up having to
put things into org.apache.spark or packages below in order to get at
private[spark] stuff. Simiarly, spark itself puts some stuff into
org.apache.hive

While this is currently necessary, it's dangerous as it creates brittle
code —and when it does break, you can't really expect the upstream project
to care that much. Really we should be advocating making stuff (spark:
management, history) public in some form, even if there is an explicit
compatibility statement around plugins saying "you are expected to rebuild
this even for point releases". That's something to discuss with the
projects, with the arguments in favour being: things are less likely to
break, and when they do break, the stack traces won't have your package
names at the top.

The cost: more obligation to expose code with stability guarantees.

FWIW, I've started a discussion on yarn-dev about the semantics of @Stable
classes there, as SLIDER-1130 showed that it didn't meet my expectations.
We may end up with some attribute to indicate "stable to use, evolving to
extend", and one for @Experimental —may go away without any warning.


>
> Regards
> JB
>
>
> On 06/01/2016 01:40 AM, Luciano Resende wrote:
>
>> I have drafted a very basic guideline for contributing new extensions for
>> Apache Bahir. My main goal is to use that as the start point to a
>> community
>> discussion that can flush out some details and make the process for
>> proposing, accepting and retiring a extension very transparent.
>>
>> Please review the initial draft at :
>> http://bahir.apache.org/contributing-extensions/
>>
>> And let's use this thread for any question, comments, feedback, etc.
>>
>>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

Re: Guideline for contributing new extensions

[Jean-Baptiste Onofré <jb...@nanthrax.net>]

Hi Luciano,

it looks good to me. The challenge could be when accepting extensions 
for different projects (Spark, Flink, Beam for instance), not necessary 
sharing the same dependencies/technologies. We will have to provide kind 
of BoM for the dependencies.

Regards
JB

On 06/01/2016 01:40 AM, Luciano Resende wrote:
> I have drafted a very basic guideline for contributing new extensions for
> Apache Bahir. My main goal is to use that as the start point to a community
> discussion that can flush out some details and make the process for
> proposing, accepting and retiring a extension very transparent.
>
> Please review the initial draft at :
> http://bahir.apache.org/contributing-extensions/
>
> And let's use this thread for any question, comments, feedback, etc.
>

-- 
Jean-Baptiste Onofr�
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com