Avro governance and Avro Enhancement Proposals

[Bruce Mitchener <br...@gmail.com>]

Hello all,

I had some discussions with cutting recently about moving to an AEP (Avro
Enhancement Proposal) process for Avro, similar to the PEP process for
Python (http://www.python.org/dev/peps/).  PEP-0001 (
http://www.python.org/dev/peps/pep-0001/) should give a pretty good (but
longer) description of this process.

In short, major and important changes would be put forward as an AEP.  An
AEP would have an editor who owned the document, worked to build consensus
and shepherds the document to approval.  Approval of an AEP would be done
via a vote of the PMC.

AEPs could be changes to the implemenations or the processes surrounding
Avro development (like how we format source code, how we handle code
reviews, how we perform releases, etc).

AEPs could be managed in the wiki or in Subversion and published to the
website.  I'm interested in hearing what people think in this area.  I'm
somewhat partial to managing them in Subversion as that:

 * Puts the history in one spot.
 * Allows us to update AEPs with the same patch as implementing a new
feature in the code.
 * Keeps things under the appropriate licensing and approval processes that
we have now, especially since anyone can edit the wiki.

As part of this, I think that it makes sense to break some things out of the
specification and out of their current locations on the website:

 * GenAvro IDL should become a new AEP.
 * An AEP should be written to describe container files and that can be
removed from the spec.
 * RPC mechanisms should be AEPs rather than part of the core spec.

As part of this, if each AEP lists which implementations support the given
AEP, it would also help solve the issue being discussed in
https://issues.apache.org/jira/browse/AVRO-358 (specifying levels of the
Avro implementations).

Thoughts?

 - Bruce

Re: Avro governance and Avro Enhancement Proposals

[Scott Carey <sc...@richrelevance.com>]

I largely agree that we are small and lightweight enough to avoid these things for much of what we do.  However, having a heavier-weight process standardized as a 'go-to' for the rare times when those tools aren't good enough is a good idea.

Apache is a "do-ocracy", IMO process like this should not get in the way of contribution, it should facilitate getting big, complicated, or contentious changes right.  I worry that potential contributors will be put off if they get the impression that they can't contribute without an AEP -- whether true or not.

There are times when a process like this is useful.  Once a feature clearly will take a long time to get consensus in JIRA, a PMC member might optionally 'promote' it to an AEP.

An AEP or something like it for Avro 2.0 proposed major changes will likely be necessary.


+1 for defining an AEP for big or contentious changes.  A default 'heavier weight' process as a go-to for some situations will be beneficial IMO.

-0 for using this process as the default way to initiate a proposal.  It should be very clear to potential contributors that this is not necessary, but may become so if the issue is contentious or complicated.  I think starting in JIRA or the mailing list, and escalating if necessary (should be rare) is a good default.  What we do now is working for 95% of what is going on. 
-1 for placing authoritative final documentation in the AEP.  A final spec for a feature once it is going to be released needs to be stand-alone so that users don't have to wade through proposals to figure out how things are supposed to work.  It might be referenced from the AEP or maintained along side it, but needs to be seen as a resource for the feature's users moreso than the resource for those that built the feature -- and if the AEP itself is treated as documentation, the result will tend to be more of the latter.
Essentially, part 4 of an AEP (pronounced 'ape'?) -- specification -- should exist elsewhere and not only in the AEP.  I can be convinced otherwise but I definitely have concerns about dual-pursposing it for end user documentation and developer process.
 
-Scott

On Jun 3, 2010, at 3:07 PM, Philip Zeyliger wrote:

> I'm -0: I actually really like the fact that there is one spec, and that
> it's the source of truth.  Since Python is being brought up as an example,
> I'll mention that my experience is that it's incredibly annoying to run into
> some documentation, and it points you to some proposal, and then you have to
> decipher what was proposed from what was implemented.
> 
> I think we're still small and lightweight enough that JIRA and the mailing
> list are sufficient for circulating most proposals.
> 
> -- Philip
> 
> On Thu, Jun 3, 2010 at 1:46 AM, Bruce Mitchener
> <br...@gmail.com>wrote:
> 
>> Hello all,
>> 
>> I had some discussions with cutting recently about moving to an AEP (Avro
>> Enhancement Proposal) process for Avro, similar to the PEP process for
>> Python (http://www.python.org/dev/peps/).  PEP-0001 (
>> http://www.python.org/dev/peps/pep-0001/) should give a pretty good (but
>> longer) description of this process.
>> 
>> In short, major and important changes would be put forward as an AEP.  An
>> AEP would have an editor who owned the document, worked to build consensus
>> and shepherds the document to approval.  Approval of an AEP would be done
>> via a vote of the PMC.
>> 
>> AEPs could be changes to the implemenations or the processes surrounding
>> Avro development (like how we format source code, how we handle code
>> reviews, how we perform releases, etc).
>> 
>> AEPs could be managed in the wiki or in Subversion and published to the
>> website.  I'm interested in hearing what people think in this area.  I'm
>> somewhat partial to managing them in Subversion as that:
>> 
>> * Puts the history in one spot.
>> * Allows us to update AEPs with the same patch as implementing a new
>> feature in the code.
>> * Keeps things under the appropriate licensing and approval processes that
>> we have now, especially since anyone can edit the wiki.
>> 
>> As part of this, I think that it makes sense to break some things out of
>> the
>> specification and out of their current locations on the website:
>> 
>> * GenAvro IDL should become a new AEP.
>> * An AEP should be written to describe container files and that can be
>> removed from the spec.
>> * RPC mechanisms should be AEPs rather than part of the core spec.
>> 
>> As part of this, if each AEP lists which implementations support the given
>> AEP, it would also help solve the issue being discussed in
>> https://issues.apache.org/jira/browse/AVRO-358 (specifying levels of the
>> Avro implementations).
>> 
>> Thoughts?
>> 
>> - Bruce
>> 

Re: Avro governance and Avro Enhancement Proposals

[Jeff Hammerbacher <ha...@cloudera.com>]

> Whatever the result of this thread is, AEPs or not, it would be nice to
> have
> some way to know what changes are being proposed / discussed / implemented.
>

Hey Bruce,

I'm sorry if it hasn't been pointed out previously, but these discussions
are easy to track down by looking at the JIRA issues tagged with the "spec"
component (ugly URL, easy to navigate to):
https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true&mode=hide&pid=12310911&sorter/order=DESC&sorter/field=priority&resolution=-1&component=12312779
.

We're all excited to have you involved and deeply committed to a rational
specification of Avro; at the current time, however, the contributor and
user base is small enough that working through spec modifications on the
JIRA is deemed appropriate by the majority of the community. I understand
that the C codebase is less nimble than, say, the Python codebase, so you
are perhaps bearing more of the burden from this less formal process for
specification changes; my apologies for that. On the other hand, you are
often not the first to find lurking features in the Avro codebase (e.g.
http://issues.apache.org/jira/browse/AVRO-447 and friends), so for that you
can be thankful.

As the Avro community grows, spec modifications will certainly become more
heavyweight; for now, I hope you'll be able to accomplish your goals for
Avro within this structure.

Thanks,
Jeff

Re: Avro governance and Avro Enhancement Proposals

[Bruce Mitchener <br...@gmail.com>]

Based on the lack of response to this, I'm left to assume that there is no
practical interest in addressing the current issues with revisions to the
specification and formalizing any structure to make these changes easier to
observe.

As such, I'll withdraw this proposal for the time being until there is
sufficient interest and energy to make such a thing worth pursuing.

 - Bruce

On Tue, Jun 8, 2010 at 9:16 AM, Bruce Mitchener
<br...@gmail.com>wrote:

> While cutting had said that there were few revisions to the spec, there are
> at least 6 being discussed in some form right now:
>
>  * 3 that were in the wiki
>  * my binary RPC proposal
>  * hammer just asked for a change to the container file format to support
> checksums
>  * jmhodges (I think) and I have talked about modifying things so that you
> can refer to a type by name before defining it as long as it is defined by
> the end of the schema.
>
> And I'm sure that if I wanted to trawl through JIRA, the lists and various
> IRC discussions, we can find other instances.
>
> That leaves aside things like GenAvro, which people have talked about
> extending but isn't part of the base Avro spec.
>
> Following along with these things requires exposing yourself to the full
> flow of changes (from JIRA) and information (mailing list, IRC) and it isn't
> terribly easy to keep track of what's being discussed, where, and how to
> give comments.
>
> Whatever the result of this thread is, AEPs or not, it would be nice to
> have some way to know what changes are being proposed / discussed /
> implemented.
>
>  - Bruce
>
>
> On Tue, Jun 8, 2010 at 2:51 AM, Matt Massie <ma...@cloudera.com> wrote:
>
>> On Mon, Jun 7, 2010 at 9:43 AM, Doug Cutting <cu...@apache.org> wrote:
>>
>> > On 06/04/2010 06:29 PM, Matt Massie wrote:
>> >
>> >> I would like to see us be a little more of a think-do-acrocy when it
>> comes
>> >> to important, high-level components of Avro (e.g. light-weight RPC).
>>  It
>> >> would be nice to have some simple consensus before code starts getting
>> >> slung
>> >> around;
>> >>
>> >
>> > I'd rather we have proofs of concept before we standardize.  I often
>> find
>> > design issues while implementing.  I like to see two working
>> implementations
>> > before we promote something as an interoperability standard.  Little
>> else of
>> > substance has gone into the spec until it's been implemented twice.
>>
>>
>> Of course we should have proofs of concept and I agree that you can find
>> design issues while implementing ideas.  You can also find design issues
>> by
>> having more than one person review the design before it's implemented.
>>
>> I'm happy to drop my support for AEPs.  The main reason I supported them
>> in
>> the first place is that I felt they would be a good way of fostering
>> communication.  If the team finds no value in them, then we should drop
>> the
>> idea for now.
>>
>>
>> > otherwise, we'll see things in our spec like "The double is
>> >> converted into a 64-bit integer using a method equivalent to Java's
>> >> doubleToLongBits and then encoded in little-endian format."  Not to say
>> we
>> >> would ever do such a thing.  :)
>> >>
>> >
>> > I don't see a fundamental problem with that.  It's a link to clear,
>> > detailed documentation that describes a standard for serializing
>> infinities
>> > and NaNs, using zeroed mantissas.  If you'd rather make the spec more
>> > standalone, then feel free to update that passage with these details.
>>  But
>> > there's nothing Java-specific in it.  This is similar to the link to
>> > Protocol Buffers to define zig-zag encoding.  It's perhaps lazy writing,
>> but
>> > I don't think it's indicative of a technically flawed specification.  Do
>> > you?
>> >
>>
>> I do not think it indicates a technically flawed specification.
>>
>> -Matt
>>
>
>

Re: Avro governance and Avro Enhancement Proposals

[Bruce Mitchener <br...@gmail.com>]

While cutting had said that there were few revisions to the spec, there are
at least 6 being discussed in some form right now:

 * 3 that were in the wiki
 * my binary RPC proposal
 * hammer just asked for a change to the container file format to support
checksums
 * jmhodges (I think) and I have talked about modifying things so that you
can refer to a type by name before defining it as long as it is defined by
the end of the schema.

And I'm sure that if I wanted to trawl through JIRA, the lists and various
IRC discussions, we can find other instances.

That leaves aside things like GenAvro, which people have talked about
extending but isn't part of the base Avro spec.

Following along with these things requires exposing yourself to the full
flow of changes (from JIRA) and information (mailing list, IRC) and it isn't
terribly easy to keep track of what's being discussed, where, and how to
give comments.

Whatever the result of this thread is, AEPs or not, it would be nice to have
some way to know what changes are being proposed / discussed / implemented.

 - Bruce

On Tue, Jun 8, 2010 at 2:51 AM, Matt Massie <ma...@cloudera.com> wrote:

> On Mon, Jun 7, 2010 at 9:43 AM, Doug Cutting <cu...@apache.org> wrote:
>
> > On 06/04/2010 06:29 PM, Matt Massie wrote:
> >
> >> I would like to see us be a little more of a think-do-acrocy when it
> comes
> >> to important, high-level components of Avro (e.g. light-weight RPC).  It
> >> would be nice to have some simple consensus before code starts getting
> >> slung
> >> around;
> >>
> >
> > I'd rather we have proofs of concept before we standardize.  I often find
> > design issues while implementing.  I like to see two working
> implementations
> > before we promote something as an interoperability standard.  Little else
> of
> > substance has gone into the spec until it's been implemented twice.
>
>
> Of course we should have proofs of concept and I agree that you can find
> design issues while implementing ideas.  You can also find design issues by
> having more than one person review the design before it's implemented.
>
> I'm happy to drop my support for AEPs.  The main reason I supported them in
> the first place is that I felt they would be a good way of fostering
> communication.  If the team finds no value in them, then we should drop the
> idea for now.
>
>
> > otherwise, we'll see things in our spec like "The double is
> >> converted into a 64-bit integer using a method equivalent to Java's
> >> doubleToLongBits and then encoded in little-endian format."  Not to say
> we
> >> would ever do such a thing.  :)
> >>
> >
> > I don't see a fundamental problem with that.  It's a link to clear,
> > detailed documentation that describes a standard for serializing
> infinities
> > and NaNs, using zeroed mantissas.  If you'd rather make the spec more
> > standalone, then feel free to update that passage with these details.
>  But
> > there's nothing Java-specific in it.  This is similar to the link to
> > Protocol Buffers to define zig-zag encoding.  It's perhaps lazy writing,
> but
> > I don't think it's indicative of a technically flawed specification.  Do
> > you?
> >
>
> I do not think it indicates a technically flawed specification.
>
> -Matt
>

Re: Avro governance and Avro Enhancement Proposals

[Matt Massie <ma...@cloudera.com>]

On Mon, Jun 7, 2010 at 9:43 AM, Doug Cutting <cu...@apache.org> wrote:

> On 06/04/2010 06:29 PM, Matt Massie wrote:
>
>> I would like to see us be a little more of a think-do-acrocy when it comes
>> to important, high-level components of Avro (e.g. light-weight RPC).  It
>> would be nice to have some simple consensus before code starts getting
>> slung
>> around;
>>
>
> I'd rather we have proofs of concept before we standardize.  I often find
> design issues while implementing.  I like to see two working implementations
> before we promote something as an interoperability standard.  Little else of
> substance has gone into the spec until it's been implemented twice.


Of course we should have proofs of concept and I agree that you can find
design issues while implementing ideas.  You can also find design issues by
having more than one person review the design before it's implemented.

I'm happy to drop my support for AEPs.  The main reason I supported them in
the first place is that I felt they would be a good way of fostering
communication.  If the team finds no value in them, then we should drop the
idea for now.


> otherwise, we'll see things in our spec like "The double is
>> converted into a 64-bit integer using a method equivalent to Java's
>> doubleToLongBits and then encoded in little-endian format."  Not to say we
>> would ever do such a thing.  :)
>>
>
> I don't see a fundamental problem with that.  It's a link to clear,
> detailed documentation that describes a standard for serializing infinities
> and NaNs, using zeroed mantissas.  If you'd rather make the spec more
> standalone, then feel free to update that passage with these details.  But
> there's nothing Java-specific in it.  This is similar to the link to
> Protocol Buffers to define zig-zag encoding.  It's perhaps lazy writing, but
> I don't think it's indicative of a technically flawed specification.  Do
> you?
>

I do not think it indicates a technically flawed specification.

-Matt

Re: Avro governance and Avro Enhancement Proposals

[Doug Cutting <cu...@apache.org>]

On 06/04/2010 06:29 PM, Matt Massie wrote:
> I would like to see us be a little more of a think-do-acrocy when it comes
> to important, high-level components of Avro (e.g. light-weight RPC).  It
> would be nice to have some simple consensus before code starts getting slung
> around;

I'd rather we have proofs of concept before we standardize.  I often 
find design issues while implementing.  I like to see two working 
implementations before we promote something as an interoperability 
standard.  Little else of substance has gone into the spec until it's 
been implemented twice.

> otherwise, we'll see things in our spec like "The double is
> converted into a 64-bit integer using a method equivalent to Java's
> doubleToLongBits and then encoded in little-endian format."  Not to say we
> would ever do such a thing.  :)

I don't see a fundamental problem with that.  It's a link to clear, 
detailed documentation that describes a standard for serializing 
infinities and NaNs, using zeroed mantissas.  If you'd rather make the 
spec more standalone, then feel free to update that passage with these 
details.  But there's nothing Java-specific in it.  This is similar to 
the link to Protocol Buffers to define zig-zag encoding.  It's perhaps 
lazy writing, but I don't think it's indicative of a technically flawed 
specification.  Do you?

Doug

Re: Avro governance and Avro Enhancement Proposals

[Matt Massie <ma...@cloudera.com>]

Re-reading my words, I should have chose them more carefully.  Not to be too
philosophical here, but when I think of "truth", I think of something that
is unchanging/timeless.  I wish I would have said the following paragraph
instead:

"Portions of the spec are evolving at different paces and have different
> levels of maturity.  It would be nice to make it clear which portions of the
> spec are "standard" and which are "evolving".  We could use AEPs as a way of
> making this clearer although we could make it clear without them too."


I agree with Scott that having an AEP process be a barrier to getting things
done would be a very bad thing.  We don't want to create any barriers to
participation.  I also agree that for open-source, and Apache specifically,
is a do-ocracy.  I think part of the issue here is that it's hard to *do*
things with C than other languages so there is a bit of a disadvantage
playing in a do-ocracy.  People don't typically reach for C when they need
to prototype something. :)

I would like to see us be a little more of a think-do-acrocy when it comes
to important, high-level components of Avro (e.g. light-weight RPC).  It
would be nice to have some simple consensus before code starts getting slung
around; otherwise, we'll see things in our spec like "The double is
converted into a 64-bit integer using a method equivalent to Java's
doubleToLongBits and then encoded in little-endian format."  Not to say we
would ever do such a thing.  :)

-Matt


On Thu, Jun 3, 2010 at 5:15 PM, Doug Cutting <cu...@apache.org> wrote:

> On 06/03/2010 04:37 PM, Matt Massie wrote:
>
>> I think it's misleading to see the spec as a single source of truth.
>>  There
>> are portions of the spec which are much more set in stone (e.g. the binary
>> serialization format) than others (e.g. lightweight RPC) are more a work
>> in
>> progress.  It would be good to make that clearer to future implementors
>> and
>> users.  I think AEPs should have a status assigned to them similar to RFCs
>> http://en.wikipedia.org/wiki/Request_for_Comments#Status.
>>
>
> The spec is already versioned.  We could add a status to each section of
> the spec if we like.  But I don't see how status is an argument against a
> having a specification document.
>
> The last incompatible change made to the spec were the file format changes
> in February, prior to any known adoption of the file format. Prior to that,
> RPC was last changed incompatibly in October, also prior to any known
> adoption of RPC.  I don't think we should add things to the spec until we
> are fairly confident they are stable.  And once something is in use, we
> should work hard to never change it incompatibly.
>
> Doug
>

Re: Avro governance and Avro Enhancement Proposals

[Doug Cutting <cu...@apache.org>]

On 06/03/2010 04:37 PM, Matt Massie wrote:
> I think it's misleading to see the spec as a single source of truth.  There
> are portions of the spec which are much more set in stone (e.g. the binary
> serialization format) than others (e.g. lightweight RPC) are more a work in
> progress.  It would be good to make that clearer to future implementors and
> users.  I think AEPs should have a status assigned to them similar to RFCs
> http://en.wikipedia.org/wiki/Request_for_Comments#Status.

The spec is already versioned.  We could add a status to each section of 
the spec if we like.  But I don't see how status is an argument against 
a having a specification document.

The last incompatible change made to the spec were the file format 
changes in February, prior to any known adoption of the file format. 
Prior to that, RPC was last changed incompatibly in October, also prior 
to any known adoption of RPC.  I don't think we should add things to the 
spec until we are fairly confident they are stable.  And once something 
is in use, we should work hard to never change it incompatibly.

Doug

Re: Avro governance and Avro Enhancement Proposals

[Matt Massie <ma...@cloudera.com>]

I'm +1 on having AEPs.  I agree we need to keep them as light-weight as
possible.  We don't want process interfering with writing code.  I like the
idea of the AEPs sitting side-by-side in subversion with the
implementations.  Jira is not a great medium for capturing standards.

We could have a single spec with well defined layers and links to the
relevant AEPs.  Initially, we might just include the AEP text but, as we
grow, we could use links/pointers instead. Even now, we have two
serialization formats (json and binary) and soon multiple RPC mechanisms.

I think it's misleading to see the spec as a single source of truth.  There
are portions of the spec which are much more set in stone (e.g. the binary
serialization format) than others (e.g. lightweight RPC) are more a work in
progress.  It would be good to make that clearer to future implementors and
users.  I think AEPs should have a status assigned to them similar to RFCs
http://en.wikipedia.org/wiki/Request_for_Comments#Status.

-Matt


On Thu, Jun 3, 2010 at 4:16 PM, Jeff Hammerbacher <ha...@cloudera.com>wrote:

> My opinion roughly coincides with Philip. I don't think Avro is large
> enough
> to require AEPs just yet. I think a single spec is great, and clarification
> of levels inside that spec would be greater.
>
> On Thu, Jun 3, 2010 at 3:07 PM, Philip Zeyliger <ph...@cloudera.com>
> wrote:
>
> > I'm -0: I actually really like the fact that there is one spec, and that
> > it's the source of truth.  Since Python is being brought up as an
> example,
> > I'll mention that my experience is that it's incredibly annoying to run
> > into
> > some documentation, and it points you to some proposal, and then you have
> > to
> > decipher what was proposed from what was implemented.
> >
> > I think we're still small and lightweight enough that JIRA and the
> mailing
> > list are sufficient for circulating most proposals.
> >
> > -- Philip
> >
> > On Thu, Jun 3, 2010 at 1:46 AM, Bruce Mitchener
> > <br...@gmail.com>wrote:
> >
> > > Hello all,
> > >
> > > I had some discussions with cutting recently about moving to an AEP
> (Avro
> > > Enhancement Proposal) process for Avro, similar to the PEP process for
> > > Python (http://www.python.org/dev/peps/).  PEP-0001 (
> > > http://www.python.org/dev/peps/pep-0001/) should give a pretty good
> (but
> > > longer) description of this process.
> > >
> > > In short, major and important changes would be put forward as an AEP.
>  An
> > > AEP would have an editor who owned the document, worked to build
> > consensus
> > > and shepherds the document to approval.  Approval of an AEP would be
> done
> > > via a vote of the PMC.
> > >
> > > AEPs could be changes to the implemenations or the processes
> surrounding
> > > Avro development (like how we format source code, how we handle code
> > > reviews, how we perform releases, etc).
> > >
> > > AEPs could be managed in the wiki or in Subversion and published to the
> > > website.  I'm interested in hearing what people think in this area.
>  I'm
> > > somewhat partial to managing them in Subversion as that:
> > >
> > >  * Puts the history in one spot.
> > >  * Allows us to update AEPs with the same patch as implementing a new
> > > feature in the code.
> > >  * Keeps things under the appropriate licensing and approval processes
> > that
> > > we have now, especially since anyone can edit the wiki.
> > >
> > > As part of this, I think that it makes sense to break some things out
> of
> > > the
> > > specification and out of their current locations on the website:
> > >
> > >  * GenAvro IDL should become a new AEP.
> > >  * An AEP should be written to describe container files and that can be
> > > removed from the spec.
> > >  * RPC mechanisms should be AEPs rather than part of the core spec.
> > >
> > > As part of this, if each AEP lists which implementations support the
> > given
> > > AEP, it would also help solve the issue being discussed in
> > > https://issues.apache.org/jira/browse/AVRO-358 (specifying levels of
> the
> > > Avro implementations).
> > >
> > > Thoughts?
> > >
> > >  - Bruce
> > >
> >
>

Re: Avro governance and Avro Enhancement Proposals

[Jeff Hammerbacher <ha...@cloudera.com>]

My opinion roughly coincides with Philip. I don't think Avro is large enough
to require AEPs just yet. I think a single spec is great, and clarification
of levels inside that spec would be greater.

On Thu, Jun 3, 2010 at 3:07 PM, Philip Zeyliger <ph...@cloudera.com> wrote:

> I'm -0: I actually really like the fact that there is one spec, and that
> it's the source of truth.  Since Python is being brought up as an example,
> I'll mention that my experience is that it's incredibly annoying to run
> into
> some documentation, and it points you to some proposal, and then you have
> to
> decipher what was proposed from what was implemented.
>
> I think we're still small and lightweight enough that JIRA and the mailing
> list are sufficient for circulating most proposals.
>
> -- Philip
>
> On Thu, Jun 3, 2010 at 1:46 AM, Bruce Mitchener
> <br...@gmail.com>wrote:
>
> > Hello all,
> >
> > I had some discussions with cutting recently about moving to an AEP (Avro
> > Enhancement Proposal) process for Avro, similar to the PEP process for
> > Python (http://www.python.org/dev/peps/).  PEP-0001 (
> > http://www.python.org/dev/peps/pep-0001/) should give a pretty good (but
> > longer) description of this process.
> >
> > In short, major and important changes would be put forward as an AEP.  An
> > AEP would have an editor who owned the document, worked to build
> consensus
> > and shepherds the document to approval.  Approval of an AEP would be done
> > via a vote of the PMC.
> >
> > AEPs could be changes to the implemenations or the processes surrounding
> > Avro development (like how we format source code, how we handle code
> > reviews, how we perform releases, etc).
> >
> > AEPs could be managed in the wiki or in Subversion and published to the
> > website.  I'm interested in hearing what people think in this area.  I'm
> > somewhat partial to managing them in Subversion as that:
> >
> >  * Puts the history in one spot.
> >  * Allows us to update AEPs with the same patch as implementing a new
> > feature in the code.
> >  * Keeps things under the appropriate licensing and approval processes
> that
> > we have now, especially since anyone can edit the wiki.
> >
> > As part of this, I think that it makes sense to break some things out of
> > the
> > specification and out of their current locations on the website:
> >
> >  * GenAvro IDL should become a new AEP.
> >  * An AEP should be written to describe container files and that can be
> > removed from the spec.
> >  * RPC mechanisms should be AEPs rather than part of the core spec.
> >
> > As part of this, if each AEP lists which implementations support the
> given
> > AEP, it would also help solve the issue being discussed in
> > https://issues.apache.org/jira/browse/AVRO-358 (specifying levels of the
> > Avro implementations).
> >
> > Thoughts?
> >
> >  - Bruce
> >
>

Re: Avro governance and Avro Enhancement Proposals

[Philip Zeyliger <ph...@cloudera.com>]

I'm -0: I actually really like the fact that there is one spec, and that
it's the source of truth.  Since Python is being brought up as an example,
I'll mention that my experience is that it's incredibly annoying to run into
some documentation, and it points you to some proposal, and then you have to
decipher what was proposed from what was implemented.

I think we're still small and lightweight enough that JIRA and the mailing
list are sufficient for circulating most proposals.

-- Philip

On Thu, Jun 3, 2010 at 1:46 AM, Bruce Mitchener
<br...@gmail.com>wrote:

> Hello all,
>
> I had some discussions with cutting recently about moving to an AEP (Avro
> Enhancement Proposal) process for Avro, similar to the PEP process for
> Python (http://www.python.org/dev/peps/).  PEP-0001 (
> http://www.python.org/dev/peps/pep-0001/) should give a pretty good (but
> longer) description of this process.
>
> In short, major and important changes would be put forward as an AEP.  An
> AEP would have an editor who owned the document, worked to build consensus
> and shepherds the document to approval.  Approval of an AEP would be done
> via a vote of the PMC.
>
> AEPs could be changes to the implemenations or the processes surrounding
> Avro development (like how we format source code, how we handle code
> reviews, how we perform releases, etc).
>
> AEPs could be managed in the wiki or in Subversion and published to the
> website.  I'm interested in hearing what people think in this area.  I'm
> somewhat partial to managing them in Subversion as that:
>
>  * Puts the history in one spot.
>  * Allows us to update AEPs with the same patch as implementing a new
> feature in the code.
>  * Keeps things under the appropriate licensing and approval processes that
> we have now, especially since anyone can edit the wiki.
>
> As part of this, I think that it makes sense to break some things out of
> the
> specification and out of their current locations on the website:
>
>  * GenAvro IDL should become a new AEP.
>  * An AEP should be written to describe container files and that can be
> removed from the spec.
>  * RPC mechanisms should be AEPs rather than part of the core spec.
>
> As part of this, if each AEP lists which implementations support the given
> AEP, it would also help solve the issue being discussed in
> https://issues.apache.org/jira/browse/AVRO-358 (specifying levels of the
> Avro implementations).
>
> Thoughts?
>
>  - Bruce
>