You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@accumulo.apache.org by Christopher <ct...@apache.org> on 2013/06/17 19:07:28 UTC
Backporting policy proposal
I propose we adopt a more structured policy beyond simple "lazy
consensus" to be apply to backporting features. Some guidelines I'd
like to see in this policy, include:
1. Back-porting bugfixes to a prior release line that is not EOL
(end-of-life) is always okay (subject to normal lazy consensus), but
it is strongly preferred to fix it first in the older branch and merge
forward to the newer one(s).
2. Back-porting performance improvements to a prior release line that
is not EOL (end-of-life) is usually okay (subject to normal lazy
consensus), so long as it does not change user-facing behavior or API.
It is still strongly preferred to add such fixes in the older branch
first, and merge forward to the newer one(s).
3. Back-porting new features and additions are to be avoided as a
general rule (see arguments for this in previous threads:
ACCUMULO-1488 and http://s.apache.org/sU5 and probably others).
4. If it is desired to back-port a new feature, then a vote on the
developer mailing list should be called, due to the additional
development and support burden the new feature may cause for all
developers.
5. Even when it is agreed that a feature should be back-ported, it
should not be done unless/until a feature is first represented in a
newer release that has gone through the testing and release process,
and can be considered stable enough to back-port. This ensures focus
is kept on the main development branch for new features, and
significantly reduces the development burden of back-porting. It also
gives us a clear idea of the target behavior for the back-ported
feature, so that it will behave in the same way as the same feature in
the later release line.
Please discuss these points, or add your own.
--
Christopher L Tubbs II
http://gravatar.com/ctubbsii
Re: Backporting policy proposal
Posted by John Vines <vi...@apache.org>.
There are flow issues though, especially when you get into the realm of
rebasing, etc. So, like SVN, there should be one established direction.
On Tue, Jun 18, 2013 at 11:26 AM, Adam Fuchs <af...@apache.org> wrote:
> On Jun 18, 2013 11:05 AM, "Josh Elser" <jo...@gmail.com> wrote:
> >
> > On 6/18/13 10:53 AM, Adam Fuchs wrote:
> >>>
> >>> >
> >>> >2. Back-porting performance improvements to a prior release line that
> >>> >is not EOL (end-of-life) is usually okay (subject to normal lazy
> >>> >consensus), so long as it does not change user-facing behavior or API.
> >>> >It is still strongly preferred to add such fixes in the older branch
> >>> >first, and merge forward to the newer one(s).
> >>
> >> Agree, although doesn't the transition to git alleviate the problems
> with
> >> order of operations?
> >>
> >
> > I don't understand what you mean by "order of operations". If you push a
> commit in 1.6.0-SNAPSHOT that should really be in 1.4.4-SNAPSHOT, Git will
> not handle this well in terms of an accurate history.
>
> I don't really understand either -- that was mostly speculation. Doesn't
> the reliance on hashes of the patch rather than commit IDs make it
> immaterial as to which branch a patch was committed to first? Admitedly, I
> am not an expert on git.
>
> Adam
>
Re: Backporting policy proposal
Posted by Josh Elser <jo...@gmail.com>.
On 6/18/13 11:26 AM, Adam Fuchs wrote:
> On Jun 18, 2013 11:05 AM, "Josh Elser" <jo...@gmail.com> wrote:
>>
>> On 6/18/13 10:53 AM, Adam Fuchs wrote:
>>>>
>>>>>
>>>>> 2. Back-porting performance improvements to a prior release line that
>>>>> is not EOL (end-of-life) is usually okay (subject to normal lazy
>>>>> consensus), so long as it does not change user-facing behavior or API.
>>>>> It is still strongly preferred to add such fixes in the older branch
>>>>> first, and merge forward to the newer one(s).
>>>
>>> Agree, although doesn't the transition to git alleviate the problems with
>>> order of operations?
>>>
>>
>> I don't understand what you mean by "order of operations". If you push a
> commit in 1.6.0-SNAPSHOT that should really be in 1.4.4-SNAPSHOT, Git will
> not handle this well in terms of an accurate history.
>
> I don't really understand either -- that was mostly speculation. Doesn't
> the reliance on hashes of the patch rather than commit IDs make it
> immaterial as to which branch a patch was committed to first? Admitedly, I
> am not an expert on git.
>
> Adam
>
No, that's incorrect. A commit is uniquely identified by a set of
attributes which, most importantly, include the parent commit. As such,
what you would have typically (happily) done with subversion (`svn merge
-r 12345 .`) has implications to the Git history graph.
The only way to accomplish this in Git is to cherry-pick that commit
which causes multiple unique commits which apply the same changes to
content. This also makes using tools like `git-bisect` much less
effective as a bug introduced in a cherry-pick'ed commit now has
multiple points of introduction instead of a single commit.
In the downtime before we hear back from INFRA, it would be good to read
through the documentation: http://git-scm.com/documentation
Re: Backporting policy proposal
Posted by Adam Fuchs <af...@apache.org>.
On Jun 18, 2013 11:05 AM, "Josh Elser" <jo...@gmail.com> wrote:
>
> On 6/18/13 10:53 AM, Adam Fuchs wrote:
>>>
>>> >
>>> >2. Back-porting performance improvements to a prior release line that
>>> >is not EOL (end-of-life) is usually okay (subject to normal lazy
>>> >consensus), so long as it does not change user-facing behavior or API.
>>> >It is still strongly preferred to add such fixes in the older branch
>>> >first, and merge forward to the newer one(s).
>>
>> Agree, although doesn't the transition to git alleviate the problems with
>> order of operations?
>>
>
> I don't understand what you mean by "order of operations". If you push a
commit in 1.6.0-SNAPSHOT that should really be in 1.4.4-SNAPSHOT, Git will
not handle this well in terms of an accurate history.
I don't really understand either -- that was mostly speculation. Doesn't
the reliance on hashes of the patch rather than commit IDs make it
immaterial as to which branch a patch was committed to first? Admitedly, I
am not an expert on git.
Adam
Re: Backporting policy proposal
Posted by Josh Elser <jo...@gmail.com>.
On 6/18/13 10:53 AM, Adam Fuchs wrote:
>> >
>> >2. Back-porting performance improvements to a prior release line that
>> >is not EOL (end-of-life) is usually okay (subject to normal lazy
>> >consensus), so long as it does not change user-facing behavior or API.
>> >It is still strongly preferred to add such fixes in the older branch
>> >first, and merge forward to the newer one(s).
> Agree, although doesn't the transition to git alleviate the problems with
> order of operations?
>
I don't understand what you mean by "order of operations". If you push a
commit in 1.6.0-SNAPSHOT that should really be in 1.4.4-SNAPSHOT, Git
will not handle this well in terms of an accurate history.
Re: Backporting policy proposal
Posted by Mike Drob <md...@mdrob.com>.
I think the point about CTR is very important, and should be weighed
carefully before making binding decisions regarding back-ports. If a
developer wants to spend time back-porting features, then I don't want to
be the one to dissuade them. Like I've heard on this list many times,
"patches welcome!"
CTR would, on the face of things, allow developers to back-port features as
they feel is appropriate. Then, if another developer has concerns with the
feature, the back-port, the implementation, or anything else, he/she would
be encouraged to start a vote on it. It might be a nice courtesy to other
developers if the intent to back-port was stated up front, but I don't
expect it to be strictly necessary.
I don't know if there are implied time-limits on reversion votes for CTR,
but that can be addressed in by-laws when the community gets around to it.
Re: Backporting policy proposal
Posted by Keith Turner <ke...@deenlo.com>.
On Tue, Jun 18, 2013 at 1:55 PM, Christopher <ct...@apache.org> wrote:
> Dealing with contentious changes might be a good addition to an
> established set of by-laws, and would certainly help alleviate some
> potential issues. However, I think we also need to establish
> guidelines for back-porting specifically, because these have the
> potential to be contentious *every* time.
>
I am not sure every back port would be contentious, it really depends on
the specifics. But even if every one were contentious, I think things
would still work out nicely in the long run. I suspect if a few
contentious changes went through reversion votes, that memes related to
back porting would be influenced by this force of natural selection.
Decisive revision votes would set precedent and influence future decisions
on back porting.
>
> --
> Christopher L Tubbs II
> http://gravatar.com/ctubbsii
>
>
> On Tue, Jun 18, 2013 at 12:05 PM, Keith Turner <ke...@deenlo.com> wrote:
> > On Tue, Jun 18, 2013 at 10:53 AM, Adam Fuchs <af...@apache.org> wrote:
> >
> >> On Jun 17, 2013 1:07 PM, "Christopher" <ct...@apache.org> wrote:
> >> >
> >> > I propose we adopt a more structured policy beyond simple "lazy
> >> > consensus" to be apply to backporting features. Some guidelines I'd
> >> > like to see in this policy, include:
> >> >
> >> > 1. Back-porting bugfixes to a prior release line that is not EOL
> >> > (end-of-life) is always okay (subject to normal lazy consensus), but
> >> > it is strongly preferred to fix it first in the older branch and merge
> >> > forward to the newer one(s).
> >>
> >> Agree.
> >>
> >> >
> >> > 2. Back-porting performance improvements to a prior release line that
> >> > is not EOL (end-of-life) is usually okay (subject to normal lazy
> >> > consensus), so long as it does not change user-facing behavior or API.
> >> > It is still strongly preferred to add such fixes in the older branch
> >> > first, and merge forward to the newer one(s).
> >>
> >> Agree, although doesn't the transition to git alleviate the problems
> with
> >> order of operations?
> >>
> >> >
> >> > 3. Back-porting new features and additions are to be avoided as a
> >> > general rule (see arguments for this in previous threads:
> >> > ACCUMULO-1488 and http://s.apache.org/sU5 and probably others).
> >>
> >> This is an overly generalized and contentious statement that strips our
> >> committers of authority. There is no reason to state this given 4 and 5
> >> below. If we want to say something like this then we should narrow it to
> >> the non contentious classes of features, even loosly defined by
> >> thresholding the costs of maintenance, documentation, API effects, etc.
> >>
> >> >
> >> > 4. If it is desired to back-port a new feature, then a vote on the
> >> > developer mailing list should be called, due to the additional
> >> > development and support burden the new feature may cause for all
> >> > developers.
> >>
> >> Again, we should have a threshold here so that less than one hour of
> code
> >> change doesn't require three days of vote tracking. While additional
> >> attention should be payed to the cost of backporting features, it is
> >> important for us to trust and enable our developers to make good
> decisions.
> >>
> >
> > Another avenue is to define a clear process dealing with contentious
> > changes. Consider the case where a developer backports a feature and
> other
> > developers think its deficient in some way. If the developer who back
> > ported will not fix the issues, then other developers are faced with the
> > prospect of fixing the issues, ignoring them and letting users deal with
> > it, or reverting the change. If no one steps up to fix it and some devs
> > are opposed to reverting it, I think we should be able to call a vote on
> > reverting the change. This process would be more inline with our
> current
> > CTR workflow. I think this approach would result in less overhead.
> >
> >
> >> >
> >> > 5. Even when it is agreed that a feature should be back-ported, it
> >> > should not be done unless/until a feature is first represented in a
> >> > newer release that has gone through the testing and release process,
> >> > and can be considered stable enough to back-port. This ensures focus
> >> > is kept on the main development branch for new features, and
> >> > significantly reduces the development burden of back-porting. It also
> >> > gives us a clear idea of the target behavior for the back-ported
> >> > feature, so that it will behave in the same way as the same feature in
> >> > the later release line.
> >>
> >> While the suggested rule would have the desire effect as listed, this is
> >> too broad a rule that eliminates much of the quick reation time that is
> >> really the whole point of backporting. If developers understand and
> agree
> >> on the costs of backporting, then do we really need a rule like this?
> >>
> >> Adam
> >>
>
Re: Backporting policy proposal
Posted by Christopher <ct...@apache.org>.
Dealing with contentious changes might be a good addition to an
established set of by-laws, and would certainly help alleviate some
potential issues. However, I think we also need to establish
guidelines for back-porting specifically, because these have the
potential to be contentious *every* time.
--
Christopher L Tubbs II
http://gravatar.com/ctubbsii
On Tue, Jun 18, 2013 at 12:05 PM, Keith Turner <ke...@deenlo.com> wrote:
> On Tue, Jun 18, 2013 at 10:53 AM, Adam Fuchs <af...@apache.org> wrote:
>
>> On Jun 17, 2013 1:07 PM, "Christopher" <ct...@apache.org> wrote:
>> >
>> > I propose we adopt a more structured policy beyond simple "lazy
>> > consensus" to be apply to backporting features. Some guidelines I'd
>> > like to see in this policy, include:
>> >
>> > 1. Back-porting bugfixes to a prior release line that is not EOL
>> > (end-of-life) is always okay (subject to normal lazy consensus), but
>> > it is strongly preferred to fix it first in the older branch and merge
>> > forward to the newer one(s).
>>
>> Agree.
>>
>> >
>> > 2. Back-porting performance improvements to a prior release line that
>> > is not EOL (end-of-life) is usually okay (subject to normal lazy
>> > consensus), so long as it does not change user-facing behavior or API.
>> > It is still strongly preferred to add such fixes in the older branch
>> > first, and merge forward to the newer one(s).
>>
>> Agree, although doesn't the transition to git alleviate the problems with
>> order of operations?
>>
>> >
>> > 3. Back-porting new features and additions are to be avoided as a
>> > general rule (see arguments for this in previous threads:
>> > ACCUMULO-1488 and http://s.apache.org/sU5 and probably others).
>>
>> This is an overly generalized and contentious statement that strips our
>> committers of authority. There is no reason to state this given 4 and 5
>> below. If we want to say something like this then we should narrow it to
>> the non contentious classes of features, even loosly defined by
>> thresholding the costs of maintenance, documentation, API effects, etc.
>>
>> >
>> > 4. If it is desired to back-port a new feature, then a vote on the
>> > developer mailing list should be called, due to the additional
>> > development and support burden the new feature may cause for all
>> > developers.
>>
>> Again, we should have a threshold here so that less than one hour of code
>> change doesn't require three days of vote tracking. While additional
>> attention should be payed to the cost of backporting features, it is
>> important for us to trust and enable our developers to make good decisions.
>>
>
> Another avenue is to define a clear process dealing with contentious
> changes. Consider the case where a developer backports a feature and other
> developers think its deficient in some way. If the developer who back
> ported will not fix the issues, then other developers are faced with the
> prospect of fixing the issues, ignoring them and letting users deal with
> it, or reverting the change. If no one steps up to fix it and some devs
> are opposed to reverting it, I think we should be able to call a vote on
> reverting the change. This process would be more inline with our current
> CTR workflow. I think this approach would result in less overhead.
>
>
>> >
>> > 5. Even when it is agreed that a feature should be back-ported, it
>> > should not be done unless/until a feature is first represented in a
>> > newer release that has gone through the testing and release process,
>> > and can be considered stable enough to back-port. This ensures focus
>> > is kept on the main development branch for new features, and
>> > significantly reduces the development burden of back-porting. It also
>> > gives us a clear idea of the target behavior for the back-ported
>> > feature, so that it will behave in the same way as the same feature in
>> > the later release line.
>>
>> While the suggested rule would have the desire effect as listed, this is
>> too broad a rule that eliminates much of the quick reation time that is
>> really the whole point of backporting. If developers understand and agree
>> on the costs of backporting, then do we really need a rule like this?
>>
>> Adam
>>
Re: Backporting policy proposal
Posted by Keith Turner <ke...@deenlo.com>.
On Tue, Jun 18, 2013 at 10:53 AM, Adam Fuchs <af...@apache.org> wrote:
> On Jun 17, 2013 1:07 PM, "Christopher" <ct...@apache.org> wrote:
> >
> > I propose we adopt a more structured policy beyond simple "lazy
> > consensus" to be apply to backporting features. Some guidelines I'd
> > like to see in this policy, include:
> >
> > 1. Back-porting bugfixes to a prior release line that is not EOL
> > (end-of-life) is always okay (subject to normal lazy consensus), but
> > it is strongly preferred to fix it first in the older branch and merge
> > forward to the newer one(s).
>
> Agree.
>
> >
> > 2. Back-porting performance improvements to a prior release line that
> > is not EOL (end-of-life) is usually okay (subject to normal lazy
> > consensus), so long as it does not change user-facing behavior or API.
> > It is still strongly preferred to add such fixes in the older branch
> > first, and merge forward to the newer one(s).
>
> Agree, although doesn't the transition to git alleviate the problems with
> order of operations?
>
> >
> > 3. Back-porting new features and additions are to be avoided as a
> > general rule (see arguments for this in previous threads:
> > ACCUMULO-1488 and http://s.apache.org/sU5 and probably others).
>
> This is an overly generalized and contentious statement that strips our
> committers of authority. There is no reason to state this given 4 and 5
> below. If we want to say something like this then we should narrow it to
> the non contentious classes of features, even loosly defined by
> thresholding the costs of maintenance, documentation, API effects, etc.
>
> >
> > 4. If it is desired to back-port a new feature, then a vote on the
> > developer mailing list should be called, due to the additional
> > development and support burden the new feature may cause for all
> > developers.
>
> Again, we should have a threshold here so that less than one hour of code
> change doesn't require three days of vote tracking. While additional
> attention should be payed to the cost of backporting features, it is
> important for us to trust and enable our developers to make good decisions.
>
Another avenue is to define a clear process dealing with contentious
changes. Consider the case where a developer backports a feature and other
developers think its deficient in some way. If the developer who back
ported will not fix the issues, then other developers are faced with the
prospect of fixing the issues, ignoring them and letting users deal with
it, or reverting the change. If no one steps up to fix it and some devs
are opposed to reverting it, I think we should be able to call a vote on
reverting the change. This process would be more inline with our current
CTR workflow. I think this approach would result in less overhead.
> >
> > 5. Even when it is agreed that a feature should be back-ported, it
> > should not be done unless/until a feature is first represented in a
> > newer release that has gone through the testing and release process,
> > and can be considered stable enough to back-port. This ensures focus
> > is kept on the main development branch for new features, and
> > significantly reduces the development burden of back-porting. It also
> > gives us a clear idea of the target behavior for the back-ported
> > feature, so that it will behave in the same way as the same feature in
> > the later release line.
>
> While the suggested rule would have the desire effect as listed, this is
> too broad a rule that eliminates much of the quick reation time that is
> really the whole point of backporting. If developers understand and agree
> on the costs of backporting, then do we really need a rule like this?
>
> Adam
>
Re: Backporting policy proposal
Posted by Christopher <ct...@apache.org>.
On Tue, Jun 18, 2013 at 10:53 AM, Adam Fuchs <af...@apache.org> wrote:
> On Jun 17, 2013 1:07 PM, "Christopher" <ct...@apache.org> wrote:
[snip agreement]
>> 2. Back-porting performance improvements to a prior release line that
>> is not EOL (end-of-life) is usually okay (subject to normal lazy
>> consensus), so long as it does not change user-facing behavior or API.
>> It is still strongly preferred to add such fixes in the older branch
>> first, and merge forward to the newer one(s).
>
> Agree, although doesn't the transition to git alleviate the problems with
> order of operations?
No, it complicates the goal of having newer versions share the entire
history of older versions in the workflow we've discussed for git.
(See Josh's comment about cherry-picking).
>> 3. Back-porting new features and additions are to be avoided as a
>> general rule (see arguments for this in previous threads:
>> ACCUMULO-1488 and http://s.apache.org/sU5 and probably others).
>
> This is an overly generalized and contentious statement that strips our
> committers of authority. There is no reason to state this given 4 and 5
> below. If we want to say something like this then we should narrow it to
> the non contentious classes of features, even loosly defined by
> thresholding the costs of maintenance, documentation, API effects, etc.
Yes, it is purposefully generalized (though I don't understand how it
is contentious... with what does it contend?), so that exceptions to
this rule can be objectively identified. It's a proposal, to be
considered in the context of #4 and #5. Establishing it as a general
guideline gives #4 and #5 meaning and scope.
It doesn't strip anybody of any authority. Committers will continue to
share authority with their other committer peers, as before. The only
change is how the authority is exercised (with active coordination
with other developers instead of passive). This just enforces engaging
with other developers on issues that have potential to disrupt the
efficiency of the development process.
>> 4. If it is desired to back-port a new feature, then a vote on the
>> developer mailing list should be called, due to the additional
>> development and support burden the new feature may cause for all
>> developers.
>
> Again, we should have a threshold here so that less than one hour of code
> change doesn't require three days of vote tracking. While additional
> attention should be payed to the cost of backporting features, it is
> important for us to trust and enable our developers to make good decisions.
What kind of threshold would you propose? And how do you objectively
(or even subjectively, with guidelines) measure the maintenance costs,
and the testing costs, the documentation costs, and the consistency
checks for compatibility during a release process? If we wanted to
rely solely on committer judgement, we could strip this item from the
proposal, and rely on #3 alone, as a general rule. However, then we're
going to have a lot of contention and split consensus whenever this
comes up, instead of simply follow a set of rules we've agreed to
follow in advance to avoid those issues.
>> 5. Even when it is agreed that a feature should be back-ported, it
>> should not be done unless/until a feature is first represented in a
>> newer release that has gone through the testing and release process,
>> and can be considered stable enough to back-port. This ensures focus
>> is kept on the main development branch for new features, and
>> significantly reduces the development burden of back-porting. It also
>> gives us a clear idea of the target behavior for the back-ported
>> feature, so that it will behave in the same way as the same feature in
>> the later release line.
>
> While the suggested rule would have the desire effect as listed, this is
> too broad a rule that eliminates much of the quick reation time that is
> really the whole point of backporting. If developers understand and agree
> on the costs of backporting, then do we really need a rule like this?
What happened with the proxy in 1.5, backported to 1.4 (Keith's
previously expressed frustration), seems to suggest that we do need
such a rule. But, this rule can be rolled into #4, in the sense that
it can be a consideration for each individual's vote. I do not see a
need to establish this as a strict "no exceptions" rule... just a
general guideline for consideration.
--
Christopher L Tubbs II
http://gravatar.com/ctubbsii
Re: Backporting policy proposal
Posted by Adam Fuchs <af...@apache.org>.
On Jun 17, 2013 1:07 PM, "Christopher" <ct...@apache.org> wrote:
>
> I propose we adopt a more structured policy beyond simple "lazy
> consensus" to be apply to backporting features. Some guidelines I'd
> like to see in this policy, include:
>
> 1. Back-porting bugfixes to a prior release line that is not EOL
> (end-of-life) is always okay (subject to normal lazy consensus), but
> it is strongly preferred to fix it first in the older branch and merge
> forward to the newer one(s).
Agree.
>
> 2. Back-porting performance improvements to a prior release line that
> is not EOL (end-of-life) is usually okay (subject to normal lazy
> consensus), so long as it does not change user-facing behavior or API.
> It is still strongly preferred to add such fixes in the older branch
> first, and merge forward to the newer one(s).
Agree, although doesn't the transition to git alleviate the problems with
order of operations?
>
> 3. Back-porting new features and additions are to be avoided as a
> general rule (see arguments for this in previous threads:
> ACCUMULO-1488 and http://s.apache.org/sU5 and probably others).
This is an overly generalized and contentious statement that strips our
committers of authority. There is no reason to state this given 4 and 5
below. If we want to say something like this then we should narrow it to
the non contentious classes of features, even loosly defined by
thresholding the costs of maintenance, documentation, API effects, etc.
>
> 4. If it is desired to back-port a new feature, then a vote on the
> developer mailing list should be called, due to the additional
> development and support burden the new feature may cause for all
> developers.
Again, we should have a threshold here so that less than one hour of code
change doesn't require three days of vote tracking. While additional
attention should be payed to the cost of backporting features, it is
important for us to trust and enable our developers to make good decisions.
>
> 5. Even when it is agreed that a feature should be back-ported, it
> should not be done unless/until a feature is first represented in a
> newer release that has gone through the testing and release process,
> and can be considered stable enough to back-port. This ensures focus
> is kept on the main development branch for new features, and
> significantly reduces the development burden of back-porting. It also
> gives us a clear idea of the target behavior for the back-ported
> feature, so that it will behave in the same way as the same feature in
> the later release line.
While the suggested rule would have the desire effect as listed, this is
too broad a rule that eliminates much of the quick reation time that is
really the whole point of backporting. If developers understand and agree
on the costs of backporting, then do we really need a rule like this?
Adam
Re: Backporting policy proposal
Posted by Josh Elser <jo...@gmail.com>.
On 06/17/2013 01:07 PM, Christopher wrote:
> I propose we adopt a more structured policy beyond simple "lazy
> consensus" to be apply to backporting features. Some guidelines I'd
> like to see in this policy, include:
>
> 1. Back-porting bugfixes to a prior release line that is not EOL
> (end-of-life) is always okay (subject to normal lazy consensus), but
> it is strongly preferred to fix it first in the older branch and merge
> forward to the newer one(s).
Agreed
> 2. Back-porting performance improvements to a prior release line that
> is not EOL (end-of-life) is usually okay (subject to normal lazy
> consensus), so long as it does not change user-facing behavior or API.
> It is still strongly preferred to add such fixes in the older branch
> first, and merge forward to the newer one(s).
Agreed
> 3. Back-porting new features and additions are to be avoided as a
> general rule (see arguments for this in previous threads:
> ACCUMULO-1488 and http://s.apache.org/sU5 and probably others).
Agree with caveat incoming on #4
> 4. If it is desired to back-port a new feature, then a vote on the
> developer mailing list should be called, due to the additional
> development and support burden the new feature may cause for all
> developers.
Mainly agree, although my opinion is more driven by wanting to not stall
out major releases. New major features, as scary as they may be, should
be addressed head-on with ourselves giving them adequate testing. Over
time, this should alleviate the desire to backport things so frequently.
> 5. Even when it is agreed that a feature should be back-ported, it
> should not be done unless/until a feature is first represented in a
> newer release that has gone through the testing and release process,
> and can be considered stable enough to back-port. This ensures focus
> is kept on the main development branch for new features, and
> significantly reduces the development burden of back-porting. It also
> gives us a clear idea of the target behavior for the back-ported
> feature, so that it will behave in the same way as the same feature in
> the later release line.
Mostly agree. I think there should be a higher-bar to reach when
including something significant in a previously released version in
regards to testing, stability and API-compatibilty. I would hope our
devs would do this on their own, but perhaps it's good to be explicit.
Again, I do like the focus on including new things in the
active-development version.
> Please discuss these points, or add your own.
>
> --
> Christopher L Tubbs II
> http://gravatar.com/ctubbsii
RE: Backporting policy proposal
Posted by Dave Marion <dl...@comcast.net>.
+1 for more structure. I like the idea of not back porting new features, it
will hopefully allow for more frequent releases of major/minor releases.
Someone mentioned bylaws for the project; easily found a few examples (see
below). One even mentions the types of changes that will be in the different
types of releases.
[1] http://hc.apache.org/bylaws.html
[2] http://pig.apache.org/bylaws.html
-----Original Message-----
From: Billie Rinaldi [mailto:billie.rinaldi@gmail.com]
Sent: Monday, June 17, 2013 5:26 PM
To: dev@accumulo.apache.org
Subject: Re: Backporting policy proposal
On Mon, Jun 17, 2013 at 10:07 AM, Christopher <ct...@apache.org> wrote:
> I propose we adopt a more structured policy beyond simple "lazy
> consensus" to be apply to backporting features. Some guidelines I'd
> like to see in this policy, include:
>
> 1. Back-porting bugfixes to a prior release line that is not EOL
> (end-of-life) is always okay (subject to normal lazy consensus), but
> it is strongly preferred to fix it first in the older branch and merge
> forward to the newer one(s).
>
> 2. Back-porting performance improvements to a prior release line that
> is not EOL (end-of-life) is usually okay (subject to normal lazy
> consensus), so long as it does not change user-facing behavior or API.
> It is still strongly preferred to add such fixes in the older branch
> first, and merge forward to the newer one(s).
>
> 3. Back-porting new features and additions are to be avoided as a
> general rule (see arguments for this in previous threads:
> ACCUMULO-1488 and http://s.apache.org/sU5 and probably others).
>
> 4. If it is desired to back-port a new feature, then a vote on the
> developer mailing list should be called, due to the additional
> development and support burden the new feature may cause for all
> developers.
>
> 5. Even when it is agreed that a feature should be back-ported, it
> should not be done unless/until a feature is first represented in a
> newer release that has gone through the testing and release process,
> and can be considered stable enough to back-port. This ensures focus
> is kept on the main development branch for new features, and
> significantly reduces the development burden of back-porting. It also
> gives us a clear idea of the target behavior for the back-ported
> feature, so that it will behave in the same way as the same feature in
> the later release line.
>
I'm not sure #5 makes sense. Certainly it's a sound idea not to do the
back-porting until the feature design has been hammered out very well.
However, in an example such as adding an iterator that we've agreed on
back-porting, whose design is clear, it wouldn't make sense to wait until
1.6.0 comes out to actually add it to the 1.5 line. I could see an argument
for placing additional testing requirements on back-ported features, like
creating full-coverage unit tests and functional tests for the new code, to
offset the risk of introducing code that has not yet gone through a full
testing cycle and release process.
I'm still deciding what I think about #4. For one, I'm reluctant to move
the discussion of the feature from the ticket to the dev list. If we do
decide to require a vote (either on ticket or dev list), we should also
decide what type of approval is appropriate (consensus [1], majority [2], or
a modified version of either, such as requiring fewer +1s but placing the
same restrictions on -1s).
Billie
[1]: http://apache.org/foundation/glossary.html#ConsensusApproval
[2]: http://apache.org/foundation/glossary.html#MajorityApproval
>
> Please discuss these points, or add your own.
>
> --
> Christopher L Tubbs II
> http://gravatar.com/ctubbsii
>
Re: Backporting policy proposal
Posted by Billie Rinaldi <bi...@gmail.com>.
On Mon, Jun 17, 2013 at 5:22 PM, Christopher <ct...@apache.org> wrote:
> #5 was borne from Keith's frustration supporting the backport of the
> proxy. It may not apply to all features.
>
I definitely agree that we should have spent more time discussing the
implications of backporting 100,000 lines of generated code. =) I would
be OK with considering #5 on a case-by-case basis for major features.
Billie
>
> I think the dev list is the appropriate place for decisions about
> inclusion of issues into particular branches, where the ticket is more
> appropriate to discuss the feature itself. The dev list seems to be
> the place where we consciously include the whole developer community,
> rather than just those interested in a particular feature
> (back-porting features seems to have the potential to impact all
> developers, plus it seems to me less likely to divide the community on
> particular features). Also, I personally track the threads in the dev
> list more closely than all the comments on all the tickets (so I
> assume others do, too). However, I don't feel strongly about this
> point, so long as there is an explicit community decision to port a
> feature to a previous version, and take on the additional costs of
> doing so, under the recognition that this diverts time and effort away
> from the next feature release version.
>
> --
> Christopher L Tubbs II
> http://gravatar.com/ctubbsii
>
>
> On Mon, Jun 17, 2013 at 5:26 PM, Billie Rinaldi
> <bi...@gmail.com> wrote:
> > On Mon, Jun 17, 2013 at 10:07 AM, Christopher <ct...@apache.org>
> wrote:
> >
> >> I propose we adopt a more structured policy beyond simple "lazy
> >> consensus" to be apply to backporting features. Some guidelines I'd
> >> like to see in this policy, include:
> >>
> >> 1. Back-porting bugfixes to a prior release line that is not EOL
> >> (end-of-life) is always okay (subject to normal lazy consensus), but
> >> it is strongly preferred to fix it first in the older branch and merge
> >> forward to the newer one(s).
> >>
> >> 2. Back-porting performance improvements to a prior release line that
> >> is not EOL (end-of-life) is usually okay (subject to normal lazy
> >> consensus), so long as it does not change user-facing behavior or API.
> >> It is still strongly preferred to add such fixes in the older branch
> >> first, and merge forward to the newer one(s).
> >>
> >> 3. Back-porting new features and additions are to be avoided as a
> >> general rule (see arguments for this in previous threads:
> >> ACCUMULO-1488 and http://s.apache.org/sU5 and probably others).
> >>
> >> 4. If it is desired to back-port a new feature, then a vote on the
> >> developer mailing list should be called, due to the additional
> >> development and support burden the new feature may cause for all
> >> developers.
> >>
> >> 5. Even when it is agreed that a feature should be back-ported, it
> >> should not be done unless/until a feature is first represented in a
> >> newer release that has gone through the testing and release process,
> >> and can be considered stable enough to back-port. This ensures focus
> >> is kept on the main development branch for new features, and
> >> significantly reduces the development burden of back-porting. It also
> >> gives us a clear idea of the target behavior for the back-ported
> >> feature, so that it will behave in the same way as the same feature in
> >> the later release line.
> >>
> >
> > I'm not sure #5 makes sense. Certainly it's a sound idea not to do the
> > back-porting until the feature design has been hammered out very well.
> > However, in an example such as adding an iterator that we've agreed on
> > back-porting, whose design is clear, it wouldn't make sense to wait until
> > 1.6.0 comes out to actually add it to the 1.5 line. I could see an
> > argument for placing additional testing requirements on back-ported
> > features, like creating full-coverage unit tests and functional tests for
> > the new code, to offset the risk of introducing code that has not yet
> gone
> > through a full testing cycle and release process.
> >
> > I'm still deciding what I think about #4. For one, I'm reluctant to move
> > the discussion of the feature from the ticket to the dev list. If we do
> > decide to require a vote (either on ticket or dev list), we should also
> > decide what type of approval is appropriate (consensus [1], majority [2],
> > or a modified version of either, such as requiring fewer +1s but placing
> > the same restrictions on -1s).
> >
> > Billie
> >
> > [1]: http://apache.org/foundation/glossary.html#ConsensusApproval
> > [2]: http://apache.org/foundation/glossary.html#MajorityApproval
> >
> >
> >>
> >> Please discuss these points, or add your own.
> >>
> >> --
> >> Christopher L Tubbs II
> >> http://gravatar.com/ctubbsii
> >>
>
Re: Backporting policy proposal
Posted by Christopher <ct...@apache.org>.
#5 was borne from Keith's frustration supporting the backport of the
proxy. It may not apply to all features.
I think the dev list is the appropriate place for decisions about
inclusion of issues into particular branches, where the ticket is more
appropriate to discuss the feature itself. The dev list seems to be
the place where we consciously include the whole developer community,
rather than just those interested in a particular feature
(back-porting features seems to have the potential to impact all
developers, plus it seems to me less likely to divide the community on
particular features). Also, I personally track the threads in the dev
list more closely than all the comments on all the tickets (so I
assume others do, too). However, I don't feel strongly about this
point, so long as there is an explicit community decision to port a
feature to a previous version, and take on the additional costs of
doing so, under the recognition that this diverts time and effort away
from the next feature release version.
--
Christopher L Tubbs II
http://gravatar.com/ctubbsii
On Mon, Jun 17, 2013 at 5:26 PM, Billie Rinaldi
<bi...@gmail.com> wrote:
> On Mon, Jun 17, 2013 at 10:07 AM, Christopher <ct...@apache.org> wrote:
>
>> I propose we adopt a more structured policy beyond simple "lazy
>> consensus" to be apply to backporting features. Some guidelines I'd
>> like to see in this policy, include:
>>
>> 1. Back-porting bugfixes to a prior release line that is not EOL
>> (end-of-life) is always okay (subject to normal lazy consensus), but
>> it is strongly preferred to fix it first in the older branch and merge
>> forward to the newer one(s).
>>
>> 2. Back-porting performance improvements to a prior release line that
>> is not EOL (end-of-life) is usually okay (subject to normal lazy
>> consensus), so long as it does not change user-facing behavior or API.
>> It is still strongly preferred to add such fixes in the older branch
>> first, and merge forward to the newer one(s).
>>
>> 3. Back-porting new features and additions are to be avoided as a
>> general rule (see arguments for this in previous threads:
>> ACCUMULO-1488 and http://s.apache.org/sU5 and probably others).
>>
>> 4. If it is desired to back-port a new feature, then a vote on the
>> developer mailing list should be called, due to the additional
>> development and support burden the new feature may cause for all
>> developers.
>>
>> 5. Even when it is agreed that a feature should be back-ported, it
>> should not be done unless/until a feature is first represented in a
>> newer release that has gone through the testing and release process,
>> and can be considered stable enough to back-port. This ensures focus
>> is kept on the main development branch for new features, and
>> significantly reduces the development burden of back-porting. It also
>> gives us a clear idea of the target behavior for the back-ported
>> feature, so that it will behave in the same way as the same feature in
>> the later release line.
>>
>
> I'm not sure #5 makes sense. Certainly it's a sound idea not to do the
> back-porting until the feature design has been hammered out very well.
> However, in an example such as adding an iterator that we've agreed on
> back-porting, whose design is clear, it wouldn't make sense to wait until
> 1.6.0 comes out to actually add it to the 1.5 line. I could see an
> argument for placing additional testing requirements on back-ported
> features, like creating full-coverage unit tests and functional tests for
> the new code, to offset the risk of introducing code that has not yet gone
> through a full testing cycle and release process.
>
> I'm still deciding what I think about #4. For one, I'm reluctant to move
> the discussion of the feature from the ticket to the dev list. If we do
> decide to require a vote (either on ticket or dev list), we should also
> decide what type of approval is appropriate (consensus [1], majority [2],
> or a modified version of either, such as requiring fewer +1s but placing
> the same restrictions on -1s).
>
> Billie
>
> [1]: http://apache.org/foundation/glossary.html#ConsensusApproval
> [2]: http://apache.org/foundation/glossary.html#MajorityApproval
>
>
>>
>> Please discuss these points, or add your own.
>>
>> --
>> Christopher L Tubbs II
>> http://gravatar.com/ctubbsii
>>
Re: Backporting policy proposal
Posted by Billie Rinaldi <bi...@gmail.com>.
On Mon, Jun 17, 2013 at 10:07 AM, Christopher <ct...@apache.org> wrote:
> I propose we adopt a more structured policy beyond simple "lazy
> consensus" to be apply to backporting features. Some guidelines I'd
> like to see in this policy, include:
>
> 1. Back-porting bugfixes to a prior release line that is not EOL
> (end-of-life) is always okay (subject to normal lazy consensus), but
> it is strongly preferred to fix it first in the older branch and merge
> forward to the newer one(s).
>
> 2. Back-porting performance improvements to a prior release line that
> is not EOL (end-of-life) is usually okay (subject to normal lazy
> consensus), so long as it does not change user-facing behavior or API.
> It is still strongly preferred to add such fixes in the older branch
> first, and merge forward to the newer one(s).
>
> 3. Back-porting new features and additions are to be avoided as a
> general rule (see arguments for this in previous threads:
> ACCUMULO-1488 and http://s.apache.org/sU5 and probably others).
>
> 4. If it is desired to back-port a new feature, then a vote on the
> developer mailing list should be called, due to the additional
> development and support burden the new feature may cause for all
> developers.
>
> 5. Even when it is agreed that a feature should be back-ported, it
> should not be done unless/until a feature is first represented in a
> newer release that has gone through the testing and release process,
> and can be considered stable enough to back-port. This ensures focus
> is kept on the main development branch for new features, and
> significantly reduces the development burden of back-porting. It also
> gives us a clear idea of the target behavior for the back-ported
> feature, so that it will behave in the same way as the same feature in
> the later release line.
>
I'm not sure #5 makes sense. Certainly it's a sound idea not to do the
back-porting until the feature design has been hammered out very well.
However, in an example such as adding an iterator that we've agreed on
back-porting, whose design is clear, it wouldn't make sense to wait until
1.6.0 comes out to actually add it to the 1.5 line. I could see an
argument for placing additional testing requirements on back-ported
features, like creating full-coverage unit tests and functional tests for
the new code, to offset the risk of introducing code that has not yet gone
through a full testing cycle and release process.
I'm still deciding what I think about #4. For one, I'm reluctant to move
the discussion of the feature from the ticket to the dev list. If we do
decide to require a vote (either on ticket or dev list), we should also
decide what type of approval is appropriate (consensus [1], majority [2],
or a modified version of either, such as requiring fewer +1s but placing
the same restrictions on -1s).
Billie
[1]: http://apache.org/foundation/glossary.html#ConsensusApproval
[2]: http://apache.org/foundation/glossary.html#MajorityApproval
>
> Please discuss these points, or add your own.
>
> --
> Christopher L Tubbs II
> http://gravatar.com/ctubbsii
>
Re: Backporting policy proposal
Posted by Christopher <ct...@apache.org>.
I should also mention another argument against back-porting features.
With our discussions about documentation, the problem of back-porting
grows worse, because now we need to bombard users with more sets of
documentation that applies to each release. Whereas, if we were more
strict about bugfix releases just being bugfixes, we wouldn't have to
republish documentation. The 1.5 user documentation would apply to all
1.5.x, not just 1.5.0, with new documentation for 1.5.1, and more with
1.5.2, etc.
So, just the problem of keeping users informed of our features,
through documentation, becomes needlessly problematic if we start
back-porting features to older release lines.
--
Christopher L Tubbs II
http://gravatar.com/ctubbsii
On Mon, Jun 17, 2013 at 4:59 PM, Christopher <ct...@apache.org> wrote:
> Our versioning standard has been for quite some time now, the standard
> Maven [major].[minor].[bugfix] versioning model. While we've perhaps
> never really followed this strictly with regards to injecting "minor"
> features into a bugfix version, it's a good model to follow, and my
> proposal here would turn that "loosely followed" standard into a more
> strict standard.
>
> This standard is already reflected in our workflow, after all... we
> release A.A.0 versions, and then move on to working on the next
> feature release A.B.0, while supporting A.A.x through bugfixes A.A.1,
> A.A.2, etc... This workflow implies that the last part of the A.B.C
> version is for bugfixes to supported releases. We just haven't
> followed this strictly. I'm hoping we do.
>
> Following this strictly means that "minor" features should be avoided
> in bugfix releases also, and that bugfixes should really be targeted
> to fix the bug without changing the overall feature set or the API or
> behavior for existing features.
>
> The kinds of cases like "bugs that are best resolved via a feature"
> seem to be very much the kind that we'd want to vote on (and would
> fall cleanly into sections 4 and 5 of my above proposal).
>
> --
> Christopher L Tubbs II
> http://gravatar.com/ctubbsii
>
>
> On Mon, Jun 17, 2013 at 4:02 PM, John Vines <jv...@gmail.com> wrote:
>> I think the issue here is that there is ambiguity on what a minor release
>> (minor minor?) is. Will 1.5.1 be only bugfixes, or are minor features
>> acceptable? If the former, what about bugs that are best resolved via
>> feature? If the latter, where is the line?
>>
>>
>> On Mon, Jun 17, 2013 at 1:07 PM, Christopher <ct...@apache.org> wrote:
>>
>>> I propose we adopt a more structured policy beyond simple "lazy
>>> consensus" to be apply to backporting features. Some guidelines I'd
>>> like to see in this policy, include:
>>>
>>> 1. Back-porting bugfixes to a prior release line that is not EOL
>>> (end-of-life) is always okay (subject to normal lazy consensus), but
>>> it is strongly preferred to fix it first in the older branch and merge
>>> forward to the newer one(s).
>>>
>>> 2. Back-porting performance improvements to a prior release line that
>>> is not EOL (end-of-life) is usually okay (subject to normal lazy
>>> consensus), so long as it does not change user-facing behavior or API.
>>> It is still strongly preferred to add such fixes in the older branch
>>> first, and merge forward to the newer one(s).
>>>
>>> 3. Back-porting new features and additions are to be avoided as a
>>> general rule (see arguments for this in previous threads:
>>> ACCUMULO-1488 and http://s.apache.org/sU5 and probably others).
>>>
>>> 4. If it is desired to back-port a new feature, then a vote on the
>>> developer mailing list should be called, due to the additional
>>> development and support burden the new feature may cause for all
>>> developers.
>>>
>>> 5. Even when it is agreed that a feature should be back-ported, it
>>> should not be done unless/until a feature is first represented in a
>>> newer release that has gone through the testing and release process,
>>> and can be considered stable enough to back-port. This ensures focus
>>> is kept on the main development branch for new features, and
>>> significantly reduces the development burden of back-porting. It also
>>> gives us a clear idea of the target behavior for the back-ported
>>> feature, so that it will behave in the same way as the same feature in
>>> the later release line.
>>>
>>> Please discuss these points, or add your own.
>>>
>>> --
>>> Christopher L Tubbs II
>>> http://gravatar.com/ctubbsii
>>>
>>
>>
>>
>> --
>> Cheers
>> ~John
Re: Backporting policy proposal
Posted by Christopher <ct...@apache.org>.
Our versioning standard has been for quite some time now, the standard
Maven [major].[minor].[bugfix] versioning model. While we've perhaps
never really followed this strictly with regards to injecting "minor"
features into a bugfix version, it's a good model to follow, and my
proposal here would turn that "loosely followed" standard into a more
strict standard.
This standard is already reflected in our workflow, after all... we
release A.A.0 versions, and then move on to working on the next
feature release A.B.0, while supporting A.A.x through bugfixes A.A.1,
A.A.2, etc... This workflow implies that the last part of the A.B.C
version is for bugfixes to supported releases. We just haven't
followed this strictly. I'm hoping we do.
Following this strictly means that "minor" features should be avoided
in bugfix releases also, and that bugfixes should really be targeted
to fix the bug without changing the overall feature set or the API or
behavior for existing features.
The kinds of cases like "bugs that are best resolved via a feature"
seem to be very much the kind that we'd want to vote on (and would
fall cleanly into sections 4 and 5 of my above proposal).
--
Christopher L Tubbs II
http://gravatar.com/ctubbsii
On Mon, Jun 17, 2013 at 4:02 PM, John Vines <jv...@gmail.com> wrote:
> I think the issue here is that there is ambiguity on what a minor release
> (minor minor?) is. Will 1.5.1 be only bugfixes, or are minor features
> acceptable? If the former, what about bugs that are best resolved via
> feature? If the latter, where is the line?
>
>
> On Mon, Jun 17, 2013 at 1:07 PM, Christopher <ct...@apache.org> wrote:
>
>> I propose we adopt a more structured policy beyond simple "lazy
>> consensus" to be apply to backporting features. Some guidelines I'd
>> like to see in this policy, include:
>>
>> 1. Back-porting bugfixes to a prior release line that is not EOL
>> (end-of-life) is always okay (subject to normal lazy consensus), but
>> it is strongly preferred to fix it first in the older branch and merge
>> forward to the newer one(s).
>>
>> 2. Back-porting performance improvements to a prior release line that
>> is not EOL (end-of-life) is usually okay (subject to normal lazy
>> consensus), so long as it does not change user-facing behavior or API.
>> It is still strongly preferred to add such fixes in the older branch
>> first, and merge forward to the newer one(s).
>>
>> 3. Back-porting new features and additions are to be avoided as a
>> general rule (see arguments for this in previous threads:
>> ACCUMULO-1488 and http://s.apache.org/sU5 and probably others).
>>
>> 4. If it is desired to back-port a new feature, then a vote on the
>> developer mailing list should be called, due to the additional
>> development and support burden the new feature may cause for all
>> developers.
>>
>> 5. Even when it is agreed that a feature should be back-ported, it
>> should not be done unless/until a feature is first represented in a
>> newer release that has gone through the testing and release process,
>> and can be considered stable enough to back-port. This ensures focus
>> is kept on the main development branch for new features, and
>> significantly reduces the development burden of back-porting. It also
>> gives us a clear idea of the target behavior for the back-ported
>> feature, so that it will behave in the same way as the same feature in
>> the later release line.
>>
>> Please discuss these points, or add your own.
>>
>> --
>> Christopher L Tubbs II
>> http://gravatar.com/ctubbsii
>>
>
>
>
> --
> Cheers
> ~John
Re: Backporting policy proposal
Posted by John Vines <jv...@gmail.com>.
I think the issue here is that there is ambiguity on what a minor release
(minor minor?) is. Will 1.5.1 be only bugfixes, or are minor features
acceptable? If the former, what about bugs that are best resolved via
feature? If the latter, where is the line?
On Mon, Jun 17, 2013 at 1:07 PM, Christopher <ct...@apache.org> wrote:
> I propose we adopt a more structured policy beyond simple "lazy
> consensus" to be apply to backporting features. Some guidelines I'd
> like to see in this policy, include:
>
> 1. Back-porting bugfixes to a prior release line that is not EOL
> (end-of-life) is always okay (subject to normal lazy consensus), but
> it is strongly preferred to fix it first in the older branch and merge
> forward to the newer one(s).
>
> 2. Back-porting performance improvements to a prior release line that
> is not EOL (end-of-life) is usually okay (subject to normal lazy
> consensus), so long as it does not change user-facing behavior or API.
> It is still strongly preferred to add such fixes in the older branch
> first, and merge forward to the newer one(s).
>
> 3. Back-porting new features and additions are to be avoided as a
> general rule (see arguments for this in previous threads:
> ACCUMULO-1488 and http://s.apache.org/sU5 and probably others).
>
> 4. If it is desired to back-port a new feature, then a vote on the
> developer mailing list should be called, due to the additional
> development and support burden the new feature may cause for all
> developers.
>
> 5. Even when it is agreed that a feature should be back-ported, it
> should not be done unless/until a feature is first represented in a
> newer release that has gone through the testing and release process,
> and can be considered stable enough to back-port. This ensures focus
> is kept on the main development branch for new features, and
> significantly reduces the development burden of back-porting. It also
> gives us a clear idea of the target behavior for the back-ported
> feature, so that it will behave in the same way as the same feature in
> the later release line.
>
> Please discuss these points, or add your own.
>
> --
> Christopher L Tubbs II
> http://gravatar.com/ctubbsii
>
--
Cheers
~John