[Math] Performance bug: Lessons to be drawn?

[Gilles Sadowski <gi...@harfang.homelinux.org>]

Hello.

Are there lessons to be drawn from the problem discovered just after the
recent release? Apart from pointing fingers, that is. :-}

I presume that people who voted for the release did as thorough a review as
they could.
And the problem is not primarily one of unit test coverage. Some use cases
will always escape the scope of unit testing: It is not reasonable to extend
the tests suite with all imaginable use cases.

One thing that comes to mind would be to announce the release candidates on
the "user" ML, gently requesting those users who intend to upgrade to test
the JAR in as many of their applications as possible. ["Let them speak now
or forever remain silent." ;-)]
For backwards-compatible releases, that does not seem to be a huge burden on
the people that use CM.

What do you think?


Regards,
Gilles

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org

Re: [Math] Performance bug: Lessons to be drawn?

[Jörg Schaible <jo...@gmx.de>]

Hi Gilles,

Gilles Sadowski wrote:

> Hello.
> 
> Are there lessons to be drawn from the problem discovered just after the
> recent release? Apart from pointing fingers, that is. :-}
> 
> I presume that people who voted for the release did as thorough a review
> as they could.
> And the problem is not primarily one of unit test coverage. Some use cases
> will always escape the scope of unit testing: It is not reasonable to
> extend the tests suite with all imaginable use cases.
> 
> One thing that comes to mind would be to announce the release candidates
> on the "user" ML, gently requesting those users who intend to upgrade to
> test the JAR in as many of their applications as possible. ["Let them
> speak now or forever remain silent." ;-)]
> For backwards-compatible releases, that does not seem to be a huge burden
> on the people that use CM.
> 
> What do you think?

So what? Simply release 3.1.1. We as PMC have to rely on the active 
component committers that they got it right. This time something slipped 
through, shit happens. No-one can expect that most of the commons PMC can 
judge a release with regard to contents. But we try to ensure the formal 
rules. These rules ensure that we should be able to get out a binary 
compatible 3.1.1 very easy.

Cheers,
Jörg


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org

Re: [Math] Performance bug: Lessons to be drawn?

[Thomas Neidhart <th...@gmail.com>]

On Mon, Dec 31, 2012 at 2:00 AM, Phil Steitz <ph...@gmail.com> wrote:

> On 12/30/12 4:01 PM, Gilles Sadowski wrote:
> > Hello.
> >
> > Are there lessons to be drawn from the problem discovered just after the
> > recent release? Apart from pointing fingers, that is. :-}
>
> Most important point - don't do that (point fingers) :)
> >
> > I presume that people who voted for the release did as thorough a review
> as
> > they could.
> > And the problem is not primarily one of unit test coverage. Some use
> cases
> > will always escape the scope of unit testing: It is not reasonable to
> extend
> > the tests suite with all imaginable use cases.
> >
> > One thing that comes to mind would be to announce the release candidates
> on
> > the "user" ML, gently requesting those users who intend to upgrade to
> test
> > the JAR in as many of their applications as possible. ["Let them speak
> now
> > or forever remain silent." ;-)]
> > For backwards-compatible releases, that does not seem to be a huge
> burden on
> > the people that use CM.
> >
> > What do you think?
>
> As I said in another post, I think the Tomcat community came up with
> a pretty good way to deal with this, which was to cut releases and
> then after the general user community had an opportunity to work
> with them, assess their stability.  See [1] and browse tomcat-dev
> archives for an idea of how this works in practice.  The basic idea
> is that it is best to get code into the hands of users ASAP and
> assess its stability after you have gotten some feedback, basically
> acknowledging that you can't possibly anticipate all of the problems
> that a new release might bring and enabling developers - and the
> user community - to move *forward* to stable releases as we identify
> and address problems in releases that do not get "stable" status.
>
> So in concrete terms, I suggest that we move to a "release early,
> release often" model with releases designated alpha when they are
> cut and subsequently voted as beta or stable.  We haven't done this
> is Commons before; but we have talked about it a few times over the
> years.  Some wisdom from markt, mturk or other tomcat devs would be
> appreciated here :)
>
> Phil
>
> [1] http://tomcat.apache.org/whichversion.html
>

Hi,

I totally agree with Phil here, for a library of considerable size as CM it
is very difficult to assess the quality when following the standard commons
release process (which mainly focuses on formal things).
Even if there is 100% code coverage, there are still areas that may cause
problems (performance / numerical stability for different data sets as the
ones envisioned by the developers, I had several examples of this).

In theory, users can grab the latest snapshots as they tend to be quite
stable, and provide feedback, but this happens only very rarely in my
experience. Maybe alpha / beta releases would give our user base more
incentives to test the code with their applications / use cases?

Thomas

Re: [Math] Performance bug: Lessons to be drawn?

[Gilles Sadowski <gi...@harfang.homelinux.org>]

> >
> > Are there lessons to be drawn from the problem discovered just after the
> > recent release? Apart from pointing fingers, that is. :-}
> 
> Most important point - don't do that (point fingers) :)
> >
> > I presume that people who voted for the release did as thorough a review as
> > they could.
> > And the problem is not primarily one of unit test coverage. Some use cases
> > will always escape the scope of unit testing: It is not reasonable to extend
> > the tests suite with all imaginable use cases.
> >
> > One thing that comes to mind would be to announce the release candidates on
> > the "user" ML, gently requesting those users who intend to upgrade to test
> > the JAR in as many of their applications as possible. ["Let them speak now
> > or forever remain silent." ;-)]
> > For backwards-compatible releases, that does not seem to be a huge burden on
> > the people that use CM.
> >
> > What do you think?
> 
> As I said in another post, I think the Tomcat community came up with
> a pretty good way to deal with this, which was to cut releases and
> then after the general user community had an opportunity to work
> with them, assess their stability.  See [1] and browse tomcat-dev
> archives for an idea of how this works in practice.  The basic idea
> is that it is best to get code into the hands of users ASAP and
> assess its stability after you have gotten some feedback, basically
> acknowledging that you can't possibly anticipate all of the problems
> that a new release might bring and enabling developers - and the
> user community - to move *forward* to stable releases as we identify
> and address problems in releases that do not get "stable" status.
> 
> So in concrete terms, I suggest that we move to a "release early,
> release often" model with releases designated alpha when they are
> cut and subsequently voted as beta or stable.  We haven't done this
> is Commons before; but we have talked about it a few times over the
> years. [...]

Sure. Lately, the few times this option was raised, it was by me, and each
time, I did not get the impression that it was warmly welcome: when I
proposed to get a release every 6 months, or after a shorter time span (i.e.
when some to-be-defined amount of work had been done), the discussion ended
with something like "we'll try to get 2 releases per year"...

There are plenty of arguments in favour of "release early, release often",
and I didn't understand the reluctance to apply it here.
Or did I just get the wrong impression? Can I really start a release
process every three months?

Of course, I agree with your above suggestion to work like they do in
Tomcat. However, I'm not optimistic that the CM's users community will
actually be as reactive...


Regards,
Gilles

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org

Re: [Math] Performance bug: Lessons to be drawn?

[Phil Steitz <ph...@gmail.com>]

On 12/30/12 4:01 PM, Gilles Sadowski wrote:
> Hello.
>
> Are there lessons to be drawn from the problem discovered just after the
> recent release? Apart from pointing fingers, that is. :-}

Most important point - don't do that (point fingers) :)
>
> I presume that people who voted for the release did as thorough a review as
> they could.
> And the problem is not primarily one of unit test coverage. Some use cases
> will always escape the scope of unit testing: It is not reasonable to extend
> the tests suite with all imaginable use cases.
>
> One thing that comes to mind would be to announce the release candidates on
> the "user" ML, gently requesting those users who intend to upgrade to test
> the JAR in as many of their applications as possible. ["Let them speak now
> or forever remain silent." ;-)]
> For backwards-compatible releases, that does not seem to be a huge burden on
> the people that use CM.
>
> What do you think?

As I said in another post, I think the Tomcat community came up with
a pretty good way to deal with this, which was to cut releases and
then after the general user community had an opportunity to work
with them, assess their stability.  See [1] and browse tomcat-dev
archives for an idea of how this works in practice.  The basic idea
is that it is best to get code into the hands of users ASAP and
assess its stability after you have gotten some feedback, basically
acknowledging that you can't possibly anticipate all of the problems
that a new release might bring and enabling developers - and the
user community - to move *forward* to stable releases as we identify
and address problems in releases that do not get "stable" status.

So in concrete terms, I suggest that we move to a "release early,
release often" model with releases designated alpha when they are
cut and subsequently voted as beta or stable.  We haven't done this
is Commons before; but we have talked about it a few times over the
years.  Some wisdom from markt, mturk or other tomcat devs would be
appreciated here :)

Phil

[1] http://tomcat.apache.org/whichversion.html

>
>
> Regards,
> Gilles
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> For additional commands, e-mail: dev-help@commons.apache.org
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org