Impact of regressions downstream

[Kathey Marsden <km...@sbcglobal.net>]

In  a previous thread Andrew said:
 >Better  to release early so new code gets into actual user's hands so
 >regressions can be flushed out sooner. Regressions happen, and no
 >release is ever go to be perfect (although that would be nice)

The reason I am so regression averse is that there is a tremendous 
impact and magnification of regressions downstream when Derby is used as 
a component of a larger piece of software.  For example if Derby is used 
as a small component in a mission critical piece of software to hold and 
manage configuration data. The configuration data is read at start up 
and during processing.   If  there is regression and a  null pointer 
exception occurs  when reading the Derby data, then that mission 
critical piece of software doesn't work at all, so the downstream 
severity is much worse.

Kathey

Re: Impact of regressions downstream

[Andrew McIntyre <mc...@gmail.com>]

On 9/13/06, Kathey Marsden <km...@sbcglobal.net> wrote:
>
> The reason I am so regression averse is that there is a tremendous
> impact and magnification of regressions downstream when Derby is used as
> a component of a larger piece of software.

I agree, and I would be surprised if there was anyone who disagreed.

It is in the interest of anyone who wants to use of Derby to be able
to rely on it as a part of their software architecture. Therefore, it
is in our best interest to keep regressions to a minimum so that
developers feel they can rely on Derby.

So, the question is: what can we, the Derby community, do about it
when faced with regressions like the ones we've seen with the recent
optimizer fixes? We must be fair and not fault of the developer who
made the original changes. Additional tests which exercised the
changes were added but they were not enough to uncover all the
problems. So our tests did not catch the problem, and it appears that
a downstream consumer's tests did not either. So it wasn't until it
got into a third party's hands, hands which did something we did not -
or could not - anticipate, that a problem was uncovered.

Is there anything that could have prevented the regression? Even if
the changes were flagged as potentially regression-causing, what can
we do? The developer making the change already made a best effort at
testing the change. Many eyes looked at the code and deemed it sound.
And so, that code was checked in.

But yet there was a regression. Could we have prevented it? The only
way that we might have prevented it that has occurred to me is that if
changes are being made to areas that are considered 'sensitive,' that
we monitor code coverage around the changes: have all the new paths
through the new or changed code been tested and verified to work
properly? Maybe other people have better suggestions, but putting the
burden of refining tests to conform to code coverage results seems an
undue burden on developers when functional tests that they have
crafted to match the change work and pass, and no one else is able to
find issues. Doing so would raise the bar considerably for working in
an area flagged as 'sensitive', which means you are less likely to get
any contributors in that area.

I agree that it is not a good thing that we introduced a regression
into 10.1, but I think we can set a great example by fixing the
regressions quickly and making a new release where the regressions are
fixed. If we can somehow make the time between when the regression is
identified, fixed, and then available in a released version approach
zero, then we as a community should be able to sleep at night without
having to worry about whether or not a recent change might have caused
a regression. :-)

I would love to hear other people's opinions on the issue, especially
with regards to improvements to how we can test and verify changes
with a high regression potential.

cheers,
andrew

p.s. A debate for elsewhere would be to discuss how SQL's declarative
style impedes functional testing of implementations written in
imperative languages.