You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by Kathey Marsden <km...@sbcglobal.net> on 2006/04/03 23:16:36 UTC
Monitoring/improving code coverage (was Re: code coverage results
for trunk - svn revision no 390306)
David W. Van Couvering wrote:
>
> Did I ask this before? Do we want to agree upon a "low water mark"
> for code coverage and send out a "Quality Regression" email if our
> testing coverage falls below that mark? I think this would have a lot
> of value.
This sounds like an interesting idea. Code coverage is an important
quality data point. What kind of granularity would it have? Would it
be just the overall number or would individual packages or files be
flagged? Also for areas that have poor coverage, how could we
encourage numbers to be brought up before or during enhancements?
Kathey
Re: Monitoring/improving code coverage (was Re: code coverage results
for trunk - svn revision no 390306)
Posted by Kathey Marsden <km...@sbcglobal.net>.
Daniel John Debrunner wrote:
>Not sure how we can get people to scratch the code coverage itch. It
>seems we can't get a lot of folks interested in fixing the 150+ bugs out
>there, never mind writing more tests that might uncover more bugs. I
>would love it if we could find a way.
>
>Bottom line is that if people don't care about code coverage they are
>not going to work on improving it.
>
>
Well, there is scratch your own itch and then there is the "Apache Way".
>From http://www.apache.org/foundation/how-it-works.html
<quote>
*Philosophy*
While there is not an official list, these six principles have been
cited as the core beliefs of philosophy behind the foundation, which is
normally referred to as "The Apache Way":
- collaborative software development
- commercial-friendly standard license
- consistently high quality software
- respectful, honest, technical-based interaction
- faithful implementation of standards
- security as a mandatory feature
All of the ASF projects share these principles.
</quote>
Code coverage, reducing bug backlog and addressing functional testing
and security issues are part of our responsibility as an ASF
project. Where do we rate for each of the six principles on a scale
of 1-10 for embedded and client? Where should we be? How do we get there?
Kathey
Re: Monitoring/improving code coverage (was Re: code coverage results
for trunk - svn revision no 390306)
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
I think it would be great to have more awareness about code coverage,
esepcially if it gets worse. Getting people to work on it is a
different question, and I agree it may be difficult. I know I would be
alarmed if someone checks in a complicated feature and the code coverage
for those packages is say 10%, and at a minimum a discussion should
ensue. But as it stands we don't even know when that happens.
I liked having a goal for each release of improving the code coverage by
some modest amount -- incremental improvement.
Another area where I think there would be value in increasing awareness
is if we have a complexity analysis tool and some package jumps in
complexity by 50% after a checkin... But that's another itch for
another email thread.
David
Daniel John Debrunner wrote:
> David W. Van Couvering wrote:
>
>
>>I like the idea of having it as a release barrier, and I also like the
>>idea of getting an email saying "Code Coverage Regression" and printing
>>out the package(s) that have regressed below a low-water mark.
>
>
> I'm not sure a release barrier will work. If the coverage is low in a
> certain area and no-one has the itch to work on it then is there no release?
>
> Think about how the coverage gets low in the first place. Someone
> contributes an improvement with some amount of testing.
>
> I think it's reasonable to reject such a contribution if there are
> no-tests. Without tests there is no easy way for a committer to
> determine if the feature even works.
>
> Now if tests are contributed that shows the feature basically works, but
> have low code coverage, is there really a justification to reject the
> contribution? The feature basically works according to the original
> contributor's itch, it's someone else's itch that more tests exist. One
> can always request more tests from the contributor, but I'm not sure you
> can force it.
>
>
>>What I am at a loss for is what the low-water mark should be. I think
>>whatever we choose, we are going to have some immediate regressions.
>>Then the question becomes, how much work are we willing to put into this
>>to get it fixed.
>>
>>One approach that comes to mind is to set a reachable goal for each
>>release as a step along the way to our ultimate goal. For right now, a
>>regression could be if any package goes 10% below what our current
>>baseline is. Then we try to raise the water each release and re-set our
>>baseline.
>
>
> Not sure how we can get people to scratch the code coverage itch. It
> seems we can't get a lot of folks interested in fixing the 150+ bugs out
> there, never mind writing more tests that might uncover more bugs. I
> would love it if we could find a way.
>
> Bottom line is that if people don't care about code coverage they are
> not going to work on improving it.
>
> Dan.
>
>
Re: Monitoring/improving code coverage (was Re: code coverage results
for trunk - svn revision no 390306)
Posted by Daniel John Debrunner <dj...@apache.org>.
David W. Van Couvering wrote:
> I like the idea of having it as a release barrier, and I also like the
> idea of getting an email saying "Code Coverage Regression" and printing
> out the package(s) that have regressed below a low-water mark.
I'm not sure a release barrier will work. If the coverage is low in a
certain area and no-one has the itch to work on it then is there no release?
Think about how the coverage gets low in the first place. Someone
contributes an improvement with some amount of testing.
I think it's reasonable to reject such a contribution if there are
no-tests. Without tests there is no easy way for a committer to
determine if the feature even works.
Now if tests are contributed that shows the feature basically works, but
have low code coverage, is there really a justification to reject the
contribution? The feature basically works according to the original
contributor's itch, it's someone else's itch that more tests exist. One
can always request more tests from the contributor, but I'm not sure you
can force it.
> What I am at a loss for is what the low-water mark should be. I think
> whatever we choose, we are going to have some immediate regressions.
> Then the question becomes, how much work are we willing to put into this
> to get it fixed.
>
> One approach that comes to mind is to set a reachable goal for each
> release as a step along the way to our ultimate goal. For right now, a
> regression could be if any package goes 10% below what our current
> baseline is. Then we try to raise the water each release and re-set our
> baseline.
Not sure how we can get people to scratch the code coverage itch. It
seems we can't get a lot of folks interested in fixing the 150+ bugs out
there, never mind writing more tests that might uncover more bugs. I
would love it if we could find a way.
Bottom line is that if people don't care about code coverage they are
not going to work on improving it.
Dan.
Re: Monitoring/improving code coverage (was Re: code coverage results
for trunk - svn revision no 390306)
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
I like the idea of having it as a release barrier, and I also like the
idea of getting an email saying "Code Coverage Regression" and printing
out the package(s) that have regressed below a low-water mark.
What I am at a loss for is what the low-water mark should be. I think
whatever we choose, we are going to have some immediate regressions.
Then the question becomes, how much work are we willing to put into this
to get it fixed.
One approach that comes to mind is to set a reachable goal for each
release as a step along the way to our ultimate goal. For right now, a
regression could be if any package goes 10% below what our current
baseline is. Then we try to raise the water each release and re-set our
baseline.
David
Rick Hillegas wrote:
> In previous lives, I've seen code-coverage metrics generated on, say, a
> monthly basis and used as a release barrier. I do not think they are
> appropriate as a barrier to checkin.
>
> Regards,
> -Rick
>
> Kathey Marsden wrote:
>
>> David W. Van Couvering wrote:
>>
>>
>>
>>> Did I ask this before? Do we want to agree upon a "low water mark"
>>> for code coverage and send out a "Quality Regression" email if our
>>> testing coverage falls below that mark? I think this would have a lot
>>> of value.
>>>
>>
>>
>> This sounds like an interesting idea. Code coverage is an important
>> quality data point. What kind of granularity would it have? Would it
>> be just the overall number or would individual packages or files be
>> flagged? Also for areas that have poor coverage, how could we
>> encourage numbers to be brought up before or during enhancements?
>>
>> Kathey
>>
>>
>>
>>
>>
>
Re: Monitoring/improving code coverage (was Re: code coverage results
for trunk - svn revision no 390306)
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
Backing out the code was a suggestion, not a rule. What is appropriate
I think depends upon the situation.
My main point is it would be good to have the information made available
to us in a timely and "in-your-face" way -- we don't have to go
searching for it, and we find out right away -- so we can discuss it as
a community and try to make the right decision.
David
Daniel John Debrunner wrote:
> David W. Van Couvering wrote:
>
>
>>I think the same could be done for code coverage regressions. If a new
>>chunk of code is checked in and the numbers drop way way down for a
>>given module, I think that is cause for concern and a committer could
>>reasonably insist that the feature is not sufficiently tested and back
>>out the change, or at least block any future checkins in that area until
>>enough tests are provided.
>
>
> So we could have applied this rule to JDBC 4.0 checkins. JDBC 4.0 code
> was checked in, the code coverage numbers decreased and could not
> increase until the tests could run under JDBC 4.0.
>
> In this case requiring the JDBC 4.0 changes be backed out until all the
> tests ran under JBDC 4.0 seems too drastic. Goes against the model of
> incremental development.
>
> Dan.
>
>
Re: Monitoring/improving code coverage (was Re: code coverage results
for trunk - svn revision no 390306)
Posted by Daniel John Debrunner <dj...@apache.org>.
David W. Van Couvering wrote:
> I think the same could be done for code coverage regressions. If a new
> chunk of code is checked in and the numbers drop way way down for a
> given module, I think that is cause for concern and a committer could
> reasonably insist that the feature is not sufficiently tested and back
> out the change, or at least block any future checkins in that area until
> enough tests are provided.
So we could have applied this rule to JDBC 4.0 checkins. JDBC 4.0 code
was checked in, the code coverage numbers decreased and could not
increase until the tests could run under JDBC 4.0.
In this case requiring the JDBC 4.0 changes be backed out until all the
tests ran under JBDC 4.0 seems too drastic. Goes against the model of
incremental development.
Dan.
Re: Monitoring/improving code coverage (was Re: code coverage results
for trunk - svn revision no 390306)
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
I think the model we are following with functionality regressions works
well. We are not necessarily preventing future checkins unless things
are perceived to be Out of Hand by one or more committers.
I think the same could be done for code coverage regressions. If a new
chunk of code is checked in and the numbers drop way way down for a
given module, I think that is cause for concern and a committer could
reasonably insist that the feature is not sufficiently tested and back
out the change, or at least block any future checkins in that area until
enough tests are provided.
So I propose having a flag raised when numbers go say 20% below current
baseline for a given package, and then it is up to the committers to
decide what action is necessary.
David
Kathey Marsden wrote:
> Rick Hillegas wrote:
>
>
>>In previous lives, I've seen code-coverage metrics generated on, say,
>>a monthly basis and used as a release barrier. I do not think they are
>>appropriate as a barrier to checkin.
>>
>
> I think since we seem to be going for very long release cycles instead
> of the release early, release often model, release barriers are not very
> effective for maintaining quality on the trunk. Also in open source,
> developers come and go so the wait until release model doesn't tend to
> work that well in that regard either. If Linux could release a new
> kernel twice a day, I tend to think that the trunk should be "ready
> for release" at any time for completed functionality and even
> incremental functionality could be covered.
>
> Kathey
>
>
Re: Monitoring/improving code coverage (was Re: code coverage results
for trunk - svn revision no 390306)
Posted by Kathey Marsden <km...@sbcglobal.net>.
Rick Hillegas wrote:
> In previous lives, I've seen code-coverage metrics generated on, say,
> a monthly basis and used as a release barrier. I do not think they are
> appropriate as a barrier to checkin.
>
I think since we seem to be going for very long release cycles instead
of the release early, release often model, release barriers are not very
effective for maintaining quality on the trunk. Also in open source,
developers come and go so the wait until release model doesn't tend to
work that well in that regard either. If Linux could release a new
kernel twice a day, I tend to think that the trunk should be "ready
for release" at any time for completed functionality and even
incremental functionality could be covered.
Kathey
Re: Monitoring/improving code coverage (was Re: code coverage results
for trunk - svn revision no 390306)
Posted by Rick Hillegas <Ri...@Sun.COM>.
In previous lives, I've seen code-coverage metrics generated on, say, a
monthly basis and used as a release barrier. I do not think they are
appropriate as a barrier to checkin.
Regards,
-Rick
Kathey Marsden wrote:
>David W. Van Couvering wrote:
>
>
>
>>Did I ask this before? Do we want to agree upon a "low water mark"
>>for code coverage and send out a "Quality Regression" email if our
>>testing coverage falls below that mark? I think this would have a lot
>>of value.
>>
>>
>
>This sounds like an interesting idea. Code coverage is an important
>quality data point. What kind of granularity would it have? Would it
>be just the overall number or would individual packages or files be
>flagged? Also for areas that have poor coverage, how could we
>encourage numbers to be brought up before or during enhancements?
>
>Kathey
>
>
>
>
>