You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by "Bernt M. Johnsen" <Be...@Sun.COM> on 2005/05/26 13:27:23 UTC
Running derbyall before submitting patches
Hi all,
The last week or two, derbyall has failed a lot the time. This makes
it hard to verify that I haven't goofed up things when doing changes
in the code (I run derbyall daily as a quality measure). To be able to
work efficiently, many of us are dependent on a reliable test suite.
So, I urge everyone who submits a patch to the Derby to run derbyall
on at least one platform/vm and not only the "relevant" tests for the
patch, even if you "know" that it should not be necessary. I think
this would save us all a lot of work (even more time saved for the
committers, I guess :-).
I know this won't guarantee that derbyall runs correctly all the time,
but the frequency of problems should at least go down.
Thanks
--
Bernt Marius Johnsen, Database Technology Group,
Sun Microsystems, Trondheim, Norway
Re: Running derbyall before submitting patches
Posted by Daniel John Debrunner <dj...@debrunners.com>.
Bernt M. Johnsen wrote:
> Hi all,
>
> The last week or two, derbyall has failed a lot the time. This makes
> it hard to verify that I haven't goofed up things when doing changes
> in the code (I run derbyall daily as a quality measure). To be able to
> work efficiently, many of us are dependent on a reliable test suite.
>
> So, I urge everyone who submits a patch to the Derby to run derbyall
> on at least one platform/vm and not only the "relevant" tests for the
> patch, even if you "know" that it should not be necessary. I think
> this would save us all a lot of work (even more time saved for the
> committers, I guess :-).
This is good advice, especially for people new to the project. Knowing
which tests might be affected by a fix comes with experience and
experience is obtained by running derbyall and seeing what fails. In
some ways it's easier just to run derbyall, I try to do that for all my
contributions, usually running overnight.
I think Satheesh and myself have requested that with any patch the
contributor should indicate which test suites have been run. And not
just for the first submission of a patch, if the patch is reworked then
the e-mail should indicate again which tests have been run. I tend to
only look at the most recent submission for a patch, not the complete
history. I think Satheesh also said that he would only look at any
patches that did indicate which tests had been run.
Another job for the committers is to stop applying any patches if the
number of tests failing on derbyall passes some threshold. Then only
commit (or revert) patches that resolve the failures.
Looking at the latest results from the Sun group, we have an average
pass rate of around 98.7% (~8 tests failing) and minimum pass rate of
around 98.0% (12 tests).
Not sure what the threshold should be, but we may be close at this
point. One or two tests failing is somewhat easy to "work-around" when
submitting patches, 8-12 is harder (just to remember which fail and
compare). Our goal should be 100% passing all the time.
Dan.
Re: Running derbyall before submitting patches
Posted by "Jean T. Anderson" <jt...@bristowhill.com>.
Myrna van Lunteren wrote:
> On 5/26/05, *Jean T. Anderson* <jta@bristowhill.com
> <ma...@bristowhill.com>> wrote:
>
> How about a general "Tests" link on the community tab to a page that
> starts listing available test resources. So far the only one is Sun
> DBTG
> test results. Is that correct?
>
> -jean
>
> yes, that's right. (sorry-I keep promising...)
>
that wasn't prodding you! :-) Anyone should feel free to post a link to
their test results.
OK, so I just added a general "Tests" link on the community tab to a new
page, derby_tests, that starts listing available test resources. I also
added a link to this new page in the
derby_downloads.html#How+to+test+Derby section.
Committed revision 178648. Changes on the web site will become visible
with the next rsync to www.apache.org (currently scheduled for every 4
hours).
-jean
Re: Running derbyall before submitting patches
Posted by Myrna van Lunteren <m....@gmail.com>.
On 5/26/05, Jean T. Anderson <jt...@bristowhill.com> wrote:
>
> How about a general "Tests" link on the community tab to a page that
> starts listing available test resources. So far the only one is Sun DBTG
> test results. Is that correct?
>
> -jean
>
> yes, that's right. (sorry-I keep promising...)
>
Re: Running derbyall before submitting patches
Posted by "Jean T. Anderson" <jt...@bristowhill.com>.
How about a general "Tests" link on the community tab to a page that
starts listing available test resources. So far the only one is Sun DBTG
test results. Is that correct?
-jean
TomohitoNakayama wrote:
> Hello.
>
> I did'nt notice the link ...
> Thank you for notifying me.
>
> I think community tab is good place for this kind of information too.
> There exists BUGS and To-Do already.
> It is natural for results of test page to exist there.
>
> Best regards.
>
> /*
>
> Tomohito Nakayama
> tomonaka@basil.ocn.ne.jp <ma...@basil.ocn.ne.jp>
> tomohito@rose.zero.ad.jp <ma...@rose.zero.ad.jp>
>
> Naka
> http://www5.ocn.ne.jp/~tomohito/TopPage.html
>
> */
>
> ----- Original Message -----
> *From:* Myrna van Lunteren <ma...@gmail.com>
> *To:* Derby Development <ma...@db.apache.org>
> *Sent:* Thursday, May 26, 2005 11:17 PM
> *Subject:* Re: Running derbyall before submitting patches
>
> On 5/26/05, *TomohitoNakayama* <tomonaka@basil.ocn.ne.jp
> <ma...@basil.ocn.ne.jp>> wrote:
>
> Hello.
>
> If possible, I hope the result of the daily-derbyall to be open
> to public,
> some where on the derby site.
> Then, we can compare it to result of local-derbyall.
> // I think derbyall twice at everyone's local is little hard ....
>
> It will be index of current quality.
>
>
> Well, I had seen this commit by Jean:
>
> Added a link to the Sun DBTG test results on derby_downloads.html
> page; see
> http://mail-archives.apache.org/mod_mbox/db-derby-dev/200505.mbox/%3c428DEB6D.8010707@Sun.COM%3e
> .
>
> But I could not find the link easily.
> I think, now that I've looked for the link, maybe this should be
> under the 'community' tab rather than downloads.
>
> And I still intend to add an excerpt from the results from IBM
> nightly testing somehow.
>
> Myrna
>
> ------------------------------------------------------------------------
>
> No virus found in this incoming message.
> Checked by AVG Anti-Virus.
> Version: 7.0.322 / Virus Database: 266.11.17 - Release Date: 2005/05/25
>
>
> ------------------------------------------------------------------------
>
> No virus found in this outgoing message.
> Checked by AVG Anti-Virus.
> Version: 7.0.322 / Virus Database: 266.11.17 - Release Date: 2005/05/25
Re: Running derbyall before submitting patches
Posted by TomohitoNakayama <to...@basil.ocn.ne.jp>.
Hello.
I did'nt notice the link ...
Thank you for notifying me.
I think community tab is good place for this kind of information too.
There exists BUGS and To-Do already.
It is natural for results of test page to exist there.
Best regards.
/*
Tomohito Nakayama
tomonaka@basil.ocn.ne.jp
tomohito@rose.zero.ad.jp
Naka
http://www5.ocn.ne.jp/~tomohito/TopPage.html
*/
----- Original Message -----
From: Myrna van Lunteren
To: Derby Development
Sent: Thursday, May 26, 2005 11:17 PM
Subject: Re: Running derbyall before submitting patches
On 5/26/05, TomohitoNakayama <to...@basil.ocn.ne.jp> wrote:
Hello.
If possible, I hope the result of the daily-derbyall to be open to
public,
some where on the derby site.
Then, we can compare it to result of local-derbyall.
// I think derbyall twice at everyone's local is little hard ....
It will be index of current quality.
Well, I had seen this commit by Jean:
Added a link to the Sun DBTG test results on derby_downloads.html page;
see
http://mail-archives.apache.org/mod_mbox/db-derby-dev/200505.mbox/%3c428DEB6D.8010707@Sun.COM%3e
.
But I could not find the link easily.
I think, now that I've looked for the link, maybe this should be under the
'community' tab rather than downloads.
And I still intend to add an excerpt from the results from IBM nightly
testing somehow.
Myrna
------------------------------------------------------------------------------
No virus found in this incoming message.
Checked by AVG Anti-Virus.
Version: 7.0.322 / Virus Database: 266.11.17 - Release Date: 2005/05/25
Re: Running derbyall before submitting patches
Posted by Myrna van Lunteren <m....@gmail.com>.
On 5/26/05, TomohitoNakayama <to...@basil.ocn.ne.jp> wrote:
>
> Hello.
>
> If possible, I hope the result of the daily-derbyall to be open to public,
> some where on the derby site.
> Then, we can compare it to result of local-derbyall.
> // I think derbyall twice at everyone's local is little hard ....
>
> It will be index of current quality.
Well, I had seen this commit by Jean:
Added a link to the Sun DBTG test results on derby_downloads.html page;
see
http://mail-archives.apache.org/mod_mbox/db-derby-dev/200505.mbox/%3c428DEB6D.8010707@Sun.COM%3e.
But I could not find the link easily.
I think, now that I've looked for the link, maybe this should be under the
'community' tab rather than downloads.
And I still intend to add an excerpt from the results from IBM nightly
testing somehow.
Myrna
Re: Running derbyall before submitting patches
Posted by TomohitoNakayama <to...@basil.ocn.ne.jp>.
Hello.
If possible, I hope the result of the daily-derbyall to be open to public,
some where on the derby site.
Then, we can compare it to result of local-derbyall.
// I think derbyall twice at everyone's local is little hard ....
It will be index of current quality.
Best regards.
/*
Tomohito Nakayama
tomonaka@basil.ocn.ne.jp
tomohito@rose.zero.ad.jp
Naka
http://www5.ocn.ne.jp/~tomohito/TopPage.html
*/
----- Original Message -----
From: "Bernt M. Johnsen" <Be...@Sun.COM>
To: <de...@db.apache.org>
Sent: Thursday, May 26, 2005 8:27 PM
Subject: Running derbyall before submitting patches
> Hi all,
>
> The last week or two, derbyall has failed a lot the time. This makes
> it hard to verify that I haven't goofed up things when doing changes
> in the code (I run derbyall daily as a quality measure). To be able to
> work efficiently, many of us are dependent on a reliable test suite.
>
> So, I urge everyone who submits a patch to the Derby to run derbyall
> on at least one platform/vm and not only the "relevant" tests for the
> patch, even if you "know" that it should not be necessary. I think
> this would save us all a lot of work (even more time saved for the
> committers, I guess :-).
>
> I know this won't guarantee that derbyall runs correctly all the time,
> but the frequency of problems should at least go down.
>
> Thanks
> --
> Bernt Marius Johnsen, Database Technology Group,
> Sun Microsystems, Trondheim, Norway
>
>
>
> --
> No virus found in this incoming message.
> Checked by AVG Anti-Virus.
> Version: 7.0.322 / Virus Database: 266.11.17 - Release Date: 2005/05/25
>
>
--
No virus found in this outgoing message.
Checked by AVG Anti-Virus.
Version: 7.0.322 / Virus Database: 266.11.17 - Release Date: 2005/05/25
Re: Running derbyall before submitting patches
Posted by Andrew McIntyre <mc...@gmail.com>.
On 5/31/05, David Van Couvering <Da...@sun.com> wrote:
> This is going to be an ongoing issue. Would it be possible to modify
> JIRA for Derby to have a workflow state that indicates a patch has been
> submitted but not committed? That way committers could query on this to
> find out what is out there.
I think marking a JIRA entry as "In Progress," by clicking Start
Progress on the right hand Workflow Actions list, once a patch has
been submitted would be a good way to track patches without needing
any modification to JIRA, just diligence on the part of those posting
patches.
andrew
Re: Running derbyall before submitting patches
Posted by David Van Couvering <Da...@Sun.COM>.
This is going to be an ongoing issue. Would it be possible to modify
JIRA for Derby to have a workflow state that indicates a patch has been
submitted but not committed? That way committers could query on this to
find out what is out there.
David
>
> We have so many patches in different states, I am losing track and am
> admittedly behind. I was wondering if someone might be willing to
> compile a list of the patches from this week that are waiting for
> committer attention and list
>
> developer
> component
> summary
> tests run
> reviewers/testers
>
> for each patch. Priority attention goes to the patch with the
> reviews/testers not blank and of course to the person that makes the
> list #:)
>
> Thanks
>
> Kathey
>
>
Re: Running derbyall before submitting patches
Posted by Kathey Marsden <km...@sbcglobal.net>.
David Van Couvering wrote:
> I think we need to have at least *one* person run derbyall besides
> the submitter of the patch. This does not necessarily have to be a
> committer.
>
> I'm not sure how this could be managed; perhaps each developer could
> "buddy up" with someone, and each of them agree to test out each
> other's patches...
>
There is also a great opportunity for lurkers, students, and others who
are interested in Derby but not making changes, to add great value and
learn by applying, reviewing, and testing out patches.
> We should probably write up a "checkin checklist", vote on it, and add
> it to the web site...
Sounds good too.
We have so many patches in different states, I am losing track and am
admittedly behind. I was wondering if someone might be willing to
compile a list of the patches from this week that are waiting for
committer attention and list
developer
component
summary
tests run
reviewers/testers
for each patch. Priority attention goes to the patch with the
reviews/testers not blank and of course to the person that makes the
list #:)
Thanks
Kathey
Re: Running derbyall before submitting patches
Posted by David Van Couvering <da...@vancouvering.com>.
Hm, it's tricky to just count on the developer submitting the patch to
run derbyall. A common mistake, for example, is that the developer
hasn't done 'svn add' for a new file, so their tests work, but the patch
submitted doesn't have all the necessary files. Also, the developer may
have old classes sitting around or who knows what funny stuff in their
environment that makes it work for them. Also also, the developer
usually tests on a specific VM and OS, and it would be great if at least
one other VM or OS were tried before committing.
What we *really* need is a good full nightly regression across all
platforms and VMS, but until we have that, I think we need to have at
least *one* person run derbyall besides the submitter of the patch.
This does not necessarily have to be a committer.
I'm not sure how this could be managed; perhaps each developer could
"buddy up" with someone, and each of them agree to test out each other's
patches...
The other thing I have noticed is that sometimes a developer forgets to
run 'svn update' and 'ant clobber' and 'ant all' before running tests.
We should probably write up a "checkin checklist", vote on it, and add
it to the web site...
David
P.S. I did run derbyall on the two recent patches I submitted. I'll add
that to my comments when I submit patches from now on.
Mike Matrigali wrote:
> me too. I run derbyALL for any thing I submit myself, but as
> satheesh has said lately I have not been able to run derbyAll for
> all the patches I have been committing for others. I do a full
> rebuild, review the change and
> run tests that look specific to the patch, and count on the submitter
> to have run all tests. Also please check the results posted to
> the list the day after your patch goes in, to see if you may have
> introduced a problem in a different environment.
>
> Remember that there are now a lot of developers out there depending
> on clean test runs, so they can figure out what affect their changes
> are having.
>
> Satheesh Bandaram wrote:
>
>
>>+10, if there is anything like that. :-)
>>
>>I used to run 'derbyAll' for each patch I tried to commit, but not able
>>to do that anymore with so many patches being submitted these days. It
>>should be everyone's responsibility to run all tests before submitting a
>>patch. I will start asking for list of tests run before committing
>>patches, if not provided already.
>>
>>Also, new people tend to miss updating multiple versions of master
>>files, if the test has different masters. (say for DerbyNet or
>>DerbyNetClient frameworks) It is also possible to have different master
>>files based on JVM being used or version.
>>
>>Satheesh
>>
>>Bernt M. Johnsen wrote:
>>
>>
>>
>>>Hi all,
>>>
>>>The last week or two, derbyall has failed a lot the time. This makes
>>>it hard to verify that I haven't goofed up things when doing changes
>>>in the code (I run derbyall daily as a quality measure). To be able to
>>>work efficiently, many of us are dependent on a reliable test suite.
>>>
>>>So, I urge everyone who submits a patch to the Derby to run derbyall
>>>on at least one platform/vm and not only the "relevant" tests for the
>>>patch, even if you "know" that it should not be necessary. I think
>>>this would save us all a lot of work (even more time saved for the
>>>committers, I guess :-).
>>>
>>>I know this won't guarantee that derbyall runs correctly all the time,
>>>but the frequency of problems should at least go down.
>>>
>>>Thanks
>>>
>>>
>>
>>
>>
Re: Running derbyall before submitting patches
Posted by Mike Matrigali <mi...@sbcglobal.net>.
me too. I run derbyALL for any thing I submit myself, but as
satheesh has said lately I have not been able to run derbyAll for
all the patches I have been committing for others. I do a full
rebuild, review the change and
run tests that look specific to the patch, and count on the submitter
to have run all tests. Also please check the results posted to
the list the day after your patch goes in, to see if you may have
introduced a problem in a different environment.
Remember that there are now a lot of developers out there depending
on clean test runs, so they can figure out what affect their changes
are having.
Satheesh Bandaram wrote:
> +10, if there is anything like that. :-)
>
> I used to run 'derbyAll' for each patch I tried to commit, but not able
> to do that anymore with so many patches being submitted these days. It
> should be everyone's responsibility to run all tests before submitting a
> patch. I will start asking for list of tests run before committing
> patches, if not provided already.
>
> Also, new people tend to miss updating multiple versions of master
> files, if the test has different masters. (say for DerbyNet or
> DerbyNetClient frameworks) It is also possible to have different master
> files based on JVM being used or version.
>
> Satheesh
>
> Bernt M. Johnsen wrote:
>
>
>>Hi all,
>>
>>The last week or two, derbyall has failed a lot the time. This makes
>>it hard to verify that I haven't goofed up things when doing changes
>>in the code (I run derbyall daily as a quality measure). To be able to
>>work efficiently, many of us are dependent on a reliable test suite.
>>
>>So, I urge everyone who submits a patch to the Derby to run derbyall
>>on at least one platform/vm and not only the "relevant" tests for the
>>patch, even if you "know" that it should not be necessary. I think
>>this would save us all a lot of work (even more time saved for the
>>committers, I guess :-).
>>
>>I know this won't guarantee that derbyall runs correctly all the time,
>>but the frequency of problems should at least go down.
>>
>>Thanks
>>
>>
>
>
>
Re: Running derbyall before submitting patches
Posted by Satheesh Bandaram <sa...@Sourcery.Org>.
+10, if there is anything like that. :-)
I used to run 'derbyAll' for each patch I tried to commit, but not able
to do that anymore with so many patches being submitted these days. It
should be everyone's responsibility to run all tests before submitting a
patch. I will start asking for list of tests run before committing
patches, if not provided already.
Also, new people tend to miss updating multiple versions of master
files, if the test has different masters. (say for DerbyNet or
DerbyNetClient frameworks) It is also possible to have different master
files based on JVM being used or version.
Satheesh
Bernt M. Johnsen wrote:
>Hi all,
>
>The last week or two, derbyall has failed a lot the time. This makes
>it hard to verify that I haven't goofed up things when doing changes
>in the code (I run derbyall daily as a quality measure). To be able to
>work efficiently, many of us are dependent on a reliable test suite.
>
>So, I urge everyone who submits a patch to the Derby to run derbyall
>on at least one platform/vm and not only the "relevant" tests for the
>patch, even if you "know" that it should not be necessary. I think
>this would save us all a lot of work (even more time saved for the
>committers, I guess :-).
>
>I know this won't guarantee that derbyall runs correctly all the time,
>but the frequency of problems should at least go down.
>
>Thanks
>
>