Contribution of/repository for testing infrastructure tools (was Re: Running a Derby JUnit test using JUnit directly)

[Kathey Marsden <km...@sbcglobal.net>]

Vemund Ostgaard wrote:

> a tool that we use internally to distribute files to remote machines 
> for testing, and as such didn't seem interesting to the community.

Actually I am curious about this tool and think maybe  the tool itself 
and other testing infrastructure tools  might really be great 
contribution to the community.    We all benefit from the testing 
results of this the efficient QA efforts in Norway. I wonder  might you 
consider contributing  these tools  so that they could be used in a more 
general purpose way and enhanced by all for the general benefit of the 
community?

I am not sure exactly where these tools would  be checked in,  but are 
certainly worthy of a place.  I recall that at one point a snapshot was 
made of the testing report scripts and then  copied and forked for the 
IBM reports and then enhanced locally in Norway. Since they were not 
checked in, they  they could not be enhanced open source style and used 
generally.  I think if contributed, they could be enhanced ultimately it 
would be easy for anyone  running tests to post them in the same format 
and link them off:
http://db.apache.org/derby/derby_tests.html

Kathey

Re: Contribution of/repository for testing infrastructure tools (was Re: Running a Derby JUnit test using JUnit directly)

[Daniel John Debrunner <dj...@apache.org>]

Ole Solberg wrote:


> I will start by moving the scripts into my Derby sandbox, e.g. under
> tools/testing/reporting as suggested, and start the cleanup.

Great.

> 
> I also have plans to make the failure analysis more "intelligent", but
> don't know yet how high on the priority list that ends up.

That's the point of making them open, you can throw out ideas and if you
can't get to them, maybe someone else can.

Thanks,
Dan.

Re: Contribution of/repository for testing infrastructure tools (was Re: Running a Derby JUnit test using JUnit directly)

[Ole Solberg <Ol...@Sun.COM>]

Ole Solberg wrote:
> Andrew McIntyre wrote:
>> On 9/6/06, Kathey Marsden <km...@sbcglobal.net> wrote:
>>
>>>
>>> Actually I am curious about this tool and think maybe  the tool itself
>>> and other testing infrastructure tools might really be great
>>> contribution to the community.
>>
> 
> The internal tool we use for testing originated as a tool to handle HA
> database testing. It allows us to define test-rigs consisting of servers
> and clients, and run sets of tests on several test-rigs in parallel.
> Tests are defined in XML-files which essentially allow us to specify
> setups and teardowns, e.g. starting and stopping databases, and JUnit
> based test methods which are run from clients against the database(s).
> Results are reported to a test result database which eases creation of
> different kinds of test reports for testers, developers and managers.
> 
> 
> I am afraid this tool would need quite some work to be open-sourced.
> We would have to do both documentation and code cleanup, removing
> dependencies to other parts of our testing infrastructure, as well as
> resolution of legal issues. We would very much like to do this, but we
> do not have resources for it at the moment.
> 
> The reporting scripts, which has already been donated in DERBY-812 in an
> earlier version, is a different thing. These are built for creating
> reports from the Derby test harness results and are independent of the
> rest of our testing infrastructure. We have also made some simple
> interfacing scripts to extract results from our internal test
> infrastructure tool into these reports.
> 
> 
>>
>> +10 - definitely would be a great contribution to the community. It
>> would allow for collaboration and enhancement of the scripts to make
>> testing easier for everyone in the community.
> Thanks!! That's nice to hear!
> 
>>
>>> I am not sure exactly where these tools would be checked in, but are
>>> certainly worthy of a place.
>>
>>
>> Tools for testing should go somewhere under tools/testing. :-)  Maybe
>> tools/testing/reporting or tools/testing/tinderbox (if applicable) or
>> something similar.
>>
>> I could check in the scripts from DERBY-812, but maybe there's a newer
>> version that's more worthy of being committed?
> 
> The newer version of the reporting scripts which also attempts to do
> some failure analysis could in the short-term be uploaded as an update
> to DERBY-812.
> I would definitely like to put this into the repository so as to allow
> enhancements of the scripts by the community!
> Anyway, I would first like to go over them to generalize possible
> site-specific stuff which may have sneaked in.
> 
> I will start by moving the scripts into my Derby sandbox, e.g. under
> tools/testing/reporting as suggested, and start the cleanup.
> 

I have re-opened DERBY-812 and uploaded a patch with a newer,
cleaned-up, version of the scripts.

The new version goes into 'tools/testing/reporting/'.
The main difference since the previous version is the addition of
a simple failure analysis, and the possibility to create plots of
testsuite duration using gnuplot.

The patch contains README files explaining how to use the scripts.

> 
> I also have plans to make the failure analysis more "intelligent", but
> don't know yet how high on the priority list that ends up.
> 
>>
>> andrew
> 
> 


-- 
Ole Solberg, Database Technology Group,
Sun Microsystems, Trondheim, Norway

Re: Contribution of/repository for testing infrastructure tools (was Re: Running a Derby JUnit test using JUnit directly)

[Ole Solberg <Ol...@Sun.COM>]

Andrew McIntyre wrote:
> On 9/6/06, Kathey Marsden <km...@sbcglobal.net> wrote:
> 
>>
>> Actually I am curious about this tool and think maybe  the tool itself
>> and other testing infrastructure tools might really be great
>> contribution to the community.
> 

The internal tool we use for testing originated as a tool to handle HA
database testing. It allows us to define test-rigs consisting of servers
and clients, and run sets of tests on several test-rigs in parallel.
Tests are defined in XML-files which essentially allow us to specify
setups and teardowns, e.g. starting and stopping databases, and JUnit
based test methods which are run from clients against the database(s).
Results are reported to a test result database which eases creation of
different kinds of test reports for testers, developers and managers.


I am afraid this tool would need quite some work to be open-sourced.
We would have to do both documentation and code cleanup, removing
dependencies to other parts of our testing infrastructure, as well as
resolution of legal issues. We would very much like to do this, but we
do not have resources for it at the moment.

The reporting scripts, which has already been donated in DERBY-812 in an
earlier version, is a different thing. These are built for creating
reports from the Derby test harness results and are independent of the
rest of our testing infrastructure. We have also made some simple
interfacing scripts to extract results from our internal test
infrastructure tool into these reports.


> 
> +10 - definitely would be a great contribution to the community. It
> would allow for collaboration and enhancement of the scripts to make
> testing easier for everyone in the community.
Thanks!! That's nice to hear!

> 
>> I am not sure exactly where these tools would be checked in, but are
>> certainly worthy of a place.
> 
> 
> Tools for testing should go somewhere under tools/testing. :-)  Maybe
> tools/testing/reporting or tools/testing/tinderbox (if applicable) or
> something similar.
> 
> I could check in the scripts from DERBY-812, but maybe there's a newer
> version that's more worthy of being committed?

The newer version of the reporting scripts which also attempts to do
some failure analysis could in the short-term be uploaded as an update
to DERBY-812.
I would definitely like to put this into the repository so as to allow
enhancements of the scripts by the community!
Anyway, I would first like to go over them to generalize possible 
site-specific stuff which may have sneaked in.

I will start by moving the scripts into my Derby sandbox, e.g. under 
tools/testing/reporting as suggested, and start the cleanup.


I also have plans to make the failure analysis more "intelligent", but 
don't know yet how high on the priority list that ends up.

> 
> andrew


-- 
Ole Solberg, Database Technology Group,
Sun Microsystems, Trondheim, Norway

Re: Contribution of/repository for testing infrastructure tools (was Re: Running a Derby JUnit test using JUnit directly)

[Andrew McIntyre <mc...@gmail.com>]

On 9/6/06, Kathey Marsden <km...@sbcglobal.net> wrote:
>
> Actually I am curious about this tool and think maybe  the tool itself
> and other testing infrastructure tools might really be great
> contribution to the community.

+10 - definitely would be a great contribution to the community. It
would allow for collaboration and enhancement of the scripts to make
testing easier for everyone in the community.

> I am not sure exactly where these tools would be checked in, but are
> certainly worthy of a place.

Tools for testing should go somewhere under tools/testing. :-)  Maybe
tools/testing/reporting or tools/testing/tinderbox (if applicable) or
something similar.

I could check in the scripts from DERBY-812, but maybe there's a newer
version that's more worthy of being committed?

andrew

Re: Contribution of/repository for testing infrastructure tools (was Re: Running a Derby JUnit test using JUnit directly)

[Mike Matrigali <mi...@sbcglobal.net>]

I second this, I really appreciate the latest test results being
posted.  It would be nice if I could use these tools to quickly
look at test results from my own machine.  It would be really helpful
to automatically know if an test failure is likely a known issue as
the current reports seem to automatically be able to determine.

Kathey Marsden wrote:
> Vemund Ostgaard wrote:
> 
>> a tool that we use internally to distribute files to remote machines 
>> for testing, and as such didn't seem interesting to the community.
> 
> 
> Actually I am curious about this tool and think maybe  the tool itself 
> and other testing infrastructure tools  might really be great 
> contribution to the community.    We all benefit from the testing 
> results of this the efficient QA efforts in Norway. I wonder  might you 
> consider contributing  these tools  so that they could be used in a more 
> general purpose way and enhanced by all for the general benefit of the 
> community?
> 
> I am not sure exactly where these tools would  be checked in,  but are 
> certainly worthy of a place.  I recall that at one point a snapshot was 
> made of the testing report scripts and then  copied and forked for the 
> IBM reports and then enhanced locally in Norway. Since they were not 
> checked in, they  they could not be enhanced open source style and used 
> generally.  I think if contributed, they could be enhanced ultimately it 
> would be easy for anyone  running tests to post them in the same format 
> and link them off:
> http://db.apache.org/derby/derby_tests.html
> 
> Kathey
> 
> 
> 
> 
> 
> 
> 
> 
>