Re: Contribution of/repository for testing infrastructure tools (was Re: Running a Derby JUnit test using JUnit directly)

[Ole Solberg <Ol...@Sun.COM>]

Ole Solberg wrote:
> Andrew McIntyre wrote:
>> On 9/6/06, Kathey Marsden <km...@sbcglobal.net> wrote:
>>
>>>
>>> Actually I am curious about this tool and think maybe  the tool itself
>>> and other testing infrastructure tools might really be great
>>> contribution to the community.
>>
> 
> The internal tool we use for testing originated as a tool to handle HA
> database testing. It allows us to define test-rigs consisting of servers
> and clients, and run sets of tests on several test-rigs in parallel.
> Tests are defined in XML-files which essentially allow us to specify
> setups and teardowns, e.g. starting and stopping databases, and JUnit
> based test methods which are run from clients against the database(s).
> Results are reported to a test result database which eases creation of
> different kinds of test reports for testers, developers and managers.
> 
> 
> I am afraid this tool would need quite some work to be open-sourced.
> We would have to do both documentation and code cleanup, removing
> dependencies to other parts of our testing infrastructure, as well as
> resolution of legal issues. We would very much like to do this, but we
> do not have resources for it at the moment.
> 
> The reporting scripts, which has already been donated in DERBY-812 in an
> earlier version, is a different thing. These are built for creating
> reports from the Derby test harness results and are independent of the
> rest of our testing infrastructure. We have also made some simple
> interfacing scripts to extract results from our internal test
> infrastructure tool into these reports.
> 
> 
>>
>> +10 - definitely would be a great contribution to the community. It
>> would allow for collaboration and enhancement of the scripts to make
>> testing easier for everyone in the community.
> Thanks!! That's nice to hear!
> 
>>
>>> I am not sure exactly where these tools would be checked in, but are
>>> certainly worthy of a place.
>>
>>
>> Tools for testing should go somewhere under tools/testing. :-)  Maybe
>> tools/testing/reporting or tools/testing/tinderbox (if applicable) or
>> something similar.
>>
>> I could check in the scripts from DERBY-812, but maybe there's a newer
>> version that's more worthy of being committed?
> 
> The newer version of the reporting scripts which also attempts to do
> some failure analysis could in the short-term be uploaded as an update
> to DERBY-812.
> I would definitely like to put this into the repository so as to allow
> enhancements of the scripts by the community!
> Anyway, I would first like to go over them to generalize possible
> site-specific stuff which may have sneaked in.
> 
> I will start by moving the scripts into my Derby sandbox, e.g. under
> tools/testing/reporting as suggested, and start the cleanup.
> 

I have re-opened DERBY-812 and uploaded a patch with a newer,
cleaned-up, version of the scripts.

The new version goes into 'tools/testing/reporting/'.
The main difference since the previous version is the addition of
a simple failure analysis, and the possibility to create plots of
testsuite duration using gnuplot.

The patch contains README files explaining how to use the scripts.

> 
> I also have plans to make the failure analysis more "intelligent", but
> don't know yet how high on the priority list that ends up.
> 
>>
>> andrew
> 
> 


-- 
Ole Solberg, Database Technology Group,
Sun Microsystems, Trondheim, Norway