Using EMMA for codecoverage analysis

[Vemund Ostgaard <Ve...@Sun.COM>]

Hi,

I've been trying to run the Derby testsuites with EMMA to measure 
codecoverage, and after troubling a little I have a few questions and 
comments.

I hit some problems both with the old test harness and the new junit 
testsuite in relation to the securitymanager. I haven't analyzed it 
fully, but both seemed to be caused by the emma codebase wanting to 
access files on the disk and/or system properties.

I also stumbled on a test for derbyrun.jar in the old harness. It forks 
a java process running derbyrun.jar with the -jar option, so no matter 
what you set your CLASSPATH env to you still only have derbyrun.jar in 
the classpath. This test then failed when running with instrumented 
jarfiles as it couldn't find the classes it needed from emma.jar. There 
might be other tests with similar problems, this was just the first one 
I stumbled on when running the derbytools subsuite.

Ideally I think the tests should be run in their normal configuration 
(with security manager if that is the default), also when running 
codecoverage measurements. It would also be good if it was possible to 
run all the tests. My immediate thought was that this would require 
modifications to policy files and possibly individual tests like the 
test for derbyrun.jar, but there may be better ways to get around these 
issues that I haven't thought about. I know that EMMA is being used by 
others in the community, how did you get around these problems?

Is there in principle anything wrong with making changes to policy files 
and/or tests to make it easier to run codecoverage measurements with the 
EMMA tool?


Vemund

Re: Using EMMA for codecoverage analysis

[Vemund Ostgaard <Ve...@Sun.COM>]

Myrna van Lunteren wrote:
> On 2/21/08, Vemund Ostgaard <Ve...@sun.com> wrote:
>   
>> I'm now working my way through the extra permissions needed for
>> junit.jar and ant-junit.jar when running with instrumented jarfiles. I
>> also think some permissions are needed when running with ant 1.7 that
>> wasn't needed when running with ant 1.6.5.
>>
>> Vemund
>>
>>     
> Are you referring to what's mentioned in DERBY-3153?
>
> I recently tried out the patch there and it didn't work well for me,
> but I didn't have time to focus on it...
>   
I hadn't seen that Jira, thank you! It looks like the same problem.

My solution was a bit different, though. By setting the "tempdir" 
attribute of the <junit> task, I got the files written to the same 
directory the test is using for the rest of its files, so I only had to 
give permissions for writing to that directory. Not that it matters much 
I guess.

Vemund

Re: Using EMMA for codecoverage analysis

[Myrna van Lunteren <m....@gmail.com>]

On 2/21/08, Vemund Ostgaard <Ve...@sun.com> wrote:
> I'm now working my way through the extra permissions needed for
> junit.jar and ant-junit.jar when running with instrumented jarfiles. I
> also think some permissions are needed when running with ant 1.7 that
> wasn't needed when running with ant 1.6.5.
>
> Vemund
>
Are you referring to what's mentioned in DERBY-3153?

I recently tried out the patch there and it didn't work well for me,
but I didn't have time to focus on it...

Myrna

Re: Using EMMA for codecoverage analysis

[Vemund Ostgaard <Ve...@Sun.COM>]

Daniel John Debrunner wrote:
> Vemund Ostgaard wrote:
>> I got suites.All to run with EMMA, but I had to add 7-8 lines to 
>> derby_tests.policy and modify a couple of tests before it ran cleanly.
>>
>> I think it would be good if this worked without editing the source, 
>> so I'll make a Jira and upload a patch with these changes.
>
> It might be good to separate out the test changes, I'd be interested 
> to understand why they needed to be changed to run under code coverage.
I will do that. I've only fixed suites.All, but I saw similar cases in 
the derbyall test as well. I think the emma instrumentation affects the 
tests in two ways:

1. Running instrumented code requires emma.jar in the classpath. If a 
test forks a new process that runs instrumented code (for instance ij) 
the new process also needs emma.jar in the classpath. The same is true 
if a test creates a new classloader that loads an instrumented class, 
emma.jar is needed in that classloaders search URL as well.

2. Code instrumented by EMMA will by default write some output, which 
some tests don't like. Using -Demma.verbosity.level=silent when starting 
junit solves this for most tests, but if a test forks a new process it 
may have to set the emma.verbosity.level for that process as well if it 
is touchy about the extra output.

It was only two tests in suites.All that needed modifications, but I 
guess we might see these two issues also in the future when new tests 
are written.
>
>> I'm thinking about adding some ant tasks for instrumenting 
>> classes/jars and running the junit-tests with EMMA, making it easier 
>> to do this.
>
> +1
I hadn't run the junit tests with ant before and had some problems 
running them with ant 1.6.5, so I switched to ant 1.7 and it is working 
better.

I'm now working my way through the extra permissions needed for 
junit.jar and ant-junit.jar when running with instrumented jarfiles. I 
also think some permissions are needed when running with ant 1.7 that 
wasn't needed when running with ant 1.6.5.

Vemund

Re: Using EMMA for codecoverage analysis

[Daniel John Debrunner <dj...@apache.org>]

Vemund Ostgaard wrote:
> I got suites.All to run with EMMA, but I had to add 7-8 lines to 
> derby_tests.policy and modify a couple of tests before it ran cleanly.
> 
> I think it would be good if this worked without editing the source, so 
> I'll make a Jira and upload a patch with these changes.

It might be good to separate out the test changes, I'd be interested to 
understand why they needed to be changed to run under code coverage.

> I'm thinking about adding some ant tasks for instrumenting classes/jars 
> and running the junit-tests with EMMA, making it easier to do this.

+1

Thanks,
Dan.

Re: Using EMMA for codecoverage analysis

[Vemund Ostgaard <Ve...@Sun.COM>]

I got suites.All to run with EMMA, but I had to add 7-8 lines to 
derby_tests.policy and modify a couple of tests before it ran cleanly.

I think it would be good if this worked without editing the source, so 
I'll make a Jira and upload a patch with these changes.

I'm thinking about adding some ant tasks for instrumenting classes/jars 
and running the junit-tests with EMMA, making it easier to do this.

Vemund

Vemund Ostgaard wrote:
> Hi,
>
> I've been trying to run the Derby testsuites with EMMA to measure 
> codecoverage, and after troubling a little I have a few questions and 
> comments.
>
> I hit some problems both with the old test harness and the new junit 
> testsuite in relation to the securitymanager. I haven't analyzed it 
> fully, but both seemed to be caused by the emma codebase wanting to 
> access files on the disk and/or system properties.
>
> I also stumbled on a test for derbyrun.jar in the old harness. It 
> forks a java process running derbyrun.jar with the -jar option, so no 
> matter what you set your CLASSPATH env to you still only have 
> derbyrun.jar in the classpath. This test then failed when running with 
> instrumented jarfiles as it couldn't find the classes it needed from 
> emma.jar. There might be other tests with similar problems, this was 
> just the first one I stumbled on when running the derbytools subsuite.
>
> Ideally I think the tests should be run in their normal configuration 
> (with security manager if that is the default), also when running 
> codecoverage measurements. It would also be good if it was possible to 
> run all the tests. My immediate thought was that this would require 
> modifications to policy files and possibly individual tests like the 
> test for derbyrun.jar, but there may be better ways to get around 
> these issues that I haven't thought about. I know that EMMA is being 
> used by others in the community, how did you get around these problems?
>
> Is there in principle anything wrong with making changes to policy 
> files and/or tests to make it easier to run codecoverage measurements 
> with the EMMA tool?
>
>
> Vemund
>
>
>