You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by Kristian Waagan <kr...@oracle.com> on 2012/06/11 23:15:48 UTC

Alternative code coverage report using JaCoCo

Hi all,

I've been looking into adding support for JaCoCo [1] for Derby, which is 
a tool used to obtain code coverage numbers. My main motivation for 
doing so has been because the development of EMMA seems to have stopped 
in 2005/2006.

Since there is some activity around code coverage in the community right 
now (i.e. GSoC 2012), I'm posting a first report right away, even though 
I haven't provided the required patch yet. The report was generated by 
running a patched build.xml and BaseTestCase. Regarding the tests, I 
added the replication tests to junit-core. Not sure how that suite test 
is run with EMMA.

Anyway, the report can be found at 
http://people.apache.org/~kristwaa/jacoco/
That location is temporary, but I'll keep it around for a while. Feel 
free to have a look and tell what you think of the report. Also, 
comparing the results with those of EMMA may be useful as a sanity check.


Regards,
-- 
Kristian

[1] http://www.eclemma.org/jacoco/

Re: Alternative code coverage report using JaCoCo

Posted by Knut Anders Hatlen <kn...@oracle.com>.
Kristian Waagan <kr...@oracle.com> writes:

> On 12.06.12 17:43, siddharth srivastava wrote:
>> Hi
>>
>> The coverage report does seem pretty simple and easy to understand.
>>
>> Comparing the jacoco [1] <
>> http://people.apache.org/~kristwaa/jacoco/org.apache.derby.iapi.jdbc/DRDAServerStarter.html>and
>> emma [2]
>> <http://dbtg.foundry.sun.com/derby/test/coverage/_files/10c.html>
>> reports for DRDAServerStarter
>
> Hmm, are the EMMA links unstable? Maybe it generates new file names
> each time the report is compiled?

Yes, the links are unstable.

It would be possible to publish the raw coverage.xml file and set up a
job on builds.apache.org to render it. I do that on my private Hudson
instance, and the EMMA plugin for Hudson does a much better job
generating reports (the reports look more like the JaCoCo reports, the
links are stable, and it generates graphs that show how the coverage
changes over time). Unfortunately, I haven't found any easy way to
export Hudson's HTML reports without having a public Hudson or Jenkins
instance.

> I'm almost sure I was taken to a different page when following [2] a
> day ago. In any case, none of the classes described were
> DRDAServerStarter :)
>
>> I was wondering if the code coverage results in jacoco are not affected
>> by the
>> Java Security Manager since security manager has been shut off by
>> default for emma's
>> code coverage results.
>
> Was there anything in particular you were thinking of?
> Note that the EMMA results should still be more complete, since it's
> being run with several JVM versions and with some additional tests.
> I'm working on improving this for JaCoCo, and we may also want to get
> a scheduled job going to get fresh numbers on a regular basis.

I think the coverage reported by EMMA might have been lower the day
Siddharth sent the message. The attached graph shows the coverage for
DRDAServerStarter since I began running the tests, and you can see that
it sometimes drops. It has been higher the last few days. Hopefully one
of the fixes we've checked in over the last week has helped, so that the
improvement is not just temporary.

Re: Alternative code coverage report using JaCoCo

Posted by Kristian Waagan <kr...@oracle.com>.
On 12.06.12 17:43, siddharth srivastava wrote:
> Hi
>
> The coverage report does seem pretty simple and easy to understand.
>
> Comparing the jacoco [1] <
> http://people.apache.org/~kristwaa/jacoco/org.apache.derby.iapi.jdbc/DRDAServerStarter.html>and
> emma [2]
> <http://dbtg.foundry.sun.com/derby/test/coverage/_files/10c.html>
> reports for DRDAServerStarter

Hmm, are the EMMA links unstable? Maybe it generates new file names each 
time the report is compiled?

I'm almost sure I was taken to a different page when following [2] a day 
ago. In any case, none of the classes described were DRDAServerStarter :)

> I was wondering if the code coverage results in jacoco are not affected
> by the
> Java Security Manager since security manager has been shut off by
> default for emma's
> code coverage results.

Was there anything in particular you were thinking of?
Note that the EMMA results should still be more complete, since it's 
being run with several JVM versions and with some additional tests.
I'm working on improving this for JaCoCo, and we may also want to get a 
scheduled job going to get fresh numbers on a regular basis.


-- 
Kristian

>
> [1]:
> http://people.apache.org/~kristwaa/jacoco/org.apache.derby.iapi.jdbc/DRDAServerStarter.html
> [2]: http://dbtg.foundry.sun.com/derby/test/coverage/_files/10c.html
>
> On 12 June 2012 14:34, Kristian Waagan <kristian.waagan@oracle.com
> <ma...@oracle.com>> wrote:
>  > On 12.06.2012 02:36, Bryan Pendleton wrote:
>  >>>
>  >>> I've been looking into adding support for JaCoCo [1] for Derby
>  >>
>  >>
>  >> Thanks for investigating this, Kristian.
>  >>
>  >> I looked at the report you published, and I found it clear and
>  >> easy to read.
>  >>
>  >> Why would we choose one tool over another? Is there a reason to
>  >> trust the reports produced by one tool more than those produced
>  >> by another? Is it easier to configure or lower-overhead to operate
>  >> one tool versus the other?
>  >
>  >
>  > Hi Bryan,
>  >
>  > Note that we don't have to choose one of the tools - we can use both
> if we
>  > want to.
>  > When I started this effort we were having problems getting EMMA to
> produce
>  > results reliably. I think that issue has been resolved now (Knut made
>  > changes to part of our test framework).
>  >
>  > I'd say both tools are just about as easy to configure, but JaCoCo
> may have
>  > a stronger focus on integration (currently ant, maven, Java agent).
>  > I don't know exactly about overhead, but my gut feeling is they're in the
>  > same league. I've tried other tools that affected performance a lot
> (i.e. it
>  > took ages to run the tests) and that required a lot of resources (either
>  > during data recording or during report generation). Note that JaCoCo
> doesn't
>  > have a separate instrumentation/compilation step.
>  > Based on a quick chat with Knut Anders offline the raw result files from
>  > JaCoCo may be somewhat smaller, but the files are so small anyway that it
>  > doesn't matter.
>  > Report generation with JaCoCo is very fast.
>  > I believe both tools are pretty simplistic in what data they record or
>  > derive, but it's enough for our current usage.
>  >
>  >
>  >>
>  >> Or are you just anticipating that Emma will fall into disrepair
>  >> over time?
>  >
>  >
>  > Yes, this is the main reason to look at another tool in my opinion.
>  > With time, EMMA may get into trouble as new versions of Java are
> released.
>  >
>  > There are also alternative ways to implement code coverage tools with
> Java
>  > 7, so in the future there may be a better tool available. However, the
>  > current level of overhead seems to be acceptable to me (when used
> with test
>  > runs).
>  >
>  >
>  > --
>  > Kristian
>  >
>  >>
>  >> thanks,
>  >>
>  >> bryan
>  >>
>  >
>
>
>
> --
> Regards
> Siddharth Srivastava


Re: Alternative code coverage report using JaCoCo

Posted by siddharth srivastava <ak...@gmail.com>.
Hi

The coverage report does seem pretty simple and easy to understand.

Comparing the jacoco [1]
<http://people.apache.org/~kristwaa/jacoco/org.apache.derby.iapi.jdbc/DRDAServerStarter.html>and
emma [2] <http://dbtg.foundry.sun.com/derby/test/coverage/_files/10c.html>reports
for DRDAServerStarter
I was wondering if the code coverage results in jacoco are not affected by
the
Java Security Manager since security manager has been shut off by default
for emma's
code coverage results.

[1]:
http://people.apache.org/~kristwaa/jacoco/org.apache.derby.iapi.jdbc/DRDAServerStarter.html
[2]: http://dbtg.foundry.sun.com/derby/test/coverage/_files/10c.html

On 12 June 2012 14:34, Kristian Waagan <kr...@oracle.com> wrote:
> On 12.06.2012 02:36, Bryan Pendleton wrote:
>>>
>>> I've been looking into adding support for JaCoCo [1] for Derby
>>
>>
>> Thanks for investigating this, Kristian.
>>
>> I looked at the report you published, and I found it clear and
>> easy to read.
>>
>> Why would we choose one tool over another? Is there a reason to
>> trust the reports produced by one tool more than those produced
>> by another? Is it easier to configure or lower-overhead to operate
>> one tool versus the other?
>
>
> Hi Bryan,
>
> Note that we don't have to choose one of the tools - we can use both if we
> want to.
> When I started this effort we were having problems getting EMMA to produce
> results reliably. I think that issue has been resolved now (Knut made
> changes to part of our test framework).
>
> I'd say both tools are just about as easy to configure, but JaCoCo may
have
> a stronger focus on integration (currently ant, maven, Java agent).
> I don't know exactly about overhead, but my gut feeling is they're in the
> same league. I've tried other tools that affected performance a lot (i.e.
it
> took ages to run the tests) and that required a lot of resources (either
> during data recording or during report generation). Note that JaCoCo
doesn't
> have a separate instrumentation/compilation step.
> Based on a quick chat with Knut Anders offline the raw result files from
> JaCoCo may be somewhat smaller, but the files are so small anyway that it
> doesn't matter.
> Report generation with JaCoCo is very fast.
> I believe both tools are pretty simplistic in what data they record or
> derive, but it's enough for our current usage.
>
>
>>
>> Or are you just anticipating that Emma will fall into disrepair
>> over time?
>
>
> Yes, this is the main reason to look at another tool in my opinion.
> With time, EMMA may get into trouble as new versions of Java are released.
>
> There are also alternative ways to implement code coverage tools with Java
> 7, so in the future there may be a better tool available. However, the
> current level of overhead seems to be acceptable to me (when used with
test
> runs).
>
>
> --
> Kristian
>
>>
>> thanks,
>>
>> bryan
>>
>



-- 
Regards
Siddharth Srivastava

Re: Alternative code coverage report using JaCoCo

Posted by Kristian Waagan <kr...@oracle.com>.
On 12.06.2012 02:36, Bryan Pendleton wrote:
>> I've been looking into adding support for JaCoCo [1] for Derby
>
> Thanks for investigating this, Kristian.
>
> I looked at the report you published, and I found it clear and
> easy to read.
>
> Why would we choose one tool over another? Is there a reason to
> trust the reports produced by one tool more than those produced
> by another? Is it easier to configure or lower-overhead to operate
> one tool versus the other?

Hi Bryan,

Note that we don't have to choose one of the tools - we can use both if 
we want to.
When I started this effort we were having problems getting EMMA to 
produce results reliably. I think that issue has been resolved now (Knut 
made changes to part of our test framework).

I'd say both tools are just about as easy to configure, but JaCoCo may 
have a stronger focus on integration (currently ant, maven, Java agent).
I don't know exactly about overhead, but my gut feeling is they're in 
the same league. I've tried other tools that affected performance a lot 
(i.e. it took ages to run the tests) and that required a lot of 
resources (either during data recording or during report generation). 
Note that JaCoCo doesn't have a separate instrumentation/compilation step.
Based on a quick chat with Knut Anders offline the raw result files from 
JaCoCo may be somewhat smaller, but the files are so small anyway that 
it doesn't matter.
Report generation with JaCoCo is very fast.
I believe both tools are pretty simplistic in what data they record or 
derive, but it's enough for our current usage.

>
> Or are you just anticipating that Emma will fall into disrepair
> over time?

Yes, this is the main reason to look at another tool in my opinion.
With time, EMMA may get into trouble as new versions of Java are released.

There are also alternative ways to implement code coverage tools with 
Java 7, so in the future there may be a better tool available. However, 
the current level of overhead seems to be acceptable to me (when used 
with test runs).


-- 
Kristian

>
> thanks,
>
> bryan
>

Re: Alternative code coverage report using JaCoCo

Posted by Bryan Pendleton <bp...@gmail.com>.
> I've been looking into adding support for JaCoCo [1] for Derby

Thanks for investigating this, Kristian.

I looked at the report you published, and I found it clear and
easy to read.

Why would we choose one tool over another? Is there a reason to
trust the reports produced by one tool more than those produced
by another? Is it easier to configure or lower-overhead to operate
one tool versus the other?

Or are you just anticipating that Emma will fall into disrepair
over time?

thanks,

bryan


Re: Alternative code coverage report using JaCoCo

Posted by Mohamed Nufail <nu...@gmail.com>.
Hi,

It definitely looks better and much more readable.

Regards,
Nufail.

On Tue, Jun 12, 2012 at 2:45 AM, Kristian Waagan <kristian.waagan@oracle.com
> wrote:

> Hi all,
>
> I've been looking into adding support for JaCoCo [1] for Derby, which is a
> tool used to obtain code coverage numbers. My main motivation for doing so
> has been because the development of EMMA seems to have stopped in 2005/2006.
>
> Since there is some activity around code coverage in the community right
> now (i.e. GSoC 2012), I'm posting a first report right away, even though I
> haven't provided the required patch yet. The report was generated by
> running a patched build.xml and BaseTestCase. Regarding the tests, I added
> the replication tests to junit-core. Not sure how that suite test is run
> with EMMA.
>
> Anyway, the report can be found at http://people.apache.org/~**
> kristwaa/jacoco/ <http://people.apache.org/%7Ekristwaa/jacoco/>
> That location is temporary, but I'll keep it around for a while. Feel free
> to have a look and tell what you think of the report. Also, comparing the
> results with those of EMMA may be useful as a sanity check.
>
>
> Regards,
> --
> Kristian
>
> [1] http://www.eclemma.org/jacoco/
>



-- 

Mohamed Nufail
Undergraduate,
Department of Computer Science & Engineering,
University of Moratuwa.
Blog: http://www.nufailm.blogspot.com/