You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@harmony.apache.org by Imran Ghory <im...@gmail.com> on 2007/08/19 00:37:39 UTC
[classlib] Code coverage
What the current status with regards to code coverage of the junit tests ?
-ve seen the wiki page but it seems a bit out of date (last october). I've
tried getting emma to work myself based upon a few of the mailing list posts
about it without much luck.
Has anyone managed to get an emma instrumented build recently ?
Imran
Re: [classlib] Code coverage
Posted by Leo Li <li...@gmail.com>.
On 8/24/07, Tim Ellison <t....@gmail.com> wrote:
>
> Leo Li wrote:
> > On 8/23/07, Rui Hu <ro...@gmail.com> wrote:
> >> I've shared the customized emma tool and its usage in [1].
> >>
> >> We can also find the reason why we need customize emma and how to do
> it.
> >> The
> >> modified source code is also provided.
> >>
> >> Any comment is highly welcomed.
> >>
> >> [1]: http://wiki.apache.org/harmony/coverageEMMA
> >
> >
> > The existence of kernel list is, as Robert says, because that Emma
> itself
> > depends on java core classes in its static initialization but
> > unfortunately in order to record testcoverage of these java core
> classes, in
> > their static initialization, emma will be referenced in order to
> register
> > some data structure for these java core classes. Thus Emma'
> initialization
> > depends on java core classes and java core classes' initialization
> depends
> > on Emma and VM detects a circular dependency and throw out
> ClassDefNotFound
> > exception.
> > In out current solution, core classes in the kernel list will not
> > register themselves to emma during instrumentation. But during
> startup, we
> > will explicitly load these core classes and Emma and then manually
> register
> > core classes afterwards.
> > But what puzzles me is that the kernel list, which is the
> dependencies of
> > Emma(The classes will be loaded when loading Emma.), is so unstable and
> is a
> > lot larger than my estimation. I once used JVMTI to record all the
> classes
> > loaded when loading Emma, and the kernel list generated by this way is
> much
> > smaller and does not work.
> > So is there a more precise way to calculate which class will be
> referenced
> > while loading Emma(The com.vladium.emma.rt.RT class)? I believe the
> > dependencies of this class should be more stable than we have met.
>
> How about "java -verbose:class ..."? That will tell you which classes
> are loaded. But you should have been able to get that info via JVMTI so
> maybe I misunderstand.
Thanks, Tim. I will try it.:)
Regards,
> Tim
>
> > 2007/8/22, Rui Hu <ro...@gmail.com>:
> >>> I've shared the coverage report in HARMONY-4665. Anyone can download
> it
> >> to
> >>> see the coverage details. Thanks.
> >>> The customized EMMA and usage will be shared later.
> >>>
> >>>
> >>> 2007/8/21, Tim Ellison < t.p.ellison@gmail.com>:
> >>>> Rui Hu wrote:
> >>>>> I think, it is not easy to automatically generate the coverage
> >> report
> >>>> of our
> >>>>> trunk. :-(
> >>>>>
> >>>>> Because Harmony is not a Java application, it's the JDK, so the
> >>>> original
> >>>>> EMMA can not be use to collect the coverage data of Harmony. We've
> >>>> modified
> >>>>> the source code of EMMA, and realize the coverage collecting for
> >>>> core-java
> >>>>> APIs. In this way, we must notice that:
> >>>>> 1. The coverage of the classes in luni-kernel.jar can not be
> >>>> collected. It's
> >>>>> not a big problem, because there're not many classes in that jar and
> >>>> the
> >>>>> coverage of those classes is considered to be very high (they're
> >>>> usually
> >>>>> used kernel classes).
> >>>>> 2. The customized EMMA needs extra list of booting classes to run,
> >> the
> >>>> class
> >>>>> list can not be too big or too small. If it's too big, the EMMA may
> >>>> not
> >>>>> work. If it's too small, the JDK can not initialize.
> >>>>> 3. The coverage of the classes in the list mentioned in #2 will be
> >>>> always
> >>>>> 100%, (only the class coverage will be 100%, the method/block
> >> coverage
> >>>> will
> >>>>> be correct), it's reasonable for the REAL booting classes, but
> >>>> incorrect for
> >>>>> the unnecessary classes. So, this is another reason why we need not
> >> a
> >>>>> too-big list.
> >>>>>
> >>>>> The problem is from the booting classes list mentioned in #2.
> >> Ideally,
> >>>> this
> >>>>> list includes the classes which is used during initialization of
> >>>> running a
> >>>>> test suite. Unfortunately, we do not have method to get the list
> >>>>> automatically, and it related with the revision of Harmony build,
> >> the
> >>>>> classes we want to collect coverage data for, and the test suite we
> >>>> want to
> >>>>> run.
> >>>>> So we now must try to get the list from cutting down the big-enough
> >>>> list
> >>>>> (usually the all-instrumented-classes-list) every time. If the test
> >>>> crash,
> >>>>> we must modify the booting classes list or remove some "problem test
> >>>> cases".
> >>>>> The only shortcut is use the current successful list as the start
> >>>> point, but
> >>>>> it can not ensure the test will run successfully, especially we
> >> change
> >>>> the
> >>>>> test suite or expand the instrumented classes list ( for example,
> >>>> instrument
> >>>>> all the classes instead of only public API classes).
> >>>>>
> >>>>> I'd like to share our customized EMMA and the experience of using
> >> it.
> >>>> It is
> >>>>> highly welcomed to share your idea to automatize this process, or
> >>>> simplify
> >>>>> the it, or make the it more stably.
> >>>> It would be great if you would share it, then perhaps other people
> >> will
> >>>> see ways to improve creating the booting classes list -- and it would
> >> be
> >>>> interesting to get reports on the current BTI test coverage.
> >>>>
> >>>> Regards,
> >>>> Tim
> >>>>
> >>>
> >>>
> >>> --
> >>> Robert Hu
> >>> China Software Development Lab, IBM
> >>>
> >>
> >>
> >> --
> >> Robert Hu
> >> China Software Development Lab, IBM
> >>
> >
> >
> >
>
--
Leo Li
China Software Development Lab, IBM
Re: [classlib] Code coverage
Posted by Tim Ellison <t....@gmail.com>.
Leo Li wrote:
> On 8/23/07, Rui Hu <ro...@gmail.com> wrote:
>> I've shared the customized emma tool and its usage in [1].
>>
>> We can also find the reason why we need customize emma and how to do it.
>> The
>> modified source code is also provided.
>>
>> Any comment is highly welcomed.
>>
>> [1]: http://wiki.apache.org/harmony/coverageEMMA
>
>
> The existence of kernel list is, as Robert says, because that Emma itself
> depends on java core classes in its static initialization but
> unfortunately in order to record testcoverage of these java core classes, in
> their static initialization, emma will be referenced in order to register
> some data structure for these java core classes. Thus Emma' initialization
> depends on java core classes and java core classes' initialization depends
> on Emma and VM detects a circular dependency and throw out ClassDefNotFound
> exception.
> In out current solution, core classes in the kernel list will not
> register themselves to emma during instrumentation. But during startup, we
> will explicitly load these core classes and Emma and then manually register
> core classes afterwards.
> But what puzzles me is that the kernel list, which is the dependencies of
> Emma(The classes will be loaded when loading Emma.), is so unstable and is a
> lot larger than my estimation. I once used JVMTI to record all the classes
> loaded when loading Emma, and the kernel list generated by this way is much
> smaller and does not work.
> So is there a more precise way to calculate which class will be referenced
> while loading Emma(The com.vladium.emma.rt.RT class)? I believe the
> dependencies of this class should be more stable than we have met.
How about "java -verbose:class ..."? That will tell you which classes
are loaded. But you should have been able to get that info via JVMTI so
maybe I misunderstand.
Regards,
Tim
> 2007/8/22, Rui Hu <ro...@gmail.com>:
>>> I've shared the coverage report in HARMONY-4665. Anyone can download it
>> to
>>> see the coverage details. Thanks.
>>> The customized EMMA and usage will be shared later.
>>>
>>>
>>> 2007/8/21, Tim Ellison < t.p.ellison@gmail.com>:
>>>> Rui Hu wrote:
>>>>> I think, it is not easy to automatically generate the coverage
>> report
>>>> of our
>>>>> trunk. :-(
>>>>>
>>>>> Because Harmony is not a Java application, it's the JDK, so the
>>>> original
>>>>> EMMA can not be use to collect the coverage data of Harmony. We've
>>>> modified
>>>>> the source code of EMMA, and realize the coverage collecting for
>>>> core-java
>>>>> APIs. In this way, we must notice that:
>>>>> 1. The coverage of the classes in luni-kernel.jar can not be
>>>> collected. It's
>>>>> not a big problem, because there're not many classes in that jar and
>>>> the
>>>>> coverage of those classes is considered to be very high (they're
>>>> usually
>>>>> used kernel classes).
>>>>> 2. The customized EMMA needs extra list of booting classes to run,
>> the
>>>> class
>>>>> list can not be too big or too small. If it's too big, the EMMA may
>>>> not
>>>>> work. If it's too small, the JDK can not initialize.
>>>>> 3. The coverage of the classes in the list mentioned in #2 will be
>>>> always
>>>>> 100%, (only the class coverage will be 100%, the method/block
>> coverage
>>>> will
>>>>> be correct), it's reasonable for the REAL booting classes, but
>>>> incorrect for
>>>>> the unnecessary classes. So, this is another reason why we need not
>> a
>>>>> too-big list.
>>>>>
>>>>> The problem is from the booting classes list mentioned in #2.
>> Ideally,
>>>> this
>>>>> list includes the classes which is used during initialization of
>>>> running a
>>>>> test suite. Unfortunately, we do not have method to get the list
>>>>> automatically, and it related with the revision of Harmony build,
>> the
>>>>> classes we want to collect coverage data for, and the test suite we
>>>> want to
>>>>> run.
>>>>> So we now must try to get the list from cutting down the big-enough
>>>> list
>>>>> (usually the all-instrumented-classes-list) every time. If the test
>>>> crash,
>>>>> we must modify the booting classes list or remove some "problem test
>>>> cases".
>>>>> The only shortcut is use the current successful list as the start
>>>> point, but
>>>>> it can not ensure the test will run successfully, especially we
>> change
>>>> the
>>>>> test suite or expand the instrumented classes list ( for example,
>>>> instrument
>>>>> all the classes instead of only public API classes).
>>>>>
>>>>> I'd like to share our customized EMMA and the experience of using
>> it.
>>>> It is
>>>>> highly welcomed to share your idea to automatize this process, or
>>>> simplify
>>>>> the it, or make the it more stably.
>>>> It would be great if you would share it, then perhaps other people
>> will
>>>> see ways to improve creating the booting classes list -- and it would
>> be
>>>> interesting to get reports on the current BTI test coverage.
>>>>
>>>> Regards,
>>>> Tim
>>>>
>>>
>>>
>>> --
>>> Robert Hu
>>> China Software Development Lab, IBM
>>>
>>
>>
>> --
>> Robert Hu
>> China Software Development Lab, IBM
>>
>
>
>
Re: [classlib] Code coverage
Posted by Leo Li <li...@gmail.com>.
On 8/23/07, Rui Hu <ro...@gmail.com> wrote:
>
> I've shared the customized emma tool and its usage in [1].
>
> We can also find the reason why we need customize emma and how to do it.
> The
> modified source code is also provided.
>
> Any comment is highly welcomed.
>
> [1]: http://wiki.apache.org/harmony/coverageEMMA
The existence of kernel list is, as Robert says, because that Emma itself
depends on java core classes in its static initialization but
unfortunately in order to record testcoverage of these java core classes, in
their static initialization, emma will be referenced in order to register
some data structure for these java core classes. Thus Emma' initialization
depends on java core classes and java core classes' initialization depends
on Emma and VM detects a circular dependency and throw out ClassDefNotFound
exception.
In out current solution, core classes in the kernel list will not
register themselves to emma during instrumentation. But during startup, we
will explicitly load these core classes and Emma and then manually register
core classes afterwards.
But what puzzles me is that the kernel list, which is the dependencies of
Emma(The classes will be loaded when loading Emma.), is so unstable and is a
lot larger than my estimation. I once used JVMTI to record all the classes
loaded when loading Emma, and the kernel list generated by this way is much
smaller and does not work.
So is there a more precise way to calculate which class will be referenced
while loading Emma(The com.vladium.emma.rt.RT class)? I believe the
dependencies of this class should be more stable than we have met.
2007/8/22, Rui Hu <ro...@gmail.com>:
> >
> > I've shared the coverage report in HARMONY-4665. Anyone can download it
> to
> > see the coverage details. Thanks.
> > The customized EMMA and usage will be shared later.
> >
> >
> > 2007/8/21, Tim Ellison < t.p.ellison@gmail.com>:
> > >
> > > Rui Hu wrote:
> > > > I think, it is not easy to automatically generate the coverage
> report
> > > of our
> > > > trunk. :-(
> > > >
> > > > Because Harmony is not a Java application, it's the JDK, so the
> > > original
> > > > EMMA can not be use to collect the coverage data of Harmony. We've
> > > modified
> > > > the source code of EMMA, and realize the coverage collecting for
> > > core-java
> > > > APIs. In this way, we must notice that:
> > > > 1. The coverage of the classes in luni-kernel.jar can not be
> > > collected. It's
> > > > not a big problem, because there're not many classes in that jar and
> > > the
> > > > coverage of those classes is considered to be very high (they're
> > > usually
> > > > used kernel classes).
> > > > 2. The customized EMMA needs extra list of booting classes to run,
> the
> > > class
> > > > list can not be too big or too small. If it's too big, the EMMA may
> > > not
> > > > work. If it's too small, the JDK can not initialize.
> > > > 3. The coverage of the classes in the list mentioned in #2 will be
> > > always
> > > > 100%, (only the class coverage will be 100%, the method/block
> coverage
> > > will
> > > > be correct), it's reasonable for the REAL booting classes, but
> > > incorrect for
> > > > the unnecessary classes. So, this is another reason why we need not
> a
> > > > too-big list.
> > > >
> > > > The problem is from the booting classes list mentioned in #2.
> Ideally,
> > > this
> > > > list includes the classes which is used during initialization of
> > > running a
> > > > test suite. Unfortunately, we do not have method to get the list
> > > > automatically, and it related with the revision of Harmony build,
> the
> > > > classes we want to collect coverage data for, and the test suite we
> > > want to
> > > > run.
> > > > So we now must try to get the list from cutting down the big-enough
> > > list
> > > > (usually the all-instrumented-classes-list) every time. If the test
> > > crash,
> > > > we must modify the booting classes list or remove some "problem test
> > > cases".
> > > >
> > > > The only shortcut is use the current successful list as the start
> > > point, but
> > > > it can not ensure the test will run successfully, especially we
> change
> > > the
> > > > test suite or expand the instrumented classes list ( for example,
> > > instrument
> > > > all the classes instead of only public API classes).
> > > >
> > > > I'd like to share our customized EMMA and the experience of using
> it.
> > > It is
> > > > highly welcomed to share your idea to automatize this process, or
> > > simplify
> > > > the it, or make the it more stably.
> > >
> > > It would be great if you would share it, then perhaps other people
> will
> > > see ways to improve creating the booting classes list -- and it would
> be
> > > interesting to get reports on the current BTI test coverage.
> > >
> > > Regards,
> > > Tim
> > >
> >
> >
> >
> > --
> > Robert Hu
> > China Software Development Lab, IBM
> >
>
>
>
> --
> Robert Hu
> China Software Development Lab, IBM
>
--
Leo Li
China Software Development Lab, IBM
Re: [classlib] Code coverage
Posted by Rui Hu <ro...@gmail.com>.
I've shared the customized emma tool and its usage in [1].
We can also find the reason why we need customize emma and how to do it. The
modified source code is also provided.
Any comment is highly welcomed.
[1]: http://wiki.apache.org/harmony/coverageEMMA
2007/8/22, Rui Hu <ro...@gmail.com>:
>
> I've shared the coverage report in HARMONY-4665. Anyone can download it to
> see the coverage details. Thanks.
> The customized EMMA and usage will be shared later.
>
>
> 2007/8/21, Tim Ellison < t.p.ellison@gmail.com>:
> >
> > Rui Hu wrote:
> > > I think, it is not easy to automatically generate the coverage report
> > of our
> > > trunk. :-(
> > >
> > > Because Harmony is not a Java application, it's the JDK, so the
> > original
> > > EMMA can not be use to collect the coverage data of Harmony. We've
> > modified
> > > the source code of EMMA, and realize the coverage collecting for
> > core-java
> > > APIs. In this way, we must notice that:
> > > 1. The coverage of the classes in luni-kernel.jar can not be
> > collected. It's
> > > not a big problem, because there're not many classes in that jar and
> > the
> > > coverage of those classes is considered to be very high (they're
> > usually
> > > used kernel classes).
> > > 2. The customized EMMA needs extra list of booting classes to run, the
> > class
> > > list can not be too big or too small. If it's too big, the EMMA may
> > not
> > > work. If it's too small, the JDK can not initialize.
> > > 3. The coverage of the classes in the list mentioned in #2 will be
> > always
> > > 100%, (only the class coverage will be 100%, the method/block coverage
> > will
> > > be correct), it's reasonable for the REAL booting classes, but
> > incorrect for
> > > the unnecessary classes. So, this is another reason why we need not a
> > > too-big list.
> > >
> > > The problem is from the booting classes list mentioned in #2. Ideally,
> > this
> > > list includes the classes which is used during initialization of
> > running a
> > > test suite. Unfortunately, we do not have method to get the list
> > > automatically, and it related with the revision of Harmony build, the
> > > classes we want to collect coverage data for, and the test suite we
> > want to
> > > run.
> > > So we now must try to get the list from cutting down the big-enough
> > list
> > > (usually the all-instrumented-classes-list) every time. If the test
> > crash,
> > > we must modify the booting classes list or remove some "problem test
> > cases".
> > >
> > > The only shortcut is use the current successful list as the start
> > point, but
> > > it can not ensure the test will run successfully, especially we change
> > the
> > > test suite or expand the instrumented classes list ( for example,
> > instrument
> > > all the classes instead of only public API classes).
> > >
> > > I'd like to share our customized EMMA and the experience of using it.
> > It is
> > > highly welcomed to share your idea to automatize this process, or
> > simplify
> > > the it, or make the it more stably.
> >
> > It would be great if you would share it, then perhaps other people will
> > see ways to improve creating the booting classes list -- and it would be
> > interesting to get reports on the current BTI test coverage.
> >
> > Regards,
> > Tim
> >
>
>
>
> --
> Robert Hu
> China Software Development Lab, IBM
>
--
Robert Hu
China Software Development Lab, IBM
Re: [classlib] Code coverage
Posted by Tim Ellison <t....@gmail.com>.
Rui Hu wrote:
> I've uploaded the new reports in the HARMONY-4665 and delete the old ones.
> There're 4 new reports:
> 1. hy5-all-api-r566713.tar.gz and hy6-all-api-r566010.tar.gz are the
> replacements of old reports in http://people.apache.org/~tellison/emma/,
> I've fixed some problem about security module to show coverage of security
> module correctly. (We can see that old report in
> http://people.apache.org/~tellison/emma/ have incorrect low coverage rate in
> security module, and now we can replace them with the 2 new reports)
> 2. hy5-hyonly-api-r566010.tar.gz and hy6-hyonly-api-r566010.tar.gz are the
> reports same as the two report above, but without the JMX, ORB, etc things
> that we get from elsewhere. (just as Tim's suggestion before). I've excluded
> classes of JMX, ORB, XML in these "hy-only-api" reports.
Thanks Robert. I uploaded your new reports to the same location
http://people.apache.org/~tellison/emma/
Regards,
Tim
Re: [classlib] Code coverage
Posted by Rui Hu <ro...@gmail.com>.
I've uploaded the new reports in the HARMONY-4665 and delete the old ones.
There're 4 new reports:
1. hy5-all-api-r566713.tar.gz and hy6-all-api-r566010.tar.gz are the
replacements of old reports in http://people.apache.org/~tellison/emma/,
I've fixed some problem about security module to show coverage of security
module correctly. (We can see that old report in
http://people.apache.org/~tellison/emma/ have incorrect low coverage rate in
security module, and now we can replace them with the 2 new reports)
2. hy5-hyonly-api-r566010.tar.gz and hy6-hyonly-api-r566010.tar.gz are the
reports same as the two report above, but without the JMX, ORB, etc things
that we get from elsewhere. (just as Tim's suggestion before). I've excluded
classes of JMX, ORB, XML in these "hy-only-api" reports.
2007/8/22, Tim Ellison <t....@gmail.com>:
>
> Rui Hu wrote:
> > I've shared the coverage report in HARMONY-4665. Anyone can download it
> to
> > see the coverage details. Thanks.
> > The customized EMMA and usage will be shared later.
>
> Thanks Robert. I have uploaded your reports here temporarily so people
> can see them easily:
>
> http://people.apache.org/~tellison/emma/
>
> The obvious things that jump out are:
> - big blocks of 0% coverage where we have dependencies from other
> projects, that are already tested, e.g. XML and ORB.
> - some areas are amazingly high, over 90% block coverage! e.g. NIO,
> Logging, JNDI.
> - a few areas that could do with a boost, e.g. RMI
>
> Overall very impressive though. I wonder what the summary numbers would
> be without the JMX, ORB, etc things that we get from elsewhere.
>
> Regards,
> Tim
>
>
--
Robert Hu
China Software Development Lab, IBM
Re: [classlib] Code coverage
Posted by Sean Qiu <se...@gmail.com>.
2007/8/22, Tim Ellison <t....@gmail.com>:
> Rui Hu wrote:
> > I've shared the coverage report in HARMONY-4665. Anyone can download it
> to
> > see the coverage details. Thanks.
> > The customized EMMA and usage will be shared later.
>
> Thanks Robert. I have uploaded your reports here temporarily so people
> can see them easily:
>
> http://people.apache.org/~tellison/emma/
>
> The obvious things that jump out are:
> - big blocks of 0% coverage where we have dependencies from other
> projects, that are already tested, e.g. XML and ORB.
> - some areas are amazingly high, over 90% block coverage! e.g. NIO,
> Logging, JNDI.
> - a few areas that could do with a boost, e.g. RMI
For the RMI, maybe we can write a scenario test in our BTI.
Because the the test need a available server to interact.
And i will try to add some junit test for it at the same time.
> Overall very impressive though. I wonder what the summary numbers would
> be without the JMX, ORB, etc things that we get from elsewhere.
>
> Regards,
> Tim
>
>
--
Sean Qiu
Re: [classlib] Code coverage
Posted by Tim Ellison <t....@gmail.com>.
Rui Hu wrote:
> I've shared the coverage report in HARMONY-4665. Anyone can download it to
> see the coverage details. Thanks.
> The customized EMMA and usage will be shared later.
Thanks Robert. I have uploaded your reports here temporarily so people
can see them easily:
http://people.apache.org/~tellison/emma/
The obvious things that jump out are:
- big blocks of 0% coverage where we have dependencies from other
projects, that are already tested, e.g. XML and ORB.
- some areas are amazingly high, over 90% block coverage! e.g. NIO,
Logging, JNDI.
- a few areas that could do with a boost, e.g. RMI
Overall very impressive though. I wonder what the summary numbers would
be without the JMX, ORB, etc things that we get from elsewhere.
Regards,
Tim
Re: [classlib] Code coverage
Posted by Rui Hu <ro...@gmail.com>.
I've shared the coverage report in HARMONY-4665. Anyone can download it to
see the coverage details. Thanks.
The customized EMMA and usage will be shared later.
2007/8/21, Tim Ellison <t....@gmail.com>:
>
> Rui Hu wrote:
> > I think, it is not easy to automatically generate the coverage report of
> our
> > trunk. :-(
> >
> > Because Harmony is not a Java application, it's the JDK, so the original
> > EMMA can not be use to collect the coverage data of Harmony. We've
> modified
> > the source code of EMMA, and realize the coverage collecting for
> core-java
> > APIs. In this way, we must notice that:
> > 1. The coverage of the classes in luni-kernel.jar can not be collected.
> It's
> > not a big problem, because there're not many classes in that jar and the
> > coverage of those classes is considered to be very high (they're usually
> > used kernel classes).
> > 2. The customized EMMA needs extra list of booting classes to run, the
> class
> > list can not be too big or too small. If it's too big, the EMMA may not
> > work. If it's too small, the JDK can not initialize.
> > 3. The coverage of the classes in the list mentioned in #2 will be
> always
> > 100%, (only the class coverage will be 100%, the method/block coverage
> will
> > be correct), it's reasonable for the REAL booting classes, but incorrect
> for
> > the unnecessary classes. So, this is another reason why we need not a
> > too-big list.
> >
> > The problem is from the booting classes list mentioned in #2. Ideally,
> this
> > list includes the classes which is used during initialization of running
> a
> > test suite. Unfortunately, we do not have method to get the list
> > automatically, and it related with the revision of Harmony build, the
> > classes we want to collect coverage data for, and the test suite we want
> to
> > run.
> > So we now must try to get the list from cutting down the big-enough list
> > (usually the all-instrumented-classes-list) every time. If the test
> crash,
> > we must modify the booting classes list or remove some "problem test
> cases".
> >
> > The only shortcut is use the current successful list as the start point,
> but
> > it can not ensure the test will run successfully, especially we change
> the
> > test suite or expand the instrumented classes list ( for example,
> instrument
> > all the classes instead of only public API classes).
> >
> > I'd like to share our customized EMMA and the experience of using it. It
> is
> > highly welcomed to share your idea to automatize this process, or
> simplify
> > the it, or make the it more stably.
>
> It would be great if you would share it, then perhaps other people will
> see ways to improve creating the booting classes list -- and it would be
> interesting to get reports on the current BTI test coverage.
>
> Regards,
> Tim
>
--
Robert Hu
China Software Development Lab, IBM
Re: [classlib] Code coverage
Posted by Sean Qiu <se...@gmail.com>.
2007/8/21, Tim Ellison <t....@gmail.com>:
>
> Rui Hu wrote:
> > I think, it is not easy to automatically generate the coverage report of
> our
> > trunk. :-(
> >
> > Because Harmony is not a Java application, it's the JDK, so the original
> > EMMA can not be use to collect the coverage data of Harmony. We've
> modified
> > the source code of EMMA, and realize the coverage collecting for
> core-java
> > APIs. In this way, we must notice that:
> > 1. The coverage of the classes in luni-kernel.jar can not be collected.
> It's
> > not a big problem, because there're not many classes in that jar and the
> > coverage of those classes is considered to be very high (they're usually
> > used kernel classes).
> > 2. The customized EMMA needs extra list of booting classes to run, the
> class
> > list can not be too big or too small. If it's too big, the EMMA may not
> > work. If it's too small, the JDK can not initialize.
> > 3. The coverage of the classes in the list mentioned in #2 will be
> always
> > 100%, (only the class coverage will be 100%, the method/block coverage
> will
> > be correct), it's reasonable for the REAL booting classes, but incorrect
> for
> > the unnecessary classes. So, this is another reason why we need not a
> > too-big list.
> >
> > The problem is from the booting classes list mentioned in #2. Ideally,
> this
> > list includes the classes which is used during initialization of running
> a
> > test suite. Unfortunately, we do not have method to get the list
> > automatically, and it related with the revision of Harmony build, the
> > classes we want to collect coverage data for, and the test suite we want
> to
> > run.
> > So we now must try to get the list from cutting down the big-enough list
> > (usually the all-instrumented-classes-list) every time. If the test
> crash,
> > we must modify the booting classes list or remove some "problem test
> cases".
> >
> > The only shortcut is use the current successful list as the start point,
> but
> > it can not ensure the test will run successfully, especially we change
> the
> > test suite or expand the instrumented classes list ( for example,
> instrument
> > all the classes instead of only public API classes).
> >
> > I'd like to share our customized EMMA and the experience of using it. It
> is
> > highly welcomed to share your idea to automatize this process, or
> simplify
> > the it, or make the it more stably.
>
> It would be great if you would share it, then perhaps other people will
> see ways to improve creating the booting classes list -- and it would be
> interesting to get reports on the current BTI test coverage.
+1
Regards,
> Tim
>
--
Sean Qiu
Re: [classlib] Code coverage
Posted by Tim Ellison <t....@gmail.com>.
Rui Hu wrote:
> I think, it is not easy to automatically generate the coverage report of our
> trunk. :-(
>
> Because Harmony is not a Java application, it's the JDK, so the original
> EMMA can not be use to collect the coverage data of Harmony. We've modified
> the source code of EMMA, and realize the coverage collecting for core-java
> APIs. In this way, we must notice that:
> 1. The coverage of the classes in luni-kernel.jar can not be collected. It's
> not a big problem, because there're not many classes in that jar and the
> coverage of those classes is considered to be very high (they're usually
> used kernel classes).
> 2. The customized EMMA needs extra list of booting classes to run, the class
> list can not be too big or too small. If it's too big, the EMMA may not
> work. If it's too small, the JDK can not initialize.
> 3. The coverage of the classes in the list mentioned in #2 will be always
> 100%, (only the class coverage will be 100%, the method/block coverage will
> be correct), it's reasonable for the REAL booting classes, but incorrect for
> the unnecessary classes. So, this is another reason why we need not a
> too-big list.
>
> The problem is from the booting classes list mentioned in #2. Ideally, this
> list includes the classes which is used during initialization of running a
> test suite. Unfortunately, we do not have method to get the list
> automatically, and it related with the revision of Harmony build, the
> classes we want to collect coverage data for, and the test suite we want to
> run.
> So we now must try to get the list from cutting down the big-enough list
> (usually the all-instrumented-classes-list) every time. If the test crash,
> we must modify the booting classes list or remove some "problem test cases".
>
> The only shortcut is use the current successful list as the start point, but
> it can not ensure the test will run successfully, especially we change the
> test suite or expand the instrumented classes list ( for example, instrument
> all the classes instead of only public API classes).
>
> I'd like to share our customized EMMA and the experience of using it. It is
> highly welcomed to share your idea to automatize this process, or simplify
> the it, or make the it more stably.
It would be great if you would share it, then perhaps other people will
see ways to improve creating the booting classes list -- and it would be
interesting to get reports on the current BTI test coverage.
Regards,
Tim
Re: [classlib] Code coverage
Posted by Rui Hu <ro...@gmail.com>.
Although I didn't use Cobertura, I found that Cobertura still need use the
"instrumented" classes when testing. It is similar as EMMA.
It's very easy to collect coverage data for a Java application, but very
hard for a JDK, because the tool (EMMA or Cobertura) itself is a Java
program and needs some core Java APIs (some instrumented classes in Harmony
JDK) to initialize. So the biggest problem is that: Java program (include
JUnit test, EMMA or Cobertura) can not initialize successfully on an
"instrumented" JDK (like Harmony). To resolve this problem, we postpone the
instrumentation phase for some kernel classes after the JDK has started
(when the kernel classes are loaded). It's what we did in the customized
EMMA.
So, if you take a look at EMMA's manual, you'll find its working process is
similar with Cobertura. And I think Cobertura will also has that kind of
problem. :-(
2007/8/21, Nathan Beyer <nb...@gmail.com>:
>
> I've never used EMMA, so I have no idea. What about Cobertura?
>
> On 8/20/07, Rui Hu <ro...@gmail.com> wrote:
> > I think, it is not easy to automatically generate the coverage report of
> our
> > trunk. :-(
> >
> > Because Harmony is not a Java application, it's the JDK, so the original
> > EMMA can not be use to collect the coverage data of Harmony. We've
> modified
> > the source code of EMMA, and realize the coverage collecting for
> core-java
> > APIs. In this way, we must notice that:
> > 1. The coverage of the classes in luni-kernel.jar can not be collected.
> It's
> > not a big problem, because there're not many classes in that jar and the
> > coverage of those classes is considered to be very high (they're usually
> > used kernel classes).
> > 2. The customized EMMA needs extra list of booting classes to run, the
> class
> > list can not be too big or too small. If it's too big, the EMMA may not
> > work. If it's too small, the JDK can not initialize.
> > 3. The coverage of the classes in the list mentioned in #2 will be
> always
> > 100%, (only the class coverage will be 100%, the method/block coverage
> will
> > be correct), it's reasonable for the REAL booting classes, but incorrect
> for
> > the unnecessary classes. So, this is another reason why we need not a
> > too-big list.
> >
> > The problem is from the booting classes list mentioned in #2. Ideally,
> this
> > list includes the classes which is used during initialization of running
> a
> > test suite. Unfortunately, we do not have method to get the list
> > automatically, and it related with the revision of Harmony build, the
> > classes we want to collect coverage data for, and the test suite we want
> to
> > run.
> > So we now must try to get the list from cutting down the big-enough list
> > (usually the all-instrumented-classes-list) every time. If the test
> crash,
> > we must modify the booting classes list or remove some "problem test
> cases".
> >
> >
> > The only shortcut is use the current successful list as the start point,
> but
> > it can not ensure the test will run successfully, especially we change
> the
> > test suite or expand the instrumented classes list ( for example,
> instrument
> > all the classes instead of only public API classes).
> >
> > I'd like to share our customized EMMA and the experience of using it. It
> is
> > highly welcomed to share your idea to automatize this process, or
> simplify
> > the it, or make the it more stably.
> >
> > Thanks in advance !
> >
> > 2007/8/20, Nathan Beyer <nb...@gmail.com>:
> > >
> > > A patch for the build scripts would be best, as the coverage could
> > > then be generated at any time by anyone.
> > >
> > > -Nathan
> > >
> > > On 8/19/07, Rui Hu <ro...@gmail.com> wrote:
> > > > I've just generated a coverage report on Harmony trunk5 and branch 6
> by
> > > EMMA
> > > > recently. Only public API classes are considered. The EMMA I used
> was
> > > > customized to collect the coverage of core java api.
> > > > The general result is:
> > > > class% --- 68% (1641/2421)
> > > > method% --- 60% (16303/27305)
> > > > block% --- 58% (330724/572138)
> > > >
> > > > The following modules are poor covered , even "zero covered":
> > > > concurrent, CORBA(ORB), XML, RMI, imageio
> > > >
> > > > coverage of awt/swing is not very high, we can improve the whole
> > > coverage
> > > > rapidly by improve these two big module.
> > > >
> > > > BTW, I need a place to publish the report (folder of many HTMLs),
> any
> > > > advice? Thanks a lot!
> > > >
> > > >
> > > >
> > > > 2007/8/19, Imran Ghory <im...@gmail.com>:
> > > > >
> > > > > What the current status with regards to code coverage of the junit
> > > tests ?
> > > > > -ve seen the wiki page but it seems a bit out of date (last
> october).
> > > I've
> > > > > tried getting emma to work myself based upon a few of the mailing
> list
> > > > > posts
> > > > > about it without much luck.
> > > > >
> > > > > Has anyone managed to get an emma instrumented build recently ?
> > > > >
> > > > > Imran
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Robert Hu
> > > > China Software Development Lab, IBM
> > > >
> > >
> >
> >
> >
> > --
> > Robert Hu
> > China Software Development Lab, IBM
> >
>
--
Robert Hu
China Software Development Lab, IBM
Re: [classlib] Code coverage
Posted by Nathan Beyer <nb...@gmail.com>.
I've never used EMMA, so I have no idea. What about Cobertura?
On 8/20/07, Rui Hu <ro...@gmail.com> wrote:
> I think, it is not easy to automatically generate the coverage report of our
> trunk. :-(
>
> Because Harmony is not a Java application, it's the JDK, so the original
> EMMA can not be use to collect the coverage data of Harmony. We've modified
> the source code of EMMA, and realize the coverage collecting for core-java
> APIs. In this way, we must notice that:
> 1. The coverage of the classes in luni-kernel.jar can not be collected. It's
> not a big problem, because there're not many classes in that jar and the
> coverage of those classes is considered to be very high (they're usually
> used kernel classes).
> 2. The customized EMMA needs extra list of booting classes to run, the class
> list can not be too big or too small. If it's too big, the EMMA may not
> work. If it's too small, the JDK can not initialize.
> 3. The coverage of the classes in the list mentioned in #2 will be always
> 100%, (only the class coverage will be 100%, the method/block coverage will
> be correct), it's reasonable for the REAL booting classes, but incorrect for
> the unnecessary classes. So, this is another reason why we need not a
> too-big list.
>
> The problem is from the booting classes list mentioned in #2. Ideally, this
> list includes the classes which is used during initialization of running a
> test suite. Unfortunately, we do not have method to get the list
> automatically, and it related with the revision of Harmony build, the
> classes we want to collect coverage data for, and the test suite we want to
> run.
> So we now must try to get the list from cutting down the big-enough list
> (usually the all-instrumented-classes-list) every time. If the test crash,
> we must modify the booting classes list or remove some "problem test cases".
>
>
> The only shortcut is use the current successful list as the start point, but
> it can not ensure the test will run successfully, especially we change the
> test suite or expand the instrumented classes list ( for example, instrument
> all the classes instead of only public API classes).
>
> I'd like to share our customized EMMA and the experience of using it. It is
> highly welcomed to share your idea to automatize this process, or simplify
> the it, or make the it more stably.
>
> Thanks in advance !
>
> 2007/8/20, Nathan Beyer <nb...@gmail.com>:
> >
> > A patch for the build scripts would be best, as the coverage could
> > then be generated at any time by anyone.
> >
> > -Nathan
> >
> > On 8/19/07, Rui Hu <ro...@gmail.com> wrote:
> > > I've just generated a coverage report on Harmony trunk5 and branch 6 by
> > EMMA
> > > recently. Only public API classes are considered. The EMMA I used was
> > > customized to collect the coverage of core java api.
> > > The general result is:
> > > class% --- 68% (1641/2421)
> > > method% --- 60% (16303/27305)
> > > block% --- 58% (330724/572138)
> > >
> > > The following modules are poor covered , even "zero covered":
> > > concurrent, CORBA(ORB), XML, RMI, imageio
> > >
> > > coverage of awt/swing is not very high, we can improve the whole
> > coverage
> > > rapidly by improve these two big module.
> > >
> > > BTW, I need a place to publish the report (folder of many HTMLs), any
> > > advice? Thanks a lot!
> > >
> > >
> > >
> > > 2007/8/19, Imran Ghory <im...@gmail.com>:
> > > >
> > > > What the current status with regards to code coverage of the junit
> > tests ?
> > > > -ve seen the wiki page but it seems a bit out of date (last october).
> > I've
> > > > tried getting emma to work myself based upon a few of the mailing list
> > > > posts
> > > > about it without much luck.
> > > >
> > > > Has anyone managed to get an emma instrumented build recently ?
> > > >
> > > > Imran
> > > >
> > >
> > >
> > >
> > > --
> > > Robert Hu
> > > China Software Development Lab, IBM
> > >
> >
>
>
>
> --
> Robert Hu
> China Software Development Lab, IBM
>
Re: [classlib] Code coverage
Posted by Rui Hu <ro...@gmail.com>.
I think, it is not easy to automatically generate the coverage report of our
trunk. :-(
Because Harmony is not a Java application, it's the JDK, so the original
EMMA can not be use to collect the coverage data of Harmony. We've modified
the source code of EMMA, and realize the coverage collecting for core-java
APIs. In this way, we must notice that:
1. The coverage of the classes in luni-kernel.jar can not be collected. It's
not a big problem, because there're not many classes in that jar and the
coverage of those classes is considered to be very high (they're usually
used kernel classes).
2. The customized EMMA needs extra list of booting classes to run, the class
list can not be too big or too small. If it's too big, the EMMA may not
work. If it's too small, the JDK can not initialize.
3. The coverage of the classes in the list mentioned in #2 will be always
100%, (only the class coverage will be 100%, the method/block coverage will
be correct), it's reasonable for the REAL booting classes, but incorrect for
the unnecessary classes. So, this is another reason why we need not a
too-big list.
The problem is from the booting classes list mentioned in #2. Ideally, this
list includes the classes which is used during initialization of running a
test suite. Unfortunately, we do not have method to get the list
automatically, and it related with the revision of Harmony build, the
classes we want to collect coverage data for, and the test suite we want to
run.
So we now must try to get the list from cutting down the big-enough list
(usually the all-instrumented-classes-list) every time. If the test crash,
we must modify the booting classes list or remove some "problem test cases".
The only shortcut is use the current successful list as the start point, but
it can not ensure the test will run successfully, especially we change the
test suite or expand the instrumented classes list ( for example, instrument
all the classes instead of only public API classes).
I'd like to share our customized EMMA and the experience of using it. It is
highly welcomed to share your idea to automatize this process, or simplify
the it, or make the it more stably.
Thanks in advance !
2007/8/20, Nathan Beyer <nb...@gmail.com>:
>
> A patch for the build scripts would be best, as the coverage could
> then be generated at any time by anyone.
>
> -Nathan
>
> On 8/19/07, Rui Hu <ro...@gmail.com> wrote:
> > I've just generated a coverage report on Harmony trunk5 and branch 6 by
> EMMA
> > recently. Only public API classes are considered. The EMMA I used was
> > customized to collect the coverage of core java api.
> > The general result is:
> > class% --- 68% (1641/2421)
> > method% --- 60% (16303/27305)
> > block% --- 58% (330724/572138)
> >
> > The following modules are poor covered , even "zero covered":
> > concurrent, CORBA(ORB), XML, RMI, imageio
> >
> > coverage of awt/swing is not very high, we can improve the whole
> coverage
> > rapidly by improve these two big module.
> >
> > BTW, I need a place to publish the report (folder of many HTMLs), any
> > advice? Thanks a lot!
> >
> >
> >
> > 2007/8/19, Imran Ghory <im...@gmail.com>:
> > >
> > > What the current status with regards to code coverage of the junit
> tests ?
> > > -ve seen the wiki page but it seems a bit out of date (last october).
> I've
> > > tried getting emma to work myself based upon a few of the mailing list
> > > posts
> > > about it without much luck.
> > >
> > > Has anyone managed to get an emma instrumented build recently ?
> > >
> > > Imran
> > >
> >
> >
> >
> > --
> > Robert Hu
> > China Software Development Lab, IBM
> >
>
--
Robert Hu
China Software Development Lab, IBM
Re: [classlib] Code coverage
Posted by Nathan Beyer <nb...@gmail.com>.
A patch for the build scripts would be best, as the coverage could
then be generated at any time by anyone.
-Nathan
On 8/19/07, Rui Hu <ro...@gmail.com> wrote:
> I've just generated a coverage report on Harmony trunk5 and branch 6 by EMMA
> recently. Only public API classes are considered. The EMMA I used was
> customized to collect the coverage of core java api.
> The general result is:
> class% --- 68% (1641/2421)
> method% --- 60% (16303/27305)
> block% --- 58% (330724/572138)
>
> The following modules are poor covered , even "zero covered":
> concurrent, CORBA(ORB), XML, RMI, imageio
>
> coverage of awt/swing is not very high, we can improve the whole coverage
> rapidly by improve these two big module.
>
> BTW, I need a place to publish the report (folder of many HTMLs), any
> advice? Thanks a lot!
>
>
>
> 2007/8/19, Imran Ghory <im...@gmail.com>:
> >
> > What the current status with regards to code coverage of the junit tests ?
> > -ve seen the wiki page but it seems a bit out of date (last october). I've
> > tried getting emma to work myself based upon a few of the mailing list
> > posts
> > about it without much luck.
> >
> > Has anyone managed to get an emma instrumented build recently ?
> >
> > Imran
> >
>
>
>
> --
> Robert Hu
> China Software Development Lab, IBM
>
Re: [classlib] Code coverage
Posted by Rui Hu <ro...@gmail.com>.
I've just generated a coverage report on Harmony trunk5 and branch 6 by EMMA
recently. Only public API classes are considered. The EMMA I used was
customized to collect the coverage of core java api.
The general result is:
class% --- 68% (1641/2421)
method% --- 60% (16303/27305)
block% --- 58% (330724/572138)
The following modules are poor covered , even "zero covered":
concurrent, CORBA(ORB), XML, RMI, imageio
coverage of awt/swing is not very high, we can improve the whole coverage
rapidly by improve these two big module.
BTW, I need a place to publish the report (folder of many HTMLs), any
advice? Thanks a lot!
2007/8/19, Imran Ghory <im...@gmail.com>:
>
> What the current status with regards to code coverage of the junit tests ?
> -ve seen the wiki page but it seems a bit out of date (last october). I've
> tried getting emma to work myself based upon a few of the mailing list
> posts
> about it without much luck.
>
> Has anyone managed to get an emma instrumented build recently ?
>
> Imran
>
--
Robert Hu
China Software Development Lab, IBM