You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ofbiz.apache.org by Adam Heath <do...@brainfood.com> on 2009/12/10 02:43:57 UTC
code coverage report for ofbiz run-tests call
I've thrown some things together this week, to implement code coverage
anaylsis for ofbiz. The code itself is not tied to any particular
code coverage tool(there's a plugin system for it). Currently I have
one written for cobertura, but that's gpl, so not compatible with apache.
The first run I just finished had some test failures. I haven't
looked to see if this is because of the coverage plugin, or some other
reason.
Anyways, without further ado, here is the output:
http://www.brainfood.com/ofbiz-coverage/
Things to note are the entity, service, and minilang components.
Those have so-so coverage.
However, just about every other component has no testing whatsoever.
Re: code coverage report for ofbiz run-tests call
Posted by Tim Ruppert <ti...@hotwaxmedia.com>.
Thanks Adam - this is really interesting information and good to see that everyone is starting to push the automated testing button!
Cheers,
Ruppert
--
Tim Ruppert
HotWax Media
http://www.hotwaxmedia.com
o:801.649.6594
f:801.649.6595
On Dec 9, 2009, at 6:43 PM, Adam Heath wrote:
> I've thrown some things together this week, to implement code coverage
> anaylsis for ofbiz. The code itself is not tied to any particular
> code coverage tool(there's a plugin system for it). Currently I have
> one written for cobertura, but that's gpl, so not compatible with apache.
>
> The first run I just finished had some test failures. I haven't
> looked to see if this is because of the coverage plugin, or some other
> reason.
>
> Anyways, without further ado, here is the output:
>
> http://www.brainfood.com/ofbiz-coverage/
>
> Things to note are the entity, service, and minilang components.
> Those have so-so coverage.
>
> However, just about every other component has no testing whatsoever.
Re: code coverage report for ofbiz run-tests call
Posted by Erwan de FERRIERES <er...@nereide.biz>.
Hi Adam and Scott,
From what we see, it seems that only the junit tests are executed.
Cobertura or any other code-coverage tool does not know how to test /use
simple methods, and are only testing java code.
From what I know, it is the same for all coverage tools, so it would be
necessary to develop our tool or plugin for OFBiz simple method ...
I was planning on doing the same on our CI server,
http://selenium.ofbiz.fr/, along with the junit and selenium results.
I'm not such a technical guy, but I'm ready to test/comment/help on any
dev you'd like to plan to execute.
cheers,
Le 10/12/2009 03:16, Scott Gray a écrit :
> Hi Adam,
>
> Looking at the results my first impression is that the coverage is
> under-reported. For example, the accounting component has quite a few
> tests but no coverage is shown at all (except for the test package
> itself). Possibly because there is lot of logic in simple methods but
> I'm 100% sure java code is also run during the tests.
>
> But still a great start and something that will be immensely useful if
> we can up the accuracy a bit.
>
> Thanks
> Scott
>
> HotWax Media
> http://www.hotwaxmedia.com
>
> On 10/12/2009, at 2:43 PM, Adam Heath wrote:
>
>> I've thrown some things together this week, to implement code coverage
>> anaylsis for ofbiz. The code itself is not tied to any particular
>> code coverage tool(there's a plugin system for it). Currently I have
>> one written for cobertura, but that's gpl, so not compatible with apache.
>>
>> The first run I just finished had some test failures. I haven't
>> looked to see if this is because of the coverage plugin, or some other
>> reason.
>>
>> Anyways, without further ado, here is the output:
>>
>> http://www.brainfood.com/ofbiz-coverage/
>>
>> Things to note are the entity, service, and minilang components.
>> Those have so-so coverage.
>>
>> However, just about every other component has no testing whatsoever.
>
--
Erwan de FERRIERES
www.nereide.biz
Re: code coverage report for ofbiz run-tests call
Posted by Adam Heath <do...@brainfood.com>.
Adrian Crum wrote:
>> http://www.brainfood.com/ofbiz-coverage/
>
> I like the holiday-themed colors. It would be cool if incomplete
> coverage could be displayed as partially eaten sugar cookies.
That's standard cobertura scheme.
Re: code coverage report for ofbiz run-tests call
Posted by Adrian Crum <ad...@hlmksw.com>.
Adam Heath wrote:
> Adrian Crum wrote:
>> Adam Heath wrote:
>>> Scott Gray wrote:
>>>> On 11/12/2009, at 6:41 AM, Adam Heath wrote:
>>>>
>>>>> Scott Gray wrote:
>>>>>> Hi Adam,
>>>>>>
>>>>>> Looking at the results my first impression is that the coverage is
>>>>>> under-reported. For example, the accounting component has quite a few
>>>>>> tests but no coverage is shown at all (except for the test package
>>>>>> itself). Possibly because there is lot of logic in simple methods but
>>>>>> I'm 100% sure java code is also run during the tests.
>>>>>>
>>>>>> But still a great start and something that will be immensely useful if
>>>>>> we can up the accuracy a bit.
>>>>> Well, it doesn't, really. If you click thru to accounting.test,
>>>>> you'll see that there aren't really that many tests. And, upon
>>>>> further investigation, the lines after the runSync calls aren't run,
>>>>> due to some exception most likely. I'm not certian if this is do to
>>>>> my changes, or if the tests themselves are broken. I'm running a
>>>>> plain test run now to check that. Plus, there actually *is* line hits
>>>>> in accounting.invoice.
>>>> The tests seem to be running fine on buildbot
>>>> (http://ci.apache.org/waterfall?show=ofbiz-trunk), I'm guessing it's the
>>>> test run problem that's causing the under reporting. There may not be
>>>> that many explicit accounting tests (even though it is a lot compared to
>>>> other components) but a lot of tests also touch accounting
>>>> indirectly. There is just no way that only 53 lines of java code are
>>>> being executed
>>>> in accounting during the full test run. I know for a fact that code is
>>>> executed from PaymentGatewayServices, FinAccountPaymentServices,
>>>> PaymentWorker, UtilAccounting and a few others during the tests.
>>> I had some other changes in that tree that were causing tests to fail.
>>> I've rerun it now, all current tests pass, and I've uploaded a new
>>> report to http://www.brainfood.com/ofbiz-coverage
>> Oops, link doesn't work.
>
> http://www.brainfood.com/ofbiz-coverage/
I like the holiday-themed colors. It would be cool if incomplete
coverage could be displayed as partially eaten sugar cookies.
Re: code coverage report for ofbiz run-tests call
Posted by Adam Heath <do...@brainfood.com>.
Adrian Crum wrote:
> Adam Heath wrote:
>> Scott Gray wrote:
>>> On 11/12/2009, at 6:41 AM, Adam Heath wrote:
>>>
>>>> Scott Gray wrote:
>>>>> Hi Adam,
>>>>>
>>>>> Looking at the results my first impression is that the coverage is
>>>>> under-reported. For example, the accounting component has quite a few
>>>>> tests but no coverage is shown at all (except for the test package
>>>>> itself). Possibly because there is lot of logic in simple methods but
>>>>> I'm 100% sure java code is also run during the tests.
>>>>>
>>>>> But still a great start and something that will be immensely useful if
>>>>> we can up the accuracy a bit.
>>>> Well, it doesn't, really. If you click thru to accounting.test,
>>>> you'll see that there aren't really that many tests. And, upon
>>>> further investigation, the lines after the runSync calls aren't run,
>>>> due to some exception most likely. I'm not certian if this is do to
>>>> my changes, or if the tests themselves are broken. I'm running a
>>>> plain test run now to check that. Plus, there actually *is* line hits
>>>> in accounting.invoice.
>>> The tests seem to be running fine on buildbot
>>> (http://ci.apache.org/waterfall?show=ofbiz-trunk), I'm guessing it's the
>>> test run problem that's causing the under reporting. There may not be
>>> that many explicit accounting tests (even though it is a lot compared to
>>> other components) but a lot of tests also touch accounting
>>> indirectly. There is just no way that only 53 lines of java code are
>>> being executed
>>> in accounting during the full test run. I know for a fact that code is
>>> executed from PaymentGatewayServices, FinAccountPaymentServices,
>>> PaymentWorker, UtilAccounting and a few others during the tests.
>>
>> I had some other changes in that tree that were causing tests to fail.
>> I've rerun it now, all current tests pass, and I've uploaded a new
>> report to http://www.brainfood.com/ofbiz-coverage
>
> Oops, link doesn't work.
http://www.brainfood.com/ofbiz-coverage/
Re: code coverage report for ofbiz run-tests call
Posted by Adrian Crum <ad...@hlmksw.com>.
Adam Heath wrote:
> Scott Gray wrote:
>> On 11/12/2009, at 6:41 AM, Adam Heath wrote:
>>
>>> Scott Gray wrote:
>>>> Hi Adam,
>>>>
>>>> Looking at the results my first impression is that the coverage is
>>>> under-reported. For example, the accounting component has quite a few
>>>> tests but no coverage is shown at all (except for the test package
>>>> itself). Possibly because there is lot of logic in simple methods but
>>>> I'm 100% sure java code is also run during the tests.
>>>>
>>>> But still a great start and something that will be immensely useful if
>>>> we can up the accuracy a bit.
>>> Well, it doesn't, really. If you click thru to accounting.test,
>>> you'll see that there aren't really that many tests. And, upon
>>> further investigation, the lines after the runSync calls aren't run,
>>> due to some exception most likely. I'm not certian if this is do to
>>> my changes, or if the tests themselves are broken. I'm running a
>>> plain test run now to check that. Plus, there actually *is* line hits
>>> in accounting.invoice.
>> The tests seem to be running fine on buildbot
>> (http://ci.apache.org/waterfall?show=ofbiz-trunk), I'm guessing it's the
>> test run problem that's causing the under reporting. There may not be
>> that many explicit accounting tests (even though it is a lot compared to
>> other components) but a lot of tests also touch accounting indirectly.
>> There is just no way that only 53 lines of java code are being executed
>> in accounting during the full test run. I know for a fact that code is
>> executed from PaymentGatewayServices, FinAccountPaymentServices,
>> PaymentWorker, UtilAccounting and a few others during the tests.
>
> I had some other changes in that tree that were causing tests to fail.
> I've rerun it now, all current tests pass, and I've uploaded a new
> report to http://www.brainfood.com/ofbiz-coverage
Oops, link doesn't work.
Re: code coverage report for ofbiz run-tests call
Posted by Scott Gray <sc...@hotwaxmedia.com>.
We could but I don't think it would help us track progress very well,
there are just too many tests that need to be created. A good start
IMO would be to go through every open bug and create a test case that
reproduces the problem and attach them as patches to be committed with
the fix. Tackling those would at least prevent known bugs from
recurring. Other than that I think people will just proceed with
creating tests for the areas that concern them most.
Regards
Scott
On 11/12/2009, at 10:13 PM, Jacques Le Roux wrote:
> Sure Scott,
>
> Good recommendation, I will remember...
> BTW, we could already open an umbrella issue in Jira and add subtask
> as things go on?
> I can't see a better way to coordinate this work
>
> Jacques
>
> From: "Scott Gray" <sc...@hotwaxmedia.com>
>> Huge tasks are prone to failure, what we really have is a lot of
>> little tasks ahead. If you're writing some code, think about
>> writing the tests that should code along with it. Hopefully it'll
>> become so common place that anyone who doesn't do it will just
>> look silly.
>> Regards
>> Scott
>> HotWax Media
>> http://www.hotwaxmedia.com
>> On 11/12/2009, at 8:59 PM, Jacques Le Roux wrote:
>>> A huge task ahead...
>>>
>>> Jacques
>>>
>>> From: "Adam Heath" <do...@brainfood.com>
>>>> Scott Gray wrote:
>>>>>> Additionally, just because a line has been noted in cobertura,
>>>>>> doesn't
>>>>>> mean all variations have been tested. Consider the case that
>>>>>> some
>>>>>> condition is doing some kind of pattern match, or looking at
>>>>>> Collection.contains or Map.containsKey. It's much simpler to
>>>>>> verify
>>>>>> that everything is tested when it is done explicitly.
>>>>> Okay I see what you mean now, it's a bad thing that coverage is
>>>>> reported
>>>>> without explicit thorough testing. Even though the indirect
>>>>> coverage is
>>>>> still better than no coverage whatsoever.
>>>> As a better example, let's say that there is only 10% coverage on
>>>> the
>>>> entire ofbiz code base. But base has 100% coverage. That other
>>>> 90%
>>>> of untested code may test parts of base that may not work, and
>>>> would
>>>> break the higher-level code.
>>>> It's easier to write tests that are close to the code being tested.
>>>> Trying to tweak a high-level test to make certain all low-level
>>>> code
>>>> is wrong is very very difficult.
>>>> Plus, if a typo gets introduced in one of those map keys, it
>>>> might not
>>>> be detected until much much later in time, when explicit tests
>>>> are not
>>>> used.
>>>> In my opinion, as each new component is activated in the ofbiz
>>>> system,
>>>> it should have it's own set of tests that move it close to 100%
>>>> coverage. So, I can test just base, and get 100%, then base+sql,
>>>> and
>>>> get 100% on base+sql, then base+sql+entity, and get 100% on
>>>> base+sql+entity, and so on. You want to make certain that earlier
>>>> components are correct before utilizing later ones, or the
>>>> entire test
>>>> run may fail spectactulary.
>>>>
>>>
>>
>
Re: code coverage report for ofbiz run-tests call
Posted by Jacques Le Roux <ja...@les7arts.com>.
Sure Scott,
Good recommendation, I will remember...
BTW, we could already open an umbrella issue in Jira and add subtask as things go on?
I can't see a better way to coordinate this work
Jacques
From: "Scott Gray" <sc...@hotwaxmedia.com>
> Huge tasks are prone to failure, what we really have is a lot of
> little tasks ahead. If you're writing some code, think about writing
> the tests that should code along with it. Hopefully it'll become so
> common place that anyone who doesn't do it will just look silly.
>
> Regards
> Scott
>
> HotWax Media
> http://www.hotwaxmedia.com
>
> On 11/12/2009, at 8:59 PM, Jacques Le Roux wrote:
>
>> A huge task ahead...
>>
>> Jacques
>>
>> From: "Adam Heath" <do...@brainfood.com>
>>> Scott Gray wrote:
>>>>> Additionally, just because a line has been noted in cobertura,
>>>>> doesn't
>>>>> mean all variations have been tested. Consider the case that some
>>>>> condition is doing some kind of pattern match, or looking at
>>>>> Collection.contains or Map.containsKey. It's much simpler to
>>>>> verify
>>>>> that everything is tested when it is done explicitly.
>>>> Okay I see what you mean now, it's a bad thing that coverage is
>>>> reported
>>>> without explicit thorough testing. Even though the indirect
>>>> coverage is
>>>> still better than no coverage whatsoever.
>>> As a better example, let's say that there is only 10% coverage on the
>>> entire ofbiz code base. But base has 100% coverage. That other 90%
>>> of untested code may test parts of base that may not work, and would
>>> break the higher-level code.
>>> It's easier to write tests that are close to the code being tested.
>>> Trying to tweak a high-level test to make certain all low-level code
>>> is wrong is very very difficult.
>>> Plus, if a typo gets introduced in one of those map keys, it might
>>> not
>>> be detected until much much later in time, when explicit tests are
>>> not
>>> used.
>>> In my opinion, as each new component is activated in the ofbiz
>>> system,
>>> it should have it's own set of tests that move it close to 100%
>>> coverage. So, I can test just base, and get 100%, then base+sql, and
>>> get 100% on base+sql, then base+sql+entity, and get 100% on
>>> base+sql+entity, and so on. You want to make certain that earlier
>>> components are correct before utilizing later ones, or the entire
>>> test
>>> run may fail spectactulary.
>>>
>>
>
>
Re: code coverage report for ofbiz run-tests call
Posted by Scott Gray <sc...@hotwaxmedia.com>.
Huge tasks are prone to failure, what we really have is a lot of
little tasks ahead. If you're writing some code, think about writing
the tests that should code along with it. Hopefully it'll become so
common place that anyone who doesn't do it will just look silly.
Regards
Scott
HotWax Media
http://www.hotwaxmedia.com
On 11/12/2009, at 8:59 PM, Jacques Le Roux wrote:
> A huge task ahead...
>
> Jacques
>
> From: "Adam Heath" <do...@brainfood.com>
>> Scott Gray wrote:
>>>> Additionally, just because a line has been noted in cobertura,
>>>> doesn't
>>>> mean all variations have been tested. Consider the case that some
>>>> condition is doing some kind of pattern match, or looking at
>>>> Collection.contains or Map.containsKey. It's much simpler to
>>>> verify
>>>> that everything is tested when it is done explicitly.
>>> Okay I see what you mean now, it's a bad thing that coverage is
>>> reported
>>> without explicit thorough testing. Even though the indirect
>>> coverage is
>>> still better than no coverage whatsoever.
>> As a better example, let's say that there is only 10% coverage on the
>> entire ofbiz code base. But base has 100% coverage. That other 90%
>> of untested code may test parts of base that may not work, and would
>> break the higher-level code.
>> It's easier to write tests that are close to the code being tested.
>> Trying to tweak a high-level test to make certain all low-level code
>> is wrong is very very difficult.
>> Plus, if a typo gets introduced in one of those map keys, it might
>> not
>> be detected until much much later in time, when explicit tests are
>> not
>> used.
>> In my opinion, as each new component is activated in the ofbiz
>> system,
>> it should have it's own set of tests that move it close to 100%
>> coverage. So, I can test just base, and get 100%, then base+sql, and
>> get 100% on base+sql, then base+sql+entity, and get 100% on
>> base+sql+entity, and so on. You want to make certain that earlier
>> components are correct before utilizing later ones, or the entire
>> test
>> run may fail spectactulary.
>>
>
Re: code coverage report for ofbiz run-tests call
Posted by Jacques Le Roux <ja...@les7arts.com>.
A huge task ahead...
Jacques
From: "Adam Heath" <do...@brainfood.com>
> Scott Gray wrote:
>>> Additionally, just because a line has been noted in cobertura, doesn't
>>> mean all variations have been tested. Consider the case that some
>>> condition is doing some kind of pattern match, or looking at
>>> Collection.contains or Map.containsKey. It's much simpler to verify
>>> that everything is tested when it is done explicitly.
>>
>> Okay I see what you mean now, it's a bad thing that coverage is reported
>> without explicit thorough testing. Even though the indirect coverage is
>> still better than no coverage whatsoever.
>
> As a better example, let's say that there is only 10% coverage on the
> entire ofbiz code base. But base has 100% coverage. That other 90%
> of untested code may test parts of base that may not work, and would
> break the higher-level code.
>
> It's easier to write tests that are close to the code being tested.
> Trying to tweak a high-level test to make certain all low-level code
> is wrong is very very difficult.
>
> Plus, if a typo gets introduced in one of those map keys, it might not
> be detected until much much later in time, when explicit tests are not
> used.
>
> In my opinion, as each new component is activated in the ofbiz system,
> it should have it's own set of tests that move it close to 100%
> coverage. So, I can test just base, and get 100%, then base+sql, and
> get 100% on base+sql, then base+sql+entity, and get 100% on
> base+sql+entity, and so on. You want to make certain that earlier
> components are correct before utilizing later ones, or the entire test
> run may fail spectactulary.
>
Re: code coverage report for ofbiz run-tests call
Posted by Adam Heath <do...@brainfood.com>.
Scott Gray wrote:
>> Additionally, just because a line has been noted in cobertura, doesn't
>> mean all variations have been tested. Consider the case that some
>> condition is doing some kind of pattern match, or looking at
>> Collection.contains or Map.containsKey. It's much simpler to verify
>> that everything is tested when it is done explicitly.
>
> Okay I see what you mean now, it's a bad thing that coverage is reported
> without explicit thorough testing. Even though the indirect coverage is
> still better than no coverage whatsoever.
As a better example, let's say that there is only 10% coverage on the
entire ofbiz code base. But base has 100% coverage. That other 90%
of untested code may test parts of base that may not work, and would
break the higher-level code.
It's easier to write tests that are close to the code being tested.
Trying to tweak a high-level test to make certain all low-level code
is wrong is very very difficult.
Plus, if a typo gets introduced in one of those map keys, it might not
be detected until much much later in time, when explicit tests are not
used.
In my opinion, as each new component is activated in the ofbiz system,
it should have it's own set of tests that move it close to 100%
coverage. So, I can test just base, and get 100%, then base+sql, and
get 100% on base+sql, then base+sql+entity, and get 100% on
base+sql+entity, and so on. You want to make certain that earlier
components are correct before utilizing later ones, or the entire test
run may fail spectactulary.
Re: code coverage report for ofbiz run-tests call
Posted by Scott Gray <sc...@hotwaxmedia.com>.
On 11/12/2009, at 12:34 PM, Adam Heath wrote:
> Scott Gray wrote:
>>> Note that framework/base has almost 100% coverage. But that's a bad
>>> thing, because it's not explicitly testing it; all that code just
>>> happens to be utilized during the rest of the test run.
>>
>> Of course explicitly testing framework/base would be much better
>> but why
>> is the current 100% coverage a bad thing? I mean implied testing is
>> better than no testing right? If I was to go in and incorrectly
>> modify
>> some of those base methods then there's a good chance some of the
>> higher
>> level tests would fail.
>
> In theory such explicit base testing could be run from the command
> line, very quickly, using the newly added 'tests' target defined in
> common.xml.
>
> Additionally, just because a line has been noted in cobertura, doesn't
> mean all variations have been tested. Consider the case that some
> condition is doing some kind of pattern match, or looking at
> Collection.contains or Map.containsKey. It's much simpler to verify
> that everything is tested when it is done explicitly.
Okay I see what you mean now, it's a bad thing that coverage is
reported without explicit thorough testing. Even though the indirect
coverage is still better than no coverage whatsoever.
Re: code coverage report for ofbiz run-tests call
Posted by Adam Heath <do...@brainfood.com>.
Scott Gray wrote:
>> Note that framework/base has almost 100% coverage. But that's a bad
>> thing, because it's not explicitly testing it; all that code just
>> happens to be utilized during the rest of the test run.
>
> Of course explicitly testing framework/base would be much better but why
> is the current 100% coverage a bad thing? I mean implied testing is
> better than no testing right? If I was to go in and incorrectly modify
> some of those base methods then there's a good chance some of the higher
> level tests would fail.
In theory such explicit base testing could be run from the command
line, very quickly, using the newly added 'tests' target defined in
common.xml.
Additionally, just because a line has been noted in cobertura, doesn't
mean all variations have been tested. Consider the case that some
condition is doing some kind of pattern match, or looking at
Collection.contains or Map.containsKey. It's much simpler to verify
that everything is tested when it is done explicitly.
Re: code coverage report for ofbiz run-tests call
Posted by Scott Gray <sc...@hotwaxmedia.com>.
On 11/12/2009, at 11:02 PM, Erwan de FERRIERES wrote:
> But if we just stay on the services, this vould give a great
> overview of their coverage.
> As we are in a SOA system, and service is the most important part of
> the system, we have to be sure they are tested.
The easiest way to achieve something like this might be to try and
extend/adapt the ArtifactInfo stuff to do it. It could inspect each
test case for service calls and report on the service coverage based
on that. We could make it so that it doesn't dig too deep and only
reports on services that are called directly from the test case and
not services that are called as a consequence of the service call e.g.
ECAs, inline service calls, etc.
Of course it wouldn't be able to report on the actual line coverage of
the service call but combined with Adam's report it might start to
give us better overall picture.
>
> For all user interface testing, validation could be done via
> storylines and scenarios, with expected results after an action, as
> David described in UBPL, for the user point of view.
>
>> From: "Erwan de FERRIERES" <er...@nereide.biz>
>>> Now the point would be to show the coverage of the simple methods
>>> and
>>> also, maybe in sometime, the selenium coverage.
>>>
>>> What would be great is finding a way to indicate which services are
>>> tested, and then display it in the webtools. This won't give an
>>> information as precise as cobertura but would add some quick display
>>> on what is or can be tested.
>>>
>>> If we agree on a syntax to use, I would be ready to add the screen
>>> in
>>> OFBiz.
>>>
>>> Cheers,
>>>
>>> Le 11/12/2009 00:21, Scott Gray a écrit :
>>> ../..
>>>>
>>>>
>>>
>>> --
>>> Erwan de FERRIERES
>>> www.nereide.biz
>>>
>>
>>
>>
>
> --
> Erwan de FERRIERES
> www.nereide.biz
Re: code coverage report for ofbiz run-tests call
Posted by Jacques Le Roux <ja...@les7arts.com>.
Erwan, Scott,
Small comments Inline...
From: "Erwan de FERRIERES" <er...@nereide.biz>
> Hi Jacques,
> inline
>
> Le 11/12/2009 08:47, Jacques Le Roux a écrit :
>> What about groovy scripts, are they handled by Cobertura?
> From what I know Cobertura needs .class files, and for the groovy files, it would mean to compile them first
> http://docs.codehaus.org/display/GROOVY/Code+Coverage+with+Cobertura
Class are created when they are used and placed in cache. As suggested Scott we could use the wonderful artifact info mechanism to
deal with this aspect and also the others (not UI)
>> And actions in Screens, is it worth do something? I guess checking
>> "static structural" files like web.xml, ofbiz-component.xml files (and
>> all xml like them, controllers, menus, commonsScreens and such) is not
>> necessary?
> I think this would mean a lot of work to adapt a tool to this. But if we just stay on the services, this vould give a great
> overview of their coverage.
> As we are in a SOA system, and service is the most important part of the system, we have to be sure they are tested.
>
> For all user interface testing, validation could be done via storylines and scenarios, with expected results after an action, as
> David described in UBPL, for the user point of view.
I guess this is more related to Selenim. Not sure how to link Cobertura and Selenium though
Jacques
>>
>> Jacques
>>
>> From: "Erwan de FERRIERES" <er...@nereide.biz>
>>> Now the point would be to show the coverage of the simple methods and
>>> also, maybe in sometime, the selenium coverage.
>>>
>>> What would be great is finding a way to indicate which services are
>>> tested, and then display it in the webtools. This won't give an
>>> information as precise as cobertura but would add some quick display
>>> on what is or can be tested.
>>>
>>> If we agree on a syntax to use, I would be ready to add the screen in
>>> OFBiz.
>>>
>>> Cheers,
>>>
>>> Le 11/12/2009 00:21, Scott Gray a écrit :
>>> ../..
>>>>
>>>>
>>>
>>> --
>>> Erwan de FERRIERES
>>> www.nereide.biz
>>>
>>
>>
>>
>
> --
> Erwan de FERRIERES
> www.nereide.biz
>
Re: java dependency hell: was Re: code coverage report for ofbiz
run-tests call
Posted by Adrian Crum <ad...@hlmksw.com>.
Adam Heath wrote:
> The coverage stuff I have done for ofbiz is based on the method I did
> it for webslinger. I wrote a shim to allow for other coverage tools
> to be used, then wrote a classpath/jar walker to find all classes
> matching a particular pattern, a dynamic temporary classloader to load
> the instrumenter, then a concrete implementation that uses Cobertura.
>
> I can check everything in, including a build.xml tweak, but can't add
> the cobertura library itself.
>
> So, with that out of the way, here's my current headache.
>
> cobertura-1.9.1 does not handle annotation definitions(@interface) in
> java files. 1.9.3 does. However, that requires upgrading from
> asm-2.2.3 to asm-3.2. But then groovy fails, 1.6.0-1.6.7 still use
> asm-2.2.3. Groovy has to be upgraded to 1.7-rc-2. But then
> webslinger fails, 'cuz it has direct links to asm, cobertura, and groovy.
>
> I've done the webslinger side, and my test cases pass, still have to
> upgrade the ofbiz stuff.
>
> ps: I love java :|
You didn't have any plans for this weekend anyway, right?
java dependency hell: was Re: code coverage report for ofbiz run-tests
call
Posted by Adam Heath <do...@brainfood.com>.
The coverage stuff I have done for ofbiz is based on the method I did
it for webslinger. I wrote a shim to allow for other coverage tools
to be used, then wrote a classpath/jar walker to find all classes
matching a particular pattern, a dynamic temporary classloader to load
the instrumenter, then a concrete implementation that uses Cobertura.
I can check everything in, including a build.xml tweak, but can't add
the cobertura library itself.
So, with that out of the way, here's my current headache.
cobertura-1.9.1 does not handle annotation definitions(@interface) in
java files. 1.9.3 does. However, that requires upgrading from
asm-2.2.3 to asm-3.2. But then groovy fails, 1.6.0-1.6.7 still use
asm-2.2.3. Groovy has to be upgraded to 1.7-rc-2. But then
webslinger fails, 'cuz it has direct links to asm, cobertura, and groovy.
I've done the webslinger side, and my test cases pass, still have to
upgrade the ofbiz stuff.
ps: I love java :|
Re: code coverage report for ofbiz run-tests call
Posted by Adam Heath <do...@brainfood.com>.
Erwan de FERRIERES wrote:
> Hi Jacques,
> inline
>
> Le 11/12/2009 08:47, Jacques Le Roux a écrit :
>> What about groovy scripts, are they handled by Cobertura?
> From what I know Cobertura needs .class files, and for the groovy files,
> it would mean to compile them first
> http://docs.codehaus.org/display/GROOVY/Code+Coverage+with+Cobertura
No, it needs a byte array that contains class data. I haven't looked
inside groovy internals in a bit, but I'm fairly certain that at some
point it creates said byte array, seeing as how the jvm requires it to
do so.
Re: code coverage report for ofbiz run-tests call
Posted by Erwan de FERRIERES <er...@nereide.biz>.
Hi Jacques,
inline
Le 11/12/2009 08:47, Jacques Le Roux a écrit :
> What about groovy scripts, are they handled by Cobertura?
From what I know Cobertura needs .class files, and for the groovy
files, it would mean to compile them first
http://docs.codehaus.org/display/GROOVY/Code+Coverage+with+Cobertura
> And actions in Screens, is it worth do something? I guess checking
> "static structural" files like web.xml, ofbiz-component.xml files (and
> all xml like them, controllers, menus, commonsScreens and such) is not
> necessary?
I think this would mean a lot of work to adapt a tool to this. But if we
just stay on the services, this vould give a great overview of their
coverage.
As we are in a SOA system, and service is the most important part of the
system, we have to be sure they are tested.
For all user interface testing, validation could be done via storylines
and scenarios, with expected results after an action, as David described
in UBPL, for the user point of view.
>
> Jacques
>
> From: "Erwan de FERRIERES" <er...@nereide.biz>
>> Now the point would be to show the coverage of the simple methods and
>> also, maybe in sometime, the selenium coverage.
>>
>> What would be great is finding a way to indicate which services are
>> tested, and then display it in the webtools. This won't give an
>> information as precise as cobertura but would add some quick display
>> on what is or can be tested.
>>
>> If we agree on a syntax to use, I would be ready to add the screen in
>> OFBiz.
>>
>> Cheers,
>>
>> Le 11/12/2009 00:21, Scott Gray a écrit :
>> ../..
>>>
>>>
>>
>> --
>> Erwan de FERRIERES
>> www.nereide.biz
>>
>
>
>
--
Erwan de FERRIERES
www.nereide.biz
Re: code coverage report for ofbiz run-tests call
Posted by Adam Heath <do...@brainfood.com>.
Jacques Le Roux wrote:
> What about groovy scripts, are they handled by Cobertura?
> And actions in Screens, is it worth do something? I guess checking
> "static structural" files like web.xml, ofbiz-component.xml files (and
> all xml like them, controllers, menus, commonsScreens and such) is not
> necessary?
This requires groovy including the line number metadata info in the
compiled bytecode, then a groovy parser that cobertura can use, so no.
Now, if there was a fancy program that supported plugins for other
languages, then that would be cool, but I don't know of one.
Re: code coverage report for ofbiz run-tests call
Posted by Jacques Le Roux <ja...@les7arts.com>.
What about groovy scripts, are they handled by Cobertura?
And actions in Screens, is it worth do something? I guess checking "static structural" files like web.xml, ofbiz-component.xml files
(and all xml like them, controllers, menus, commonsScreens and such) is not necessary?
Jacques
From: "Erwan de FERRIERES" <er...@nereide.biz>
> Now the point would be to show the coverage of the simple methods and also, maybe in sometime, the selenium coverage.
>
> What would be great is finding a way to indicate which services are tested, and then display it in the webtools. This won't give
> an information as precise as cobertura but would add some quick display on what is or can be tested.
>
> If we agree on a syntax to use, I would be ready to add the screen in OFBiz.
>
> Cheers,
>
> Le 11/12/2009 00:21, Scott Gray a écrit :
> ../..
>>
>>
>
> --
> Erwan de FERRIERES
> www.nereide.biz
>
Re: code coverage report for ofbiz run-tests call
Posted by Erwan de FERRIERES <er...@nereide.biz>.
Now the point would be to show the coverage of the simple methods and
also, maybe in sometime, the selenium coverage.
What would be great is finding a way to indicate which services are
tested, and then display it in the webtools. This won't give an
information as precise as cobertura but would add some quick display on
what is or can be tested.
If we agree on a syntax to use, I would be ready to add the screen in OFBiz.
Cheers,
Le 11/12/2009 00:21, Scott Gray a écrit :
../..
>
>
--
Erwan de FERRIERES
www.nereide.biz
Re: code coverage report for ofbiz run-tests call
Posted by Scott Gray <sc...@hotwaxmedia.com>.
On 11/12/2009, at 11:55 AM, Adam Heath wrote:
> Scott Gray wrote:
>> On 11/12/2009, at 6:41 AM, Adam Heath wrote:
>>
>>> Scott Gray wrote:
>>>> Hi Adam,
>>>>
>>>> Looking at the results my first impression is that the coverage is
>>>> under-reported. For example, the accounting component has quite
>>>> a few
>>>> tests but no coverage is shown at all (except for the test package
>>>> itself). Possibly because there is lot of logic in simple
>>>> methods but
>>>> I'm 100% sure java code is also run during the tests.
>>>>
>>>> But still a great start and something that will be immensely
>>>> useful if
>>>> we can up the accuracy a bit.
>>>
>>> Well, it doesn't, really. If you click thru to accounting.test,
>>> you'll see that there aren't really that many tests. And, upon
>>> further investigation, the lines after the runSync calls aren't run,
>>> due to some exception most likely. I'm not certian if this is do to
>>> my changes, or if the tests themselves are broken. I'm running a
>>> plain test run now to check that. Plus, there actually *is* line
>>> hits
>>> in accounting.invoice.
>>
>> The tests seem to be running fine on buildbot
>> (http://ci.apache.org/waterfall?show=ofbiz-trunk), I'm guessing
>> it's the
>> test run problem that's causing the under reporting. There may not
>> be
>> that many explicit accounting tests (even though it is a lot
>> compared to
>> other components) but a lot of tests also touch accounting
>> indirectly.
>> There is just no way that only 53 lines of java code are being
>> executed
>> in accounting during the full test run. I know for a fact that
>> code is
>> executed from PaymentGatewayServices, FinAccountPaymentServices,
>> PaymentWorker, UtilAccounting and a few others during the tests.
>
> I had some other changes in that tree that were causing tests to fail.
> I've rerun it now, all current tests pass, and I've uploaded a new
> report to http://www.brainfood.com/ofbiz-coverage
Cool thanks, looks more in line with what I was expecting.
>
> Note that framework/base has almost 100% coverage. But that's a bad
> thing, because it's not explicitly testing it; all that code just
> happens to be utilized during the rest of the test run.
Of course explicitly testing framework/base would be much better but
why is the current 100% coverage a bad thing? I mean implied testing
is better than no testing right? If I was to go in and incorrectly
modify some of those base methods then there's a good chance some of
the higher level tests would fail.
>
> Total coverage increased from 7% to 14%.
Re: code coverage report for ofbiz run-tests call
Posted by Adam Heath <do...@brainfood.com>.
Scott Gray wrote:
> On 11/12/2009, at 6:41 AM, Adam Heath wrote:
>
>> Scott Gray wrote:
>>> Hi Adam,
>>>
>>> Looking at the results my first impression is that the coverage is
>>> under-reported. For example, the accounting component has quite a few
>>> tests but no coverage is shown at all (except for the test package
>>> itself). Possibly because there is lot of logic in simple methods but
>>> I'm 100% sure java code is also run during the tests.
>>>
>>> But still a great start and something that will be immensely useful if
>>> we can up the accuracy a bit.
>>
>> Well, it doesn't, really. If you click thru to accounting.test,
>> you'll see that there aren't really that many tests. And, upon
>> further investigation, the lines after the runSync calls aren't run,
>> due to some exception most likely. I'm not certian if this is do to
>> my changes, or if the tests themselves are broken. I'm running a
>> plain test run now to check that. Plus, there actually *is* line hits
>> in accounting.invoice.
>
> The tests seem to be running fine on buildbot
> (http://ci.apache.org/waterfall?show=ofbiz-trunk), I'm guessing it's the
> test run problem that's causing the under reporting. There may not be
> that many explicit accounting tests (even though it is a lot compared to
> other components) but a lot of tests also touch accounting indirectly.
> There is just no way that only 53 lines of java code are being executed
> in accounting during the full test run. I know for a fact that code is
> executed from PaymentGatewayServices, FinAccountPaymentServices,
> PaymentWorker, UtilAccounting and a few others during the tests.
I had some other changes in that tree that were causing tests to fail.
I've rerun it now, all current tests pass, and I've uploaded a new
report to http://www.brainfood.com/ofbiz-coverage
Note that framework/base has almost 100% coverage. But that's a bad
thing, because it's not explicitly testing it; all that code just
happens to be utilized during the rest of the test run.
Total coverage increased from 7% to 14%.
>
> Regards
> Scott
Re: code coverage report for ofbiz run-tests call
Posted by Jacques Le Roux <ja...@les7arts.com>.
From: "Scott Gray" <sc...@hotwaxmedia.com>
> On 11/12/2009, at 6:41 AM, Adam Heath wrote:
>
>> Scott Gray wrote:
>>> Hi Adam,
>>>
>>> Looking at the results my first impression is that the coverage is
>>> under-reported. For example, the accounting component has quite a
>>> few
>>> tests but no coverage is shown at all (except for the test package
>>> itself). Possibly because there is lot of logic in simple methods
>>> but
>>> I'm 100% sure java code is also run during the tests.
>>>
>>> But still a great start and something that will be immensely useful
>>> if
>>> we can up the accuracy a bit.
>>
>> Well, it doesn't, really. If you click thru to accounting.test,
>> you'll see that there aren't really that many tests. And, upon
>> further investigation, the lines after the runSync calls aren't run,
>> due to some exception most likely. I'm not certian if this is do to
>> my changes, or if the tests themselves are broken. I'm running a
>> plain test run now to check that. Plus, there actually *is* line hits
>> in accounting.invoice.
>
> The tests seem to be running fine on buildbot (http://ci.apache.org/waterfall?show=ofbiz-trunk
> ), I'm guessing it's the test run problem that's causing the under
> reporting. There may not be that many explicit accounting tests (even
> though it is a lot compared to other components) but a lot of tests
> also touch accounting indirectly. There is just no way that only 53
> lines of java code are being executed in accounting during the full
> test run. I know for a fact that code is executed from
> PaymentGatewayServices, FinAccountPaymentServices, PaymentWorker,
> UtilAccounting and a few others during the tests.
>
> Regards
> Scott
At least, I love to see all this green!
Jacques
Re: code coverage report for ofbiz run-tests call
Posted by Scott Gray <sc...@hotwaxmedia.com>.
On 11/12/2009, at 6:41 AM, Adam Heath wrote:
> Scott Gray wrote:
>> Hi Adam,
>>
>> Looking at the results my first impression is that the coverage is
>> under-reported. For example, the accounting component has quite a
>> few
>> tests but no coverage is shown at all (except for the test package
>> itself). Possibly because there is lot of logic in simple methods
>> but
>> I'm 100% sure java code is also run during the tests.
>>
>> But still a great start and something that will be immensely useful
>> if
>> we can up the accuracy a bit.
>
> Well, it doesn't, really. If you click thru to accounting.test,
> you'll see that there aren't really that many tests. And, upon
> further investigation, the lines after the runSync calls aren't run,
> due to some exception most likely. I'm not certian if this is do to
> my changes, or if the tests themselves are broken. I'm running a
> plain test run now to check that. Plus, there actually *is* line hits
> in accounting.invoice.
The tests seem to be running fine on buildbot (http://ci.apache.org/waterfall?show=ofbiz-trunk
), I'm guessing it's the test run problem that's causing the under
reporting. There may not be that many explicit accounting tests (even
though it is a lot compared to other components) but a lot of tests
also touch accounting indirectly. There is just no way that only 53
lines of java code are being executed in accounting during the full
test run. I know for a fact that code is executed from
PaymentGatewayServices, FinAccountPaymentServices, PaymentWorker,
UtilAccounting and a few others during the tests.
Regards
Scott
Re: code coverage report for ofbiz run-tests call
Posted by Jacques Le Roux <ja...@les7arts.com>.
From: "Adam Heath" <do...@brainfood.com>
> Additionally, the way the classloaders are constructed is a bit odd.
> The bootstrap loader only has ofbiz.jar. The code then creates a
> classloader will *all* jars in base/lib, recursively. After the
> containers and components are started, yet another classloader is
> created, this time with proper libs and all other ofbiz-$foo.jar.
> Both of those classloaders update the java.class.path system property.
> This is a bug, as the instrumentation changes all ofbiz-$foo.jar into
> a new jar in /tmp, so the value of the property is different each
> time. Additionally, the second created classloader should not load
> the base component again, as that is sub-optimal.
>
> I'm thinking that component code should really move to start.
>
> Another bug, is that due to one of the above bugs, the first
> classloader is not instrumented at all. Since this is the classloader
> that contains framework/base, you don't see any line hits in the report.
Since the the start (my start with OFBiz) I have always wondered about complexity there
Jacques
Re: code coverage report for ofbiz run-tests call
Posted by Adam Heath <do...@brainfood.com>.
Scott Gray wrote:
> Hi Adam,
>
> Looking at the results my first impression is that the coverage is
> under-reported. For example, the accounting component has quite a few
> tests but no coverage is shown at all (except for the test package
> itself). Possibly because there is lot of logic in simple methods but
> I'm 100% sure java code is also run during the tests.
>
> But still a great start and something that will be immensely useful if
> we can up the accuracy a bit.
Well, it doesn't, really. If you click thru to accounting.test,
you'll see that there aren't really that many tests. And, upon
further investigation, the lines after the runSync calls aren't run,
due to some exception most likely. I'm not certian if this is do to
my changes, or if the tests themselves are broken. I'm running a
plain test run now to check that. Plus, there actually *is* line hits
in accounting.invoice.
Additionally, the way the classloaders are constructed is a bit odd.
The bootstrap loader only has ofbiz.jar. The code then creates a
classloader will *all* jars in base/lib, recursively. After the
containers and components are started, yet another classloader is
created, this time with proper libs and all other ofbiz-$foo.jar.
Both of those classloaders update the java.class.path system property.
This is a bug, as the instrumentation changes all ofbiz-$foo.jar into
a new jar in /tmp, so the value of the property is different each
time. Additionally, the second created classloader should not load
the base component again, as that is sub-optimal.
I'm thinking that component code should really move to start.
Another bug, is that due to one of the above bugs, the first
classloader is not instrumented at all. Since this is the classloader
that contains framework/base, you don't see any line hits in the report.
Re: code coverage report for ofbiz run-tests call
Posted by Scott Gray <sc...@hotwaxmedia.com>.
Hi Adam,
Looking at the results my first impression is that the coverage is
under-reported. For example, the accounting component has quite a few
tests but no coverage is shown at all (except for the test package
itself). Possibly because there is lot of logic in simple methods but
I'm 100% sure java code is also run during the tests.
But still a great start and something that will be immensely useful if
we can up the accuracy a bit.
Thanks
Scott
HotWax Media
http://www.hotwaxmedia.com
On 10/12/2009, at 2:43 PM, Adam Heath wrote:
> I've thrown some things together this week, to implement code coverage
> anaylsis for ofbiz. The code itself is not tied to any particular
> code coverage tool(there's a plugin system for it). Currently I have
> one written for cobertura, but that's gpl, so not compatible with
> apache.
>
> The first run I just finished had some test failures. I haven't
> looked to see if this is because of the coverage plugin, or some other
> reason.
>
> Anyways, without further ado, here is the output:
>
> http://www.brainfood.com/ofbiz-coverage/
>
> Things to note are the entity, service, and minilang components.
> Those have so-so coverage.
>
> However, just about every other component has no testing whatsoever.
Re: code coverage report for ofbiz run-tests call
Posted by Adrian Crum <ad...@hlmksw.com>.
I am working on unit tests for the framework/base component. I hope to
have them committed soon.
Thanks for the info!
-Adrian
Adam Heath wrote:
> I've thrown some things together this week, to implement code coverage
> anaylsis for ofbiz. The code itself is not tied to any particular
> code coverage tool(there's a plugin system for it). Currently I have
> one written for cobertura, but that's gpl, so not compatible with apache.
>
> The first run I just finished had some test failures. I haven't
> looked to see if this is because of the coverage plugin, or some other
> reason.
>
> Anyways, without further ado, here is the output:
>
> http://www.brainfood.com/ofbiz-coverage/
>
> Things to note are the entity, service, and minilang components.
> Those have so-so coverage.
>
> However, just about every other component has no testing whatsoever.
>