|
I've thrown some things together this week, to implement code coverage
anaylsis for ofbiz. The code itself is not tied to any particular code coverage tool(there's a plugin system for it). Currently I have one written for cobertura, but that's gpl, so not compatible with apache. The first run I just finished had some test failures. I haven't looked to see if this is because of the coverage plugin, or some other reason. Anyways, without further ado, here is the output: http://www.brainfood.com/ofbiz-coverage/ Things to note are the entity, service, and minilang components. Those have so-so coverage. However, just about every other component has no testing whatsoever. |
|
Thanks Adam - this is really interesting information and good to see that everyone is starting to push the automated testing button!
Cheers, Ruppert -- Tim Ruppert HotWax Media http://www.hotwaxmedia.com o:801.649.6594 f:801.649.6595 On Dec 9, 2009, at 6:43 PM, Adam Heath wrote: > I've thrown some things together this week, to implement code coverage > anaylsis for ofbiz. The code itself is not tied to any particular > code coverage tool(there's a plugin system for it). Currently I have > one written for cobertura, but that's gpl, so not compatible with apache. > > The first run I just finished had some test failures. I haven't > looked to see if this is because of the coverage plugin, or some other > reason. > > Anyways, without further ado, here is the output: > > http://www.brainfood.com/ofbiz-coverage/ > > Things to note are the entity, service, and minilang components. > Those have so-so coverage. > > However, just about every other component has no testing whatsoever. |
|
In reply to this post by Adam Heath-2
Hi Adam,
Looking at the results my first impression is that the coverage is under-reported. For example, the accounting component has quite a few tests but no coverage is shown at all (except for the test package itself). Possibly because there is lot of logic in simple methods but I'm 100% sure java code is also run during the tests. But still a great start and something that will be immensely useful if we can up the accuracy a bit. Thanks Scott HotWax Media http://www.hotwaxmedia.com On 10/12/2009, at 2:43 PM, Adam Heath wrote: > I've thrown some things together this week, to implement code coverage > anaylsis for ofbiz. The code itself is not tied to any particular > code coverage tool(there's a plugin system for it). Currently I have > one written for cobertura, but that's gpl, so not compatible with > apache. > > The first run I just finished had some test failures. I haven't > looked to see if this is because of the coverage plugin, or some other > reason. > > Anyways, without further ado, here is the output: > > http://www.brainfood.com/ofbiz-coverage/ > > Things to note are the entity, service, and minilang components. > Those have so-so coverage. > > However, just about every other component has no testing whatsoever. |
|
Hi Adam and Scott,
From what we see, it seems that only the junit tests are executed. Cobertura or any other code-coverage tool does not know how to test /use simple methods, and are only testing java code. From what I know, it is the same for all coverage tools, so it would be necessary to develop our tool or plugin for OFBiz simple method ... I was planning on doing the same on our CI server, http://selenium.ofbiz.fr/, along with the junit and selenium results. I'm not such a technical guy, but I'm ready to test/comment/help on any dev you'd like to plan to execute. cheers, Le 10/12/2009 03:16, Scott Gray a écrit : > Hi Adam, > > Looking at the results my first impression is that the coverage is > under-reported. For example, the accounting component has quite a few > tests but no coverage is shown at all (except for the test package > itself). Possibly because there is lot of logic in simple methods but > I'm 100% sure java code is also run during the tests. > > But still a great start and something that will be immensely useful if > we can up the accuracy a bit. > > Thanks > Scott > > HotWax Media > http://www.hotwaxmedia.com > > On 10/12/2009, at 2:43 PM, Adam Heath wrote: > >> I've thrown some things together this week, to implement code coverage >> anaylsis for ofbiz. The code itself is not tied to any particular >> code coverage tool(there's a plugin system for it). Currently I have >> one written for cobertura, but that's gpl, so not compatible with apache. >> >> The first run I just finished had some test failures. I haven't >> looked to see if this is because of the coverage plugin, or some other >> reason. >> >> Anyways, without further ado, here is the output: >> >> http://www.brainfood.com/ofbiz-coverage/ >> >> Things to note are the entity, service, and minilang components. >> Those have so-so coverage. >> >> However, just about every other component has no testing whatsoever. > -- Erwan de FERRIERES www.nereide.biz |
|
In reply to this post by Adam Heath-2
I am working on unit tests for the framework/base component. I hope to
have them committed soon. Thanks for the info! -Adrian Adam Heath wrote: > I've thrown some things together this week, to implement code coverage > anaylsis for ofbiz. The code itself is not tied to any particular > code coverage tool(there's a plugin system for it). Currently I have > one written for cobertura, but that's gpl, so not compatible with apache. > > The first run I just finished had some test failures. I haven't > looked to see if this is because of the coverage plugin, or some other > reason. > > Anyways, without further ado, here is the output: > > http://www.brainfood.com/ofbiz-coverage/ > > Things to note are the entity, service, and minilang components. > Those have so-so coverage. > > However, just about every other component has no testing whatsoever. > |
|
In reply to this post by Scott Gray-2
Scott Gray wrote:
> Hi Adam, > > Looking at the results my first impression is that the coverage is > under-reported. For example, the accounting component has quite a few > tests but no coverage is shown at all (except for the test package > itself). Possibly because there is lot of logic in simple methods but > I'm 100% sure java code is also run during the tests. > > But still a great start and something that will be immensely useful if > we can up the accuracy a bit. Well, it doesn't, really. If you click thru to accounting.test, you'll see that there aren't really that many tests. And, upon further investigation, the lines after the runSync calls aren't run, due to some exception most likely. I'm not certian if this is do to my changes, or if the tests themselves are broken. I'm running a plain test run now to check that. Plus, there actually *is* line hits in accounting.invoice. Additionally, the way the classloaders are constructed is a bit odd. The bootstrap loader only has ofbiz.jar. The code then creates a classloader will *all* jars in base/lib, recursively. After the containers and components are started, yet another classloader is created, this time with proper libs and all other ofbiz-$foo.jar. Both of those classloaders update the java.class.path system property. This is a bug, as the instrumentation changes all ofbiz-$foo.jar into a new jar in /tmp, so the value of the property is different each time. Additionally, the second created classloader should not load the base component again, as that is sub-optimal. I'm thinking that component code should really move to start. Another bug, is that due to one of the above bugs, the first classloader is not instrumented at all. Since this is the classloader that contains framework/base, you don't see any line hits in the report. |
|
On 11/12/2009, at 6:41 AM, Adam Heath wrote:
> Scott Gray wrote: >> Hi Adam, >> >> Looking at the results my first impression is that the coverage is >> under-reported. For example, the accounting component has quite a >> few >> tests but no coverage is shown at all (except for the test package >> itself). Possibly because there is lot of logic in simple methods >> but >> I'm 100% sure java code is also run during the tests. >> >> But still a great start and something that will be immensely useful >> if >> we can up the accuracy a bit. > > Well, it doesn't, really. If you click thru to accounting.test, > you'll see that there aren't really that many tests. And, upon > further investigation, the lines after the runSync calls aren't run, > due to some exception most likely. I'm not certian if this is do to > my changes, or if the tests themselves are broken. I'm running a > plain test run now to check that. Plus, there actually *is* line hits > in accounting.invoice. ), I'm guessing it's the test run problem that's causing the under reporting. There may not be that many explicit accounting tests (even though it is a lot compared to other components) but a lot of tests also touch accounting indirectly. There is just no way that only 53 lines of java code are being executed in accounting during the full test run. I know for a fact that code is executed from PaymentGatewayServices, FinAccountPaymentServices, PaymentWorker, UtilAccounting and a few others during the tests. Regards Scott |
|
Administrator
|
From: "Scott Gray" <[hidden email]>
> On 11/12/2009, at 6:41 AM, Adam Heath wrote: > >> Scott Gray wrote: >>> Hi Adam, >>> >>> Looking at the results my first impression is that the coverage is >>> under-reported. For example, the accounting component has quite a >>> few >>> tests but no coverage is shown at all (except for the test package >>> itself). Possibly because there is lot of logic in simple methods >>> but >>> I'm 100% sure java code is also run during the tests. >>> >>> But still a great start and something that will be immensely useful >>> if >>> we can up the accuracy a bit. >> >> Well, it doesn't, really. If you click thru to accounting.test, >> you'll see that there aren't really that many tests. And, upon >> further investigation, the lines after the runSync calls aren't run, >> due to some exception most likely. I'm not certian if this is do to >> my changes, or if the tests themselves are broken. I'm running a >> plain test run now to check that. Plus, there actually *is* line hits >> in accounting.invoice. > > The tests seem to be running fine on buildbot (http://ci.apache.org/waterfall?show=ofbiz-trunk > ), I'm guessing it's the test run problem that's causing the under > reporting. There may not be that many explicit accounting tests (even > though it is a lot compared to other components) but a lot of tests > also touch accounting indirectly. There is just no way that only 53 > lines of java code are being executed in accounting during the full > test run. I know for a fact that code is executed from > PaymentGatewayServices, FinAccountPaymentServices, PaymentWorker, > UtilAccounting and a few others during the tests. > > Regards > Scott At least, I love to see all this green! Jacques |
|
In reply to this post by Scott Gray-2
Scott Gray wrote:
> On 11/12/2009, at 6:41 AM, Adam Heath wrote: > >> Scott Gray wrote: >>> Hi Adam, >>> >>> Looking at the results my first impression is that the coverage is >>> under-reported. For example, the accounting component has quite a few >>> tests but no coverage is shown at all (except for the test package >>> itself). Possibly because there is lot of logic in simple methods but >>> I'm 100% sure java code is also run during the tests. >>> >>> But still a great start and something that will be immensely useful if >>> we can up the accuracy a bit. >> >> Well, it doesn't, really. If you click thru to accounting.test, >> you'll see that there aren't really that many tests. And, upon >> further investigation, the lines after the runSync calls aren't run, >> due to some exception most likely. I'm not certian if this is do to >> my changes, or if the tests themselves are broken. I'm running a >> plain test run now to check that. Plus, there actually *is* line hits >> in accounting.invoice. > > The tests seem to be running fine on buildbot > (http://ci.apache.org/waterfall?show=ofbiz-trunk), I'm guessing it's the > test run problem that's causing the under reporting. There may not be > that many explicit accounting tests (even though it is a lot compared to > other components) but a lot of tests also touch accounting indirectly. > There is just no way that only 53 lines of java code are being executed > in accounting during the full test run. I know for a fact that code is > executed from PaymentGatewayServices, FinAccountPaymentServices, > PaymentWorker, UtilAccounting and a few others during the tests. I had some other changes in that tree that were causing tests to fail. I've rerun it now, all current tests pass, and I've uploaded a new report to http://www.brainfood.com/ofbiz-coverage Note that framework/base has almost 100% coverage. But that's a bad thing, because it's not explicitly testing it; all that code just happens to be utilized during the rest of the test run. Total coverage increased from 7% to 14%. > > Regards > Scott |
|
Adam Heath wrote:
> Scott Gray wrote: >> On 11/12/2009, at 6:41 AM, Adam Heath wrote: >> >>> Scott Gray wrote: >>>> Hi Adam, >>>> >>>> Looking at the results my first impression is that the coverage is >>>> under-reported. For example, the accounting component has quite a few >>>> tests but no coverage is shown at all (except for the test package >>>> itself). Possibly because there is lot of logic in simple methods but >>>> I'm 100% sure java code is also run during the tests. >>>> >>>> But still a great start and something that will be immensely useful if >>>> we can up the accuracy a bit. >>> Well, it doesn't, really. If you click thru to accounting.test, >>> you'll see that there aren't really that many tests. And, upon >>> further investigation, the lines after the runSync calls aren't run, >>> due to some exception most likely. I'm not certian if this is do to >>> my changes, or if the tests themselves are broken. I'm running a >>> plain test run now to check that. Plus, there actually *is* line hits >>> in accounting.invoice. >> The tests seem to be running fine on buildbot >> (http://ci.apache.org/waterfall?show=ofbiz-trunk), I'm guessing it's the >> test run problem that's causing the under reporting. There may not be >> that many explicit accounting tests (even though it is a lot compared to >> other components) but a lot of tests also touch accounting indirectly. >> There is just no way that only 53 lines of java code are being executed >> in accounting during the full test run. I know for a fact that code is >> executed from PaymentGatewayServices, FinAccountPaymentServices, >> PaymentWorker, UtilAccounting and a few others during the tests. > > I had some other changes in that tree that were causing tests to fail. > I've rerun it now, all current tests pass, and I've uploaded a new > report to http://www.brainfood.com/ofbiz-coverage Oops, link doesn't work. |
|
Adrian Crum wrote:
> Adam Heath wrote: >> Scott Gray wrote: >>> On 11/12/2009, at 6:41 AM, Adam Heath wrote: >>> >>>> Scott Gray wrote: >>>>> Hi Adam, >>>>> >>>>> Looking at the results my first impression is that the coverage is >>>>> under-reported. For example, the accounting component has quite a few >>>>> tests but no coverage is shown at all (except for the test package >>>>> itself). Possibly because there is lot of logic in simple methods but >>>>> I'm 100% sure java code is also run during the tests. >>>>> >>>>> But still a great start and something that will be immensely useful if >>>>> we can up the accuracy a bit. >>>> Well, it doesn't, really. If you click thru to accounting.test, >>>> you'll see that there aren't really that many tests. And, upon >>>> further investigation, the lines after the runSync calls aren't run, >>>> due to some exception most likely. I'm not certian if this is do to >>>> my changes, or if the tests themselves are broken. I'm running a >>>> plain test run now to check that. Plus, there actually *is* line hits >>>> in accounting.invoice. >>> The tests seem to be running fine on buildbot >>> (http://ci.apache.org/waterfall?show=ofbiz-trunk), I'm guessing it's the >>> test run problem that's causing the under reporting. There may not be >>> that many explicit accounting tests (even though it is a lot compared to >>> other components) but a lot of tests also touch accounting >>> indirectly. There is just no way that only 53 lines of java code are >>> being executed >>> in accounting during the full test run. I know for a fact that code is >>> executed from PaymentGatewayServices, FinAccountPaymentServices, >>> PaymentWorker, UtilAccounting and a few others during the tests. >> >> I had some other changes in that tree that were causing tests to fail. >> I've rerun it now, all current tests pass, and I've uploaded a new >> report to http://www.brainfood.com/ofbiz-coverage > > Oops, link doesn't work. http://www.brainfood.com/ofbiz-coverage/ |
|
Adam Heath wrote:
> Adrian Crum wrote: >> Adam Heath wrote: >>> Scott Gray wrote: >>>> On 11/12/2009, at 6:41 AM, Adam Heath wrote: >>>> >>>>> Scott Gray wrote: >>>>>> Hi Adam, >>>>>> >>>>>> Looking at the results my first impression is that the coverage is >>>>>> under-reported. For example, the accounting component has quite a few >>>>>> tests but no coverage is shown at all (except for the test package >>>>>> itself). Possibly because there is lot of logic in simple methods but >>>>>> I'm 100% sure java code is also run during the tests. >>>>>> >>>>>> But still a great start and something that will be immensely useful if >>>>>> we can up the accuracy a bit. >>>>> Well, it doesn't, really. If you click thru to accounting.test, >>>>> you'll see that there aren't really that many tests. And, upon >>>>> further investigation, the lines after the runSync calls aren't run, >>>>> due to some exception most likely. I'm not certian if this is do to >>>>> my changes, or if the tests themselves are broken. I'm running a >>>>> plain test run now to check that. Plus, there actually *is* line hits >>>>> in accounting.invoice. >>>> The tests seem to be running fine on buildbot >>>> (http://ci.apache.org/waterfall?show=ofbiz-trunk), I'm guessing it's the >>>> test run problem that's causing the under reporting. There may not be >>>> that many explicit accounting tests (even though it is a lot compared to >>>> other components) but a lot of tests also touch accounting >>>> indirectly. There is just no way that only 53 lines of java code are >>>> being executed >>>> in accounting during the full test run. I know for a fact that code is >>>> executed from PaymentGatewayServices, FinAccountPaymentServices, >>>> PaymentWorker, UtilAccounting and a few others during the tests. >>> I had some other changes in that tree that were causing tests to fail. >>> I've rerun it now, all current tests pass, and I've uploaded a new >>> report to http://www.brainfood.com/ofbiz-coverage >> Oops, link doesn't work. > > http://www.brainfood.com/ofbiz-coverage/ I like the holiday-themed colors. It would be cool if incomplete coverage could be displayed as partially eaten sugar cookies. |
|
In reply to this post by Adam Heath-2
On 11/12/2009, at 11:55 AM, Adam Heath wrote:
> Scott Gray wrote: >> On 11/12/2009, at 6:41 AM, Adam Heath wrote: >> >>> Scott Gray wrote: >>>> Hi Adam, >>>> >>>> Looking at the results my first impression is that the coverage is >>>> under-reported. For example, the accounting component has quite >>>> a few >>>> tests but no coverage is shown at all (except for the test package >>>> itself). Possibly because there is lot of logic in simple >>>> methods but >>>> I'm 100% sure java code is also run during the tests. >>>> >>>> But still a great start and something that will be immensely >>>> useful if >>>> we can up the accuracy a bit. >>> >>> Well, it doesn't, really. If you click thru to accounting.test, >>> you'll see that there aren't really that many tests. And, upon >>> further investigation, the lines after the runSync calls aren't run, >>> due to some exception most likely. I'm not certian if this is do to >>> my changes, or if the tests themselves are broken. I'm running a >>> plain test run now to check that. Plus, there actually *is* line >>> hits >>> in accounting.invoice. >> >> The tests seem to be running fine on buildbot >> (http://ci.apache.org/waterfall?show=ofbiz-trunk), I'm guessing >> it's the >> test run problem that's causing the under reporting. There may not >> be >> that many explicit accounting tests (even though it is a lot >> compared to >> other components) but a lot of tests also touch accounting >> indirectly. >> There is just no way that only 53 lines of java code are being >> executed >> in accounting during the full test run. I know for a fact that >> code is >> executed from PaymentGatewayServices, FinAccountPaymentServices, >> PaymentWorker, UtilAccounting and a few others during the tests. > > I had some other changes in that tree that were causing tests to fail. > I've rerun it now, all current tests pass, and I've uploaded a new > report to http://www.brainfood.com/ofbiz-coverage > > Note that framework/base has almost 100% coverage. But that's a bad > thing, because it's not explicitly testing it; all that code just > happens to be utilized during the rest of the test run. Of course explicitly testing framework/base would be much better but why is the current 100% coverage a bad thing? I mean implied testing is better than no testing right? If I was to go in and incorrectly modify some of those base methods then there's a good chance some of the higher level tests would fail. > > Total coverage increased from 7% to 14%. |
|
In reply to this post by Adrian Crum
Adrian Crum wrote:
>> http://www.brainfood.com/ofbiz-coverage/ > > I like the holiday-themed colors. It would be cool if incomplete > coverage could be displayed as partially eaten sugar cookies. That's standard cobertura scheme. |
|
In reply to this post by Scott Gray-2
Scott Gray wrote:
>> Note that framework/base has almost 100% coverage. But that's a bad >> thing, because it's not explicitly testing it; all that code just >> happens to be utilized during the rest of the test run. > > Of course explicitly testing framework/base would be much better but why > is the current 100% coverage a bad thing? I mean implied testing is > better than no testing right? If I was to go in and incorrectly modify > some of those base methods then there's a good chance some of the higher > level tests would fail. In theory such explicit base testing could be run from the command line, very quickly, using the newly added 'tests' target defined in common.xml. Additionally, just because a line has been noted in cobertura, doesn't mean all variations have been tested. Consider the case that some condition is doing some kind of pattern match, or looking at Collection.contains or Map.containsKey. It's much simpler to verify that everything is tested when it is done explicitly. |
|
On 11/12/2009, at 12:34 PM, Adam Heath wrote:
> Scott Gray wrote: >>> Note that framework/base has almost 100% coverage. But that's a bad >>> thing, because it's not explicitly testing it; all that code just >>> happens to be utilized during the rest of the test run. >> >> Of course explicitly testing framework/base would be much better >> but why >> is the current 100% coverage a bad thing? I mean implied testing is >> better than no testing right? If I was to go in and incorrectly >> modify >> some of those base methods then there's a good chance some of the >> higher >> level tests would fail. > > In theory such explicit base testing could be run from the command > line, very quickly, using the newly added 'tests' target defined in > common.xml. > > Additionally, just because a line has been noted in cobertura, doesn't > mean all variations have been tested. Consider the case that some > condition is doing some kind of pattern match, or looking at > Collection.contains or Map.containsKey. It's much simpler to verify > that everything is tested when it is done explicitly. reported without explicit thorough testing. Even though the indirect coverage is still better than no coverage whatsoever. |
|
Scott Gray wrote:
>> Additionally, just because a line has been noted in cobertura, doesn't >> mean all variations have been tested. Consider the case that some >> condition is doing some kind of pattern match, or looking at >> Collection.contains or Map.containsKey. It's much simpler to verify >> that everything is tested when it is done explicitly. > > Okay I see what you mean now, it's a bad thing that coverage is reported > without explicit thorough testing. Even though the indirect coverage is > still better than no coverage whatsoever. As a better example, let's say that there is only 10% coverage on the entire ofbiz code base. But base has 100% coverage. That other 90% of untested code may test parts of base that may not work, and would break the higher-level code. It's easier to write tests that are close to the code being tested. Trying to tweak a high-level test to make certain all low-level code is wrong is very very difficult. Plus, if a typo gets introduced in one of those map keys, it might not be detected until much much later in time, when explicit tests are not used. In my opinion, as each new component is activated in the ofbiz system, it should have it's own set of tests that move it close to 100% coverage. So, I can test just base, and get 100%, then base+sql, and get 100% on base+sql, then base+sql+entity, and get 100% on base+sql+entity, and so on. You want to make certain that earlier components are correct before utilizing later ones, or the entire test run may fail spectactulary. |
|
In reply to this post by Scott Gray-2
Now the point would be to show the coverage of the simple methods and
also, maybe in sometime, the selenium coverage. What would be great is finding a way to indicate which services are tested, and then display it in the webtools. This won't give an information as precise as cobertura but would add some quick display on what is or can be tested. If we agree on a syntax to use, I would be ready to add the screen in OFBiz. Cheers, Le 11/12/2009 00:21, Scott Gray a écrit : ../.. > > -- Erwan de FERRIERES www.nereide.biz |
|
Administrator
|
What about groovy scripts, are they handled by Cobertura?
And actions in Screens, is it worth do something? I guess checking "static structural" files like web.xml, ofbiz-component.xml files (and all xml like them, controllers, menus, commonsScreens and such) is not necessary? Jacques From: "Erwan de FERRIERES" <[hidden email]> > Now the point would be to show the coverage of the simple methods and also, maybe in sometime, the selenium coverage. > > What would be great is finding a way to indicate which services are tested, and then display it in the webtools. This won't give > an information as precise as cobertura but would add some quick display on what is or can be tested. > > If we agree on a syntax to use, I would be ready to add the screen in OFBiz. > > Cheers, > > Le 11/12/2009 00:21, Scott Gray a écrit : > ../.. >> >> > > -- > Erwan de FERRIERES > www.nereide.biz > |
|
Administrator
|
In reply to this post by Adam Heath-2
From: "Adam Heath" <[hidden email]>
> Additionally, the way the classloaders are constructed is a bit odd. > The bootstrap loader only has ofbiz.jar. The code then creates a > classloader will *all* jars in base/lib, recursively. After the > containers and components are started, yet another classloader is > created, this time with proper libs and all other ofbiz-$foo.jar. > Both of those classloaders update the java.class.path system property. > This is a bug, as the instrumentation changes all ofbiz-$foo.jar into > a new jar in /tmp, so the value of the property is different each > time. Additionally, the second created classloader should not load > the base component again, as that is sub-optimal. > > I'm thinking that component code should really move to start. > > Another bug, is that due to one of the above bugs, the first > classloader is not instrumented at all. Since this is the classloader > that contains framework/base, you don't see any line hits in the report. Since the the start (my start with OFBiz) I have always wondered about complexity there Jacques |
| Free forum by Nabble | Edit this page |
