You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Bhuvaneswaran A <bh...@collab.net> on 2009/10/22 15:22:51 UTC
[PATCH] Junit report for Subversion tests
Hello,
Please find attached a script to generate the Junit report based on
Subversion test results. The junit reports can be used to integrate the
Subversion build process with continuous integration tools like Hudson.
[[
New script to generate the junit reports for Subversion test cases. The
junit reports can be used by continuous integration tools like Hudson
for building and testing Subversion.
* tools/dev/gen_junit_report.py
The junit reports are generated by parsing the tests log file,
tests.log. The test results, PASS, FAIL and SKIP are marked as
appropriate in the junit reports. The XFAIL tests are considered as
PASS. This script can take options: --log-file and --output-dir.
The value for --log-file option denotes the log file to parse for
generating the junit reports.
The value for --output-dir option denotes the directory to store the
junit reports. A separate file is created in this directory for each
test case.
]]
--
Bhuvaneswaran A
CollabNet Software P Ltd. | www.collab.net
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2410275
Re: [PATCH] Junit report for Subversion tests
Posted by Mark Phippard <ma...@gmail.com>.
On Thu, Oct 22, 2009 at 2:17 PM, Bhuvaneswaran A <bh...@collab.net> wrote:
> On Thu, 2009-10-22 at 13:49 -0400, Mark Phippard wrote:
>>
>> Before we backport it, let's be sure we have finished fine-tuning it
>> on trunk. For example, I do not think the times it is reporting is
>> right. 12 minutes for all the tests? I do not think so. Also see
>> this page that has times:
>>
>> http://orac.ece.utexas.edu:8080/job/svn-trunk/294/testReport/(root)/
>>
>> Being able to see trends in the test times is one of the things I am
>> hoping to get out of this so it would be good to be sure it is right.
>
> It's the time taken by "make check" process, yes! It seem to be a server
> with 4GB RAM having *4* processors, each running "Intel(R) Core(TM)2
> Quad CPU @ 2.66GHz". Hyrum, correct me if i'm wrong.
>
> The sum of all >1 seconds test cases sum up to 12.6 minutes.
It did not run 62 commit tests in 0 seconds and there are a lot of
tests like that. Also, the tests are running single threaded so 4 CPU
will not help.
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2410358
Re: [PATCH] Junit report for Subversion tests
Posted by Gavin Baumanis <ga...@thespidernet.com>.
Thanks.
I'll remove it from the "To Do List".
Gavin.
On 09/11/2009, at 17:04 , Bhuvaneswaran A wrote:
> On Sat, 2009-11-07 at 20:30 +1100, Gavin 'Beau' Baumanis wrote:
>> I have logged issue #3521
>> http://subversion.tigris.org/issues/show_bug.cgi?id=3521
>>
>> Gavin.
>
> Gavin,
> The script along with follow-up fixes are already committed to trunk.
> See r40187 and r40228 for details.
>
>> On 03/11/2009, at 10:18 , Gavin Baumanis wrote:
>>
>>> Hi All,
>>>
>>> Although there is a patch submission - there was a quite a bit of
>>> discussion too.
>>> Though there doesn't seem to have been an agreement or a subsequent
>>> patch submission (if at all needed).
>>> But since it has been a week now without activity I thought I would
>>> bring it back up the list - in case it had fallen off people's
>>> radar.
>>>
>>> Gavin.
>>>
>>> On 27/10/2009, at 14:55 , Branko Cibej wrote:
>>>
>>>> Hyrum K. Wright wrote:
>>>>> On Oct 22, 2009, at 1:17 PM, Bhuvaneswaran A wrote:
>>>>>
>>>>>
>>>>>> The sum of all >1 seconds test cases sum up to 12.6 minutes.
>>>>>>
>>>>>
>>>>> That does seem rather quick. I just ran 'make check' on a
>>>>> separate
>>>>> working copy on the same machine, and got something different:
>>>>> real 32m51.756s
>>>>> user 6m40.321s
>>>>> sys 6m27.504s
>>>>>
>>>>> I don't know that this is a problem with the junit converter
>>>>> script,
>>>>> or perhaps the test running reporting bogus ELAPSED times.
>>>>>
>>>>
>>>> Apparently it's quite reasonable. As I posted elsewhere, I got a
>>>> similar
>>>> "real" time on my laptop (with tmpfs and cleanups and parallel test
>>>> executions), but my user time was almost 22 minutes. And I have an
>>>> elapsed time for commit-tests.py under one minute. While it's
>>>> easy to
>>>> imagine this being twice as fast on a real machine with something
>>>> more
>>>> than this trundle for a CPU, I can hardly imagine how that would be
>>>> truncated to 0 seconds.
>>>>
>>>> -- Brane
>>>>
>>>> ------------------------------------------------------
>>>> http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2411585
>>>
>>> ------------------------------------------------------
>>> http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2413974
>>
>
>
> --
> Bhuvaneswaran A
> CollabNet Software P Ltd. | www.collab.net
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2415737
Re: [PATCH] Junit report for Subversion tests
Posted by Bhuvaneswaran A <bh...@collab.net>.
On Sat, 2009-11-07 at 20:30 +1100, Gavin 'Beau' Baumanis wrote:
> I have logged issue #3521
> http://subversion.tigris.org/issues/show_bug.cgi?id=3521
>
> Gavin.
Gavin,
The script along with follow-up fixes are already committed to trunk.
See r40187 and r40228 for details.
> On 03/11/2009, at 10:18 , Gavin Baumanis wrote:
>
> > Hi All,
> >
> > Although there is a patch submission - there was a quite a bit of
> > discussion too.
> > Though there doesn't seem to have been an agreement or a subsequent
> > patch submission (if at all needed).
> > But since it has been a week now without activity I thought I would
> > bring it back up the list - in case it had fallen off people's radar.
> >
> > Gavin.
> >
> > On 27/10/2009, at 14:55 , Branko Cibej wrote:
> >
> >> Hyrum K. Wright wrote:
> >>> On Oct 22, 2009, at 1:17 PM, Bhuvaneswaran A wrote:
> >>>
> >>>
> >>>> The sum of all >1 seconds test cases sum up to 12.6 minutes.
> >>>>
> >>>
> >>> That does seem rather quick. I just ran 'make check' on a separate
> >>> working copy on the same machine, and got something different:
> >>> real 32m51.756s
> >>> user 6m40.321s
> >>> sys 6m27.504s
> >>>
> >>> I don't know that this is a problem with the junit converter script,
> >>> or perhaps the test running reporting bogus ELAPSED times.
> >>>
> >>
> >> Apparently it's quite reasonable. As I posted elsewhere, I got a
> >> similar
> >> "real" time on my laptop (with tmpfs and cleanups and parallel test
> >> executions), but my user time was almost 22 minutes. And I have an
> >> elapsed time for commit-tests.py under one minute. While it's easy to
> >> imagine this being twice as fast on a real machine with something
> >> more
> >> than this trundle for a CPU, I can hardly imagine how that would be
> >> truncated to 0 seconds.
> >>
> >> -- Brane
> >>
> >> ------------------------------------------------------
> >> http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2411585
> >
> > ------------------------------------------------------
> > http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2413974
>
--
Bhuvaneswaran A
CollabNet Software P Ltd. | www.collab.net
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2415681
Re: [PATCH] Junit report for Subversion tests
Posted by Gavin Baumanis <ga...@thespidernet.com>.
I have logged issue #3521
http://subversion.tigris.org/issues/show_bug.cgi?id=3521
Gavin.
On 03/11/2009, at 10:18 , Gavin Baumanis wrote:
> Hi All,
>
> Although there is a patch submission - there was a quite a bit of
> discussion too.
> Though there doesn't seem to have been an agreement or a subsequent
> patch submission (if at all needed).
> But since it has been a week now without activity I thought I would
> bring it back up the list - in case it had fallen off people's radar.
>
> Gavin.
>
> On 27/10/2009, at 14:55 , Branko Cibej wrote:
>
>> Hyrum K. Wright wrote:
>>> On Oct 22, 2009, at 1:17 PM, Bhuvaneswaran A wrote:
>>>
>>>
>>>> The sum of all >1 seconds test cases sum up to 12.6 minutes.
>>>>
>>>
>>> That does seem rather quick. I just ran 'make check' on a separate
>>> working copy on the same machine, and got something different:
>>> real 32m51.756s
>>> user 6m40.321s
>>> sys 6m27.504s
>>>
>>> I don't know that this is a problem with the junit converter script,
>>> or perhaps the test running reporting bogus ELAPSED times.
>>>
>>
>> Apparently it's quite reasonable. As I posted elsewhere, I got a
>> similar
>> "real" time on my laptop (with tmpfs and cleanups and parallel test
>> executions), but my user time was almost 22 minutes. And I have an
>> elapsed time for commit-tests.py under one minute. While it's easy to
>> imagine this being twice as fast on a real machine with something
>> more
>> than this trundle for a CPU, I can hardly imagine how that would be
>> truncated to 0 seconds.
>>
>> -- Brane
>>
>> ------------------------------------------------------
>> http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2411585
>
> ------------------------------------------------------
> http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2413974
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2415375
Re: [PATCH] Junit report for Subversion tests
Posted by Gavin Baumanis <ga...@thespidernet.com>.
Hi All,
Although there is a patch submission - there was a quite a bit of
discussion too.
Though there doesn't seem to have been an agreement or a subsequent
patch submission (if at all needed).
But since it has been a week now without activity I thought I would
bring it back up the list - in case it had fallen off people's radar.
Gavin.
On 27/10/2009, at 14:55 , Branko Cibej wrote:
> Hyrum K. Wright wrote:
>> On Oct 22, 2009, at 1:17 PM, Bhuvaneswaran A wrote:
>>
>>
>>> The sum of all >1 seconds test cases sum up to 12.6 minutes.
>>>
>>
>> That does seem rather quick. I just ran 'make check' on a separate
>> working copy on the same machine, and got something different:
>> real 32m51.756s
>> user 6m40.321s
>> sys 6m27.504s
>>
>> I don't know that this is a problem with the junit converter script,
>> or perhaps the test running reporting bogus ELAPSED times.
>>
>
> Apparently it's quite reasonable. As I posted elsewhere, I got a
> similar
> "real" time on my laptop (with tmpfs and cleanups and parallel test
> executions), but my user time was almost 22 minutes. And I have an
> elapsed time for commit-tests.py under one minute. While it's easy to
> imagine this being twice as fast on a real machine with something more
> than this trundle for a CPU, I can hardly imagine how that would be
> truncated to 0 seconds.
>
> -- Brane
>
> ------------------------------------------------------
> http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2411585
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2413974
Re: [PATCH] Junit report for Subversion tests
Posted by Branko Cibej <br...@xbc.nu>.
Hyrum K. Wright wrote:
> On Oct 22, 2009, at 1:17 PM, Bhuvaneswaran A wrote:
>
>
>> The sum of all >1 seconds test cases sum up to 12.6 minutes.
>>
>
> That does seem rather quick. I just ran 'make check' on a separate
> working copy on the same machine, and got something different:
> real 32m51.756s
> user 6m40.321s
> sys 6m27.504s
>
> I don't know that this is a problem with the junit converter script,
> or perhaps the test running reporting bogus ELAPSED times.
>
Apparently it's quite reasonable. As I posted elsewhere, I got a similar
"real" time on my laptop (with tmpfs and cleanups and parallel test
executions), but my user time was almost 22 minutes. And I have an
elapsed time for commit-tests.py under one minute. While it's easy to
imagine this being twice as fast on a real machine with something more
than this trundle for a CPU, I can hardly imagine how that would be
truncated to 0 seconds.
-- Brane
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2411585
Re: [PATCH] Junit report for Subversion tests
Posted by Mark Phippard <ma...@gmail.com>.
On Mon, Oct 26, 2009 at 10:02 AM, Bhuvaneswaran A <bh...@collab.net> wrote:
> On Mon, 2009-10-26 at 08:54 -0400, Mark Phippard wrote:
>>
>> I do not understand how this accounts for such a big discrepancy in
>> the time reported? For example, as I pointed out, the entire
>> commit_tests.py reports as taking 0ms. How is including and not
>> rounding the milliseconds going to suddenly make that reports as the
>> several minutes that it in fact does take?
>
> I'll explain the issue using the commit_tests.py test suite.
>
> Originally, the commit_tests.py test suite has taken 39 seconds to
> complete. In total, there are 62 tests comprising this test suite. The
> time taken by each test is set as 0 seconds in the junit file instead of
> 0.62 seconds (39/62). Due to this fact, the Hudson set the total time
> taken for this test suite as 0 (0*62). It is irrespective of the fact
> that the actual time for this test suite is defined as 39 seconds in the
> junit file, globally.
>
>> As someone else speculated, is it possible that the test setup and
>> teardown is not being included in the times?
>
> The ELAPSED: line in tests.log prints the *exact* time taken to execute
> each test suite. If the setup/teardown part is not performed within
> START/END of test suite, then yes, it's not included.
>
> I'll look into this, once we are done with the known bug. Thanks!
OK, thanks. I thought those tests took a few minutes on my system,
but if they run that fast, then maybe that explains it?
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2411395
Re: [PATCH] Junit report for Subversion tests
Posted by Bhuvaneswaran A <bh...@collab.net>.
On Mon, 2009-10-26 at 22:09 +0100, Hyrum K. Wright wrote:
>
> On Oct 26, 2009, at 3:02 PM, Bhuvaneswaran A wrote:
>
> > On Mon, 2009-10-26 at 08:54 -0400, Mark Phippard wrote:
> >>
> >> I do not understand how this accounts for such a big discrepancy in
> >> the time reported? For example, as I pointed out, the entire
> >> commit_tests.py reports as taking 0ms. How is including and not
> >> rounding the milliseconds going to suddenly make that reports as
> the
> >> several minutes that it in fact does take?
> >
> > I'll explain the issue using the commit_tests.py test suite.
> >
> > Originally, the commit_tests.py test suite has taken 39 seconds to
> > complete. In total, there are 62 tests comprising this test suite.
> The
>
> I find it really hard to believe that the commit test suite only
> takes
> 39 seconds to run.
That's what the /var/lib/hudson/jobs/job-name/workspace/co-dir/tests.log
says:
ELAPSED: commit_tests.py 00:00:39
> (In other news, the admin of orac tells me it had a disk failure, so
> it may be offline for a bit.)
Sounds like it's not up even after 4hrs.
--
Bhuvaneswaran A
CollabNet Software P Ltd. | www.collab.net
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2411568
Re: [PATCH] Junit report for Subversion tests
Posted by "Hyrum K. Wright" <hy...@hyrumwright.org>.
On Oct 26, 2009, at 3:02 PM, Bhuvaneswaran A wrote:
> On Mon, 2009-10-26 at 08:54 -0400, Mark Phippard wrote:
>>
>> I do not understand how this accounts for such a big discrepancy in
>> the time reported? For example, as I pointed out, the entire
>> commit_tests.py reports as taking 0ms. How is including and not
>> rounding the milliseconds going to suddenly make that reports as the
>> several minutes that it in fact does take?
>
> I'll explain the issue using the commit_tests.py test suite.
>
> Originally, the commit_tests.py test suite has taken 39 seconds to
> complete. In total, there are 62 tests comprising this test suite. The
I find it really hard to believe that the commit test suite only takes
39 seconds to run.
(In other news, the admin of orac tells me it had a disk failure, so
it may be offline for a bit.)
> time taken by each test is set as 0 seconds in the junit file
> instead of
> 0.62 seconds (39/62). Due to this fact, the Hudson set the total time
> taken for this test suite as 0 (0*62). It is irrespective of the fact
> that the actual time for this test suite is defined as 39 seconds in
> the
> junit file, globally.
>
>> As someone else speculated, is it possible that the test setup and
>> teardown is not being included in the times?
>
> The ELAPSED: line in tests.log prints the *exact* time taken to
> execute
> each test suite. If the setup/teardown part is not performed within
> START/END of test suite, then yes, it's not included.
>
> I'll look into this, once we are done with the known bug. Thanks!
> --
> Bhuvaneswaran A
> CollabNet Software P Ltd. | www.collab.net
>
> ------------------------------------------------------
> http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2411393
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2411527
Re: [PATCH] Junit report for Subversion tests
Posted by Bhuvaneswaran A <bh...@collab.net>.
On Mon, 2009-10-26 at 08:54 -0400, Mark Phippard wrote:
>
> I do not understand how this accounts for such a big discrepancy in
> the time reported? For example, as I pointed out, the entire
> commit_tests.py reports as taking 0ms. How is including and not
> rounding the milliseconds going to suddenly make that reports as the
> several minutes that it in fact does take?
I'll explain the issue using the commit_tests.py test suite.
Originally, the commit_tests.py test suite has taken 39 seconds to
complete. In total, there are 62 tests comprising this test suite. The
time taken by each test is set as 0 seconds in the junit file instead of
0.62 seconds (39/62). Due to this fact, the Hudson set the total time
taken for this test suite as 0 (0*62). It is irrespective of the fact
that the actual time for this test suite is defined as 39 seconds in the
junit file, globally.
> As someone else speculated, is it possible that the test setup and
> teardown is not being included in the times?
The ELAPSED: line in tests.log prints the *exact* time taken to execute
each test suite. If the setup/teardown part is not performed within
START/END of test suite, then yes, it's not included.
I'll look into this, once we are done with the known bug. Thanks!
--
Bhuvaneswaran A
CollabNet Software P Ltd. | www.collab.net
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2411393
Re: [PATCH] Junit report for Subversion tests
Posted by Mark Phippard <ma...@gmail.com>.
On Mon, Oct 26, 2009 at 6:44 AM, Bhuvaneswaran A <bh...@collab.net> wrote:
> On Thu, 2009-10-22 at 14:11 -0500, Hyrum K. Wright wrote:
>>
>> That does seem rather quick. I just ran 'make check' on a separate
>> working copy on the same machine, and got something different:
>> real 32m51.756s
>> user 6m40.321s
>> sys 6m27.504s
>>
>> I don't know that this is a problem with the junit converter script,
>> or perhaps the test running reporting bogus ELAPSED times.
>
> This seem to be a problem with combination of both scripts. As there is
> no way to calculate the time taken for EACH TEST, we calculate it based
> on total time taken. However it is not done correctly.
>
> For instance, consider a test suite that has 10 tests taking 15 seconds
> to complete. We don't know the time taken for 10 tests individually. We
> only know that it took 15 seconds for all 10 tests together. We
> calculate the time taken for individual tests as 1 instead of 1.5. It's
> due to a bug in gen_junit_report.py script.
>
> On the other hand, since we do not account for milliseconds, those tests
> are recorded as 0 ms for individual tests.
>
> The Hudson as such, does not recognise the time specified for each test
> suite in the junit report. Instead it calculates the time taken for each
> test suite based on time taken for each test individually. Due to this
> bug, in the above use case, the Hudson would report 10 seconds, because
> according to the junit report, each test has taken 1 second.
>
> Bottom line, I see two issues to be fixed here:
> a) Include milliseconds in the tests log file, tests.log.
> b) Do not round the result while calculating the time taken for each
> test.
I do not understand how this accounts for such a big discrepancy in
the time reported? For example, as I pointed out, the entire
commit_tests.py reports as taking 0ms. How is including and not
rounding the milliseconds going to suddenly make that reports as the
several minutes that it in fact does take?
As someone else speculated, is it possible that the test setup and
teardown is not being included in the times?
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2411373
Re: [PATCH] Junit report for Subversion tests
Posted by Bhuvaneswaran A <bh...@collab.net>.
On Thu, 2009-10-22 at 14:11 -0500, Hyrum K. Wright wrote:
>
> That does seem rather quick. I just ran 'make check' on a separate
> working copy on the same machine, and got something different:
> real 32m51.756s
> user 6m40.321s
> sys 6m27.504s
>
> I don't know that this is a problem with the junit converter script,
> or perhaps the test running reporting bogus ELAPSED times.
This seem to be a problem with combination of both scripts. As there is
no way to calculate the time taken for EACH TEST, we calculate it based
on total time taken. However it is not done correctly.
For instance, consider a test suite that has 10 tests taking 15 seconds
to complete. We don't know the time taken for 10 tests individually. We
only know that it took 15 seconds for all 10 tests together. We
calculate the time taken for individual tests as 1 instead of 1.5. It's
due to a bug in gen_junit_report.py script.
On the other hand, since we do not account for milliseconds, those tests
are recorded as 0 ms for individual tests.
The Hudson as such, does not recognise the time specified for each test
suite in the junit report. Instead it calculates the time taken for each
test suite based on time taken for each test individually. Due to this
bug, in the above use case, the Hudson would report 10 seconds, because
according to the junit report, each test has taken 1 second.
Bottom line, I see two issues to be fixed here:
a) Include milliseconds in the tests log file, tests.log.
b) Do not round the result while calculating the time taken for each
test.
Attached a patch for issue {a} in another thread.
> Another problem: on some of the test information pages, I see stuff
> like:
> lock_tests-py.lock_tests.py 1: lock a file and verify that it's
> locked
> The html entity looks a bit out-of-place.
We do escape the xml entities {<,>,',",&}. They seem to work well with
the test names when they are displayed in the report. In the final
screen where we see the status of the test, it does not play along well.
I'll see to fix this.
Thank you.
--
Bhuvaneswaran A
CollabNet Software P Ltd. | www.collab.net
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2411335
Re: [PATCH] Junit report for Subversion tests
Posted by Bhuvaneswaran A <bh...@collab.net>.
On Thu, 2009-10-22 at 22:09 +0200, Branko Čibej wrote:
>
> Upon further consideration, I think that we should be recording
> milliseconds for elapsed times, too.
Yes, I agree. I'll attach a patch for this. Thank you.
--
Bhuvaneswaran A
CollabNet Software P Ltd. | www.collab.net
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2411327
Re: [PATCH] Junit report for Subversion tests
Posted by Branko Cibej <br...@xbc.nu>.
Hyrum K. Wright wrote:
> On Oct 22, 2009, at 1:17 PM, Bhuvaneswaran A wrote:
>
>
>> On Thu, 2009-10-22 at 13:49 -0400, Mark Phippard wrote:
>>
>>> Before we backport it, let's be sure we have finished fine-tuning it
>>> on trunk. For example, I do not think the times it is reporting is
>>> right. 12 minutes for all the tests? I do not think so. Also see
>>> this page that has times:
>>>
>>> http://orac.ece.utexas.edu:8080/job/svn-trunk/294/testReport/(root)/
>>>
>
> Kinda hopin' not to put that URL in the email archive just yet...
>
>
>>> Being able to see trends in the test times is one of the things I am
>>> hoping to get out of this so it would be good to be sure it is right.
>>>
>> It's the time taken by "make check" process, yes! It seem to be a
>> server
>> with 4GB RAM having *4* processors, each running "Intel(R) Core(TM)2
>> Quad CPU @ 2.66GHz". Hyrum, correct me if i'm wrong.
>>
>
> This is indeed the configuration of the machine.
>
>
>> The sum of all >1 seconds test cases sum up to 12.6 minutes.
>>
>
> That does seem rather quick. I just ran 'make check' on a separate
> working copy on the same machine, and got something different:
> real 32m51.756s
> user 6m40.321s
> sys 6m27.504s
>
> I don't know that this is a problem with the junit converter script,
> or perhaps the test running reporting bogus ELAPSED times.
>
I do believe that the elapsed times added in the patch did not take
account of the test setup and especially cleanup times. The latter could
easily account for half of the test runtime.
Upon further consideration, I think that we should be recording
milliseconds for elapsed times, too.
-- Brane
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2410405
Re: [PATCH] Junit report for Subversion tests
Posted by "Hyrum K. Wright" <hy...@hyrumwright.org>.
On Oct 22, 2009, at 1:17 PM, Bhuvaneswaran A wrote:
> On Thu, 2009-10-22 at 13:49 -0400, Mark Phippard wrote:
>>
>> Before we backport it, let's be sure we have finished fine-tuning it
>> on trunk. For example, I do not think the times it is reporting is
>> right. 12 minutes for all the tests? I do not think so. Also see
>> this page that has times:
>>
>> http://orac.ece.utexas.edu:8080/job/svn-trunk/294/testReport/(root)/
Kinda hopin' not to put that URL in the email archive just yet...
>>
>> Being able to see trends in the test times is one of the things I am
>> hoping to get out of this so it would be good to be sure it is right.
>
> It's the time taken by "make check" process, yes! It seem to be a
> server
> with 4GB RAM having *4* processors, each running "Intel(R) Core(TM)2
> Quad CPU @ 2.66GHz". Hyrum, correct me if i'm wrong.
This is indeed the configuration of the machine.
> The sum of all >1 seconds test cases sum up to 12.6 minutes.
That does seem rather quick. I just ran 'make check' on a separate
working copy on the same machine, and got something different:
real 32m51.756s
user 6m40.321s
sys 6m27.504s
I don't know that this is a problem with the junit converter script,
or perhaps the test running reporting bogus ELAPSED times.
Another problem: on some of the test information pages, I see stuff
like:
lock_tests-py.lock_tests.py 1: lock a file and verify that it's
locked
The html entity looks a bit out-of-place.
-Hyrum
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2410374
Re: [PATCH] Junit report for Subversion tests
Posted by Bhuvaneswaran A <bh...@collab.net>.
On Thu, 2009-10-22 at 13:49 -0400, Mark Phippard wrote:
>
> Before we backport it, let's be sure we have finished fine-tuning it
> on trunk. For example, I do not think the times it is reporting is
> right. 12 minutes for all the tests? I do not think so. Also see
> this page that has times:
>
> http://orac.ece.utexas.edu:8080/job/svn-trunk/294/testReport/(root)/
>
> Being able to see trends in the test times is one of the things I am
> hoping to get out of this so it would be good to be sure it is right.
It's the time taken by "make check" process, yes! It seem to be a server
with 4GB RAM having *4* processors, each running "Intel(R) Core(TM)2
Quad CPU @ 2.66GHz". Hyrum, correct me if i'm wrong.
The sum of all >1 seconds test cases sum up to 12.6 minutes.
--
Bhuvaneswaran A
CollabNet Software P Ltd. | www.collab.net
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2410342
Re: [PATCH] Junit report for Subversion tests
Posted by Mark Phippard <ma...@gmail.com>.
On Thu, Oct 22, 2009 at 1:38 PM, Bhuvaneswaran A <bh...@collab.net> wrote:
> On Thu, 2009-10-22 at 10:31 -0500, Hyrum K. Wright wrote:
>> We've been using this to test a sample Hudson instance for
>> Subversion
>> trunk and 1.6.x. I asked Bhuvan to send the patch here so I could
>> +1
>> it to live in the repo. Version controlled, open source, useful for
>> posterity, and all that.
>>
>> +1 to commit.
>
> Thank you, Hyrum. Committed this script in r40187. May I: (a) nominate
> this script for 1.6.x branch. (b) include your vote, in addition to mine
> in 1.6.x/STATUS?
Before we backport it, let's be sure we have finished fine-tuning it
on trunk. For example, I do not think the times it is reporting is
right. 12 minutes for all the tests? I do not think so. Also see
this page that has times:
http://orac.ece.utexas.edu:8080/job/svn-trunk/294/testReport/(root)/
Being able to see trends in the test times is one of the things I am
hoping to get out of this so it would be good to be sure it is right.
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2410330
Re: [PATCH] Junit report for Subversion tests
Posted by Bhuvaneswaran A <bh...@collab.net>.
On Thu, 2009-10-22 at 10:31 -0500, Hyrum K. Wright wrote:
> We've been using this to test a sample Hudson instance for
> Subversion
> trunk and 1.6.x. I asked Bhuvan to send the patch here so I could
> +1
> it to live in the repo. Version controlled, open source, useful for
> posterity, and all that.
>
> +1 to commit.
Thank you, Hyrum. Committed this script in r40187. May I: (a) nominate
this script for 1.6.x branch. (b) include your vote, in addition to mine
in 1.6.x/STATUS?
--
Bhuvaneswaran A
CollabNet Software P Ltd. | www.collab.net
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2410325
Re: [PATCH] Junit report for Subversion tests
Posted by "Hyrum K. Wright" <hy...@hyrumwright.org>.
We've been using this to test a sample Hudson instance for Subversion
trunk and 1.6.x. I asked Bhuvan to send the patch here so I could +1
it to live in the repo. Version controlled, open source, useful for
posterity, and all that.
+1 to commit.
-Hyrum
On Oct 22, 2009, at 10:22 AM, Bhuvaneswaran A wrote:
> Hello,
> Please find attached a script to generate the Junit report based on
> Subversion test results. The junit reports can be used to integrate
> the
> Subversion build process with continuous integration tools like
> Hudson.
>
> [[
> New script to generate the junit reports for Subversion test cases.
> The
> junit reports can be used by continuous integration tools like Hudson
> for building and testing Subversion.
>
> * tools/dev/gen_junit_report.py
> The junit reports are generated by parsing the tests log file,
> tests.log. The test results, PASS, FAIL and SKIP are marked as
> appropriate in the junit reports. The XFAIL tests are considered as
> PASS. This script can take options: --log-file and --output-dir.
>
> The value for --log-file option denotes the log file to parse for
> generating the junit reports.
>
> The value for --output-dir option denotes the directory to store the
> junit reports. A separate file is created in this directory for each
> test case.
> ]]
> --
> Bhuvaneswaran A
> CollabNet Software P Ltd. | www.collab.net
>
> ------------------------------------------------------
> http://subversion.tigris.org/ds/viewMessage.do?
> dsForumId=462&dsMessageId=2410275<gen-junit.diff>
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2410278