You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Bert Huijben <rh...@sharpsvn.net> on 2009/03/29 21:40:26 UTC
Test results on buildbot
Recently the test framework was changed to support work-in-progress
tests. When this support was completed the Windows buildbot was offline.
Since a few days the buildbot is back online (Thanks D.J. Heap!), and
for the last few days the Windows tests don't report an error in the
summary:
===============
At least one test was SKIPPED, checking M:\svn-auto-test\fsfs\tests.log
SKIP: cache-test.exe 2: basic memcache svn_cache test
SKIP: cache-test.exe 3: memcache svn_cache with very long keys
SKIP: ra-local-test.exe 4: svn_ra_local__split_URL: invalid host names
SKIP: commit_tests.py 34: tabs in paths
SKIP: prop_tests.py 23: issue #2581: propset shouldn't touch symlink
perms
SKIP: schedule_tests.py 4: revert: add some executable files
SKIP: copy_tests.py 10: executable bit should be preserved when copying
At least one test XFAILED, checking M:\svn-auto-test\fsfs\tests.log
XFAIL: fs-test.exe 18: merging commit
[[needs to be written to match new merge() algorithm
expectations]]
XFAIL: locks-test.exe 9: able to reserve a name (lock non-existent path)
XFAIL: locks-test.exe 10: directory locks (kinda)
XFAIL: tree-conflict-data-test.exe 3: detect broken tree conflict data
XFAIL: basic_tests.py 38: remotely remove directories from two
repositories
XFAIL: update_tests.py 50: tree conflicts 2.3: skip on 2nd update
XFAIL: switch_tests.py 10: switch a file to a dir and back to the file
XFAIL: log_tests.py 21: test log -c on range of changes
XFAIL: diff_tests.py 28: diff a renamed directory
XFAIL: diff_tests.py 49: diff URL against working copy with local mods
XFAIL: diff_tests.py 50: diff -r1 of removed file to its local addition
Summary of test results:
659 tests PASSED
7 tests SKIPPED
11 tests XFAILED (1 WORK-IN-PROGRESS)
02:46 PM
=========================
But the testresult is still failed (red) and broken builds are reported
to the svn-breakage list.
If I look in the raw log file I find 'FAIL:' lines for the
work-in-progress patch tests, but those aren't reported in the summary
(Maybe because it doesn't look for 'FAIL:' lines?).
Could somebody please look at this issue?
See http://www.mobsol.be/buildbot/waterfall for the last report.
Thanks,
Bert
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=1474339
Re: Test results on buildbot
Posted by Lieven Govaerts <sv...@mobsol.be>.
On Mon, Mar 30, 2009 at 1:08 AM, Branko Cibej <br...@xbc.nu> wrote:
> Bert Huijben wrote:
>> Recently the test framework was changed to support work-in-progress
>> tests. When this support was completed the Windows buildbot was offline.
>>
>> Since a few days the buildbot is back online (Thanks D.J. Heap!), and
>> for the last few days the Windows tests don't report an error in the
>> summary:
..
>
> Hm. I made sure that any failures reported from wimpy tests did not
> cause the tests to exit with non-zero status. Individual per-test
> results should be XPASS and XFAIL, too.
>
> IMHO buildbots should look at test exit status, not grep the log file;
> run-tests.py already does that better.
>
Buildbot does not and has never relied on parsing the log files. The
only indication it uses of failure state is by checking the return
value of the script ran by the slave. Note that in most cases this is
not 'make check' directly, but something like svncheck.sh or
svncheck.bat.
Lieven
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=1480832
Re: Test results on buildbot
Posted by Branko Cibej <br...@xbc.nu>.
Branko Čibej wrote:
> Bert Huijben wrote:
>
>> Recently the test framework was changed to support work-in-progress
>> tests. When this support was completed the Windows buildbot was offline.
>>
>> Since a few days the buildbot is back online (Thanks D.J. Heap!), and
>> for the last few days the Windows tests don't report an error in the
>> summary:
>>
>> ===============
>>
>>
> [...]
>
>> =========================
>>
>> But the testresult is still failed (red) and broken builds are reported
>> to the svn-breakage list.
>>
>> If I look in the raw log file I find 'FAIL:' lines for the
>> work-in-progress patch tests, but those aren't reported in the summary
>> (Maybe because it doesn't look for 'FAIL:' lines?).
>>
>> Could somebody please look at this issue?
>>
>>
>
> Hm. I made sure that any failures reported from wimpy tests did not
> cause the tests to exit with non-zero status. Individual per-test
> results should be XPASS and XFAIL, too.
>
> IMHO buildbots should look at test exit status, not grep the log file;
> run-tests.py already does that better.
>
So it turns out that Arfrever changed those patch tests from Wimp to
SkipUnless i36564 n rand they're failing on Windows again.
Tut, tut. Do not over shoulder your code throw, my young padawan. Check
buildbots. :)
I *am* very much surprised though that those failures do not appear in
the test summary.
-- Brane
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=1475199
Re: Test results on buildbot
Posted by Branko Cibej <br...@xbc.nu>.
Bert Huijben wrote:
> Recently the test framework was changed to support work-in-progress
> tests. When this support was completed the Windows buildbot was offline.
>
> Since a few days the buildbot is back online (Thanks D.J. Heap!), and
> for the last few days the Windows tests don't report an error in the
> summary:
>
> ===============
>
[...]
> =========================
>
> But the testresult is still failed (red) and broken builds are reported
> to the svn-breakage list.
>
> If I look in the raw log file I find 'FAIL:' lines for the
> work-in-progress patch tests, but those aren't reported in the summary
> (Maybe because it doesn't look for 'FAIL:' lines?).
>
> Could somebody please look at this issue?
>
Hm. I made sure that any failures reported from wimpy tests did not
cause the tests to exit with non-zero status. Individual per-test
results should be XPASS and XFAIL, too.
IMHO buildbots should look at test exit status, not grep the log file;
run-tests.py already does that better.
-- Brane
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=1475068