You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Senthil Kumaran S <se...@collab.net> on 2008/03/20 10:36:28 UTC
Random test failures with PARALLEL
Hi,
When we run the subversion tests with the PARALLEL=1 option we get test
failures in a random fashion. 90% (at least for me) of the time when I run the
tests (in Debian etch GNU/Linux, over ra_dav), it fails at different points.
Since the failures are random, it becomes difficult to track down the actual
cause of the failures.
Following is a stack trace of the failures during one of the runs,
basic_tests.py:
<snip>
CMD: svn proplist
svn-test-work/working_copies/basic_tests-17/dirA/dirB/new_file --verbose
--config-dir
/home/stylesen/subversion-dev/trunk/virtual-neon/subversion/tests/cmdline/svn-test-work/local_tmp/config
--password rayjandom --no-auth-cache --username jrandom <TIME = 0.147680>
CMD: svn status -v -u -q svn-test-work/working_copies/basic_tests-17
--config-dir
/home/stylesen/subversion-dev/trunk/virtual-neon/subversion/tests/cmdline/svn-test-work/local_tmp/config
--password rayjandom --no-auth-cache --username jrandom <TIME = 0.117506>
../subversion/libsvn_ra_neon/util.c:560: (apr_err=175002)
svn: Server sent unexpected return value (403 Forbidden) in response to OPTIONS
request for 'http://localhost:28671/svn-test-work/repositories/basic_tests-17'
Traceback (most recent call last):
File
"/home/stylesen/subversion-dev/trunk/subversion/tests/cmdline/svntest/main.py",
line 1066, in run
rc = apply(self.pred.run, (), kw)
File
"/home/stylesen/subversion-dev/trunk/subversion/tests/cmdline/svntest/testcase.py",
line 121, in run
return self.func(sandbox)
File
"/home/stylesen/subversion-dev/trunk/virtual-neon/../subversion/tests/cmdline/basic_tests.py",
line 1295, in basic_import
None, None, 1)
File
"/home/stylesen/subversion-dev/trunk/subversion/tests/cmdline/svntest/actions.py",
line 704, in run_and_verify_update
check_props)
File
"/home/stylesen/subversion-dev/trunk/subversion/tests/cmdline/svntest/actions.py",
line 616, in verify_update
run_and_verify_status(wc_dir_name, status_tree)
File
"/home/stylesen/subversion-dev/trunk/subversion/tests/cmdline/svntest/actions.py",
line 1056, in run_and_verify_status
wc_dir_name)
File
"/home/stylesen/subversion-dev/trunk/subversion/tests/cmdline/svntest/main.py",
line 495, in run_svn
return run_command(svn_binary, error_expected, 0,
File
"/home/stylesen/subversion-dev/trunk/subversion/tests/cmdline/svntest/main.py",
line 314, in run_command
None, *varargs)
File
"/home/stylesen/subversion-dev/trunk/subversion/tests/cmdline/svntest/main.py",
line 436, in run_command_stdin
raise Failure
Failure
FAIL: basic_tests.py 17: basic import of single new file
</snip>
diff_test.py:
<snip>
CMD: svnadmin create svn-test-work/repositories/diff_tests-27 --bdb-txn-nosync
<TIME = 1.162752>
CMD: svnadmin dump "svn-test-work/local_tmp/repos" | svnadmin load
"svn-test-work/repositories/diff_tests-27" --ignore-uuid \<TIME = 0.017950>
CMD: svn co http://localhost:28671/svn-test-work/repositories/diff_tests-27
svn-test-work/working_copies/diff_tests-27 --con\fig-dir
/home/stylesen/subversion-dev/trunk/virtual-neon/subversion/tests/cmdline/svn-test-work/local_tmp/config
--password \rayjandom --no-auth-cache --username jrandom <TIME = 0.576778>
../subversion/libsvn_ra_neon/util.c:560: (apr_err=175002)
svn: Server sent unexpected return value (403 Forbidden) in response to OPTIONS
request for 'http://localhost:28671/svn-test\-work/repositories/diff_tests-27'
Traceback (most recent call last):
File
"/home/stylesen/subversion-dev/trunk/subversion/tests/cmdline/svntest/main.py",
line 1066, in run
rc = apply(self.pred.run, (), kw)
File
"/home/stylesen/subversion-dev/trunk/subversion/tests/cmdline/svntest/testcase.py",
line 121, in run
return self.func(sandbox)
File
"/home/stylesen/subversion-dev/trunk/virtual-neon/../subversion/tests/cmdline/diff_tests.py",
line 1905, in diff_sche\dule_delete
sbox.build()
File
"/home/stylesen/subversion-dev/trunk/subversion/tests/cmdline/svntest/main.py",
line 921, in build
if actions.make_repo_and_wc(self, create_wc, read_only):
File
"/home/stylesen/subversion-dev/trunk/subversion/tests/cmdline/svntest/actions.py",
line 1282, in make_repo_and_wc
expected_wc)
File
"/home/stylesen/subversion-dev/trunk/subversion/tests/cmdline/svntest/actions.py",
line 377, in run_and_verify_checko\ut
URL, wc_dir_name, *args)
File
"/home/stylesen/subversion-dev/trunk/subversion/tests/cmdline/svntest/main.py",
line 495, in run_svn
return run_command(svn_binary, error_expected, 0,
File
"/home/stylesen/subversion-dev/trunk/subversion/tests/cmdline/svntest/main.py",
line 314, in run_command
None, *varargs)
File
"/home/stylesen/subversion-dev/trunk/subversion/tests/cmdline/svntest/main.py",
line 436, in run_command_stdin
raise Failure
Failure
FAIL: diff_tests.py 27: scheduled deleted
</snip>
stat_tests.py
<snip>
CMD: svnadmin create svn-test-work/repositories/stat_tests-11 --bdb-txn-nosync
<TIME = 0.485893>
CMD: svnadmin dump "svn-test-work/local_tmp/repos" | svnadmin load
"svn-test-work/repositories/stat_tests-11" --ignore-uuid \<TIME = 0.005457>
CMD: svn co http://localhost:28671/svn-test-work/repositories/stat_tests-11
svn-test-work/working_copies/stat_tests-11 --con\fig-dir
/home/stylesen/subversion-dev/trunk/virtual-neon/subversion/tests/cmdline/svn-test-work/local_tmp/config
--password \rayjandom --no-auth-cache --username jrandom <TIME = 0.618778>
CMD: svn add newfile --config-dir
/home/stylesen/subversion-dev/trunk/virtual-neon/subversion/tests/cmdline/svn-test-work/lo\cal_tmp/config
--password rayjandom --no-auth-cache --username jrandom <TIME = 0.396752>
CMD: svn ci -m "log msg" --config-dir
/home/stylesen/subversion-dev/trunk/virtual-neon/subversion/tests/cmdline/svn-test-wor\k/local_tmp/config
--password rayjandom --no-auth-cache --username jrandom <TIME = 0.361931>
../subversion/libsvn_client/commit.c:916: (apr_err=175002)
svn: Commit failed (details follow):
../subversion/libsvn_ra_neon/util.c:560: (apr_err=175002)
svn: Server sent unexpected return value (403 Forbidden) in response to OPTIONS
request for 'http://localhost:28671/svn-test\-work/repositories/stat_tests-11'
Traceback (most recent call last):
File
"/home/stylesen/subversion-dev/trunk/subversion/tests/cmdline/svntest/main.py",
line 1066, in run
rc = apply(self.pred.run, (), kw)
File
"/home/stylesen/subversion-dev/trunk/subversion/tests/cmdline/svntest/testcase.py",
line 121, in run
return self.func(sandbox)
File
"/home/stylesen/subversion-dev/trunk/virtual-neon/../subversion/tests/cmdline/stat_tests.py",
line 495, in status_uni\nvited_parent_directory
'ci', '-m', 'log msg')
File
"/home/stylesen/subversion-dev/trunk/subversion/tests/cmdline/svntest/main.py",
line 495, in run_svn
return run_command(svn_binary, error_expected, 0,
File
"/home/stylesen/subversion-dev/trunk/subversion/tests/cmdline/svntest/main.py",
line 314, in run_command
None, *varargs)
File
"/home/stylesen/subversion-dev/trunk/subversion/tests/cmdline/svntest/main.py",
line 436, in run_command_stdin
raise Failure
Failure
FAIL: stat_tests.py 11: status -u on outdated, added file shows only that
</snip>
--
Senthil Kumaran S
http://www.stylesen.org/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Random test failures with PARALLEL
Posted by Senthil Kumaran S <se...@collab.net>.
David Glasser wrote:
> I'm getting this issue as well.
>
> I'm not sure if I follow your diagnosis. The different parallel tests
> are in separate processes (the only threading is in a dispatching
> process); I don't see why this would make the STDIN_LINES larger than
> one process. Plus we're clearly getting a 403 from the server.
That was just my random guess. If I come across something interesting I ll keep
the list posted.
--
Senthil Kumaran S
http://www.stylesen.org/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Random test failures with PARALLEL
Posted by David Glasser <gl...@davidglasser.net>.
On Thu, Mar 20, 2008 at 5:22 AM, Senthil Kumaran S <se...@collab.net> wrote:
> Lieven Govaerts wrote:
> > Relevant extract from apache error log:
> > [Thu Mar 20 09:56:19 2008] [error] [client 127.0.0.1] The URI does not
> > contain the name of a repository. [403, #190001]
> >
> > access.log:
> > 127.0.0.1 - jrandom [20/Mar/2008:09:56:19 +0100] "OPTIONS
> > /svn-test-work/repositories HTTP/1.1" 403 322 "-" "SVN/1.6.0 (dev build)
> > neon/0.25.5"
> >
> > So, why is there no repository name?
>
> I could see in all the test failures stacktrace we have this line:
>
> <snip>
>
> File
> "/home/stylesen/subversion-dev/trunk/subversion/tests/cmdline/svntest/main.py",
> line 436, in run_command_stdin
> raise Failure
> </snip>
>
> which in turn gives a 403 response in apache. The doc string of
> run_command_stdin says:
>
> " this should not be very large, as if the program outputs more than the OS
> is willing to buffer, this will deadlock, with both Python and
> COMMAND waiting to write to each other for ever"
>
> So my assumption is when the tests are run in parallel the STDIN_LINES created
> by different processes are large and raises a failure randomly for the above
> reason?
I'm getting this issue as well.
I'm not sure if I follow your diagnosis. The different parallel tests
are in separate processes (the only threading is in a dispatching
process); I don't see why this would make the STDIN_LINES larger than
one process. Plus we're clearly getting a 403 from the server.
In the past, this has been because we've mixed in authz-using tests
(which can't be run in parallel) with other tests. I don't see that
here though.
--dave
--
David Glasser | glasser@davidglasser.net | http://www.davidglasser.net/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Random test failures with PARALLEL
Posted by Senthil Kumaran S <se...@collab.net>.
Lieven Govaerts wrote:
> Relevant extract from apache error log:
> [Thu Mar 20 09:56:19 2008] [error] [client 127.0.0.1] The URI does not
> contain the name of a repository. [403, #190001]
>
> access.log:
> 127.0.0.1 - jrandom [20/Mar/2008:09:56:19 +0100] "OPTIONS
> /svn-test-work/repositories HTTP/1.1" 403 322 "-" "SVN/1.6.0 (dev build)
> neon/0.25.5"
>
> So, why is there no repository name?
I could see in all the test failures stacktrace we have this line:
<snip>
File
"/home/stylesen/subversion-dev/trunk/subversion/tests/cmdline/svntest/main.py",
line 436, in run_command_stdin
raise Failure
</snip>
which in turn gives a 403 response in apache. The doc string of
run_command_stdin says:
" this should not be very large, as if the program outputs more than the OS
is willing to buffer, this will deadlock, with both Python and
COMMAND waiting to write to each other for ever"
So my assumption is when the tests are run in parallel the STDIN_LINES created
by different processes are large and raises a failure randomly for the above
reason?
--
Senthil Kumaran S
http://www.stylesen.org/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Random test failures with PARALLEL
Posted by Lieven Govaerts <sv...@mobsol.be>.
Senthil Kumaran S wrote:
> Hi,
>
> When we run the subversion tests with the PARALLEL=1 option we get test
> failures in a random fashion. 90% (at least for me) of the time when I
> run the tests (in Debian etch GNU/Linux, over ra_dav), it fails at
> different points. Since the failures are random, it becomes difficult to
> track down the actual cause of the failures.
>
> Following is a stack trace of the failures during one of the runs,
>
> basic_tests.py:
> <snip>
> CMD: svn proplist
> svn-test-work/working_copies/basic_tests-17/dirA/dirB/new_file --verbose
> --config-dir
> /home/stylesen/subversion-dev/trunk/virtual-neon/subversion/tests/cmdline/svn-test-work/local_tmp/config
> --password rayjandom --no-auth-cache --username jrandom <TIME = 0.147680>
> CMD: svn status -v -u -q svn-test-work/working_copies/basic_tests-17
> --config-dir
> /home/stylesen/subversion-dev/trunk/virtual-neon/subversion/tests/cmdline/svn-test-work/local_tmp/config
> --password rayjandom --no-auth-cache --username jrandom <TIME = 0.117506>
> ../subversion/libsvn_ra_neon/util.c:560: (apr_err=175002)
> svn: Server sent unexpected return value (403 Forbidden) in response to
> OPTIONS request for
> 'http://localhost:28671/svn-test-work/repositories/basic_tests-17'
I don't have time to look into this in the coming hours, just want to
add that I get the same "403 forbidden" error in basic test 1 sometimes
(let's say 1 run out of 10).
Relevant extract from apache error log:
[Thu Mar 20 09:56:19 2008] [error] [client 127.0.0.1] The URI does not
contain the name of a repository. [403, #190001]
access.log:
127.0.0.1 - jrandom [20/Mar/2008:09:56:19 +0100] "OPTIONS
/svn-test-work/repositories HTTP/1.1" 403 322 "-" "SVN/1.6.0 (dev build)
neon/0.25.5"
So, why is there no repository name?
Lieven
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org