You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Nathan Hartman <ha...@gmail.com> on 2020/01/02 15:45:25 UTC
Re: Makefile.svn: a dumb question
On Sat, Dec 28, 2019 at 12:13 PM Daniel Shahaf <d....@daniel.shahaf.name> wrote:
> Nathan Hartman wrote on Fri, Dec 27, 2019 at 15:39:10 -0500:
> > On Fri, Dec 27, 2019 at 12:41 AM Daniel Shahaf <d....@daniel.shahaf.name> wrote:
> > > Forgot to say this earlier, but you might try running a threaded svnserve. In
> > > plain trunk that's «make svnserveautocheck THREADED=1»; I don't know how to do
> > > that in stsp's distro.
> >
> > I'll give it a try. I have to study how the makefile invokes the
> > regression tests.
> >
> > There is a variable THREADING in the makefile; it defaults to yes and
> > activates threading and thread-safety stuff in apr, gettext, ruby, and
> > sqlite. It has no effect on the Subversion build or tests.
>
> To be clear, I was referring to the variable THREADED in Subversion's Makefile,
> not to anything in stsp's distro. Perhaps this? —
>
> Index: tools/dev/unix-build/Makefile.svn
> ===================================================================
> --- tools/dev/unix-build/Makefile.svn (revision 1871981)
> +++ tools/dev/unix-build/Makefile.svn (working copy)
> @@ -2165,6 +2165,7 @@
> env LD_LIBRARY_PATH=$(LD_LIBRARY_PATH) $(LIB_PTHREAD_HACK) \
> env MAKEFLAGS= make check PARALLEL=$(PARALLEL) CLEANUP=$(CLEANUP) \
> EXCLUSIVE_WC_LOCKS=$(EXCLUSIVE_WC_LOCKS) \
> + THREADED=$(THREADED) \
> SVN_BIN_DIR=$(SVN_PREFIX)/bin \
> MEMCACHED_SERVER=$(MEMCACHED_SERVER) $1 FS_TYPE=$$fs; \
> for log in tests.log fails.log; do \
>
> (If it works for you, feel free to commit it, but note the comment at the top
> of the file.)
Daniel, as you suggested previously (w/patch above), I added
"THREADED=$(THREADED)" to pass-through THREADED from stsp's
Makefile.svn to Subversion's Makefile (which in turn passes it to
svnserveautocheck.sh, which then invokes svnserve with -T). I think
that's the correct behavior and it should be committed. Hopefully
it will help us get better test coverage with and without threading.
(I'll commit it a little later today.)
Regarding the one test failure I had (autoprop_tests.py 7 with
[svn x bdb]), I've had two more since, both with bdb. These are
intermittent database errors and occur in varying places. (All fsfs
tests have passed every time.)
Before investigating too much further, I see that Makefile.svn is
downloading an antique bdb 4.7.25. From the Subversion 1.13.0-rc1
signing emails [1], I see that Brane runs his tests with the newest,
18.1.32. So I'll try that and see if the failures go away (assuming I
succeed to actually build it -- that's a big major version jump).
[1] https://s.apache.org/q75t8
Just for completeness (because I know I'll lose the logs later), this
is a list of the bdb test failures I've had thus far:
* autoprop_tests.py 7 with [svn x bdb] (mentioned previously)
* prop_tests.py 3 with [local x bdb]
* merge_tree_conflict_tests.py 3 with [svn x bdb]
Details of the latter two are below:
**********************************************************************
[[[
Begin test: local x bdb
...
[098/123] prop_tests.py..................................................FAILURE
...
At least one test FAILED, checking
/home/nate/ramdrive/svndev/svn-trunk/tests.log
FAIL: prop_tests.py 3: receive properties via update
Summary of test results:
2496 tests PASSED
157 tests SKIPPED
94 tests XFAILED (17 WORK-IN-PROGRESS)
1 test FAILED
Python version: 3.7.5.
SUMMARY: Some tests failed
]]]
From fails.log.svn-check-local-bdb:
[[[
W: * Dumped revision 0.
subversion/svnadmin/svnadmin.c:1475,
subversion/libsvn_repos/dump.c:2175,
subversion/libsvn_delta/path_driver.c:190,
subversion/libsvn_delta/path_driver.c:328,
subversion/libsvn_repos/replay.c:597,
subversion/libsvn_repos/dump.c:1714,
subversion/libsvn_repos/dump.c:1538,
subversion/libsvn_subr/stream.c:194,
subversion/libsvn_fs_base/reps-strings.c:981,
subversion/libsvn_fs_base/reps-strings.c:873,
subversion/libsvn_fs_base/reps-strings.c:477,
subversion/libsvn_fs_base/bdb/strings-table.c:208,
subversion/libsvn_fs_base/bdb/strings-table.c:96,
subversion/libsvn_fs_base/bdb/bdb-err.c:85,
subversion/libsvn_fs_base/bdb/bdb-err.c:79: (apr_err=SVN_ERR_FS_BERKELEY_DB)
svnadmin: E160029: Berkeley DB error for filesystem
'svn-test-work/local_tmp/repos/db' while creating cursor for reading a
string:
DB_RUNRECOVERY: Fatal error, run database recovery
subversion/libsvn_fs_base/bdb/env.c:230: (apr_err=SVN_ERR_FS_BERKELEY_DB)
svnadmin: E160029: bdb: PANIC: fatal region error detected; run recovery
W: CMD: /home/nate/ramdrive/svndev/prefix/svn-trunk/bin/svnadmin dump
svn-test-work/local_tmp/repos terminated by signal 6
W: CWD: /home/nate/ramdrive/svndev/svn-trunk/subversion/tests/cmdline
W: EXCEPTION: SVNProcessTerminatedBySignal
Traceback (most recent call last):
File "/home/nate/ramdrive/svndev/svn-trunk/subversion/tests/cmdline/svntest/main.py",
line 1931, in run
rc = self.pred.run(sandbox)
File "/home/nate/ramdrive/svndev/svn-trunk/subversion/tests/cmdline/svntest/testcase.py",
line 178, in run
result = self.func(sandbox)
File "/home/nate/ramdrive/svndev/svn-trunk/subversion/tests/cmdline/prop_tests.py",
line 147, in update_props
sbox.build()
File "/home/nate/ramdrive/svndev/svn-trunk/subversion/tests/cmdline/svntest/sandbox.py",
line 164, in build
minor_version, tree)
File "/home/nate/ramdrive/svndev/svn-trunk/subversion/tests/cmdline/svntest/actions.py",
line 1994, in make_repo_and_wc
guarantee_greek_repository(sbox.repo_dir, minor_version)
File "/home/nate/ramdrive/svndev/svn-trunk/subversion/tests/cmdline/svntest/actions.py",
line 193, in guarantee_greek_repository
_guarantee_repos(path, main.pristine_greek_repos_dir, minor_version)
File "/home/nate/ramdrive/svndev/svn-trunk/subversion/tests/cmdline/svntest/actions.py",
line 178, in _guarantee_repos
failed = main.copy_repos(repos_dir, path, 1, 1, minor_version)
File "/home/nate/ramdrive/svndev/svn-trunk/subversion/tests/cmdline/svntest/main.py",
line 1246, in copy_repos
dump_stdout, dump_stderr, dump_exit_code = wait_on_pipe(dump_kid, True)
File "/home/nate/ramdrive/svndev/svn-trunk/subversion/tests/cmdline/svntest/main.py",
line 557, in wait_on_pipe
raise SVNProcessTerminatedBySignal
svntest.main.SVNProcessTerminatedBySignal
FAIL: prop_tests.py 3: receive properties via update
]]]
**********************************************************************
[[[
Begin test: svn x bdb
...
[091/123] merge_tree_conflict_tests.py...................................FAILURE
...
At least one test FAILED, checking
/home/nate/ramdrive/svndev/svn-trunk/tests.log
FAIL: merge_tree_conflict_tests.py 3: merge on deleted directory in target
Summary of test results:
2531 tests PASSED
119 tests SKIPPED
97 tests XFAILED (17 WORK-IN-PROGRESS)
1 test FAILED
Python version: 3.7.5.
SUMMARY: Some tests failed
]]]
From fails.log.svn-check-svn-bdb:
[[[
W: * Dumped revision 0.
subversion/svnadmin/svnadmin.c:1475,
subversion/libsvn_repos/dump.c:2175,
subversion/libsvn_delta/path_driver.c:190,
subversion/libsvn_delta/path_driver.c:328,
subversion/libsvn_repos/replay.c:534,
subversion/libsvn_fs/fs-loader.c:1223,
subversion/libsvn_fs_base/tree.c:1193,
subversion/libsvn_fs_base/tree.c:1165,
subversion/libsvn_fs_base/tree.c:1026,
subversion/libsvn_fs_base/trail.c:218,
subversion/libsvn_fs_base/bdb/bdb-err.c:85,
subversion/libsvn_fs_base/bdb/bdb-err.c:79: (apr_err=SVN_ERR_FS_BERKELEY_DB)
svnadmin: E160029: Berkeley DB error for filesystem
'svn-test-work/local_tmp/repos/db' while checkpointing after Berkeley
DB transaction:
DB_RUNRECOVERY: Fatal error, run database recovery
subversion/libsvn_fs_base/bdb/env.c:230: (apr_err=SVN_ERR_FS_BERKELEY_DB)
svnadmin: E160029: bdb: PANIC: fatal region error detected; run recovery
W: CMD: /home/nate/ramdrive/svndev/prefix/svn-trunk/bin/svnadmin dump
svn-test-work/local_tmp/repos terminated by signal 6
W: CWD: /home/nate/ramdrive/svndev/svn-trunk/subversion/tests/cmdline
W: EXCEPTION: SVNProcessTerminatedBySignal
Traceback (most recent call last):
File "/home/nate/ramdrive/svndev/svn-trunk/subversion/tests/cmdline/svntest/main.py",
line 1931, in run
rc = self.pred.run(sandbox)
File "/home/nate/ramdrive/svndev/svn-trunk/subversion/tests/cmdline/svntest/testcase.py",
line 258, in run
return self._delegate.run(sandbox)
File "/home/nate/ramdrive/svndev/svn-trunk/subversion/tests/cmdline/svntest/testcase.py",
line 178, in run
result = self.func(sandbox)
File "/home/nate/ramdrive/svndev/svn-trunk/subversion/tests/cmdline/merge_tree_conflict_tests.py",
line 266, in merge_tree_deleted_in_target
sbox.build()
File "/home/nate/ramdrive/svndev/svn-trunk/subversion/tests/cmdline/svntest/sandbox.py",
line 164, in build
minor_version, tree)
File "/home/nate/ramdrive/svndev/svn-trunk/subversion/tests/cmdline/svntest/actions.py",
line 1994, in make_repo_and_wc
guarantee_greek_repository(sbox.repo_dir, minor_version)
File "/home/nate/ramdrive/svndev/svn-trunk/subversion/tests/cmdline/svntest/actions.py",
line 193, in guarantee_greek_repository
_guarantee_repos(path, main.pristine_greek_repos_dir, minor_version)
File "/home/nate/ramdrive/svndev/svn-trunk/subversion/tests/cmdline/svntest/actions.py",
line 178, in _guarantee_repos
failed = main.copy_repos(repos_dir, path, 1, 1, minor_version)
File "/home/nate/ramdrive/svndev/svn-trunk/subversion/tests/cmdline/svntest/main.py",
line 1246, in copy_repos
dump_stdout, dump_stderr, dump_exit_code = wait_on_pipe(dump_kid, True)
File "/home/nate/ramdrive/svndev/svn-trunk/subversion/tests/cmdline/svntest/main.py",
line 557, in wait_on_pipe
raise SVNProcessTerminatedBySignal
svntest.main.SVNProcessTerminatedBySignal
FAIL: merge_tree_conflict_tests.py 3: merge on deleted directory in target
]]]
**********************************************************************
Happy New Year,
Nathan
Re: Makefile.svn: a dumb question
Posted by Nathan Hartman <ha...@gmail.com>.
On Thu, Jan 2, 2020 at 1:03 PM Daniel Shahaf <d....@daniel.shahaf.name> wrote:
>
> Nathan Hartman wrote on Thu, 02 Jan 2020 17:44 +00:00:
> > On Thu, Jan 2, 2020 at 11:35 AM Daniel Shahaf <d....@daniel.shahaf.name> wrote:
> > > Nathan Hartman wrote on Thu, Jan 02, 2020 at 10:45:25 -0500:
> > > > Regarding the one test failure I had (autoprop_tests.py 7 with
> > > > [svn x bdb]), I've had two more since, both with bdb. These are
> > > > intermittent database errors and occur in varying places. (All fsfs
> > > > tests have passed every time.)
> > >
> > > Actually, all three backtraces so far are in sbox.build(), so something like
> > > this might be useful:
> >
> > I'm a bit confused here. Yes, the Python backtraces show it coming from
> > sbox.build but aren't these being instigated by the BDB errors/panics
> > (which are reported as a E160029)?
>
> sbox.build() calls svnadmin, which exits with E160029 (and, apparently,
> SIGABRT), which becomes an uncaught Python exception, which triggers the
> stack trace, which lists sbox.build().
>
> The point is, the errors always happened during sbox.build(), never
> during any other part of the test functions [the ones listed in
> «test_list»], so you should be able to reproduce the errors much more
> quickly by running the test suite with that patch. For example, you
> could run the test suite with the older bdb, establish how many FAILs
> you get per 2000 test functions, and then upgrade bdb and try again.
>
> Makes sense?
Yes, this makes sense now. Thank you for explaining in detail.
I'll try it soon and will post my findings...
Thanks,
Nathan
Re: Makefile.svn: a dumb question
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Nathan Hartman wrote on Thu, 02 Jan 2020 17:44 +00:00:
> On Thu, Jan 2, 2020 at 11:35 AM Daniel Shahaf <d....@daniel.shahaf.name> wrote:
> > Nathan Hartman wrote on Thu, Jan 02, 2020 at 10:45:25 -0500:
> > > Regarding the one test failure I had (autoprop_tests.py 7 with
> > > [svn x bdb]), I've had two more since, both with bdb. These are
> > > intermittent database errors and occur in varying places. (All fsfs
> > > tests have passed every time.)
> >
> > Actually, all three backtraces so far are in sbox.build(), so something like
> > this might be useful:
>
> I'm a bit confused here. Yes, the Python backtraces show it coming from
> sbox.build but aren't these being instigated by the BDB errors/panics
> (which are reported as a E160029)?
sbox.build() calls svnadmin, which exits with E160029 (and, apparently,
SIGABRT), which becomes an uncaught Python exception, which triggers the
stack trace, which lists sbox.build().
The point is, the errors always happened during sbox.build(), never
during any other part of the test functions [the ones listed in
«test_list»], so you should be able to reproduce the errors much more
quickly by running the test suite with that patch. For example, you
could run the test suite with the older bdb, establish how many FAILs
you get per 2000 test functions, and then upgrade bdb and try again.
Makes sense?
Re: Makefile.svn: a dumb question
Posted by Nathan Hartman <ha...@gmail.com>.
On Thu, Jan 2, 2020 at 11:35 AM Daniel Shahaf <d....@daniel.shahaf.name>
wrote:
> Nathan Hartman wrote on Thu, Jan 02, 2020 at 10:45:25 -0500:
> > Regarding the one test failure I had (autoprop_tests.py 7 with
> > [svn x bdb]), I've had two more since, both with bdb. These are
> > intermittent database errors and occur in varying places. (All fsfs
> > tests have passed every time.)
>
> Actually, all three backtraces so far are in sbox.build(), so something
> like
> this might be useful:
>
> [[[
> Index: subversion/tests/cmdline/basic_tests.py
> ===================================================================
> --- subversion/tests/cmdline/basic_tests.py (revision 1871981)
> +++ subversion/tests/cmdline/basic_tests.py (working copy)
> @@ -3319,6 +3319,7 @@
> null_prop_update_last_changed_revision,
> filtered_ls_top_level_path,
> ]
> +test_list = [ None ] + [ lambda sbox: sbox.build() ] * 2000
>
> if __name__ == '__main__':
> svntest.main.run_tests(test_list)
> ]]]
>
> … then pass «TESTS="subversion/tests/cmdline/basic_tests.py"» to make.
>
> (I didn't test that patch, so it might complain about a missing __doc__
> attribute, but you get the idea.)
>
> Cheers,
>
> Daniel
> (And you can use «operator.methodcaller('build')» to squeeze a few more
> cycles there)
I'm a bit confused here. Yes, the Python backtraces show it coming from
sbox.build but aren't these being instigated by the BDB errors/panics
(which are reported as a E160029)?
Thanks,
Nathan
Re: Makefile.svn: a dumb question
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Nathan Hartman wrote on Thu, Jan 02, 2020 at 10:45:25 -0500:
> Regarding the one test failure I had (autoprop_tests.py 7 with
> [svn x bdb]), I've had two more since, both with bdb. These are
> intermittent database errors and occur in varying places. (All fsfs
> tests have passed every time.)
Actually, all three backtraces so far are in sbox.build(), so something like
this might be useful:
[[[
Index: subversion/tests/cmdline/basic_tests.py
===================================================================
--- subversion/tests/cmdline/basic_tests.py (revision 1871981)
+++ subversion/tests/cmdline/basic_tests.py (working copy)
@@ -3319,6 +3319,7 @@
null_prop_update_last_changed_revision,
filtered_ls_top_level_path,
]
+test_list = [ None ] + [ lambda sbox: sbox.build() ] * 2000
if __name__ == '__main__':
svntest.main.run_tests(test_list)
]]]
… then pass «TESTS="subversion/tests/cmdline/basic_tests.py"» to make.
(I didn't test that patch, so it might complain about a missing __doc__
attribute, but you get the idea.)
Cheers,
Daniel
(And you can use «operator.methodcaller('build')» to squeeze a few more cycles there)
Re: Makefile.svn: a dumb question
Posted by Stefan Sperling <st...@elego.de>.
On Thu, Jan 02, 2020 at 10:45:25AM -0500, Nathan Hartman wrote:
> Before investigating too much further, I see that Makefile.svn is
> downloading an antique bdb 4.7.25. From the Subversion 1.13.0-rc1
> signing emails [1], I see that Brane runs his tests with the newest,
> 18.1.32. So I'll try that and see if the failures go away (assuming I
> succeed to actually build it -- that's a big major version jump).
If I'm not mistaken, newer BDB versions don't allow a non-threaded
build of BDB. Which is why this wasn't updated.