You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Daniel Sahlberg <da...@gmail.com> on 2025/01/06 21:32:30 UTC
Re: svnserve frequently die at assertion in utf.c
Den ons 18 dec. 2024 kl 16:43 skrev Guido Jäkel <G....@dnb.de>:
>
> > Did I understand you correctly that you were seeing the issue before the
> upgrade as well, but more frequent now?
> Yes, we saw it also before during the last 12m using the "previous"
> version (lxc container image) of our subversion service. But this was to
> seldom to really care about. Now, with the "current" version -- refreshed
> this days -- it happens within hours. To build software here, there's also
> an about one year more current build chain or system libs like the glibc.
> This could cause a mutex issue to become more problematic because of
> different timings.
>
> We never observe the issue in our test or approval stage, but there's no
> "real world load" on the subversion service.
>
> The container is also executed together with a whole bunch on one of our
> blade server with 112 cores, but the Subversion container is limited to two
> cores.
>
> > Reading up on it, I see that this is exactly the same failure you got.
> I'm leaning towards an APR issue (possibly related to the exact
> kernel/libpthread you are using), I never got around to float it on the APR
> list since it is still quite a bit of "Subversion code" involved. My next
> step would be to try to write a minimum reproduction application with a
> bunch of threads all randomy trying to get and release the same mutex.
>
> If you provide me the source code I'm probably able to compile an run it
> here, too.
>
I have attached my test program which tries to exercise the mutex
libraries. The program contains a simple linked list with 100 threads
adding 100 items to the end of the list. In the end, the list should
contain 100 items for each thread with a counter incrementing by 1 (per
thread), but with threads intermixed randomly. There is of course a race
condition updating the pointer to the last node of the linked list where
two threads mustn't update it simultaneously, so it is protected by a mutex.
The Makefile must be adjusted to point to the correct locations of the APR
library.
After compiling, the program can be executed with no argument (which
defaults to using a mutex) or with a single "n" to run without mutex.
Sample run:
[[[
$ ./main
Creating mutexes and threads... Done
Waiting for threads to finish... Finished
Checking for errors... No errors!
$ ./main n
Creating mutexes and threads... Done
Waiting for threads to finish... Finished
Checking for errors...
Thread 32 missing 6 found 7
Thread 48 missing 2 found 3
Thread 30 missing 14 found 15
]]]
To me, this proves that my basic program is correct and that with a mutex,
the shared memory is protected and without the mutex we get errors. I have
tested this under both Ubuntu and Guix with the same results.
The Guix environment is the same environment where I have been able to
reproduce the assert() in Subversion which started the discussion. So it
seems that the mutex code is not the culprit. But I still don't understand
why Subversion, running on Ubuntu with threading enabled can pass the test
suite while the same code running under Guix is failing.
I would expect that my test program also succeeds under Gentoo, but it
would still be interesting to see.
...
I realise we accidentally had some discussion offlist, so for the benefit
of the list I will repeat one question I've already asked:
- Is it possible to run the Subversion test suite (make check -jN
PARALLEL=N, where N is the number of parallell threads during testing) and
does it randomly fail? If you check the tests.log file after a failure, do
you see the same assertion as in svnserve?
Cheers,
Daniel
Re: svnserve frequently die at assertion in utf.c
Posted by Daniel Sahlberg <da...@gmail.com>.
Big thanks to both Sean and Guido for their help in tracking down the issue.
It seems indeed to be a bug in the Subversion code. I've added
dev@subversion.apache.org for further debugging - this goes out of scope of
the users list.
I believe several threads are accessing the same "translator"
simultaneously and the error occurs when they try to put it back to the
cache.
I've been able to reproduce the issue on Debian 10 (running under
VirtualBox), so I can check this using the ThreadSanitizer.
Kind regards,
Daniel
Den tis 7 jan. 2025 kl 10:47 skrev Jäkel, Guido <G....@dnb.de>:
> Building and running the test suite non-parallel shows up a single failure
>
> # make clean
> # EXTRA_CFLAGS="-fsanitize=thread" EXTRA_LDFLAGS="$EXTRA_CFLAGS" make check
> [...]
> [028/122]
> locks-test...............................................................................................................................FAILURE
> [...]
>
> tests.log
> [...]
> PASS: locks-test 13: lock callback error
> svn_tests: E200006: Expected error but got SVN_NO_ERROR
> FAIL: locks-test 14: lock/unlock when 'write-lock' couldn't be obtained
> [...]
>
>
>
> But running it with just two threads rises the issue nearly at once:
>
> # make -j 2 PARALLEL=2 check
> [...]
> [001/122]
> auth-test................................................................................................................................success
> [002/122]
> authz-test...............................................................................................................................success
> [003/122]
> bit-array-test...........................................................................................................................success
> [004/122]
> cache-test...............................................................................................................................success
> [005/122]
> checksum-test............................................................................................................................success
> [006/122]
> client-test..............................................................................................................................FAILURE
> [007/122]
> compat-test..............................................................................................................................success
> [008/122]
> compress-test............................................................................................................................success
> [009/122]
> config-test..............................................................................................................................success
> [010/122]
> conflict-data-test.......................................................................................................................success
> [011/122]
> conflicts-test...........................................................................................................................success
> [012/122]
> crypto-test..............................................................................................................................success
> [013/122]
> db-test..................................................................................................................................success
> [014/122]
> diff-diff3-test..........................................................................................................................FAILURE
> [015/122]
> dirent_uri-test..........................................................................................................................success
> [016/122]
> dump-load-test...........................................................................................................................FAILURE
> [017/122]
> entries-compat-test......................................................................................................................FAILURE
> [018/122]
> error-code-test..........................................................................................................................success
> [019/122]
> error-test...............................................................................................................................success
> [020/122]
> filesize-test............................................................................................................................success
> [021/122]
> fs-fs-pack-test..........................................................................................................................FAILURE
> [022/122]
> fs-fs-private-test.......................................................................................................................FAILURE
> [023/122]
> fs-sequential-test.......................................................................................................................success
> [024/122]
> fs-test..................................................................................................................................FAILURE
> [025/122]
> fs-x-pack-test...........................................................................................................................success
> [026/122]
> hashdump-test............................................................................................................................success
> [027/122]
> io-test....................................................................................................................FAILURE
> [028/122]
> locks-test...............................................................................................................................FAILURE
> [029/122]
> mergeinfo-test...........................................................................................................................FAILURE
> [030/122]
> mtcc-test................................................................................................................................FAILURE
> [031/122]
> op-depth-test............................................................................................................................FAILURE
> [032/122]
> opt-test.................................................................................................................................success
> [033/122]
> packed-data-test.........................................................................................................................success
> [034/122]
> parse-diff-test..........................................................................................................................success
> [...]
>
>
> And the thread sanitizer mention about function names I observered as
> failed asserts in the strace sessions:
>
> tests.log
> [...]
> START: checksum-test
> PASS: checksum-test 1: checksum parse
> PASS: checksum-test 2: checksum emptiness
> PASS: checksum-test 3: zero checksum matching
> PASS: checksum-test 4: zlib expansion test (zlib regression)
> PASS: checksum-test 5: zero checksum cross-type matching
> PASS: checksum-test 6: checksum (de-)serialization
> PASS: checksum-test 7: checksum parse all zero
> PASS: checksum-test 8: read from checksummed stream
> PASS: checksum-test 9: reset checksummed stream
> END: checksum-test
> ELAPSED: checksum-test 0:00:00.651333
>
> START: client-test
> ==================
> WARNING: ThreadSanitizer: data race (pid=10993)
> Write of size 8 at 0x7b9000006338 by thread T2 (mutexes: write M0):
> #0 put_xlate_handle_node <null> (libsvn_subr-1.so+0x8105e)
> #1 svn_utf_cstring_from_utf8 <null> (libsvn_subr-1.so+0x8163d)
> #2 svn_path_cstring_from_utf8 <null> (libsvn_subr-1.so+0x60a93)
> #3 svn_opt__arg_canonicalize_path <null> (libsvn_subr-1.so+0x5ab57)
> #4 svn_client_args_to_target_array2 <null> (libsvn_client-1.so+0x20861)
> #5 test_args_to_target_array <null> (client-test+0x7d31)
> #6 test_thread <null> (libsvn_test-1.so.0+0x873f)
> #7 <null> <null> (libtsan.so.2+0x34247)
>
> Previous read of size 8 at 0x7b9000006338 by thread T3:
> #0 put_xlate_handle_node <null> (libsvn_subr-1.so+0x80f93)
> #1 svn_utf_cstring_from_utf8 <null> (libsvn_subr-1.so+0x8163d)
> #2 svn_path_cstring_from_utf8 <null> (libsvn_subr-1.so+0x60a93)
> #3 svn_io_dir_open <null> (libsvn_subr-1.so+0x4a089)
> #4 svn_io_get_dirents3 <null> (libsvn_subr-1.so+0x4a1bd)
> #5 svn_io_remove_dir2 <null> (libsvn_subr-1.so+0x4fc8a)
> #6 svn_test__create_repos2 <null> (libsvn_test-1.so.0+0x5bfc)
> #7 svn_test__create_repos <null> (libsvn_test-1.so.0+0x5edd)
> #8 create_greek_repos <null> (client-test+0x3f74)
> #9 test_foreign_repos_copy <null> (client-test+0x6e84)
> #10 test_thread <null> (libsvn_test-1.so.0+0x87cf)
> #11 <null> <null> (libtsan.so.2+0x34247)
>
> Location is heap block of size 8192 at 0x7b9000006000 allocated by main
> thread:
> #0 malloc <null> (libtsan.so.2+0x39137)
> #1 <null> <null> (libapr-1.so.0+0x24171)
> #2 svn_utf_initialize2 <null> (libsvn_subr-1.so+0x81704)
> #3 svn_test_main <null> (libsvn_test-1.so.0+0x91b0)
> #4 main <null> (client-test+0x3e0e)
>
> Mutex M0 (0x7b90000060b0) created at:
> #0 pthread_mutex_init <null> (libtsan.so.2+0x3a616)
> #1 apr_thread_mutex_create <null> (libapr-1.so.0+0x23d1d)
> #2 svn_utf_initialize2 <null> (libsvn_subr-1.so+0x81717)
> #3 svn_test_main <null> (libsvn_test-1.so.0+0x91b0)
> #4 main <null> (client-test+0x3e0e)
>
> Thread T2 (tid=11009, running) created by main thread at:
> #0 pthread_create <null> (libtsan.so.2+0x39fe9)
> #1 svn_test_main <null> (libsvn_test-1.so.0+0x9efd)
> #2 main <null> (client-test+0x3e0e)
>
> Thread T3 (tid=11010, running) created by main thread at:
> #0 pthread_create <null> (libtsan.so.2+0x39fe9)
> #1 svn_test_main <null> (libsvn_test-1.so.0+0x9efd)
> #2 main <null> (client-test+0x3e0e)
>
> SUMMARY: ThreadSanitizer: data race
> (/var/tmp/portage/dev-vcs/subversion-1.14.5/work/subversion-1.14.5/subversion/libsvn_subr/.libs/libsvn_subr-1.so+0x8105e)
> in put_xlate_handle_node
> ==================
> [...]
>
> I interrupt the test suite after some time, the full tests.log so far is
> appended.
>
>
> > Now, I'll try to apply this to the subversion Makefile and run the test
> suite...
>
Re: AW: svnserve frequently die at assertion in utf.c
Posted by Branko Čibej <br...@apache.org>.
On 7. 1. 25 10:47, Jäkel, Guido wrote:
> Building and running the test suite non-parallel shows up a single failure
>
> # make clean
> # EXTRA_CFLAGS="-fsanitize=thread" EXTRA_LDFLAGS="$EXTRA_CFLAGS" make check
> [...]
> [028/122] locks-test...............................................................................................................................FAILURE
> [...]
>
> tests.log
> [...]
> PASS: locks-test 13: lock callback error
> svn_tests: E200006: Expected error but got SVN_NO_ERROR
> FAIL: locks-test 14: lock/unlock when 'write-lock' couldn't be obtained
> [...]
I'm late to the party but I just want to point out that these are *two*
errors, not one: an unexpected pass and a failure. The PASS should
really be an XPASS (but I don't remember if we support that in the C tests).
-- Brane
Re: svnserve frequently die at assertion in utf.c
Posted by Daniel Sahlberg <da...@gmail.com>.
Big thanks to both Sean and Guido for their help in tracking down the issue.
It seems indeed to be a bug in the Subversion code. I've added
dev@subversion.apache.org for further debugging - this goes out of scope of
the users list.
I believe several threads are accessing the same "translator"
simultaneously and the error occurs when they try to put it back to the
cache.
I've been able to reproduce the issue on Debian 10 (running under
VirtualBox), so I can check this using the ThreadSanitizer.
Kind regards,
Daniel
Den tis 7 jan. 2025 kl 10:47 skrev Jäkel, Guido <G....@dnb.de>:
> Building and running the test suite non-parallel shows up a single failure
>
> # make clean
> # EXTRA_CFLAGS="-fsanitize=thread" EXTRA_LDFLAGS="$EXTRA_CFLAGS" make check
> [...]
> [028/122]
> locks-test...............................................................................................................................FAILURE
> [...]
>
> tests.log
> [...]
> PASS: locks-test 13: lock callback error
> svn_tests: E200006: Expected error but got SVN_NO_ERROR
> FAIL: locks-test 14: lock/unlock when 'write-lock' couldn't be obtained
> [...]
>
>
>
> But running it with just two threads rises the issue nearly at once:
>
> # make -j 2 PARALLEL=2 check
> [...]
> [001/122]
> auth-test................................................................................................................................success
> [002/122]
> authz-test...............................................................................................................................success
> [003/122]
> bit-array-test...........................................................................................................................success
> [004/122]
> cache-test...............................................................................................................................success
> [005/122]
> checksum-test............................................................................................................................success
> [006/122]
> client-test..............................................................................................................................FAILURE
> [007/122]
> compat-test..............................................................................................................................success
> [008/122]
> compress-test............................................................................................................................success
> [009/122]
> config-test..............................................................................................................................success
> [010/122]
> conflict-data-test.......................................................................................................................success
> [011/122]
> conflicts-test...........................................................................................................................success
> [012/122]
> crypto-test..............................................................................................................................success
> [013/122]
> db-test..................................................................................................................................success
> [014/122]
> diff-diff3-test..........................................................................................................................FAILURE
> [015/122]
> dirent_uri-test..........................................................................................................................success
> [016/122]
> dump-load-test...........................................................................................................................FAILURE
> [017/122]
> entries-compat-test......................................................................................................................FAILURE
> [018/122]
> error-code-test..........................................................................................................................success
> [019/122]
> error-test...............................................................................................................................success
> [020/122]
> filesize-test............................................................................................................................success
> [021/122]
> fs-fs-pack-test..........................................................................................................................FAILURE
> [022/122]
> fs-fs-private-test.......................................................................................................................FAILURE
> [023/122]
> fs-sequential-test.......................................................................................................................success
> [024/122]
> fs-test..................................................................................................................................FAILURE
> [025/122]
> fs-x-pack-test...........................................................................................................................success
> [026/122]
> hashdump-test............................................................................................................................success
> [027/122]
> io-test....................................................................................................................FAILURE
> [028/122]
> locks-test...............................................................................................................................FAILURE
> [029/122]
> mergeinfo-test...........................................................................................................................FAILURE
> [030/122]
> mtcc-test................................................................................................................................FAILURE
> [031/122]
> op-depth-test............................................................................................................................FAILURE
> [032/122]
> opt-test.................................................................................................................................success
> [033/122]
> packed-data-test.........................................................................................................................success
> [034/122]
> parse-diff-test..........................................................................................................................success
> [...]
>
>
> And the thread sanitizer mention about function names I observered as
> failed asserts in the strace sessions:
>
> tests.log
> [...]
> START: checksum-test
> PASS: checksum-test 1: checksum parse
> PASS: checksum-test 2: checksum emptiness
> PASS: checksum-test 3: zero checksum matching
> PASS: checksum-test 4: zlib expansion test (zlib regression)
> PASS: checksum-test 5: zero checksum cross-type matching
> PASS: checksum-test 6: checksum (de-)serialization
> PASS: checksum-test 7: checksum parse all zero
> PASS: checksum-test 8: read from checksummed stream
> PASS: checksum-test 9: reset checksummed stream
> END: checksum-test
> ELAPSED: checksum-test 0:00:00.651333
>
> START: client-test
> ==================
> WARNING: ThreadSanitizer: data race (pid=10993)
> Write of size 8 at 0x7b9000006338 by thread T2 (mutexes: write M0):
> #0 put_xlate_handle_node <null> (libsvn_subr-1.so+0x8105e)
> #1 svn_utf_cstring_from_utf8 <null> (libsvn_subr-1.so+0x8163d)
> #2 svn_path_cstring_from_utf8 <null> (libsvn_subr-1.so+0x60a93)
> #3 svn_opt__arg_canonicalize_path <null> (libsvn_subr-1.so+0x5ab57)
> #4 svn_client_args_to_target_array2 <null> (libsvn_client-1.so+0x20861)
> #5 test_args_to_target_array <null> (client-test+0x7d31)
> #6 test_thread <null> (libsvn_test-1.so.0+0x873f)
> #7 <null> <null> (libtsan.so.2+0x34247)
>
> Previous read of size 8 at 0x7b9000006338 by thread T3:
> #0 put_xlate_handle_node <null> (libsvn_subr-1.so+0x80f93)
> #1 svn_utf_cstring_from_utf8 <null> (libsvn_subr-1.so+0x8163d)
> #2 svn_path_cstring_from_utf8 <null> (libsvn_subr-1.so+0x60a93)
> #3 svn_io_dir_open <null> (libsvn_subr-1.so+0x4a089)
> #4 svn_io_get_dirents3 <null> (libsvn_subr-1.so+0x4a1bd)
> #5 svn_io_remove_dir2 <null> (libsvn_subr-1.so+0x4fc8a)
> #6 svn_test__create_repos2 <null> (libsvn_test-1.so.0+0x5bfc)
> #7 svn_test__create_repos <null> (libsvn_test-1.so.0+0x5edd)
> #8 create_greek_repos <null> (client-test+0x3f74)
> #9 test_foreign_repos_copy <null> (client-test+0x6e84)
> #10 test_thread <null> (libsvn_test-1.so.0+0x87cf)
> #11 <null> <null> (libtsan.so.2+0x34247)
>
> Location is heap block of size 8192 at 0x7b9000006000 allocated by main
> thread:
> #0 malloc <null> (libtsan.so.2+0x39137)
> #1 <null> <null> (libapr-1.so.0+0x24171)
> #2 svn_utf_initialize2 <null> (libsvn_subr-1.so+0x81704)
> #3 svn_test_main <null> (libsvn_test-1.so.0+0x91b0)
> #4 main <null> (client-test+0x3e0e)
>
> Mutex M0 (0x7b90000060b0) created at:
> #0 pthread_mutex_init <null> (libtsan.so.2+0x3a616)
> #1 apr_thread_mutex_create <null> (libapr-1.so.0+0x23d1d)
> #2 svn_utf_initialize2 <null> (libsvn_subr-1.so+0x81717)
> #3 svn_test_main <null> (libsvn_test-1.so.0+0x91b0)
> #4 main <null> (client-test+0x3e0e)
>
> Thread T2 (tid=11009, running) created by main thread at:
> #0 pthread_create <null> (libtsan.so.2+0x39fe9)
> #1 svn_test_main <null> (libsvn_test-1.so.0+0x9efd)
> #2 main <null> (client-test+0x3e0e)
>
> Thread T3 (tid=11010, running) created by main thread at:
> #0 pthread_create <null> (libtsan.so.2+0x39fe9)
> #1 svn_test_main <null> (libsvn_test-1.so.0+0x9efd)
> #2 main <null> (client-test+0x3e0e)
>
> SUMMARY: ThreadSanitizer: data race
> (/var/tmp/portage/dev-vcs/subversion-1.14.5/work/subversion-1.14.5/subversion/libsvn_subr/.libs/libsvn_subr-1.so+0x8105e)
> in put_xlate_handle_node
> ==================
> [...]
>
> I interrupt the test suite after some time, the full tests.log so far is
> appended.
>
>
> > Now, I'll try to apply this to the subversion Makefile and run the test
> suite...
>
AW: svnserve frequently die at assertion in utf.c
Posted by "Jäkel, Guido" <G....@dnb.de>.
Building and running the test suite non-parallel shows up a single failure
# make clean
# EXTRA_CFLAGS="-fsanitize=thread" EXTRA_LDFLAGS="$EXTRA_CFLAGS" make check
[...]
[028/122] locks-test...............................................................................................................................FAILURE
[...]
tests.log
[...]
PASS: locks-test 13: lock callback error
svn_tests: E200006: Expected error but got SVN_NO_ERROR
FAIL: locks-test 14: lock/unlock when 'write-lock' couldn't be obtained
[...]
But running it with just two threads rises the issue nearly at once:
# make -j 2 PARALLEL=2 check
[...]
[001/122] auth-test................................................................................................................................success
[002/122] authz-test...............................................................................................................................success
[003/122] bit-array-test...........................................................................................................................success
[004/122] cache-test...............................................................................................................................success
[005/122] checksum-test............................................................................................................................success
[006/122] client-test..............................................................................................................................FAILURE
[007/122] compat-test..............................................................................................................................success
[008/122] compress-test............................................................................................................................success
[009/122] config-test..............................................................................................................................success
[010/122] conflict-data-test.......................................................................................................................success
[011/122] conflicts-test...........................................................................................................................success
[012/122] crypto-test..............................................................................................................................success
[013/122] db-test..................................................................................................................................success
[014/122] diff-diff3-test..........................................................................................................................FAILURE
[015/122] dirent_uri-test..........................................................................................................................success
[016/122] dump-load-test...........................................................................................................................FAILURE
[017/122] entries-compat-test......................................................................................................................FAILURE
[018/122] error-code-test..........................................................................................................................success
[019/122] error-test...............................................................................................................................success
[020/122] filesize-test............................................................................................................................success
[021/122] fs-fs-pack-test..........................................................................................................................FAILURE
[022/122] fs-fs-private-test.......................................................................................................................FAILURE
[023/122] fs-sequential-test.......................................................................................................................success
[024/122] fs-test..................................................................................................................................FAILURE
[025/122] fs-x-pack-test...........................................................................................................................success
[026/122] hashdump-test............................................................................................................................success
[027/122] io-test....................................................................................................................FAILURE
[028/122] locks-test...............................................................................................................................FAILURE
[029/122] mergeinfo-test...........................................................................................................................FAILURE
[030/122] mtcc-test................................................................................................................................FAILURE
[031/122] op-depth-test............................................................................................................................FAILURE
[032/122] opt-test.................................................................................................................................success
[033/122] packed-data-test.........................................................................................................................success
[034/122] parse-diff-test..........................................................................................................................success
[...]
And the thread sanitizer mention about function names I observered as failed asserts in the strace sessions:
tests.log
[...]
START: checksum-test
PASS: checksum-test 1: checksum parse
PASS: checksum-test 2: checksum emptiness
PASS: checksum-test 3: zero checksum matching
PASS: checksum-test 4: zlib expansion test (zlib regression)
PASS: checksum-test 5: zero checksum cross-type matching
PASS: checksum-test 6: checksum (de-)serialization
PASS: checksum-test 7: checksum parse all zero
PASS: checksum-test 8: read from checksummed stream
PASS: checksum-test 9: reset checksummed stream
END: checksum-test
ELAPSED: checksum-test 0:00:00.651333
START: client-test
==================
WARNING: ThreadSanitizer: data race (pid=10993)
Write of size 8 at 0x7b9000006338 by thread T2 (mutexes: write M0):
#0 put_xlate_handle_node <null> (libsvn_subr-1.so+0x8105e)
#1 svn_utf_cstring_from_utf8 <null> (libsvn_subr-1.so+0x8163d)
#2 svn_path_cstring_from_utf8 <null> (libsvn_subr-1.so+0x60a93)
#3 svn_opt__arg_canonicalize_path <null> (libsvn_subr-1.so+0x5ab57)
#4 svn_client_args_to_target_array2 <null> (libsvn_client-1.so+0x20861)
#5 test_args_to_target_array <null> (client-test+0x7d31)
#6 test_thread <null> (libsvn_test-1.so.0+0x873f)
#7 <null> <null> (libtsan.so.2+0x34247)
Previous read of size 8 at 0x7b9000006338 by thread T3:
#0 put_xlate_handle_node <null> (libsvn_subr-1.so+0x80f93)
#1 svn_utf_cstring_from_utf8 <null> (libsvn_subr-1.so+0x8163d)
#2 svn_path_cstring_from_utf8 <null> (libsvn_subr-1.so+0x60a93)
#3 svn_io_dir_open <null> (libsvn_subr-1.so+0x4a089)
#4 svn_io_get_dirents3 <null> (libsvn_subr-1.so+0x4a1bd)
#5 svn_io_remove_dir2 <null> (libsvn_subr-1.so+0x4fc8a)
#6 svn_test__create_repos2 <null> (libsvn_test-1.so.0+0x5bfc)
#7 svn_test__create_repos <null> (libsvn_test-1.so.0+0x5edd)
#8 create_greek_repos <null> (client-test+0x3f74)
#9 test_foreign_repos_copy <null> (client-test+0x6e84)
#10 test_thread <null> (libsvn_test-1.so.0+0x87cf)
#11 <null> <null> (libtsan.so.2+0x34247)
Location is heap block of size 8192 at 0x7b9000006000 allocated by main thread:
#0 malloc <null> (libtsan.so.2+0x39137)
#1 <null> <null> (libapr-1.so.0+0x24171)
#2 svn_utf_initialize2 <null> (libsvn_subr-1.so+0x81704)
#3 svn_test_main <null> (libsvn_test-1.so.0+0x91b0)
#4 main <null> (client-test+0x3e0e)
Mutex M0 (0x7b90000060b0) created at:
#0 pthread_mutex_init <null> (libtsan.so.2+0x3a616)
#1 apr_thread_mutex_create <null> (libapr-1.so.0+0x23d1d)
#2 svn_utf_initialize2 <null> (libsvn_subr-1.so+0x81717)
#3 svn_test_main <null> (libsvn_test-1.so.0+0x91b0)
#4 main <null> (client-test+0x3e0e)
Thread T2 (tid=11009, running) created by main thread at:
#0 pthread_create <null> (libtsan.so.2+0x39fe9)
#1 svn_test_main <null> (libsvn_test-1.so.0+0x9efd)
#2 main <null> (client-test+0x3e0e)
Thread T3 (tid=11010, running) created by main thread at:
#0 pthread_create <null> (libtsan.so.2+0x39fe9)
#1 svn_test_main <null> (libsvn_test-1.so.0+0x9efd)
#2 main <null> (client-test+0x3e0e)
SUMMARY: ThreadSanitizer: data race (/var/tmp/portage/dev-vcs/subversion-1.14.5/work/subversion-1.14.5/subversion/libsvn_subr/.libs/libsvn_subr-1.so+0x8105e) in put_xlate_handle_node
==================
[...]
I interrupt the test suite after some time, the full tests.log so far is appended.
> Now, I'll try to apply this to the subversion Makefile and run the test suite...
Re: svnserve frequently die at assertion in utf.c
Posted by Sean McBride <se...@rogue-research.com>.
On 7 Jan 2025, at 2:07, Guido Jäkel wrote:
> I'm no developer, but Google probably tell me to add the right flags:
Yes, -fsanitize=thread is the main flag. You should also build in debug so that you get filenames and line numbers.
Are you saying svn's own test suite does not pass under Thread Sanitizer? That's a bit shocking, but if true, is definitely something the svn devs should check out. TSan has very few false positives, and is an amazing tool. A must for any C/C++ multithreaded code.
Sean
Re: svnserve frequently die at assertion in utf.c
Posted by Guido Jäkel <G....@DNB.DE>.
Dear Sean,
I'm no developer, but Google probably tell me to add the right flags:
root@testsub0 /var/tmp/portage/dev-vcs/subversion-1.14.5/work/subversion-1.14.5/testcase # diff Makefile{.20250107-073455,} -u
--- Makefile.20250107-073455 2025-01-07 07:34:55.000000000 +0100
+++ Makefile 2025-01-07 07:59:56.481738320 +0100
@@ -1,5 +1,5 @@
-CFLAGS=-I/usr/include/apr-1.0
-LDFLAGS=-L/usr/lib/x86_64-linux-gnu
+CFLAGS=-I/usr/include/apr-1 -fsanitize=thread
+LDFLAGS=-L/usr/lib/x86_64-linux-gnu -fsanitize=thread
LDLIBS=-lapr-1
all: main
For the testcase, this catches problems:
root@testsub0 /var/tmp/portage/dev-vcs/subversion-1.14.5/work/subversion-1.14.5/testcase # ./main n |& tail -50
Thread T3 (tid=58578, running) created by main thread at:
#0 pthread_create <null> (libtsan.so.2+0x39fe9)
#1 main <null> (main+0x1cb1)
Thread T2 (tid=58577, running) created by main thread at:
#0 pthread_create <null> (libtsan.so.2+0x39fe9)
#1 main <null> (main+0x1cb1)
SUMMARY: ThreadSanitizer: data race (/var/tmp/portage/dev-vcs/subversion-1.14.5/work/subversion-1.14.5/testcase/main+0x1695) in threadfunc
==================
==================
WARNING: ThreadSanitizer: data race (pid=58573)
Write of size 8 at 0x7b0400003848 by thread T1:
#0 threadfunc <null> (main+0x167e)
#1 <null> <null> (libtsan.so.2+0x34247)
Previous write of size 8 at 0x7b0400003848 by thread T10:
#0 threadfunc <null> (main+0x167e)
#1 <null> <null> (libtsan.so.2+0x34247)
As if synchronized via sleep:
#0 usleep <null> (libtsan.so.2+0x5acbd)
#1 threadfunc <null> (main+0x1794)
#2 <null> <null> (libtsan.so.2+0x34247)
Location is heap block of size 16 at 0x7b0400003840 allocated by thread T7:
#0 malloc <null> (libtsan.so.2+0x39137)
#1 threadfunc <null> (main+0x1532)
#2 <null> <null> (libtsan.so.2+0x34247)
Thread T1 (tid=58576, running) created by main thread at:
#0 pthread_create <null> (libtsan.so.2+0x39fe9)
#1 main <null> (main+0x1cb1)
Thread T10 (tid=58585, running) created by main thread at:
#0 pthread_create <null> (libtsan.so.2+0x39fe9)
#1 main <null> (main+0x1cb1)
Thread T7 (tid=58582, running) created by main thread at:
#0 pthread_create <null> (libtsan.so.2+0x39fe9)
#1 main <null> (main+0x1cb1)
SUMMARY: ThreadSanitizer: data race (/var/tmp/portage/dev-vcs/subversion-1.14.5/work/subversion-1.14.5/testcase/main+0x167e) in threadfunc
==================
ThreadSanitizer: reported 4 warnings
Creating mutex and threads... Done
Waiting for threads to finish... Finished
Checking for errors...
Thread 1 missing 0 found 4
Now, I'll try to apply this to the subversion Makefile and run the test suite...
On 07.01.25 05:11, Sean McBride wrote:
> On 6 Jan 2025, at 16:32, Daniel Sahlberg wrote:
>
>> To me, this proves that my basic program is correct and that with a mutex, the shared memory is protected and without the mutex we get errors. I have tested this under both Ubuntu and Guix with the same results.
>
> If you really want to prove correctness, build with thread sanitizer; that'll help catch any race conditions.
>
> Sean
Re: svnserve frequently die at assertion in utf.c
Posted by Sean McBride <se...@rogue-research.com>.
On 6 Jan 2025, at 16:32, Daniel Sahlberg wrote:
> To me, this proves that my basic program is correct and that with a mutex, the shared memory is protected and without the mutex we get errors. I have tested this under both Ubuntu and Guix with the same results.
If you really want to prove correctness, build with thread sanitizer; that'll help catch any race conditions.
Sean