You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by J Robert Ray <jr...@imageworks.com> on 2004/10/07 23:29:37 UTC
mass rearrange and berkeley db lock exhaustion
Hi all,
I am developing a script (using the svn perl bindings) to do a mass
rearrange of a large software project. In very simple terms, my script
does many, many SVN::Client::ls() (non-recursive) calls, with the
occasional SVN::Client::mkdir() and SVN::Client::move().
These are the only calls I am making, and they are all against
respository URLs, I am not using a working copy.
As I run my script, and it is off sniffing around the repository via ls,
I am running 'watch db_stat -c' in another terminal.
""
150 Number of current lockers.
1681 Maximum number of lockers at any one time.
""
The 'Number of current lockers' count grows rapidly and eventually meets
up with the max, and the script dies with a 400 error. I can always go
to the apache server log and dig up a message like this one:
(20014)Error string not specified yet: Berkeley DB error while opening
'representations' table for filesystem
/mnt/d3/subversion/move_test/db:\nCannot allocate memory
Vital stats:
server: svn 1.0.6, apache 2.0.59
client: svn 1.1.0, perl 5.8.0, swig 1.3.21
berkeley db: 4.2.52
Any suggestions on how I can avoid running out of lockers?
I have tried creating a new SVN::Client object to use after n operations
to see if that would flush out old transactions and free up locks. This
doesn't really seem to help. Sometimes the script will run for a few
minutes before running out of locks, sometimes it dies within a few seconds.
After reading the berkeley db docs, I understand that the number of
lockers in use is akin to the number of concurrent transactions. As I
am only calling ls(), mkdir(), and move() serially, and I am operating
directly on the repository, I should in theory only ever have one active
transaction at a time.
Maybe unrelated, my script segfaults periodically in libneon _init().
Here's a sample stack trace:
#0 0x407c9cec in _init ()
from
/net/soft_scratch/users/jrray/rpm/BUILD/subversion-1.1.0/neon/src/.libs/libneon.so.24
#1 0x407d1101 in ne_sock_readline (sock=0xc40ade8, buf=0xc404be0 "",
buflen=8192)
at ne_socket.c:595
#2 0x407cb49c in read_status_line (req=0xc404bb8, status=0xc406cdc,
retry=0)
at ne_request.c:921
#3 0x407cb726 in send_request (req=0xc404bb8, request=0x0) at
ne_request.c:991
#4 0x407cbee7 in ne_begin_request (req=0xc404bb8) at ne_request.c:1179
#5 0x407cc17f in ne_request_dispatch (req=0xc404bb8) at ne_request.c:1263
#6 0x406c57ee in parsed_request (sess=0xc3fe900, method=0x406c7407
"OPTIONS",
url=0xc38e2c0 "/svn/XXXX/xxxx/tags", body=0xc408d70 "",
body_file=0x460,
set_parser=0, elements=0x460, use_neon_shim=1,
validate_compat_cb=0x460,
startelm_compat_cb=0x460, endelm_compat_cb=0x460, startelm_cb=0x460,
cdata_cb=0x460, endelm_cb=0x460, baton=0xbfffd748, extra_headers=0x0,
status_code=0x0, pool=0x83201d0) at subversion/libsvn_ra_dav/util.c:603
#7 0x406c5c3c in svn_ra_dav__parsed_request_compat (sess=0x460,
method=0x460 <Address 0x460 out of bounds>,
url=0x460 <Address 0x460 out of bounds>,
body=0x460 <Address 0x460 out of bounds>, body_file=0x460,
set_parser=0x460,
elements=0x460, validate_cb=0x460, startelm_cb=0x460, endelm_cb=0x460,
baton=0x460, extra_headers=0x460, status_code=0x460, pool=0x460)
at subversion/libsvn_ra_dav/util.c:716
#8 0x406c209c in svn_ra_dav__get_activity_collection (activity_coll=0x460,
ras=0x460, url=0x460 <Address 0x460 out of bounds>, pool=0x460)
at subversion/libsvn_ra_dav/options.c:126
#9 0x406bab8a in get_activity_collection (cc=0xc3fa650,
collection=0xbfffd7cc,
force=0, pool=0x83201d0) at subversion/libsvn_ra_dav/commit.c:302
#10 0x406bac7b in create_activity (cc=0xc3fa650, pool=0x83201d0)
at subversion/libsvn_ra_dav/commit.c:332
#11 0x406bc43f in svn_ra_dav__get_commit_editor (session_baton=0xc3fa5a0,
editor=0x0, edit_baton=0x460, log_msg=0xc3fa490 "", callback=0x460,
callback_baton=0x460, pool=0x83201d0) at
subversion/libsvn_ra_dav/commit.c:1265
#12 0x40433e49 in mkdir_urls (commit_info=0x460, paths=0x460, ctx=0xc3ac548,
pool=0x83201d0) at subversion/libsvn_client/add.c:576
#13 0x4043412c in svn_client_mkdir (commit_info=0x460, paths=0xc3fa1b8,
ctx=0xc3ac548, pool=0x83201d0) at subversion/libsvn_client/add.c:607
#14 0x40958677 in _wrap_svn_client_mkdir (cv=0x82ecba0) at svn_client.c:3412
#15 0x080a314f in Perl_pp_entersub ()
#16 0x0809d3aa in Perl_runops_standard ()
#17 0x0805f117 in S_run_body ()
#18 0x0805eeaa in perl_run ()
#19 0x0805c8e3 in main ()
#20 0x40064914 in __libc_start_main () from /lib/libc.so.6
Thanks,
- Robert
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: mass rearrange and berkeley db lock exhaustion
Posted by kf...@collab.net.
J Robert Ray <jr...@imageworks.com> writes:
> Script is attached. I can't share my repository, it's corporate
> source code and >3GB anyway...
J, just thought you'd want to know:
Mike Pilato is right now running your script on a copy of the
Subversion repository, and we're seeing the lock counts climb madly!
This is great. I hope we'll be able to fix this bug soon...
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: mass rearrange and berkeley db lock exhaustion
Posted by J Robert Ray <jr...@imageworks.com>.
kfogel@collab.net wrote:
> J Robert Ray <jr...@imageworks.com> writes:
>
>>I am developing a script (using the svn perl bindings) to do a mass
>>rearrange of a large software project. In very simple terms, my
>>script does many, many SVN::Client::ls() (non-recursive) calls, with
>>the occasional SVN::Client::mkdir() and SVN::Client::move().
>>
>>These are the only calls I am making, and they are all against
>>respository URLs, I am not using a working copy.
>>
>>As I run my script, and it is off sniffing around the repository via
>>ls, I am running 'watch db_stat -c' in another terminal.
>>
>>""
>>150 Number of current lockers.
>>1681 Maximum number of lockers at any one time.
>>""
>>
>>The 'Number of current lockers' count grows rapidly and eventually
>>meets up with the max, and the script dies with a 400 error. I can
>>always go to the apache server log and dig up a message like this one:
>>
>>(20014)Error string not specified yet: Berkeley DB error while opening
>>'representations' table for filesystem
>>/mnt/d3/subversion/move_test/db:\nCannot allocate memory
>
>
> This is very exciting! We've encountered the same error message from
> Subversion's own repository a couple of times, and have heard
> occasional reports of it happening elsewhere. But this is the first
> thing resembling a reproduction recipe we've gotten.
>
> Can you share your script with us? (And the repository would be good
> too, although probably just looking at the script would be enough to
> start tracing down the bug in Subversion.)
Script is attached. I can't share my repository, it's corporate source
code and >3GB anyway...
>
>
>>Vital stats:
>>server: svn 1.0.6, apache 2.0.59
>>client: svn 1.1.0, perl 5.8.0, swig 1.3.21
>>berkeley db: 4.2.52
>
>
> If you upgrade the server to 1.1.0, does it still have the problem?
> (If you have time to try it with svnserve and svn://, that could be
> useful information too.)
Yes, 1.1.0 client and server has the same problem. When I try using
svn://, I get a slightly better diagnostic output. This error is
returned to the client:
Berkeley DB error: Berkeley DB error while opening 'nodes' table for
filesystem /net/soft_scratch/users/jrray/software_copy/db:
Cannot allocate memory: bdb: Logging region out of memory; you may need
to increase its size at ./svn_restructure.pl line 46
My DB_CONFIG settings are currently at factory defaults.
After the above failure, 'svnadmin recover' can no longer recover the
database:
Repository lock acquired.
Please wait; recovering the repository may take some time...
svn: DB_RUNRECOVERY: Fatal error, run database recovery
svn: bdb: Logging region out of memory; you may need to increase its size
svn: bdb: Recovery function for LSN 17571 340797 failed on backward pass
svn: bdb: PANIC: Cannot allocate memory
svn: bdb: PANIC: fatal region error detected; run recovery
svn: bdb: PANIC: fatal region error detected; run recovery
...
In a different test repository when I had this problem I bumped up the
DB_CONFIG values to look like this:
set_cachesize 0 8388608 8
set_lg_regionmax 524288
set_lg_bsize 2097152
set_lg_max 8388608
After that I could recover the database, and my script runs without
hitting the lock limit it seems. I'm ready to retry this with another
test repository if you want to guide me through some experiments.
This is all starting to look like a berkeley db configuration issue to
me and not necessarily a subversion bug, but that's your determination
to make.
There is still that neon problem, though.
Thanks for your help,
- Robert
Re: mass rearrange and berkeley db lock exhaustion
Posted by J Robert Ray <jr...@imageworks.com>.
kfogel@collab.net wrote:
> J Robert Ray <jr...@imageworks.com> writes:
>
>>Maybe unrelated, my script segfaults periodically in libneon
>>_init(). Here's a sample stack trace:
>
>
> Oooh. No idea what's going on there, sorry. I suspect the neon init
> segfaults are unrelated to the server-side lock count problem, unless
> you notice some correlation between them (?).
Hi Karl,
I'm waiting for approval from my manager to share my script with this
list. In the meantime, I'll copy here an email I sent privately to Ben.
This is focused more on the segfault problem (and severe memory leak).
I have a couple more observations to share with you.
I have moved my test environment up to 1.10 client/server and migrated
over to fsfs, to side-step the berkeley db grief.
Still getting nailed by the periodic segfault, I hacked up my build
script to force neon to be compiled with just '-g', to better debug the
problem. After doing so, the segfault goes away and is replaced by a
"friendlier" error message:
RA layer request failed: PROPFIND request failed on '/svn/repo/moo':
PROPFIND of '/svn/repo/moo': Could not read status line: Bad file
descriptor (http://pippen) at ./svn_restructure.pl line 39
Okay, time to try to bypass neon. I switched over to using file://...
First hurdle was the is_canonical assertion.
$svn->ls('http://foo/svn/repo/bar') fails assertion.
$svn->ls('http://foo/svn/repo/bar/') passes assertion.
$svn->ls('file:///repo/bar') passes assertion.
$svn->ls('file:///repo/bar/') fails assertion.
This is frustrating! The svn command line tool doesn't care either way.
But no matter, I adjusted my script to make it happy.
The big surprise here is when using file:// URLs it leaks memory like
crazy. It ate up all 2GB of ram in less than a minute of runtime
(client side). So much for that idea.
Here's a stack trace for the 'RA layer request...' problem I mentioned
above:
Breakpoint 1, aborted (req=0xae10620,
doing=0x407f754b "Could not read status line", code=-1)
at ne_request.c:197
197 ne_session *sess = req->session;
(gdb) where
#0 aborted (req=0xae10620, doing=0x407f754b "Could not read status line",
code=-1) at ne_request.c:197
#1 0x407e7705 in read_status_line (req=0xae10620, status=0xae12744,
retry=0)
at ne_request.c:923
#2 0x407e7a45 in send_request (req=0xae10620, request=0xadf4220)
at ne_request.c:995
#3 0x407e8240 in ne_begin_request (req=0xae10620) at ne_request.c:1179
#4 0x407e8541 in ne_request_dispatch (req=0xae10620) at ne_request.c:1263
#5 0x406e1497 in parsed_request (sess=0xae0a420,
method=0x406e389d "PROPFIND", url=0xade9258 "/svn/birps_test",
body=0xad81b68 "<?xml version=\"1.0\" encoding=\"utf-8\"?><propfind
xmlns=\"DAV:\"><prop><version-controlled-configuration
xmlns=\"DAV:\"/><resourcetype xmlns=\"DAV:\"/><baseline-relative-path
xmlns=\"http://subversion.tigris.o"..., body_file=0x0,
set_parser=0x406dddb1 <set_parser>, elements=0x406e5660,
use_neon_shim=1, validate_compat_cb=0x406dd764 <validate_element>,
startelm_compat_cb=0x406dd8c1 <start_element>,
endelm_compat_cb=0x406dda41 <end_element>, startelm_cb=0, cdata_cb=0,
endelm_cb=0, baton=0xbfffdcc0, extra_headers=0xade9268,
status_code=0x0,
pool=0x8320d18) at subversion/libsvn_ra_dav/util.c:603
#6 0x406e1848 in svn_ra_dav__parsed_request_compat (sess=0xae0a420,
method=0x406e389d "PROPFIND", url=0xade9258 "/svn/birps_test",
body=0xad81b68 "<?xml version=\"1.0\" encoding=\"utf-8\"?><propfind
xmlns=\"DAV:\"><prop><version-controlled-configuration
xmlns=\"DAV:\"/><resourcetype xmlns=\"DAV:\"/><baseline-relative-path
xmlns=\"http://subversion.tigris.o"..., body_file=0x0,
set_parser=0x406dddb1 <set_parser>, elements=0x406e5660,
validate_cb=0x406dd764 <validate_element>,
startelm_cb=0x406dd8c1 <start_element>,
endelm_cb=0x406dda41 <end_element>, baton=0xbfffdcc0,
extra_headers=0xade9268, status_code=0x0, pool=0x8320d18)
at subversion/libsvn_ra_dav/util.c:716
#7 0x406de04d in svn_ra_dav__get_props (results=0xbfffdd40, sess=0xae0a420,
url=0xade9258 "/svn/birps_test", depth=0, label=0x0,
which_props=0x406e5520, pool=0x8320d18)
at subversion/libsvn_ra_dav/props.c:527
#8 0x406de0f6 in svn_ra_dav__get_props_resource (rsrc=0xbfffde5c,
sess=0xae0a420, url=0xade9238 "/svn/birps_test", label=0x0,
which_props=0x406e5520, pool=0x8320d18)
at subversion/libsvn_ra_dav/props.c:554
#9 0x406de364 in svn_ra_dav__get_starting_props (rsrc=0xbfffde5c,
sess=0xae0a420, url=0xade9238 "/svn/birps_test", label=0x0,
pool=0x8320d18) at subversion/libsvn_ra_dav/props.c:626
#10 0x406de3f6 in svn_ra_dav__search_for_starting_props (rsrc=0xbfffde5c,
missing_path=0xbfffde4c, sess=0xae0a420, url=0xadae6a8
"/svn/birps_test",
pool=0x8320d18) at subversion/libsvn_ra_dav/props.c:656
#11 0x406de653 in svn_ra_dav__get_baseline_props (bc_relative=0xbfffdea8,
bln_rsrc=0xbfffdebc, sess=0xae0a420, url=0xadae6a8 "/svn/birps_test",
revision=-1, which_props=0x406e5548, pool=0x8320d18)
at subversion/libsvn_ra_dav/props.c:765
#12 0x406de904 in svn_ra_dav__get_baseline_info (is_dir=0x0, bc_url=0x0,
bc_relative=0x0, latest_rev=0xbfffdfd4, sess=0xae0a420,
url=0xadae6a8 "/svn/birps_test", revision=-1, pool=0x8320d18)
at subversion/libsvn_ra_dav/props.c:893
#13 0x406d7f2b in svn_ra_dav__get_latest_revnum (session_baton=0xade91c0,
latest_revnum=0xbfffdfd4, pool=0x8320d18)
at subversion/libsvn_ra_dav/fetch.c:1011
#14 0x4044d24c in svn_client__get_revision_number (revnum=0xbfffdfd4,
ra_lib=0x406e57e0, sess=0xade91c0, revision=0xbfffe120, path=0x0,
pool=0x8320d18) at subversion/libsvn_client/revisions.c:68
#15 0x4043c76f in repos_to_repos_copy (commit_info=0xbfffe11c,
src_url=0xadadc48
"http://softboss:81/svn/birps_test/Af/tags/birps-rel-3_0_62/libAfAll",
src_revision=0xbfffe120,
dst_url=0xacd0b50
"http://softboss:81/svn/birps_test/birps/tags/birps-rel-3_0_62/src/Af/libAfAll",
ctx=0xa731280, is_move=1, pool=0x8320d18)
at subversion/libsvn_client/copy.c:370
#16 0x4043e42e in setup_copy (commit_info=0xbfffe11c,
src_path=0xadadc48
"http://softboss:81/svn/birps_test/Af/tags/birps-rel-3_0_62/libAfAll",
src_revision=0xbfffe120,
dst_path=0xacd0b50
"http://softboss:81/svn/birps_test/birps/tags/birps-rel-3_0_62/src/Af/libAfAll",
is_move=1, force=0, ctx=0xa731280, pool=0x8320d18)
at subversion/libsvn_client/copy.c:1115
#17 0x4043e4d9 in svn_client_move (commit_info=0xbfffe11c,
src_path=0xadadc48
"http://softboss:81/svn/birps_test/Af/tags/birps-rel-3_0_62/libAfAll",
src_revision=0xbfffe120,
dst_path=0xacd0b50
"http://softboss:81/svn/birps_test/birps/tags/birps-rel-3_0_62/src/Af/libAfAll",
force=0, ctx=0xa731280, pool=0x8320d18)
at subversion/libsvn_client/copy.c:1151
#18 0x4097b909 in _wrap_svn_client_move (cv=0x82ed360) at svn_client.c:5436
#19 0x080a314f in Perl_pp_entersub ()
#20 0x0809d3aa in Perl_runops_standard ()
#21 0x0805f117 in S_run_body ()
#22 0x0805eeaa in perl_run ()
#23 0x0805c8e3 in main ()
#24 0x40064914 in __libc_start_main () from /lib/libc.so.6
(gdb) up
#1 0x407e7705 in read_status_line (req=0xae10620, status=0xae12744,
retry=0)
at ne_request.c:923
923 int aret = aborted(req, _("Could not read status line"),
ret);
(gdb) p *req
$1 = {method = 0xadadc08 "PROPFIND", uri = 0xadbacc0 "/svn/birps_test",
headers = 0xadf4ce8, body_cb = 0x407e6370 <body_string_send>,
body_ud = 0xae10620, body = {fd = 181934952, buf = {
buffer = 0xad81b68 "<?xml version=\"1.0\"
encoding=\"utf-8\"?><propfind
xmlns=\"DAV:\"><prop><version-controlled-configuration
xmlns=\"DAV:\"/><resourcetype xmlns=\"DAV:\"/><baseline-relative-path
xmlns=\"http://subversion.tigris.o"..., pnt = 0xad81c94 "", left = 0}},
body_size = 300, body_progress = 0,
respbuf = '\0' <repeats 8191 times>, resp = {length = 0, left = 0,
chunk_left = 0, total = 0, mode = R_TILLEOF}, header_catchers = 0x0,
private = 0xadf4d98, header_handlers = {0x0, 0x0, 0x0, 0x0, 0xad72350,
0x0 <repeats 14 times>, 0xadb8d60, 0x0, 0x0, 0xadbacd8,
0x0 <repeats 12 times>, 0xadd9148, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
0x0, 0x0, 0x0, 0xae01088, 0xad72320, 0x0, 0x0, 0x0, 0x0, 0x0},
body_readers = 0xadd9178, method_is_head = 0, use_expect100 = 0,
can_persist = 0, session = 0xae0a420, status = {major_version = 0,
minor_version = 0, code = 0, klass = 0, reason_phrase = 0x0}}
(gdb) p *req->session
$2 = {socket = 0xae16850, connected = 1, persisted = 0, is_http11 = 0,
scheme = 0xae00e30 "http", server = {hostname = 0xadae690 "softboss",
port = 81, address = 0xae011a0, current = 0xadf41e8,
hostport = 0xae07318 "softboss:81"}, proxy = {hostname = 0x0, port = 0,
address = 0x0, current = 0x0, hostport = 0x0}, use_proxy = 0,
no_persist = 0, use_ssl = 0, in_connect = 0, expect100_works = -1,
progress_cb = 0, progress_ud = 0x0, notify_cb = 0, notify_ud = 0x0,
rdtimeout = 3600, create_req_hooks = 0xadf4cd0, pre_send_hooks =
0xae00e60,
post_send_hooks = 0xae00e78, destroy_req_hooks = 0xadf4d20,
destroy_sess_hooks = 0xadf4d38, private = 0xadbaaa0,
user_agent = 0xadf4818 "User-Agent: SVN/1.1.0 (dev build, r2)
neon/0.24.7\r\n", ssl_verify_fn = 0, ssl_verify_ud = 0x0, ssl_provide_fn
= 0,
ssl_provide_ud = 0x0, error = "Unknown error.", '\0' <repeats 8177
times>}
(gdb) p *req->session->socket
$3 = {fd = 1122, error = "Bad file descriptor", '\0' <repeats 180 times>,
progress_ud = 0x0, rdtimeout = 3600, ops = 0x407fa094,
buffer = '\0' <repeats 4095 times>, bufpos = 0xae16928 "", bufavail = 0}
(gdb)
I'll leave the gdb session open for a while if you want me to poke at
anything in particular.
Cheers,
- Robert
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: mass rearrange and berkeley db lock exhaustion
Posted by kf...@collab.net.
J Robert Ray <jr...@imageworks.com> writes:
> I am developing a script (using the svn perl bindings) to do a mass
> rearrange of a large software project. In very simple terms, my
> script does many, many SVN::Client::ls() (non-recursive) calls, with
> the occasional SVN::Client::mkdir() and SVN::Client::move().
>
> These are the only calls I am making, and they are all against
> respository URLs, I am not using a working copy.
>
> As I run my script, and it is off sniffing around the repository via
> ls, I am running 'watch db_stat -c' in another terminal.
>
> ""
> 150 Number of current lockers.
> 1681 Maximum number of lockers at any one time.
> ""
>
> The 'Number of current lockers' count grows rapidly and eventually
> meets up with the max, and the script dies with a 400 error. I can
> always go to the apache server log and dig up a message like this one:
>
> (20014)Error string not specified yet: Berkeley DB error while opening
> 'representations' table for filesystem
> /mnt/d3/subversion/move_test/db:\nCannot allocate memory
This is very exciting! We've encountered the same error message from
Subversion's own repository a couple of times, and have heard
occasional reports of it happening elsewhere. But this is the first
thing resembling a reproduction recipe we've gotten.
Can you share your script with us? (And the repository would be good
too, although probably just looking at the script would be enough to
start tracing down the bug in Subversion.)
> Vital stats:
> server: svn 1.0.6, apache 2.0.59
> client: svn 1.1.0, perl 5.8.0, swig 1.3.21
> berkeley db: 4.2.52
If you upgrade the server to 1.1.0, does it still have the problem?
(If you have time to try it with svnserve and svn://, that could be
useful information too.)
> Any suggestions on how I can avoid running out of lockers?
Based on what you've described so far, this is just a bug in the
server -- the lock count shouldn't be climbing so high in the first
place. So I don't have a suggestion as to a workaround, but I'm
hoping we can fix this, assuming we can reproduce it.
> I have tried creating a new SVN::Client object to use after n
> operations to see if that would flush out old transactions and free up
> locks. This doesn't really seem to help. Sometimes the script will
> run for a few minutes before running out of locks, sometimes it dies
> within a few seconds.
>
> After reading the berkeley db docs, I understand that the number of
> lockers in use is akin to the number of concurrent transactions. As I
> am only calling ls(), mkdir(), and move() serially, and I am operating
> directly on the repository, I should in theory only ever have one
> active transaction at a time.
Right. Except, it seems, Subversion is leaving Berkeley transactions
open when it shouldn't.
> Maybe unrelated, my script segfaults periodically in libneon
> _init(). Here's a sample stack trace:
Oooh. No idea what's going on there, sorry. I suspect the neon init
segfaults are unrelated to the server-side lock count problem, unless
you notice some correlation between them (?).
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org