You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Alex Le Dain <al...@psi.com.au> on 2006/05/25 08:10:59 UTC
TortoiseSVN Crashes SvnServe
Hi List,
I'm running svnserve under Win2000 server with Berkley database. There
are three repositories hosted. The database has been fine for ~2 years
now. I've upgraded on several occasions migrating from v1.0.1-2, v1.1.4,
and 1.3.0 (with changing the db over to use Berkley 4.3).
On occasions the svnserve application would abort without warning, say
about once every month or more (repository me), every 6 months
(repository ee) and only ever once (repository sw). Certainly liveable
and the message was always 'abnormal program termination'. On occasion
I'd have to do a recover, but there were never any of the panics
mentioned in the troubleshooting guide and the data all seemed to be
there so I was never overly concerned.
However, after upgrading svnserve to 1.3.0 these 'crashes' have appeared
to occur much more frequently. Unfortunately they now occur about every
10th access of the repositories (via the TortoiseSVN repobrowser; there
are no failures during commit, or update transactions).
Interestingly I cannot cause the crash using the command line client
(svn ls "svn://x.x.x.x/me" [even repeatedly from a batch file]) and only
see it if access is via tortoiseSVN's repobrowser. The way to reproduce
the crash is repeatedly and randomly jump all over the repo browser
window clicking on folders to open them up. I can crash the svnserve in
as little as three clicks and at worst it takes about 20-30 clicks. I
know that means it's probably TSVN, but I also fail to see why the
server should be this vulnerable unless TSVN is doing something really
illegal. Even the most recent version of TSVN (1.3.3.6219) brings the
most recent version of svnserve (1.3.1) down.
I have now upgraded the server to 1.3.3, cleared all the dead logs, and
removed the failed transactions (of which their weren't that many). The
number of failed transactions was not correlated with the crashability
of the repository. It still happens though so I'm concerned about where
exactly the problem may be - could it even be the database itself?
N.B. I've cross posted this to the TSVN and SVN groups because although
it is a TSVN initiated crash, I think the SVN server should be more
resilient to this type of attack.
cheers, Alex.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: TortoiseSVN Crashes SvnServe
Posted by "D.J. Heap" <dj...@gmail.com>.
On 5/29/06, Alex Le Dain <al...@psi.com.au> wrote:
[snip]
>
> This small test was done with svnserve and one batch file (Win2K SP4,
> 2.4GHz, 512MB), the other batch running on WinXp (SP2, 3GHz, 1GB). Both
> single CPU's.
>
> The deployment test on our database (which does crash svnserve) is:
> svnserve and one copy of the batch file (Win2K Server, SP3, 2.4GHz,
> 512MB) and other batch running on Win2K (SP4, 2.4GHz, 512MB). Both
> single CPU's.
>
I can't recreate this problem on trunk. I've got 6 never-ending 'svn
ls -v svn://localhost/tr' scripts running against a bdb repository and
have had no errors so far -- they've been running around 30 minutes.
I believe Brane (and maybe others) did a lot of work to improve the
robustness of our bdb backend if you have a new enough BDB (4.4 or
newer).
FYI I'm using Apache 2.0.58, VC 2005 Express, zlib 1.2.3, OpenSSL
0.9.8, BDB 4.4.20, neon 0.26.1 and Subversion trunk. The 1.4.x branch
probably includes all the pertinent changes as well.
DJ
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: TortoiseSVN Crashes SvnServe
Posted by Alex Le Dain <al...@psi.com.au>.
D.J. Heap wrote:
>
> What kind of machine? Multiprocessor?
This small test was done with svnserve and one batch file (Win2K SP4,
2.4GHz, 512MB), the other batch running on WinXp (SP2, 3GHz, 1GB). Both
single CPU's.
The deployment test on our database (which does crash svnserve) is:
svnserve and one copy of the batch file (Win2K Server, SP3, 2.4GHz,
512MB) and other batch running on Win2K (SP4, 2.4GHz, 512MB). Both
single CPU's.
cheers, Alex.
--
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: TortoiseSVN Crashes SvnServe
Posted by "D.J. Heap" <dj...@gmail.com>.
On 5/28/06, Alex Le Dain <al...@psi.com.au> wrote:
> D.J. Heap wrote:
> > I've had rare sporadic crashes in the test suite with BDB, but since
> > they were not consistently reproduceable and I don't use BDB, I
> > haven't tried to chase them down. If you can provide a full batch
> > script to create the repo, stress it, etc to repro it, I'll try it out
> > on BDB 4.4.20 and Subversion trunk on Win32.
>
> Thanks DJ,
>
> The repository was created with :
>
> cd\svn
> md repos
> bin\svnadmin create --fs-type bdb repos/testbdb
> bin\svnserve -d -r repos/
>
> and the spamming script is just:
>
> svn ls -v "svn://192.168.0.66/ts"
>
> repeated 50 times in a .cmd file.
What kind of machine? Multiprocessor?
>
> I am considering converting to fsfs because of a couple of comments that
> have been made, but a dump load cycle on ~10GB of data will take a while ;-)
Yes it will...IIRC, our largest repo was around 1GB when I converted
and took an hourish. Definitely get good backups and do a test run
first, though.
>
> What are you experiences with 'checkout speed' and 'finalization delay
> may cause client timeouts' and 'scaleability with number of entries',
> because in the comparison table
> (http://svnbook.red-bean.com/nightly/en/svn.reposadmin.html#svn.reposadmin.basics.backends)
> these seem to be the only real cons.
At the time I converted, fsfs was as fast or faster in all the cases I
tried (checkout was virtually the same speed and other things were
noticeably faster) and we ended up with a smaller repo as well. Since
then some things have been optimized better on both repo types, I
believe, so those results may not be valid anymore.
We've never had any troubles at all since the conversion, but our
biggest repo is only 1.5GB or so and we don't have any really huge
files. But do some testing with your data files and see how it goes
-- it's pretty easy to setup and test.
>
> Our group uses tortoiseSVN and the server is on ntfs, and I don't think
> going to a command line based client is an option at all.
>
> I am going to look into DB_CONFIG and see if there is anything obvious I
> can tweak.
>
I think I just added a zero to some of the interesting-looking
parameters and things got better. Locks and log file block sizes are
what comes to mind, but it was a long time ago. Some of the
parameters are dependant on each other, and so must be increased
together, if I remember correctly. And they affect memory usage, of
course.
DJ
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: TortoiseSVN Crashes SvnServe
Posted by Alex Le Dain <al...@psi.com.au>.
D.J. Heap wrote:
> I've had rare sporadic crashes in the test suite with BDB, but since
> they were not consistently reproduceable and I don't use BDB, I
> haven't tried to chase them down. If you can provide a full batch
> script to create the repo, stress it, etc to repro it, I'll try it out
> on BDB 4.4.20 and Subversion trunk on Win32.
Thanks DJ,
The repository was created with :
cd\svn
md repos
bin\svnadmin create --fs-type bdb repos/testbdb
bin\svnserve -d -r repos/
and the spamming script is just:
svn ls -v "svn://192.168.0.66/ts"
repeated 50 times in a .cmd file.
I am considering converting to fsfs because of a couple of comments that
have been made, but a dump load cycle on ~10GB of data will take a while ;-)
What are you experiences with 'checkout speed' and 'finalization delay
may cause client timeouts' and 'scaleability with number of entries',
because in the comparison table
(http://svnbook.red-bean.com/nightly/en/svn.reposadmin.html#svn.reposadmin.basics.backends)
these seem to be the only real cons.
Our group uses tortoiseSVN and the server is on ntfs, and I don't think
going to a command line based client is an option at all.
I am going to look into DB_CONFIG and see if there is anything obvious I
can tweak.
cheers, Alex.
--
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: TortoiseSVN Crashes SvnServe
Posted by "D.J. Heap" <dj...@gmail.com>.
On 5/28/06, Alex Le Dain <al...@psi.com.au> wrote:
[snip]
> All these errors are relatively benign, the server process did not exit.
> I then ran two copies of the spamming batch file, one on the server
> machine and one from another machine. I still wasn't able to crash the
> server, but there were a lot of the same 'Resource Device' types of
> errors. Making a repository using the fsfs database did not exhibit the
> problems.
>
> According to the tigris site I need a buddy to register this as a bug.
> Can anyone verify these faults and help me add the bug?
>
I've had rare sporadic crashes in the test suite with BDB, but since
they were not consistently reproduceable and I don't use BDB, I
haven't tried to chase them down. If you can provide a full batch
script to create the repo, stress it, etc to repro it, I'll try it out
on BDB 4.4.20 and Subversion trunk on Win32.
Personally, I found in the past (and why I switched to fsfs) that BDB
is sensitive to load based on its config parameters -- the Subversion
defaults used to be pretty low (it was a year or two ago and I'm not
sure about that anymore) and I had to tweak the parameters in the
DB_CONFIG file to fit load in regards to number of locks, lock memory,
etc. I don't remember all the details but they are doc'd in BDB
somewhere and I would often get errors somewhat similar to what you've
reported if I hit a BDB Subversion database hard and caused it to
croak because it hit some limit. Recovery usually brought it back to
life with no data loss. But fsfs has been far simpler and easier to
use and maintain for me (having very little experience with BDB).
DJ
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: TortoiseSVN Crashes SvnServe
Posted by Alex Le Dain <al...@psi.com.au>.
Hi List,
I still haven't solved my crashing problem and now am certain that it is
a bug in the svnserve program.
I created a brand new bdb repository on a win2k machine and installed
the 1.3.1 server. The spamming tool I used was a batch file consisting
of 50 copies of the line shown in the error below. Both the server and
the spamming batch are running on the same machine. This is the result
of the 11 th line of the batch file the first time it was ever run. Note
that the repository is completely empty, nothings has ever accessed it
other than the create process.
T:\bin>svn ls -v "svn://x.x.x.x/ts"
svn: Berkeley DB error for filesystem D:/Svn/repos/ts/db while opening
'changes' table:
Resource device
svn: bdb: D:/Svn/repos/ts/db\changes: Resource device
The second time I ran the batch file, this was observed on line 3:
T:\bin>svn ls -v "svn://x.x.x.x/ts"
svn: Berkeley DB error for filesystem D:/Svn/repos/ts/db while opening
'transactions' table:
Resource device
svn: bdb: D:/Svn/repos/ts/db\transactions: Resource device
and this on line 15:
T:\bin>svn ls -v "svn://x.x.x.x/ts"
svn: Berkeley DB error for filesystem D:/Svn/repos/ts/db while opening
'changes' table:
Resource device
and this on line 43:
T:\bin>svn ls -v "svn://192.168.0.66/ts"
svn: Berkeley DB error for filesystem D:/Svn/repos/ts/db while opening
'copies' table:
Resource device
All these errors are relatively benign, the server process did not exit.
I then ran two copies of the spamming batch file, one on the server
machine and one from another machine. I still wasn't able to crash the
server, but there were a lot of the same 'Resource Device' types of
errors. Making a repository using the fsfs database did not exhibit the
problems.
According to the tigris site I need a buddy to register this as a bug.
Can anyone verify these faults and help me add the bug?
cheers, Alex.
--
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: TortoiseSVN Crashes SvnServe
Posted by Alex Le Dain <al...@psi.com.au>.
Stefan Küng wrote:
>> see it if access is via tortoiseSVN's repobrowser. The way to
>> reproduce the crash is repeatedly and randomly jump all over the repo
>> browser window clicking on folders to open them up. I can crash the
>> svnserve in as little as three clicks and at worst it takes about
>> 20-30 clicks. I know that means it's probably TSVN, but I also fail to
>> see why the
>
>
> It simply *can't* be TSVN:
> 1. it's svnserve that's crashing, not TSVN. TSVN has bugs of its own :)
> 2. TSVN uses the Subversion library for all repository/working copy
> operations. We don't do anything ourselves here.
Yeah, upon reflection last night I'd come to the same conclusion. It
would appear (and that's not to say its the case) that svnserve may be
crashing because TSVN opens several connections at the same time. I've
verified using Ethereal that the conversations with svnserve are
complete (and interlaced acording to the ethereal data); that is the
crash does not occur within a conversation, but it would appear that
svnserve dies just after a conversation has occurred.
It would also appear that svnserve is not cleaning up something nicely
when hit by multiple connections (*), but I guess it doesn't rule out
the possibility of a problem with the database either. However, against
that is the fact that the repobrowser induced crash can occur at
different points in the 'tree', so it wouldn't appear to be related to
location.
(*) Given it's a server I would have thought that the cleanup code for
the converstaion would be quite stable, perhaps this is a result of some
database call not being closed out? To do with v4.3 of the BDB?
> Try
> svn ls -v "svn://x.x.x.x/me"
>
> the repobrowser always calls 'ls' with the '-v' switch (actually, we're
> calling svn_client_ls(), not the CL client, but you get the idea).
Tried this, 50 calls from a batch program running on two different
systems (one my dev machine, the other the server itself), effectively
spamming svnserve. I received a fatal error once and had to do a
recovery. The very next test produced this error:
svn: Berkeley DB error for filesystem E:/Svn/repos/me/db while opening
'nodes' table:
Resource device
and this
svn: Berkeley DB error for filesystem E:/Svn/repos/me/db while opening
'nodes' table:
Resource device
but only while both batch files were running. These errors did not
suggest a reovery, but I did one anyway before running the test again.
After a few of the nodes errors I finally got:
svn: Can't connect to host '192.168.0.128': No connection could be made
because the target mach
ine actively refused it.
Indicating that svnserve had crashed. Lookis like the crash is entirely
on the svn side.
cheers, Alex.
--
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: TortoiseSVN Crashes SvnServe
Posted by Stefan Küng <to...@gmail.com>.
Alex Le Dain wrote:
> I'm running svnserve under Win2000 server with Berkley database. There
> are three repositories hosted. The database has been fine for ~2 years
> now. I've upgraded on several occasions migrating from v1.0.1-2, v1.1.4,
> and 1.3.0 (with changing the db over to use Berkley 4.3).
>
> On occasions the svnserve application would abort without warning, say
> about once every month or more (repository me), every 6 months
> (repository ee) and only ever once (repository sw). Certainly liveable
> and the message was always 'abnormal program termination'. On occasion
> I'd have to do a recover, but there were never any of the panics
> mentioned in the troubleshooting guide and the data all seemed to be
> there so I was never overly concerned.
But you should have reported that on the Subversion mailing list. You
see, bugs that no one reports aren't going to be fixed.
> However, after upgrading svnserve to 1.3.0 these 'crashes' have appeared
> to occur much more frequently. Unfortunately they now occur about every
> 10th access of the repositories (via the TortoiseSVN repobrowser; there
> are no failures during commit, or update transactions).
>
> Interestingly I cannot cause the crash using the command line client
> (svn ls "svn://x.x.x.x/me" [even repeatedly from a batch file]) and only
Try
svn ls -v "svn://x.x.x.x/me"
the repobrowser always calls 'ls' with the '-v' switch (actually, we're
calling svn_client_ls(), not the CL client, but you get the idea).
> see it if access is via tortoiseSVN's repobrowser. The way to reproduce
> the crash is repeatedly and randomly jump all over the repo browser
> window clicking on folders to open them up. I can crash the svnserve in
> as little as three clicks and at worst it takes about 20-30 clicks. I
> know that means it's probably TSVN, but I also fail to see why the
It simply *can't* be TSVN:
1. it's svnserve that's crashing, not TSVN. TSVN has bugs of its own :)
2. TSVN uses the Subversion library for all repository/working copy
operations. We don't do anything ourselves here.
> server should be this vulnerable unless TSVN is doing something really
> illegal. Even the most recent version of TSVN (1.3.3.6219) brings the
> most recent version of svnserve (1.3.1) down.
You're going after this from the wrong direction. Always start with that
what's *not* working.
Stefan
--
___
oo // \\ "De Chelonian Mobile"
(_,\/ \_/ \ TortoiseSVN
\ \_/_\_/> The coolest Interface to (Sub)Version Control
/_/ \_\ http://tortoisesvn.tigris.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: TortoiseSVN Crashes SvnServe
Posted by Alex Le Dain <al...@psi.com.au>.
Ryan Schmidt wrote:
> I recommend converting your repository from BDB to FSFS. Instructions
> are in the FAQ:
>
> http://subversion.tigris.org/faq.html#bdb-fsfs-convert
I was going to try that. Is the problem related to access time or
something? One other point is that the repositories are large: me
~6.5GB, sw ~2.6GB and ee ~1.2 GB.
> Also note that the latest released version of Subversion at this time
> is 1.3.1. Is that the version of svnserve you have on the server?
Sorry, my mistake. Yes it is 1.3.1.
--
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: TortoiseSVN Crashes SvnServe
Posted by Garrett Rooney <ro...@electricjellyfish.net>.
On 5/25/06, Ryan Schmidt <su...@ryandesign.com> wrote:
> I recommend converting your repository from BDB to FSFS. Instructions
> are in the FAQ:
>
> http://subversion.tigris.org/faq.html#bdb-fsfs-convert
For crying out loud Ryan, not every problem can be addressed by a
conversion to fsfs. There's zero indication here that his problem is
at all related to berkeley db.
-garrett
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: TortoiseSVN Crashes SvnServe
Posted by Ryan Schmidt <su...@ryandesign.com>.
On May 25, 2006, at 10:10, Alex Le Dain wrote:
> I'm running svnserve under Win2000 server with Berkley database.
> There are three repositories hosted. The database has been fine for
> ~2 years now. I've upgraded on several occasions migrating from
> v1.0.1-2, v1.1.4, and 1.3.0 (with changing the db over to use
> Berkley 4.3).
>
> On occasions the svnserve application would abort without warning,
> say about once every month or more (repository me), every 6 months
> (repository ee) and only ever once (repository sw). Certainly
> liveable and the message was always 'abnormal program termination'.
> On occasion I'd have to do a recover, but there were never any of
> the panics mentioned in the troubleshooting guide and the data all
> seemed to be there so I was never overly concerned.
>
> However, after upgrading svnserve to 1.3.0 these 'crashes' have
> appeared to occur much more frequently. Unfortunately they now
> occur about every 10th access of the repositories (via the
> TortoiseSVN repobrowser; there are no failures during commit, or
> update transactions).
>
> Interestingly I cannot cause the crash using the command line
> client (svn ls "svn://x.x.x.x/me" [even repeatedly from a batch
> file]) and only see it if access is via tortoiseSVN's repobrowser.
> The way to reproduce the crash is repeatedly and randomly jump all
> over the repo browser window clicking on folders to open them up. I
> can crash the svnserve in as little as three clicks and at worst it
> takes about 20-30 clicks. I know that means it's probably TSVN, but
> I also fail to see why the server should be this vulnerable unless
> TSVN is doing something really illegal. Even the most recent
> version of TSVN (1.3.3.6219) brings the most recent version of
> svnserve (1.3.1) down.
>
> I have now upgraded the server to 1.3.3, cleared all the dead logs,
> and removed the failed transactions (of which their weren't that
> many). The number of failed transactions was not correlated with
> the crashability of the repository. It still happens though so I'm
> concerned about where exactly the problem may be - could it even be
> the database itself?
I recommend converting your repository from BDB to FSFS. Instructions
are in the FAQ:
http://subversion.tigris.org/faq.html#bdb-fsfs-convert
Also note that the latest released version of Subversion at this time
is 1.3.1. Is that the version of svnserve you have on the server?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org