You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Andreas Braathen <an...@cern.ch> on 2009/01/26 09:36:16 UTC
SVN stops responding - how to debug?
Hi everyone,
When I try to do any operation at all in subversion (e.g. commit,
check-out or similar), it just stops working.
We are three different users that all get the same problem, timeout. And
the services which svn is installed
on is working properly (got this verified). What is the way in
subversion to see what operations are being
performed in the background so I can try to find the root of the problems ?
$ svn co file:///afs/repository/project
(no response at all, stops here and unable to ctrl+c or stop it)
Kind regards,
Andreas Braathen
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=1054090
To unsubscribe from this discussion, e-mail: [users-unsubscribe@subversion.tigris.org].
Re: SVN stops responding - how to debug?
Posted by Hari Kodungallur <hk...@gmail.com>.
On Mon, Jan 26, 2009 at 1:02 PM, Andreas Braathen
<an...@cern.ch>wrote:
> -----Original Message-----
> From: Tyler Roscoe [mailto:tyler@cryptio.net]
> Sent: Mon 26/01/2009 20:12
> To: Andreas Braathen
> Cc: users@subversion.tigris.org
> Subject: Re: SVN stops responding - how to debug?
>
> On Mon, Jan 26, 2009 at 10:36:16AM +0100, Andreas Braathen wrote:
> > When I try to do any operation at all in subversion (e.g. commit,
> > check-out or similar), it just stops working.
> > We are three different users that all get the same problem, timeout. And
> > the services which svn is installed
> > on is working properly (got this verified). What is the way in
> > subversion to see what operations are being
> > performed in the background so I can try to find the root of the problems
> ?
> >
> > $ svn co file:///afs/repository/project
> > (no response at all, stops here and unable to ctrl+c or stop it)
>
> That looks like a network-mounted file system. Can you access that path
> in your shell? If not, you've found your problem.
>
> tyler
>
> Unfortunately I can access it in my shell. The problem is not the network
> nor the client. Its either the SVN repository configuration or files/locks.
> Is there no way to check what subversion is really doing without performing
> real depth 'strace' and gnu debuggers that provide tons of unnecessary
> information?
>
Does this: http://subversion.tigris.org/faq.html#nfs apply to you? (Even if
it did, I am not quite sure svn will stop responding, but just checking
anyway).
regards,
-Hari
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=1056447
To unsubscribe from this discussion, e-mail: [users-unsubscribe@subversion.tigris.org].
Re: SVN stops responding - how to debug?
Posted by Erik Huelsmann <eh...@gmail.com>.
On Tue, Jan 27, 2009 at 9:06 PM, Tyler <ty...@cryptio.net> wrote:
> On Tue, Jan 27, 2009 at 02:09:29PM +0100, Andreas Braathen wrote:
>> futex seems to be a fast lock mechanism in Unix? Can I for instance try
>> to turn of the locking to make
>> it "less reliable" for a period to check this theory?
>>
>> >open("/afs/project/db/__db.001", O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE,
>> >0666) = -1 EEXIST (Fileexists)
>> >open("/afs/project/db/__db.001", O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE,
>> >0666) = -1 EEXIST (Fileexists)
>> >open("/afs/project/db/__db.001", O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE,
>> >0666) = -1 EEXIST (Fileexists)
>> >open("/afs/project/db/__db.001", O_RDWR|O_LARGEFILE) = 4
>> >fcntl64(4, F_SETFD, FD_CLOEXEC) = 0
>> >fstat64(4, {st_mode=S_IFREG|0644, st_size=16384, ...}) = 0
>> >close(4) = 0
>> >open("/afsproject/db/__db.001", O_RDWR|O_LARGEFILE) = 4
>> >fcntl64(4, F_SETFD, FD_CLOEXEC) = 0
>> >mmap2(NULL, 16384, PROT_READ|PROT_WRITE, MAP_SHARED, 4, 0) = 0xb7cb7000
>> >close(4) = 0
>> >futex(0xb7cb7000, FUTEX_WAIT, 2, NULL
>
> Maybe someone understands this more deeply than I do but I think you're
> on the right track here. The fact that you say you haven't changed
> anything else in your workflow definitely makes me want to dig into the
> network-mounted filesystem as the source of the problems.
>
> Turning off hard locks is a good starting point.
>
> There's also a million other experiments you could try: sync the repo
> onto a local disk and see if the problem persists. Make sure you can
> access /afs/project/db/__db.001 with non-svn tools. Etc. etc.
You're aware that BDB on a networked filesystem is EXTREMELY
DANGEROUS? Right? From the filenames, I conclude you're using BDB and
from the path I conclude - like Tyler - that it's a networked file
system.
It's an unsupported and very much explicitly dangerous setup. I'm
sorry, but you have to quit using this setup if you like your
software. BTW: with FSFS, networked setups *are* supported, but *only*
if the network FS supports locks; meaning that - as far as I'm
informed - AFS is not an option. NFS only works for some well-written
v3 implementations.
Bye,
Erik.
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=1060131
To unsubscribe from this discussion, e-mail: [users-unsubscribe@subversion.tigris.org].
Re: SVN stops responding - how to debug?
Posted by Tyler <ty...@cryptio.net>.
On Tue, Jan 27, 2009 at 02:09:29PM +0100, Andreas Braathen wrote:
> futex seems to be a fast lock mechanism in Unix? Can I for instance try
> to turn of the locking to make
> it "less reliable" for a period to check this theory?
>
> >open("/afs/project/db/__db.001", O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE,
> >0666) = -1 EEXIST (Fileexists)
> >open("/afs/project/db/__db.001", O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE,
> >0666) = -1 EEXIST (Fileexists)
> >open("/afs/project/db/__db.001", O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE,
> >0666) = -1 EEXIST (Fileexists)
> >open("/afs/project/db/__db.001", O_RDWR|O_LARGEFILE) = 4
> >fcntl64(4, F_SETFD, FD_CLOEXEC) = 0
> >fstat64(4, {st_mode=S_IFREG|0644, st_size=16384, ...}) = 0
> >close(4) = 0
> >open("/afsproject/db/__db.001", O_RDWR|O_LARGEFILE) = 4
> >fcntl64(4, F_SETFD, FD_CLOEXEC) = 0
> >mmap2(NULL, 16384, PROT_READ|PROT_WRITE, MAP_SHARED, 4, 0) = 0xb7cb7000
> >close(4) = 0
> >futex(0xb7cb7000, FUTEX_WAIT, 2, NULL
Maybe someone understands this more deeply than I do but I think you're
on the right track here. The fact that you say you haven't changed
anything else in your workflow definitely makes me want to dig into the
network-mounted filesystem as the source of the problems.
Turning off hard locks is a good starting point.
There's also a million other experiments you could try: sync the repo
onto a local disk and see if the problem persists. Make sure you can
access /afs/project/db/__db.001 with non-svn tools. Etc. etc.
good luck,
tyler
Re: SVN stops responding - how to debug?
Posted by Andreas Braathen <an...@cern.ch>.
Tyler Roscoe wrote:
> On Mon, Jan 26, 2009 at 10:02:44PM +0100, Andreas Braathen wrote:
>
>> Unfortunately I can access it in my shell. The problem is not the network
>> nor the client. Its either the SVN repository configuration or files/locks.
>> Is there no way to check what subversion is really doing without performing
>> real depth 'strace' and gnu debuggers that provide tons of unnecessary information?
>>
>
> The nice thing about using strace to solve this problem is you probably
> don't care about the tons of unnecessary information; you only need to
> know what's happening right before svn locks up. I would start by at
> least giving strace a try.
>
> tyler
>
Here is the last lines in the strace output after running it. I'm not
that familiar with these commands but
futex seems to be a fast lock mechanism in Unix? Can I for instance try
to turn of the locking to make
it "less reliable" for a period to check this theory?
> open("/afs/project/db/__db.001", O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE,
> 0666) = -1 EEXIST (Fileexists)
> open("/afs/project/db/__db.001", O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE,
> 0666) = -1 EEXIST (Fileexists)
> open("/afs/project/db/__db.001", O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE,
> 0666) = -1 EEXIST (Fileexists)
> open("/afs/project/db/__db.001", O_RDWR|O_LARGEFILE) = 4
> fcntl64(4, F_SETFD, FD_CLOEXEC) = 0
> fstat64(4, {st_mode=S_IFREG|0644, st_size=16384, ...}) = 0
> close(4) = 0
> open("/afsproject/db/__db.001", O_RDWR|O_LARGEFILE) = 4
> fcntl64(4, F_SETFD, FD_CLOEXEC) = 0
> mmap2(NULL, 16384, PROT_READ|PROT_WRITE, MAP_SHARED, 4, 0) = 0xb7cb7000
> close(4) = 0
> futex(0xb7cb7000, FUTEX_WAIT, 2, NULL
>
> Does this: http://subversion.tigris.org/faq.html#nfs apply to you?
> (Even if it did, I am not quite sure svn will stop responding, but
> just checking anyway).
It applies in some way because AFS is quite similar to NFS. But the
thing is that SVN has been working for over 1 year now without having
any problems
with it. And since the client nor the server have been updated, I think
it has to do with some changes in the configuration or un removable
locks .. (?)
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=1058883
To unsubscribe from this discussion, e-mail: [users-unsubscribe@subversion.tigris.org].
Re: SVN stops responding - how to debug?
Posted by Tyler <ty...@cryptio.net>.
On Mon, Jan 26, 2009 at 10:02:44PM +0100, Andreas Braathen wrote:
> Unfortunately I can access it in my shell. The problem is not the network
> nor the client. Its either the SVN repository configuration or files/locks.
> Is there no way to check what subversion is really doing without performing
> real depth 'strace' and gnu debuggers that provide tons of unnecessary information?
The nice thing about using strace to solve this problem is you probably
don't care about the tons of unnecessary information; you only need to
know what's happening right before svn locks up. I would start by at
least giving strace a try.
tyler
RE: SVN stops responding - how to debug?
Posted by Andreas Braathen <an...@cern.ch>.
-----Original Message-----
From: Tyler Roscoe [mailto:tyler@cryptio.net]
Sent: Mon 26/01/2009 20:12
To: Andreas Braathen
Cc: users@subversion.tigris.org
Subject: Re: SVN stops responding - how to debug?
On Mon, Jan 26, 2009 at 10:36:16AM +0100, Andreas Braathen wrote:
> When I try to do any operation at all in subversion (e.g. commit,
> check-out or similar), it just stops working.
> We are three different users that all get the same problem, timeout. And
> the services which svn is installed
> on is working properly (got this verified). What is the way in
> subversion to see what operations are being
> performed in the background so I can try to find the root of the problems ?
>
> $ svn co file:///afs/repository/project
> (no response at all, stops here and unable to ctrl+c or stop it)
That looks like a network-mounted file system. Can you access that path
in your shell? If not, you've found your problem.
tyler
Unfortunately I can access it in my shell. The problem is not the network
nor the client. Its either the SVN repository configuration or files/locks.
Is there no way to check what subversion is really doing without performing
real depth 'strace' and gnu debuggers that provide tons of unnecessary information?
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=1056015
To unsubscribe from this discussion, e-mail: [users-unsubscribe@subversion.tigris.org].
Re: SVN stops responding - how to debug?
Posted by Tyler <ty...@cryptio.net>.
On Mon, Jan 26, 2009 at 10:36:16AM +0100, Andreas Braathen wrote:
> When I try to do any operation at all in subversion (e.g. commit,
> check-out or similar), it just stops working.
> We are three different users that all get the same problem, timeout. And
> the services which svn is installed
> on is working properly (got this verified). What is the way in
> subversion to see what operations are being
> performed in the background so I can try to find the root of the problems ?
>
> $ svn co file:///afs/repository/project
> (no response at all, stops here and unable to ctrl+c or stop it)
That looks like a network-mounted file system. Can you access that path
in your shell? If not, you've found your problem.
tyler