You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by John Duprey <jo...@gmail.com> on 2005/09/16 17:48:04 UTC
Bug: apache/mod_svn intermittently creates transaction dirs with incorrect permissions or ownership such that the transaction breaks.
I'm about to file a subversion bug in relation to apache/mod_svn
intermittently creating transaction dirs with incorrect permissions or
ownership such that the transaction breaks.
Many times during the day commits are failing with an error message like the
following:
*[Wed Aug 31 12:55:55 2005] [error] [client
192.189.224.121<http://192.189.224.121>
<http://192.189.224.121>] *
*Could not create activity *
*/svn/ResultsPlus/!svn/act/a24ba835-0155-2042-a734-13818c448007. [500, #0] *
*[Wed Aug 31 12:55:55 2005] [error] [client
192.189.224.121<http://192.189.224.121>
<http://192.189.224.121>] *
*could not begin a transaction [500, #13] *
*[Wed Aug 31 12:55:55 2005] [error] [client
192.189.224.121<http://192.189.224.121>
<http://192.189.224.121>] *
*Can't open file
'/svnroot/repos/ResultsPlus/db/transactions/90-1.txn/node.0.0':
*
*Permission denied [500, #13]
*
Inspection on the server side reveals one of 2 scenarios:
1. The transaction directory (e.g.
/svnroot/repos/reposname/db/transactions/90-1.txn) has the wrong
permissions and cannot be populated by the apache user (apache) - i.e.
drw-rwSrw-
2. The props file in the transaction directory (e.g.
/svnroot/repos/reposname/db/transactions/90-1.txn/props) is owned by
the root user and ONLY user has write permissions
If the commit is retried, it may or may not succeed. To temporarily fix the
problem, I must restart apache, and delete the broken transaction
directories.
I am now running the latest stable apache(httpd-2.0.54) and subversion(
subversion-1.2.3) built from scratch. (This problem was first observed with
prebuilt RPMS.) I am running Intel RedHat Enterprise Server 3 (rhel-3). I
have audited the system extensively in an effort to identify any external
processes that could be causing this problem and found now.
I have a little bash script that will repeatedly modify, commit, and sleep 1
second that tests subversion. I can reproduce this error within 10 - 30
commits. I posted this problem before (
http://svn.haxx.se/users/archive-2005-09/0228.shtml). More details can be
found in that post. This problem was reported by another user as well in
June (http://svn.haxx.se/users/archive-2005-06/1629.shtml) -- on Solaris. He
was able to work around the problem by using setfacl however, rhel-3 (kernel
2.4) doesn't appear to support setfacl completely.
If anyone can help me debug this problem further I'd appreciated it. I'm
willing to try almost anything - a debug version of subversion, apache etc.
If you'd like more information, please ask me. Otherwise, I'd welcome any
tips for successfully logging a bug on
subversion.trigris.org<http://subversion.trigris.org>.
I feel I've exhausted all configuration possibilities between
apache-subversion. I've tried creating a test repository from scratch -
ensuring all commands out-of-apache-web-server were done as apache (sudo -u
apache svnadmin ...). I've tried different file system locations. I've
removed all but svn functionality from apache. I've used the simplest svn
configuration possible..
Lost,
-John Duprey
Re: Bug: apache/mod_svn intermittently creates transaction dirs with incorrect permissions or ownership such that the transaction breaks.
Posted by John Duprey <jo...@gmail.com>.
>
> In my apache configuration (which is 1.3, but I imagine 2.x on UNIX is
> similar), there's a single parent process run as root, and a number of
> children running as 'nobody'. The children handle requests, while the
> parent binds to port 80 and manages the children.
I have verified that the children are running as user apache, the parent as
root.. but I have not run something to check continuously:
<PRE>
root 11106 0.0 0.1 14420 6428 ? S 16:02 0:00 /usr/local/bin/httpd -k start
apache 11107 0.0 0.1 14828 7188 ? S 16:02 0:00 \_ /usr/local/bin/httpd -k
start
apache 11108 0.0 0.1 14556 6564 ? S 16:02 0:00 \_ /usr/local/bin/httpd -k
start
apache 11109 0.0 0.2 16372 8500 ? S 16:02 0:00 \_ /usr/local/bin/httpd -k
start
apache 11110 0.0 0.1 14420 6492 ? S 16:02 0:00 \_ /usr/local/bin/httpd -k
start
apache 11111 0.0 0.1 14420 6492 ? S 16:02 0:00 \_ /usr/local/bin/httpd -k
start
apache 11113 0.0 0.1 14420 6496 ? S 16:02 0:00 \_ /usr/local/bin/httpd -k
start
</PRE>
Could you add something to your monitoring script to capture the whole
> process tree every couple of seconds? It's always possible, I suppose,
> that one of the apache children is erroneously running as root.
I can try it.
You said you were originally running RHEL3. I assume that this means we
> can discount any SELinux interaction?
After scouring the server myself, I asked a dedicated admin to look too. I
don't think there's external processes or security configurations that are
messing with this.
It seems like either apache or mod_svn breaking each other.
Is it ever possible that the parent process (that is running as root) would
handle some part of the request?? ...Or perhaps apache is mis-configured,
although I've whittled the config down to a bare minimum and still got this
error.
Regards,
> Malcolm
>
Re: Bug: apache/mod_svn intermittently creates transaction dirs with incorrect permissions or ownership such that the transaction breaks.
Posted by John Duprey <jo...@gmail.com>.
>
> In my apache configuration (which is 1.3, but I imagine 2.x on UNIX is
> similar), there's a single parent process run as root, and a number of
> children running as 'nobody'. The children handle requests, while the
> parent binds to port 80 and manages the children.
I have verified that the children are running as user apache, the parent as
root.. but I have not run something to check continuously:
<PRE>
root 11106 0.0 0.1 14420 6428 ? S 16:02 0:00 /usr/local/bin/httpd -k start
apache 11107 0.0 0.1 14828 7188 ? S 16:02 0:00 \_ /usr/local/bin/httpd -k
start
apache 11108 0.0 0.1 14556 6564 ? S 16:02 0:00 \_ /usr/local/bin/httpd -k
start
apache 11109 0.0 0.2 16372 8500 ? S 16:02 0:00 \_ /usr/local/bin/httpd -k
start
apache 11110 0.0 0.1 14420 6492 ? S 16:02 0:00 \_ /usr/local/bin/httpd -k
start
apache 11111 0.0 0.1 14420 6492 ? S 16:02 0:00 \_ /usr/local/bin/httpd -k
start
apache 11113 0.0 0.1 14420 6496 ? S 16:02 0:00 \_ /usr/local/bin/httpd -k
start
</PRE>
Could you add something to your monitoring script to capture the whole
> process tree every couple of seconds? It's always possible, I suppose,
> that one of the apache children is erroneously running as root.
I can try it.
You said you were originally running RHEL3. I assume that this means we
> can discount any SELinux interaction?
After scouring the server myself, I asked a dedicated admin to look too. I
don't think there's external processes or security configurations that are
messing with this.
It seems like either apache or mod_svn breaking each other.
Is it ever possible that the parent process (that is running as root) would
handle some part of the request?? ...Or perhaps apache is mis-configured,
although I've whittled the config down to a bare minimum and still got this
error.
Regards,
> Malcolm
>
Re: Bug: apache/mod_svn intermittently creates transaction dirs with incorrect permissions or ownership such that the transaction breaks.
Posted by Malcolm Rowe <ma...@farside.org.uk>.
On Fri, Sep 16, 2005 at 01:48:04PM -0400, John Duprey wrote:
> I'm about to file a subversion bug in relation to apache/mod_svn
> intermittently creating transaction dirs with incorrect permissions or
> ownership such that the transaction breaks.
>
> [...]
> Inspection on the server side reveals one of 2 scenarios:
>
> 2. The props file in the transaction directory (e.g.
> /svnroot/repos/reposname/db/transactions/90-1.txn/props) is owned by
> the root user and ONLY user has write permissions
I don't use mod_dav_svn myself (I run over svn://), but as a general
principle, if files are being created that are owned by root, they
_must_ be being created by a process running as root.
In my apache configuration (which is 1.3, but I imagine 2.x on UNIX is
similar), there's a single parent process run as root, and a number of
children running as 'nobody'. The children handle requests, while the
parent binds to port 80 and manages the children.
Could you add something to your monitoring script to capture the whole
process tree every couple of seconds? It's always possible, I suppose,
that one of the apache children is erroneously running as root.
Alternatively, if there is another process running as root that you've
not been able to find, that might be another way to find it. [Did you
check root's crontab?]
You said you were originally running RHEL3. I assume that this means we
can discount any SELinux interaction?
Regards,
Malcolm
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Bug: apache/mod_svn intermittently creates transaction dirs with
incorrect permissions or ownership such that the transaction breaks.
Posted by Michael Sinz <Mi...@sinz.org>.
John Duprey wrote:
> Michael
>
> Could you please send the script - I wrote my own "bash the server hard"
> script and other than finding an annoying behavior of "svn mkdir" it has
> yet to show a problem. (I only did 9000 or so revisions with it,
> many in
> parallel, and no problems) All over HTTP/DAV
>
>
> Nothing sexy here:
> while [ 1 ] ; do echo `date` >> JWD.txt; svn ci -m "A test commit from home."; sleep 1; done
>
> But for my problem, it exposes it quickly.
Hmmm... I have now run that for 1 hour against both of my test servers. They
are both Apache 2.0.54 and Subversion 1.2.3 builds - one on an old (very old)
GCC 2.95 based Linux 2.4.x and the other on a relatively modern GCC 3.4 based
Linux 2.6.x system, always across the network. (And I even tried HTTP and HTTPS,
not that I would think it would matter)
But then again, my test script did not show any problems either.
My guess is that there is something strange with the Apache configuration and
the MPM/Prefork/etc configuration you are using and the base thread is ending
up doing some of the work (rather than just the workers - since I assume you
are running Apache as root and having it drop priv to the apache user)
You may want to look at http://svn.sinz.com/svn/Insurrection/trunk/insurrection.conf
as it will show you basically the configuration that I run on sinz.com (a
production server that has not shown this problem but I have not hit as hard
due to the fact that it is *production*) I use basically the same setup on
my test systems.
--
Michael Sinz Technology and Engineering Director/Consultant
"Starting Startups" mailto:michael.sinz@sinz.org
My place on the web http://www.sinz.org/Michael.Sinz
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Bug: apache/mod_svn intermittently creates transaction dirs with incorrect permissions or ownership such that the transaction breaks.
Posted by John Duprey <jo...@gmail.com>.
Michael
Could you please send the script - I wrote my own "bash the server hard"
> script and other than finding an annoying behavior of "svn mkdir" it has
> yet to show a problem. (I only did 9000 or so revisions with it, many in
> parallel, and no problems) All over HTTP/DAV
Nothing sexy here:
while [ 1 ] ; do echo `date` >> JWD.txt; svn ci -m "A test commit from
home."; sleep 1; done
But for my problem, it exposes it quickly.
PS - here is the script I ran - it is *very* crude but it did the job...
Thanks.
>
Re: Bug: apache/mod_svn intermittently creates transaction dirs with
incorrect permissions or ownership such that the transaction breaks.
Posted by Michael Sinz <Mi...@sinz.org>.
John Duprey wrote:
> I'm about to file a subversion bug in relation to apache/mod_svn
> intermittently creating transaction dirs with incorrect permissions or
> ownership such that the transaction breaks.
>
> Many times during the day commits are failing with an error message like
> the following:
>
> /[Wed Aug 31 12:55:55 2005] [error] [client 192.189.224.121
> <http://192.189.224.121><http://192.189.224.121
> <http://192.189.224.121/>>] /
> /Could not create activity /
> //svn/ResultsPlus/!svn/act/a24ba835-0155-2042-a734-13818c448007. [500,
> #0] /
> /[Wed Aug 31 12:55:55 2005] [error] [client 192.189.224.121
> <http://192.189.224.121><http://192.189.224.121
> <http://192.189.224.121/>>] /
> /could not begin a transaction [500, #13] /
> /[Wed Aug 31 12:55:55 2005] [error] [client 192.189.224.121
> <http://192.189.224.121><http://192.189.224.121
> <http://192.189.224.121/>>] /
> /Can't open file
> '/svnroot/repos/ResultsPlus/db/transactions/90-1.txn/node.0.0': /
> /Permission denied [500, #13]
> /
> Inspection on the server side reveals one of 2 scenarios:
>
> 1. The transaction directory (e.g.
> /svnroot/repos/reposname/db/transactions/90-1.txn) has the wrong
> permissions and cannot be populated by the apache user (apache) -
> i.e. drw-rwSrw-
> 2. The props file in the transaction directory (e.g.
> /svnroot/repos/reposname/db/transactions/90-1.txn/props) is owned
> by the root user and ONLY user has write permissions
>
> If the commit is retried, it may or may not succeed. To temporarily fix
> the problem, I must restart apache, and delete the broken transaction
> directories.
>
> I am now running the latest stable apache(httpd-2.0.54) and
> subversion(subversion-1.2.3) built from scratch. (This problem was
> first observed with prebuilt RPMS.) I am running Intel RedHat
> Enterprise Server 3 (rhel-3). I have audited the system extensively in
> an effort to identify any external processes that could be causing this
> problem and found now.
>
> I have a little bash script that will repeatedly modify, commit, and
> sleep 1 second that tests subversion. I can reproduce this error within
> 10 - 30 commits. I posted this problem before
Could you please send the script - I wrote my own "bash the server hard"
script and other than finding an annoying behavior of "svn mkdir" it has
yet to show a problem. (I only did 9000 or so revisions with it, many in
parallel, and no problems) All over HTTP/DAV
PS - here is the script I ran - it is *very* crude but it did the job...
PPS - once I get some time, I will investigate the svn mkdir problem - basically,
it fails sometimes but not because of the directory already existing but with
a message that says that "." is out of date - which can not be true since I
have not even checked anything out yet...
--
Michael Sinz Technology and Engineering Director/Consultant
"Starting Startups" mailto:michael.sinz@sinz.org
My place on the web http://www.sinz.org/Michael.Sinz
Re: Bug: apache/mod_svn intermittently creates transaction dirs with incorrect permissions or ownership such that the transaction breaks.
Posted by Malcolm Rowe <ma...@farside.org.uk>.
On Fri, Sep 16, 2005 at 01:48:04PM -0400, John Duprey wrote:
> I'm about to file a subversion bug in relation to apache/mod_svn
> intermittently creating transaction dirs with incorrect permissions or
> ownership such that the transaction breaks.
>
> [...]
> Inspection on the server side reveals one of 2 scenarios:
>
> 2. The props file in the transaction directory (e.g.
> /svnroot/repos/reposname/db/transactions/90-1.txn/props) is owned by
> the root user and ONLY user has write permissions
I don't use mod_dav_svn myself (I run over svn://), but as a general
principle, if files are being created that are owned by root, they
_must_ be being created by a process running as root.
In my apache configuration (which is 1.3, but I imagine 2.x on UNIX is
similar), there's a single parent process run as root, and a number of
children running as 'nobody'. The children handle requests, while the
parent binds to port 80 and manages the children.
Could you add something to your monitoring script to capture the whole
process tree every couple of seconds? It's always possible, I suppose,
that one of the apache children is erroneously running as root.
Alternatively, if there is another process running as root that you've
not been able to find, that might be another way to find it. [Did you
check root's crontab?]
You said you were originally running RHEL3. I assume that this means we
can discount any SELinux interaction?
Regards,
Malcolm
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Bug: apache/mod_svn intermittently creates transaction dirs with
incorrect permissions or ownership such that the transaction breaks.
Posted by Michael Sinz <Mi...@sinz.org>.
John Duprey wrote:
> I'm about to file a subversion bug in relation to apache/mod_svn
> intermittently creating transaction dirs with incorrect permissions or
> ownership such that the transaction breaks.
>
> Many times during the day commits are failing with an error message like
> the following:
>
> /[Wed Aug 31 12:55:55 2005] [error] [client 192.189.224.121
> <http://192.189.224.121><http://192.189.224.121
> <http://192.189.224.121/>>] /
> /Could not create activity /
> //svn/ResultsPlus/!svn/act/a24ba835-0155-2042-a734-13818c448007. [500,
> #0] /
> /[Wed Aug 31 12:55:55 2005] [error] [client 192.189.224.121
> <http://192.189.224.121><http://192.189.224.121
> <http://192.189.224.121/>>] /
> /could not begin a transaction [500, #13] /
> /[Wed Aug 31 12:55:55 2005] [error] [client 192.189.224.121
> <http://192.189.224.121><http://192.189.224.121
> <http://192.189.224.121/>>] /
> /Can't open file
> '/svnroot/repos/ResultsPlus/db/transactions/90-1.txn/node.0.0': /
> /Permission denied [500, #13]
> /
> Inspection on the server side reveals one of 2 scenarios:
>
> 1. The transaction directory (e.g.
> /svnroot/repos/reposname/db/transactions/90-1.txn) has the wrong
> permissions and cannot be populated by the apache user (apache) -
> i.e. drw-rwSrw-
> 2. The props file in the transaction directory (e.g.
> /svnroot/repos/reposname/db/transactions/90-1.txn/props) is owned
> by the root user and ONLY user has write permissions
>
> If the commit is retried, it may or may not succeed. To temporarily fix
> the problem, I must restart apache, and delete the broken transaction
> directories.
>
> I am now running the latest stable apache(httpd-2.0.54) and
> subversion(subversion-1.2.3) built from scratch. (This problem was
> first observed with prebuilt RPMS.) I am running Intel RedHat
> Enterprise Server 3 (rhel-3). I have audited the system extensively in
> an effort to identify any external processes that could be causing this
> problem and found now.
>
> I have a little bash script that will repeatedly modify, commit, and
> sleep 1 second that tests subversion. I can reproduce this error within
> 10 - 30 commits. I posted this problem before
Could you please send the script - I wrote my own "bash the server hard"
script and other than finding an annoying behavior of "svn mkdir" it has
yet to show a problem. (I only did 9000 or so revisions with it, many in
parallel, and no problems) All over HTTP/DAV
PS - here is the script I ran - it is *very* crude but it did the job...
PPS - once I get some time, I will investigate the svn mkdir problem - basically,
it fails sometimes but not because of the directory already existing but with
a message that says that "." is out of date - which can not be true since I
have not even checked anything out yet...
--
Michael Sinz Technology and Engineering Director/Consultant
"Starting Startups" mailto:michael.sinz@sinz.org
My place on the web http://www.sinz.org/Michael.Sinz