E165001 pre-commit hook failed

[Patrik Jonsson <co...@familjenjonsson.org>]

Hi all,

I've been banging my head against this problem for a day and I need some
help. We recently updated the machine hosting our svn repo, and this broke
commits using svn+ssh. Here's the setup:

 * Machine runs debian 3.16.7, svn 1.8.10.
 * svn runs as user "www-data" (the apache user).
 * svn+ssh access uses a forced ssh command which sudos to the www-data
user and executes an svnserve wrapper.
 * svnserve wrapper sets umask to 022 and then executes "svnserve -t" with
a specific tunnel user.

This setup is identical to what it was on the old machine. However, there
must be something different about it, because now https commits work fine,
but the svn+ssh commits give the error:

Transmitting file data .svn: E165001: Commit failed (details follow):
svn: E165001: Commit blocked by pre-commit hook (exit code 255) with no
output.

If I access the repository directly using file:// and sudo to the www-data
user when executing svn, commits work fine. This, in combination with the
fact that https access works, makes me conclude it is not a permissions or
hook problem on any of the files since all these access methods run as the
www-data user. Nevertheless, the error comes from the hook, because if I
remove the hook file completely, the failure moves to the post-commit hook.

It's not a problem finding !#/bin/sh either, because I tried replacing the
hook with a compiled C program that just returns 1, and I still got the 255
return code.

When I attempt to commit, I can see successful authentication in the
syslog, like:

sudo: <user> : TTY=unknown ; PWD=/home/<user> ; USER=www-data ;
GROUP=www-data ; COMMAND=/usr/local/bin/svnserve-wrapper
sudo: pam_unix(sudo:session): session opened for user www-data by (uid=0)
sshd[26903]: Received disconnect from <ip>: 11: disconnected by user
sshd[26897]: pam_unix(sshd:session): session closed for user <user>

The svnserve log file gets this (the name of the repo here is "test")

<user> test open 2 cap=(edit-pipeline svndiff1 absent-entries depth
mergeinfo log-revprops) / SVN/1.8.10%20(x86_64-pc-linux-gnu) -

and then nothing. (I don't know what the "open" command does, it's not
included in the list of commands on e.g.
http://svnbook.red-bean.com/en/1.8/svn.serverconfig.operational-logging.html
)

I've seen some similar reports of this, but no suggestions apart from
permissions or corrupted hook files, which this can't be. I don't even know
how to proceed with debugging this. Is it possible to see what svnserve
attempts to do with the hook file? Since it's spawned on demand, I don't
know how to attach to it with a debugger, or where in the source code this
error originates.

Any ideas would be much appreciated,

Regards,

/Patrik J.

Re: E165001 pre-commit hook failed

[Patrik Jonsson <co...@familjenjonsson.org>]

No, SElinux isn't installed.

I figured it out, finally. It was pretty obscure:

The ssh login is done as user X, which then force-executes the svnserve
wrapper using sudo to the www-data user. When svnserve attempts to execute
the hook, one of the things done in the call to svn_io_start_cmd3 is to set
the path to ".". The apr_proc_create call then attempts to chdir to this
path after forking. Since the call originated in a login as user X, the cwd
was /home/X. The www-data user, however, did not have permissions to chdir
to /home/X, so the chdir(".") call failed with EACCES and the forked
process returned -1 (ie 255) and bailed on even attempting to execute the
pre-commit hook.

I think this is pretty bad in that it fails on "cd .", a completely useless
operation, and there is no diagnostic whatsoever that indicates that the
cwd has anything to do with anything, since the hook itself is being
executed with an absolute path. It should probably be mentioned somewhere
that svnserve needs to have access permissions to the cwd even if it
doesn't want to do anything with it.

Cheers,

/Patrik



On Sun, Jul 31, 2016 at 5:15 PM, David Chapman <dc...@acm.org> wrote:

> On 7/31/2016 2:30 PM, Patrik Jonsson wrote:
>
>> Hi all,
>>
>> I've been banging my head against this problem for a day and I need some
>> help. We recently updated the machine hosting our svn repo, and this broke
>> commits using svn+ssh. Here's the setup:
>>
>>  * Machine runs debian 3.16.7, svn 1.8.10.
>>  * svn runs as user "www-data" (the apache user).
>>  * svn+ssh access uses a forced ssh command which sudos to the www-data
>> user and executes an svnserve wrapper.
>>  * svnserve wrapper sets umask to 022 and then executes "svnserve -t"
>> with a specific tunnel user.
>>
>> This setup is identical to what it was on the old machine. However, there
>> must be something different about it, because now https commits work fine,
>> but the svn+ssh commits give the error:
>>
>> Transmitting file data .svn: E165001: Commit failed (details follow):
>> svn: E165001: Commit blocked by pre-commit hook (exit code 255) with no
>> output.
>>
>> If I access the repository directly using file:// and sudo to the
>> www-data user when executing svn, commits work fine. This, in combination
>> with the fact that https access works, makes me conclude it is not a
>> permissions or hook problem on any of the files since all these access
>> methods run as the www-data user. Nevertheless, the error comes from the
>> hook, because if I remove the hook file completely, the failure moves to
>> the post-commit hook.
>>
>> It's not a problem finding !#/bin/sh either, because I tried replacing
>> the hook with a compiled C program that just returns 1, and I still got the
>> 255 return code.
>>
>> When I attempt to commit, I can see successful authentication in the
>> syslog, like:
>>
>> sudo: <user> : TTY=unknown ; PWD=/home/<user> ; USER=www-data ;
>> GROUP=www-data ; COMMAND=/usr/local/bin/svnserve-wrapper
>> sudo: pam_unix(sudo:session): session opened for user www-data by (uid=0)
>> sshd[26903]: Received disconnect from <ip>: 11: disconnected by user
>> sshd[26897]: pam_unix(sshd:session): session closed for user <user>
>>
>> The svnserve log file gets this (the name of the repo here is "test")
>>
>> <user> test open 2 cap=(edit-pipeline svndiff1 absent-entries depth
>> mergeinfo log-revprops) / SVN/1.8.10%20(x86_64-pc-linux-gnu) -
>>
>> and then nothing. (I don't know what the "open" command does, it's not
>> included in the list of commands on e.g.
>> http://svnbook.red-bean.com/en/1.8/svn.serverconfig.operational-logging.html
>> )
>>
>> I've seen some similar reports of this, but no suggestions apart from
>> permissions or corrupted hook files, which this can't be. I don't even know
>> how to proceed with debugging this. Is it possible to see what svnserve
>> attempts to do with the hook file? Since it's spawned on demand, I don't
>> know how to attach to it with a debugger, or where in the source code this
>> error originates.
>>
>> Any ideas would be much appreciated,
>>
>> Regards,
>>
>> /Patrik J.
>>
>>
> Is SELinux enabled on the new server?  I've seen some oddball permission
> problems result when upgrading Linux systems if SELinux is enabled on the
> new server but not the old.  I don't use svnserve, so I can't offer
> specific advice other than the security context in which the new user runs
> may be different than that of svnserve, and SELinux may be blocking it.  On
> Red Hat/CentOS, you would look in "/var/log/audit/audit.log" for signs of
> trouble.  I don't know if that is the location of the SELinux log files
> under Debian.
>
> In particular, watch out for files (scripts, configuration files) copied
> directly from an older server without SELinux into a new server with
> SELinux.  They don't get a context appropriate to the directory in which
> you put them.  I use Apache, and I had to track down these files afterward
> and fix them one by one - very painful. This isn't just a Subversion
> problem but is a general Apache problem.
>
> If you do have SELinux running, a quick way to determine whether you have
> a security context problem is to turn SELinux off briefly.
>
> --
>     David Chapman      dcchapman@acm.org
>     Chapman Consulting -- San Jose, CA
>     Software Development Done Right.
>     www.chapman-consulting-sj.com
>
>

Re: E165001 pre-commit hook failed

[Nico Kadel-Garcia <nk...@gmail.com>]

On Sun, Jul 31, 2016 at 11:15 PM, David Chapman <dc...@acm.org> wrote:
> On 7/31/2016 2:30 PM, Patrik Jonsson wrote:
>>
>> Hi all,
>>
>> I've been banging my head against this problem for a day and I need some
>> help. We recently updated the machine hosting our svn repo, and this broke
>> commits using svn+ssh. Here's the setup:
>>
>>  * Machine runs debian 3.16.7, svn 1.8.10.
>>  * svn runs as user "www-data" (the apache user).
>>  * svn+ssh access uses a forced ssh command which sudos to the www-data
>> user and executes an svnserve wrapper.
>>  * svnserve wrapper sets umask to 022 and then executes "svnserve -t" with
>> a specific tunnel user.
>>
>> This setup is identical to what it was on the old machine. However, there
>> must be something different about it, because now https commits work fine,
>> but the svn+ssh commits give the error:
>>
>> Transmitting file data .svn: E165001: Commit failed (details follow):
>> svn: E165001: Commit blocked by pre-commit hook (exit code 255) with no
>> output.
>>
>> If I access the repository directly using file:// and sudo to the www-data
>> user when executing svn, commits work fine. This, in combination with the
>> fact that https access works, makes me conclude it is not a permissions or
>> hook problem on any of the files since all these access methods run as the
>> www-data user. Nevertheless, the error comes from the hook, because if I
>> remove the hook file completely, the failure moves to the post-commit hook.
>>
>> It's not a problem finding !#/bin/sh either, because I tried replacing the
>> hook with a compiled C program that just returns 1, and I still got the 255
>> return code.
>>
>> When I attempt to commit, I can see successful authentication in the
>> syslog, like:
>>
>> sudo: <user> : TTY=unknown ; PWD=/home/<user> ; USER=www-data ;
>> GROUP=www-data ; COMMAND=/usr/local/bin/svnserve-wrapper
>> sudo: pam_unix(sudo:session): session opened for user www-data by (uid=0)
>> sshd[26903]: Received disconnect from <ip>: 11: disconnected by user
>> sshd[26897]: pam_unix(sshd:session): session closed for user <user>
>>
>> The svnserve log file gets this (the name of the repo here is "test")
>>
>> <user> test open 2 cap=(edit-pipeline svndiff1 absent-entries depth
>> mergeinfo log-revprops) / SVN/1.8.10%20(x86_64-pc-linux-gnu) -
>>
>> and then nothing. (I don't know what the "open" command does, it's not
>> included in the list of commands on e.g.
>> http://svnbook.red-bean.com/en/1.8/svn.serverconfig.operational-logging.html)
>>
>> I've seen some similar reports of this, but no suggestions apart from
>> permissions or corrupted hook files, which this can't be. I don't even know
>> how to proceed with debugging this. Is it possible to see what svnserve
>> attempts to do with the hook file? Since it's spawned on demand, I don't
>> know how to attach to it with a debugger, or where in the source code this
>> error originates.
>>
>> Any ideas would be much appreciated,
>>
>> Regards,
>>
>> /Patrik J.
>>
>
> Is SELinux enabled on the new server?  I've seen some oddball permission
> problems result when upgrading Linux systems if SELinux is enabled on the
> new server but not the old.  I don't use svnserve, so I can't offer specific
> advice other than the security context in which the new user runs may be
> different than that of svnserve, and SELinux may be blocking it.  On Red
> Hat/CentOS, you would look in "/var/log/audit/audit.log" for signs of
> trouble.  I don't know if that is the location of the SELinux log files
> under Debian.
>
> In particular, watch out for files (scripts, configuration files) copied
> directly from an older server without SELinux into a new server with
> SELinux.  They don't get a context appropriate to the directory in which you
> put them.  I use Apache, and I had to track down these files afterward and
> fix them one by one - very painful. This isn't just a Subversion problem but
> is a general Apache problem.
>
> If you do have SELinux running, a quick way to determine whether you have a
> security context problem is to turn SELinux off briefly.

Set it to "warning", not completely disabled, or you won'g get logs in
/var/log/audit.log.

Re: E165001 pre-commit hook failed

[David Chapman <dc...@acm.org>]

On 7/31/2016 2:30 PM, Patrik Jonsson wrote:
> Hi all,
>
> I've been banging my head against this problem for a day and I need 
> some help. We recently updated the machine hosting our svn repo, and 
> this broke commits using svn+ssh. Here's the setup:
>
>  * Machine runs debian 3.16.7, svn 1.8.10.
>  * svn runs as user "www-data" (the apache user).
>  * svn+ssh access uses a forced ssh command which sudos to the 
> www-data user and executes an svnserve wrapper.
>  * svnserve wrapper sets umask to 022 and then executes "svnserve -t" 
> with a specific tunnel user.
>
> This setup is identical to what it was on the old machine. However, 
> there must be something different about it, because now https commits 
> work fine, but the svn+ssh commits give the error:
>
> Transmitting file data .svn: E165001: Commit failed (details follow):
> svn: E165001: Commit blocked by pre-commit hook (exit code 255) with 
> no output.
>
> If I access the repository directly using file:// and sudo to the 
> www-data user when executing svn, commits work fine. This, in 
> combination with the fact that https access works, makes me conclude 
> it is not a permissions or hook problem on any of the files since all 
> these access methods run as the www-data user. Nevertheless, the error 
> comes from the hook, because if I remove the hook file completely, the 
> failure moves to the post-commit hook.
>
> It's not a problem finding !#/bin/sh either, because I tried replacing 
> the hook with a compiled C program that just returns 1, and I still 
> got the 255 return code.
>
> When I attempt to commit, I can see successful authentication in the 
> syslog, like:
>
> sudo: <user> : TTY=unknown ; PWD=/home/<user> ; USER=www-data ; 
> GROUP=www-data ; COMMAND=/usr/local/bin/svnserve-wrapper
> sudo: pam_unix(sudo:session): session opened for user www-data by (uid=0)
> sshd[26903]: Received disconnect from <ip>: 11: disconnected by user
> sshd[26897]: pam_unix(sshd:session): session closed for user <user>
>
> The svnserve log file gets this (the name of the repo here is "test")
>
> <user> test open 2 cap=(edit-pipeline svndiff1 absent-entries depth 
> mergeinfo log-revprops) / SVN/1.8.10%20(x86_64-pc-linux-gnu) -
>
> and then nothing. (I don't know what the "open" command does, it's not 
> included in the list of commands on e.g. 
> http://svnbook.red-bean.com/en/1.8/svn.serverconfig.operational-logging.html)
>
> I've seen some similar reports of this, but no suggestions apart from 
> permissions or corrupted hook files, which this can't be. I don't even 
> know how to proceed with debugging this. Is it possible to see what 
> svnserve attempts to do with the hook file? Since it's spawned on 
> demand, I don't know how to attach to it with a debugger, or where in 
> the source code this error originates.
>
> Any ideas would be much appreciated,
>
> Regards,
>
> /Patrik J.
>

Is SELinux enabled on the new server?  I've seen some oddball permission 
problems result when upgrading Linux systems if SELinux is enabled on 
the new server but not the old.  I don't use svnserve, so I can't offer 
specific advice other than the security context in which the new user 
runs may be different than that of svnserve, and SELinux may be blocking 
it.  On Red Hat/CentOS, you would look in "/var/log/audit/audit.log" for 
signs of trouble.  I don't know if that is the location of the SELinux 
log files under Debian.

In particular, watch out for files (scripts, configuration files) copied 
directly from an older server without SELinux into a new server with 
SELinux.  They don't get a context appropriate to the directory in which 
you put them.  I use Apache, and I had to track down these files 
afterward and fix them one by one - very painful. This isn't just a 
Subversion problem but is a general Apache problem.

If you do have SELinux running, a quick way to determine whether you 
have a security context problem is to turn SELinux off briefly.

-- 
     David Chapman      dcchapman@acm.org
     Chapman Consulting -- San Jose, CA
     Software Development Done Right.
     www.chapman-consulting-sj.com