svnlook fails as subprocess, but succeeds when run directly?

[Karl Fogel <kf...@newton.ch.collab.net>]

Okay, Ben and I have been debugging a bizarre problem with svnlook.
We happen to be testing over ra_local, but the same problem happened
(on the server) using ra_dav.

First, set up a hooks/pre-commit script in your repository that
attempts to retrieve the log using `svnlook'.  For example, mine
contains:

   REPOS=${1}
   TXN=${2}
   SVNLOOK=/home/kfogel/src/subversion/subversion/svnlook/svnlook
   LOG=`${SVNLOOK} ${REPOS} txn ${TXN} log`
   echo "${LOG}" | grep "[a-zA-Z0-9]" > /dev/null || exit 1
   exit 0

(Don't forget to do "chmod a+x repo/hooks/pre-commit".)

This is the standard "Make sure there's a non-empty log message."
script, essentially the same one we're using on svn.collab.net.

Now try to commit with a log message that contains the character "é".
See the commit fail:

   $ cat msg
    [... see the message, including 'é' ...]
   $ svn ci -F msg
   Sending        iota
   Transmitting file data .
   subversion/libsvn_subr/utf.c:194
   apr_error: #22, src_err 0 : <Invalid argument>
     failure during string recoding
   
   subversion/libsvn_client/commit.c:685
   svn_error: #21086 : <A repository hook failed.>
     Commit failed (details follow):
   
   subversion/libsvn_repos/hooks.c:131
   svn_error: #21086 : <A repository hook failed.>
     pre-commit hook return non-zero status.  Aborting txn.
   
   $ 

Note that the top error is being printed directly to stderr by
svnlook.  The next two errors are from ra_local -- the middle error
wraps the last error.  They simply report that the pre-commit hook
script failed.  (Subversion doesn't know why the hook failed, it just
knows that the hook exited with non-zero exit status.)

So, it appears that `svnlook' got an error converting the log message
(on the txn) from UTF-8 to native encoding.

That's already weird, because there should be no problem converting
from UTF-8's "é" to ISO-8859-1's "é".  My client does that all the
time, so does Ben's.  Our locales are both Latin 1.

At this point, Ben and I naturally wanted to try running `svnlook' on
the txn by hand.  This required us to first make a temporary local mod
such that the txn would be left around after a failed commit, instead
of being aborted.  We did this by commenting out the line

   svn_fs_abort_txn (eb->txn);

in subversion/libsvn_ra_local/commit_editor.c:close_edit().  Then we
tried the commit again, watched it failed in the same way as before
(which we expected), and then we did this:

   $ svnlook /path/to/repo txn a log

And it printed out the complete log message, including the "é", with
no problem!

So can anyone think of a reason why svnlook would be able to do
charset transcoding when run by hand from the command line, but not
when invoked by a script (which itself was invoked by Subversion)?

(Yes, when http://subversion.tigris.org/issues/show_bug.cgi?id=807 is
fixed, this problem will become mostly invisible.  But although that
issue needs to be fixed for other reasons, it doesn't really help us
here -- it would just paper over a deeper problem.)

Thanks in advance,
-Karl

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: svnlook fails as subprocess, but succeeds when run directly?

[cm...@collab.net]

Blair Zajac <bl...@orcaware.com> writes:

> Karl Fogel wrote:
> > So can anyone think of a reason why svnlook would be able to do
> > charset transcoding when run by hand from the command line, but not
> > when invoked by a script (which itself was invoked by Subversion)?
> 
> The only thing I can think of is a different set of environmental
> variables, $LANG, etc.  But if it happens over ra_local, then
> why would this happen?  Does svn do any environment cleaning?

We are running the hooks *without* inheriting the environment.  See
libsvn_repos/hooks.c:run_cmd_with_output().

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: svnlook fails as subprocess, but succeeds when run directly?

[Blair Zajac <bl...@orcaware.com>]

Karl Fogel wrote:
> So can anyone think of a reason why svnlook would be able to do
> charset transcoding when run by hand from the command line, but not
> when invoked by a script (which itself was invoked by Subversion)?

The only thing I can think of is a different set of environmental
variables, $LANG, etc.  But if it happens over ra_local, then
why would this happen?  Does svn do any environment cleaning?

Best,
Blair

-- 
Blair Zajac <bl...@orcaware.com>
Web and OS performance plots - http://www.orcaware.com/orca/

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: svnlook fails as subprocess, but succeeds when run directly?

[Nuutti Kotivuori <na...@iki.fi>]

Karl Fogel wrote:
> In the longer term, I think we should just inherit the environment
> in the hook subprocesses.  Can anyone see any disadvantage or risk
> to doing so?

Kind of.

In the case of Apache and mod_dav_svn, no - except that sometimes it
would probably be easier to have a better way of setting the
environment than just mucking about with the parent environment.

But in the case of ra_local, this sounds a bit bad - since if there's
a script that uses svnlook there, then it means that it may depends on
the user committing that how the output looks like.

I think care should be had in trying to invoke the hook script in an
exactly similar manner regardless of who or from where it is invoked
from. On the other hand, it seems like a very good practice to try and
make the hook script itself behave the same, regardless of how it is
started.

In any case, just pointing it out, if no-one thought of this.

-- Naked


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: svnlook fails as subprocess, but succeeds when run directly?

[Karl Fogel <kf...@newton.ch.collab.net>]

brane@xbc.nu writes:
> Maybe because we don't call the hooks with APR_PROGRAM_ENV, so they don't get
> the real LC_CTYPE, and the default one is "C", so that the
> setlocale(LC_CTYPE, "") call in svnlook/main.c is actually a no-op?
> 
> Just guessing ...

Yup, that was the problem.  We don't pass the environment when we run
the hooks, so LC_CTYPE, LANG, whatever, are not set for the hook
scripts.

As a workaround, I've temporarily set

   LANG=en_US

in the live pre-commit and post-commit scripts on svn.collab.net.  As
you can see from revision 2734, that solved the problem :-).

In the longer term, I think we should just inherit the environment in
the hook subprocesses.  Can anyone see any disadvantage or risk to
doing so?

-Karl

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: svnlook fails as subprocess, but succeeds when run directly?

[br...@xbc.nu]

Quoting Karl Fogel <kf...@newton.ch.collab.net>:

> So can anyone think of a reason why svnlook would be able to do
> charset transcoding when run by hand from the command line, but not
> when invoked by a script (which itself was invoked by Subversion)?

Maybe because we don't call the hooks with APR_PROGRAM_ENV, so they don't get
the real LC_CTYPE, and the default one is "C", so that the setlocale(LC_CTYPE,
"") call in svnlook/main.c is actually a no-op?

Just guessing ...


    Brane

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org