You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by "C. Michael Pilato" <cm...@collab.net> on 2010/01/04 13:43:09 UTC

Apache, Subversion hooks, and locales

CollabNet has a customer who's having trouble with locking files with
Chinese names.  They are hosting with Apache, and when they attempt to lock
such a file, the server process hangs and the client sits doing nothing
until a timeout kicks in and breaks the connection.  (I can easily reproduce
the situation myself, too.)

The problem occurs as Subversion attempts to convert all the UTF-8 pre-lock
hook parameters (which include the to-be-locked path) into the locale's
character set before using them to invoke the hook script itself.  The
problem is that even though the system locale is a UTF-8-ish one, Apache has
set the locale to the baseline "C" locale.  UTF-8 Chinese characters can't
be represented in mere ASCII, so the conversion fails.  Subversion (on the
server) creates an error when svn_io_start_cmd() is trying to convert those
hook parameters, and stores the error in an 'err' local variable in
run_hook_cmd():

   err = svn_io_start_cmd(&cmd_proc, ".", cmd, args, FALSE,
                          stdin_handle, result ? write_outhandle : null_handle,
                          write_errhandle, pool);

But this codepath still allows us to fall into this future block of code,
where we're trying to read output from the hook (which, by the way, was
never actually launched):

   if (result)
      {
        svn_stringbuf_t *native_stdout;
        SVN_ERR(svn_stringbuf_from_aprfile(&native_stdout, read_outhandle,
pool));

I believe this is the cause of the Apache process hang.  And I suspect that
we can fix this.

But what about original problem with encodings?  If I recall correctly,
Apache is intentionally locale-ignorant.  Like, not just "I don't know about
the system locale", but "I'd like to ignore the system locale.  On purpose.
 For reals."  Would an appropriate workaround for this problem be to
introduce a mod_dav_svn configuration directive which allows admins to
dictate the locale of their hook environment?

-- 
C. Michael Pilato <cm...@collab.net>
CollabNet   <>   www.collab.net   <>   Distributed Development On Demand


Re: Apache, Subversion hooks, and locales

Posted by "C. Michael Pilato" <cm...@collab.net>.
Just stumbled across issue #2487, which tracks some of this same discussion:
http://subversion.tigris.org/issues/show_bug.cgi?id=2487

Is the environment-variable-based workaround included in that issue worth
"productizing"?

C. Michael Pilato wrote:
> CollabNet has a customer who's having trouble with locking files with
> Chinese names.  They are hosting with Apache, and when they attempt to lock
> such a file, the server process hangs and the client sits doing nothing
> until a timeout kicks in and breaks the connection.  (I can easily reproduce
> the situation myself, too.)
> 
> The problem occurs as Subversion attempts to convert all the UTF-8 pre-lock
> hook parameters (which include the to-be-locked path) into the locale's
> character set before using them to invoke the hook script itself.  The
> problem is that even though the system locale is a UTF-8-ish one, Apache has
> set the locale to the baseline "C" locale.  UTF-8 Chinese characters can't
> be represented in mere ASCII, so the conversion fails.  Subversion (on the
> server) creates an error when svn_io_start_cmd() is trying to convert those
> hook parameters, and stores the error in an 'err' local variable in
> run_hook_cmd():
> 
>    err = svn_io_start_cmd(&cmd_proc, ".", cmd, args, FALSE,
>                           stdin_handle, result ? write_outhandle : null_handle,
>                           write_errhandle, pool);
> 
> But this codepath still allows us to fall into this future block of code,
> where we're trying to read output from the hook (which, by the way, was
> never actually launched):
> 
>    if (result)
>       {
>         svn_stringbuf_t *native_stdout;
>         SVN_ERR(svn_stringbuf_from_aprfile(&native_stdout, read_outhandle,
> pool));
> 
> I believe this is the cause of the Apache process hang.  And I suspect that
> we can fix this.
> 
> But what about original problem with encodings?  If I recall correctly,
> Apache is intentionally locale-ignorant.  Like, not just "I don't know about
> the system locale", but "I'd like to ignore the system locale.  On purpose.
>  For reals."  Would an appropriate workaround for this problem be to
> introduce a mod_dav_svn configuration directive which allows admins to
> dictate the locale of their hook environment?
> 


-- 
C. Michael Pilato <cm...@collab.net>
CollabNet   <>   www.collab.net   <>   Distributed Development On Demand


Re: Apache, Subversion hooks, and locales

Posted by Branko Čibej <br...@xbc.nu>.
Jack Repenning wrote:
> On Jan 4, 2010, at 5:48 AM, Mark Phippard wrote:
>
>> Could we just declare that the paths are passed as UTF-8 strings?
>
> Any chance of declaring that the paths are not only UTF-8, but also
> *composed* UTF-8?

That's an even bigger can of worms; we don't even guarantee that for
paths in the WC or in the repository, last time I looked we just blindly
used whatever form the underlying OS or apr-iconv conversion functions
supplied. And I recall problems with Mac OS X because of that.

It would indeed be nice if we guaranteed to have all paths internally
and in the repository and in hook scripts in form-C, *and* convert to
whatever the OS expects in the WC. Frankly this involves pendantic
fiddling that I for one would not be up to dealing with right now ...

-- Brane

Re: Apache, Subversion hooks, and locales

Posted by Jack Repenning <jr...@collab.net>.
On Jan 4, 2010, at 5:48 AM, Mark Phippard wrote:

> Could we just declare that the paths are passed as UTF-8 strings?

Any chance of declaring that the paths are not only UTF-8, but also  
*composed* UTF-8?

-==-
Jack Repenning
Chief Technology Officer
CollabNet, Inc.
8000 Marina Boulevard, Suite 600
Brisbane, California 94005
office: +1 650.228.2562
twitter: http://twitter.com/jrep







Re: Apache, Subversion hooks, and locales

Posted by Stefan Sperling <st...@elego.de>.
On Tue, Jan 31, 2012 at 09:19:12PM +0100, Erik Huelsmann wrote:
> Beware that you don't depend on setlocale() not having been called
> though: at least one of the popular mod_<programming language> modules
> *does* use setlocale(). (I think it was php5.)

Right. This won't be a problem with the patches I've attached to the issue.

Re: Apache, Subversion hooks, and locales

Posted by Erik Huelsmann <eh...@gmail.com>.
> Given that httpd is avoiding setlocale() we're pretty much left without
> locale support in mod_dav_svn.

Beware that you don't depend on setlocale() not having been called
though: at least one of the popular mod_<programming language> modules
*does* use setlocale(). (I think it was php5.)

Other than that: completely agreed.


Bye,

Erik.

Re: Apache, Subversion hooks, and locales

Posted by Stefan Sperling <st...@elego.de>.
Revining this old thread.

On Mon, Jan 04, 2010 at 08:59:50AM -0500, C. Michael Pilato wrote:
> We have the ability technically to pass UTF-8 strings to the hooks.  Add
> flags down the code stack towards svn_io_start_cmd() that say "Don't convert
> the parameters", or somesuch.  But is it the right thing to do?  On the plus
> side, it relieves us of any concern with locale handling in these parts.
> But this puts the conversion burden on each and every hook script.  And keep
> in mind that it's not just the input to the hooks at stake here -- we also
> need to consider the output from them (stdout and stderr messaging).  Do we
> then also expect the hooks to generate UTF-8 output instead of output in the
> system's preferred locale (so we can package it up correctly for
> transmission back to the client)?

I think this is what we'll need to expect people to do.

How can hook scripts even tell what the native encoding is?
They cannot know because their environment is empty.

Given that httpd is avoiding setlocale() we're pretty much left without
locale support in mod_dav_svn. No locale support means that there is no
concept of a "native" character set. All you've got without locales
is the standard character set of the C programming language (ASCII),
and the ability to use any user-specified character sets and convert
between them using iconv().

I've attached two patches to issue #2487 which implement this approach.
I don't really like the use of an environment variable to pass
information from mod_dav_svn to libsvn_repos. But it cannot do much
harm and might also come in handy in other situations, so why not...

Re: Apache, Subversion hooks, and locales

Posted by "C. Michael Pilato" <cm...@collab.net>.
Mark Phippard wrote:
> On Mon, Jan 4, 2010 at 8:43 AM, C. Michael Pilato <cm...@collab.net> wrote:
> 
>> But what about original problem with encodings?  If I recall correctly,
>> Apache is intentionally locale-ignorant.  Like, not just "I don't know about
>> the system locale", but "I'd like to ignore the system locale.  On purpose.
>>  For reals."  Would an appropriate workaround for this problem be to
>> introduce a mod_dav_svn configuration directive which allows admins to
>> dictate the locale of their hook environment?
> 
> Could we just declare that the paths are passed as UTF-8 strings?
> Most of the hooks do not receive paths, right?  So this is kind of
> specific to the lock hooks.

We have the ability technically to pass UTF-8 strings to the hooks.  Add
flags down the code stack towards svn_io_start_cmd() that say "Don't convert
the parameters", or somesuch.  But is it the right thing to do?  On the plus
side, it relieves us of any concern with locale handling in these parts.
But this puts the conversion burden on each and every hook script.  And keep
in mind that it's not just the input to the hooks at stake here -- we also
need to consider the output from them (stdout and stderr messaging).  Do we
then also expect the hooks to generate UTF-8 output instead of output in the
system's preferred locale (so we can package it up correctly for
transmission back to the client)?

-- 
C. Michael Pilato <cm...@collab.net>
CollabNet   <>   www.collab.net   <>   Distributed Development On Demand


Re: Apache, Subversion hooks, and locales

Posted by Mark Phippard <ma...@gmail.com>.
On Mon, Jan 4, 2010 at 8:43 AM, C. Michael Pilato <cm...@collab.net> wrote:

> But what about original problem with encodings?  If I recall correctly,
> Apache is intentionally locale-ignorant.  Like, not just "I don't know about
> the system locale", but "I'd like to ignore the system locale.  On purpose.
>  For reals."  Would an appropriate workaround for this problem be to
> introduce a mod_dav_svn configuration directive which allows admins to
> dictate the locale of their hook environment?

Could we just declare that the paths are passed as UTF-8 strings?
Most of the hooks do not receive paths, right?  So this is kind of
specific to the lock hooks.

-- 
Thanks

Mark Phippard
http://markphip.blogspot.com/