You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@subversion.apache.org by ri...@apache.org on 2025/05/20 18:31:35 UTC

svn commit: r1925722 - in /subversion/branches/utf8-cmdline-prototype/subversion: include/svn_client.h libsvn_client/cmdline.c

Author: rinrab
Date: Tue May 20 18:31:35 2025
New Revision: 1925722

URL: http://svn.apache.org/viewvc?rev=1925722&view=rev
Log:
On the 'utf8-cmdline-prototype' branch: add a new function that retrieves
target list from args, instead of converting the encoding, assumes that
args have already been converted.

* subversion/include/svn_client.h
  (svn_client_args_to_target_array_utf8): Declare symbol.
* subversion/libsvn_client/cmdline.c
  (args_to_target_array): Factor out an internal helper.
  (svn_client_args_to_target_array2): Implement through args_to_target_array().
  (svn_client_args_to_target_array_utf8): Implement function.

No functional changes in the cmdline.

Modified:
    subversion/branches/utf8-cmdline-prototype/subversion/include/svn_client.h
    subversion/branches/utf8-cmdline-prototype/subversion/libsvn_client/cmdline.c

Modified: subversion/branches/utf8-cmdline-prototype/subversion/include/svn_client.h
URL: http://svn.apache.org/viewvc/subversion/branches/utf8-cmdline-prototype/subversion/include/svn_client.h?rev=1925722&r1=1925721&r2=1925722&view=diff
==============================================================================
--- subversion/branches/utf8-cmdline-prototype/subversion/include/svn_client.h (original)
+++ subversion/branches/utf8-cmdline-prototype/subversion/include/svn_client.h Tue May 20 18:31:35 2025
@@ -1197,6 +1197,20 @@ svn_client_args_to_target_array(apr_arra
                                 svn_client_ctx_t *ctx,
                                 apr_pool_t *pool);
 
+/**
+ * Similar to svn_client_args_to_target_array2() but assuming that the targets
+ * in @a os are already UTF-8 encoded.
+ *
+ * @since New in 1.15.0
+ */
+svn_error_t *
+svn_client_args_to_target_array_utf8(apr_array_header_t **targets_p,
+                                     apr_getopt_t *os,
+                                     const apr_array_header_t *known_targets,
+                                     svn_client_ctx_t *ctx,
+                                     svn_boolean_t keep_last_origpath_on_truepath_collision,
+                                     apr_pool_t *pool);
+
 /** @} group end: Client command-line processing */
 
 /** @} */

Modified: subversion/branches/utf8-cmdline-prototype/subversion/libsvn_client/cmdline.c
URL: http://svn.apache.org/viewvc/subversion/branches/utf8-cmdline-prototype/subversion/libsvn_client/cmdline.c?rev=1925722&r1=1925721&r2=1925722&view=diff
==============================================================================
--- subversion/branches/utf8-cmdline-prototype/subversion/libsvn_client/cmdline.c (original)
+++ subversion/branches/utf8-cmdline-prototype/subversion/libsvn_client/cmdline.c Tue May 20 18:31:35 2025
@@ -110,15 +110,16 @@ check_root_url_of_target(const char **ro
    return SVN_NO_ERROR;
 }
 
-/* Note: This is substantially copied from svn_opt__args_to_target_array() in
- * order to move to libsvn_client while maintaining backward compatibility. */
-svn_error_t *
-svn_client_args_to_target_array2(apr_array_header_t **targets_p,
-                                 apr_getopt_t *os,
-                                 const apr_array_header_t *known_targets,
-                                 svn_client_ctx_t *ctx,
-                                 svn_boolean_t keep_last_origpath_on_truepath_collision,
-                                 apr_pool_t *pool)
+/* Internal helper for public interfaces svn_client_args_to_target_array2
+ * and svn_client_args_to_target_array_utf8.
+ */
+static svn_error_t *
+args_to_target_array(apr_array_header_t **targets_p,
+                     apr_array_header_t *utf8_targets,
+                     const apr_array_header_t *known_targets,
+                     svn_client_ctx_t *ctx,
+                     svn_boolean_t keep_last_origpath_on_truepath_collision,
+                     apr_pool_t *pool)
 {
   int i;
   svn_boolean_t rel_url_found = FALSE;
@@ -135,14 +136,9 @@ svn_client_args_to_target_array2(apr_arr
      If any of the targets are relative urls, then set the rel_url_found
      flag.*/
 
-  for (; os->ind < os->argc; os->ind++)
+  for (i = 0; i < utf8_targets->nelts; i++)
     {
-      /* The apr_getopt targets are still in native encoding. */
-      const char *raw_target = os->argv[os->ind];
-      const char *utf8_target;
-
-      SVN_ERR(svn_utf_cstring_to_utf8(&utf8_target,
-                                      raw_target, pool));
+      const char *utf8_target = APR_ARRAY_IDX(utf8_targets, i, const char *);
 
       if (svn_path_is_repos_relative_url(utf8_target))
         rel_url_found = TRUE;
@@ -370,3 +366,54 @@ svn_client_args_to_target_array2(apr_arr
 
   return SVN_NO_ERROR;
 }
+
+/* Note: This is substantially copied from svn_opt__args_to_target_array() in
+ * order to move to libsvn_client while maintaining backward compatibility. */
+svn_error_t *
+svn_client_args_to_target_array2(apr_array_header_t **targets_p,
+                                 apr_getopt_t *os,
+                                 const apr_array_header_t *known_targets,
+                                 svn_client_ctx_t *ctx,
+                                 svn_boolean_t keep_last_origpath_on_truepath_collision,
+                                 apr_pool_t *pool)
+{
+  apr_array_header_t *utf8_input_targets =
+      apr_array_make(pool, DEFAULT_ARRAY_SIZE, sizeof(const char *));
+
+  for (; os->ind < os->argc; os->ind++)
+    {
+      /* The apr_getopt targets are still in native encoding. */
+      const char *raw_target = os->argv[os->ind];
+      const char *utf8_target;
+
+      SVN_ERR(svn_utf_cstring_to_utf8(&utf8_target,
+                                      raw_target, pool));
+
+      APR_ARRAY_PUSH(utf8_input_targets, const char *) = utf8_target;
+    }
+
+  return svn_error_trace(
+      args_to_target_array(targets_p, utf8_input_targets, known_targets, ctx,
+                           keep_last_origpath_on_truepath_collision, pool));
+}
+
+svn_error_t *
+svn_client_args_to_target_array_utf8(apr_array_header_t **targets_p,
+                                     apr_getopt_t *os,
+                                     const apr_array_header_t *known_targets,
+                                     svn_client_ctx_t *ctx,
+                                     svn_boolean_t keep_last_origpath_on_truepath_collision,
+                                     apr_pool_t *pool)
+{
+  apr_array_header_t *utf8_input_targets =
+      apr_array_make(pool, DEFAULT_ARRAY_SIZE, sizeof(const char *));
+
+  for (; os->ind < os->argc; os->ind++)
+    {
+      APR_ARRAY_PUSH(utf8_input_targets, const char *) = os->argv[os->ind];
+    }
+
+  return svn_error_trace(
+      args_to_target_array(targets_p, utf8_input_targets, known_targets, ctx,
+                           keep_last_origpath_on_truepath_collision, pool));
+}



Re: svn commit: r1925722 - in /subversion/branches/utf8-cmdline-prototype/subversion: include/svn_client.h libsvn_client/cmdline.c

Posted by Branko Čibej <br...@apache.org>.
On 20. 5. 25 20:31, rinrab@apache.org wrote:
> Author: rinrab
> Date: Tue May 20 18:31:35 2025
> New Revision: 1925722
>
> URL:http://svn.apache.org/viewvc?rev=1925722&view=rev
> Log:
> On the 'utf8-cmdline-prototype' branch: add a new function that retrieves
> target list from args, instead of converting the encoding, assumes that
> args have already been converted.


Once again, why don't you use apr_app_initialize() that does all that 
and more, instead of perpetuating the same problem? As long as this is 
used on Windows from main() instead of wmain(), there's no way it will 
work consistently.


-- Brane

Re: svn commit: r1925722 - in /subversion/branches/utf8-cmdline-prototype/subversion: include/svn_client.h libsvn_client/cmdline.c

Posted by Daniel Sahlberg <da...@gmail.com>.
Den tis 27 maj 2025 kl 09:37 skrev Branko Čibej <br...@apache.org>:

> On 26. 5. 25 23:15, Timofei Zhakov wrote:
>
> On Mon, May 26, 2025 at 10:07 PM Branko Čibej <br...@apache.org> wrote:
>
>> On 26. 5. 25 21:25, Timofei Zhakov wrote:
>>
>> On Mon, May 26, 2025 at 8:29 PM Branko Čibej <br...@apache.org> wrote:
>>
>>> On 26. 5. 25 14:07, Timofei Zhakov wrote:
>>>
>>
>> This is exactly what happened with svn_cmdline__get_cstring_argv(). I
>> added a new function svn_cmdline__get_utf8_argv(), that returns a utf8
>> string instead. Then we can switch the one used in each executable
>> one-by-one.
>>
>>
>> Those are private functions, you can do whatever you want with them, as
>> long as you end up with only one version of each instead of, I dunno, five.
>>
>>
> Those are also transitional functions needed to migrate all usages
> smoothly. Probably it makes sense to drop the old ones then, but it would
> not be a big deal for us in future...
>
>
>
> We have, what, 11 command-line tools? That's 11 one-line changes to make
> the transition to the correct behaviour. It doesn't have to be done all at
> once, but it shouldn't take years, either. Once all of those are fixed, we
> should remove the unused private functions. If we make any new command-line
> tools, we'll just have to get it right the first time. :)
>
>
I believe there is a point in doing stuff incrementally - especially since
this is a branch. There is nothing wrong with making checkpoint commits.

I agree with your overall concern about the naming of the functions - but
why not:
- Keep on as Timofei has started by adding a new _utf8_-named function
(just to keep it clear what it is doing)
- Migrate a few command line tools at a time to make sure we don't run into
anything unexpected
- When everything is done and the old _cstring_ version isn't used anymore,
just remove it from the source
- Removing _utf8_
- Committing each step along the way

This way it is easy to check if/when something goes wrong. Especially since
the GHAs keep testing each commit along the way.

- When everything looks good we can check the diff against trunk and if it
all looks good we can make a single big commit to trunk with the functions
just magically behaving correctly and with appropriate names. Then we can
delete the branch and pretend it never existed.

A very experienced guy once asked for a bit of patience and that things
might not get fixed right this second. I second that.

Cheers,
Daniel

Re: svn commit: r1925722 - in /subversion/branches/utf8-cmdline-prototype/subversion: include/svn_client.h libsvn_client/cmdline.c

Posted by Branko Čibej <br...@apache.org>.
On 26. 5. 25 23:15, Timofei Zhakov wrote:
> On Mon, May 26, 2025 at 10:07 PM Branko Čibej <br...@apache.org> wrote:
>
>     On 26. 5. 25 21:25, Timofei Zhakov wrote:
>>     On Mon, May 26, 2025 at 8:29 PM Branko Čibej <br...@apache.org>
>>     wrote:
>>
>>         On 26. 5. 25 14:07, Timofei Zhakov wrote:
>>
>>
>>     This is exactly what happened with
>>     svn_cmdline__get_cstring_argv(). I added a new function
>>     svn_cmdline__get_utf8_argv(), that returns a utf8 string instead.
>>     Then we can switch the one used in each executable one-by-one.
>
>     Those are private functions, you can do whatever you want with
>     them, as long as you end up with only one version of each instead
>     of, I dunno, five.
>
>
> Those are also transitional functions needed to migrate all usages 
> smoothly. Probably it makes sense to drop the old ones then, but it 
> would not be a big deal for us in future...


We have, what, 11 command-line tools? That's 11 one-line changes to make 
the transition to the correct behaviour. It doesn't have to be done all 
at once, but it shouldn't take years, either. Once all of those are 
fixed, we should remove the unused private functions. If we make any new 
command-line tools, we'll just have to get it right the first time. :)

-- Brane

Re: svn commit: r1925722 - in /subversion/branches/utf8-cmdline-prototype/subversion: include/svn_client.h libsvn_client/cmdline.c

Posted by Timofei Zhakov <ti...@chemodax.net>.
On Mon, May 26, 2025 at 10:07 PM Branko Čibej <br...@apache.org> wrote:

> On 26. 5. 25 21:25, Timofei Zhakov wrote:
>
> On Mon, May 26, 2025 at 8:29 PM Branko Čibej <br...@apache.org> wrote:
>
>> On 26. 5. 25 14:07, Timofei Zhakov wrote:
>>
>> On Tue, May 20, 2025 at 9:29 PM Branko Čibej <br...@apache.org> wrote:
>>
>>> On 20. 5. 25 20:31, rinrab@apache.org wrote:
>>>
>>> Author: rinrab
>>> Date: Tue May 20 18:31:35 2025
>>> New Revision: 1925722
>>>
>>> URL: http://svn.apache.org/viewvc?rev=1925722&view=rev
>>> Log:
>>> On the 'utf8-cmdline-prototype' branch: add a new function that retrieves
>>> target list from args, instead of converting the encoding, assumes that
>>> args have already been converted.
>>>
>>>
>>>
>>> Once again, why don't you use apr_app_initialize() that does all that
>>> and more, instead of perpetuating the same problem? As long as this is used
>>> on Windows from main() instead of wmain(), there's no way it will work
>>> consistently.
>>>
>>
>> First I wanna mention that this exact change is still needed even if we
>> decide to use apr app to host the binary. At this point there is no
>> difference between using apr app and svn's implementation of wmain and main
>> wrappers.
>>
>> Also, I'd like to argue about using apr app in general. The subversion's
>> command line wasn't using this library during the whole history (as far as
>> I know), so I don't think it's correct to start using it now. What would we
>> do if we need some functionality apr app doesn't implement? I mean that
>> would be a quite huge change...
>>
>> In my opinion, apr app is not something fully ready to use in subversion.
>> For example, it doesn't wrap main on unix platforms, while it still can be
>> executed with non-utf8-ized arguments. In wmain() apr_conv_ucs2_to_utf8()
>> just ignores the error, which is bad (on win32).
>>
>> And finally, the code we have right now is pretty good, and it seems
>> possible to support non-ASCII arguments (which I'm working on).
>>
>> I hope this makes some sense.
>>
>>
>>
>> OK, fine, but I still don't see where you're converting the argument list
>> directly from UTF-16-LE to UTF-8. svn_cmdline__win32_get_cstring_argv
>> doesn't do that, and it should.
>>
>> That's the main problem on Windows: that function is converting to the
>> local multibyte character set, then svn_client_args_to_target_array2
>> converts /that/ to UTF-8. That's fine-ish on Unix, where we don't convert
>> twice, but completely brain-dead on Windows. And I freely admit I had a lot
>> do do with that brain-deadedness.
>>
>> This is what should be happening, IMO:
>>
>>    1. deprecate svn_client_args_to_target_array2
>>    2. rename your svn_client_args_to_target_array_utf8
>>    svn_client_args_to_target_array3 and make it explicitly require that the
>>    args are already in UTF-8
>>    3. svn_cmdline__win32_get_cstring_argv must convert to UTF-8
>>    4. svn_cmdline__default_get_cstring_argv must *also* convert to UTF-8
>>
>>
>> You've done steps 1 and 2, 3 should be trivial and 4 will make the result
>> compatible with 3.
>>
>> -- Brane
>>
>
> IMHO, it's much better to follow my way instead, in which I'm adding utf8
> alternatives for those functions, without deprecating old ones, let's call
> them cstring.
>
>
>
> The old ones are wrong, period. Subversion's internal character set has
> been UTF-8 practically from day 1. svn_client_args_to_target_array2 is
> violating that convention. We have conversion functions.
>
>
>
Oh, exactly! You are right and this finally makes sense. I didn't initially
think about that. Let me quickly fix that...


>
> This is exactly what happened with svn_cmdline__get_cstring_argv(). I
> added a new function svn_cmdline__get_utf8_argv(), that returns a utf8
> string instead. Then we can switch the one used in each executable
> one-by-one.
>
>
> Those are private functions, you can do whatever you want with them, as
> long as you end up with only one version of each instead of, I dunno, five.
>
>
Those are also transitional functions needed to migrate all usages
smoothly. Probably it makes sense to drop the old ones then, but it would
not be a big deal for us in future...


>
> The same thing with svn_client_args_to_target_array_utf8, and also this
> difference in names is also helpful to differentiate between cstring and
> utf8 versions much more than version bump. Also the old one has not been
> deprecated, because someone may still want this function to convert
> encoding.
>
>
> Except for the encoding conversion functions, *none* of our API has utf8
> in the name, because that's understood and implied.
> svn_client_args_to_target_array2 is one of the few exceptions that converts
> between encoding without saying so explicitly in the name, and it's been a
> pain to deal with precisely because of that.
>
> Our API requires that strings are in UTF-8. The fewer exceptions we have
> to that, the better.
>
>
Now, I see.


>
> Let's not overcomplicate things if possible.
>
>
> Yes, exactly. Let's not maintain multiple incompatible versions of the
> same function. I completely agree.
>

 Yay!


>
> -- Brane
>


-- 
Timofei Zhakov

Re: svn commit: r1925722 - in /subversion/branches/utf8-cmdline-prototype/subversion: include/svn_client.h libsvn_client/cmdline.c

Posted by Branko Čibej <br...@apache.org>.
On 26. 5. 25 21:25, Timofei Zhakov wrote:
> On Mon, May 26, 2025 at 8:29 PM Branko Čibej <br...@apache.org> wrote:
>
>     On 26. 5. 25 14:07, Timofei Zhakov wrote:
>>     On Tue, May 20, 2025 at 9:29 PM Branko Čibej <br...@apache.org>
>>     wrote:
>>
>>         On 20. 5. 25 20:31, rinrab@apache.org wrote:
>>>         Author: rinrab
>>>         Date: Tue May 20 18:31:35 2025
>>>         New Revision: 1925722
>>>
>>>         URL:http://svn.apache.org/viewvc?rev=1925722&view=rev  <http://svn.apache.org/viewvc?rev=1925722&view=rev>
>>>         Log:
>>>         On the 'utf8-cmdline-prototype' branch: add a new function that retrieves
>>>         target list from args, instead of converting the encoding, assumes that
>>>         args have already been converted.
>>
>>
>>         Once again, why don't you use apr_app_initialize() that does
>>         all that and more, instead of perpetuating the same problem?
>>         As long as this is used on Windows from main() instead of
>>         wmain(), there's no way it will work consistently.
>>
>>
>>     First I wanna mention that this exact change is still needed even
>>     if we decide to use apr app to host the binary. At this point
>>     there is no difference between using apr app and svn's
>>     implementation of wmain and main wrappers.
>>
>>     Also, I'd like to argue about using apr app in general. The
>>     subversion's command line wasn't using this library during the
>>     whole history (as far as I know), so I don't think it's correct
>>     to start using it now. What would we do if we need some
>>     functionality apr app doesn't implement? I mean that would be a
>>     quite huge change...
>>
>>     In my opinion, apr app is not something fully ready to use in
>>     subversion. For example, it doesn't wrap main on unix platforms,
>>     while it still can be executed with non-utf8-ized arguments. In
>>     wmain() apr_conv_ucs2_to_utf8() just ignores the error, which is
>>     bad (on win32).
>>
>>     And finally, the code we have right now is pretty good, and it
>>     seems possible to support non-ASCII arguments (which I'm working
>>     on).
>>
>>     I hope this makes some sense.
>
>
>     OK, fine, but I still don't see where you're converting the
>     argument list directly from UTF-16-LE to UTF-8.
>     svn_cmdline__win32_get_cstring_argv doesn't do that, and it should.
>
>     That's the main problem on Windows: that function is converting to
>     the local multibyte character set, then
>     svn_client_args_to_target_array2 converts /that/ to UTF-8. That's
>     fine-ish on Unix, where we don't convert twice, but completely
>     brain-dead on Windows. And I freely admit I had a lot do do with
>     that brain-deadedness.
>
>     This is what should be happening, IMO:
>
>      1. deprecate svn_client_args_to_target_array2
>      2. rename your svn_client_args_to_target_array_utf8
>         svn_client_args_to_target_array3 and make it explicitly
>         require that the args are already in UTF-8
>      3. svn_cmdline__win32_get_cstring_argv must convert to UTF-8
>      4. svn_cmdline__default_get_cstring_argv must *also* convert to UTF-8
>
>
>     You've done steps 1 and 2, 3 should be trivial and 4 will make the
>     result compatible with 3.
>
>     -- Brane
>
>
> IMHO, it's much better to follow my way instead, in which I'm adding 
> utf8 alternatives for those functions, without deprecating old ones, 
> let's call them cstring.


The old ones are wrong, period. Subversion's internal character set has 
been UTF-8 practically from day 1. svn_client_args_to_target_array2 is 
violating that convention. We have conversion functions.


>
> This is exactly what happened with svn_cmdline__get_cstring_argv(). I 
> added a new function svn_cmdline__get_utf8_argv(), that returns a utf8 
> string instead. Then we can switch the one used in each executable 
> one-by-one.

Those are private functions, you can do whatever you want with them, as 
long as you end up with only one version of each instead of, I dunno, five.

>
> The same thing with svn_client_args_to_target_array_utf8, and also 
> this difference in names is also helpful to differentiate between 
> cstring and utf8 versions much more than version bump. Also the old 
> one has not been deprecated, because someone may still want this 
> function to convert encoding.

Except for the encoding conversion functions, /none/ of our API has utf8 
in the name, because that's understood and implied. 
svn_client_args_to_target_array2 is one of the few exceptions that 
converts between encoding without saying so explicitly in the name, and 
it's been a pain to deal with precisely because of that.

Our API requires that strings are in UTF-8. The fewer exceptions we have 
to that, the better.


> Let's not overcomplicate things if possible.

Yes, exactly. Let's not maintain multiple incompatible versions of the 
same function. I completely agree.

-- Brane

Re: svn commit: r1925722 - in /subversion/branches/utf8-cmdline-prototype/subversion: include/svn_client.h libsvn_client/cmdline.c

Posted by Timofei Zhakov <ti...@chemodax.net>.
On Mon, May 26, 2025 at 8:29 PM Branko Čibej <br...@apache.org> wrote:

> On 26. 5. 25 14:07, Timofei Zhakov wrote:
>
> On Tue, May 20, 2025 at 9:29 PM Branko Čibej <br...@apache.org> wrote:
>
>> On 20. 5. 25 20:31, rinrab@apache.org wrote:
>>
>> Author: rinrab
>> Date: Tue May 20 18:31:35 2025
>> New Revision: 1925722
>>
>> URL: http://svn.apache.org/viewvc?rev=1925722&view=rev
>> Log:
>> On the 'utf8-cmdline-prototype' branch: add a new function that retrieves
>> target list from args, instead of converting the encoding, assumes that
>> args have already been converted.
>>
>>
>>
>> Once again, why don't you use apr_app_initialize() that does all that and
>> more, instead of perpetuating the same problem? As long as this is used on
>> Windows from main() instead of wmain(), there's no way it will work
>> consistently.
>>
>
> First I wanna mention that this exact change is still needed even if we
> decide to use apr app to host the binary. At this point there is no
> difference between using apr app and svn's implementation of wmain and main
> wrappers.
>
> Also, I'd like to argue about using apr app in general. The subversion's
> command line wasn't using this library during the whole history (as far as
> I know), so I don't think it's correct to start using it now. What would we
> do if we need some functionality apr app doesn't implement? I mean that
> would be a quite huge change...
>
> In my opinion, apr app is not something fully ready to use in subversion.
> For example, it doesn't wrap main on unix platforms, while it still can be
> executed with non-utf8-ized arguments. In wmain() apr_conv_ucs2_to_utf8()
> just ignores the error, which is bad (on win32).
>
> And finally, the code we have right now is pretty good, and it seems
> possible to support non-ASCII arguments (which I'm working on).
>
> I hope this makes some sense.
>
>
>
> OK, fine, but I still don't see where you're converting the argument list
> directly from UTF-16-LE to UTF-8. svn_cmdline__win32_get_cstring_argv
> doesn't do that, and it should.
>
> That's the main problem on Windows: that function is converting to the
> local multibyte character set, then svn_client_args_to_target_array2
> converts /that/ to UTF-8. That's fine-ish on Unix, where we don't convert
> twice, but completely brain-dead on Windows. And I freely admit I had a lot
> do do with that brain-deadedness.
>
> This is what should be happening, IMO:
>
>    1. deprecate svn_client_args_to_target_array2
>    2. rename your svn_client_args_to_target_array_utf8
>    svn_client_args_to_target_array3 and make it explicitly require that the
>    args are already in UTF-8
>    3. svn_cmdline__win32_get_cstring_argv must convert to UTF-8
>    4. svn_cmdline__default_get_cstring_argv must *also* convert to UTF-8
>
>
> You've done steps 1 and 2, 3 should be trivial and 4 will make the result
> compatible with 3.
>
> -- Brane
>

IMHO, it's much better to follow my way instead, in which I'm adding utf8
alternatives for those functions, without deprecating old ones, let's call
them cstring.

This is exactly what happened with svn_cmdline__get_cstring_argv(). I added
a new function svn_cmdline__get_utf8_argv(), that returns a utf8 string
instead. Then we can switch the one used in each executable one-by-one.

This approach makes it so we can smoothly switch all programs to use utf8
args, which simplifies the process of hacking this whole thing a lot.

The same thing with svn_client_args_to_target_array_utf8, and also this
difference in names is also helpful to differentiate between cstring and
utf8 versions much more than version bump. Also the old one has not been
deprecated, because someone may still want this function to convert
encoding.

Let's not overcomplicate things if possible.

-- 
Timofei Zhakov

Re: svn commit: r1925722 - in /subversion/branches/utf8-cmdline-prototype/subversion: include/svn_client.h libsvn_client/cmdline.c

Posted by Timofei Zhakov <ti...@chemodax.net>.
On Mon, May 26, 2025 at 8:29 PM Branko Čibej <br...@apache.org> wrote:

> On 26. 5. 25 14:07, Timofei Zhakov wrote:
>
> On Tue, May 20, 2025 at 9:29 PM Branko Čibej <br...@apache.org> wrote:
>
>> On 20. 5. 25 20:31, rinrab@apache.org wrote:
>>
>> Author: rinrab
>> Date: Tue May 20 18:31:35 2025
>> New Revision: 1925722
>>
>> URL: http://svn.apache.org/viewvc?rev=1925722&view=rev
>> Log:
>> On the 'utf8-cmdline-prototype' branch: add a new function that retrieves
>> target list from args, instead of converting the encoding, assumes that
>> args have already been converted.
>>
>>
>>
>> Once again, why don't you use apr_app_initialize() that does all that and
>> more, instead of perpetuating the same problem? As long as this is used on
>> Windows from main() instead of wmain(), there's no way it will work
>> consistently.
>>
>
> First I wanna mention that this exact change is still needed even if we
> decide to use apr app to host the binary. At this point there is no
> difference between using apr app and svn's implementation of wmain and main
> wrappers.
>
> Also, I'd like to argue about using apr app in general. The subversion's
> command line wasn't using this library during the whole history (as far as
> I know), so I don't think it's correct to start using it now. What would we
> do if we need some functionality apr app doesn't implement? I mean that
> would be a quite huge change...
>
> In my opinion, apr app is not something fully ready to use in subversion.
> For example, it doesn't wrap main on unix platforms, while it still can be
> executed with non-utf8-ized arguments. In wmain() apr_conv_ucs2_to_utf8()
> just ignores the error, which is bad (on win32).
>
> And finally, the code we have right now is pretty good, and it seems
> possible to support non-ASCII arguments (which I'm working on).
>
> I hope this makes some sense.
>
>
>
> OK, fine, but I still don't see where you're converting the argument list
> directly from UTF-16-LE to UTF-8. svn_cmdline__win32_get_cstring_argv
> doesn't do that, and it should.
>
> That's the main problem on Windows: that function is converting to the
> local multibyte character set, then svn_client_args_to_target_array2
> converts /that/ to UTF-8. That's fine-ish on Unix, where we don't convert
> twice, but completely brain-dead on Windows. And I freely admit I had a lot
> do do with that brain-deadedness.
>
> This is what should be happening, IMO:
>
>    1. deprecate svn_client_args_to_target_array2
>    2. rename your svn_client_args_to_target_array_utf8
>    svn_client_args_to_target_array3 and make it explicitly require that the
>    args are already in UTF-8
>    3. svn_cmdline__win32_get_cstring_argv must convert to UTF-8
>    4. svn_cmdline__default_get_cstring_argv must *also* convert to UTF-8
>
>
> You've done steps 1 and 2, 3 should be trivial and 4 will make the result
> compatible with 3.
>
> -- Brane
>

IMHO, it's much better to follow my way instead, in which I'm adding utf8
alternatives for those functions, without deprecating old ones, let's call
them cstring.

This is exactly what happened with svn_cmdline__get_cstring_argv(). I added
a new function svn_cmdline__get_utf8_argv(), that returns a utf8 string
instead. Then we can switch the one used in each executable one-by-one.

This approach makes it so we can smoothly switch all programs to use utf8
args, which simplifies the process of hacking this whole thing a lot.

The same thing with svn_client_args_to_target_array_utf8, and also this
difference in names is also helpful to differentiate between cstring and
utf8 versions much more than version bump. Also the old one has not been
deprecated, because someone may still want this function to convert
encoding.

Let's not overcomplicate things if possible.

-- 
Timofei Zhakov

Re: svn commit: r1925722 - in /subversion/branches/utf8-cmdline-prototype/subversion: include/svn_client.h libsvn_client/cmdline.c

Posted by Branko Čibej <br...@apache.org>.
On 26. 5. 25 14:07, Timofei Zhakov wrote:
> On Tue, May 20, 2025 at 9:29 PM Branko Čibej <br...@apache.org> wrote:
>
>     On 20. 5. 25 20:31, rinrab@apache.org wrote:
>>     Author: rinrab
>>     Date: Tue May 20 18:31:35 2025
>>     New Revision: 1925722
>>
>>     URL:http://svn.apache.org/viewvc?rev=1925722&view=rev  <http://svn.apache.org/viewvc?rev=1925722&view=rev>
>>     Log:
>>     On the 'utf8-cmdline-prototype' branch: add a new function that retrieves
>>     target list from args, instead of converting the encoding, assumes that
>>     args have already been converted.
>
>
>     Once again, why don't you use apr_app_initialize() that does all
>     that and more, instead of perpetuating the same problem? As long
>     as this is used on Windows from main() instead of wmain(), there's
>     no way it will work consistently.
>
>
> First I wanna mention that this exact change is still needed even if 
> we decide to use apr app to host the binary. At this point there is no 
> difference between using apr app and svn's implementation of wmain and 
> main wrappers.
>
> Also, I'd like to argue about using apr app in general. The 
> subversion's command line wasn't using this library during the whole 
> history (as far as I know), so I don't think it's correct to start 
> using it now. What would we do if we need some functionality apr app 
> doesn't implement? I mean that would be a quite huge change...
>
> In my opinion, apr app is not something fully ready to use in 
> subversion. For example, it doesn't wrap main on unix platforms, while 
> it still can be executed with non-utf8-ized arguments. In wmain() 
> apr_conv_ucs2_to_utf8() just ignores the error, which is bad (on win32).
>
> And finally, the code we have right now is pretty good, and it seems 
> possible to support non-ASCII arguments (which I'm working on).
>
> I hope this makes some sense.


OK, fine, but I still don't see where you're converting the argument 
list directly from UTF-16-LE to UTF-8. 
svn_cmdline__win32_get_cstring_argv doesn't do that, and it should.

That's the main problem on Windows: that function is converting to the 
local multibyte character set, then svn_client_args_to_target_array2 
converts /that/ to UTF-8. That's fine-ish on Unix, where we don't 
convert twice, but completely brain-dead on Windows. And I freely admit 
I had a lot do do with that brain-deadedness.

This is what should be happening, IMO:

 1. deprecate svn_client_args_to_target_array2
 2. rename your svn_client_args_to_target_array_utf8
    svn_client_args_to_target_array3 and make it explicitly require that
    the args are already in UTF-8
 3. svn_cmdline__win32_get_cstring_argv must convert to UTF-8
 4. svn_cmdline__default_get_cstring_argv must *also* convert to UTF-8


You've done steps 1 and 2, 3 should be trivial and 4 will make the 
result compatible with 3.

-- Brane

Re: svn commit: r1925722 - in /subversion/branches/utf8-cmdline-prototype/subversion: include/svn_client.h libsvn_client/cmdline.c

Posted by Timofei Zhakov <ti...@chemodax.net>.
On Tue, May 20, 2025 at 9:29 PM Branko Čibej <br...@apache.org> wrote:

> On 20. 5. 25 20:31, rinrab@apache.org wrote:
>
> Author: rinrab
> Date: Tue May 20 18:31:35 2025
> New Revision: 1925722
>
> URL: http://svn.apache.org/viewvc?rev=1925722&view=rev
> Log:
> On the 'utf8-cmdline-prototype' branch: add a new function that retrieves
> target list from args, instead of converting the encoding, assumes that
> args have already been converted.
>
>
>
> Once again, why don't you use apr_app_initialize() that does all that and
> more, instead of perpetuating the same problem? As long as this is used on
> Windows from main() instead of wmain(), there's no way it will work
> consistently.
>

First I wanna mention that this exact change is still needed even if we
decide to use apr app to host the binary. At this point there is no
difference between using apr app and svn's implementation of wmain and main
wrappers.

Also, I'd like to argue about using apr app in general. The subversion's
command line wasn't using this library during the whole history (as far as
I know), so I don't think it's correct to start using it now. What would we
do if we need some functionality apr app doesn't implement? I mean that
would be a quite huge change...

In my opinion, apr app is not something fully ready to use in subversion.
For example, it doesn't wrap main on unix platforms, while it still can be
executed with non-utf8-ized arguments. In wmain() apr_conv_ucs2_to_utf8()
just ignores the error, which is bad (on win32).

And finally, the code we have right now is pretty good, and it seems
possible to support non-ASCII arguments (which I'm working on).

I hope this makes some sense.

-- 
Timofei Zhakov

Re: svn commit: r1925722 - in /subversion/branches/utf8-cmdline-prototype/subversion: include/svn_client.h libsvn_client/cmdline.c

Posted by Timofei Zhakov <ti...@chemodax.net>.
On Tue, May 20, 2025 at 9:29 PM Branko Čibej <br...@apache.org> wrote:

> On 20. 5. 25 20:31, rinrab@apache.org wrote:
>
> Author: rinrab
> Date: Tue May 20 18:31:35 2025
> New Revision: 1925722
>
> URL: http://svn.apache.org/viewvc?rev=1925722&view=rev
> Log:
> On the 'utf8-cmdline-prototype' branch: add a new function that retrieves
> target list from args, instead of converting the encoding, assumes that
> args have already been converted.
>
>
>
> Once again, why don't you use apr_app_initialize() that does all that and
> more, instead of perpetuating the same problem? As long as this is used on
> Windows from main() instead of wmain(), there's no way it will work
> consistently.
>

First I wanna mention that this exact change is still needed even if we
decide to use apr app to host the binary. At this point there is no
difference between using apr app and svn's implementation of wmain and main
wrappers.

Also, I'd like to argue about using apr app in general. The subversion's
command line wasn't using this library during the whole history (as far as
I know), so I don't think it's correct to start using it now. What would we
do if we need some functionality apr app doesn't implement? I mean that
would be a quite huge change...

In my opinion, apr app is not something fully ready to use in subversion.
For example, it doesn't wrap main on unix platforms, while it still can be
executed with non-utf8-ized arguments. In wmain() apr_conv_ucs2_to_utf8()
just ignores the error, which is bad (on win32).

And finally, the code we have right now is pretty good, and it seems
possible to support non-ASCII arguments (which I'm working on).

I hope this makes some sense.

-- 
Timofei Zhakov

Re: svn commit: r1925722 - in /subversion/branches/utf8-cmdline-prototype/subversion: include/svn_client.h libsvn_client/cmdline.c

Posted by Branko Čibej <br...@apache.org>.
On 20. 5. 25 20:31, rinrab@apache.org wrote:
> Author: rinrab
> Date: Tue May 20 18:31:35 2025
> New Revision: 1925722
>
> URL:http://svn.apache.org/viewvc?rev=1925722&view=rev
> Log:
> On the 'utf8-cmdline-prototype' branch: add a new function that retrieves
> target list from args, instead of converting the encoding, assumes that
> args have already been converted.


Once again, why don't you use apr_app_initialize() that does all that 
and more, instead of perpetuating the same problem? As long as this is 
used on Windows from main() instead of wmain(), there's no way it will 
work consistently.


-- Brane