You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Mark Phippard <ma...@gmail.com> on 2007/05/30 16:29:00 UTC
SVN Win32 Developers -- need some help
I think we really need to address the problems of APR ICONV in our
next release. Please see this thread for some references:
http://subversion.tigris.org/servlets/BrowseList?list=dev&by=thread&from=582337
This library has always been a problem, when there are multiple
clients installed and they used different compilers. But now with a
mixture of APR ICONV 0.9.x and 1.1.x out there this problem is going
to escalate. It crashes Subversion, which means it can also crash
clients using Subversion.
Stefan King did the work of creating an initial patch that replaces
the dependency with Win32 API's. I think we just need some consensus
that this is what we should do, and then get a developer to do it.
I think this is important and we are going to have problems if we do
not address this for 1.5.
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: SVN Win32 Developers -- need some help
Posted by Mark Phippard <ma...@gmail.com>.
On 6/2/07, D.J. Heap <dj...@gmail.com> wrote:
> On 5/30/07, Mark Phippard <ma...@gmail.com> wrote:
> > I think we really need to address the problems of APR ICONV in our
> > next release. Please see this thread for some references:
> >
> > http://subversion.tigris.org/servlets/BrowseList?list=dev&by=thread&from=582337
> >
> > This library has always been a problem, when there are multiple
> > clients installed and they used different compilers. But now with a
> > mixture of APR ICONV 0.9.x and 1.1.x out there this problem is going
> > to escalate. It crashes Subversion, which means it can also crash
> > clients using Subversion.
> >
> > Stefan King did the work of creating an initial patch that replaces
> > the dependency with Win32 API's. I think we just need some consensus
> > that this is what we should do, and then get a developer to do it.
> >
> > I think this is important and we are going to have problems if we do
> > not address this for 1.5.
> >
>
>
> I would love to see this go in as well. Using an unpatched apr-iconv
> on Windows is problematic at best, due to the global environment
> variable (as has been explained already).
>
> Unfortunately, I don't know much about multi-language issues and so
> I'm not sure I can help much. But I am very much for using Stefan's
> implementation if it works well, and I will help build and test where
> I can.
Great. It sounds like there is general consensus that this would be a
good improvement, assuming we can make the code work.
Have you seen the patches Stefan previously submitted?
http://subversion.tigris.org/servlets/ReadMsg?list=dev&msgNo=106781
Perhaps that could be a starting point. Not sure if the patch still
applies cleanly.
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: SVN Win32 Developers -- need some help
Posted by "D.J. Heap" <dj...@gmail.com>.
On 5/30/07, Mark Phippard <ma...@gmail.com> wrote:
> I think we really need to address the problems of APR ICONV in our
> next release. Please see this thread for some references:
>
> http://subversion.tigris.org/servlets/BrowseList?list=dev&by=thread&from=582337
>
> This library has always been a problem, when there are multiple
> clients installed and they used different compilers. But now with a
> mixture of APR ICONV 0.9.x and 1.1.x out there this problem is going
> to escalate. It crashes Subversion, which means it can also crash
> clients using Subversion.
>
> Stefan King did the work of creating an initial patch that replaces
> the dependency with Win32 API's. I think we just need some consensus
> that this is what we should do, and then get a developer to do it.
>
> I think this is important and we are going to have problems if we do
> not address this for 1.5.
>
I would love to see this go in as well. Using an unpatched apr-iconv
on Windows is problematic at best, due to the global environment
variable (as has been explained already).
Unfortunately, I don't know much about multi-language issues and so
I'm not sure I can help much. But I am very much for using Stefan's
implementation if it works well, and I will help build and test where
I can.
DJ
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: SVN Win32 Developers -- need some help
Posted by Mark Phippard <ma...@gmail.com>.
On 6/3/07, Martin Hauner <ma...@gmx.net> wrote:
> Hi,
>
> Mark Phippard wrote:
> > I think we really need to address the problems of APR ICONV in our
> > next release. Please see this thread for some references:
> >
> > http://subversion.tigris.org/servlets/BrowseList?list=dev&by=thread&from=582337
> >
> >
> > This library has always been a problem, when there are multiple
> > clients installed and they used different compilers. But now with a
> > mixture of APR ICONV 0.9.x and 1.1.x out there this problem is going
> > to escalate. It crashes Subversion, which means it can also crash
> > clients using Subversion.
>
> I'm simply applying a small patch (already sent to the list, this is a
> recurring issue ;) that makes apr iconv first look into a subfolder iconv
> that is in the same dir as the libapriconv.dll.
>
> I'm doing this for years now and I have never seen any problems with it
> having svn, tsvn and subcommander on my system.
>
> I think Stefan used a similar patch in the past. I don't know why he
> changed it.
>
> Since subcommander/submerge is using the same iconv based code on
> Unix/Windows i'm not in favour of removing apr-iconv on Windows.
I look at it differently. On every platform except Windows, we are
using the iconv library from the OS. On Windows we require apr-iconv,
and the 200+ extra libraries that drags along with it. There is also
no precedent for Subversion applying a patch to a third party library
as part of its build process. Sure you can do it for your product,
but Subversion is not likely going to do it for the binaries it
releases. There are already many reports on the users@ list of
crashes because of the incompatabilities.
I think that using the translation services provided by Windows makes
us more consistent in our approach and also makes our app easier to
distribute as well as more robust in that it will not be as prone to
crashes.
The patch you talk about has been around for years and for whatever
reason the apr-iconv maintainers have chosen not to apply it. I think
it is time to look for an alternative.
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: SVN Win32 Developers -- need some help
Posted by Martin Hauner <ma...@gmx.net>.
Hi,
Mark Phippard wrote:
> I think we really need to address the problems of APR ICONV in our
> next release. Please see this thread for some references:
>
> http://subversion.tigris.org/servlets/BrowseList?list=dev&by=thread&from=582337
>
>
> This library has always been a problem, when there are multiple
> clients installed and they used different compilers. But now with a
> mixture of APR ICONV 0.9.x and 1.1.x out there this problem is going
> to escalate. It crashes Subversion, which means it can also crash
> clients using Subversion.
I'm simply applying a small patch (already sent to the list, this is a
recurring issue ;) that makes apr iconv first look into a subfolder iconv
that is in the same dir as the libapriconv.dll.
I'm doing this for years now and I have never seen any problems with it
having svn, tsvn and subcommander on my system.
I think Stefan used a similar patch in the past. I don't know why he
changed it.
Since subcommander/submerge is using the same iconv based code on
Unix/Windows i'm not in favour of removing apr-iconv on Windows.
--
Martin
Subcommander 1.2.2 - http://subcommander.tigris.org
a cross platform Win32/Unix/MacOSX subversion GUI client & diff/merge tool.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: SVN Win32 Developers -- need some help
Posted by "D.J. Heap" <dj...@gmail.com>.
On 6/30/07, Mark Phippard <ma...@gmail.com> wrote:
[snip]
>
> Does the test suite have any tests that exercise this?
I don't think so.
>
> Ivan had said he was going to look into it, but I do not think he has.
> Ideally, I think someone should put together the patch that they
> think works completely and just commit it. Then we can bang on it and
> see if there are issues.
Ok, I can at least do a build and get some binaries out there for
people to test out.
DJ
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: SVN Win32 Developers -- need some help
Posted by Mark Phippard <ma...@gmail.com>.
On 6/30/07, D.J. Heap <dj...@gmail.com> wrote:
> On 6/5/07, eg <eg...@gmail.com> wrote:
> > Mark Phippard wrote:
> >
> > >
> > > Even in his objection Brane said "if this is true". All I (and
> > > Stefan) am saying is we need a more specific objection. If there are
> > > real problems that we cannot resolve, then I am sure we would all be
> > > against this. But we know there are problems with apr_iconv too, so
> > > let's just understand what the problems are.
> > >
> >
> > In an attempt to help here, I have done a bit of hunting around to see
> > if I could substantiate any publicly reported issues. I have mentioned
> > the few that I have found in other posts. Stefan has already responded
> > that these should not be issues for Subversion.
> >
> > The only other thing I could find for utf8 conversion, relates to
> > handling of "bad inputs" related to lone surrogate halves when passing
> > them to WideCharToMultiByte() on older versions of Windows (alluded to
> > in blog comments from:
> > http://blogs.msdn.com/michkap/archive/2006/01/20/515238.aspx).
> > Apparently this is tightened up now (on Vista?).
> >
> > I suspect this is also not an issue for this usage in any case, but
> > someone else more familiar with svn usage should confirm that.
> >
> > If there are any issues related to using Win32
> > MultiByteToWideChar()/WideCharToMultiByte() in place of iconv() under
> > Windows, I cannot find them. Older versions of Windows (95/98/Me) will
> > need to install and use the MSLU (Microsoft Layer for unicode).
>
>
> So, I have not been following along very carefully, but what needs to
> happen now? Do a build and have people using different languages test
> it?
>
> I can apply the patch and build the binary zips if other people have
> the means to test them out a bit.
>
> Or is there something else that needs to be done?
Does the test suite have any tests that exercise this?
Ivan had said he was going to look into it, but I do not think he has.
Ideally, I think someone should put together the patch that they
think works completely and just commit it. Then we can bang on it and
see if there are issues.
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: SVN Win32 Developers -- need some help
Posted by "D.J. Heap" <dj...@gmail.com>.
On 6/5/07, eg <eg...@gmail.com> wrote:
> Mark Phippard wrote:
>
> >
> > Even in his objection Brane said "if this is true". All I (and
> > Stefan) am saying is we need a more specific objection. If there are
> > real problems that we cannot resolve, then I am sure we would all be
> > against this. But we know there are problems with apr_iconv too, so
> > let's just understand what the problems are.
> >
>
> In an attempt to help here, I have done a bit of hunting around to see
> if I could substantiate any publicly reported issues. I have mentioned
> the few that I have found in other posts. Stefan has already responded
> that these should not be issues for Subversion.
>
> The only other thing I could find for utf8 conversion, relates to
> handling of "bad inputs" related to lone surrogate halves when passing
> them to WideCharToMultiByte() on older versions of Windows (alluded to
> in blog comments from:
> http://blogs.msdn.com/michkap/archive/2006/01/20/515238.aspx).
> Apparently this is tightened up now (on Vista?).
>
> I suspect this is also not an issue for this usage in any case, but
> someone else more familiar with svn usage should confirm that.
>
> If there are any issues related to using Win32
> MultiByteToWideChar()/WideCharToMultiByte() in place of iconv() under
> Windows, I cannot find them. Older versions of Windows (95/98/Me) will
> need to install and use the MSLU (Microsoft Layer for unicode).
So, I have not been following along very carefully, but what needs to
happen now? Do a build and have people using different languages test
it?
I can apply the patch and build the binary zips if other people have
the means to test them out a bit.
Or is there something else that needs to be done?
DJ
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: SVN Win32 Developers -- need some help
Posted by eg <eg...@gmail.com>.
Mark Phippard wrote:
>
> Even in his objection Brane said "if this is true". All I (and
> Stefan) am saying is we need a more specific objection. If there are
> real problems that we cannot resolve, then I am sure we would all be
> against this. But we know there are problems with apr_iconv too, so
> let's just understand what the problems are.
>
In an attempt to help here, I have done a bit of hunting around to see
if I could substantiate any publicly reported issues. I have mentioned
the few that I have found in other posts. Stefan has already responded
that these should not be issues for Subversion.
The only other thing I could find for utf8 conversion, relates to
handling of "bad inputs" related to lone surrogate halves when passing
them to WideCharToMultiByte() on older versions of Windows (alluded to
in blog comments from:
http://blogs.msdn.com/michkap/archive/2006/01/20/515238.aspx).
Apparently this is tightened up now (on Vista?).
I suspect this is also not an issue for this usage in any case, but
someone else more familiar with svn usage should confirm that.
If there are any issues related to using Win32
MultiByteToWideChar()/WideCharToMultiByte() in place of iconv() under
Windows, I cannot find them. Older versions of Windows (95/98/Me) will
need to install and use the MSLU (Microsoft Layer for unicode).
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: SVN Win32 Developers -- need some help
Posted by Mark Phippard <ma...@gmail.com>.
On 6/4/07, Ivan Zhakov <ch...@gmail.com> wrote:
> On 6/5/07, Mark Phippard <ma...@gmail.com> wrote:
> > On 6/4/07, Ivan Zhakov <ch...@gmail.com> wrote:
> > > On 5/30/07, Mark Phippard <ma...@gmail.com> wrote:
> > > I totally agree with you that Subversion should able use Win32 api
> > > instead iconv for encoding conversion on Windows. Maintaining iconv on
> > > windows is hell now. But as I remember last discussion on this topic
> > > blocked by Branko (http://svn.haxx.se/dev/archive-2005-10/0683.shtml):
> > > >> Anyway... before proceeding any further with this patch, I'd like to see
> > > >> some definite answers about the recent report that Windows translation
> > > >> functions aren't reliable for all encodings. If that's true, repllacing
> > > >> apr_iconv would be taking a step backwards (and would therefore generate
> > > >> a -1 from me).
> >
> > I think to be fair here, someone needs to raise a more specific
> > objection, such as pointing to an actual flaw not a hypothetical one.
> > I could turn this around and point to dozens of known crashes of
> > Windows because we use apr_iconv.
> It was not my objection :) I'm just pointing to Brane objection and his -1.
> Personally I don't see problems to use win32 api, even it doesn't
> support some encoding. User will get problems with these encodings on
> OS level anyway.
Even in his objection Brane said "if this is true". All I (and
Stefan) am saying is we need a more specific objection. If there are
real problems that we cannot resolve, then I am sure we would all be
against this. But we know there are problems with apr_iconv too, so
let's just understand what the problems are.
> > > I consider that right way is to teach apr-iconv use win32 api instead
> > > if doing this in Subversion itself. Also in apr-iconv we can hardcode
> > > names of encodings supported by Windows (as far I know Windows
> > > supports limited set of encodings)
> > >
> > > Also I had some technical comments on Stefan's patch. But I'm ready to
> > > solve it if we get some decision in concept.
> >
> > There is so little code here, why involve apr_iconv? I think we
> > should just dump it.
> >
> I consider that OS abstraction is apr_iconv responsibility. I don't
> like idea to have list of hardcoded encoding names in Subversion code.
> We need to list them to support --encoding option. But I could change
> my mind.
>
> So we have to argue Branko and choose location of this code: in
> Subversion or in apr-iconv.
My knowledge in this area always bounces back and forth. I thought we
were not using apr_iconv on other platforms. Is the issue that we do
use it, but if "iconv" is available that gets used instead?
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: SVN Win32 Developers -- need some help
Posted by Ivan Zhakov <ch...@gmail.com>.
On 6/5/07, Mark Phippard <ma...@gmail.com> wrote:
> On 6/4/07, Ivan Zhakov <ch...@gmail.com> wrote:
> > On 5/30/07, Mark Phippard <ma...@gmail.com> wrote:
> > I totally agree with you that Subversion should able use Win32 api
> > instead iconv for encoding conversion on Windows. Maintaining iconv on
> > windows is hell now. But as I remember last discussion on this topic
> > blocked by Branko (http://svn.haxx.se/dev/archive-2005-10/0683.shtml):
> > >> Anyway... before proceeding any further with this patch, I'd like to see
> > >> some definite answers about the recent report that Windows translation
> > >> functions aren't reliable for all encodings. If that's true, repllacing
> > >> apr_iconv would be taking a step backwards (and would therefore generate
> > >> a -1 from me).
>
> I think to be fair here, someone needs to raise a more specific
> objection, such as pointing to an actual flaw not a hypothetical one.
> I could turn this around and point to dozens of known crashes of
> Windows because we use apr_iconv.
It was not my objection :) I'm just pointing to Brane objection and his -1.
Personally I don't see problems to use win32 api, even it doesn't
support some encoding. User will get problems with these encodings on
OS level anyway.
>
> > I consider that right way is to teach apr-iconv use win32 api instead
> > if doing this in Subversion itself. Also in apr-iconv we can hardcode
> > names of encodings supported by Windows (as far I know Windows
> > supports limited set of encodings)
> >
> > Also I had some technical comments on Stefan's patch. But I'm ready to
> > solve it if we get some decision in concept.
>
> There is so little code here, why involve apr_iconv? I think we
> should just dump it.
>
I consider that OS abstraction is apr_iconv responsibility. I don't
like idea to have list of hardcoded encoding names in Subversion code.
We need to list them to support --encoding option. But I could change
my mind.
So we have to argue Branko and choose location of this code: in
Subversion or in apr-iconv.
--
Ivan Zhakov
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: SVN Win32 Developers -- need some help
Posted by Mark Phippard <ma...@gmail.com>.
On 6/4/07, Ivan Zhakov <ch...@gmail.com> wrote:
> On 5/30/07, Mark Phippard <ma...@gmail.com> wrote:
> I totally agree with you that Subversion should able use Win32 api
> instead iconv for encoding conversion on Windows. Maintaining iconv on
> windows is hell now. But as I remember last discussion on this topic
> blocked by Branko (http://svn.haxx.se/dev/archive-2005-10/0683.shtml):
> >> Anyway... before proceeding any further with this patch, I'd like to see
> >> some definite answers about the recent report that Windows translation
> >> functions aren't reliable for all encodings. If that's true, repllacing
> >> apr_iconv would be taking a step backwards (and would therefore generate
> >> a -1 from me).
I think to be fair here, someone needs to raise a more specific
objection, such as pointing to an actual flaw not a hypothetical one.
I could turn this around and point to dozens of known crashes of
Windows because we use apr_iconv.
> I consider that right way is to teach apr-iconv use win32 api instead
> if doing this in Subversion itself. Also in apr-iconv we can hardcode
> names of encodings supported by Windows (as far I know Windows
> supports limited set of encodings)
>
> Also I had some technical comments on Stefan's patch. But I'm ready to
> solve it if we get some decision in concept.
There is so little code here, why involve apr_iconv? I think we
should just dump it.
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: SVN Win32 Developers -- need some help
Posted by Ivan Zhakov <ch...@gmail.com>.
On 5/30/07, Mark Phippard <ma...@gmail.com> wrote:
> I think we really need to address the problems of APR ICONV in our
> next release. Please see this thread for some references:
>
> http://subversion.tigris.org/servlets/BrowseList?list=dev&by=thread&from=582337
>
> This library has always been a problem, when there are multiple
> clients installed and they used different compilers. But now with a
> mixture of APR ICONV 0.9.x and 1.1.x out there this problem is going
> to escalate. It crashes Subversion, which means it can also crash
> clients using Subversion.
>
> Stefan King did the work of creating an initial patch that replaces
> the dependency with Win32 API's. I think we just need some consensus
> that this is what we should do, and then get a developer to do it.
>
> I think this is important and we are going to have problems if we do
> not address this for 1.5.
>
Hi Mark,
I totally agree with you that Subversion should able use Win32 api
instead iconv for encoding conversion on Windows. Maintaining iconv on
windows is hell now. But as I remember last discussion on this topic
blocked by Branko (http://svn.haxx.se/dev/archive-2005-10/0683.shtml):
>> Anyway... before proceeding any further with this patch, I'd like to see
>> some definite answers about the recent report that Windows translation
>> functions aren't reliable for all encodings. If that's true, repllacing
>> apr_iconv would be taking a step backwards (and would therefore generate
>> a -1 from me).
I consider that right way is to teach apr-iconv use win32 api instead
if doing this in Subversion itself. Also in apr-iconv we can hardcode
names of encodings supported by Windows (as far I know Windows
supports limited set of encodings)
Also I had some technical comments on Stefan's patch. But I'm ready to
solve it if we get some decision in concept.
--
Ivan Zhakov
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: SVN Win32 Developers -- need some help
Posted by eg <eg...@gmail.com>.
Stefan Küng wrote:
>
>>> Doesn't this mean that, in a sense, it has been effectively field tested
>>> under various locales for some time?
>
> Yes, it has been tested for some time now. But since that change is only
> in the TSVN trunk, not that many people are really using it yet.
That's good to know... for some reason I thought it was in the 1.4.x
releases already.
>
>>> The only objection in the original thread seems to come from someone's
>>> recollections about incomplete utf8 support under Windows.
>
> Yes, and such arguments simply can't be opposed. You see, there is no
> way to prove that it really works *exactly* as apr-iconv, at least not
> that I know of a way.
I agree. I am guessing / hoping the original objection was an attempt to
ping people to see if anyone had real knowledge of such issues.
>
>>> I recall PostgreSql running into something like this as well so I
>>> googled over there and came up with the following:
>>>
>>> http://archives.postgresql.org/pgsql-odbc/2006-03/msg00227.php
>>> and
>>> http://pginstaller.projects.postgresql.org/faq/FAQ_windows.html#2.6
>>>
>>> Perhaps there is some useful "facts" there.
>
> If you read those you will see that it's not really a windows problem
> but a problem with code pages. Of course it's not possible to convert
> all utf8 sequences into one codepage. And windows has its own local
> codepages (because they're not really standardized (is that a word?)).
> And since each OS has its own local (custom) codepages, you can get into
> troubles. But: as it also says there: both MultiByteToWideChar() and
> WideCharToMultiByte() work just fine, and that's all we use.
>
Thats good to know it shouldn't affect SVN. It is the only tangible
objection / issue that I have come across with respect to this topic on
Windows.
We have been using MultiBytoToWideChar() and WideCharToMultiByte() in
various projects targeting various locales (not really much Asian or
middle eastern languages) and haven't come across any gotchas (note: our
distribution is much, much smaller than the numbers quoted for Tortoise).
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: SVN Win32 Developers -- need some help
Posted by Stefan Küng <to...@gmail.com>.
eg wrote:
> Stefan Küng wrote:
>>
>> If you read those you will see that it's not really a windows problem
>> but a problem with code pages. Of course it's not possible to convert
>> all utf8 sequences into one codepage. And windows has its own local
>> codepages (because they're not really standardized (is that a word?)).
>> And since each OS has its own local (custom) codepages, you can get
>> into troubles. But: as it also says there: both MultiByteToWideChar()
>> and WideCharToMultiByte() work just fine, and that's all we use.
>>
>
> Did a search of the MS Knowledgebase and only came across the following
> reported problem with respect to Japanese Shift-JIS. See the following
> link:
>
> http://support.microsoft.com/kb/170559/en-us
>
> Is this an issue for this approach?
Not really: that happens when converting from the codepage 932 to UTF16.
But since the Subversion lib only converts from/to utf8, this is not an
issue. It can be an issue for the command line client since that one is
not compiled as an utf16 application on windows, but it would also be an
issue with apr-iconv: it can use the same codepage (932) and would
suffer from the same problems.
Stefan
--
___
oo // \\ "De Chelonian Mobile"
(_,\/ \_/ \ TortoiseSVN
\ \_/_\_/> The coolest Interface to (Sub)Version Control
/_/ \_\ http://tortoisesvn.net
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: SVN Win32 Developers -- need some help
Posted by eg <eg...@gmail.com>.
Stefan Küng wrote:
>
> If you read those you will see that it's not really a windows problem
> but a problem with code pages. Of course it's not possible to convert
> all utf8 sequences into one codepage. And windows has its own local
> codepages (because they're not really standardized (is that a word?)).
> And since each OS has its own local (custom) codepages, you can get into
> troubles. But: as it also says there: both MultiByteToWideChar() and
> WideCharToMultiByte() work just fine, and that's all we use.
>
Did a search of the MS Knowledgebase and only came across the following
reported problem with respect to Japanese Shift-JIS. See the following link:
http://support.microsoft.com/kb/170559/en-us
Is this an issue for this approach?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: SVN Win32 Developers -- need some help
Posted by Stefan Küng <to...@gmail.com>.
Mark Phippard wrote:
>> TortoiseSVN "seems" to be one of the main ide/tools of choice for
>> Windows SVN users... this is surmised based on the fact that there have
>> now been over 4,500,000 downloads of it from Sourceforge.
>>
>> I don't know how many of these are non-english users, but some of the
>> main Tortoise devs seem to be.
Yes, I am from Switzerland, and we have some non-ASCII chars in our
language (e.g., öäü, and of course some french chars too, since
Switzerland has four languages).
I can also tell that even Chinese and Russian chars are correctly converted.
>> Doesn't this mean that, in a sense, it has been effectively field tested
>> under various locales for some time?
Yes, it has been tested for some time now. But since that change is only
in the TSVN trunk, not that many people are really using it yet.
>> The only objection in the original thread seems to come from someone's
>> recollections about incomplete utf8 support under Windows.
Yes, and such arguments simply can't be opposed. You see, there is no
way to prove that it really works *exactly* as apr-iconv, at least not
that I know of a way.
>> I recall PostgreSql running into something like this as well so I
>> googled over there and came up with the following:
>>
>> http://archives.postgresql.org/pgsql-odbc/2006-03/msg00227.php
>> and
>> http://pginstaller.projects.postgresql.org/faq/FAQ_windows.html#2.6
>>
>> Perhaps there is some useful "facts" there.
If you read those you will see that it's not really a windows problem
but a problem with code pages. Of course it's not possible to convert
all utf8 sequences into one codepage. And windows has its own local
codepages (because they're not really standardized (is that a word?)).
And since each OS has its own local (custom) codepages, you can get into
troubles. But: as it also says there: both MultiByteToWideChar() and
WideCharToMultiByte() work just fine, and that's all we use.
> Stefan did point out one limitation currently and that is support for
> the --encoding option on the command line. This is not an issue for
> TortoiseSVN because it does not expose that option to users. But
> apparently that option allows you to specify an arbitrary encoding and
> he did not find an API that he could use for that.
>
> It is possible he did not try very hard since he did not need it.
> Personally, I'd rather lose that option in the Windows command line
> than live with APR ICONV any longer.
Well, we could add some kind of table which maps the code-page strings
to the windows code-page defines. But since I had no need for that, I
skipped it (also because I had no intention to go through all the
trouble again to get my patch accepted).
Stefan
--
___
oo // \\ "De Chelonian Mobile"
(_,\/ \_/ \ TortoiseSVN
\ \_/_\_/> The coolest Interface to (Sub)Version Control
/_/ \_\ http://tortoisesvn.net
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: SVN Win32 Developers -- need some help
Posted by Mark Phippard <ma...@gmail.com>.
On 5/30/07, eg <eg...@gmail.com> wrote:
> Mark Phippard wrote:
> > I think we really need to address the problems of APR ICONV in our
> > next release. Please see this thread for some references:
> >
> > http://subversion.tigris.org/servlets/BrowseList?list=dev&by=thread&from=582337
> >
> >
> > This library has always been a problem, when there are multiple
> > clients installed and they used different compilers. But now with a
> > mixture of APR ICONV 0.9.x and 1.1.x out there this problem is going
> > to escalate. It crashes Subversion, which means it can also crash
> > clients using Subversion.
> >
> > Stefan King did the work of creating an initial patch that replaces
> > the dependency with Win32 API's. I think we just need some consensus
> > that this is what we should do, and then get a developer to do it.
> >
> > I think this is important and we are going to have problems if we do
> > not address this for 1.5.
> >
>
> If I understand correctly, Stefan already patches TortoiseSVN to do this.
>
> TortoiseSVN "seems" to be one of the main ide/tools of choice for
> Windows SVN users... this is surmised based on the fact that there have
> now been over 4,500,000 downloads of it from Sourceforge.
>
> I don't know how many of these are non-english users, but some of the
> main Tortoise devs seem to be.
>
> Doesn't this mean that, in a sense, it has been effectively field tested
> under various locales for some time?
>
> The only objection in the original thread seems to come from someone's
> recollections about incomplete utf8 support under Windows.
>
> I recall PostgreSql running into something like this as well so I
> googled over there and came up with the following:
>
> http://archives.postgresql.org/pgsql-odbc/2006-03/msg00227.php
> and
> http://pginstaller.projects.postgresql.org/faq/FAQ_windows.html#2.6
>
> Perhaps there is some useful "facts" there.
Stefan did point out one limitation currently and that is support for
the --encoding option on the command line. This is not an issue for
TortoiseSVN because it does not expose that option to users. But
apparently that option allows you to specify an arbitrary encoding and
he did not find an API that he could use for that.
It is possible he did not try very hard since he did not need it.
Personally, I'd rather lose that option in the Windows command line
than live with APR ICONV any longer.
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: SVN Win32 Developers -- need some help
Posted by eg <eg...@gmail.com>.
Mark Phippard wrote:
> I think we really need to address the problems of APR ICONV in our
> next release. Please see this thread for some references:
>
> http://subversion.tigris.org/servlets/BrowseList?list=dev&by=thread&from=582337
>
>
> This library has always been a problem, when there are multiple
> clients installed and they used different compilers. But now with a
> mixture of APR ICONV 0.9.x and 1.1.x out there this problem is going
> to escalate. It crashes Subversion, which means it can also crash
> clients using Subversion.
>
> Stefan King did the work of creating an initial patch that replaces
> the dependency with Win32 API's. I think we just need some consensus
> that this is what we should do, and then get a developer to do it.
>
> I think this is important and we are going to have problems if we do
> not address this for 1.5.
>
If I understand correctly, Stefan already patches TortoiseSVN to do this.
TortoiseSVN "seems" to be one of the main ide/tools of choice for
Windows SVN users... this is surmised based on the fact that there have
now been over 4,500,000 downloads of it from Sourceforge.
I don't know how many of these are non-english users, but some of the
main Tortoise devs seem to be.
Doesn't this mean that, in a sense, it has been effectively field tested
under various locales for some time?
The only objection in the original thread seems to come from someone's
recollections about incomplete utf8 support under Windows.
I recall PostgreSql running into something like this as well so I
googled over there and came up with the following:
http://archives.postgresql.org/pgsql-odbc/2006-03/msg00227.php
and
http://pginstaller.projects.postgresql.org/faq/FAQ_windows.html#2.6
Perhaps there is some useful "facts" there.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: SVN Win32 Developers -- need some help
Posted by eg <eg...@gmail.com>.
Mark Phippard wrote:
> I think we really need to address the problems of APR ICONV in our
> next release. Please see this thread for some references:
>
> http://subversion.tigris.org/servlets/BrowseList?list=dev&by=thread&from=582337
>
>
> This library has always been a problem, when there are multiple
> clients installed and they used different compilers. But now with a
> mixture of APR ICONV 0.9.x and 1.1.x out there this problem is going
> to escalate. It crashes Subversion, which means it can also crash
> clients using Subversion.
>
> Stefan King did the work of creating an initial patch that replaces
> the dependency with Win32 API's. I think we just need some consensus
> that this is what we should do, and then get a developer to do it.
>
> I think this is important and we are going to have problems if we do
> not address this for 1.5.
>
If I understand correctly, Stefan already patches TortoiseSVN to do this.
TortoiseSVN "seems" to be one of the main ide/tools of choice for
Windows SVN users... this is surmised based on the fact that there have
now been over 4,500,000 downloads of it from Sourceforge.
I don't know how many of these are non-english users, but some of the
main Tortoise devs seem to be.
Doesn't this mean that, in a sense, it has been effectively field tested
under various locales for some time?
The only objection in the original thread seems to come from someone's
recollections about incomplete utf8 support under Windows.
I recall PostgreSql running into something like this as well so I
googled over there and came up with the following:
http://archives.postgresql.org/pgsql-odbc/2006-03/msg00227.php
and
http://pginstaller.projects.postgresql.org/faq/FAQ_windows.html#2.6
Perhaps there is some useful "facts" there.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: SVN Win32 Developers -- need some help
Posted by Norbert Unterberg <nu...@gmail.com>.
2007/6/4, Stefan Küng <to...@gmail.com>:
> * it returns an error if in svn_utf_cstring_to_utf8_ex2 the 'frompage'
> param is not APR_DEFAULT_CHARSET or APR_LOCALE_CHARSET (or the
> equivalent svn_utf_cstring_from_utf8_ex2 and 'topage'). That's because I
> don't know of an API which can convert an iconv codepage string into the
> define required by the Windows APIs.
Maybe the MIME database in the registry could help? It maps MIME
enconding names to code page identifiers.
HKEY_CLASSES_ROOT\Mime\Database\Charset
There is also a COM API named "MLang" that deals with this MIME database:
http://msdn2.microsoft.com/en-us/library/aa767865.aspx
but I do not have any experience with that API.
Norbert
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: SVN Win32 Developers -- need some help
Posted by Stefan Küng <to...@gmail.com>.
Mark Phippard wrote:
> I think we really need to address the problems of APR ICONV in our
> next release. Please see this thread for some references:
>
> http://subversion.tigris.org/servlets/BrowseList?list=dev&by=thread&from=582337
>
>
> This library has always been a problem, when there are multiple
> clients installed and they used different compilers. But now with a
> mixture of APR ICONV 0.9.x and 1.1.x out there this problem is going
> to escalate. It crashes Subversion, which means it can also crash
> clients using Subversion.
>
> Stefan King did the work of creating an initial patch that replaces
> the dependency with Win32 API's. I think we just need some consensus
> that this is what we should do, and then get a developer to do it.
>
> I think this is important and we are going to have problems if we do
> not address this for 1.5.
Patch attached.
This works very well for TSVN already.
Some things you should know about this patch:
* it returns an error if in svn_utf_cstring_to_utf8_ex2 the 'frompage'
param is not APR_DEFAULT_CHARSET or APR_LOCALE_CHARSET (or the
equivalent svn_utf_cstring_from_utf8_ex2 and 'topage'). That's because I
don't know of an API which can convert an iconv codepage string into the
define required by the Windows APIs.
* it uses calloc() and free() for memory allocation. I could have used
the apr memory pools for that, but the memory needed there is only
needed for two lines of code and can be freed after that. I think using
an apr sub-pool is much 'heavier' than a clean calloc()/free(). And
since this is Windows-Only code anyway, we don't have to use the
platform independent apr pools.
Stefan
--
___
oo // \\ "De Chelonian Mobile"
(_,\/ \_/ \ TortoiseSVN
\ \_/_\_/> The coolest Interface to (Sub)Version Control
/_/ \_\ http://tortoisesvn.net