You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Thomas Åkesson <th...@akesson.cc> on 2012/02/14 12:36:23 UTC
Re: {SPAM 03.5} Re: [RFC] Non-normalizing Unicode Composition Awareness
Thanks Julian and Markus for providing feedback.
I am not commenting below because all the feedback is very good and I will try to address it as best I can in the next iteration. Describing the behaviour changes to the WC is the most challenging since I lack that kind of detailed knowledge. I will instead try to draft the structure of that section to make it easier for someone with that level of detail to assist.
Regarding use cases, what can I say... it was towards the end of a long stretch.
I think it would help with the upcoming iterations if I could move this "document" into the wiki. If you find that this first draft shows promise, please consider granting edit access in the wiki. My user name is "Thomas Åkesson", which exercises the Unicode awareness of MoinMoin...
/Thomas Å.
On 14 feb 2012, at 11:25, Julian Foad wrote:
> Hi Thomas. It's fantastic that you're taking the trouble to write up this proposal. That's just what we need. Just a few initial comments below...
>
> Thomas Åkesson wrote:
>
>> Context
>> ===
>>
>> [...] A unicode string (e.g. a file name) can be represented
>> in 2 normalized forms (NFC/NFD) or mixed, i.e. multiple such
>> characters where some are composed and others decomposed (rare).
>
>
> What's "rare"? We have to assume that input is in mixed composition in any system that doesn't explicitly normalize it, which (I think) includes most operating systems. While it may be rare for any single string to contain characters in both compositions, it is very common to be processing a string that *might* have characters in both compositions -- in other words, that is not guaranteed to be normalized. I think it would be clearer to drop the "(rare)" and just say "... normalized forms (NFC/NFD) or mixed (not normalized).".
>
>
>> A minority of file systems (currently Mac OS X HFS+ only) will
>> normalize the paths. In the case of HFS+, the path will be
>> normalized into NFD and it will even be given back that way when
>> listing the filesystem.
>
>
> Drop the word "even"? The statement is not surprising.
>
>
> [...]
>
>> Similarities to case-sensitivity
>> ===
>>
>> - If two Unicode strings differ only by letter case/composition,
>
> Drop "/composition" -- it's the subject of the following sentence.
>
>> on some
> computer systems they refer to the same file, while on
>> other systems
> they refer to different files. The same applies
>> if two Unicode strings
> differ only by composition.
>
>
>> [...]
>
>> Client Changes
>> ===
>>
>> [...] An abstraction between the repository path and the file
>> system path can be achieved by ensuring that there is a column
>> in wc.db that contains the file system path in exactly the same
>> form that the file system gives back. APIs in wc needs to be
>> extended to ensure that all interaction with the file system is
>> performed with the file system path.
>
> [...]
>
> This part seems to be the heart of the whole proposal. You describe the data that we need, but the behaviour will also need to be described in detail. Presumably much of the behaviour is boring and obvious (when we check out a new path and create it on disk, we store the disk path), but I'm sure there will be some less obvious parts (do we need to find out what the disk path of an 'excluded' node would be, even though we're not actually creating it on disk, for example).
>
>
>> Use Cases
>> ===
>>
>> This change will only affect use cases which rely on creating
>> paths that look like duplicates but use different unicode
>> composition. It is highly unlikely anyone is relying on this..
>
>
> Uh... it sounds like you are saying there are no interesting use cases for this proposal! No, on the contrary, this proposal also affects checking out and using a WC on Mac HFS+ where the repository paths were created on another system and are not in NFD, and it allows that case to work. That's the more interesting use case, is it not? It's definitely worth writing out the interesting case in full, including steps like checkout (or update) that brings in a non-NFD path, create a new file on the Mac, and commit.
>
> - Julian
>
Re: [RFC] Non-normalizing Unicode Composition Awareness
Posted by Philip Martin <ph...@wandisco.com>.
Thomas Åkesson <th...@akesson.cc> writes:
> If you, or someone else with WC insight, could provide some details on
> when/how conversions in the opposite direction is performed (e.g. svn
> stat and most commands taking path arguments), that would be
> incredibly useful to me. I would like to explore the option to somehow
> work around the "irreversible problem".
The Subversion libraries generally deal with UTF-8 paths exclusively.
When the use runs
$ svn stat foo
the path 'foo' is typed in some local encoding. The program converts the
path from the local encoding to UTF-8 before passing it to the client or
working copy libraries. These library then treats the path as UTF-8 in
almost most cases. When making a system call such as stat() they pass
the UTF-8 path to the Subversion low-level IO functions in libsvn_subr.
These low-level functions convert the UTF-8 to the local encoding before
making the system calls.
--
uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com
Re: [RFC] Non-normalizing Unicode Composition Awareness
Posted by Thomas Åkesson <th...@bafast.se>.
On 9 nov 2012, at 14:28, "C. Michael Pilato" <cm...@collab.net> wrote:
> On 11/09/2012 07:49 AM, Branko Čibej wrote:
>> On 09.11.2012 12:28, Thomas Åkesson wrote:
>> I'm currently doing the grunt work of implementing the collation (done)
>> and the LIKE and GLOB operators that we'll need (in progress). The next,
>> and biggest, step will be to review the client and WC libraries to make
>> sure that paths sent to the server always come from the wc.db, not from
>> disk.
>
> I'm not closely following this problem or solution, but how does the above
> play out for "svn import", "svn mkdir IRI", "svn delete IRI", etc? (If this
> is documented somewhere, a reference by way of response would suffice.)
http://wiki.apache.org/subversion/NonNormalizingUnicodeCompositionAwareness
The draft proposes that the server does not discriminate any composition, apart from ensuring that creation of new name collisions is not allowed.
Ensuring that paths come from wc.db applies to existing object. We can discuss whether Mac client should normalize to NFC, but that would be an option in my opinion.
/Thomas Å.
Re: [RFC] Non-normalizing Unicode Composition Awareness
Posted by Branko Čibej <br...@wandisco.com>.
On 09.11.2012 14:28, C. Michael Pilato wrote:
> On 11/09/2012 07:49 AM, Branko Čibej wrote:
>> On 09.11.2012 12:28, Thomas Åkesson wrote:
>> I'm currently doing the grunt work of implementing the collation (done)
>> and the LIKE and GLOB operators that we'll need (in progress). The next,
>> and biggest, step will be to review the client and WC libraries to make
>> sure that paths sent to the server always come from the wc.db, not from
>> disk.
> I'm not closely following this problem or solution, but how does the above
> play out for "svn import", "svn mkdir IRI", "svn delete IRI", etc? (If this
> is documented somewhere, a reference by way of response would suffice.)
Since these are server-side operations with no working copy involvement,
and I'm doing this strictly client-side for now, nothing would change.
This is a problem that we'll eventually have to solve on the server as
well. I don't believe it would be correct for the client to verify that
such operations do not create normalization conflicts on the server.
As a matter of interest, a server-side solution is one of the features
we identified for FSv2; although there's no reason to wait for that. In
FSv2, I envision all names being stored twice, once in their original
form, and once NFC-normalized, for indexing. The reason for that is that
I expect server CPU cycles to be more expensive than server storage, and
it therefore makes sense to avoid using a relatively expensive
normalizing collation in the server metadata index.
This /may/ turn out to be an issue for client working copy performance,
too; but for now I've elected to assume that collation won't have a
noticeable effect. If it does, we'll look at other solutions.
-- Brane
--
Branko Čibej
Director of Subversion | WANdisco | www.wandisco.com
Re: [RFC] Non-normalizing Unicode Composition Awareness
Posted by "C. Michael Pilato" <cm...@collab.net>.
On 11/09/2012 07:49 AM, Branko Čibej wrote:
> On 09.11.2012 12:28, Thomas Åkesson wrote:
> I'm currently doing the grunt work of implementing the collation (done)
> and the LIKE and GLOB operators that we'll need (in progress). The next,
> and biggest, step will be to review the client and WC libraries to make
> sure that paths sent to the server always come from the wc.db, not from
> disk.
I'm not closely following this problem or solution, but how does the above
play out for "svn import", "svn mkdir IRI", "svn delete IRI", etc? (If this
is documented somewhere, a reference by way of response would suffice.)
--
C. Michael Pilato <cm...@collab.net>
CollabNet <> www.collab.net <> Enterprise Cloud Development
Re: [RFC] Non-normalizing Unicode Composition Awareness
Posted by Branko Čibej <br...@wandisco.com>.
On 09.11.2012 12:28, Thomas Åkesson wrote:
> Today, I noticed that Branko started some implementation in a branch. Looks like a collation based on utf8proc is in the making? I think that would make a lot of sense because the ICU extension poses some challenges in the build process and we might not need all that functionality that it provides.
Hi Thomas,
Yes, I started a branch that's intended to fix the normalization
problem. I selected utf8proc because we really don't need ICU (I can't
see a serious need for language-specific case folding, for example, nor
for Unicode regular expressions). Furthermore, utf8proc can be easily
embedded into Subversion so it doesn't present another dependency that
users would have to worry about.
I'm currently doing the grunt work of implementing the collation (done)
and the LIKE and GLOB operators that we'll need (in progress). The next,
and biggest, step will be to review the client and WC libraries to make
sure that paths sent to the server always come from the wc.db, not from
disk.
One open question is what to do about (historical) collisions in
existing repositories, but I don't think that issue is important enough
to resolve now.
It'll take a while, but I hope to be able to finish the work in time for
1.8. If not ... well then, it'll be in 1.9.
-- Brane
--
Branko Čibej
Director of Subversion | WANdisco | www.wandisco.com
Re: [RFC] Non-normalizing Unicode Composition Awareness
Posted by Thomas Åkesson <th...@akesson.cc>.
Revisiting this thread after a few months. Last spring, I did some work in the Wiki designing a proposal for resolving the Mac Unicode issues in a Non-normalizing manner. I ran out of time, but the thought process has been ongoing.
A couple of weeks ago at Subversion Live in London, I had the opportunity to discuss with a number of people. Although there were some different opinions on the matter, I think we concluded that we are actually relatively well aligned on the core idea.
The proposal I drafted this spring (in the Wiki) proposed that a couple of columns were added to the WC in order to store normalized paths. Since a couple of months the concept of using a Sqlite collation has seemed more appealing. Last week, I did a test with the Sqlite ICU extension (available in sqlite source repository) which turned out to be quite encouraging. With such a collation, it is possible to perform equals in SQL statements that match paths in a Unicode composition aware manner and therefore return rows regardless what composition the paths have.
This would be very useful, for instance, when given a filesystem path attempting to locate the corresponding node in wc.db. That is basically half the issue with Mac working copies.
Today, I noticed that Branko started some implementation in a branch. Looks like a collation based on utf8proc is in the making? I think that would make a lot of sense because the ICU extension poses some challenges in the build process and we might not need all that functionality that it provides.
I started a wiki page about unicode collation. I will append more info:
http://wiki.apache.org/subversion/UnicodeCollation
Also note the tiny test repo attached to:
http://wiki.apache.org/subversion/NonNormalizingUnicodeCompositionAwareness
Cheers,
Thomas Å.
Re: [RFC] Non-normalizing Unicode Composition Awareness
Posted by Thomas Åkesson <th...@akesson.cc>.
Hi Philip,
Thanks for your comments in the wiki article. They raised some important points and potentially an idea that might simplify the solution.
> All three paths are in UTF-8 but NFC/NFD is not currently specified. local_relpath/parent_relpath get converted from UTF-8 to whatever locale encoding is in use whenever they are used to access the filesystem.
This is not unlike what we need to do for HFS+. We could consider UTF8-MAC to be a distinct encoding. There is the major caveat that this conversion is irreversible (since the normalization is not specified in the repo/wc.db).
If you, or someone else with WC insight, could provide some details on when/how conversions in the opposite direction is performed (e.g. svn stat and most commands taking path arguments), that would be incredibly useful to me. I would like to explore the option to somehow work around the "irreversible problem".
It would also be useful if someone could point me to where in the WC code the conversion from UTF-8 to locale encoding is performed.
Thanks!
/Thomas Å.
On 17 apr 2012, at 05:24, Thomas Åkesson wrote:
> Hi,
> A bit of a status update on the wiki article:
> http://wiki.apache.org/subversion/NonNormalizingUnicodeCompositionAwareness
>
> Received some comments from Daniel, which I have tried to address. Thanks.
>
> I have written a bash script which demonstrates the concept of "Alternative 1" with regards to how the local_relpath column is handled by checkout/update.
>
> From the wiki:
> ---
> This alternative can be simulated using the attached script localrelpath2nfd.sh. This provides a Working Copy equivalent to what a checkout should produce if this alternative was implemented in Subversion itself:
>
> svn co ...
> svn stat #Shows any problematic items as missing and unversioned
> localrelpath2nfd.sh
> svn stat #Should be clean apart from misperception that some items are switched
> ---
>
> This script can be used to investigate how other subcommands are affected and determine what needs to be done. It is possible to make commits but updates to normalisation-dependent nodes will fail since this script is not inside the update code.
>
> I intend to use this script to take the design to the next level of detail. First, I would like some feedback from people with in-depth knowledge of the WC and preferably get some idea on what the community thinks about the approach.
>
> /Thomas Å.
>
>
> On 26 mar 2012, at 04:14, Thomas Åkesson <th...@akesson.cc> wrote:
>
>> Hi,
>> Sorry about the delay, had a release to sort out...
>>
>> I have moved the proposal into the wiki:
>> http://wiki.apache.org/subversion/NonNormalizingUnicodeCompositionAwareness
>>
>> The comments from Julian and Markus have been implemented and I have added more information to the "Client Changes" section as well as more structure and TODO-notes.
>>
>> I would really appreciate if someone with more insight into WC-NG could provide input on some of the TODO items (or things that have been completely overlooked).
>>
>> Thanks,
>> Thomas Å.
>>
>>
>> On 21 feb 2012, at 09:55, Daniel Shahaf wrote:
>>
>>> I've granted you write access to the wiki.
>>>
>>> Thomas Åkesson wrote on Tue, Feb 14, 2012 at 12:36:23 +0100:
>>>> Thanks Julian and Markus for providing feedback.
>>>>
>>>> I am not commenting below because all the feedback is very good and I will try to address it as best I can in the next iteration. Describing the behaviour changes to the WC is the most challenging since I lack that kind of detailed knowledge. I will instead try to draft the structure of that section to make it easier for someone with that level of detail to assist.
>>>>
>>>> Regarding use cases, what can I say... it was towards the end of a long stretch.
>>>>
>>>> I think it would help with the upcoming iterations if I could move this "document" into the wiki. If you find that this first draft shows promise, please consider granting edit access in the wiki. My user name is "Thomas Åkesson", which exercises the Unicode awareness of MoinMoin...
>>>>
>>>> /Thomas Å.
>>>>
>>>>
>>>> On 14 feb 2012, at 11:25, Julian Foad wrote:
>>>>
>>>>> Hi Thomas. It's fantastic that you're taking the trouble to write up this proposal. That's just what we need. Just a few initial comments below...
>>>>>
>>>>> Thomas Åkesson wrote:
>>>>>
>>>>>> Context
>>>>>> ===
>>>>>>
>>>>>> [...] A unicode string (e.g. a file name) can be represented
>>>>>> in 2 normalized forms (NFC/NFD) or mixed, i.e. multiple such
>>>>>> characters where some are composed and others decomposed (rare).
>>>>>
>>>>>
>>>>> What's "rare"? We have to assume that input is in mixed composition in any system that doesn't explicitly normalize it, which (I think) includes most operating systems. While it may be rare for any single string to contain characters in both compositions, it is very common to be processing a string that *might* have characters in both compositions -- in other words, that is not guaranteed to be normalized. I think it would be clearer to drop the "(rare)" and just say "... normalized forms (NFC/NFD) or mixed (not normalized).".
>>>>>
>>>>>
>>>>>> A minority of file systems (currently Mac OS X HFS+ only) will
>>>>>> normalize the paths. In the case of HFS+, the path will be
>>>>>> normalized into NFD and it will even be given back that way when
>>>>>> listing the filesystem.
>>>>>
>>>>>
>>>>> Drop the word "even"? The statement is not surprising.
>>>>>
>>>>>
>>>>> [...]
>>>>>
>>>>>> Similarities to case-sensitivity
>>>>>> ===
>>>>>>
>>>>>> - If two Unicode strings differ only by letter case/composition,
>>>>>
>>>>> Drop "/composition" -- it's the subject of the following sentence.
>>>>>
>>>>>> on some
>>>>> computer systems they refer to the same file, while on
>>>>>> other systems
>>>>> they refer to different files. The same applies
>>>>>> if two Unicode strings
>>>>> differ only by composition.
>>>>>
>>>>>
>>>>>> [...]
>>>>>
>>>>>> Client Changes
>>>>>> ===
>>>>>>
>>>>>> [...] An abstraction between the repository path and the file
>>>>>> system path can be achieved by ensuring that there is a column
>>>>>> in wc.db that contains the file system path in exactly the same
>>>>>> form that the file system gives back. APIs in wc needs to be
>>>>>> extended to ensure that all interaction with the file system is
>>>>>> performed with the file system path.
>>>>>
>>>>> [...]
>>>>>
>>>>> This part seems to be the heart of the whole proposal. You describe the data that we need, but the behaviour will also need to be described in detail. Presumably much of the behaviour is boring and obvious (when we check out a new path and create it on disk, we store the disk path), but I'm sure there will be some less obvious parts (do we need to find out what the disk path of an 'excluded' node would be, even though we're not actually creating it on disk, for example).
>>>>>
>>>>>
>>>>>> Use Cases
>>>>>> ===
>>>>>>
>>>>>> This change will only affect use cases which rely on creating
>>>>>> paths that look like duplicates but use different unicode
>>>>>> composition. It is highly unlikely anyone is relying on this..
>>>>>
>>>>>
>>>>> Uh... it sounds like you are saying there are no interesting use cases for this proposal! No, on the contrary, this proposal also affects checking out and using a WC on Mac HFS+ where the repository paths were created on another system and are not in NFD, and it allows that case to work. That's the more interesting use case, is it not? It's definitely worth writing out the interesting case in full, including steps like checkout (or update) that brings in a non-NFD path, create a new file on the Mac, and commit.
>>>>>
>>>>> - Julian
>>>>>
>>>>
>>
>
Re: [RFC] Non-normalizing Unicode Composition Awareness
Posted by Stefan Sperling <st...@elego.de>.
On Tue, Apr 17, 2012 at 05:24:53AM +0200, Thomas Åkesson wrote:
> I intend to use this script to take the design to the next level of detail. First, I would like some feedback from people with in-depth knowledge of the WC and preferably get some idea on what the community thinks about the approach.
I am interested but I don't have time to digest the details right now.
If you don't get any responses within a couple of days please ping me
and I'll take a look. Thanks!
Re: [RFC] Non-normalizing Unicode Composition Awareness
Posted by Thomas Åkesson <th...@akesson.cc>.
Hi,
A bit of a status update on the wiki article:
http://wiki.apache.org/subversion/NonNormalizingUnicodeCompositionAwareness
Received some comments from Daniel, which I have tried to address. Thanks.
I have written a bash script which demonstrates the concept of "Alternative 1" with regards to how the local_relpath column is handled by checkout/update.
From the wiki:
---
This alternative can be simulated using the attached script localrelpath2nfd.sh. This provides a Working Copy equivalent to what a checkout should produce if this alternative was implemented in Subversion itself:
svn co ...
svn stat #Shows any problematic items as missing and unversioned
localrelpath2nfd.sh
svn stat #Should be clean apart from misperception that some items are switched
---
This script can be used to investigate how other subcommands are affected and determine what needs to be done. It is possible to make commits but updates to normalisation-dependent nodes will fail since this script is not inside the update code.
I intend to use this script to take the design to the next level of detail. First, I would like some feedback from people with in-depth knowledge of the WC and preferably get some idea on what the community thinks about the approach.
/Thomas Å.
On 26 mar 2012, at 04:14, Thomas Åkesson <th...@akesson.cc> wrote:
> Hi,
> Sorry about the delay, had a release to sort out...
>
> I have moved the proposal into the wiki:
> http://wiki.apache.org/subversion/NonNormalizingUnicodeCompositionAwareness
>
> The comments from Julian and Markus have been implemented and I have added more information to the "Client Changes" section as well as more structure and TODO-notes.
>
> I would really appreciate if someone with more insight into WC-NG could provide input on some of the TODO items (or things that have been completely overlooked).
>
> Thanks,
> Thomas Å.
>
>
> On 21 feb 2012, at 09:55, Daniel Shahaf wrote:
>
>> I've granted you write access to the wiki.
>>
>> Thomas Åkesson wrote on Tue, Feb 14, 2012 at 12:36:23 +0100:
>>> Thanks Julian and Markus for providing feedback.
>>>
>>> I am not commenting below because all the feedback is very good and I will try to address it as best I can in the next iteration. Describing the behaviour changes to the WC is the most challenging since I lack that kind of detailed knowledge. I will instead try to draft the structure of that section to make it easier for someone with that level of detail to assist.
>>>
>>> Regarding use cases, what can I say... it was towards the end of a long stretch.
>>>
>>> I think it would help with the upcoming iterations if I could move this "document" into the wiki. If you find that this first draft shows promise, please consider granting edit access in the wiki. My user name is "Thomas Åkesson", which exercises the Unicode awareness of MoinMoin...
>>>
>>> /Thomas Å.
>>>
>>>
>>> On 14 feb 2012, at 11:25, Julian Foad wrote:
>>>
>>>> Hi Thomas. It's fantastic that you're taking the trouble to write up this proposal. That's just what we need. Just a few initial comments below...
>>>>
>>>> Thomas Åkesson wrote:
>>>>
>>>>> Context
>>>>> ===
>>>>>
>>>>> [...] A unicode string (e.g. a file name) can be represented
>>>>> in 2 normalized forms (NFC/NFD) or mixed, i.e. multiple such
>>>>> characters where some are composed and others decomposed (rare).
>>>>
>>>>
>>>> What's "rare"? We have to assume that input is in mixed composition in any system that doesn't explicitly normalize it, which (I think) includes most operating systems. While it may be rare for any single string to contain characters in both compositions, it is very common to be processing a string that *might* have characters in both compositions -- in other words, that is not guaranteed to be normalized. I think it would be clearer to drop the "(rare)" and just say "... normalized forms (NFC/NFD) or mixed (not normalized).".
>>>>
>>>>
>>>>> A minority of file systems (currently Mac OS X HFS+ only) will
>>>>> normalize the paths. In the case of HFS+, the path will be
>>>>> normalized into NFD and it will even be given back that way when
>>>>> listing the filesystem.
>>>>
>>>>
>>>> Drop the word "even"? The statement is not surprising.
>>>>
>>>>
>>>> [...]
>>>>
>>>>> Similarities to case-sensitivity
>>>>> ===
>>>>>
>>>>> - If two Unicode strings differ only by letter case/composition,
>>>>
>>>> Drop "/composition" -- it's the subject of the following sentence.
>>>>
>>>>> on some
>>>> computer systems they refer to the same file, while on
>>>>> other systems
>>>> they refer to different files. The same applies
>>>>> if two Unicode strings
>>>> differ only by composition.
>>>>
>>>>
>>>>> [...]
>>>>
>>>>> Client Changes
>>>>> ===
>>>>>
>>>>> [...] An abstraction between the repository path and the file
>>>>> system path can be achieved by ensuring that there is a column
>>>>> in wc.db that contains the file system path in exactly the same
>>>>> form that the file system gives back. APIs in wc needs to be
>>>>> extended to ensure that all interaction with the file system is
>>>>> performed with the file system path.
>>>>
>>>> [...]
>>>>
>>>> This part seems to be the heart of the whole proposal. You describe the data that we need, but the behaviour will also need to be described in detail. Presumably much of the behaviour is boring and obvious (when we check out a new path and create it on disk, we store the disk path), but I'm sure there will be some less obvious parts (do we need to find out what the disk path of an 'excluded' node would be, even though we're not actually creating it on disk, for example).
>>>>
>>>>
>>>>> Use Cases
>>>>> ===
>>>>>
>>>>> This change will only affect use cases which rely on creating
>>>>> paths that look like duplicates but use different unicode
>>>>> composition. It is highly unlikely anyone is relying on this..
>>>>
>>>>
>>>> Uh... it sounds like you are saying there are no interesting use cases for this proposal! No, on the contrary, this proposal also affects checking out and using a WC on Mac HFS+ where the repository paths were created on another system and are not in NFD, and it allows that case to work. That's the more interesting use case, is it not? It's definitely worth writing out the interesting case in full, including steps like checkout (or update) that brings in a non-NFD path, create a new file on the Mac, and commit.
>>>>
>>>> - Julian
>>>>
>>>
>
Re: [RFC] Non-normalizing Unicode Composition Awareness
Posted by Thomas Åkesson <th...@akesson.cc>.
Hi,
Sorry about the delay, had a release to sort out...
I have moved the proposal into the wiki:
http://wiki.apache.org/subversion/NonNormalizingUnicodeCompositionAwareness
The comments from Julian and Markus have been implemented and I have added more information to the "Client Changes" section as well as more structure and TODO-notes.
I would really appreciate if someone with more insight into WC-NG could provide input on some of the TODO items (or things that have been completely overlooked).
Thanks,
Thomas Å.
On 21 feb 2012, at 09:55, Daniel Shahaf wrote:
> I've granted you write access to the wiki.
>
> Thomas Åkesson wrote on Tue, Feb 14, 2012 at 12:36:23 +0100:
>> Thanks Julian and Markus for providing feedback.
>>
>> I am not commenting below because all the feedback is very good and I will try to address it as best I can in the next iteration. Describing the behaviour changes to the WC is the most challenging since I lack that kind of detailed knowledge. I will instead try to draft the structure of that section to make it easier for someone with that level of detail to assist.
>>
>> Regarding use cases, what can I say... it was towards the end of a long stretch.
>>
>> I think it would help with the upcoming iterations if I could move this "document" into the wiki. If you find that this first draft shows promise, please consider granting edit access in the wiki. My user name is "Thomas Åkesson", which exercises the Unicode awareness of MoinMoin...
>>
>> /Thomas Å.
>>
>>
>> On 14 feb 2012, at 11:25, Julian Foad wrote:
>>
>>> Hi Thomas. It's fantastic that you're taking the trouble to write up this proposal. That's just what we need. Just a few initial comments below...
>>>
>>> Thomas Åkesson wrote:
>>>
>>>> Context
>>>> ===
>>>>
>>>> [...] A unicode string (e.g. a file name) can be represented
>>>> in 2 normalized forms (NFC/NFD) or mixed, i.e. multiple such
>>>> characters where some are composed and others decomposed (rare).
>>>
>>>
>>> What's "rare"? We have to assume that input is in mixed composition in any system that doesn't explicitly normalize it, which (I think) includes most operating systems. While it may be rare for any single string to contain characters in both compositions, it is very common to be processing a string that *might* have characters in both compositions -- in other words, that is not guaranteed to be normalized. I think it would be clearer to drop the "(rare)" and just say "... normalized forms (NFC/NFD) or mixed (not normalized).".
>>>
>>>
>>>> A minority of file systems (currently Mac OS X HFS+ only) will
>>>> normalize the paths. In the case of HFS+, the path will be
>>>> normalized into NFD and it will even be given back that way when
>>>> listing the filesystem.
>>>
>>>
>>> Drop the word "even"? The statement is not surprising.
>>>
>>>
>>> [...]
>>>
>>>> Similarities to case-sensitivity
>>>> ===
>>>>
>>>> - If two Unicode strings differ only by letter case/composition,
>>>
>>> Drop "/composition" -- it's the subject of the following sentence.
>>>
>>>> on some
>>> computer systems they refer to the same file, while on
>>>> other systems
>>> they refer to different files. The same applies
>>>> if two Unicode strings
>>> differ only by composition.
>>>
>>>
>>>> [...]
>>>
>>>> Client Changes
>>>> ===
>>>>
>>>> [...] An abstraction between the repository path and the file
>>>> system path can be achieved by ensuring that there is a column
>>>> in wc.db that contains the file system path in exactly the same
>>>> form that the file system gives back. APIs in wc needs to be
>>>> extended to ensure that all interaction with the file system is
>>>> performed with the file system path.
>>>
>>> [...]
>>>
>>> This part seems to be the heart of the whole proposal. You describe the data that we need, but the behaviour will also need to be described in detail. Presumably much of the behaviour is boring and obvious (when we check out a new path and create it on disk, we store the disk path), but I'm sure there will be some less obvious parts (do we need to find out what the disk path of an 'excluded' node would be, even though we're not actually creating it on disk, for example).
>>>
>>>
>>>> Use Cases
>>>> ===
>>>>
>>>> This change will only affect use cases which rely on creating
>>>> paths that look like duplicates but use different unicode
>>>> composition. It is highly unlikely anyone is relying on this..
>>>
>>>
>>> Uh... it sounds like you are saying there are no interesting use cases for this proposal! No, on the contrary, this proposal also affects checking out and using a WC on Mac HFS+ where the repository paths were created on another system and are not in NFD, and it allows that case to work. That's the more interesting use case, is it not? It's definitely worth writing out the interesting case in full, including steps like checkout (or update) that brings in a non-NFD path, create a new file on the Mac, and commit.
>>>
>>> - Julian
>>>
>>
Re: {SPAM 03.5} Re: [RFC] Non-normalizing Unicode Composition
Awareness
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
I've granted you write access to the wiki.
Thomas Åkesson wrote on Tue, Feb 14, 2012 at 12:36:23 +0100:
> Thanks Julian and Markus for providing feedback.
>
> I am not commenting below because all the feedback is very good and I will try to address it as best I can in the next iteration. Describing the behaviour changes to the WC is the most challenging since I lack that kind of detailed knowledge. I will instead try to draft the structure of that section to make it easier for someone with that level of detail to assist.
>
> Regarding use cases, what can I say... it was towards the end of a long stretch.
>
> I think it would help with the upcoming iterations if I could move this "document" into the wiki. If you find that this first draft shows promise, please consider granting edit access in the wiki. My user name is "Thomas Åkesson", which exercises the Unicode awareness of MoinMoin...
>
> /Thomas Å.
>
>
> On 14 feb 2012, at 11:25, Julian Foad wrote:
>
> > Hi Thomas. It's fantastic that you're taking the trouble to write up this proposal. That's just what we need. Just a few initial comments below...
> >
> > Thomas Åkesson wrote:
> >
> >> Context
> >> ===
> >>
> >> [...] A unicode string (e.g. a file name) can be represented
> >> in 2 normalized forms (NFC/NFD) or mixed, i.e. multiple such
> >> characters where some are composed and others decomposed (rare).
> >
> >
> > What's "rare"? We have to assume that input is in mixed composition in any system that doesn't explicitly normalize it, which (I think) includes most operating systems. While it may be rare for any single string to contain characters in both compositions, it is very common to be processing a string that *might* have characters in both compositions -- in other words, that is not guaranteed to be normalized. I think it would be clearer to drop the "(rare)" and just say "... normalized forms (NFC/NFD) or mixed (not normalized).".
> >
> >
> >> A minority of file systems (currently Mac OS X HFS+ only) will
> >> normalize the paths. In the case of HFS+, the path will be
> >> normalized into NFD and it will even be given back that way when
> >> listing the filesystem.
> >
> >
> > Drop the word "even"? The statement is not surprising.
> >
> >
> > [...]
> >
> >> Similarities to case-sensitivity
> >> ===
> >>
> >> - If two Unicode strings differ only by letter case/composition,
> >
> > Drop "/composition" -- it's the subject of the following sentence.
> >
> >> on some
> > computer systems they refer to the same file, while on
> >> other systems
> > they refer to different files. The same applies
> >> if two Unicode strings
> > differ only by composition.
> >
> >
> >> [...]
> >
> >> Client Changes
> >> ===
> >>
> >> [...] An abstraction between the repository path and the file
> >> system path can be achieved by ensuring that there is a column
> >> in wc.db that contains the file system path in exactly the same
> >> form that the file system gives back. APIs in wc needs to be
> >> extended to ensure that all interaction with the file system is
> >> performed with the file system path.
> >
> > [...]
> >
> > This part seems to be the heart of the whole proposal. You describe the data that we need, but the behaviour will also need to be described in detail. Presumably much of the behaviour is boring and obvious (when we check out a new path and create it on disk, we store the disk path), but I'm sure there will be some less obvious parts (do we need to find out what the disk path of an 'excluded' node would be, even though we're not actually creating it on disk, for example).
> >
> >
> >> Use Cases
> >> ===
> >>
> >> This change will only affect use cases which rely on creating
> >> paths that look like duplicates but use different unicode
> >> composition. It is highly unlikely anyone is relying on this..
> >
> >
> > Uh... it sounds like you are saying there are no interesting use cases for this proposal! No, on the contrary, this proposal also affects checking out and using a WC on Mac HFS+ where the repository paths were created on another system and are not in NFD, and it allows that case to work. That's the more interesting use case, is it not? It's definitely worth writing out the interesting case in full, including steps like checkout (or update) that brings in a non-NFD path, create a new file on the Mac, and commit.
> >
> > - Julian
> >
>