You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by kf...@collab.net on 2004/12/01 16:33:30 UTC
Re: issue #1954 (was: Re: Supporting non-XML-safe pathnames)
Thanks folks for clarifying my mistaken idea of what is valid in XML
1.0 and 1.1. I now grok the situation...
VK Sameer <sa...@collab.net> writes:
> Wouldn't this depend on SVN's definition of a valid pathname? Could that
> be settled first?
I'm happy with just UTF-8, that is, no control chars except LF, CR,
and TAB.
(In one mail, Julian Reschke wrote "SP" where I think he meant "LF".
Obviously, we would allow SP, it's just that it's not a control
character, whereas LF is. Both are allowed.)
> Once the definition is settled, if a valid pathname contains UTF8
> characters not in the XML 1.0 legal character set, then it would require
> escaping/base-64 encoding before use as an attribute value.
Okay. +1 on prohibiting control characters except LF, CR, TAB. I
doubt any users are going to suffer much if we do that. There doesn't
seem to be much opposition to it on this list either (am I forgetting
anyone?), and clearly it'll make Peter Lundblad very happy :-).
Does anyone object to VK Sameer proceeding along those lines?
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: issue #1954 (was: Re: Supporting non-XML-safe pathnames)
Posted by kf...@collab.net.
VK Sameer <sa...@collab.net> writes:
> On Thu, 2004-12-02 at 03:49, kfogel@collab.net wrote:
> > "Peter N. Lundblad" <pe...@famlundblad.se> writes:
> > > > Valid Subversion paths are UTF8 strings, but with no control chars
> > > > allowed except for TAB.
> > > >
> > > > That is what I meant to +1.
> > >
> > > +1. We can loosen this if we want when the world is XML 1.1. But htat's
> > > another year I suppose.
> >
> > Cool.
> >
> > Sameer -- you okay with all this?
>
> Karl, absolutely, consensus at last! I'll come up with a patch with tab
> in the list of invalid Subversion pathname characters. Unless somebody
> objects ...
Whups! I'm not sure if you mistyped above, or if you misunderstood
what I wrote. My language was technically accurate, but perhaps
needlessly terse. Let's try again:
Valid Subversion paths are a *subset* of valid UTF8 strings.
Specifically, they are UTF8 strings with no control characters.
Except, one particular control character is allowed: TAB. All
other control characters are prohibited.
So, TAB *is* allowed. Other control chars are not. Still +1, right?
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: issue #1954 (was: Re: Supporting non-XML-safe pathnames)
Posted by VK Sameer <sa...@collab.net>.
On Thu, 2004-12-02 at 03:49, kfogel@collab.net wrote:
> "Peter N. Lundblad" <pe...@famlundblad.se> writes:
> > > Valid Subversion paths are UTF8 strings, but with no control chars
> > > allowed except for TAB.
> > >
> > > That is what I meant to +1.
> >
> > +1. We can loosen this if we want when the world is XML 1.1. But htat's
> > another year I suppose.
>
> Cool.
>
> Sameer -- you okay with all this?
Karl, absolutely, consensus at last! I'll come up with a patch with tab
in the list of invalid Subversion pathname characters. Unless somebody
objects ...
Regards
Sameer
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: issue #1954 (was: Re: Supporting non-XML-safe pathnames)
Posted by kf...@collab.net.
"Peter N. Lundblad" <pe...@famlundblad.se> writes:
> > Valid Subversion paths are UTF8 strings, but with no control chars
> > allowed except for TAB.
> >
> > That is what I meant to +1.
>
> +1. We can loosen this if we want when the world is XML 1.1. But htat's
> another year I suppose.
Cool.
Sameer -- you okay with all this?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: issue #1954
Posted by Julian Reschke <ju...@gmx.de>.
Peter N. Lundblad wrote:
> +1. We can loosen this if we want when the world is XML 1.1. But htat's
> another year I suppose.
It may never happen, looking at the traction it currently gets. But
that's a discussion for xml-dev :-)
Julian
--
<green/>bytes GmbH -- http://www.greenbytes.de -- tel:+492512807760
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: issue #1954 (was: Re: Supporting non-XML-safe pathnames)
Posted by "Peter N. Lundblad" <pe...@famlundblad.se>.
On Wed, 1 Dec 2004 kfogel@collab.net wrote:
> > > Okay. +1 on prohibiting control characters except LF, CR, TAB. I
> > > doubt any users are going to suffer much if we do that. There doesn't
> > > seem to be much opposition to it on this list either (am I forgetting
> > > anyone?), and clearly it'll make Peter Lundblad very happy :-).
> >
> > Hehehehe... Somewhere, I rembember having seen something about newlines
> > and the dump format. If there is a problem there, we need to ban it as
> > well. I don't know. Else, +1 from me as well.
>
> AAAAaaargh. I misspoke. Let me try again:
>
> Valid Subversion paths are UTF8 strings, but with no control chars
> allowed except for TAB.
>
> That is what I meant to +1.
>
+1. We can loosen this if we want when the world is XML 1.1. But htat's
another year I suppose.
Regards,
//Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: issue #1954
Posted by VK Sameer <sa...@collab.net>.
On Sat, 2004-12-04 at 23:38, kfogel@collab.net wrote:
> Branko Čibej <br...@xbc.nu> writes:
> > kfogel@collab.net wrote:
> > >Are you asking about the svn_ctype_* functions? They support exactly
> > >what they're documented to support: all of UTF8, including both the
> > >ASCII and non-ASCII parts of UTF8.
> >
> > Ah... actually, those functions can't classify a UTF-8 multibyte
> > sequence. They can only tell you if a particular byte is /valid/ in
> > UTF-8, not if the whole sequence is actually valid. We have
> > utf_validate.c for that.
>
> Sorry, yes. I was being careless; I knew we had the functionality in
> Subversion, I just didn't bother to think about exactly *where*. VK
> Sameer, hope I didn't waste any of your time!
No problem, Karl, I hadn't thought of multi-byte sequences yet :)
Sameer
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: issue #1954
Posted by kf...@collab.net.
Branko Čibej <br...@xbc.nu> writes:
> kfogel@collab.net wrote:
> >Are you asking about the svn_ctype_* functions? They support exactly
> >what they're documented to support: all of UTF8, including both the
> >ASCII and non-ASCII parts of UTF8.
>
> Ah... actually, those functions can't classify a UTF-8 multibyte
> sequence. They can only tell you if a particular byte is /valid/ in
> UTF-8, not if the whole sequence is actually valid. We have
> utf_validate.c for that.
Sorry, yes. I was being careless; I knew we had the functionality in
Subversion, I just didn't bother to think about exactly *where*. VK
Sameer, hope I didn't waste any of your time!
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: issue #1954
Posted by Branko Čibej <br...@xbc.nu>.
kfogel@collab.net wrote:
>Are you asking about the svn_ctype_* functions? They support exactly
>what they're documented to support: all of UTF8, including both the
>ASCII and non-ASCII parts of UTF8.
>
Ah... actually, those functions can't classify a UTF-8 multibyte
sequence. They can only tell you if a particular byte is /valid/ in
UTF-8, not if the whole sequence is actually valid. We have
utf_validate.c for that.
-- Brane
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: issue #1954
Posted by kf...@collab.net.
VK Sameer <sa...@collab.net> writes:
> Sorry, I mis-stated the problem - the idea is to create a file and
> directory with non-XML-safe+non-ASCII pathnames, not non-ASCII content.
Ah. Well, we can't include that in the repository, for obvious
reasons, so the option below...
> > Sure, that's one way (depending on how the tests work, it may turn out
> > to be better to just include the data within the Python test file
> > itself).
>
> OK, it's probably easiest to create Unicode strings in Python.
... is probably the best way.
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: issue #1954
Posted by VK Sameer <sa...@collab.net>.
On Mon, 2004-12-06 at 11:41, kfogel@collab.net wrote:
> VK Sameer <sa...@collab.net> writes:
> > > Or just tell us what example strings you need, and we can attach
> > > a file to the issue, containing those strings.
> >
> > I'd like to add creating a directory and file with non-ASCII characters
> > to the regression tests.
Sorry, I mis-stated the problem - the idea is to create a file and
directory with non-XML-safe+non-ASCII pathnames, not non-ASCII content.
> Sure, that's one way (depending on how the tests work, it may turn out
> to be better to just include the data within the Python test file
> itself).
OK, it's probably easiest to create Unicode strings in Python.
> There's no need to resolve those details now, though. The main thing
> is the new svn code, and the new test code. I've attached some UTF8
> test data to issue #1954, let me know if it helps. (Also, I forgot
> that the COMMITTERS file in the top level of the Subversion trunk is a
> UTF8 file too.)
Thanks
Sameer
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: issue #1954
Posted by kf...@collab.net.
VK Sameer <sa...@collab.net> writes:
> > Or just tell us what example strings you need, and we can attach
> > a file to the issue, containing those strings.
>
> I'd like to add creating a directory and file with non-ASCII characters
> to the regression tests.
Sure, that's one way (depending on how the tests work, it may turn out
to be better to just include the data within the Python test file
itself).
There's no need to resolve those details now, though. The main thing
is the new svn code, and the new test code. I've attached some UTF8
test data to issue #1954, let me know if it helps. (Also, I forgot
that the COMMITTERS file in the top level of the Subversion trunk is a
UTF8 file too.)
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: issue #1954
Posted by VK Sameer <sa...@collab.net>.
On Sat, 2004-12-04 at 00:03, kfogel@collab.net wrote:
> VK Sameer <sa...@collab.net> writes:
>
> Yes, #1954 is for 1.2. It need not go into 1.1.2 IMHO. Basically,
> just make your changes work with trunk, and everything should be fine.
OK.
> Btw, what editing environment do you use?
vim
> Or just tell us what example strings you need, and we can attach
> a file to the issue, containing those strings.
I'd like to add creating a directory and file with non-ASCII characters
to the regression tests.
Thanks
Sameer
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: issue #1954
Posted by kf...@collab.net.
VK Sameer <sa...@collab.net> writes:
> That would be very nice. A couple of questions, though. Is 1954 going to
> go into 1.1.2 or 1.2? svn_ctype_* functions have been marked with @since
> 1.2, so does relying on them for is_valid_pathname() automatically mean
> a 1.2 release?
Yes, #1954 is for 1.2. It need not go into 1.1.2 IMHO. Basically,
just make your changes work with trunk, and everything should be fine.
> Also, will/do those functions support non-ASCII, valid XML characters? I
> don't know how to enter those characters, and hence haven't been able to
> test myself.
Are you asking about the svn_ctype_* functions? They support exactly
what they're documented to support: all of UTF8, including both the
ASCII and non-ASCII parts of UTF8. Some of these characters are not
valid in XML, such as most of the control chars. And we've decided
not to support the few control chars that *are* valid XML anyway.
(Again, just to be clear, when I say "valid in XML", I mean "able to
be represented in XML, although XML-standard escape codes may be
necessary for some characters such as <, >, etc".)
Btw, what editing environment do you use? If it's GNU Emacs, I can
help you create non-ASCII characters. Or just tell us what example
strings you need, and we can attach a file to the issue, containing
those strings.
> OK, will do. Hope to send a first cut at the patch soon :)
Looking forward to it, thanks!
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: issue #1954
Posted by VK Sameer <sa...@collab.net>.
On Thu, 2004-12-02 at 22:20, kfogel@collab.net wrote:
> Branko Čibej <br...@xbc.nu> writes:
> > Julian Foad wrote:
> > > This is now sounding like a very good proposal.
> > >
> > > Let me just stick my neck out and ask, quoting C-Mike Pilato, "GACK!
> > > Exceptions! Why in Heaven's name must there always be exceptions?!"
> > > Not this this necessarily deserves such a strong reaction, but it's
> > > always worth asking ourselves this question.
> > >
> > > Is there any known substantial reason for allowing TAB? I know it's
> > > easy enough to allow it, and some people might have file names with
> > > TAB in them ... but I'd like a stronger reason such as evidence that
> > > people actually do have such file names (and not just in those weird
> > > cases where they also have other control characters). Another
> > > strong reason could be compatibility with some existing filename
> > > standard.
> > >
> > > If there is no strong reason, then I would suggest not allowing TAB.
> > > As I keep saying recently, it's very easy to relax a requirement in
> > > future if the need arises, whereas it is hard to tighten it.
> >
> > Not allowing tabs would also simplify the validity test to the point of
> >
> > svn_ctype_isutf8(c) && !svn_ctype_iscntrl(c)
> >
> > And I can add a macro that'll do this in a /single/ lookup into
> > svn_ctype_table. :-)
That would be very nice. A couple of questions, though. Is 1954 going to
go into 1.1.2 or 1.2? svn_ctype_* functions have been marked with @since
1.2, so does relying on them for is_valid_pathname() automatically mean
a 1.2 release?
Also, will/do those functions support non-ASCII, valid XML characters? I
don't know how to enter those characters, and hence haven't been able to
test myself.
> If we want to start out disallowing TABS and see how that works out,
> that's fine with me. This would mean my mail to VK Sameer just now...
>
> > Whups! I'm not sure if you mistyped above, or if you misunderstood
> > what I wrote. My language was technically accurate, but perhaps
> > needlessly terse. Let's try again:
> >
> > Valid Subversion paths are a *subset* of valid UTF8 strings.
> > Specifically, they are UTF8 strings with no control characters.
> > Except, one particular control character is allowed: TAB. All
> > other control characters are prohibited.
> >
> > So, TAB *is* allowed. Other control chars are not. Still +1, right?
>
> ...would be amended: TAB would disallowed after all.
>
> Since I'm +1 on both ways, and don't even have a strong opinion on the
> question, I'll drop out at this point. VK Sameer, go ahead and do
> whichever you think is best :-). I guess if you don't have a strong
> preference either, then just prohibit TABS, because it's one less
> exception to document.
OK, will do. Hope to send a first cut at the patch soon :)
Regards
Sameer
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: issue #1954
Posted by kf...@collab.net.
Branko Čibej <br...@xbc.nu> writes:
> Julian Foad wrote:
> > This is now sounding like a very good proposal.
> >
> > Let me just stick my neck out and ask, quoting C-Mike Pilato, "GACK!
> > Exceptions! Why in Heaven's name must there always be exceptions?!"
> > Not this this necessarily deserves such a strong reaction, but it's
> > always worth asking ourselves this question.
> >
> > Is there any known substantial reason for allowing TAB? I know it's
> > easy enough to allow it, and some people might have file names with
> > TAB in them ... but I'd like a stronger reason such as evidence that
> > people actually do have such file names (and not just in those weird
> > cases where they also have other control characters). Another
> > strong reason could be compatibility with some existing filename
> > standard.
> >
> > If there is no strong reason, then I would suggest not allowing TAB.
> > As I keep saying recently, it's very easy to relax a requirement in
> > future if the need arises, whereas it is hard to tighten it.
>
> Not allowing tabs would also simplify the validity test to the point of
>
> svn_ctype_isutf8(c) && !svn_ctype_iscntrl(c)
>
> And I can add a macro that'll do this in a /single/ lookup into
> svn_ctype_table. :-)
If we want to start out disallowing TABS and see how that works out,
that's fine with me. This would mean my mail to VK Sameer just now...
> Whups! I'm not sure if you mistyped above, or if you misunderstood
> what I wrote. My language was technically accurate, but perhaps
> needlessly terse. Let's try again:
>
> Valid Subversion paths are a *subset* of valid UTF8 strings.
> Specifically, they are UTF8 strings with no control characters.
> Except, one particular control character is allowed: TAB. All
> other control characters are prohibited.
>
> So, TAB *is* allowed. Other control chars are not. Still +1, right?
...would be amended: TAB would disallowed after all.
Since I'm +1 on both ways, and don't even have a strong opinion on the
question, I'll drop out at this point. VK Sameer, go ahead and do
whichever you think is best :-). I guess if you don't have a strong
preference either, then just prohibit TABS, because it's one less
exception to document.
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: issue #1954
Posted by Branko Čibej <br...@xbc.nu>.
Julian Foad wrote:
> kfogel@collab.net wrote:
>
>> Valid Subversion paths are UTF8 strings, but with no control chars
>> allowed except for TAB.
>
>
> This is now sounding like a very good proposal.
>
> Let me just stick my neck out and ask, quoting C-Mike Pilato, "GACK!
> Exceptions! Why in Heaven's name must there always be exceptions?!"
> Not this this necessarily deserves such a strong reaction, but it's
> always worth asking ourselves this question.
>
> Is there any known substantial reason for allowing TAB? I know it's
> easy enough to allow it, and some people might have file names with
> TAB in them ... but I'd like a stronger reason such as evidence that
> people actually do have such file names (and not just in those weird
> cases where they also have other control characters). Another strong
> reason could be compatibility with some existing filename standard.
>
> If there is no strong reason, then I would suggest not allowing TAB.
> As I keep saying recently, it's very easy to relax a requirement in
> future if the need arises, whereas it is hard to tighten it.
Not allowing tabs would also simplify the validity test to the point of
svn_ctype_isutf8(c) && !svn_ctype_iscntrl(c)
And I can add a macro that'll do this in a /single/ lookup into
svn_ctype_table. :-)
-- Brane
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: issue #1954
Posted by Julian Foad <ju...@btopenworld.com>.
kfogel@collab.net wrote:
> Valid Subversion paths are UTF8 strings, but with no control chars
> allowed except for TAB.
This is now sounding like a very good proposal.
Let me just stick my neck out and ask, quoting C-Mike Pilato, "GACK!
Exceptions! Why in Heaven's name must there always be exceptions?!" Not this
this necessarily deserves such a strong reaction, but it's always worth asking
ourselves this question.
Is there any known substantial reason for allowing TAB? I know it's easy
enough to allow it, and some people might have file names with TAB in them ...
but I'd like a stronger reason such as evidence that people actually do have
such file names (and not just in those weird cases where they also have other
control characters). Another strong reason could be compatibility with some
existing filename standard.
If there is no strong reason, then I would suggest not allowing TAB. As I keep
saying recently, it's very easy to relax a requirement in future if the need
arises, whereas it is hard to tighten it.
- Julian
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: issue #1954 (was: Re: Supporting non-XML-safe pathnames)
Posted by kf...@collab.net.
"Peter N. Lundblad" <pe...@famlundblad.se> writes:
> > I'm happy with just UTF-8, that is, no control chars except LF, CR,
> > and TAB.
>
> By "just UTF-8" it seems like you're still mixing what's valid UTF-8 and
> what are valid XML 1.0 characters. Just wanting to make sure there is no
> confusion left here. Control chars are valid UTF-8 but invalid in XML 1.0,
> wherever they occur. You just can't have them, without another layer of
> encoding. And a correct XML parser shouldn't give the application any
> control chars (except the whitespace ones named above).
Yes. Sorry -- I'm totally clear on it now, but earlier today I was
still wrecked among heathen dreams.
What you say above matches my understanding.
> > Okay. +1 on prohibiting control characters except LF, CR, TAB. I
> > doubt any users are going to suffer much if we do that. There doesn't
> > seem to be much opposition to it on this list either (am I forgetting
> > anyone?), and clearly it'll make Peter Lundblad very happy :-).
>
> Hehehehe... Somewhere, I rembember having seen something about newlines
> and the dump format. If there is a problem there, we need to ban it as
> well. I don't know. Else, +1 from me as well.
AAAAaaargh. I misspoke. Let me try again:
Valid Subversion paths are UTF8 strings, but with no control chars
allowed except for TAB.
That is what I meant to +1.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: issue #1954 (was: Re: Supporting non-XML-safe pathnames)
Posted by "Peter N. Lundblad" <pe...@famlundblad.se>.
On Wed, 1 Dec 2004 kfogel@collab.net wrote:
> VK Sameer <sa...@collab.net> writes:
> > Wouldn't this depend on SVN's definition of a valid pathname? Could that
> > be settled first?
>
> I'm happy with just UTF-8, that is, no control chars except LF, CR,
> and TAB.
>
By "just UTF-8" it seems like you're still mixing what's valid UTF-8 and
what are valid XML 1.0 characters. Just wanting to make sure there is no
confusion left here. Control chars are valid UTF-8 but invalid in XML 1.0,
wherever they occur. You just can't have them, without another layer of
encoding. And a correct XML parser shouldn't give the application any
control chars (except the whitespace ones named above).
> > Once the definition is settled, if a valid pathname contains UTF8
> > characters not in the XML 1.0 legal character set, then it would require
> > escaping/base-64 encoding before use as an attribute value.
>
> Okay. +1 on prohibiting control characters except LF, CR, TAB. I
> doubt any users are going to suffer much if we do that. There doesn't
> seem to be much opposition to it on this list either (am I forgetting
> anyone?), and clearly it'll make Peter Lundblad very happy :-).
>
Hehehehe... Somewhere, I rembember having seen something about newlines
and the dump format. If there is a problem there, we need to ban it as
well. I don't know. Else, +1 from me as well.
Regards,
//Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org