You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Marcus Sundman <su...@iki.fi> on 2004/06/24 14:23:48 UTC
How does subversion handle encodings?
I've searched all subversion docs I could find but the only mention of
encoding I have found is regarding commit messages.
People on a project often use different system. This is especially common
for OSS projects where people often are in very different parts of the
world. Different platforms/locales use different line breaks and text
encodings, and most developer tools expect to find all files in whatever
happens to be the default local system text encoding and using the line
break convention of the local OS. Therefore we are faced with three
options:
A) Get all systems to standardize on one encoding and one type of line
breaks.
B) Build support for different encodings and line breaks into each tool and
supply the necessary metadata at each invokation of each such tool.
C) Transcode text files in input/output so that they are always in the
default local system encoding and using the line breaks convention of the
local OS.
I think the first two options are out of the question while the third one is
relatively easy to implement, so the decision seems like a no-brainer to
me.
Most content management (CM) systems already convert line breaks in text
files between the unix, mac and windows types. Text encoding is even more
important since it has far reaching consequences (if you have written "foo"
you do not want it to say "bar" when it's in production code) and errors
can be very hard to detect. Despite this fact many CM systems seem to
ignore the issue completely, relying on projects adopting some kind of
hybrid policy between A and B above, and communicating the needed metadata
(i.e. what encodings and line breaks to use for which files) per email or
even face to face. As can be expected, the results of such ad-hoc practices
can be very ugly.
So, how does subversion tackle these issues?
- Marcus Sundman
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: How does subversion handle encodings?
Posted by Marcus Sundman <su...@iki.fi>.
> the "Content-Type" specification as per RFC 2045 and 2046, including the
> handling of "encoding" attribute of text types
Typo: "encoding"=>"charset" (MIME thinks character encoding is called
"charset" although it is more than just a character repertoire).
- Marcus Sundman
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: How does subversion handle encodings?
Posted by Marcus Sundman <su...@iki.fi>.
On Monday 28 June 2004 20:00, Michael W Thelen wrote:
> * Marcus Sundman <su...@iki.fi> [2004-06-28 04:12]:
> > One thing is still not clear: Can subversion have a "type" metadata
> > associated with each file? (Not storing that crucial piece of info
> > would be really, really stupid since that would render most text files
> > useless as text files. That is, unless you send that piece of info in
> > some other way to everyone using those text files.)
>
> Yep, check out the svn:mime-type property:
> http://svnbook.red-bean.com/svnbook/ch07s02.html#svn-ch-7-sect-2.3.2
Great! I suppose the attribute fully conforms to the "Content-Type"
specification as per RFC 2045 and 2046, including the handling of
"encoding" attribute of text types (i.e. if not present then it is assumed
to be "us-ascii" (=7-bit ASCII)).
Does the various (visual) diff, merge et al. utilities take the encoding
into consideration when translating the bytes to characters?
- Marcus Sundman
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: How does subversion handle encodings?
Posted by Michael W Thelen <th...@cs.utah.edu>.
* Marcus Sundman <su...@iki.fi> [2004-06-28 04:12]:
> > Subversion, by default, will never touch file contents, period. No EOL
> > conversions, no keyword expansions, no nothing. By default, Subversion
> > treats all file contents as opaque collections of bytes.
> >
> > [...]
> >
> > * Subversion does absolutely nothing in terms of trying to understand
> > the encoding of a file's text (ascii vs. ucs2 vs. utf8, etc.) Bytes are
> > bytes. At most, it will change line-endings if you ask.
> >
> > Does that answer your questions?
>
> Mostly, although it was not the answer I was hoping for.
> One thing is still not clear: Can subversion have a "type" metadata
> associated with each file? (Not storing that crucial piece of info would be
> really, really stupid since that would render most text files useless as
> text files. That is, unless you send that piece of info in some other way
> to everyone using those text files.)
Yep, check out the svn:mime-type property:
http://svnbook.red-bean.com/svnbook/ch07s02.html#svn-ch-7-sect-2.3.2
-- Mike
--
Michael W. Thelen
Experience is what you get when you don't get what you want.
-- Dan Stanford
Re: How does subversion handle encodings?
Posted by Ulrich Eckhardt <ec...@satorlaser.com>.
Marcus Sundman wrote:
> On Thursday 24 June 2004 17:35, Ben Collins-Sussman wrote:
> > * Subversion does absolutely nothing in terms of trying to understand
> > the encoding of a file's text (ascii vs. ucs2 vs. utf8, etc.) Bytes are
> > bytes. At most, it will change line-endings if you ask.
> >
> > Does that answer your questions?
>
> Mostly, although it was not the answer I was hoping for.
> One thing is still not clear: Can subversion have a "type" metadata
> associated with each file?
I believe you can have 'properties' attached to a file. You'd just have to
make other tools agree to honor these properties.
> (Not storing that crucial piece of info would be
> really, really stupid since that would render most text files useless as
> text files. That is, unless you send that piece of info in some other way
> to everyone using those text files.)
I'd vote for "always use UTF-8" for files that need more than ASCII. That way
you're fine. Maybe adding a three-byte marker (codepoint 0xfeff encoded in
UTF-8) at the beginning of the file would help, too, as it's common practice
and understood by many tools.
just my 2cc
Uli
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: How does subversion handle encodings?
Posted by Marcus Sundman <su...@iki.fi>.
On Thursday 24 June 2004 17:35, Ben Collins-Sussman wrote:
> On Thu, 2004-06-24 at 09:23, Marcus Sundman wrote:
> > So, how does subversion tackle these issues?
>
> Subversion, by default, will never touch file contents, period. No EOL
> conversions, no keyword expansions, no nothing. By default, Subversion
> treats all file contents as opaque collections of bytes.
>
> [...]
>
> * Subversion does absolutely nothing in terms of trying to understand
> the encoding of a file's text (ascii vs. ucs2 vs. utf8, etc.) Bytes are
> bytes. At most, it will change line-endings if you ask.
>
> Does that answer your questions?
Mostly, although it was not the answer I was hoping for.
One thing is still not clear: Can subversion have a "type" metadata
associated with each file? (Not storing that crucial piece of info would be
really, really stupid since that would render most text files useless as
text files. That is, unless you send that piece of info in some other way
to everyone using those text files.)
- Marcus Sundman
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: How does subversion handle encodings?
Posted by Marcus Sundman <su...@iki.fi>.
On Sunday 27 June 2004 20:03, Greg Hudson wrote:
> Marcus Sundman wrote:
> > Most content management (CM) systems already convert line breaks in
> > text files between the unix, mac and windows types. Text encoding is
> > even more important since it has far reaching consequences (if you
> > have written "foo" you do not want it to say "bar" when it's in
> > production code) and errors can be very hard to detect. Despite this
> > fact many CM systems seem to ignore the issue completely [...]
>
> Ben wrote:
> > [A description of Subversion's current features in this area,
> > confirming that Subversion punts on character encodings of file
> > contents]
>
> There are a few things going on here:
>
> * Traditionally, version control systems have been used to manage
> source code, which is traditionally written in ASCII. Localized
> user messages typically come from PO files or the
> language-specific equivalent, which are stored in UTF-8 or in
> mixed charsets, not in the local charset.
The actual source code might be written in 7-bit us-ascii, but string
constants, comments, annotations etc. are not.
> * Although many development teams mix Windows and Mac or Unix
> machines, I bet not so many mix people using different charset
> encodings.
You'd certainly lose that bet. Although Windows-1252 and ISO-8859-1 are
quite similar they are not the same. E.g. a long dash in Windows-1252 is a
control character in ISO-8859-1. And then when the windows guys use command
line utilities they are suddenly using cp850 which is completely different.
(You can of course "chcp 1252", but that leads to a plethora of other
problems.)
Then we have many people working in different languages and they want to use
UTF-8 or similar multi-byte encoding, which isn't well supported on other
platforms.
All in all, many people are NOT using same character encodings, even though
they might think they are.
> At first glance, it would be consistent with Subversion's current
> feature offerings, and not a tremendous amount of code, to add a
> feature where you can set svn:encoding to "native" or to an LC_CHARSET
> value, and Subversion would transcode the file's contents from UTF-8
> to the stated encoding after newline and keyword translation.
I think the correct place for the encoding is in the "svn:mime-type" (see
RFC 2045), and then you'd need a boolean attribute "svn:auto-filter" which
indicates whether to transcode (including changing line breaks) on
input/output or not.
> The moment I use a character which
> can't be represented in your encoding, you can no longer check out
> that file properly.
That is already the case. There are workarounds, though. E.g., you could
replace each unsupported character with "{uN}" where N is the unicode code.
> > Therefore we are faced with three options:
> > A) Get all systems to standardize on one encoding and one type of
> > line breaks.
> >
> > I think the first two options are out of the question [...]
>
> While standardizing on one type of line breaks is likely to remain
> painful for a long time, standardizing on one encoding (specifically,
> UTF-8) seems like a great idea. Why do you say it's out of the
> question?
Great idea, yes, but still painful. Good luck trying to get UTF-8 to work
properly on windows or even on linux. Of course this would be best, but I
just don't see it as very realistic within a reasonable time frame.
Then once everyone have switched to UTF-8 we get the "line break issue"
fixed for free, since there is only one proper line break in unicode,
namely \u2028.
- Marcus Sundman
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: How does subversion handle encodings?
Posted by Greg Hudson <gh...@MIT.EDU>.
Marcus Sundman wrote:
> Most content management (CM) systems already convert line breaks in
> text files between the unix, mac and windows types. Text encoding is
> even more important since it has far reaching consequences (if you
> have written "foo" you do not want it to say "bar" when it's in
> production code) and errors can be very hard to detect. Despite this
> fact many CM systems seem to ignore the issue completely [...]
Ben wrote:
> [A description of Subversion's current features in this area,
> confirming that Subversion punts on character encodings of file
> contents]
There are a few things going on here:
* Traditionally, version control systems have been used to manage
source code, which is traditionally written in ASCII. Localized
user messages typically come from PO files or the
language-specific equivalent, which are stored in UTF-8 or in
mixed charsets, not in the local charset.
* Although many development teams mix Windows and Mac or Unix
machines, I bet not so many mix people using different charset
encodings.
* A version control system is supposed to be about versioning, not
so much about file interchange. (Perhaps a "CM system" is also
about file interchange, but Subversion isn't a CM system.) Adding
just the newline translation functions was bothersome to many
Subversion developers.
At first glance, it would be consistent with Subversion's current
feature offerings, and not a tremendous amount of code, to add a
feature where you can set svn:encoding to "native" or to an LC_CHARSET
value, and Subversion would transcode the file's contents from UTF-8
to the stated encoding after newline and keyword translation. (This
would make it extra-important to fix the way "svn diff" works, so that
it translates the text-base to wc format and diffs against the wc
file, instead of detranslating the wc file to text-base format and
diffing against the text base.)
I'd worry that such a feature would wind up being more trouble to
users than it was worth, though. The moment I use a character which
can't be represented in your encoding, you can no longer check out
that file properly.
Marcus wrote:
> Therefore we are faced with three options:
> A) Get all systems to standardize on one encoding and one type of
> line breaks.
> I think the first two options are out of the question [...]
While standardizing on one type of line breaks is likely to remain
painful for a long time, standardizing on one encoding (specifically,
UTF-8) seems like a great idea. Why do you say it's out of the
question?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: How does subversion handle encodings?
Posted by Ben Collins-Sussman <su...@collab.net>.
On Thu, 2004-06-24 at 09:23, Marcus Sundman wrote:
> So, how does subversion tackle these issues?
Subversion, by default, will never touch file contents, period. No EOL
conversions, no keyword expansions, no nothing. By default, Subversion
treats all file contents as opaque collections of bytes.
* If you want EOL translation or keyword expansion to happen on certain
files, you must activate these features using the svn:eol-style and
svn:keywords properties, documented in chapter 7.
* The Subversion repository demands that all paths and commit messages
be stored in UTF8. This means that when committing, the client will
attempt to convert local paths (and commit messages) to UTF8, and when
doing a checkout/update, it will attempt to convert UTF8 paths back into
native locale. So if a user commits a Chinese filename, that's fine,
but then you better make sure every other user has a locale capable of
representing that filename.
* Subversion does absolutely nothing in terms of trying to understand
the encoding of a file's text (ascii vs. ucs2 vs. utf8, etc.) Bytes are
bytes. At most, it will change line-endings if you ask.
Does that answer your questions?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org