XMLCh and wchar_t, again

[Andy Heninger <an...@jtcsv.com>]

I would like to reopen the earlier discussion regarding the
what the type of XMLCh is, or should be, and how it relates
to wchar_t

The question is important right now because there will be
a new version of XML4C coming out at the end of the month
from IBM, and we will make a stable xerces-c release at
the same time, to take advantage of the XML4C testing.  If
there's going to be a change, now is a good time to do it.

Here is how thing are in xerces-c 3.2

Platform      wchar_t    wchar_t    XMLCh   XMLCh
              size       encoding   size    encoding 
=====================================================
Windows       16         utf-16     16      utf-16
Windows w 
  Borland C++ 32           ???      16      utf-16
Linux         32         UCS-4      32      utf-16
Solaris       32         UCS-4      32      utf-16
HPUX          32   locale-dependent 16      utf-16
AIX           16         utf-16     16      utf-16
390           32?  extended ebcdic  16      utf-16


On Windows and AIX, we have perfect compatibility between
XMLCh * strings and wchar_t * strings.

On Linux and Solaris, we have limited compatibility.  There
will be failures on any Unicode characters with a
value greater than 64K.  While such characters are not yet
widely used, they have been defined, and new ones continue
to be defined.  Software systems that purport to support
Unicode, but fail on these characters, are setting themselves
up for trouble as time progresses.

On HPUX and IBM System 390s, there is no compatibility between
XMLCh and wchar_t

Considering the above, portable code can never make any
assumptions that any useful relationship exists between
any form of Unicode encoding and wchar_t.


Another factor,
the DOM has memory footprint problems with the 32 bit XMLCh
on Solaris and Linux.  When using the DOM, the entire XML
document resides in memory, and using  a 32 bit XMLCh rather
than 16 bit exactly doubles the storage requirement for the
text itself  (there's additional memory needed for the document
structure that is not affected, so the overall total is not
doubled, but still, the hit is very significant.)


I think that we should switch back to a uniform 16 bit XMLCh
on all platforms.  (This is how Xerces was first released)
Since XMLCh is encoded as UTF-16, 16 bit storage is
guaranteed to work; any Unicode character > 64k will be
encoded as a surrogate pair of utf-16 values.

The downside of making this change is that applications on Solaris 
or Linux that pass XMLCh * strings to library functions expecting
a wchar_t * will fail.  But such applications are already
suspect - the code is non-portable, and even on platforms
where it "works", it will still fail if a character code > 64 k is
encountered.

I'd like to hear everyone's opinion.
(Dean, I already know that you disagree, but fire away, just 
to make sure that the opposing view is given a good airing.)

Andy Heninger
IBM XML Technology Group, Cupertino, CA
heninger@us.ibm.com

Re: XMLCh and wchar_t, again

[Dean Roddey <dr...@charmedquark.com>]

"I also find the ability to transcode from one charset to another to be
useless in all but the EBCDIC case. "

You don't, but others probably do. And its not just in and out that's
important, its also being able to deal with all data internally using one
set of code, which is certainly going to be important for many applications.

"Here's the crazy idea: how about building a family of XML/XSLT pairs around
the eight bit character encodings? One each for Latin1-Latin1, UTF8-UTF8,
ShiftJIS, etc.."

I honestly think that would be insanely difficult, both in implementation
(since you still have to have the UTF-16, UCS-4 variations as well), and in
build and test cycles.

--------------------------
Dean Roddey
The CIDLib C++ Frameworks
Charmed Quark Software
droddey@charmedquark.com
http://www.charmedquark.com

"You young, and you gotcha health. Whatchoo wanna job fer?"

Re: XMLCh and wchar_t, again

[Jon Smirl <jo...@mediaone.net>]

I'm working in an environment which is pretty much totally Latin1. Unicode
needs could be handled by setting up a parallel server. What I'd like to do
is keep pure Latin1 through my complete cycle. Database, XML, XSLT, browser.

The database engine I work with, Sybase, does not support UCS2/4 it is UTF8
based (or Latin1, etc). Since XML/XSLT only do UCS2 internally I'm switching
my databases from Latin1 to UTF8. I figure than since Unicode is being
forced on to me I might as well start to support it. I'm still stuck with
two transcodes: UTF8-UCS2, UCS2-Latin1 and double memory
consumption/processing time because of two bytes vs one.

I also find the ability to transcode from one charset to another to be
useless in all but the EBCDIC case. There's just no purpose in transcoding a
Unicode document written in Japanese into Latin1. The only other purpose I
can see is a mixed charset documents, but can a browser display these?

Here's the crazy idea: how about building a family of XML/XSLT pairs around
the eight bit character encodings? One each for Latin1-Latin1, UTF8-UTF8,
ShiftJIS, etc..

Jon Smirl
jonsmirl@mediaone.net

Re: XMLCh and wchar_t, again

[Dean Roddey <dr...@charmedquark.com>]

If there is not a desire to make the parser spit out UCS-4 when that's the
native format, then I guess there isn't much else to do if you aren't
willing to just live with the issue as it stands. If I were a Linux
developer though, I'd not be too happy if I had to do yet another transcode
of the data in order to use it, when in almost all cases it would have been
prefectly happy anyway if it had just been transcoded into a 4 byte XMLCh to
begin with.

--------------------------
Dean Roddey
The CIDLib C++ Frameworks
Charmed Quark Software
droddey@charmedquark.com
http://www.charmedquark.com

"You young, and you gotcha health. Whatchoo wanna job fer?"


----- Original Message -----
From: "Andy Heninger" <an...@jtcsv.com>
To: <xe...@xml.apache.org>
Sent: Tuesday, August 15, 2000 4:37 PM
Subject: XMLCh and wchar_t, again



> I'd like to hear everyone's opinion.
> (Dean, I already know that you disagree, but fire away, just
> to make sure that the opposing view is given a good airing.)
>