Character set ignorance

[Richard Rowell <ri...@bowmansystems.com>]

I recieved my first "real" XML file for import today from a windows
user, it was marked as being UTF-8 encoded.  It however had 0x92
everywhere that there should have been a single quote.  Xerces did not
seem to care, but Xalan puked immediatly.  My ignorance about character
sets is monumental and I'm assuming most of my clients will be in the
same boat.  Can anyone point me towards some gentle documentation about
charactersets in XML and possibly what encoding they windows folks
should specify if they are gonna use characters like a forward tick in
place of single quotes?

Re: Character set ignorance

["Ushakov, Sergey N" <us...@int.com.ru>]

Richard, it is still difficult to make any conclusions regarding your issue
as the information you have posted misses such important parts as XML and
XSL prologues/declarations. If you post something more complete it would be
much easier... The same about verbatim complaints from Xalan.

Regards,
Sergey



----- Original Message -----
From: "Richard Rowell" <ri...@bowmansystems.com>
To: "xalan-j-users" <xa...@xml.apache.org>
Sent: Monday, September 30, 2002 6:44 PM
Subject: Re: Character set ignorance

On Thu, 2002-09-26 at 19:05, Ushakov, Sergey N wrote:
> Richard, could you post some minimal XML and XSL input and corresponding
> output? It would be easier to judge then...
>

The offending record looked like this:
<case_assessment date_added="2001-06-26T00:00:00"
date_updated="2001-06-26T00:00:00">
             <notes>I don~Rt believe he is clean an...</notes>
</case_assessment>


The ~R is how vim displays the forward tick (my keyboard does not have a
key for that character).  I am suprised that xerces did not reject the
file, as I pre-process with a simple class that is just validating
Xerces SAX parser (to assure the input is valid XML before it ever gets
into the import process). I had hoped it would have kicked out an
exception since this is clearly not UTF8.

Is there any way to verify that the input is UTF8 before it gets to
Xalan?  If Xerces can not/will not point out problems with character
sets, perhaps someone knows of some other tool that will?

Re: Character set ignorance

[Richard Rowell <ri...@bowmansystems.com>]

On Thu, 2002-09-26 at 19:05, Ushakov, Sergey N wrote:
> Richard, could you post some minimal XML and XSL input and corresponding
> output? It would be easier to judge then...
> 

The offending record looked like this:
<case_assessment date_added="2001-06-26T00:00:00"
date_updated="2001-06-26T00:00:00">
             <notes>I don~Rt believe he is clean an...</notes>
</case_assessment>


The ~R is how vim displays the forward tick (my keyboard does not have a
key for that character).  I am suprised that xerces did not reject the
file, as I pre-process with a simple class that is just validating
Xerces SAX parser (to assure the input is valid XML before it ever gets
into the import process). I had hoped it would have kicked out an
exception since this is clearly not UTF8.  

Is there any way to verify that the input is UTF8 before it gets to
Xalan?  If Xerces can not/will not point out problems with character
sets, perhaps someone knows of some other tool that will?

Re: Character set ignorance

["Ushakov, Sergey N" <us...@int.com.ru>]

Richard, could you post some minimal XML and XSL input and corresponding
output? It would be easier to judge then...

Regards,
Sergey


----- Original Message -----
From: "Richard Rowell" <ri...@bowmansystems.com>
To: "xalan-j-users" <xa...@xml.apache.org>
Sent: Thursday, September 26, 2002 10:10 PM
Subject: Character set ignorance

Re: Character set ignorance

[Joseph Kesselman <ke...@us.ibm.com>]

Only the basic ASCII-range quotes and apostrophies can be used to delimit 
XML attributes, not the fancy word-processor versions that some Windows 
tools insist on hitting us with. See the XML spec. 

If you've got an encoding which maps 0x92 to one or the other of these, 
you may be able to get away with it... but then you have to specify that 
encoding as part of the document's <?xml?> declaration.

I'm rather amazed that you say Xerces didn't complain. I suspect you're 
fooling yourself, since Xalan normally uses Xerces to read the file.

______________________________________
Joe Kesselman  / IBM Research