You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@maven.apache.org by Dan Fabulich <da...@fabulich.com> on 2008/02/11 21:49:48 UTC
Nulls in XML
I didn't know this, so I imagine others might not. The string "�" is
invalid XML. The character is simply not allowed in XML in any
representation. XML 1.0 standard blocks most of the characters under x20,
allowing only x9 xA and xD. XML 1.1 allows x1-x20, but still blocks x0.
http://www.w3.org/TR/1998/REC-xml-19980210#charsets
http://www.w3.org/TR/xml11/#charsets
This creates an interesting problem for serializing Java strings
containing the null character, e.g. "\u0000", or for other non-whitespace
control characters like the bell character "\u0007". We've got an
integration test for this case in Surefire, and it does entirely the wrong
thing (SUREFIRE-455).
In the patch submitted to that bug, Todor throws away nulls in his XML
escaper, silently omitting them from the output; all other control
characters (even the 1.0-illegal ones) pass through. That doesn't seem
right, especially when we're talking about test results! (Expected "" but
was "" ... Just imagine how painful it would be to track something like
that down.)
But neither does it seem right to insert "�" when it's illegal XML.
Notably, Java will cheerfully print � in XML if you tell it to do so,
and many parsers will figure out what to do with it just fine; the same
applies to "".
Thoughts? Should we emit "�", standards-be-damned? Silently omit the
character? Print a "?" instead? Something else?
-Dan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Nulls in XML
Posted by Benjamin Bentmann <be...@udo.edu>.
> Should we emit "�", standards-be-damned?
Dangerous: If some parser goes fully-XML-compliant, they will blame Surefire
again. So just emitting "�" seems not really a long-term solution.
> (Expected "" but was "" ... Just imagine how painful it would be to track
> something like that down.)
> [...]
> Silently omit the character?
I fully agree, that is painful.
> Print a "?" instead?
Well, if altering the original string is acceptable, why not replace invalid
characters with dedicated codes such that users can deduct their original
contents? For example, replace invalid characters with their literal Unicode
escape, e.g. print
Expected "" but was "\u0000"
instead of
Expected "" but was "?"
> Something else?
If retaining the original string contents is the ulitmate goal, a reliable
approach would be to preprocess strings with some encoding scheme (Base64,
C-style escaping, etc.) before emitting to the XML. Of course, this would
require a decoding on the other side and also some additions to the XML
format (i.e. different attribute to hold the encoded string or additional
attribute to flag some other attribute/element as being encoded).
Regards,
Benjamin Bentmann
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Nulls in XML
Posted by Jochen Wiedmann <jo...@gmail.com>.
On Feb 12, 2008 7:17 PM, Dan Fabulich <da...@fabulich.com> wrote:
> When a test fails (by throwing a Throwable) Surefire needs to output the
> message string of the failure in the XML test result output; Java strings
> may contain the null character \u0000. Therefore, not all Java strings
> can be saved as XML, and so the task is impossible.
Understood. My personal recommendation for such cases would be to use
the same encoding (\u0000), as it is very readable for Java
programmers.
Jochen
--
Look, that's why there's rules, understand? So that you think before
you break 'em.
-- (Terry Pratchett, Thief of Time)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Nulls in XML
Posted by Dan Fabulich <da...@fabulich.com>.
Jochen Wiedmann wrote:
> As he wrote, these characters are illegal, regardless of encoding,
> CDATA, and so on. XML is a text format and ASCII 0 is a binary
> character.
>
> The suggested way to embed binary data into XML is using the base64 encoding.
>
> Apart from that, I do not understand the use case. Where do we need to
> embed binary data into XML?
When a test fails (by throwing a Throwable) Surefire needs to output the
message string of the failure in the XML test result output; Java strings
may contain the null character \u0000. Therefore, not all Java strings
can be saved as XML, and so the task is impossible.
The trivial example is:
junit.framework.Assert.fail("\u0000");
A more realistic example would be:
String value = blah(); // returns \u0000
junit.framework.Assert.assertEquals("", value);
My current best resolution for this is to deliberately double-encode these
characters, so you'd see in the XML:
<failure type="junit.framework.AssertionFailedError" message="&#0;">
That's literally wrong, but I don't know of anything better to do. I've
filed this as bug SUREFIRE-456, targeted for "Future" because I have no
idea when/if we'd ever fix this.
-Dan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Nulls in XML
Posted by Jochen Wiedmann <jo...@gmail.com>.
On Feb 12, 2008 1:26 AM, Jason van Zyl <ja...@maven.org> wrote:
> Where did you run into this? I think not showing what it actually is
> makes it immediately not obvious what's going wrong. So I'm for
> showing what it actually is. Can you just wrap in CDATA?
As he wrote, these characters are illegal, regardless of encoding,
CDATA, and so on. XML is a text format and ASCII 0 is a binary
character.
The suggested way to embed binary data into XML is using the base64 encoding.
Apart from that, I do not understand the use case. Where do we need to
embed binary data into XML?
Jochen
--
Look, that's why there's rules, understand? So that you think before
you break 'em.
-- (Terry Pratchett, Thief of Time)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Nulls in XML
Posted by Dan Fabulich <da...@fabulich.com>.
Jason van Zyl wrote:
> Where did you run into this?
One of our integration tests says:
junit.framework.Assert.fail("\u0000");
When it's as bald as that it's not very important, but it's considerably
more likely when you Assert.assertEquals(expected, actual) where they both
contain control characters (e.g. expected \u0001 but was \u0002).
> I think not showing what it actually is makes it immediately not obvious
> what's going wrong. So I'm for showing what it actually is.
I agree, but my research suggests that it's impossible. :-(
> Can you just wrap in CDATA?
Nope, you can't put a null character in a CDATA block... the parser will
object. (For reasons like this SGML had "NDATA" and "SDATA" blocks in
addition to CDATA.)
[Of course you can write: <![CDATA[�]]> but that's because the parser
ignores the entity entirely. That parses as 5 characters instead of one
null character.]
-Dan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Nulls in XML
Posted by Jason van Zyl <ja...@maven.org>.
On 11-Feb-08, at 12:49 PM, Dan Fabulich wrote:
>
> I didn't know this, so I imagine others might not. The string
> "�" is invalid XML. The character is simply not allowed in XML
> in any representation. XML 1.0 standard blocks most of the
> characters under x20, allowing only x9 xA and xD. XML 1.1 allows x1-
> x20, but still blocks x0.
>
> http://www.w3.org/TR/1998/REC-xml-19980210#charsets
> http://www.w3.org/TR/xml11/#charsets
>
> This creates an interesting problem for serializing Java strings
> containing the null character, e.g. "\u0000", or for other non-
> whitespace control characters like the bell character "\u0007".
> We've got an integration test for this case in Surefire, and it does
> entirely the wrong thing (SUREFIRE-455).
>
> In the patch submitted to that bug, Todor throws away nulls in his
> XML escaper, silently omitting them from the output; all other
> control characters (even the 1.0-illegal ones) pass through. That
> doesn't seem right, especially when we're talking about test
> results! (Expected "" but was "" ... Just imagine how painful it
> would be to track something like that down.)
>
> But neither does it seem right to insert "�" when it's illegal
> XML. Notably, Java will cheerfully print � in XML if you tell it
> to do so, and many parsers will figure out what to do with it just
> fine; the same applies to "".
>
> Thoughts? Should we emit "�", standards-be-damned? Silently
> omit the character? Print a "?" instead? Something else?
>
Where did you run into this? I think not showing what it actually is
makes it immediately not obvious what's going wrong. So I'm for
showing what it actually is. Can you just wrap in CDATA?
> -Dan
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
Thanks,
Jason
----------------------------------------------------------
Jason van Zyl
Founder, Apache Maven
jason at sonatype dot com
----------------------------------------------------------
happiness is like a butterfly: the more you chase it, the more it will
elude you, but if you turn your attention to other things, it will come
and sit softly on your shoulder ...
-- Thoreau
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Nulls in XML
Posted by Dan Fabulich <da...@fabulich.com>.
Dan Fabulich wrote:
> But neither does it seem right to insert "�" when it's illegal XML.
> Notably, Java will cheerfully print � in XML if you tell it to do so, and
> many parsers will figure out what to do with it just fine; the same applies
> to "".
Notably, Java's XML parser does NOT know what to do with "�" or
"", despite the fact that Java will emit the character if you ask it
to.
org.xml.sax.SAXParseException: Character reference "�" is an invalid XML character.
at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:236)
at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:215)
at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:386)
at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:316)
at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1438)
at com.sun.org.apache.xerces.internal.impl.XMLScanner.scanCharReferenceValue(XMLScanner.java:1304)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanCharReference(XMLDocumentFragmentScannerImpl.java:1259)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFragmentScannerImpl.java:1753)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:368)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:834)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764)
at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:148)
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1242)
at javax.xml.parsers.SAXParser.parse(SAXParser.java:375)
at javax.xml.parsers.SAXParser.parse(SAXParser.java:311)
-Dan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org