You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@maven.apache.org by Dan Fabulich <da...@fabulich.com> on 2008/02/11 21:49:48 UTC

Nulls in XML

I didn't know this, so I imagine others might not.  The string "&#x0;" is 
invalid XML.  The character is simply not allowed in XML in any 
representation.  XML 1.0 standard blocks most of the characters under x20, 
allowing only x9 xA and xD.  XML 1.1 allows x1-x20, but still blocks x0.

http://www.w3.org/TR/1998/REC-xml-19980210#charsets
http://www.w3.org/TR/xml11/#charsets

This creates an interesting problem for serializing Java strings 
containing the null character, e.g. "\u0000", or for other non-whitespace 
control characters like the bell character "\u0007".  We've got an 
integration test for this case in Surefire, and it does entirely the wrong 
thing (SUREFIRE-455).

In the patch submitted to that bug, Todor throws away nulls in his XML 
escaper, silently omitting them from the output; all other control 
characters (even the 1.0-illegal ones) pass through.  That doesn't seem 
right, especially when we're talking about test results!  (Expected "" but 
was "" ... Just imagine how painful it would be to track something like 
that down.)

But neither does it seem right to insert "&#x0;" when it's illegal XML. 
Notably, Java will cheerfully print &#x0; in XML if you tell it to do so, 
and many parsers will figure out what to do with it just fine; the same 
applies to "&#x07;".

Thoughts?  Should we emit "&#x0;", standards-be-damned?  Silently omit the 
character?  Print a "?" instead?  Something else?

-Dan

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org


Re: Nulls in XML

Posted by Benjamin Bentmann <be...@udo.edu>.
> Should we emit "&#x0;", standards-be-damned?

Dangerous: If some parser goes fully-XML-compliant, they will blame Surefire 
again. So just emitting "&#x0;" seems not really a long-term solution.

> (Expected "" but was "" ... Just imagine how painful it would be to track 
> something like that down.)
> [...]
> Silently omit the character?

I fully agree, that is painful.

> Print a "?" instead?

Well, if altering the original string is acceptable, why not replace invalid 
characters with dedicated codes such that users can deduct their original 
contents? For example, replace invalid characters with their literal Unicode 
escape, e.g. print
  Expected "" but was "\u0000"
instead of
  Expected "" but was "?"

> Something else?

If retaining the original string contents is the ulitmate goal, a reliable 
approach would be to preprocess strings with some encoding scheme (Base64, 
C-style escaping, etc.) before emitting to the XML. Of course, this would 
require a decoding on the other side and also some additions to the XML 
format (i.e. different attribute to hold the encoded string or additional 
attribute to flag some other attribute/element as being encoded).

Regards,



Benjamin Bentmann 


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org


Re: Nulls in XML

Posted by Jochen Wiedmann <jo...@gmail.com>.
On Feb 12, 2008 7:17 PM, Dan Fabulich <da...@fabulich.com> wrote:

> When a test fails (by throwing a Throwable) Surefire needs to output the
> message string of the failure in the XML test result output; Java strings
> may contain the null character \u0000.  Therefore, not all Java strings
> can be saved as XML, and so the task is impossible.

Understood. My personal recommendation for such cases would be to use
the same encoding (\u0000), as it is very readable for Java
programmers.


Jochen


-- 
Look, that's why there's rules, understand? So that you think before
you break 'em.

    -- (Terry Pratchett, Thief of Time)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org


Re: Nulls in XML

Posted by Dan Fabulich <da...@fabulich.com>.
Jochen Wiedmann wrote:

> As he wrote, these characters are illegal, regardless of encoding,
> CDATA, and so on. XML is a text format and ASCII 0 is a binary
> character.
>
> The suggested way to embed binary data into XML is using the base64 encoding.
>
> Apart from that, I do not understand the use case. Where do we need to
> embed binary data into XML?

When a test fails (by throwing a Throwable) Surefire needs to output the 
message string of the failure in the XML test result output; Java strings 
may contain the null character \u0000.  Therefore, not all Java strings 
can be saved as XML, and so the task is impossible.

The trivial example is:

   junit.framework.Assert.fail("\u0000");

A more realistic example would be:

   String value = blah(); // returns \u0000
   junit.framework.Assert.assertEquals("", value);

My current best resolution for this is to deliberately double-encode these 
characters, so you'd see in the XML:

   <failure type="junit.framework.AssertionFailedError" message="&amp;#0;">

That's literally wrong, but I don't know of anything better to do.  I've 
filed this as bug SUREFIRE-456, targeted for "Future" because I have no 
idea when/if we'd ever fix this.

-Dan

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org


Re: Nulls in XML

Posted by Jochen Wiedmann <jo...@gmail.com>.
On Feb 12, 2008 1:26 AM, Jason van Zyl <ja...@maven.org> wrote:

> Where did you run into this? I think not showing what it actually is
> makes it immediately not obvious what's going wrong. So I'm for
> showing what it actually is. Can you just wrap in CDATA?

As he wrote, these characters are illegal, regardless of encoding,
CDATA, and so on. XML is a text format and ASCII 0 is a binary
character.

The suggested way to embed binary data into XML is using the base64 encoding.

Apart from that, I do not understand the use case. Where do we need to
embed binary data into XML?


Jochen



-- 
Look, that's why there's rules, understand? So that you think before
you break 'em.

    -- (Terry Pratchett, Thief of Time)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org


Re: Nulls in XML

Posted by Dan Fabulich <da...@fabulich.com>.
Jason van Zyl wrote:

> Where did you run into this?

One of our integration tests says:
junit.framework.Assert.fail("\u0000");

When it's as bald as that it's not very important, but it's considerably 
more likely when you Assert.assertEquals(expected, actual) where they both 
contain control characters (e.g. expected \u0001 but was \u0002).

> I think not showing what it actually is makes it immediately not obvious 
> what's going wrong. So I'm for showing what it actually is.

I agree, but my research suggests that it's impossible.  :-(

> Can you just wrap in CDATA?

Nope, you can't put a null character in a CDATA block... the parser will 
object.  (For reasons like this SGML had "NDATA" and "SDATA" blocks in 
addition to CDATA.)

[Of course you can write: <![CDATA[&#x0;]]> but that's because the parser 
ignores the entity entirely.  That parses as 5 characters instead of one 
null character.]

-Dan

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org


Re: Nulls in XML

Posted by Jason van Zyl <ja...@maven.org>.
On 11-Feb-08, at 12:49 PM, Dan Fabulich wrote:

>
> I didn't know this, so I imagine others might not.  The string  
> "&#x0;" is invalid XML.  The character is simply not allowed in XML  
> in any representation.  XML 1.0 standard blocks most of the  
> characters under x20, allowing only x9 xA and xD.  XML 1.1 allows x1- 
> x20, but still blocks x0.
>
> http://www.w3.org/TR/1998/REC-xml-19980210#charsets
> http://www.w3.org/TR/xml11/#charsets
>
> This creates an interesting problem for serializing Java strings  
> containing the null character, e.g. "\u0000", or for other non- 
> whitespace control characters like the bell character "\u0007".   
> We've got an integration test for this case in Surefire, and it does  
> entirely the wrong thing (SUREFIRE-455).
>
> In the patch submitted to that bug, Todor throws away nulls in his  
> XML escaper, silently omitting them from the output; all other  
> control characters (even the 1.0-illegal ones) pass through.  That  
> doesn't seem right, especially when we're talking about test  
> results!  (Expected "" but was "" ... Just imagine how painful it  
> would be to track something like that down.)
>
> But neither does it seem right to insert "&#x0;" when it's illegal  
> XML. Notably, Java will cheerfully print &#x0; in XML if you tell it  
> to do so, and many parsers will figure out what to do with it just  
> fine; the same applies to "&#x07;".
>
> Thoughts?  Should we emit "&#x0;", standards-be-damned?  Silently  
> omit the character?  Print a "?" instead?  Something else?
>

Where did you run into this? I think not showing what it actually is  
makes it immediately not obvious what's going wrong. So I'm for  
showing what it actually is. Can you just wrap in CDATA?

> -Dan
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>

Thanks,

Jason

----------------------------------------------------------
Jason van Zyl
Founder,  Apache Maven
jason at sonatype dot com
----------------------------------------------------------

happiness is like a butterfly: the more you chase it, the more it will
elude you, but if you turn your attention to other things, it will come
and sit softly on your shoulder ...

-- Thoreau 




---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org


Re: Nulls in XML

Posted by Dan Fabulich <da...@fabulich.com>.
Dan Fabulich wrote:

> But neither does it seem right to insert "&#x0;" when it's illegal XML. 
> Notably, Java will cheerfully print &#x0; in XML if you tell it to do so, and 
> many parsers will figure out what to do with it just fine; the same applies 
> to "&#x07;".

Notably, Java's XML parser does NOT know what to do with "&#x0;" or 
"&#x7;", despite the fact that Java will emit the character if you ask it 
to.

org.xml.sax.SAXParseException: Character reference "&#x0" is an invalid XML character.
 	at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:236)
 	at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:215)
 	at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:386)
 	at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:316)
 	at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1438)
 	at com.sun.org.apache.xerces.internal.impl.XMLScanner.scanCharReferenceValue(XMLScanner.java:1304)
 	at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanCharReference(XMLDocumentFragmentScannerImpl.java:1259)
 	at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFragmentScannerImpl.java:1753)
 	at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:368)
 	at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:834)
 	at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764)
 	at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:148)
 	at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1242)
 	at javax.xml.parsers.SAXParser.parse(SAXParser.java:375)
 	at javax.xml.parsers.SAXParser.parse(SAXParser.java:311)

-Dan

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org