You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@uima.apache.org by "Adam Lally (JIRA)" <ui...@incubator.apache.org> on 2007/01/16 00:06:27 UTC
[jira] Created: (UIMA-194) XCasToCasDataSaxHandlerTest gives
failure when run under Sun Java 1.4.2
XCasToCasDataSaxHandlerTest gives failure when run under Sun Java 1.4.2
-----------------------------------------------------------------------
Key: UIMA-194
URL: https://issues.apache.org/jira/browse/UIMA-194
Project: UIMA
Issue Type: Bug
Components: Core Java Framework
Environment: Sun Java 1.4.2_12
Reporter: Adam Lally
Priority: Minor
Possible newline issue? An assertEquals of two strings fails, yet the strings appear identical in the compare viewer.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (UIMA-194) Tools highlight incorrect annotation
offsets due to XML serialization bug in Sun Java 1.4.2
Posted by "Marshall Schor (JIRA)" <ui...@incubator.apache.org>.
[ https://issues.apache.org/jira/browse/UIMA-194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12476757 ]
Marshall Schor commented on UIMA-194:
-------------------------------------
In http://www.w3.org/TR/2003/WD-xslt-xquery-serialization-20030502/#xml-output , for XSLT 2.0, XQUERY 1.0 serialization requires that "CR characters in text nodes should be written as 
 or an equivalent; while CR, NL, and TAB characters in attribute nodes should be output respectively as 
, 
, and 	, or their equivalents."
This is required for XSLT 2, but maybe not for 1. The spec uses the word "should" rather than "must".
I wonder if we could run a little test at startup and warn the user if this wasn't happening with a suggested fix to change Java levels etc.
> Tools highlight incorrect annotation offsets due to XML serialization bug in Sun Java 1.4.2
> -------------------------------------------------------------------------------------------
>
> Key: UIMA-194
> URL: https://issues.apache.org/jira/browse/UIMA-194
> Project: UIMA
> Issue Type: Bug
> Components: Documentation, Tools
> Environment: Sun Java 1.4.2_12
> Reporter: Adam Lally
> Priority: Minor
> Fix For: 2.2
>
>
> The XML serialization support in Sun Java 1.4.2_12 doesn't serialize CR characters to XML. As a result, if the document text contains CR characters, XCAS or XMI serialization will cause them to be lost, resulting in incorrect annotation offsets. This is exposed in the DocumentAnalyzer, with the highlighting being incorrect if the input document contains CR characters.
> Unit test failure occurred in XCasToCasDataHandlerTest, but the test was modified so it passes. In that test an assertEquals of two strings fails, yet the strings appear identical in the compare viewer.
> The problem does not occur in Sun Java 1.5 or later, or with IBM Java.
> Probably a documentation update is an appropriate way to address this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
RE: [jira] Commented: (UIMA-194) Tools highlight incorrect annotation offsets due to XML serialization bug in Sun Java 1.4.2
Posted by Pascal Coupet <pa...@temis.com>.
This is an issue we bumped into several times. XML and XSL
transformations are not nice with spaces and end of lines encoding. I'm
not sure if it's the fault of specs or is some tools doen't work
properly on this. What we found is that a reliable way to preserve
formatting is to put text zone within a CDATA section.
There is an interesting article on white spaces in XML and XSL on
xml.com (http://www.xml.com/pub/a/2001/11/07/whitespace.html). It seems
that it's possible to design XSL transformations respecting white spaces
using xsl:preserve-space.
Pascal
-----Original Message-----
From: Adam Lally (JIRA) [mailto:uima-dev@incubator.apache.org]
Sent: Tuesday, February 27, 2007 10:00 AM
To: uima-dev@incubator.apache.org
Subject: [jira] Commented: (UIMA-194) Tools highlight incorrect
annotation offsets due to XML serialization bug in Sun Java 1.4.2
[
https://issues.apache.org/jira/browse/UIMA-194?page=com.atlassian.jira.p
lugin.system.issuetabpanels:comment-tabpanel#action_12476189 ]
Adam Lally commented on UIMA-194:
---------------------------------
Actually Adam needs the historical background too. :) Is there a part
of the spec that actually says that XML parsers can do whatever they
want? I'd be curious to see that.
As far as XML parsing is concerned, the XML spec requires that the
parser return all newlines (meaning unescpaed newline characters) in the
input as single #xA characters, regardless of what was in the input [1].
For serializaiton, we use XSLT. I wasn't able to quickly find a clear
statement on newlines in the XSLT spec just now, but it would seem quite
bizarre to me if an XSLT transform that copied a text node from an input
document was allowed to drop #xD escape sequences that occurred in that
input text node. That's essentially the situation we have here, I
think.
In any case Xalan seems to think it at least was undesirable behavior
(if not a bug), since between Java 1.4 and 1.5 (both of which bundle
Xalan) they "fixed" it.
[1] http://www.w3.org/TR/REC-xml/#sec-line-ends
> Tools highlight incorrect annotation offsets due to XML serialization
bug in Sun Java 1.4.2
>
------------------------------------------------------------------------
-------------------
>
> Key: UIMA-194
> URL: https://issues.apache.org/jira/browse/UIMA-194
> Project: UIMA
> Issue Type: Bug
> Components: Documentation, Tools
> Environment: Sun Java 1.4.2_12
> Reporter: Adam Lally
> Priority: Minor
> Fix For: 2.2
>
>
> The XML serialization support in Sun Java 1.4.2_12 doesn't serialize
CR characters to XML. As a result, if the document text contains CR
characters, XCAS or XMI serialization will cause them to be lost,
resulting in incorrect annotation offsets. This is exposed in the
DocumentAnalyzer, with the highlighting being incorrect if the input
document contains CR characters.
> Unit test failure occurred in XCasToCasDataHandlerTest, but the test
was modified so it passes. In that test an assertEquals of two strings
fails, yet the strings appear identical in the compare viewer.
> The problem does not occur in Sun Java 1.5 or later, or with IBM Java.
> Probably a documentation update is an appropriate way to address this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (UIMA-194) Tools highlight incorrect annotation
offsets due to XML serialization bug in Sun Java 1.4.2
Posted by "Adam Lally (JIRA)" <ui...@incubator.apache.org>.
[ https://issues.apache.org/jira/browse/UIMA-194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12476189 ]
Adam Lally commented on UIMA-194:
---------------------------------
Actually Adam needs the historical background too. :) Is there a part of the spec that actually says that XML parsers can do whatever they want? I'd be curious to see that.
As far as XML parsing is concerned, the XML spec requires that the parser return all newlines (meaning unescpaed newline characters) in the input as single #xA characters, regardless of what was in the input [1].
For serializaiton, we use XSLT. I wasn't able to quickly find a clear statement on newlines in the XSLT spec just now, but it would seem quite bizarre to me if an XSLT transform that copied a text node from an input document was allowed to drop #xD escape sequences that occurred in that input text node. That's essentially the situation we have here, I think.
In any case Xalan seems to think it at least was undesirable behavior (if not a bug), since between Java 1.4 and 1.5 (both of which bundle Xalan) they "fixed" it.
[1] http://www.w3.org/TR/REC-xml/#sec-line-ends
> Tools highlight incorrect annotation offsets due to XML serialization bug in Sun Java 1.4.2
> -------------------------------------------------------------------------------------------
>
> Key: UIMA-194
> URL: https://issues.apache.org/jira/browse/UIMA-194
> Project: UIMA
> Issue Type: Bug
> Components: Documentation, Tools
> Environment: Sun Java 1.4.2_12
> Reporter: Adam Lally
> Priority: Minor
> Fix For: 2.2
>
>
> The XML serialization support in Sun Java 1.4.2_12 doesn't serialize CR characters to XML. As a result, if the document text contains CR characters, XCAS or XMI serialization will cause them to be lost, resulting in incorrect annotation offsets. This is exposed in the DocumentAnalyzer, with the highlighting being incorrect if the input document contains CR characters.
> Unit test failure occurred in XCasToCasDataHandlerTest, but the test was modified so it passes. In that test an assertEquals of two strings fails, yet the strings appear identical in the compare viewer.
> The problem does not occur in Sun Java 1.5 or later, or with IBM Java.
> Probably a documentation update is an appropriate way to address this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Commented: (UIMA-194) Tools highlight incorrect annotation offsets due to XML serialization bug in Sun Java 1.4.2
Posted by Adam Lally <al...@alum.rpi.edu>.
On 2/28/07, Marshall Schor <ms...@schor.com> wrote:
> The XML Spec requires all 3 combinations of CR, CRLF, or LF be converted
> to #xA character when "parsing" an XML document. See section 2.11,
> End-of-Line handling
>
> The XSLT spec doesn't say how these are to be "output". Michael Kay's
> book on XSLT 2.0 says, on pg 389, Rules for Text Output, "The way in
> which line endings are output (for example LF or CRLF) is not defined;
> the implementation might choose to do the default line-ending
> conventions of the platform on which it is running".
>
There may be a subtle difference between escaped newlines and
unescaped newlines. I can certainly understand that an XSLT processor
could output a "real" newline character any way it wanted, because
from the XML parser's perspective, it makes absolutely no difference -
each newline will be considered just an LF when it is parsed.
To me the question is what happens if the input XML document contains
an escaped &#D; sequence. Is the XSLT processor really allowed to
drop it if it copies this text to the output? That makes no sense to
me (not that this means it isn't possible the spec was poorly written
and allows it, but it seems like the implementers now at least agree
with me that this wouldn't make sense to actually implement it that
way).
> It would be nice to get a fix for this which worked on all
> implementations. Perhaps something like: do a "test" at start up to see
> what convention is in use on this particular platform/software level,
> and then appropriately "compensate" for it.
>
Not really sure what you intend to do based on a platform-specific
test here. I think we would essentially have to implement the XML
serialization code ourselves rather than using XSLT, which is no easy
task.
There's a workaround for this issue. Use a newer version of Java, or
if you're really committed to Sun Java 1.4, get a newer version of
Xalan. I suppose, now that we're an Apache project, we can ship Xalan
with UIMA if we wanted to. I just don't think UIMA should be in the
business of reimplementing XML serialization because that Xalan
version that's several years old has a bug (or undeseriable beahvior
if you prefer).
-Adam
Re: [jira] Commented: (UIMA-194) Tools highlight incorrect annotation
offsets due to XML serialization bug in Sun Java 1.4.2
Posted by Marshall Schor <ms...@schor.com>.
The XML Spec requires all 3 combinations of CR, CRLF, or LF be converted
to #xA character when "parsing" an XML document. See section 2.11,
End-of-Line handling
The XSLT spec doesn't say how these are to be "output". Michael Kay's
book on XSLT 2.0 says, on pg 389, Rules for Text Output, "The way in
which line endings are output (for example LF or CRLF) is not defined;
the implementation might choose to do the default line-ending
conventions of the platform on which it is running".
It would be nice to get a fix for this which worked on all
implementations. Perhaps something like: do a "test" at start up to see
what convention is in use on this particular platform/software level,
and then appropriately "compensate" for it.
-Marshall
Adam Lally (JIRA) wrote:
> [ https://issues.apache.org/jira/browse/UIMA-194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12476190 ]
>
> Adam Lally commented on UIMA-194:
> ---------------------------------
>
> Actually Adam needs the historical background too. :) Is there a part of the spec that actually says that XML parsers can do whatever they want? I'd be curious to see that.
>
> As far as XML parsing is concerned, the XML spec requires that the parser return all newlines (meaning unescpaed newline characters) in the input as single #xA characters, regardless of what was in the input [1].
>
> For serializaiton, we use XSLT. I wasn't able to quickly find a clear statement on newlines in the XSLT spec just now, but it would seem quite bizarre to me if an XSLT transform that copied a text node from an input document was allowed to drop #xD escape sequences that occurred in that input text node. That's essentially the situation we have here, I think.
>
> In any case Xalan seems to think it at least was undesirable behavior (if not a bug), since between Java 1.4 and 1.5 (both of which bundle Xalan) they "fixed" it.
>
>
> [1] http://www.w3.org/TR/REC-xml/#sec-line-ends
>
>
>> Tools highlight incorrect annotation offsets due to XML serialization bug in Sun Java 1.4.2
>> -------------------------------------------------------------------------------------------
>>
>> Key: UIMA-194
>> URL: https://issues.apache.org/jira/browse/UIMA-194
>> Project: UIMA
>> Issue Type: Bug
>> Components: Documentation, Tools
>> Environment: Sun Java 1.4.2_12
>> Reporter: Adam Lally
>> Priority: Minor
>> Fix For: 2.2
>>
>>
>> The XML serialization support in Sun Java 1.4.2_12 doesn't serialize CR characters to XML. As a result, if the document text contains CR characters, XCAS or XMI serialization will cause them to be lost, resulting in incorrect annotation offsets. This is exposed in the DocumentAnalyzer, with the highlighting being incorrect if the input document contains CR characters.
>> Unit test failure occurred in XCasToCasDataHandlerTest, but the test was modified so it passes. In that test an assertEquals of two strings fails, yet the strings appear identical in the compare viewer.
>> The problem does not occur in Sun Java 1.5 or later, or with IBM Java.
>> Probably a documentation update is an appropriate way to address this.
>>
>
>
[jira] Commented: (UIMA-194) Tools highlight incorrect annotation
offsets due to XML serialization bug in Sun Java 1.4.2
Posted by "Adam Lally (JIRA)" <ui...@incubator.apache.org>.
[ https://issues.apache.org/jira/browse/UIMA-194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12476190 ]
Adam Lally commented on UIMA-194:
---------------------------------
Actually Adam needs the historical background too. :) Is there a part of the spec that actually says that XML parsers can do whatever they want? I'd be curious to see that.
As far as XML parsing is concerned, the XML spec requires that the parser return all newlines (meaning unescpaed newline characters) in the input as single #xA characters, regardless of what was in the input [1].
For serializaiton, we use XSLT. I wasn't able to quickly find a clear statement on newlines in the XSLT spec just now, but it would seem quite bizarre to me if an XSLT transform that copied a text node from an input document was allowed to drop #xD escape sequences that occurred in that input text node. That's essentially the situation we have here, I think.
In any case Xalan seems to think it at least was undesirable behavior (if not a bug), since between Java 1.4 and 1.5 (both of which bundle Xalan) they "fixed" it.
[1] http://www.w3.org/TR/REC-xml/#sec-line-ends
> Tools highlight incorrect annotation offsets due to XML serialization bug in Sun Java 1.4.2
> -------------------------------------------------------------------------------------------
>
> Key: UIMA-194
> URL: https://issues.apache.org/jira/browse/UIMA-194
> Project: UIMA
> Issue Type: Bug
> Components: Documentation, Tools
> Environment: Sun Java 1.4.2_12
> Reporter: Adam Lally
> Priority: Minor
> Fix For: 2.2
>
>
> The XML serialization support in Sun Java 1.4.2_12 doesn't serialize CR characters to XML. As a result, if the document text contains CR characters, XCAS or XMI serialization will cause them to be lost, resulting in incorrect annotation offsets. This is exposed in the DocumentAnalyzer, with the highlighting being incorrect if the input document contains CR characters.
> Unit test failure occurred in XCasToCasDataHandlerTest, but the test was modified so it passes. In that test an assertEquals of two strings fails, yet the strings appear identical in the compare viewer.
> The problem does not occur in Sun Java 1.5 or later, or with IBM Java.
> Probably a documentation update is an appropriate way to address this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (UIMA-194) Tools highlight incorrect annotation
offsets due to XML serialization bug in Sun Java 1.4.2
Posted by "Adam Lally (JIRA)" <ui...@incubator.apache.org>.
[ https://issues.apache.org/jira/browse/UIMA-194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Adam Lally updated UIMA-194:
----------------------------
Component/s: (was: Core Java Framework)
Tools
Documentation
Fix Version/s: 2.2
Description:
The XML serialization support in Sun Java 1.4.2_12 doesn't serialize CR characters to XML. As a result, if the document text contains CR characters, XCAS or XMI serialization will cause them to be lost, resulting in incorrect annotation offsets. This is exposed in the DocumentAnalyzer, with the highlighting being incorrect if the input document contains CR characters.
Unit test failure occurred in XCasToCasDataHandlerTest, but the test was modified so it passes. In that test an assertEquals of two strings fails, yet the strings appear identical in the compare viewer.
The problem does not occur in Sun Java 1.5 or later, or with IBM Java.
Probably a documentation update is an appropriate way to address this.
was:Possible newline issue? An assertEquals of two strings fails, yet the strings appear identical in the compare viewer.
Summary: Tools highlight incorrect annotation offsets due to XML serialization bug in Sun Java 1.4.2 (was: XCasToCasDataSaxHandlerTest gives failure when run under Sun Java 1.4.2)
> Tools highlight incorrect annotation offsets due to XML serialization bug in Sun Java 1.4.2
> -------------------------------------------------------------------------------------------
>
> Key: UIMA-194
> URL: https://issues.apache.org/jira/browse/UIMA-194
> Project: UIMA
> Issue Type: Bug
> Components: Documentation, Tools
> Environment: Sun Java 1.4.2_12
> Reporter: Adam Lally
> Priority: Minor
> Fix For: 2.2
>
>
> The XML serialization support in Sun Java 1.4.2_12 doesn't serialize CR characters to XML. As a result, if the document text contains CR characters, XCAS or XMI serialization will cause them to be lost, resulting in incorrect annotation offsets. This is exposed in the DocumentAnalyzer, with the highlighting being incorrect if the input document contains CR characters.
> Unit test failure occurred in XCasToCasDataHandlerTest, but the test was modified so it passes. In that test an assertEquals of two strings fails, yet the strings appear identical in the compare viewer.
> The problem does not occur in Sun Java 1.5 or later, or with IBM Java.
> Probably a documentation update is an appropriate way to address this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Closed: (UIMA-194) Tools highlight incorrect annotation
offsets due to XML serialization bug in Sun Java 1.4.2
Posted by "Thilo Goetz (JIRA)" <ui...@incubator.apache.org>.
[ https://issues.apache.org/jira/browse/UIMA-194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thilo Goetz closed UIMA-194.
----------------------------
Resolution: Fixed
"Fixed" by adding a note about the errant JVM behavior to the docs.
> Tools highlight incorrect annotation offsets due to XML serialization bug in Sun Java 1.4.2
> -------------------------------------------------------------------------------------------
>
> Key: UIMA-194
> URL: https://issues.apache.org/jira/browse/UIMA-194
> Project: UIMA
> Issue Type: Bug
> Components: Documentation, Tools
> Environment: Sun Java 1.4.2_12
> Reporter: Adam Lally
> Assignee: Thilo Goetz
> Priority: Minor
> Fix For: 2.2
>
>
> The XML serialization support in Sun Java 1.4.2_12 doesn't serialize CR characters to XML. As a result, if the document text contains CR characters, XCAS or XMI serialization will cause them to be lost, resulting in incorrect annotation offsets. This is exposed in the DocumentAnalyzer, with the highlighting being incorrect if the input document contains CR characters.
> Unit test failure occurred in XCasToCasDataHandlerTest, but the test was modified so it passes. In that test an assertEquals of two strings fails, yet the strings appear identical in the compare viewer.
> The problem does not occur in Sun Java 1.5 or later, or with IBM Java.
> Probably a documentation update is an appropriate way to address this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (UIMA-194) Tools highlight incorrect annotation
offsets due to XML serialization bug in Sun Java 1.4.2
Posted by "Thilo Goetz (JIRA)" <ui...@incubator.apache.org>.
[ https://issues.apache.org/jira/browse/UIMA-194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12476152 ]
Thilo Goetz commented on UIMA-194:
----------------------------------
Some historical background (for people who are not Adam ;-): this issue has plagued us in the past, and may do so again in the future. The issue here is that this behavior is not really a bug. According to the w3c xml spec, an xml processor is free to do with eol characters whatever it wants. So we may not only have this issue on serialization, but also on deserialization.
One way to deal with this in a general way would be to offer an option to base64 encode the document text on serialization. That way we could circumvent the vagaries of xml.
> Tools highlight incorrect annotation offsets due to XML serialization bug in Sun Java 1.4.2
> -------------------------------------------------------------------------------------------
>
> Key: UIMA-194
> URL: https://issues.apache.org/jira/browse/UIMA-194
> Project: UIMA
> Issue Type: Bug
> Components: Documentation, Tools
> Environment: Sun Java 1.4.2_12
> Reporter: Adam Lally
> Priority: Minor
> Fix For: 2.2
>
>
> The XML serialization support in Sun Java 1.4.2_12 doesn't serialize CR characters to XML. As a result, if the document text contains CR characters, XCAS or XMI serialization will cause them to be lost, resulting in incorrect annotation offsets. This is exposed in the DocumentAnalyzer, with the highlighting being incorrect if the input document contains CR characters.
> Unit test failure occurred in XCasToCasDataHandlerTest, but the test was modified so it passes. In that test an assertEquals of two strings fails, yet the strings appear identical in the compare viewer.
> The problem does not occur in Sun Java 1.5 or later, or with IBM Java.
> Probably a documentation update is an appropriate way to address this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (UIMA-194) Tools highlight incorrect annotation
offsets due to XML serialization bug in Sun Java 1.4.2
Posted by "Thilo Goetz (JIRA)" <ui...@incubator.apache.org>.
[ https://issues.apache.org/jira/browse/UIMA-194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thilo Goetz reassigned UIMA-194:
--------------------------------
Assignee: Thilo Goetz
> Tools highlight incorrect annotation offsets due to XML serialization bug in Sun Java 1.4.2
> -------------------------------------------------------------------------------------------
>
> Key: UIMA-194
> URL: https://issues.apache.org/jira/browse/UIMA-194
> Project: UIMA
> Issue Type: Bug
> Components: Documentation, Tools
> Environment: Sun Java 1.4.2_12
> Reporter: Adam Lally
> Assignee: Thilo Goetz
> Priority: Minor
> Fix For: 2.2
>
>
> The XML serialization support in Sun Java 1.4.2_12 doesn't serialize CR characters to XML. As a result, if the document text contains CR characters, XCAS or XMI serialization will cause them to be lost, resulting in incorrect annotation offsets. This is exposed in the DocumentAnalyzer, with the highlighting being incorrect if the input document contains CR characters.
> Unit test failure occurred in XCasToCasDataHandlerTest, but the test was modified so it passes. In that test an assertEquals of two strings fails, yet the strings appear identical in the compare viewer.
> The problem does not occur in Sun Java 1.5 or later, or with IBM Java.
> Probably a documentation update is an appropriate way to address this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.