You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@santuario.apache.org by "steve (Created) (JIRA)" <ji...@apache.org> on 2012/03/26 19:42:29 UTC
[jira] [Created] (SANTUARIO-307) utf8 encode is broken
utf8 encode is broken
---------------------
Key: SANTUARIO-307
URL: https://issues.apache.org/jira/browse/SANTUARIO-307
Project: Santuario
Issue Type: Bug
Components: Java
Affects Versions: Java 1.4.4
Environment: Ubuntu 10.04 LTS 32 buit
JRE 5.x
Reporter: steve
Assignee: Colm O hEigeartaigh
This code:
if ((c >= 0xD800 && c <= 0xDBFF) || (c >= 0xDC00 && c <= 0xDFFF) ){
//No Surrogates in sun java
out.write(0x3f);
return;
}
from UtfHelpper.writeCharToUtf8 and other methods in UtfHelpper seems to be excluding these 3 unicode blocks:
http://www.fileformat.info/info/unicode/block/high_surrogates/index.htm
http://www.fileformat.info/info/unicode/block/high_private_use_surrogates/index.htm
http://www.fileformat.info/info/unicode/block/low_surrogates/index.htm
The problem is that some characters from other sections fall in that range when encoded in UTF-16. For example this character 0x0002000B encoded as UTF16 is 0xD840 0xDC0B.
http://www.fileformat.info/info/unicode/char/2000b/index.htm
This causes the output to be corrupted as ? in somecases.
here is a code sample:
Canonicalizer c14n = Canonicalizer.getInstance(Canonicalizer.ALGO_ID_C14N_EXCL_OMIT_COMMENTS);
String doc1 = "<a>\u4E1F</a>";
String doc2 = "<a>\uD840\uDC0B</a>";
System.out.println("doc1 before:" + doc1);
byte [] output = c14n.canonicalize(doc1.getBytes("UTF8"));
System.out.println("doc1 after:" + new String(output, "UTF8"));
System.out.println("doc2 before:" + doc2);
output = c14n.canonicalize(doc2.getBytes("UTF8"));
System.out.println("doc2 after:" + new String(output, "UTF8"));
the output is:
doc1 before:<a>丟</a>
doc1 after:<a>丟</a>
doc2 before:<a>𠀋</a>
doc2 after:<a>??</a>
Notice that "doc2 after" corrupted, as <a>??</a> instead of <a>𠀋</a>
Based on the code there does not see mot be any workaround the issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (SANTUARIO-307) utf8 encode is broken
Posted by "steve (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SANTUARIO-307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
steve updated SANTUARIO-307:
----------------------------
Environment:
Ubuntu 10.04 LTS 32 bit
JRE 5.x
was:
Ubuntu 10.04 LTS 32 buit
JRE 5.x
> utf8 encode is broken
> ---------------------
>
> Key: SANTUARIO-307
> URL: https://issues.apache.org/jira/browse/SANTUARIO-307
> Project: Santuario
> Issue Type: Bug
> Components: Java
> Affects Versions: Java 1.4.4
> Environment: Ubuntu 10.04 LTS 32 bit
> JRE 5.x
> Reporter: steve
> Assignee: Colm O hEigeartaigh
> Labels: Surrogates, UtfHelpper, utf8
>
> This code:
> if ((c >= 0xD800 && c <= 0xDBFF) || (c >= 0xDC00 && c <= 0xDFFF) ){
> //No Surrogates in sun java
> out.write(0x3f);
> return;
> }
> from UtfHelpper.writeCharToUtf8 and other methods in UtfHelpper seems to be excluding these 3 unicode blocks:
> http://www.fileformat.info/info/unicode/block/high_surrogates/index.htm
> http://www.fileformat.info/info/unicode/block/high_private_use_surrogates/index.htm
> http://www.fileformat.info/info/unicode/block/low_surrogates/index.htm
> The problem is that some characters from other sections fall in that range when encoded in UTF-16. For example this character 0x0002000B encoded as UTF16 is 0xD840 0xDC0B.
> http://www.fileformat.info/info/unicode/char/2000b/index.htm
> This causes the output to be corrupted as ? in somecases.
> here is a code sample:
> Canonicalizer c14n = Canonicalizer.getInstance(Canonicalizer.ALGO_ID_C14N_EXCL_OMIT_COMMENTS);
> String doc1 = "<a>\u4E1F</a>";
> String doc2 = "<a>\uD840\uDC0B</a>";
> System.out.println("doc1 before:" + doc1);
> byte [] output = c14n.canonicalize(doc1.getBytes("UTF8"));
> System.out.println("doc1 after:" + new String(output, "UTF8"));
> System.out.println("doc2 before:" + doc2);
> output = c14n.canonicalize(doc2.getBytes("UTF8"));
> System.out.println("doc2 after:" + new String(output, "UTF8"));
> the output is:
> doc1 before:<a>丟</a>
> doc1 after:<a>丟</a>
> doc2 before:<a>𠀋</a>
> doc2 after:<a>??</a>
> Notice that "doc2 after" corrupted, as <a>??</a> instead of <a>𠀋</a>
> Based on the code there does not see mot be any workaround the issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SANTUARIO-307) utf8 encode is broken
Posted by "Colm O hEigeartaigh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SANTUARIO-307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13281463#comment-13281463 ]
Colm O hEigeartaigh commented on SANTUARIO-307:
-----------------------------------------------
> For example this character 0x0002000B encoded as UTF16 is 0xD840 0xDC0B.
The problem with this case is that the CanonicalizerBase steps through each character to be output, and so it ends up encoding each character separately. Based on this approach I'm not sure I see an easy resolution to this problem.
Colm.
> utf8 encode is broken
> ---------------------
>
> Key: SANTUARIO-307
> URL: https://issues.apache.org/jira/browse/SANTUARIO-307
> Project: Santuario
> Issue Type: Bug
> Components: Java
> Affects Versions: Java 1.4.4
> Environment: Ubuntu 10.04 LTS 32 bit
> JRE 5.x
> Reporter: steve
> Assignee: Colm O hEigeartaigh
> Labels: Surrogates, UtfHelpper, utf8
>
> This code:
> if ((c >= 0xD800 && c <= 0xDBFF) || (c >= 0xDC00 && c <= 0xDFFF) ){
> //No Surrogates in sun java
> out.write(0x3f);
> return;
> }
> from UtfHelpper.writeCharToUtf8 and other methods in UtfHelpper seems to be excluding these 3 unicode blocks:
> http://www.fileformat.info/info/unicode/block/high_surrogates/index.htm
> http://www.fileformat.info/info/unicode/block/high_private_use_surrogates/index.htm
> http://www.fileformat.info/info/unicode/block/low_surrogates/index.htm
> The problem is that some characters from other sections fall in that range when encoded in UTF-16. For example this character 0x0002000B encoded as UTF16 is 0xD840 0xDC0B.
> http://www.fileformat.info/info/unicode/char/2000b/index.htm
> This causes the output to be corrupted as ? in some cases.
> here is a code sample:
> Canonicalizer c14n = Canonicalizer.getInstance(Canonicalizer.ALGO_ID_C14N_EXCL_OMIT_COMMENTS);
> String doc1 = "<a>\u4E1F</a>";
> String doc2 = "<a>\uD840\uDC0B</a>";
> System.out.println("doc1 before:" + doc1);
> byte [] output = c14n.canonicalize(doc1.getBytes("UTF8"));
> System.out.println("doc1 after:" + new String(output, "UTF8"));
> System.out.println("doc2 before:" + doc2);
> output = c14n.canonicalize(doc2.getBytes("UTF8"));
> System.out.println("doc2 after:" + new String(output, "UTF8"));
> the output is:
> doc1 before:<a>丟</a>
> doc1 after:<a>丟</a>
> doc2 before:<a>𠀋</a>
> doc2 after:<a>??</a>
> Notice that "doc2 after" corrupted, as <a>??</a> instead of <a>𠀋</a>
> Based on the code there does not seem to be any workaround.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (SANTUARIO-307) utf8 encode is broken
Posted by "steve (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SANTUARIO-307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
steve updated SANTUARIO-307:
----------------------------
Description:
This code:
if ((c >= 0xD800 && c <= 0xDBFF) || (c >= 0xDC00 && c <= 0xDFFF) ){
//No Surrogates in sun java
out.write(0x3f);
return;
}
from UtfHelpper.writeCharToUtf8 and other methods in UtfHelpper seems to be excluding these 3 unicode blocks:
http://www.fileformat.info/info/unicode/block/high_surrogates/index.htm
http://www.fileformat.info/info/unicode/block/high_private_use_surrogates/index.htm
http://www.fileformat.info/info/unicode/block/low_surrogates/index.htm
The problem is that some characters from other sections fall in that range when encoded in UTF-16. For example this character 0x0002000B encoded as UTF16 is 0xD840 0xDC0B.
http://www.fileformat.info/info/unicode/char/2000b/index.htm
This causes the output to be corrupted as ? in some cases.
here is a code sample:
Canonicalizer c14n = Canonicalizer.getInstance(Canonicalizer.ALGO_ID_C14N_EXCL_OMIT_COMMENTS);
String doc1 = "<a>\u4E1F</a>";
String doc2 = "<a>\uD840\uDC0B</a>";
System.out.println("doc1 before:" + doc1);
byte [] output = c14n.canonicalize(doc1.getBytes("UTF8"));
System.out.println("doc1 after:" + new String(output, "UTF8"));
System.out.println("doc2 before:" + doc2);
output = c14n.canonicalize(doc2.getBytes("UTF8"));
System.out.println("doc2 after:" + new String(output, "UTF8"));
the output is:
doc1 before:<a>丟</a>
doc1 after:<a>丟</a>
doc2 before:<a>𠀋</a>
doc2 after:<a>??</a>
Notice that "doc2 after" corrupted, as <a>??</a> instead of <a>𠀋</a>
Based on the code there does not seem to be any workaround.
was:
This code:
if ((c >= 0xD800 && c <= 0xDBFF) || (c >= 0xDC00 && c <= 0xDFFF) ){
//No Surrogates in sun java
out.write(0x3f);
return;
}
from UtfHelpper.writeCharToUtf8 and other methods in UtfHelpper seems to be excluding these 3 unicode blocks:
http://www.fileformat.info/info/unicode/block/high_surrogates/index.htm
http://www.fileformat.info/info/unicode/block/high_private_use_surrogates/index.htm
http://www.fileformat.info/info/unicode/block/low_surrogates/index.htm
The problem is that some characters from other sections fall in that range when encoded in UTF-16. For example this character 0x0002000B encoded as UTF16 is 0xD840 0xDC0B.
http://www.fileformat.info/info/unicode/char/2000b/index.htm
This causes the output to be corrupted as ? in somecases.
here is a code sample:
Canonicalizer c14n = Canonicalizer.getInstance(Canonicalizer.ALGO_ID_C14N_EXCL_OMIT_COMMENTS);
String doc1 = "<a>\u4E1F</a>";
String doc2 = "<a>\uD840\uDC0B</a>";
System.out.println("doc1 before:" + doc1);
byte [] output = c14n.canonicalize(doc1.getBytes("UTF8"));
System.out.println("doc1 after:" + new String(output, "UTF8"));
System.out.println("doc2 before:" + doc2);
output = c14n.canonicalize(doc2.getBytes("UTF8"));
System.out.println("doc2 after:" + new String(output, "UTF8"));
the output is:
doc1 before:<a>丟</a>
doc1 after:<a>丟</a>
doc2 before:<a>𠀋</a>
doc2 after:<a>??</a>
Notice that "doc2 after" corrupted, as <a>??</a> instead of <a>𠀋</a>
Based on the code there does not see mot be any workaround the issue.
> utf8 encode is broken
> ---------------------
>
> Key: SANTUARIO-307
> URL: https://issues.apache.org/jira/browse/SANTUARIO-307
> Project: Santuario
> Issue Type: Bug
> Components: Java
> Affects Versions: Java 1.4.4
> Environment: Ubuntu 10.04 LTS 32 bit
> JRE 5.x
> Reporter: steve
> Assignee: Colm O hEigeartaigh
> Labels: Surrogates, UtfHelpper, utf8
>
> This code:
> if ((c >= 0xD800 && c <= 0xDBFF) || (c >= 0xDC00 && c <= 0xDFFF) ){
> //No Surrogates in sun java
> out.write(0x3f);
> return;
> }
> from UtfHelpper.writeCharToUtf8 and other methods in UtfHelpper seems to be excluding these 3 unicode blocks:
> http://www.fileformat.info/info/unicode/block/high_surrogates/index.htm
> http://www.fileformat.info/info/unicode/block/high_private_use_surrogates/index.htm
> http://www.fileformat.info/info/unicode/block/low_surrogates/index.htm
> The problem is that some characters from other sections fall in that range when encoded in UTF-16. For example this character 0x0002000B encoded as UTF16 is 0xD840 0xDC0B.
> http://www.fileformat.info/info/unicode/char/2000b/index.htm
> This causes the output to be corrupted as ? in some cases.
> here is a code sample:
> Canonicalizer c14n = Canonicalizer.getInstance(Canonicalizer.ALGO_ID_C14N_EXCL_OMIT_COMMENTS);
> String doc1 = "<a>\u4E1F</a>";
> String doc2 = "<a>\uD840\uDC0B</a>";
> System.out.println("doc1 before:" + doc1);
> byte [] output = c14n.canonicalize(doc1.getBytes("UTF8"));
> System.out.println("doc1 after:" + new String(output, "UTF8"));
> System.out.println("doc2 before:" + doc2);
> output = c14n.canonicalize(doc2.getBytes("UTF8"));
> System.out.println("doc2 after:" + new String(output, "UTF8"));
> the output is:
> doc1 before:<a>丟</a>
> doc1 after:<a>丟</a>
> doc2 before:<a>𠀋</a>
> doc2 after:<a>??</a>
> Notice that "doc2 after" corrupted, as <a>??</a> instead of <a>𠀋</a>
> Based on the code there does not seem to be any workaround.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira