You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@xalan.apache.org by "maxfortun (via GitHub)" <gi...@apache.org> on 2024/01/24 18:46:06 UTC
[PR] updated [xalan-java]
maxfortun opened a new pull request, #166:
URL: https://github.com/apache/xalan-java/pull/166
https://issues.apache.org/jira/browse/XALANJ-2725
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@xalan.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@xalan.apache.org
For additional commands, e-mail: dev-help@xalan.apache.org
Re: [PR] UTF16 buffer boundary XALANJ-2725 [xalan-java]
Posted by "maxfortun (via GitHub)" <gi...@apache.org>.
maxfortun commented on code in PR #166:
URL: https://github.com/apache/xalan-java/pull/166#discussion_r1465447856
##########
serializer/src/main/java/org/apache/xml/serializer/ToStream.java:
##########
@@ -1595,23 +1599,40 @@ else if (m_encodingInfo.isInEncoding(ch)) {
// not in the normal ASCII range, we also
// just leave it get added on to the clean characters
}
- else if (Encodings.isHighUTF16Surrogate(ch) && i < end-1 && Encodings.isLowUTF16Surrogate(chars[i+1])) {
- // So, this is a (valid) surrogate pair
- if (! m_encodingInfo.isInEncoding(ch, chars[i+1])) {
- int codepoint = Encodings.toCodePoint(ch, chars[i+1]);
- writeOutCleanChars(chars, i, lastDirtyCharProcessed);
- writer.write("&#");
- writer.write(Integer.toString(codepoint));
- writer.write(';');
- lastDirtyCharProcessed = i+1;
- }
- i++; // skip the low surrogate, too
+ else if (Encodings.isHighUTF16Surrogate(ch)) {
+ // Store for later processing. We may be at the end of a buffer,
+ // and must wait till low surrogate arrives
+ // before we can do anything with this.
+ writeOutCleanChars(chars, i, lastDirtyCharProcessed);
+ m_highUTF16Surrogate = ch;
+ lastDirtyCharProcessed = i;
+ }
+ else if (m_highUTF16Surrogate != 0 && Encodings.isLowUTF16Surrogate(ch)) {
+ // The complete utf16 byte sequence is now available and may be serialized.
+ if (! m_encodingInfo.isInEncoding(m_highUTF16Surrogate, ch)) {
+ int codepoint = Encodings.toCodePoint(m_highUTF16Surrogate, ch);
+ writer.write("&#");
+ writer.write(Integer.toString(codepoint));
+ writer.write(';');
+ } else {
+ writer.write(m_highUTF16Surrogate);
+ writer.write(ch);
+ }
+ lastDirtyCharProcessed = i;
+ m_highUTF16Surrogate = 0;
}
else {
// This is a fallback plan, we get here if the
// encoding doesn't contain ch and it's not part
// of a surrogate pair
// The right thing is to write out an entity
+ if(m_highUTF16Surrogate != 0) {
Review Comment:
Btw, I did encounter this scenario, that's why I coded this. But in theory this should not be happening.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@xalan.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@xalan.apache.org
For additional commands, e-mail: dev-help@xalan.apache.org
Re: [PR] UTF16 buffer boundary XALANJ-2725 [xalan-java]
Posted by "jkesselm (via GitHub)" <gi...@apache.org>.
jkesselm commented on PR #166:
URL: https://github.com/apache/xalan-java/pull/166#issuecomment-1908848362
Looks more plausible, though I'm still not sure writing out an NCE for an isolated high surrogate is a win.
I still sorta hate the idea of adding Still More Conditionals to the rendering loop, but I don't see a great alternative.
This doesn't seem to fix what slipped between 2419 and master, alas. Lemme work on that and get back to you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@xalan.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@xalan.apache.org
For additional commands, e-mail: dev-help@xalan.apache.org
Re: [PR] UTF16 buffer boundary XALANJ-2725 [xalan-java]
Posted by "maxfortun (via GitHub)" <gi...@apache.org>.
maxfortun commented on code in PR #166:
URL: https://github.com/apache/xalan-java/pull/166#discussion_r1465446880
##########
serializer/src/main/java/org/apache/xml/serializer/ToStream.java:
##########
@@ -1595,23 +1599,40 @@ else if (m_encodingInfo.isInEncoding(ch)) {
// not in the normal ASCII range, we also
// just leave it get added on to the clean characters
}
- else if (Encodings.isHighUTF16Surrogate(ch) && i < end-1 && Encodings.isLowUTF16Surrogate(chars[i+1])) {
- // So, this is a (valid) surrogate pair
- if (! m_encodingInfo.isInEncoding(ch, chars[i+1])) {
- int codepoint = Encodings.toCodePoint(ch, chars[i+1]);
- writeOutCleanChars(chars, i, lastDirtyCharProcessed);
- writer.write("&#");
- writer.write(Integer.toString(codepoint));
- writer.write(';');
- lastDirtyCharProcessed = i+1;
- }
- i++; // skip the low surrogate, too
+ else if (Encodings.isHighUTF16Surrogate(ch)) {
+ // Store for later processing. We may be at the end of a buffer,
+ // and must wait till low surrogate arrives
+ // before we can do anything with this.
+ writeOutCleanChars(chars, i, lastDirtyCharProcessed);
+ m_highUTF16Surrogate = ch;
+ lastDirtyCharProcessed = i;
+ }
+ else if (m_highUTF16Surrogate != 0 && Encodings.isLowUTF16Surrogate(ch)) {
+ // The complete utf16 byte sequence is now available and may be serialized.
+ if (! m_encodingInfo.isInEncoding(m_highUTF16Surrogate, ch)) {
+ int codepoint = Encodings.toCodePoint(m_highUTF16Surrogate, ch);
+ writer.write("&#");
+ writer.write(Integer.toString(codepoint));
+ writer.write(';');
+ } else {
+ writer.write(m_highUTF16Surrogate);
+ writer.write(ch);
+ }
+ lastDirtyCharProcessed = i;
+ m_highUTF16Surrogate = 0;
}
else {
// This is a fallback plan, we get here if the
// encoding doesn't contain ch and it's not part
// of a surrogate pair
// The right thing is to write out an entity
+ if(m_highUTF16Surrogate != 0) {
Review Comment:
This is probably not a valid scenario and should throw an error. I am not certain what is the correct way to address a situation where we have seen a high surrogate not followed by a low one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@xalan.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@xalan.apache.org
For additional commands, e-mail: dev-help@xalan.apache.org
Re: [PR] UTF16 buffer boundary XALANJ-2725 [xalan-java]
Posted by "maxfortun (via GitHub)" <gi...@apache.org>.
maxfortun commented on code in PR #166:
URL: https://github.com/apache/xalan-java/pull/166#discussion_r1465445384
##########
serializer/src/main/java/org/apache/xml/serializer/ToStream.java:
##########
@@ -1595,23 +1599,40 @@ else if (m_encodingInfo.isInEncoding(ch)) {
// not in the normal ASCII range, we also
// just leave it get added on to the clean characters
}
- else if (Encodings.isHighUTF16Surrogate(ch) && i < end-1 && Encodings.isLowUTF16Surrogate(chars[i+1])) {
- // So, this is a (valid) surrogate pair
- if (! m_encodingInfo.isInEncoding(ch, chars[i+1])) {
- int codepoint = Encodings.toCodePoint(ch, chars[i+1]);
- writeOutCleanChars(chars, i, lastDirtyCharProcessed);
- writer.write("&#");
- writer.write(Integer.toString(codepoint));
- writer.write(';');
- lastDirtyCharProcessed = i+1;
- }
- i++; // skip the low surrogate, too
+ else if (Encodings.isHighUTF16Surrogate(ch)) {
+ // Store for later processing. We may be at the end of a buffer,
+ // and must wait till low surrogate arrives
+ // before we can do anything with this.
+ writeOutCleanChars(chars, i, lastDirtyCharProcessed);
+ m_highUTF16Surrogate = ch;
+ lastDirtyCharProcessed = i;
+ }
+ else if (m_highUTF16Surrogate != 0 && Encodings.isLowUTF16Surrogate(ch)) {
+ // The complete utf16 byte sequence is now available and may be serialized.
+ if (! m_encodingInfo.isInEncoding(m_highUTF16Surrogate, ch)) {
Review Comment:
1602: If ch is high surrogate, dump clean chars and retain ch. This works within the same buffer or across buffer boundaries
1610: if we have a retained high surrogate and ch is a valid low surrogate, we have 2 use cases.
1. encoding doesn't support the multibyte char, escape entity.
2. encoding does support the multibyte char, output chars as is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@xalan.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@xalan.apache.org
For additional commands, e-mail: dev-help@xalan.apache.org
Re: [PR] UTF16 buffer boundary XALANJ-2725 [xalan-java]
Posted by "kubycsolutions (via GitHub)" <gi...@apache.org>.
kubycsolutions commented on PR #166:
URL: https://github.com/apache/xalan-java/pull/166#issuecomment-2368829197
I believe a fix for this issue has already been committed; if you agree, we should close this pull request.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@xalan.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@xalan.apache.org
For additional commands, e-mail: dev-help@xalan.apache.org
Re: [PR] UTF16 buffer boundary XALANJ-2725 [xalan-java]
Posted by "jkesselm (via GitHub)" <gi...@apache.org>.
jkesselm commented on code in PR #166:
URL: https://github.com/apache/xalan-java/pull/166#discussion_r1465440629
##########
serializer/src/main/java/org/apache/xml/serializer/ToStream.java:
##########
@@ -1595,23 +1599,40 @@ else if (m_encodingInfo.isInEncoding(ch)) {
// not in the normal ASCII range, we also
// just leave it get added on to the clean characters
}
- else if (Encodings.isHighUTF16Surrogate(ch) && i < end-1 && Encodings.isLowUTF16Surrogate(chars[i+1])) {
- // So, this is a (valid) surrogate pair
- if (! m_encodingInfo.isInEncoding(ch, chars[i+1])) {
- int codepoint = Encodings.toCodePoint(ch, chars[i+1]);
- writeOutCleanChars(chars, i, lastDirtyCharProcessed);
- writer.write("&#");
- writer.write(Integer.toString(codepoint));
- writer.write(';');
- lastDirtyCharProcessed = i+1;
- }
- i++; // skip the low surrogate, too
+ else if (Encodings.isHighUTF16Surrogate(ch)) {
+ // Store for later processing. We may be at the end of a buffer,
+ // and must wait till low surrogate arrives
+ // before we can do anything with this.
+ writeOutCleanChars(chars, i, lastDirtyCharProcessed);
+ m_highUTF16Surrogate = ch;
+ lastDirtyCharProcessed = i;
+ }
+ else if (m_highUTF16Surrogate != 0 && Encodings.isLowUTF16Surrogate(ch)) {
+ // The complete utf16 byte sequence is now available and may be serialized.
+ if (! m_encodingInfo.isInEncoding(m_highUTF16Surrogate, ch)) {
Review Comment:
I'm a bit confused by your intent here.
The high surrogate is never in any of the encodings, and should never be output by itself, so this case is firing incorrectly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@xalan.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@xalan.apache.org
For additional commands, e-mail: dev-help@xalan.apache.org
Re: [PR] UTF16 buffer boundary XALANJ-2725 [xalan-java]
Posted by "jkesselm (via GitHub)" <gi...@apache.org>.
jkesselm commented on code in PR #166:
URL: https://github.com/apache/xalan-java/pull/166#discussion_r1465440629
##########
serializer/src/main/java/org/apache/xml/serializer/ToStream.java:
##########
@@ -1595,23 +1599,40 @@ else if (m_encodingInfo.isInEncoding(ch)) {
// not in the normal ASCII range, we also
// just leave it get added on to the clean characters
}
- else if (Encodings.isHighUTF16Surrogate(ch) && i < end-1 && Encodings.isLowUTF16Surrogate(chars[i+1])) {
- // So, this is a (valid) surrogate pair
- if (! m_encodingInfo.isInEncoding(ch, chars[i+1])) {
- int codepoint = Encodings.toCodePoint(ch, chars[i+1]);
- writeOutCleanChars(chars, i, lastDirtyCharProcessed);
- writer.write("&#");
- writer.write(Integer.toString(codepoint));
- writer.write(';');
- lastDirtyCharProcessed = i+1;
- }
- i++; // skip the low surrogate, too
+ else if (Encodings.isHighUTF16Surrogate(ch)) {
+ // Store for later processing. We may be at the end of a buffer,
+ // and must wait till low surrogate arrives
+ // before we can do anything with this.
+ writeOutCleanChars(chars, i, lastDirtyCharProcessed);
+ m_highUTF16Surrogate = ch;
+ lastDirtyCharProcessed = i;
+ }
+ else if (m_highUTF16Surrogate != 0 && Encodings.isLowUTF16Surrogate(ch)) {
+ // The complete utf16 byte sequence is now available and may be serialized.
+ if (! m_encodingInfo.isInEncoding(m_highUTF16Surrogate, ch)) {
Review Comment:
I'm a bit confused by your intent here.
The high surrogate is never in any of the encodings, and should never be output by itself, so this case is firing incorrectly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@xalan.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@xalan.apache.org
For additional commands, e-mail: dev-help@xalan.apache.org