You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@fory.apache.org by ch...@apache.org on 2026/10/06 15:46:11 UTC
(fory) branch main updated: perf(java): copy plain runs after the first escape when writing JSON strings (#4122)
This is an automated email from the ASF dual-hosted git repository.
chaokunyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/fory.git
The following commit(s) were added to refs/heads/main by this push:
new f53dd7f15 perf(java): copy plain runs after the first escape when writing JSON strings (#4122)
f53dd7f15 is described below
commit f53dd7f15976be4673fa61cfa2ac70f8939f03ae
Author: Pavel Ptashyts <49...@users.noreply.github.com>
AuthorDate: Tue Oct 6 17:46:06 2026 +0200
perf(java): copy plain runs after the first escape when writing JSON strings (#4122)
## Why?
This is the writing side of #4055 / #4058. Once a string being written
contains its first character that needs escaping (a quote, a backslash,
a control character, or non-ASCII text with `escapeNonAscii`), both JSON
writers write the rest of it one character at a time through
`writeEscapedChar` - a switch, a capacity check and one store per
character - and never return to a bulk copy. The cost of a string
therefore follows the distance from that character to its end, not the
number of characters to escape: one quote near the front of a 20 KB
string makes it 13 times slower to write than with no quote at all. In
`Utf8JsonWriter` a compact latin1 string longer than the fixed-size fast
paths is also rescanned with `charAt` up to that character, so even a
quote at the very end costs 5 times. Details and measurements in #4121.
## What does this PR do?
- `Utf8JsonWriter`: a compact latin1 `String` that the fast paths reject
now goes from `writeStringChars` to `writeLatin1StringEscaped`, which
copies each plain run a word at a time with the existing
`isJsonAsciiWord` predicate, writes only the byte that stopped it
through `writeEscapedChar`, and resumes. The character tails
(`writeStringSlow` for `String` and `CharSequence`) copy the plain ASCII
run up to the next stop with one capacity check per run, then handle the
stop character exactly as before.
- `StringJsonWriter`: the same in all four tails.
`writeLatin1StringSlow` and `writeLatin1StringUtf16Slow` copy plain
words and resume the word copy after a byte above 0x7F, running the word
predicate at most once per eight bytes. In `writeStringSlow` the run
follows the current output coder, because a character above 0xFF
upgrades the output to UTF16 in the middle of a string; the UTF16 runs
(`writeStringSlow`, `writeStringUtf16Slow`) clear `latin1Output` when
they copy such a character.
- Every run copy reserves capacity for at most `RUN_CHUNK` (8192)
characters at a time, the way `writeEscapedUtf16` already bounds its
reservation, so `(length - i) << 1` or `pos + (length - i)` cannot
overflow for a huge string and skip `grow` before the word stores. A run
that ends at a chunk boundary rather than at a stop just continues. The
copy is inlined into each tail: a helper call per stop measurably slowed
strings with many escapes.
- The hot `writeString(String)` methods and their fixed-size fast paths
are untouched; all new code is in the cold paths, following the
C2-layout notes in both classes. The output is byte-identical.
- Tests in `JsonStringTest`:
- `writeEscapeFollowedByOtherText` compares both writers with
`JsonStringEscaper` (a character-at-a-time reference) for a stop
character (quote, backslash, newline, control character, latin1 above
0x7F) followed by latin1 text, 0x7F/0x80, text outside latin1, a
surrogate pair and further escapes, with 0-20 characters before it and
0-8 after it so stops land in every lane of a word, with and without
trailing text, for `String` and `CharSequence` input (each on a fresh
writer), utf8 output and LATIN1 and UTF16 string output, with and
without `escapeNonAscii`. Accepting a quote inside the latin1 run, or
ignoring `escapeNonAscii` in the UTF16 run, makes it fail.
- `writeRunsAcrossReservationChunks`: a quote, latin1 above 0x7F, text
outside latin1 and a surrogate pair at 8192 +/- 9 characters after an
escape, so they land on and around a chunk boundary.
- `writeRejectsUnpairedSurrogateAfterAnEscape`: an unpaired high or low
surrogate after an escape, inside a long plain run and at the end, is
still rejected by both writers, with and without `escapeNonAscii`.
- `writeEscapeThenWideTextAfterUtf16Reset`: a writer reset after a UTF16
result starts in UTF16 while assuming its output fits latin1; the run
copy must withdraw that assumption. Removing the `latin1Output` clear
makes it fail.
- The old code was correct but slow, so these tests pin behaviour and
also pass on it.
## Related issues
- Fixes #4121
- Writing counterpart of #4055 / #4058
## AI Contribution Checklist
- [x] Substantial AI assistance was used in this PR: `yes`
- [x] If `yes`, I included a completed [AI Contribution
Checklist](https://github.com/apache/fory/blob/main/AI_POLICY.md#9-contributor-checklist-for-ai-assisted-prs)
in this PR description and the required `AI Usage Disclosure`.
- [x] If `yes`, my PR description includes the required `ai_review`
summary and screenshot evidence or equivalent persisted links of the
final clean AI review results from both fresh reviewers described in
`AI_POLICY.md`, the Fory-guided reviewer and the independent general
reviewer, on the current PR diff or current HEAD after the latest code
changes.
Completed checklist from `AI_POLICY.md` section 9:
- [x] Substantial AI assistance was used in this PR: `yes`
- [x] I included the standardized `AI Usage Disclosure` block below.
- [x] I can explain and defend all important changes without AI help.
- [x] I reviewed AI-assisted code changes line by line before
submission.
- [x] I completed line-by-line self-review first and fixed issues before
requesting AI review.
- [x] I ran two fresh AI review agents on the current PR diff or current
HEAD after the latest code changes: one Fory-guided reviewer using
`AGENTS.md` and `.agents/ci-and-pr.md`, and one independent general
reviewer in a separate clean-context review session that was not pointed
to `.agents/ci-and-pr.md` or any copied Fory-specific review checklist.
- [x] I addressed all AI review comments and repeated the review loop
until both ai reviewers reported no further actionable comments.
- [x] I attached equivalent persisted links of the final clean AI review
results from both fresh reviewers on the current PR head in this PR
body.
- [x] I ran adequate human verification and recorded evidence (checks
run locally, pass/fail summary, and confirmation I reviewed results).
- [x] I added/updated tests and specs where required.
- [x] I validated protocol/performance impacts with evidence when
applicable.
- [x] I verified licensing and provenance compliance.
```text
AI Usage Disclosure
- substantial_ai_assistance: yes
- scope: investigation, code drafting, tests, benchmark harness, PR text
- affected_files_or_subsystems: java/fory-json writers (Utf8JsonWriter, StringJsonWriter) and JsonStringTest
- ai_review: line-by-line self-review of the AI-drafted change is pending by the contributor (see unchecked items above). Two-reviewer loop on fresh sessions for every round, six rounds: Fory-guided reviewer Codex gpt-6-astra (xhigh), independent general reviewer Claude Fable 5.1. Fixed from review: each input type written by a fresh writer in the test so a CharSequence reaches the mid-string LATIN1-to-UTF16 upgrade itself; 0x7F/0x80 tails and strings ending right after a stop; surrog [...]
- ai_review_artifacts: Fory-guided reviewer: https://github.com/apache/fory/pull/4122#issuecomment-5997450696 ; independent general reviewer: https://github.com/apache/fory/pull/4122#issuecomment-5997451342 (both on commit 72008b9e9a15f2f9fe1de2fdd3cf529a849056dd, the current head, final round, no further actionable comments)
- human_verification: see "Verification" below
- performance_verification: see "Benchmark" below
- provenance_license_confirmation: all code written for this PR against the Apache Fory sources; no third-party code introduced
```
## Verification
JDK 25.0.2, Windows 11 x86_64, from `java/`:
- `mvn -pl fory-json -am install -DskipTests`: success
- `mvn -pl fory-json test`: 1276 tests, 0 failures, 0 errors, 0 skipped
- `JsonStringTest#writeEscapeFollowedByOtherText` with a quote accepted
inside the latin1 run of `Utf8JsonWriter`, and with `escapeNonAscii`
ignored in the `StringJsonWriter` UTF16 run copy: fails, as intended
- `JsonStringTest#writeEscapeThenWideTextAfterUtf16Reset` with the
`latin1Output` clear removed from the UTF16 run copy: fails (`中` comes
back as `-`), as intended
- `mvn -pl fory-json spotless:check checkstyle:check`: success, 0
Checkstyle violations
## Does this PR introduce any user-facing change?
- [x] Does this PR introduce any public API change? No.
- [x] Does this PR introduce any binary protocol compatibility change?
No. The JSON output is byte-identical.
## Benchmark
Standalone harness from #4121 (not JMH): a bid whose `adm` field is a
19998-character VAST XML string, pure ASCII, 612 quotes and 345
newlines, the first quote at character 15. Variants change only the
characters that need escaping inside `adm`, keeping its length.
`ForyJson.builder().build()`, best of 7 rounds of 5000 writes after
30000 warm-up writes, one JVM per build, builds alternating three times
on an otherwise idle machine; JDK 25.0.2. Range over the three runs,
microseconds per write:
| variant | `toJsonBytes` main f85877f2 | `toJsonBytes` this PR |
`toJson` main | `toJson` this PR |
| --- | ---: | ---: | ---: | ---: |
| nothing to escape | 3.6-3.8 | 3.6-3.7 | 3.4-3.5 | 3.4-3.5 |
| one quote at the very end | 18.7-19.0 | 6.0-6.4 | 3.3-3.4 | 3.3-3.4 |
| one quote at character 1 | 44.5-45.0 | 4.2-4.6 | 33.8-36.8 | 3.7-4.0 |
| the real string | 44.3-45.0 | 9.9-10.4 | 36.5-40.7 | 14.6-15.8 |
A bean of eight short strings, four of them with one escape each:
286-291 ns -> 181-191 ns (`toJsonBytes`), 255-272 ns -> 231-235 ns
(`toJson`). The output digest is identical between the builds in every
case. For reference, Jackson 3.1 `writeValueAsString` takes 12.5 us on
the string without escapes and 25.0 us on the real one.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <no...@anthropic.com>
Co-authored-by: chaokunyang <sh...@gmail.com>
---
.../apache/fory/json/writer/StringJsonWriter.java | 255 +++++++++++++++++++--
.../apache/fory/json/writer/Utf8JsonWriter.java | 99 +++++++-
.../java/org/apache/fory/json/JsonStringTest.java | 132 +++++++++++
3 files changed, 458 insertions(+), 28 deletions(-)
diff --git a/java/fory-json/src/main/java/org/apache/fory/json/writer/StringJsonWriter.java b/java/fory-json/src/main/java/org/apache/fory/json/writer/StringJsonWriter.java
index 8fa7df05c..f65eb249e 100644
--- a/java/fory-json/src/main/java/org/apache/fory/json/writer/StringJsonWriter.java
+++ b/java/fory-json/src/main/java/org/apache/fory/json/writer/StringJsonWriter.java
@@ -78,6 +78,9 @@ public final class StringJsonWriter extends JsonWriter implements Appendable {
private static final byte[] NEGATIVE_INFINITY_BYTES =
"\"-Infinity\"".getBytes(StandardCharsets.ISO_8859_1);
private static final long DECIMAL_8 = 100_000_000L;
+ // Bound each run reservation. Capacity checks must subtract the cursor from buffer length;
+ // adding even this bounded increment can overflow near the maximum array size.
+ private static final int RUN_CHUNK = 8192;
private static final int[] DIGIT_TRIPLES = new int[1000];
private static final int[] DIGIT_QUADS = new int[10000];
private static final int[] HEX_PAIRS = new int[256];
@@ -1910,13 +1913,52 @@ public final class StringJsonWriter extends JsonWriter implements Appendable {
}
private void writeStringSlow(String value, int index, int length) {
- for (int i = index; i < length; i++) {
- char ch = value.charAt(i);
+ int i = index;
+ while (i < length) {
+ // Copy the plain run up to the next character that needs escaping or a surrogate. One
+ // reservation covers at most RUN_CHUNK characters, keeping its byte count bounded. A
+ // character above 0xFF upgrades the output to UTF16 in the middle of the string, so the copy
+ // follows the current coder; in LATIN1 output it stops at such a character.
+ int limit = length - i > RUN_CHUNK ? i + RUN_CHUNK : length;
+ int pos = position;
+ char ch;
+ if (coder == LATIN1) {
+ if (limit - i > buffer.length - pos) {
+ grow(limit - i);
+ }
+ byte[] bytes = buffer;
+ int maxPlain = escapeNonAscii ? 0x7f : 0xff;
+ while (i < limit && isJsonLatin1(ch = value.charAt(i)) && ch <= maxPlain) {
+ bytes[pos++] = (byte) ch;
+ i++;
+ }
+ } else {
+ int additional = (limit - i) << 1;
+ if (additional > buffer.length - pos) {
+ grow(additional);
+ }
+ byte[] bytes = buffer;
+ int maxPlain = escapeNonAscii ? 0x7f : 0xffff;
+ int widest = 0;
+ while (i < limit && isJsonUtf16(ch = value.charAt(i)) && ch <= maxPlain) {
+ widest |= ch;
+ pos = putUtf16Char(bytes, pos, ch);
+ i++;
+ }
+ if (widest > 0xff) {
+ latin1Output = false;
+ }
+ }
+ position = pos;
+ if (i == limit) {
+ continue;
+ }
+ ch = value.charAt(i++);
if (Character.isHighSurrogate(ch)) {
- if (i + 1 >= length) {
+ if (i >= length) {
throw new ForyJsonException("Unpaired high surrogate in string");
}
- char low = value.charAt(++i);
+ char low = value.charAt(i++);
if (!Character.isLowSurrogate(low)) {
throw new ForyJsonException("Unpaired high surrogate in string");
}
@@ -1937,13 +1979,52 @@ public final class StringJsonWriter extends JsonWriter implements Appendable {
}
private void writeStringSlow(CharSequence value, int index, int length) {
- for (int i = index; i < length; i++) {
- char ch = value.charAt(i);
+ int i = index;
+ while (i < length) {
+ // Copy the plain run up to the next character that needs escaping or a surrogate. One
+ // reservation covers at most RUN_CHUNK characters, keeping its byte count bounded. A
+ // character above 0xFF upgrades the output to UTF16 in the middle of the string, so the copy
+ // follows the current coder; in LATIN1 output it stops at such a character.
+ int limit = length - i > RUN_CHUNK ? i + RUN_CHUNK : length;
+ int pos = position;
+ char ch;
+ if (coder == LATIN1) {
+ if (limit - i > buffer.length - pos) {
+ grow(limit - i);
+ }
+ byte[] bytes = buffer;
+ int maxPlain = escapeNonAscii ? 0x7f : 0xff;
+ while (i < limit && isJsonLatin1(ch = value.charAt(i)) && ch <= maxPlain) {
+ bytes[pos++] = (byte) ch;
+ i++;
+ }
+ } else {
+ int additional = (limit - i) << 1;
+ if (additional > buffer.length - pos) {
+ grow(additional);
+ }
+ byte[] bytes = buffer;
+ int maxPlain = escapeNonAscii ? 0x7f : 0xffff;
+ int widest = 0;
+ while (i < limit && isJsonUtf16(ch = value.charAt(i)) && ch <= maxPlain) {
+ widest |= ch;
+ pos = putUtf16Char(bytes, pos, ch);
+ i++;
+ }
+ if (widest > 0xff) {
+ latin1Output = false;
+ }
+ }
+ position = pos;
+ if (i == limit) {
+ continue;
+ }
+ ch = value.charAt(i++);
if (Character.isHighSurrogate(ch)) {
- if (i + 1 >= length) {
+ if (i >= length) {
throw new ForyJsonException("Unpaired high surrogate in string");
}
- char low = value.charAt(++i);
+ char low = value.charAt(i++);
if (!Character.isLowSurrogate(low)) {
throw new ForyJsonException("Unpaired high surrogate in string");
}
@@ -1963,9 +2044,51 @@ public final class StringJsonWriter extends JsonWriter implements Appendable {
writeByteRaw((byte) '"');
}
+ /**
+ * Writes the rest of a compact Latin1 string into LATIN1 output after the first byte that needs
+ * escaping. Plain runs between such bytes are copied a word at a time, so the cost follows the
+ * number of escapes rather than the distance from the first one to the end; a word the predicate
+ * rejects for a byte above 0x7F is copied byte by byte up to its end, so the predicate runs at
+ * most once per eight bytes. One reservation covers at most RUN_CHUNK bytes, keeping its byte
+ * count bounded. Only reached without non-ASCII escaping: bytes above 0x7F are copied as they
+ * are.
+ */
private void writeLatin1StringSlow(byte[] value, int index, int length) {
- for (int i = index; i < length; i++) {
- writeEscapedChar((char) (value[i] & 0xff));
+ int i = index;
+ while (i < length) {
+ int limit = length - i > RUN_CHUNK ? i + RUN_CHUNK : length;
+ int pos = position;
+ int additional = limit - i;
+ if (additional > buffer.length - pos) {
+ grow(additional);
+ }
+ byte[] bytes = buffer;
+ int wordEnd = limit - Long.BYTES;
+ while (i < limit) {
+ int windowEnd = limit;
+ if (i <= wordEnd) {
+ long word = LittleEndian.getInt64(value, i);
+ if (isJsonAsciiWord(word)) {
+ LittleEndian.putInt64(bytes, pos, word);
+ pos += Long.BYTES;
+ i += Long.BYTES;
+ continue;
+ }
+ windowEnd = i + Long.BYTES;
+ }
+ while (i < windowEnd && isJsonLatin1Byte(value[i])) {
+ bytes[pos++] = value[i];
+ i++;
+ }
+ if (i < windowEnd) {
+ break;
+ }
+ }
+ position = pos;
+ if (i == limit) {
+ continue;
+ }
+ writeEscapedChar((char) (value[i++] & 0xff));
}
writeByteRaw((byte) '"');
}
@@ -2224,21 +2347,88 @@ public final class StringJsonWriter extends JsonWriter implements Appendable {
writeUtf16ByteNoEnsure((byte) '"');
}
+ /**
+ * Writes the rest of a compact Latin1 string into UTF16 output after the first byte that needs
+ * escaping. Plain runs between such bytes are copied a word at a time, so the cost follows the
+ * number of escapes rather than the distance from the first one to the end; a word the predicate
+ * rejects for a byte above 0x7F is copied byte by byte up to its end, so the predicate runs at
+ * most once per eight bytes. One reservation covers at most RUN_CHUNK bytes, keeping its byte
+ * count bounded. Only reached without non-ASCII escaping: bytes above 0x7F are copied as they
+ * are.
+ */
private void writeLatin1StringUtf16Slow(byte[] value, int index, int length) {
- for (int i = index; i < length; i++) {
- writeEscapedChar((char) (value[i] & 0xff));
+ int i = index;
+ while (i < length) {
+ int limit = length - i > RUN_CHUNK ? i + RUN_CHUNK : length;
+ int pos = position;
+ int additional = (limit - i) << 1;
+ if (additional > buffer.length - pos) {
+ grow(additional);
+ }
+ byte[] bytes = buffer;
+ int wordEnd = limit - Long.BYTES;
+ while (i < limit) {
+ int windowEnd = limit;
+ if (i <= wordEnd) {
+ long word = LittleEndian.getInt64(value, i);
+ if (isJsonAsciiWord(word)) {
+ putLatin1WordAsUtf16(bytes, pos, word);
+ pos += Long.BYTES << 1;
+ i += Long.BYTES;
+ continue;
+ }
+ windowEnd = i + Long.BYTES;
+ }
+ while (i < windowEnd && isJsonLatin1Byte(value[i])) {
+ pos = putUtf16Char(bytes, pos, (char) (value[i] & 0xff));
+ i++;
+ }
+ if (i < windowEnd) {
+ break;
+ }
+ }
+ position = pos;
+ if (i == limit) {
+ continue;
+ }
+ writeEscapedChar((char) (value[i++] & 0xff));
}
writeByteRaw((byte) '"');
}
private void writeStringUtf16Slow(String value, int index, int length) {
- for (int i = index; i < length; i++) {
- char ch = value.charAt(i);
+ int i = index;
+ while (i < length) {
+ // Copy the plain run up to the next character that needs escaping or a surrogate. One
+ // reservation covers at most RUN_CHUNK characters, keeping its byte count bounded.
+ int limit = length - i > RUN_CHUNK ? i + RUN_CHUNK : length;
+ int pos = position;
+ int additional = (limit - i) << 1;
+ if (additional > buffer.length - pos) {
+ grow(additional);
+ }
+ byte[] bytes = buffer;
+ int maxPlain = escapeNonAscii ? 0x7f : 0xffff;
+ int widest = 0;
+ char ch;
+ while (i < limit && isJsonUtf16(ch = value.charAt(i)) && ch <= maxPlain) {
+ widest |= ch;
+ pos = putUtf16Char(bytes, pos, ch);
+ i++;
+ }
+ if (widest > 0xff) {
+ latin1Output = false;
+ }
+ position = pos;
+ if (i == limit) {
+ continue;
+ }
+ ch = value.charAt(i++);
if (Character.isHighSurrogate(ch)) {
- if (i + 1 >= length) {
+ if (i >= length) {
throw new ForyJsonException("Unpaired high surrogate in string");
}
- char low = value.charAt(++i);
+ char low = value.charAt(i++);
if (!Character.isLowSurrogate(low)) {
throw new ForyJsonException("Unpaired high surrogate in string");
}
@@ -2258,13 +2448,38 @@ public final class StringJsonWriter extends JsonWriter implements Appendable {
}
private void writeStringUtf16Slow(CharSequence value, int index, int length) {
- for (int i = index; i < length; i++) {
- char ch = value.charAt(i);
+ int i = index;
+ while (i < length) {
+ // Copy the plain run up to the next character that needs escaping or a surrogate. One
+ // reservation covers at most RUN_CHUNK characters, keeping its byte count bounded.
+ int limit = length - i > RUN_CHUNK ? i + RUN_CHUNK : length;
+ int pos = position;
+ int additional = (limit - i) << 1;
+ if (additional > buffer.length - pos) {
+ grow(additional);
+ }
+ byte[] bytes = buffer;
+ int maxPlain = escapeNonAscii ? 0x7f : 0xffff;
+ int widest = 0;
+ char ch;
+ while (i < limit && isJsonUtf16(ch = value.charAt(i)) && ch <= maxPlain) {
+ widest |= ch;
+ pos = putUtf16Char(bytes, pos, ch);
+ i++;
+ }
+ if (widest > 0xff) {
+ latin1Output = false;
+ }
+ position = pos;
+ if (i == limit) {
+ continue;
+ }
+ ch = value.charAt(i++);
if (Character.isHighSurrogate(ch)) {
- if (i + 1 >= length) {
+ if (i >= length) {
throw new ForyJsonException("Unpaired high surrogate in string");
}
- char low = value.charAt(++i);
+ char low = value.charAt(i++);
if (!Character.isLowSurrogate(low)) {
throw new ForyJsonException("Unpaired high surrogate in string");
}
diff --git a/java/fory-json/src/main/java/org/apache/fory/json/writer/Utf8JsonWriter.java b/java/fory-json/src/main/java/org/apache/fory/json/writer/Utf8JsonWriter.java
index 9738476a5..1b87eba0c 100644
--- a/java/fory-json/src/main/java/org/apache/fory/json/writer/Utf8JsonWriter.java
+++ b/java/fory-json/src/main/java/org/apache/fory/json/writer/Utf8JsonWriter.java
@@ -87,6 +87,9 @@ public final class Utf8JsonWriter extends JsonWriter implements Appendable {
private static final byte[] NEGATIVE_INFINITY_BYTES =
"\"-Infinity\"".getBytes(java.nio.charset.StandardCharsets.ISO_8859_1);
private static final long EIGHT_DIGITS = 100_000_000L;
+ // Bound each run reservation. Capacity checks must subtract the cursor from buffer length;
+ // adding even this bounded increment can overflow near the maximum array size.
+ private static final int RUN_CHUNK = 8192;
private static final int[] HEX_PAIRS = new int[256];
private static final long UTF16_ASCII_MASK = 0xFF80FF80FF80FF80L;
private static final char[] DIGIT_PAIRS = new char[256];
@@ -1141,6 +1144,11 @@ public final class Utf8JsonWriter extends JsonWriter implements Appendable {
}
private void writeStringChars(String value) {
+ if (STRING_BYTES_BACKED
+ && StringSerializer.isLatin1Coder(StringSerializer.getStringCoder(value))) {
+ writeLatin1StringEscaped(StringSerializer.getStringBytes(value));
+ return;
+ }
int length = value.length();
int pos = position;
int additional = length + 2;
@@ -2492,14 +2500,71 @@ public final class Utf8JsonWriter extends JsonWriter implements Appendable {
}
}
+ /**
+ * Writes a compact Latin1 string that the fast paths rejected because it needs escaping or holds
+ * a byte above 0x7F. Plain runs between those bytes are copied a word at a time, so the cost
+ * follows the number of stops rather than the distance from the first one to the end.
+ */
+ private void writeLatin1StringEscaped(byte[] value) {
+ writeByteRaw((byte) '"');
+ int length = value.length;
+ int i = 0;
+ while (i < length) {
+ // One reservation covers at most RUN_CHUNK bytes, keeping its byte count bounded.
+ int limit = length - i > RUN_CHUNK ? i + RUN_CHUNK : length;
+ int pos = position;
+ if (limit - i > buffer.length - pos) {
+ grow(limit - i);
+ }
+ byte[] bytes = buffer;
+ int wordEnd = limit - Long.BYTES;
+ while (i <= wordEnd) {
+ long word = LittleEndian.getInt64(value, i);
+ if (!isJsonAsciiWord(word)) {
+ break;
+ }
+ LittleEndian.putInt64(bytes, pos, word);
+ pos += Long.BYTES;
+ i += Long.BYTES;
+ }
+ while (i < limit && isJsonAsciiByte(value[i])) {
+ bytes[pos++] = value[i++];
+ }
+ position = pos;
+ if (i == limit) {
+ continue;
+ }
+ writeEscapedChar((char) (value[i++] & 0xff));
+ }
+ writeByteRaw((byte) '"');
+ }
+
private void writeStringSlow(String value, int index, int length) {
- for (int i = index; i < length; i++) {
- char ch = value.charAt(i);
+ int i = index;
+ while (i < length) {
+ // Copy the plain ASCII run up to the next character that needs escaping or encoding. One
+ // reservation covers at most RUN_CHUNK characters, keeping its byte count bounded.
+ int limit = length - i > RUN_CHUNK ? i + RUN_CHUNK : length;
+ int pos = position;
+ if (limit - i > buffer.length - pos) {
+ grow(limit - i);
+ }
+ byte[] bytes = buffer;
+ char ch;
+ while (i < limit && isJsonAscii(ch = value.charAt(i))) {
+ bytes[pos++] = (byte) ch;
+ i++;
+ }
+ position = pos;
+ if (i == limit) {
+ continue;
+ }
+ ch = value.charAt(i++);
if (Character.isHighSurrogate(ch)) {
- if (i + 1 >= length) {
+ if (i >= length) {
throw new ForyJsonException("Unpaired high surrogate in string");
}
- char low = value.charAt(++i);
+ char low = value.charAt(i++);
if (!Character.isLowSurrogate(low)) {
throw new ForyJsonException("Unpaired high surrogate in string");
}
@@ -2519,13 +2584,31 @@ public final class Utf8JsonWriter extends JsonWriter implements Appendable {
}
private void writeStringSlow(CharSequence value, int index, int length) {
- for (int i = index; i < length; i++) {
- char ch = value.charAt(i);
+ int i = index;
+ while (i < length) {
+ // Copy the plain ASCII run up to the next character that needs escaping or encoding. One
+ // reservation covers at most RUN_CHUNK characters, keeping its byte count bounded.
+ int limit = length - i > RUN_CHUNK ? i + RUN_CHUNK : length;
+ int pos = position;
+ if (limit - i > buffer.length - pos) {
+ grow(limit - i);
+ }
+ byte[] bytes = buffer;
+ char ch;
+ while (i < limit && isJsonAscii(ch = value.charAt(i))) {
+ bytes[pos++] = (byte) ch;
+ i++;
+ }
+ position = pos;
+ if (i == limit) {
+ continue;
+ }
+ ch = value.charAt(i++);
if (Character.isHighSurrogate(ch)) {
- if (i + 1 >= length) {
+ if (i >= length) {
throw new ForyJsonException("Unpaired high surrogate in string");
}
- char low = value.charAt(++i);
+ char low = value.charAt(i++);
if (!Character.isLowSurrogate(low)) {
throw new ForyJsonException("Unpaired high surrogate in string");
}
diff --git a/java/fory-json/src/test/java/org/apache/fory/json/JsonStringTest.java b/java/fory-json/src/test/java/org/apache/fory/json/JsonStringTest.java
index 1eea3eeac..790dee254 100644
--- a/java/fory-json/src/test/java/org/apache/fory/json/JsonStringTest.java
+++ b/java/fory-json/src/test/java/org/apache/fory/json/JsonStringTest.java
@@ -64,6 +64,7 @@ import org.apache.fory.json.reader.JsonReader;
import org.apache.fory.json.reader.Latin1JsonReader;
import org.apache.fory.json.reader.Utf16JsonReader;
import org.apache.fory.json.reader.Utf8JsonReader;
+import org.apache.fory.json.writer.JsonStringEscaper;
import org.apache.fory.json.writer.JsonWriter;
import org.apache.fory.json.writer.StringJsonWriter;
import org.apache.fory.json.writer.Utf8JsonWriter;
@@ -1071,6 +1072,137 @@ public class JsonStringTest extends ForyJsonTestModels {
}
}
+ /**
+ * After a character that needs escaping, plain text is copied in runs, and each run has to stop
+ * wherever the character-at-a-time path would have handled the character itself - another escape,
+ * latin1 text above 0x7F, text outside latin1 and a surrogate pair - for a String and a
+ * CharSequence, for utf8 output and for LATIN1 and UTF16 string output, with and without escaping
+ * non-ASCII text. {@link JsonStringEscaper} is the character-at-a-time reference.
+ */
+ @Test
+ public void writeEscapeFollowedByOtherText() {
+ // Text outside latin1 as the first stop upgrades LATIN1 string output to UTF16 mid-string, so
+ // the escapes in the tail that follows go through the UTF16 run copy of the same string.
+ String[] stops = {"\"", "\\", "\n", "\u0001", "é", "中", "😀"};
+ String[] tails = {
+ "a",
+ "é",
+ "ÿ",
+ "\u007f\u0080",
+ "中文",
+ "😀",
+ "aé中文b",
+ "é" + repeat('c', 40),
+ "\"\\\t" + repeat('d', 9) + "\""
+ };
+ // An empty end makes the string finish right after its last stop character.
+ String[] ends = {"", repeat('b', 7)};
+ for (boolean escapeNonAscii : new boolean[] {false, true}) {
+ JsonConfig config = newJsonBuilder().escapeNonAscii(escapeNonAscii).build().config();
+ for (String stop : stops) {
+ for (String tail : tails) {
+ for (String end : ends) {
+ for (int lead = 0; lead <= 20; lead++) {
+ for (int gap = 0; gap <= 8; gap++) {
+ String value = repeat('a', lead) + stop + repeat('c', gap) + tail + end;
+ assertWrittenLikeEscaper(config, value);
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * The run copy sits in front of the surrogate checks, so an unpaired surrogate after an escape
+ * must still be rejected, both inside a long plain run and at the end of the string.
+ */
+ @Test
+ public void writeRejectsUnpairedSurrogateAfterAnEscape() {
+ String plain = repeat('a', 24);
+ for (boolean escapeNonAscii : new boolean[] {false, true}) {
+ JsonConfig config = newJsonBuilder().escapeNonAscii(escapeNonAscii).build().config();
+ for (String value :
+ new String[] {
+ "x\n" + plain + "\uD83D" + plain,
+ "x\n" + plain + "\uDE00" + plain,
+ "x\n" + plain + "\uD83D",
+ "x\n中" + plain + "\uD83D" + plain
+ }) {
+ Utf8JsonWriter utf8 =
+ new Utf8JsonWriter(config, newUtf8Writer().typeResolver(), new byte[1]);
+ assertThrows(ForyJsonException.class, () -> utf8.writeString(value));
+ assertThrows(ForyJsonException.class, () -> utf8.writeString(new StringBuilder(value)));
+ StringJsonWriter string =
+ new StringJsonWriter(config, newStringWriter().typeResolver(), new byte[1]);
+ assertThrows(ForyJsonException.class, () -> string.writeString(value));
+ assertThrows(ForyJsonException.class, () -> string.writeString(new StringBuilder(value)));
+ }
+ }
+ }
+
+ /**
+ * A plain run longer than one reservation chunk is copied in several, so a stop character, latin1
+ * text above 0x7F, text outside latin1 or a surrogate pair right at a chunk boundary must come
+ * out the same as anywhere else.
+ */
+ @Test
+ public void writeRunsAcrossReservationChunks() {
+ int chunk = 8192;
+ String[] boundaryTexts = {"\"", "é", "中", "😀"};
+ for (boolean escapeNonAscii : new boolean[] {false, true}) {
+ JsonConfig config = newJsonBuilder().escapeNonAscii(escapeNonAscii).build().config();
+ for (String text : boundaryTexts) {
+ for (int offset = -9; offset <= 9; offset++) {
+ String value = "x\n" + repeat('a', chunk + offset) + text + repeat('b', chunk + 20);
+ assertWrittenLikeEscaper(config, value);
+ }
+ }
+ }
+ }
+
+ /**
+ * A writer reset after a UTF16 result starts the next document in UTF16 while still assuming its
+ * output fits latin1; the run copy after an escape must withdraw that assumption when it copies a
+ * character above 0xFF, or the result is narrowed and the character is lost.
+ */
+ @Test
+ public void writeEscapeThenWideTextAfterUtf16Reset() {
+ StringJsonWriter writer = newStringWriter(new byte[16]);
+ for (CharSequence value : new CharSequence[] {"a\"中", new StringBuilder("a\"中")}) {
+ writer.writeRawValue("中");
+ assertEquals(writer.toJson(), "中");
+ writer.reset();
+ writer.writeString(value);
+ assertEquals(writer.toJson(), "\"a\\\"中\"");
+ writer.reset();
+ }
+ }
+
+ private static void assertWrittenLikeEscaper(JsonConfig config, String value) {
+ String expected =
+ new String(
+ JsonStringEscaper.utf8Value(value, config.escapeNonAscii()), StandardCharsets.UTF_8);
+ // Each input type gets a fresh writer, so a String never leaves the output coder upgraded for
+ // the CharSequence that follows it.
+ for (CharSequence input : new CharSequence[] {value, new StringBuilder(value)}) {
+ Utf8JsonWriter utf8 = new Utf8JsonWriter(config, newUtf8Writer().typeResolver(), new byte[1]);
+ utf8.writeString(input);
+ assertEquals(new String(utf8.toJsonBytes(), StandardCharsets.UTF_8), expected, value);
+ StringJsonWriter latin1 =
+ new StringJsonWriter(config, newStringWriter().typeResolver(), new byte[1]);
+ latin1.writeString(input);
+ assertEquals(latin1.toJson(), expected, value);
+ // Raw text outside latin1 switches string output to UTF16 first, whatever the escape setting.
+ StringJsonWriter utf16 =
+ new StringJsonWriter(config, newStringWriter().typeResolver(), new byte[1]);
+ utf16.writeRawValue("中");
+ utf16.writeString(input);
+ assertEquals(utf16.toJson(), "中" + expected, value);
+ }
+ }
+
@Test(dataProvider = "enableCodegen")
public void readUnicodeFieldNames(boolean codegen) {
ForyJson json = newJson(codegen);
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org