You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@fory.apache.org by ch...@apache.org on 2026/09/20 03:54:08 UTC
(fory) branch main updated: perf(java): share JSON reader and writer byte buffers (#4061)
This is an automated email from the ASF dual-hosted git repository.
chaokunyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/fory.git
The following commit(s) were added to refs/heads/main by this push:
new c0b0464f0 perf(java): share JSON reader and writer byte buffers (#4061)
c0b0464f0 is described below
commit c0b0464f0594e9d2ef7520d756b1b58883a88ea1
Author: Shawn Yang <sh...@gmail.com>
AuthorDate: Sun Sep 20 11:54:04 2026 +0800
perf(java): share JSON reader and writer byte buffers (#4061)
## Why?
## What does this PR do?
Reuse buffer between writer/reader to reduce memory footprint
## Related issues
#4058
## AI Contribution Checklist
- [ ] Substantial AI assistance was used in this PR: `yes` / `no`
- [ ] If `yes`, I included a completed [AI Contribution
Checklist](https://github.com/apache/fory/blob/main/AI_POLICY.md#9-contributor-checklist-for-ai-assisted-prs)
in this PR description and the required `AI Usage Disclosure`.
- [ ] If `yes`, my PR description includes the required `ai_review`
summary and screenshot evidence or equivalent persisted links of the
final clean AI review results from both fresh reviewers described in
`AI_POLICY.md`, the Fory-guided reviewer and the independent general
reviewer, on the current PR diff or current HEAD after the latest code
changes.
## Does this PR introduce any user-facing change?
- [ ] Does this PR introduce any public API change?
- [ ] Does this PR introduce any binary protocol compatibility change?
## Benchmark
---
.agents/languages/java.md | 9 +
docs/json/object-mapping.md | 5 +-
.../main/java/org/apache/fory/json/ForyJson.java | 110 +++++--
.../java/org/apache/fory/json/ForyJsonBuilder.java | 6 +-
.../apache/fory/json/reader/Latin1JsonReader.java | 66 ++--
.../apache/fory/json/reader/Utf16JsonReader.java | 64 ++--
.../apache/fory/json/reader/Utf8JsonReader.java | 62 ++--
.../apache/fory/json/writer/StringJsonWriter.java | 26 +-
.../apache/fory/json/writer/Utf8JsonWriter.java | 15 +-
.../apache/fory/json/ForyJsonByteRangeTest.java | 6 +-
.../apache/fory/json/ForyJsonConcurrencyTest.java | 44 ++-
.../org/apache/fory/json/JsonContainerTest.java | 4 +-
.../apache/fory/json/JsonFieldNameCacheTest.java | 8 +-
.../java/org/apache/fory/json/JsonScalarTest.java | 82 ++---
.../java/org/apache/fory/json/JsonStringTest.java | 351 +++++++++++++++------
.../org/apache/fory/json/JsonTemporalTest.java | 110 ++++---
.../java/org/apache/fory/json/JsonTestSupport.java | 8 +-
.../apache/fory/json/reader/ZoneIdCacheTest.java | 10 +-
18 files changed, 619 insertions(+), 367 deletions(-)
diff --git a/.agents/languages/java.md b/.agents/languages/java.md
index fa3911ee6..192a651ce 100644
--- a/.agents/languages/java.md
+++ b/.agents/languages/java.md
@@ -24,6 +24,15 @@ Load this file when changing anything under `java/` or when Java drives a cross-
- `fory-core` targets Java 8 bytecode and `fory-format` targets Java 11 bytecode. Do not use newer APIs in those modules.
- `fory-json` must not depend on or reference `jdk.incubator.vector`, including production and
multi-release sources, module descriptors, Maven wiring, and optional runtime paths.
+- Each Fory JSON `JsonState` owns one shared main byte buffer while idle and transfers it to the
+ selected concrete reader or writer for a root operation. Reclaim the latest grown or swapped
+ array before clearing that component's buffer reference, including on failure. Keep hot paths
+ on concrete byte-array fields; do not add a buffer holder, reader/writer backreference, or
+ per-value handoff. Preserve the separate UTF16 input mirror and String writer widening scratch
+ because they can be used at the same time as the main buffer.
+- JSON reader reset methods take the decode buffer explicitly and assign it directly. The caller
+ owns initial allocation; do not add a decode-buffer null check, fallback allocation, or separate
+ setter to root setup. Keep capacity growth in the existing decoding owner.
- Put a `fory-json` optimization in the earliest multi-release overlay whose public JDK APIs
support it. In particular, `Math.multiplyHigh` and `VarHandle` implementations belong in the
Java 9 overlay, not the Java 25 overlay; keep only the Java 8 compatibility implementation in
diff --git a/docs/json/object-mapping.md b/docs/json/object-mapping.md
index acde2f3b7..14d8c0593 100644
--- a/docs/json/object-mapping.md
+++ b/docs/json/object-mapping.md
@@ -237,7 +237,7 @@ An explicit `JsonProperty.include` overrides the builder default. See
| `withPropertyNamingStrategy(strategy)` | `LOWER_CAMEL_CASE` | Name properties without an explicit `JsonProperty` name |
| `withMaxCachedFieldNames(int)` | `DEFAULT_MAX_CACHED_FIELD_NAMES` (`8192`) | Field-name cache entries per reader; zero disables caching |
| `withConcurrencyLevel(int)` | `max(1, 2 * processors)` | Maximum concurrent root operations |
-| `withBufferSizeLimitBytes(int)` | 2 MiB | Maximum reusable capacity retained by each pooled writer |
+| `withBufferSizeLimitBytes(int)` | 2 MiB | Maximum retained output/string-decoding buffer capacity |
| `registerCodec(type, codec)` | None | Replace an eligible exact class's complete JSON codec |
| `registerMixin(mixinType)` | None | Apply one annotation Mixin to its exact declared target |
@@ -252,7 +252,8 @@ their own output shape.
Concurrency-level and buffer-retention limits must be positive. The cached-field-name limit
applies independently to each reader; zero disables this cache. It bounds only cached field names,
not names accepted from the input. The buffer-retention setting does not limit JSON input or output
-size, only reusable writer storage retained after an operation.
+size. It limits the capacity of each reusable output or string-decoding buffer after an operation,
+not the total memory used by the instance.
For class loading, type policy, nesting depth, graph-memory limits, and external input controls,
see [Fory JSON Security](security.md).
diff --git a/java/fory-json/src/main/java/org/apache/fory/json/ForyJson.java b/java/fory-json/src/main/java/org/apache/fory/json/ForyJson.java
index 4cebd03ca..75a90977c 100644
--- a/java/fory-json/src/main/java/org/apache/fory/json/ForyJson.java
+++ b/java/fory-json/src/main/java/org/apache/fory/json/ForyJson.java
@@ -170,7 +170,7 @@ public final class ForyJson {
public String toJson(Object value) {
PooledState entry = acquire();
JsonState state = entry.state;
- StringJsonWriter writer = state.stringWriter;
+ StringJsonWriter writer = state.stringWriter();
try {
state.typeResolver.lockJIT();
try {
@@ -186,7 +186,8 @@ public final class ForyJson {
return writer.toJson();
} finally {
try {
- writer.reset();
+ state.buffer = writer.getBuffer();
+ writer.clear();
} finally {
release(entry);
}
@@ -236,7 +237,7 @@ public final class ForyJson {
public byte[] toJsonBytes(Object value) {
PooledState entry = acquire();
JsonState state = entry.state;
- Utf8JsonWriter writer = state.utf8Writer;
+ Utf8JsonWriter writer = state.utf8Writer();
try {
state.typeResolver.lockJIT();
try {
@@ -252,7 +253,8 @@ public final class ForyJson {
return writer.toJsonBytes();
} finally {
try {
- writer.reset();
+ state.buffer = writer.getBuffer();
+ writer.clear();
} finally {
release(entry);
}
@@ -299,7 +301,7 @@ public final class ForyJson {
Objects.requireNonNull(output, "output");
PooledState entry = acquire();
JsonState state = entry.state;
- Utf8JsonWriter writer = state.utf8Writer;
+ Utf8JsonWriter writer = state.utf8Writer();
try {
state.typeResolver.lockJIT();
try {
@@ -316,7 +318,8 @@ public final class ForyJson {
writer.writeTo(output);
} finally {
try {
- writer.reset();
+ state.buffer = writer.getBuffer();
+ writer.clear();
} finally {
release(entry);
}
@@ -354,7 +357,7 @@ public final class ForyJson {
private String toJsonDeclared(Object value, Class<?> type) {
PooledState entry = acquire();
JsonState state = entry.state;
- StringJsonWriter writer = state.stringWriter;
+ StringJsonWriter writer = state.stringWriter();
try {
state.typeResolver.lockJIT();
try {
@@ -365,7 +368,8 @@ public final class ForyJson {
return writer.toJson();
} finally {
try {
- writer.reset();
+ state.buffer = writer.getBuffer();
+ writer.clear();
} finally {
release(entry);
}
@@ -375,7 +379,7 @@ public final class ForyJson {
private String toJsonDeclared(Object value, TypeRef<?> type) {
PooledState entry = acquire();
JsonState state = entry.state;
- StringJsonWriter writer = state.stringWriter;
+ StringJsonWriter writer = state.stringWriter();
try {
state.typeResolver.lockJIT();
try {
@@ -386,7 +390,8 @@ public final class ForyJson {
return writer.toJson();
} finally {
try {
- writer.reset();
+ state.buffer = writer.getBuffer();
+ writer.clear();
} finally {
release(entry);
}
@@ -396,7 +401,7 @@ public final class ForyJson {
private byte[] toJsonBytesDeclared(Object value, Class<?> type) {
PooledState entry = acquire();
JsonState state = entry.state;
- Utf8JsonWriter writer = state.utf8Writer;
+ Utf8JsonWriter writer = state.utf8Writer();
try {
state.typeResolver.lockJIT();
try {
@@ -408,7 +413,8 @@ public final class ForyJson {
return writer.toJsonBytes();
} finally {
try {
- writer.reset();
+ state.buffer = writer.getBuffer();
+ writer.clear();
} finally {
release(entry);
}
@@ -418,7 +424,7 @@ public final class ForyJson {
private byte[] toJsonBytesDeclared(Object value, TypeRef<?> type) {
PooledState entry = acquire();
JsonState state = entry.state;
- Utf8JsonWriter writer = state.utf8Writer;
+ Utf8JsonWriter writer = state.utf8Writer();
try {
state.typeResolver.lockJIT();
try {
@@ -429,7 +435,8 @@ public final class ForyJson {
return writer.toJsonBytes();
} finally {
try {
- writer.reset();
+ state.buffer = writer.getBuffer();
+ writer.clear();
} finally {
release(entry);
}
@@ -440,7 +447,7 @@ public final class ForyJson {
Objects.requireNonNull(output, "output");
PooledState entry = acquire();
JsonState state = entry.state;
- Utf8JsonWriter writer = state.utf8Writer;
+ Utf8JsonWriter writer = state.utf8Writer();
try {
state.typeResolver.lockJIT();
try {
@@ -451,7 +458,8 @@ public final class ForyJson {
writer.writeTo(output);
} finally {
try {
- writer.reset();
+ state.buffer = writer.getBuffer();
+ writer.clear();
} finally {
release(entry);
}
@@ -462,7 +470,7 @@ public final class ForyJson {
Objects.requireNonNull(output, "output");
PooledState entry = acquire();
JsonState state = entry.state;
- Utf8JsonWriter writer = state.utf8Writer;
+ Utf8JsonWriter writer = state.utf8Writer();
try {
state.typeResolver.lockJIT();
try {
@@ -473,7 +481,8 @@ public final class ForyJson {
writer.writeTo(output);
} finally {
try {
- writer.reset();
+ state.buffer = writer.getBuffer();
+ writer.clear();
} finally {
release(entry);
}
@@ -779,7 +788,14 @@ public final class ForyJson {
}
private void release(PooledState entry) {
- entry.release();
+ try {
+ JsonState state = entry.state;
+ if (state.buffer.length > state.bufferSizeLimitBytes) {
+ state.buffer = new byte[state.bufferSizeLimitBytes];
+ }
+ } finally {
+ entry.release();
+ }
}
private PooledState acquireContended(int slotIndex) {
@@ -829,6 +845,8 @@ public final class ForyJson {
}
private Object readJavaStringValue(String json, Class<?> type, JsonState state) {
+ // Older JDKs read the coder through Unsafe, which can crash on a null String.
+ Objects.requireNonNull(json, "json");
if (StringSerializer.isBytesBackedString()) {
byte coder = StringSerializer.getStringCoder(json);
if (StringSerializer.isLatin1Coder(coder)) {
@@ -844,6 +862,8 @@ public final class ForyJson {
}
private Object readJavaStringValue(String json, TypeRef<?> type, JsonState state) {
+ // Validate before the Unsafe coder access, as in the Class-based root path.
+ Objects.requireNonNull(json, "json");
if (StringSerializer.isBytesBackedString()) {
byte coder = StringSerializer.getStringCoder(json);
if (StringSerializer.isLatin1Coder(coder)) {
@@ -1012,6 +1032,12 @@ public final class ForyJson {
private final Utf8JsonReader utf8Reader;
private final Latin1JsonReader latin1Reader;
private final Utf16JsonReader utf16Reader;
+ // Only the leased root's concrete reader or writer holds this array while active. Reclaim
+ // its latest array before clearing it; an idle component must not retain an old growth array.
+ // Keep the UTF16 input mirror and String writer's widening scratch separate: each can be live
+ // alongside the main workspace. No buffer holder or reader/writer backreference is needed.
+ private byte[] buffer;
+ private final int bufferSizeLimitBytes;
private byte[] charBackedUtf16Bytes;
private Class<?> lastRuntimeRootType;
private JsonTypeInfo lastRuntimeRootInfo;
@@ -1024,21 +1050,37 @@ public final class ForyJson {
private JsonState(JsonConfig config, JsonSharedRegistry sharedRegistry) {
typeResolver = new JsonTypeResolver(sharedRegistry);
- utf8Writer = new Utf8JsonWriter(config, typeResolver, new byte[INITIAL_BUFFER_SIZE]);
- stringWriter = new StringJsonWriter(config, typeResolver, new byte[INITIAL_BUFFER_SIZE]);
+ bufferSizeLimitBytes = config.bufferSizeLimitBytes();
+ buffer = new byte[Math.min(INITIAL_BUFFER_SIZE, bufferSizeLimitBytes)];
+ utf8Writer = new Utf8JsonWriter(config, typeResolver, null);
+ stringWriter = new StringJsonWriter(config, typeResolver, null);
utf8Reader = new Utf8JsonReader(config, typeResolver);
latin1Reader = new Latin1JsonReader(config, typeResolver);
utf16Reader = new Utf16JsonReader(config, typeResolver);
charBackedUtf16Bytes = EMPTY_BYTES;
}
+ private Utf8JsonWriter utf8Writer() {
+ utf8Writer.setBuffer(buffer);
+ buffer = null;
+ return utf8Writer;
+ }
+
+ private StringJsonWriter stringWriter() {
+ stringWriter.setBuffer(buffer);
+ buffer = null;
+ return stringWriter;
+ }
+
private Latin1JsonReader latin1Reader(String input) {
- latin1Reader.reset(input);
+ latin1Reader.reset(input, buffer);
+ buffer = null;
return latin1Reader;
}
private Utf16JsonReader utf16Reader(String input) {
- utf16Reader.reset(input);
+ utf16Reader.reset(input, buffer);
+ buffer = null;
return utf16Reader;
}
@@ -1060,30 +1102,46 @@ public final class ForyJson {
}
// JDK 8 char[]-backed Strings are converted once so parsing still uses UTF16 byte loads.
StringSerializer.copyStringCharsToBytes(input, bytes);
- utf16Reader.reset(input, bytes);
+ utf16Reader.reset(input, bytes, buffer);
+ buffer = null;
return utf16Reader;
}
private Utf8JsonReader utf8Reader(byte[] input) {
// Keep full-array roots on the direct reset so the existing hot path does not pay the range
// validation branches.
- utf8Reader.reset(input);
+ utf8Reader.reset(input, buffer);
+ buffer = null;
return utf8Reader;
}
private Utf8JsonReader utf8Reader(byte[] input, int offset, int length) {
- utf8Reader.reset(input, offset, length);
+ utf8Reader.reset(input, offset, length, buffer);
+ buffer = null;
return utf8Reader;
}
// Clear only readers reset by the current public parse entry; clearing the unused readers shows
// up on small byte-input parses and does not release additional retained input.
private void clearStringReaders() {
+ // Only the selected representation borrowed the workspace. Setup can fail before either
+ // reader is selected, in which case the state still owns its array.
+ byte[] latin1Bytes = latin1Reader.getStringDecodeBuffer();
+ byte[] utf16Bytes = utf16Reader.getStringDecodeBuffer();
+ if (latin1Bytes != null) {
+ buffer = latin1Bytes;
+ } else if (utf16Bytes != null) {
+ buffer = utf16Bytes;
+ }
latin1Reader.clear();
utf16Reader.clear();
}
private void clearUtf8Reader() {
+ byte[] bytes = utf8Reader.getStringDecodeBuffer();
+ if (bytes != null) {
+ buffer = bytes;
+ }
utf8Reader.clear();
}
diff --git a/java/fory-json/src/main/java/org/apache/fory/json/ForyJsonBuilder.java b/java/fory-json/src/main/java/org/apache/fory/json/ForyJsonBuilder.java
index baf99891c..48e5b6849 100644
--- a/java/fory-json/src/main/java/org/apache/fory/json/ForyJsonBuilder.java
+++ b/java/fory-json/src/main/java/org/apache/fory/json/ForyJsonBuilder.java
@@ -231,10 +231,10 @@ public final class ForyJsonBuilder {
}
/**
- * Sets the maximum byte-buffer capacity retained by each pooled String and UTF-8 writer.
+ * Sets the maximum retained capacity of each reusable output or string-decoding byte buffer.
*
- * <p>This bounds reusable writer storage after a root operation; it does not limit JSON output
- * size.
+ * <p>This bounds retained buffer capacity after a root operation; it does not limit JSON input or
+ * output size or the total memory used by this instance.
*/
public ForyJsonBuilder withBufferSizeLimitBytes(int bufferSizeLimitBytes) {
if (bufferSizeLimitBytes < 1) {
diff --git a/java/fory-json/src/main/java/org/apache/fory/json/reader/Latin1JsonReader.java b/java/fory-json/src/main/java/org/apache/fory/json/reader/Latin1JsonReader.java
index b26d420e6..ac6c5b48f 100644
--- a/java/fory-json/src/main/java/org/apache/fory/json/reader/Latin1JsonReader.java
+++ b/java/fory-json/src/main/java/org/apache/fory/json/reader/Latin1JsonReader.java
@@ -47,19 +47,11 @@ import org.apache.fory.serializer.StringSerializer;
* input or the reusable decode buffer.
*
* <p>This concrete owner implements representation-specific token probes, packed digit parsing,
- * string decoding, and field hashing. {@link #clear()} releases the input reference and bounds the
- * retained decode workspace before the owning pooled state is reused.
+ * string decoding, and field hashing. {@link #clear()} releases the input and decode workspace
+ * references after the owning pooled state has reclaimed the workspace.
*/
public final class Latin1JsonReader extends JsonReader {
private static final byte[] EMPTY_BYTES = new byte[0];
- private static final int INITIAL_STRING_DECODE_BUFFER_SIZE = 1024;
-
- /** The floor the decode buffer is shrunk back to; it is kept until it is twice what is needed. */
- private static final int RETAINED_STRING_DECODE_BUFFER_SIZE = 8192;
-
- /** The ceiling on what a single document can keep alive in a pooled reader between parses. */
- private static final int MAX_RETAINED_STRING_DECODE_BUFFER_SIZE = 1 << 17;
-
private static final boolean LITTLE_ENDIAN = NativeByteOrder.IS_LITTLE_ENDIAN;
private static final long BYTE_ONES = 0x0101010101010101L;
private static final int INT_BYTE_ONES = 0x01010101;
@@ -85,13 +77,9 @@ public final class Latin1JsonReader extends JsonReader {
// JSON syntax bytes are ASCII, so hot token checks can compare signed bytes directly.
// Latin1 string content and field-name hashing must keep unsigned byte conversion.
private byte[] input;
- private byte[] stringDecodeBuffer = new byte[INITIAL_STRING_DECODE_BUFFER_SIZE];
-
- /**
- * Bytes of the longest string decoded since the last {@link #clear()}, so a steady workload keeps
- * them.
- */
- private int stringDecodeHighWater;
+ // The caller supplies decode storage on every reset; avoid a redundant null check or allocation
+ // on pooled root setup. Decoding owns any subsequent growth.
+ private byte[] stringDecodeBuffer;
// Keep the cache after hot representation fields; an inherited reference shifts their offsets.
private final FieldNameCache fieldNameCache;
@@ -288,7 +276,6 @@ public final class Latin1JsonReader extends JsonReader {
outBytes = ensureStringDecodeCapacity(outBytes, out + 2);
out = putUtf16Char(outBytes, out, ch);
}
- recordStringDecodeUse(out);
return decodedQuotedText(outBytes, out, true);
}
@@ -375,24 +362,30 @@ public final class Latin1JsonReader extends JsonReader {
return candidate;
}
- public Latin1JsonReader(JsonConfig config, JsonTypeResolver typeResolver, byte[] input) {
+ public Latin1JsonReader(
+ JsonConfig config, JsonTypeResolver typeResolver, byte[] input, byte[] decodeBuffer) {
this(config, typeResolver);
- reset(input);
+ reset(input, decodeBuffer);
}
- public Latin1JsonReader(JsonConfig config, JsonTypeResolver typeResolver, String input) {
+ public Latin1JsonReader(
+ JsonConfig config, JsonTypeResolver typeResolver, String input, byte[] decodeBuffer) {
this(config, typeResolver);
- reset(input);
+ reset(input, decodeBuffer);
}
- public Latin1JsonReader reset(byte[] input) {
+ /** Resets the input and borrows the caller's non-null decode buffer until {@link #clear()}. */
+ public Latin1JsonReader reset(byte[] input, byte[] decodeBuffer) {
+ stringDecodeBuffer = decodeBuffer;
this.input = input;
position = 0;
reset();
return this;
}
- public Latin1JsonReader reset(String input) {
+ /** Resets the input and borrows the caller's non-null decode buffer until {@link #clear()}. */
+ public Latin1JsonReader reset(String input, byte[] decodeBuffer) {
+ stringDecodeBuffer = decodeBuffer;
if (!StringSerializer.isBytesBackedString()) {
throw new IllegalStateException("Latin1JsonReader requires byte-backed strings");
}
@@ -410,27 +403,13 @@ public final class Latin1JsonReader extends JsonReader {
reset();
input = EMPTY_BYTES;
position = 0;
- // Keep what the last document needed, so consecutive large documents stop re-growing the
- // buffer, but never pin more than the ceiling in a pooled reader.
- int keep =
- Math.max(
- RETAINED_STRING_DECODE_BUFFER_SIZE,
- Math.min(stringDecodeHighWater, MAX_RETAINED_STRING_DECODE_BUFFER_SIZE));
- if (stringDecodeBuffer.length > MAX_RETAINED_STRING_DECODE_BUFFER_SIZE
- || stringDecodeBuffer.length - keep >= keep) {
- stringDecodeBuffer = new byte[keep];
- }
- stringDecodeHighWater = 0;
+ stringDecodeBuffer = null;
}
- /**
- * Both decode paths end here, so a buffer grown for quoted text is kept the same way a String's
- * is.
- */
- private void recordStringDecodeUse(int length) {
- if (length > stringDecodeHighWater) {
- stringDecodeHighWater = length;
- }
+ /** Returns the current storage, including any growth, before {@link #clear()} detaches it. */
+ @Internal
+ public byte[] getStringDecodeBuffer() {
+ return stringDecodeBuffer;
}
@Override
@@ -2885,7 +2864,6 @@ public final class Latin1JsonReader extends JsonReader {
}
private String finishDecodedString(byte[] bytes, int length, boolean utf16) {
- recordStringDecodeUse(length);
// The decode buffer is reader-owned reusable storage; returned Strings must own exact bytes.
byte[] result = new byte[length];
System.arraycopy(bytes, 0, result, 0, length);
diff --git a/java/fory-json/src/main/java/org/apache/fory/json/reader/Utf16JsonReader.java b/java/fory-json/src/main/java/org/apache/fory/json/reader/Utf16JsonReader.java
index d88282629..a7b135b13 100644
--- a/java/fory-json/src/main/java/org/apache/fory/json/reader/Utf16JsonReader.java
+++ b/java/fory-json/src/main/java/org/apache/fory/json/reader/Utf16JsonReader.java
@@ -24,6 +24,7 @@ import java.time.LocalDate;
import java.time.OffsetDateTime;
import java.time.ZoneOffset;
import java.util.Arrays;
+import org.apache.fory.annotation.Internal;
import org.apache.fory.json.JsonConfig;
import org.apache.fory.json.meta.JsonFieldInfo;
import org.apache.fory.json.meta.JsonFieldNameHash;
@@ -45,18 +46,10 @@ import org.apache.fory.serializer.StringSerializer;
* available when no byte mirror exists. Returned values never retain the reusable decode buffer.
*
* <p>This concrete owner implements UTF16 token probes, packed digit parsing, string decoding, and
- * field hashing. {@link #clear()} releases both borrowed representations and bounds the retained
- * decode workspace before the pooled state is reused.
+ * field hashing. {@link #clear()} releases both borrowed representations and the decode workspace
+ * after the owning pooled state has reclaimed the workspace.
*/
public final class Utf16JsonReader extends JsonReader {
- private static final int INITIAL_STRING_DECODE_BUFFER_SIZE = 1024;
-
- /** The floor the decode buffer is shrunk back to; it is kept until it is twice what is needed. */
- private static final int RETAINED_STRING_DECODE_BUFFER_SIZE = 8192;
-
- /** The ceiling on what a single document can keep alive in a pooled reader between parses. */
- private static final int MAX_RETAINED_STRING_DECODE_BUFFER_SIZE = 1 << 17;
-
private static final boolean LITTLE_ENDIAN = NativeByteOrder.IS_LITTLE_ENDIAN;
private static final long BYTE_ONES = 0x0101010101010101L;
private static final long BYTE_HIGH_BITS = 0x8080808080808080L;
@@ -81,13 +74,9 @@ public final class Utf16JsonReader extends JsonReader {
private String input;
private byte[] bytes;
private int length;
- private byte[] stringDecodeBuffer = new byte[INITIAL_STRING_DECODE_BUFFER_SIZE];
-
- /**
- * Bytes of the longest string decoded since the last {@link #clear()}, so a steady workload keeps
- * them.
- */
- private int stringDecodeHighWater;
+ // The caller supplies decode storage on every reset; avoid a redundant null check or allocation
+ // on pooled root setup. Decoding owns any subsequent growth.
+ private byte[] stringDecodeBuffer;
// Keep the cache after hot representation fields; an inherited reference shifts their offsets.
private final FieldNameCache fieldNameCache;
@@ -284,7 +273,6 @@ public final class Utf16JsonReader extends JsonReader {
outBytes = ensureStringDecodeCapacity(outBytes, out + 2);
out = putUtf16Char(outBytes, out, ch);
}
- recordStringDecodeUse(out);
return decodedQuotedText(outBytes, out, true);
}
@@ -378,12 +366,15 @@ public final class Utf16JsonReader extends JsonReader {
return candidate;
}
- public Utf16JsonReader(JsonConfig config, JsonTypeResolver typeResolver, String input) {
+ public Utf16JsonReader(
+ JsonConfig config, JsonTypeResolver typeResolver, String input, byte[] decodeBuffer) {
this(config, typeResolver);
- reset(input);
+ reset(input, decodeBuffer);
}
- public Utf16JsonReader reset(String input) {
+ /** Resets the input and borrows the caller's non-null decode buffer until {@link #clear()}. */
+ public Utf16JsonReader reset(String input, byte[] decodeBuffer) {
+ stringDecodeBuffer = decodeBuffer;
this.input = input;
if (StringSerializer.isBytesBackedString()) {
byte coder = StringSerializer.getStringCoder(input);
@@ -403,13 +394,15 @@ public final class Utf16JsonReader extends JsonReader {
}
/**
- * Resets this reader with a String and its exact native-layout UTF16 byte mirror.
+ * Resets this reader with a String, its exact native-layout UTF16 byte mirror, and the caller's
+ * non-null decode buffer.
*
* <p>The caller owns creating the mirror. Its first {@code input.length() * 2} bytes must encode
* the same UTF16 code units in the layout used by {@link StringSerializer}; this method validates
* capacity but does not compare the mirror on the hot setup path.
*/
- public Utf16JsonReader reset(String input, byte[] bytes) {
+ public Utf16JsonReader reset(String input, byte[] bytes, byte[] decodeBuffer) {
+ stringDecodeBuffer = decodeBuffer;
int length = input.length();
if (length > (Integer.MAX_VALUE >>> 1)) {
throw new IllegalArgumentException("String is too large");
@@ -431,27 +424,13 @@ public final class Utf16JsonReader extends JsonReader {
bytes = null;
length = 0;
position = 0;
- // Keep what the last document needed, so consecutive large documents stop re-growing the
- // buffer, but never pin more than the ceiling in a pooled reader.
- int keep =
- Math.max(
- RETAINED_STRING_DECODE_BUFFER_SIZE,
- Math.min(stringDecodeHighWater, MAX_RETAINED_STRING_DECODE_BUFFER_SIZE));
- if (stringDecodeBuffer.length > MAX_RETAINED_STRING_DECODE_BUFFER_SIZE
- || stringDecodeBuffer.length - keep >= keep) {
- stringDecodeBuffer = new byte[keep];
- }
- stringDecodeHighWater = 0;
+ stringDecodeBuffer = null;
}
- /**
- * Both decode paths end here, so a buffer grown for quoted text is kept the same way a String's
- * is.
- */
- private void recordStringDecodeUse(int length) {
- if (length > stringDecodeHighWater) {
- stringDecodeHighWater = length;
- }
+ /** Returns the current storage, including any growth, before {@link #clear()} detaches it. */
+ @Internal
+ public byte[] getStringDecodeBuffer() {
+ return stringDecodeBuffer;
}
@Override
@@ -2265,7 +2244,6 @@ public final class Utf16JsonReader extends JsonReader {
}
private String finishDecodedString(byte[] outBytes, int length, boolean utf16) {
- recordStringDecodeUse(length);
// The decode buffer is reader-owned reusable storage; returned Strings must own exact bytes.
byte[] result = new byte[length];
System.arraycopy(outBytes, 0, result, 0, length);
diff --git a/java/fory-json/src/main/java/org/apache/fory/json/reader/Utf8JsonReader.java b/java/fory-json/src/main/java/org/apache/fory/json/reader/Utf8JsonReader.java
index 2582c96b3..1545751d4 100644
--- a/java/fory-json/src/main/java/org/apache/fory/json/reader/Utf8JsonReader.java
+++ b/java/fory-json/src/main/java/org/apache/fory/json/reader/Utf8JsonReader.java
@@ -65,8 +65,8 @@ import org.apache.fory.serializer.StringSerializer;
* and never retain the input or reusable decode buffer.
*
* <p>This concrete owner implements UTF-8 token probes, packed digit parsing, string decoding, and
- * field hashing. {@link #clear()} releases the input and bounds the retained decode workspace
- * before the owning pooled state is reused.
+ * field hashing. {@link #clear()} releases the input and decode workspace references after the
+ * owning pooled state has reclaimed the workspace.
*/
public final class Utf8JsonReader extends JsonReader {
private static final byte[] EMPTY_BYTES = new byte[0];
@@ -79,14 +79,6 @@ public final class Utf8JsonReader extends JsonReader {
private static final int[] NANO_SCALE = {
1, 10, 100, 1000, 10000, 100000, 1000000, 10000000, 100000000, 1000000000
};
- private static final int INITIAL_STRING_DECODE_BUFFER_SIZE = 1024;
-
- /** The floor the decode buffer is shrunk back to; it is kept until it is twice what is needed. */
- private static final int RETAINED_STRING_DECODE_BUFFER_SIZE = 8192;
-
- /** The ceiling on what a single document can keep alive in a pooled reader between parses. */
- private static final int MAX_RETAINED_STRING_DECODE_BUFFER_SIZE = 1 << 17;
-
private static final boolean LITTLE_ENDIAN = NativeByteOrder.IS_LITTLE_ENDIAN;
private static final long BYTE_ONES = 0x0101010101010101L;
private static final int INT_BYTE_ONES = 0x01010101;
@@ -131,13 +123,9 @@ public final class Utf8JsonReader extends JsonReader {
// UTF-8 string decoding must keep unsigned byte conversion for non-ASCII content.
private byte[] input;
private int inputLimit;
- private byte[] stringDecodeBuffer = new byte[INITIAL_STRING_DECODE_BUFFER_SIZE];
-
- /**
- * Bytes of the longest string decoded since the last {@link #clear()}, so a steady workload keeps
- * them.
- */
- private int stringDecodeHighWater;
+ // The caller supplies decode storage on every reset; avoid a redundant null check or allocation
+ // on pooled root setup. Decoding owns any subsequent growth.
+ private byte[] stringDecodeBuffer;
// Keep the cache after hot representation fields; an inherited reference shifts their offsets.
private final FieldNameCache fieldNameCache;
@@ -392,7 +380,6 @@ public final class Utf8JsonReader extends JsonReader {
out = putUtf16Char(outBytes, out, Character.lowSurrogate(codePoint));
}
}
- recordStringDecodeUse(out);
return decodedQuotedText(outBytes, out, true);
}
@@ -517,12 +504,15 @@ public final class Utf8JsonReader extends JsonReader {
return candidate;
}
- public Utf8JsonReader(JsonConfig config, JsonTypeResolver typeResolver, byte[] input) {
+ public Utf8JsonReader(
+ JsonConfig config, JsonTypeResolver typeResolver, byte[] input, byte[] decodeBuffer) {
this(config, typeResolver);
- reset(input);
+ reset(input, decodeBuffer);
}
- public Utf8JsonReader reset(byte[] input) {
+ /** Resets the input and borrows the caller's non-null decode buffer until {@link #clear()}. */
+ public Utf8JsonReader reset(byte[] input, byte[] decodeBuffer) {
+ stringDecodeBuffer = decodeBuffer;
this.input = input;
inputLimit = input.length;
position = 0;
@@ -530,9 +520,10 @@ public final class Utf8JsonReader extends JsonReader {
return this;
}
- /** Resets this reader to a logical range of a borrowed byte array. */
+ /** Resets to a logical input range and borrows the caller's non-null decode buffer. */
@Internal
- public Utf8JsonReader reset(byte[] input, int offset, int length) {
+ public Utf8JsonReader reset(byte[] input, int offset, int length, byte[] decodeBuffer) {
+ stringDecodeBuffer = decodeBuffer;
int inputLength = input.length;
if ((offset | length) < 0 || offset > inputLength - length) {
throwInvalidByteRange(offset, length);
@@ -554,27 +545,13 @@ public final class Utf8JsonReader extends JsonReader {
input = EMPTY_BYTES;
inputLimit = 0;
position = 0;
- // Keep what the last document needed, so consecutive large documents stop re-growing the
- // buffer, but never pin more than the ceiling in a pooled reader.
- int keep =
- Math.max(
- RETAINED_STRING_DECODE_BUFFER_SIZE,
- Math.min(stringDecodeHighWater, MAX_RETAINED_STRING_DECODE_BUFFER_SIZE));
- if (stringDecodeBuffer.length > MAX_RETAINED_STRING_DECODE_BUFFER_SIZE
- || stringDecodeBuffer.length - keep >= keep) {
- stringDecodeBuffer = new byte[keep];
- }
- stringDecodeHighWater = 0;
+ stringDecodeBuffer = null;
}
- /**
- * Both decode paths end here, so a buffer grown for quoted text is kept the same way a String's
- * is.
- */
- private void recordStringDecodeUse(int length) {
- if (length > stringDecodeHighWater) {
- stringDecodeHighWater = length;
- }
+ /** Returns the current storage, including any growth, before {@link #clear()} detaches it. */
+ @Internal
+ public byte[] getStringDecodeBuffer() {
+ return stringDecodeBuffer;
}
@Override
@@ -4738,7 +4715,6 @@ public final class Utf8JsonReader extends JsonReader {
}
private String finishDecodedString(byte[] bytes, int length, boolean utf16) {
- recordStringDecodeUse(length);
// Strings must not share the reader-owned decode buffer; the buffer is reused by later reads.
byte[] result = new byte[length];
System.arraycopy(bytes, 0, result, 0, length);
diff --git a/java/fory-json/src/main/java/org/apache/fory/json/writer/StringJsonWriter.java b/java/fory-json/src/main/java/org/apache/fory/json/writer/StringJsonWriter.java
index 02062d2a8..0a98764d0 100644
--- a/java/fory-json/src/main/java/org/apache/fory/json/writer/StringJsonWriter.java
+++ b/java/fory-json/src/main/java/org/apache/fory/json/writer/StringJsonWriter.java
@@ -36,6 +36,7 @@ import java.time.temporal.TemporalAccessor;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
+import org.apache.fory.annotation.Internal;
import org.apache.fory.json.ForyJsonException;
import org.apache.fory.json.JsonConfig;
import org.apache.fory.json.meta.JsonFieldInfo;
@@ -131,8 +132,8 @@ public final class StringJsonWriter extends JsonWriter implements Appendable {
public StringJsonWriter(JsonConfig config, JsonTypeResolver typeResolver, byte[] buffer) {
super(config, typeResolver);
- this.buffer = initialBuffer(buffer);
- scratch = new byte[this.buffer.length];
+ this.buffer = buffer == null ? null : initialBuffer(buffer);
+ scratch = new byte[buffer == null ? 0 : this.buffer.length];
bufferSizeLimitBytes = config.bufferSizeLimitBytes();
decimalBuilder = newDecimalBuilder();
}
@@ -140,7 +141,7 @@ public final class StringJsonWriter extends JsonWriter implements Appendable {
@Override
public void reset() {
super.reset();
- if (buffer.length > bufferSizeLimitBytes) {
+ if (buffer != null && buffer.length > bufferSizeLimitBytes) {
buffer = new byte[bufferSizeLimitBytes];
}
if (scratch.length > bufferSizeLimitBytes) {
@@ -151,6 +152,25 @@ public final class StringJsonWriter extends JsonWriter implements Appendable {
position = 0;
}
+ /** Borrows the owning execution state's output storage for one root operation. */
+ @Internal
+ public void setBuffer(byte[] buffer) {
+ this.buffer = buffer;
+ }
+
+ /** Returns the current output storage after growth or a UTF16 widening swap. */
+ @Internal
+ public byte[] getBuffer() {
+ return buffer;
+ }
+
+ /** Resets operation state and detaches storage after the execution state has reclaimed it. */
+ @Internal
+ public void clear() {
+ buffer = null;
+ reset();
+ }
+
public String toJson() {
// The returned String may zero-copy this exact array, so pooled writer storage is never
// exposed.
diff --git a/java/fory-json/src/main/java/org/apache/fory/json/writer/Utf8JsonWriter.java b/java/fory-json/src/main/java/org/apache/fory/json/writer/Utf8JsonWriter.java
index 0686f1abf..25425f19f 100644
--- a/java/fory-json/src/main/java/org/apache/fory/json/writer/Utf8JsonWriter.java
+++ b/java/fory-json/src/main/java/org/apache/fory/json/writer/Utf8JsonWriter.java
@@ -163,7 +163,7 @@ public final class Utf8JsonWriter extends JsonWriter implements Appendable {
@Override
public void reset() {
super.reset();
- if (buffer.length > bufferSizeLimitBytes) {
+ if (buffer != null && buffer.length > bufferSizeLimitBytes) {
buffer = new byte[bufferSizeLimitBytes];
}
position = 0;
@@ -174,6 +174,19 @@ public final class Utf8JsonWriter extends JsonWriter implements Appendable {
return buffer;
}
+ /** Borrows the owning execution state's output storage for one root operation. */
+ @Internal
+ public void setBuffer(byte[] buffer) {
+ this.buffer = buffer;
+ }
+
+ /** Resets operation state and detaches storage after the execution state has reclaimed it. */
+ @Internal
+ public void clear() {
+ buffer = null;
+ reset();
+ }
+
@Internal
public int getPosition() {
return position;
diff --git a/java/fory-json/src/test/java/org/apache/fory/json/ForyJsonByteRangeTest.java b/java/fory-json/src/test/java/org/apache/fory/json/ForyJsonByteRangeTest.java
index d66c7f205..921da397d 100644
--- a/java/fory-json/src/test/java/org/apache/fory/json/ForyJsonByteRangeTest.java
+++ b/java/fory-json/src/test/java/org/apache/fory/json/ForyJsonByteRangeTest.java
@@ -91,20 +91,20 @@ public class ForyJsonByteRangeTest {
}
bytes[offset + length] = '{';
bytes[offset + length + 1] = '}';
- reader.reset(bytes, offset, length + 2);
+ reader.reset(bytes, offset, length + 2, reader.getStringDecodeBuffer());
assertEquals(reader.consumeToken('{'), true);
assertEquals(reader.position(), offset + length + 1);
reader.expectNextToken('}');
reader.finish();
- reader.reset(bytes, offset, length);
+ reader.reset(bytes, offset, length, reader.getStringDecodeBuffer());
assertEquals(reader.consumeToken('{'), false);
assertEquals(reader.position(), offset + length);
}
}
for (int value = 0; value < 256; value++) {
byte[] bytes = {' ', (byte) value, '{'};
- reader.reset(bytes);
+ reader.reset(bytes, reader.getStringDecodeBuffer());
boolean space = value == ' ' || value == '\n' || value == '\r' || value == '\t';
assertEquals(reader.consumeToken('{'), space || value == '{');
assertEquals(reader.position(), space ? 3 : value == '{' ? 2 : 1);
diff --git a/java/fory-json/src/test/java/org/apache/fory/json/ForyJsonConcurrencyTest.java b/java/fory-json/src/test/java/org/apache/fory/json/ForyJsonConcurrencyTest.java
index e1d77a8f9..d1c421286 100644
--- a/java/fory-json/src/test/java/org/apache/fory/json/ForyJsonConcurrencyTest.java
+++ b/java/fory-json/src/test/java/org/apache/fory/json/ForyJsonConcurrencyTest.java
@@ -25,6 +25,7 @@ import static org.testng.Assert.assertThrows;
import static org.testng.Assert.assertTrue;
import static org.testng.Assert.fail;
+import java.nio.charset.StandardCharsets;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicReference;
@@ -59,7 +60,7 @@ public class ForyJsonConcurrencyTest {
new Thread(
() -> {
try {
- assertEquals(json.toJson(new BlockingValue()), "null");
+ assertEquals(json.toJson(new BlockingValue()), "[\"held\\n\"]");
} catch (Throwable t) {
firstFailure.set(t);
}
@@ -97,6 +98,43 @@ public class ForyJsonConcurrencyTest {
assertFailure(secondFailure.get());
}
+ @Test
+ public void concurrentBufferIsolation() throws Exception {
+ CountDownLatch rootEntered = new CountDownLatch(1);
+ CountDownLatch releaseRoot = new CountDownLatch(1);
+ ForyJson json =
+ ForyJson.builder()
+ .withConcurrencyLevel(2)
+ .registerCodec(BlockingValue.class, new BlockingCodec(rootEntered, releaseRoot))
+ .build();
+ AtomicReference<Throwable> failure = new AtomicReference<>();
+ Thread writer =
+ new Thread(
+ () -> {
+ try {
+ assertEquals(json.toJson(new BlockingValue()), "[\"held\\n\"]");
+ } catch (Throwable t) {
+ failure.set(t);
+ }
+ });
+ writer.start();
+ try {
+ await(rootEntered);
+ // The first state holds a partially written document while the second state's readers
+ // decode into their workspace. Neither may overwrite the paused writer's bytes.
+ assertEquals(json.fromJson("\"overwritten\\n\"", String.class), "overwritten\n");
+ assertEquals(
+ json.fromJson("\"overwritten\\n\"".getBytes(StandardCharsets.UTF_8), String.class),
+ "overwritten\n");
+ assertEquals(json.fromJson("\"\u4e2d\\n\"", String.class), "\u4e2d\n");
+ } finally {
+ releaseRoot.countDown();
+ writer.join(TimeUnit.SECONDS.toMillis(30));
+ }
+ assertTrue(!writer.isAlive(), "Writer did not finish");
+ assertFailure(failure.get());
+ }
+
private static void await(CountDownLatch latch) throws InterruptedException {
assertTrue(latch.await(30, TimeUnit.SECONDS), "Timed out waiting for test coordination");
}
@@ -135,6 +173,8 @@ public class ForyJsonConcurrencyTest {
@Override
public void write(JsonWriter writer, BlockingValue value) {
+ writer.writeArrayStart();
+ writer.writeString("held\n");
entered.countDown();
try {
assertTrue(release.await(30, TimeUnit.SECONDS), "Timed out waiting to release root codec");
@@ -142,7 +182,7 @@ public class ForyJsonConcurrencyTest {
Thread.currentThread().interrupt();
throw new AssertionError(e);
}
- writer.writeNull();
+ writer.writeArrayEnd();
}
@Override
diff --git a/java/fory-json/src/test/java/org/apache/fory/json/JsonContainerTest.java b/java/fory-json/src/test/java/org/apache/fory/json/JsonContainerTest.java
index 36ee3c88c..09497eea6 100644
--- a/java/fory-json/src/test/java/org/apache/fory/json/JsonContainerTest.java
+++ b/java/fory-json/src/test/java/org/apache/fory/json/JsonContainerTest.java
@@ -126,7 +126,7 @@ public class JsonContainerTest extends ForyJsonTestModels {
byte[] input = new byte[offset + token.length + 8];
Arrays.fill(input, (byte) '9');
System.arraycopy(token, 0, input, offset, token.length);
- reader.reset(input, offset, token.length);
+ reader.reset(input, offset, token.length, reader.getStringDecodeBuffer());
assertEquals(reader.readFieldNameInt(), value);
reader.finish();
}
@@ -187,7 +187,7 @@ public class JsonContainerTest extends ForyJsonTestModels {
byte[] input = new byte[offset + token.length + 8];
Arrays.fill(input, (byte) '9');
System.arraycopy(token, 0, input, offset, token.length);
- reader.reset(input, offset, token.length);
+ reader.reset(input, offset, token.length, reader.getStringDecodeBuffer());
assertEquals(reader.readFieldNameLong(), value);
reader.finish();
}
diff --git a/java/fory-json/src/test/java/org/apache/fory/json/JsonFieldNameCacheTest.java b/java/fory-json/src/test/java/org/apache/fory/json/JsonFieldNameCacheTest.java
index 0feb7b187..7a30bc259 100644
--- a/java/fory-json/src/test/java/org/apache/fory/json/JsonFieldNameCacheTest.java
+++ b/java/fory-json/src/test/java/org/apache/fory/json/JsonFieldNameCacheTest.java
@@ -206,7 +206,7 @@ public class JsonFieldNameCacheTest {
(Latin1JsonReader)
JsonTestSupport.pooledStateField(
json, (homeIndex + 1) % JsonTestSupport.pooledStateCount(json), "latin1Reader");
- String sharedName = other.reset(latin1Bytes("\"b\"")).readFieldName();
+ String sharedName = other.reset(latin1Bytes("\"b\""), new byte[1024]).readFieldName();
CachedFieldName shared = registry.cachedFieldName(JsonFieldNameHash.hash("b"));
assertNotNull(shared);
assertSame(shared.name(), sharedName);
@@ -243,7 +243,7 @@ public class JsonFieldNameCacheTest {
(Latin1JsonReader)
JsonTestSupport.pooledStateField(
json, (homeIndex + 1) % JsonTestSupport.pooledStateCount(json), "latin1Reader");
- String sharedName = other.reset(latin1Bytes("\"q\"")).readFieldName();
+ String sharedName = other.reset(latin1Bytes("\"q\""), new byte[1024]).readFieldName();
CachedFieldName shared = registry.cachedFieldName(JsonFieldNameHash.hash("q"));
assertNotNull(shared);
assertSame(shared.name(), sharedName);
@@ -277,8 +277,8 @@ public class JsonFieldNameCacheTest {
JsonTestSupport.pooledStateField(
json, (homeIndex + 1) % JsonTestSupport.pooledStateCount(json), "latin1Reader");
- String first = current.reset(latin1Bytes("\"shared\"")).readFieldName();
- String second = other.reset(latin1Bytes("\"shared\"")).readFieldName();
+ String first = current.reset(latin1Bytes("\"shared\""), new byte[1024]).readFieldName();
+ String second = other.reset(latin1Bytes("\"shared\""), new byte[1024]).readFieldName();
assertSame(second, first);
}
diff --git a/java/fory-json/src/test/java/org/apache/fory/json/JsonScalarTest.java b/java/fory-json/src/test/java/org/apache/fory/json/JsonScalarTest.java
index a6de76422..9372c116c 100644
--- a/java/fory-json/src/test/java/org/apache/fory/json/JsonScalarTest.java
+++ b/java/fory-json/src/test/java/org/apache/fory/json/JsonScalarTest.java
@@ -119,17 +119,17 @@ public class JsonScalarTest extends ForyJsonTestModels {
Arrays.fill(bytes, (byte) 'x');
System.arraycopy(encoded, 0, bytes, offset, encoded.length);
for (int length = 0; length < encoded.length; length++) {
- reader.reset(bytes, offset, length);
+ reader.reset(bytes, offset, length, reader.getStringDecodeBuffer());
// A complete token remains in the backing array beyond the declared input slice.
assertThrows(ForyJsonException.class, () -> reader.readBooleanValue());
}
- reader.reset(bytes, offset, encoded.length);
+ reader.reset(bytes, offset, encoded.length, reader.getStringDecodeBuffer());
assertEquals(reader.readBooleanValue(), token.contains("true"));
reader.finish();
for (int index = 0; index < encoded.length; index++) {
byte saved = bytes[offset + index];
bytes[offset + index] = 'x';
- reader.reset(bytes, offset, encoded.length);
+ reader.reset(bytes, offset, encoded.length, reader.getStringDecodeBuffer());
assertThrows(
ForyJsonException.class,
() -> {
@@ -161,9 +161,9 @@ public class JsonScalarTest extends ForyJsonTestModels {
byte[] sliced = new byte[offset + bytes.length + 8];
System.arraycopy(bytes, 0, sliced, offset, bytes.length);
Utf8JsonReader reader = newUtf8Reader(sliced);
- reader.reset(sliced, offset, whitespace.length());
+ reader.reset(sliced, offset, whitespace.length(), reader.getStringDecodeBuffer());
assertThrows(RuntimeException.class, reader::peekToken);
- reader.reset(sliced, offset, bytes.length);
+ reader.reset(sliced, offset, bytes.length, reader.getStringDecodeBuffer());
assertEquals(reader.peekToken(), token.charAt(0));
assertEquals(reader.position(), offset + whitespace.length());
reader.skipValue();
@@ -547,7 +547,7 @@ public class JsonScalarTest extends ForyJsonTestModels {
byte[] input = new byte[offset + token.length + 8];
Arrays.fill(input, (byte) '9');
System.arraycopy(token, 0, input, offset, token.length);
- reader.reset(input, offset, token.length);
+ reader.reset(input, offset, token.length, reader.getStringDecodeBuffer());
assertSame(codec.readUtf8(reader), value);
reader.finish();
}
@@ -760,13 +760,16 @@ public class JsonScalarTest extends ForyJsonTestModels {
for (int value : new int[] {0, -1, 17, 123456789, Integer.MIN_VALUE, Integer.MAX_VALUE}) {
String number = Integer.toString(value);
for (String token : new String[] {number, '"' + number + '"'}) {
- reader.reset((token + ",17").getBytes(StandardCharsets.US_ASCII));
+ reader.reset(
+ (token + ",17").getBytes(StandardCharsets.US_ASCII), reader.getStringDecodeBuffer());
assertEquals(reader.readIntTokenValue(), value);
reader.expectNextToken(',');
assertEquals(reader.readIntTokenValue(), 17);
reader.finish();
}
- reader.reset(('"' + number + "\":17").getBytes(StandardCharsets.US_ASCII));
+ reader.reset(
+ ('"' + number + "\":17").getBytes(StandardCharsets.US_ASCII),
+ reader.getStringDecodeBuffer());
assertEquals(reader.readFieldNameInt(), value);
reader.expectNextToken(':');
assertEquals(reader.readIntTokenValue(), 17);
@@ -774,10 +777,12 @@ public class JsonScalarTest extends ForyJsonTestModels {
for (String suffix : new String[] {".0", "e0", "E+1"}) {
String invalid = number + suffix;
for (String token : new String[] {invalid, '"' + invalid + '"'}) {
- reader.reset(token.getBytes(StandardCharsets.US_ASCII));
+ reader.reset(token.getBytes(StandardCharsets.US_ASCII), reader.getStringDecodeBuffer());
assertThrows(RuntimeException.class, reader::readIntTokenValue);
}
- reader.reset(('"' + invalid + '"').getBytes(StandardCharsets.US_ASCII));
+ reader.reset(
+ ('"' + invalid + '"').getBytes(StandardCharsets.US_ASCII),
+ reader.getStringDecodeBuffer());
assertThrows(RuntimeException.class, reader::readFieldNameInt);
}
}
@@ -812,11 +817,12 @@ public class JsonScalarTest extends ForyJsonTestModels {
byte[] input = new byte[offset + bytes.length + 8];
Arrays.fill(input, (byte) '9');
System.arraycopy(bytes, 0, input, offset, bytes.length);
- reader.reset(input, offset, bytes.length);
+ reader.reset(input, offset, bytes.length, reader.getStringDecodeBuffer());
assertEquals(reader.readIntTokenValue(), expected);
reader.finish();
}
- reader.reset((token + ",17").getBytes(StandardCharsets.US_ASCII));
+ reader.reset(
+ (token + ",17").getBytes(StandardCharsets.US_ASCII), reader.getStringDecodeBuffer());
assertEquals(reader.readIntTokenValue(), expected);
reader.expectNextToken(',');
assertEquals(reader.readIntTokenValue(), 17);
@@ -900,10 +906,10 @@ public class JsonScalarTest extends ForyJsonTestModels {
byte[] bytes = new byte[offset + encoded.length + 8];
Arrays.fill(bytes, (byte) '9');
System.arraycopy(encoded, 0, bytes, offset, encoded.length);
- reader.reset(bytes, offset, token.length());
+ reader.reset(bytes, offset, token.length(), reader.getStringDecodeBuffer());
assertEquals(reader.readLongValue(), expected);
reader.finish();
- reader.reset(bytes, offset, encoded.length);
+ reader.reset(bytes, offset, encoded.length, reader.getStringDecodeBuffer());
assertEquals(reader.readLongValue(), expected);
reader.expectNextToken(',');
assertEquals(reader.readInt(), 17);
@@ -1465,12 +1471,12 @@ public class JsonScalarTest extends ForyJsonTestModels {
byte[] bytes = new byte[offset + value.length + 8];
Arrays.fill(bytes, (byte) '9');
System.arraycopy(value, 0, bytes, offset, value.length);
- reader.reset(bytes, offset, value.length);
+ reader.reset(bytes, offset, value.length, reader.getStringDecodeBuffer());
assertEquals(reader.readBigDecimal(), expected);
reader.finish();
}
byte[] sequence = (token + ",123456789").getBytes(StandardCharsets.UTF_8);
- reader.reset(sequence);
+ reader.reset(sequence, reader.getStringDecodeBuffer());
assertEquals(reader.readBigDecimal(), expected);
reader.expectNextToken(',');
assertEquals(reader.readIntValue(), 123456789);
@@ -1506,11 +1512,13 @@ public class JsonScalarTest extends ForyJsonTestModels {
byte[] bytes = new byte[offset + value.length + 8];
Arrays.fill(bytes, (byte) '9');
System.arraycopy(value, 0, bytes, offset, value.length);
- reader.reset(bytes, offset, value.length);
+ reader.reset(bytes, offset, value.length, reader.getStringDecodeBuffer());
assertEquals(reader.readBigDecimal(), expected);
reader.finish();
}
- reader.reset((token + ",123456789").getBytes(StandardCharsets.UTF_8));
+ reader.reset(
+ (token + ",123456789").getBytes(StandardCharsets.UTF_8),
+ reader.getStringDecodeBuffer());
assertEquals(reader.readBigDecimal(), expected);
reader.expectNextToken(',');
assertEquals(reader.readIntValue(), 123456789);
@@ -1549,11 +1557,13 @@ public class JsonScalarTest extends ForyJsonTestModels {
byte[] bytes = new byte[offset + value.length + 16];
Arrays.fill(bytes, (byte) '9');
System.arraycopy(value, 0, bytes, offset, value.length);
- reader.reset(bytes, offset, value.length);
+ reader.reset(bytes, offset, value.length, reader.getStringDecodeBuffer());
assertEquals(reader.readBigDecimal(), expected);
reader.finish();
}
- reader.reset((token + ",123456789").getBytes(StandardCharsets.UTF_8));
+ reader.reset(
+ (token + ",123456789").getBytes(StandardCharsets.UTF_8),
+ reader.getStringDecodeBuffer());
assertEquals(reader.readBigDecimal(), expected);
reader.expectNextToken(',');
assertEquals(reader.readIntValue(), 123456789);
@@ -1639,12 +1649,12 @@ public class JsonScalarTest extends ForyJsonTestModels {
BigInteger value =
boundary.add(BigInteger.valueOf(delta)).multiply(BigInteger.valueOf(sign));
byte[] input = value.toString().getBytes(StandardCharsets.US_ASCII);
- reader.reset(input, 0, input.length);
+ reader.reset(input, 0, input.length, reader.getStringDecodeBuffer());
expected.add(value);
actual.add(reader.readBigInteger());
BigDecimal decimal = new BigDecimal(value, 7);
input = decimal.toPlainString().getBytes(StandardCharsets.US_ASCII);
- reader.reset(input, 0, input.length);
+ reader.reset(input, 0, input.length, reader.getStringDecodeBuffer());
expected.add(value);
actual.add(reader.readBigDecimal().unscaledValue());
}
@@ -1674,7 +1684,7 @@ public class JsonScalarTest extends ForyJsonTestModels {
Arrays.fill(input, (byte) '9');
System.arraycopy(token, 0, input, offset, token.length);
Utf8JsonReader reader = newUtf8Reader(input);
- reader.reset(input, offset, token.length);
+ reader.reset(input, offset, token.length, reader.getStringDecodeBuffer());
assertEquals(reader.readNumberAsString(), number);
}
}
@@ -1696,7 +1706,7 @@ public class JsonScalarTest extends ForyJsonTestModels {
byte[] token = (number + ",17").getBytes(StandardCharsets.US_ASCII);
byte[] bytes = new byte[offset + token.length];
System.arraycopy(token, 0, bytes, offset, token.length);
- reader.reset(bytes, offset, token.length);
+ reader.reset(bytes, offset, token.length, reader.getStringDecodeBuffer());
assertEquals(reader.readNumberAsString(), number);
reader.expect(',');
assertEquals(reader.readInt(), 17);
@@ -1708,7 +1718,7 @@ public class JsonScalarTest extends ForyJsonTestModels {
byte[] bytes = "1234567812345678,17".getBytes(StandardCharsets.US_ASCII);
bytes[8 + lane] = (byte) ch;
Latin1JsonReader reference = newLatin1Reader(bytes);
- reader.reset(bytes);
+ reader.reset(bytes, reader.getStringDecodeBuffer());
String expected;
try {
expected = reference.readNumberAsString();
@@ -1757,7 +1767,7 @@ public class JsonScalarTest extends ForyJsonTestModels {
byte[] token = (value + ",17").getBytes(StandardCharsets.US_ASCII);
byte[] bytes = new byte[offset + token.length + 8];
System.arraycopy(token, 0, bytes, offset, token.length);
- reader.reset(bytes, offset, token.length);
+ reader.reset(bytes, offset, token.length, reader.getStringDecodeBuffer());
Number actual = reader.readNumber();
assertEquals(actual.getClass(), expected.getClass());
assertEquals(actual, expected);
@@ -1775,7 +1785,7 @@ public class JsonScalarTest extends ForyJsonTestModels {
new String[] {"-9223372036854775808", "92233720368547758080", "1.25e-100", "-0.0"}) {
byte[] bytes = value.getBytes(StandardCharsets.US_ASCII);
for (int length = 0; length <= bytes.length; length++) {
- reader.reset(bytes, 0, length);
+ reader.reset(bytes, 0, length, reader.getStringDecodeBuffer());
Latin1JsonReader reference = newLatin1Reader(Arrays.copyOf(bytes, length));
Number expected;
try {
@@ -1807,7 +1817,7 @@ public class JsonScalarTest extends ForyJsonTestModels {
byte[] bytes = "1234567812345678123,17".getBytes(StandardCharsets.US_ASCII);
bytes[8 + lane] = (byte) ch;
Latin1JsonReader reference = newLatin1Reader(bytes);
- reader.reset(bytes);
+ reader.reset(bytes, reader.getStringDecodeBuffer());
Number expected;
try {
expected = reference.readNumber();
@@ -1836,11 +1846,11 @@ public class JsonScalarTest extends ForyJsonTestModels {
byte[] input = new byte[offset + token.length + 16];
Arrays.fill(input, (byte) '9');
System.arraycopy(token, 0, input, offset, token.length);
- reader.reset(input, offset, token.length);
+ reader.reset(input, offset, token.length, reader.getStringDecodeBuffer());
assertEquals(reader.readBigInteger(), expected);
byte[] suffix = ",17 ".getBytes(StandardCharsets.US_ASCII);
System.arraycopy(suffix, 0, input, offset + token.length, suffix.length);
- reader.reset(input, offset, token.length + suffix.length);
+ reader.reset(input, offset, token.length + suffix.length, reader.getStringDecodeBuffer());
assertEquals(reader.readBigInteger(), expected);
reader.expectNextToken(',');
assertEquals(reader.readInt(), 17);
@@ -3170,7 +3180,7 @@ public class JsonScalarTest extends ForyJsonTestModels {
byte[] bytes = new byte[offset + token.length() + 3];
byte[] value = (token + ",17").getBytes(StandardCharsets.US_ASCII);
System.arraycopy(value, 0, bytes, offset, value.length);
- reader.reset(bytes, offset, value.length);
+ reader.reset(bytes, offset, value.length, reader.getStringDecodeBuffer());
assertEquals(Float.floatToRawIntBits(reader.readFloatTokenValue()), expected);
reader.expect(',');
assertEquals(reader.readInt(), 17);
@@ -3207,13 +3217,13 @@ public class JsonScalarTest extends ForyJsonTestModels {
byte[] input = (token + ",17 ").getBytes(StandardCharsets.US_ASCII);
byte[] bytes = new byte[offset + input.length];
System.arraycopy(input, 0, bytes, offset, input.length);
- reader.reset(bytes, offset, input.length);
+ reader.reset(bytes, offset, input.length, reader.getStringDecodeBuffer());
assertEquals(
Double.doubleToRawLongBits(reader.readDoubleTokenValue()), expected, token);
reader.expectNextToken(',');
assertEquals(reader.readInt(), 17);
Arrays.fill(bytes, offset + token.length(), bytes.length, (byte) '9');
- reader.reset(bytes, offset, token.length());
+ reader.reset(bytes, offset, token.length(), reader.getStringDecodeBuffer());
assertEquals(
Double.doubleToRawLongBits(reader.readDoubleTokenValue()), expected, token);
reader.finish();
@@ -3253,13 +3263,13 @@ public class JsonScalarTest extends ForyJsonTestModels {
byte[] input = (token + ",17 ").getBytes(StandardCharsets.US_ASCII);
byte[] bytes = new byte[offset + input.length];
System.arraycopy(input, 0, bytes, offset, input.length);
- reader.reset(bytes, offset, input.length);
+ reader.reset(bytes, offset, input.length, reader.getStringDecodeBuffer());
assertEquals(
Float.floatToRawIntBits(reader.readFloatTokenValue()), expected, token);
reader.expectNextToken(',');
assertEquals(reader.readInt(), 17);
Arrays.fill(bytes, offset + token.length(), bytes.length, (byte) '9');
- reader.reset(bytes, offset, token.length());
+ reader.reset(bytes, offset, token.length(), reader.getStringDecodeBuffer());
assertEquals(
Float.floatToRawIntBits(reader.readFloatTokenValue()), expected, token);
reader.finish();
@@ -4386,7 +4396,7 @@ public class JsonScalarTest extends ForyJsonTestModels {
private static Utf16JsonReader utf16Reader(String input) {
byte[] bytes = new byte[input.length() << 1];
StringSerializer.copyStringCharsToBytes(input, bytes);
- return newUtf16Reader().reset(input, bytes);
+ return newUtf16Reader().reset(input, bytes, new byte[1024]);
}
private static void assertGeneratedFloatingFields(GeneratedFloatingFields value) {
diff --git a/java/fory-json/src/test/java/org/apache/fory/json/JsonStringTest.java b/java/fory-json/src/test/java/org/apache/fory/json/JsonStringTest.java
index aab34999c..e47834360 100644
--- a/java/fory-json/src/test/java/org/apache/fory/json/JsonStringTest.java
+++ b/java/fory-json/src/test/java/org/apache/fory/json/JsonStringTest.java
@@ -26,12 +26,18 @@ import static org.apache.fory.json.JsonTestSupport.newUtf16Reader;
import static org.apache.fory.json.JsonTestSupport.newUtf8Reader;
import static org.testng.Assert.assertEquals;
import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertNull;
+import static org.testng.Assert.assertSame;
import static org.testng.Assert.assertThrows;
import static org.testng.Assert.assertTrue;
+import java.io.IOException;
+import java.io.OutputStream;
import java.lang.reflect.Field;
import java.nio.charset.StandardCharsets;
import java.util.Arrays;
+import org.apache.fory.json.codec.AbstractJsonValueCodec;
+import org.apache.fory.json.codec.JsonValueCodec;
import org.apache.fory.json.data.CharValue;
import org.apache.fory.json.data.Kind;
import org.apache.fory.json.data.Nested;
@@ -42,12 +48,15 @@ import org.apache.fory.json.data.UnicodeKind;
import org.apache.fory.json.data.UnicodeMatrix;
import org.apache.fory.json.data.UnicodeValues;
import org.apache.fory.json.meta.JsonFieldNameHash;
+import org.apache.fory.json.reader.JsonReader;
import org.apache.fory.json.reader.Latin1JsonReader;
import org.apache.fory.json.reader.Utf16JsonReader;
import org.apache.fory.json.reader.Utf8JsonReader;
+import org.apache.fory.json.writer.JsonWriter;
import org.apache.fory.json.writer.StringJsonWriter;
import org.apache.fory.json.writer.Utf8JsonWriter;
import org.apache.fory.memory.NativeByteOrder;
+import org.apache.fory.reflect.TypeRef;
import org.apache.fory.serializer.StringSerializer;
import org.testng.annotations.Test;
@@ -62,7 +71,7 @@ public class JsonStringTest extends ForyJsonTestModels {
for (int offset = 0; offset < 8; offset++) {
byte[] input = new byte[offset + token.length + 8];
System.arraycopy(token, 0, input, offset, token.length);
- reader.reset(input, offset, token.length);
+ reader.reset(input, offset, token.length, reader.getStringDecodeBuffer());
if (raw >= 0x20 && raw < 0x80 && raw != '"' && raw != '\\') {
assertEquals(reader.readFieldNameHash(), JsonFieldNameHash.hash(prefix + (char) raw));
reader.expectNextToken(':');
@@ -105,14 +114,14 @@ public class JsonStringTest extends ForyJsonTestModels {
for (int offset = 0; offset < 8; offset++) {
byte[] bytes = new byte[offset + encoded.length + 8];
System.arraycopy(encoded, 0, bytes, offset, encoded.length);
- reader.reset(bytes, offset, encoded.length);
+ reader.reset(bytes, offset, encoded.length, reader.getStringDecodeBuffer());
assertEquals(reader.readFieldNameHash(), JsonFieldNameHash.hash(name.toString()));
reader.expectNextToken(':');
assertEquals(reader.readInt(), 17);
reader.finish();
if (mask == (1 << length) - 1) {
for (int end = 0; end < nameEnd; end++) {
- reader.reset(bytes, offset, end);
+ reader.reset(bytes, offset, end, reader.getStringDecodeBuffer());
assertThrows(ForyJsonException.class, reader::readFieldNameHash);
}
}
@@ -140,7 +149,7 @@ public class JsonStringTest extends ForyJsonTestModels {
for (int offset = 0; offset < 8; offset++) {
byte[] bytes = new byte[offset + encoded.length + 8];
System.arraycopy(encoded, 0, bytes, offset, encoded.length);
- reader.reset(bytes, offset, encoded.length);
+ reader.reset(bytes, offset, encoded.length, reader.getStringDecodeBuffer());
assertEquals(reader.readFieldNameHash(), JsonFieldNameHash.hash(name));
reader.expectNextToken(':');
assertEquals(reader.readNextIntValue(), 17);
@@ -163,7 +172,7 @@ public class JsonStringTest extends ForyJsonTestModels {
for (int lane = 0; lane < 4; lane++) {
token[3 + lane] = (byte) digits.charAt((value >>> (12 - lane * 4)) & 15);
}
- reader.reset(token);
+ reader.reset(token, reader.getStringDecodeBuffer());
assertEquals(reader.readString(), String.valueOf((char) value));
reader.expectNextToken(',');
assertEquals(reader.readInt(), 17);
@@ -173,7 +182,7 @@ public class JsonStringTest extends ForyJsonTestModels {
for (int value = 0; value < 256; value++) {
Arrays.fill(token, 3, 7, (byte) '0');
token[3 + lane] = (byte) value;
- reader.reset(token);
+ reader.reset(token, reader.getStringDecodeBuffer());
int digit = value < 128 ? Character.digit((char) value, 16) : -1;
if (digit < 0) {
assertThrows(ForyJsonException.class, reader::readString);
@@ -200,17 +209,17 @@ public class JsonStringTest extends ForyJsonTestModels {
byte[] bytes = new byte[offset + encoded.length + 8];
System.arraycopy(encoded, 0, bytes, offset, encoded.length);
for (int length = 0; length < encoded.length; length++) {
- reader.reset(bytes, offset, length);
+ reader.reset(bytes, offset, length, reader.getStringDecodeBuffer());
assertThrows(ForyJsonException.class, () -> reader.readString());
}
- reader.reset(bytes, offset, encoded.length);
+ reader.reset(bytes, offset, encoded.length, reader.getStringDecodeBuffer());
assertEquals(reader.readString(), (prefix.isEmpty() ? "" : "\u0100") + value);
reader.finish();
for (int digit = 0; digit < 4; digit++) {
int index = offset + prefix.length() + 3 + digit;
byte saved = bytes[index];
bytes[index] = 'x';
- reader.reset(bytes, offset, encoded.length);
+ reader.reset(bytes, offset, encoded.length, reader.getStringDecodeBuffer());
assertThrows(ForyJsonException.class, () -> reader.readString());
bytes[index] = saved;
}
@@ -233,10 +242,10 @@ public class JsonStringTest extends ForyJsonTestModels {
byte[] bytes = new byte[offset + encoded.length + 8];
System.arraycopy(encoded, 0, bytes, offset, encoded.length);
for (int length = 0; length < encoded.length - 3; length++) {
- reader.reset(bytes, offset, length);
+ reader.reset(bytes, offset, length, reader.getStringDecodeBuffer());
assertThrows(ForyJsonException.class, reader::readString);
}
- reader.reset(bytes, offset, encoded.length);
+ reader.reset(bytes, offset, encoded.length, reader.getStringDecodeBuffer());
assertEquals(reader.readString(), "\u0100" + first + second);
reader.expectNextToken(',');
assertEquals(reader.readInt(), 17);
@@ -246,7 +255,7 @@ public class JsonStringTest extends ForyJsonTestModels {
int index = offset + 9 + escape * 6 + digit;
byte saved = bytes[index];
bytes[index] = 'x';
- reader.reset(bytes, offset, encoded.length);
+ reader.reset(bytes, offset, encoded.length, reader.getStringDecodeBuffer());
assertThrows(ForyJsonException.class, reader::readString);
bytes[index] = saved;
}
@@ -261,7 +270,9 @@ public class JsonStringTest extends ForyJsonTestModels {
text.append("\\u20ac\\n\\u0000\\uD834\\uDD1E\\uabcd\\uFFFF");
expected.append("\u20ac\n\u0000\uD834\uDD1E\uabcd\uFFFF");
}
- reader.reset(text.append('"').toString().getBytes(StandardCharsets.US_ASCII));
+ reader.reset(
+ text.append('"').toString().getBytes(StandardCharsets.US_ASCII),
+ reader.getStringDecodeBuffer());
assertEquals(reader.readString(), expected.toString());
reader.finish();
}
@@ -310,7 +321,7 @@ public class JsonStringTest extends ForyJsonTestModels {
String input = "\"music \uD834\uDD1E\"";
byte[] bytes = new byte[input.length() << 1];
StringSerializer.copyStringCharsToBytes(input, bytes);
- Utf16JsonReader reader = newUtf16Reader().reset(input, bytes);
+ Utf16JsonReader reader = newUtf16Reader().reset(input, bytes, new byte[1024]);
assertEquals(reader.readString(), "music \uD834\uDD1E");
reader.finish();
}
@@ -329,13 +340,13 @@ public class JsonStringTest extends ForyJsonTestModels {
for (int offset = 0; offset < 8; offset++) {
byte[] bytes = new byte[offset + token.length + 8];
System.arraycopy(token, 0, bytes, offset, token.length);
- reader.reset(bytes, offset, token.length);
+ reader.reset(bytes, offset, token.length, reader.getStringDecodeBuffer());
assertEquals(reader.readFieldNameHash(), JsonFieldNameHash.hash(name));
reader.expectNextToken(':');
assertEquals(reader.readInt(), 17);
reader.finish();
for (int length = 0; length < token.length - 3; length++) {
- reader.reset(bytes, offset, length);
+ reader.reset(bytes, offset, length, reader.getStringDecodeBuffer());
assertThrows(RuntimeException.class, reader::readFieldNameHash);
}
}
@@ -503,17 +514,25 @@ public class JsonStringTest extends ForyJsonTestModels {
.build();
String value = repeat('a', bufferLimit + 1);
- StringJsonWriter stringWriter = (StringJsonWriter) currentStateField(json, "stringWriter");
- stringWriter.writeString(value);
- assertTrue(writerBufferLength(stringWriter) > bufferLimit);
- stringWriter.reset();
- assertEquals(writerBufferLength(stringWriter), bufferLimit);
-
- Utf8JsonWriter utf8Writer = (Utf8JsonWriter) currentStateField(json, "utf8Writer");
- utf8Writer.writeString(value);
- assertTrue(writerBufferLength(utf8Writer) > bufferLimit);
- utf8Writer.reset();
- assertEquals(writerBufferLength(utf8Writer), bufferLimit);
+ assertEquals(json.toJson(value), "\"" + value + "\"");
+ assertEquals(sharedBuffer(json).length, bufferLimit);
+ assertEquals(new String(json.toJsonBytes(value), StandardCharsets.UTF_8), "\"" + value + "\"");
+ assertEquals(sharedBuffer(json).length, bufferLimit);
+ assertEquals(json.fromJson("\"\\n" + value + "\"", String.class), "\n" + value);
+ assertEquals(sharedBuffer(json).length, bufferLimit);
+ assertBuffersDetached(json);
+
+ for (int limit : new int[] {1, 17}) {
+ ForyJson small =
+ ForyJson.builder().withConcurrencyLevel(1).withBufferSizeLimitBytes(limit).build();
+ for (int i = 0; i < 2; i++) {
+ assertEquals(small.toJson(1.25), "1.25");
+ assertEquals(small.fromJson(small.toJsonBytes("text\n"), String.class), "text\n");
+ assertEquals(small.toJson("\u4e2d"), "\"\u4e2d\"");
+ assertEquals(sharedBuffer(small).length, limit);
+ assertBuffersDetached(small);
+ }
+ }
ForyJson defaults = ForyJson.builder().build();
assertEquals(writerBufferLimit(currentStateField(defaults, "stringWriter")), 2 * 1024 * 1024);
@@ -573,74 +592,203 @@ public class JsonStringTest extends ForyJsonTestModels {
assertThrows(ForyJsonException.class, () -> json.fromJson("\"\\uD83D\"", String.class));
}
- /**
- * A buffer the recent documents needed is kept, so a stream of large strings stops re-growing it,
- * and it is released once a later document no longer needs it.
- */
@Test
- public void readerDecodeBufferIsKeptWhileNeeded() throws Exception {
- String latin1Input = "\"" + repeat('a', 9000) + "\\n\"";
- String smallInput = "\"" + repeat('a', 8) + "\\n\"";
- if (StringSerializer.isBytesBackedString()
- && StringSerializer.isLatin1Coder(StringSerializer.getStringCoder(latin1Input))) {
- Latin1JsonReader latin1Reader = newLatin1Reader(latin1Input);
- assertEquals(latin1Reader.readString(), repeat('a', 9000) + "\n");
- int grown = readerBufferLength(latin1Reader);
- assertTrue(grown > 8192);
- latin1Reader.clear();
- assertEquals(readerBufferLength(latin1Reader), grown);
- latin1Reader.reset(latin1Input);
- assertEquals(latin1Reader.readString(), repeat('a', 9000) + "\n");
- assertEquals(readerBufferLength(latin1Reader), grown);
- latin1Reader.clear();
- latin1Reader.reset(smallInput);
- assertEquals(latin1Reader.readString(), repeat('a', 8) + "\n");
- latin1Reader.clear();
- assertEquals(readerBufferLength(latin1Reader), 8192);
+ public void readerResetAfterClear() {
+ String document = "\"text\\n\"";
+ byte[] bytes = document.getBytes(StandardCharsets.UTF_8);
+ Utf8JsonReader utf8 = newUtf8Reader(bytes);
+ Latin1JsonReader latin1 = newLatin1Reader(bytes);
+ Utf16JsonReader utf16 = newUtf16Reader(document);
+ for (int i = 0; i < 2; i++) {
+ assertEquals(utf8.readString(), "text\n");
+ assertEquals(latin1.readString(), "text\n");
+ assertEquals(utf16.readString(), "text\n");
+ byte[] utf8Buffer = utf8.getStringDecodeBuffer();
+ byte[] latin1Buffer = latin1.getStringDecodeBuffer();
+ byte[] utf16Buffer = utf16.getStringDecodeBuffer();
+ utf8.clear();
+ latin1.clear();
+ utf16.clear();
+ utf8.reset(bytes, utf8Buffer);
+ latin1.reset(bytes, latin1Buffer);
+ utf16.reset(document, utf16Buffer);
+ assertSame(utf8.getStringDecodeBuffer(), utf8Buffer);
+ assertSame(latin1.getStringDecodeBuffer(), latin1Buffer);
+ assertSame(utf16.getStringDecodeBuffer(), utf16Buffer);
}
+ }
- // A buffer no document should pin: it is released whatever the shrink threshold says.
- String hugeInput = "\"\\n" + repeat('a', 200000) + "\"";
- Latin1JsonReader hugeReader = newLatin1Reader(hugeInput.getBytes(StandardCharsets.ISO_8859_1));
- assertEquals(hugeReader.readString(), "\n" + repeat('a', 200000));
- int hugeGrown = readerBufferLength(hugeReader);
- assertTrue(hugeGrown > 131072 && hugeGrown < 262144);
- hugeReader.clear();
- assertTrue(readerBufferLength(hugeReader) <= 131072);
-
- // The quoted text view shares the same buffer, so growing it there has to count the same way.
- Utf8JsonReader quotedReader = newUtf8Reader(latin1Input.getBytes(StandardCharsets.UTF_8));
- assertEquals(quotedReader.readQuotedText().toString(), repeat('a', 9000) + "\n");
- int quotedGrown = readerBufferLength(quotedReader);
- assertTrue(quotedGrown > 8192);
- quotedReader.clear();
- assertEquals(readerBufferLength(quotedReader), quotedGrown);
-
- Utf8JsonReader utf8Reader = newUtf8Reader(latin1Input.getBytes(StandardCharsets.UTF_8));
- assertEquals(utf8Reader.readString(), repeat('a', 9000) + "\n");
- int utf8Grown = readerBufferLength(utf8Reader);
- assertTrue(utf8Grown > 8192);
- utf8Reader.clear();
- assertEquals(readerBufferLength(utf8Reader), utf8Grown);
- utf8Reader.reset(smallInput.getBytes(StandardCharsets.UTF_8));
- assertEquals(utf8Reader.readString(), repeat('a', 8) + "\n");
- utf8Reader.clear();
- assertEquals(readerBufferLength(utf8Reader), 8192);
-
- String utf16Input = "\"\u4e2d\u6587" + repeat('b', 9000) + "\\n\"";
- Utf16JsonReader utf16Reader = newUtf16Reader(utf16Input);
- assertEquals(utf16Reader.readString(), "\u4e2d\u6587" + repeat('b', 9000) + "\n");
- int grown = readerBufferLength(utf16Reader);
- assertTrue(grown > 8192);
- utf16Reader.clear();
- assertEquals(readerBufferLength(utf16Reader), grown);
- String smallUtf16 = "\"\u4e2d" + repeat('b', 8) + "\\n\"";
- byte[] smallUtf16Bytes = new byte[smallUtf16.length() << 1];
- StringSerializer.copyStringCharsToBytes(smallUtf16, smallUtf16Bytes);
- utf16Reader.reset(smallUtf16, smallUtf16Bytes);
- assertEquals(utf16Reader.readString(), "\u4e2d" + repeat('b', 8) + "\n");
- utf16Reader.clear();
- assertEquals(readerBufferLength(utf16Reader), 8192);
+ @Test(dataProvider = "enableCodegen")
+ public void sharedBufferReuse(boolean codegen) {
+ BufferReuseCodec codec = new BufferReuseCodec();
+ ForyJson json =
+ ForyJson.builder()
+ .withCodegen(codegen)
+ .withAsyncCompilation(false)
+ .withConcurrencyLevel(1)
+ .registerCodec(QuotedValue.class, codec)
+ .build();
+ codec.buffer = sharedBuffer(json);
+ String value = repeat('a', 9000) + "\n";
+ String document = "\"" + repeat('a', 9000) + "\\n\"";
+ String decoded =
+ json.fromJson(document.getBytes(StandardCharsets.UTF_8), QuotedValue.class).text;
+ byte[] grown = sharedBuffer(json);
+ assertSame(grown, codec.buffer);
+ assertTrue(grown.length > 8192);
+ assertEquals(decoded, value);
+ assertBuffersDetached(json);
+
+ String encodedString = json.toJson(new QuotedValue(value));
+ assertEquals(encodedString, document);
+ assertSame(sharedBuffer(json), codec.buffer);
+ byte[] encodedBytes = json.toJsonBytes(new QuotedValue(value));
+ assertSame(sharedBuffer(json), codec.buffer);
+ assertEquals(json.fromJson(document, QuotedValue.class).text, value);
+ assertSame(sharedBuffer(json), codec.buffer);
+
+ // Widening swaps the String writer's main and scratch arrays. The state must reclaim the
+ // new main array, while later reads must not change either detached result above.
+ String unicode = "\u4e2d\u6587" + repeat('b', 9000) + "\n";
+ String unicodeDocument = json.toJson(new QuotedValue(unicode));
+ byte[] widened = sharedBuffer(json);
+ assertSame(widened, codec.buffer);
+ assertEquals(json.fromJson(unicodeDocument, QuotedValue.class).text, unicode);
+ assertSame(sharedBuffer(json), widened);
+ assertEquals(
+ json.fromJson(unicodeDocument.getBytes(StandardCharsets.UTF_8), QuotedValue.class).text,
+ unicode);
+ assertSame(sharedBuffer(json), widened);
+ json.toJsonBytes(new QuotedValue("overwritten"));
+ assertSame(sharedBuffer(json), codec.buffer);
+ assertEquals(decoded, value);
+ assertEquals(encodedString, document);
+ assertEquals(new String(encodedBytes, StandardCharsets.UTF_8), document);
+ assertBuffersDetached(json);
+ }
+
+ private static final class BufferReuseCodec implements JsonValueCodec<QuotedValue> {
+ private byte[] buffer;
+
+ // Check the borrowed array before writing and the reclaimed array afterward. Java 8's
+ // char[] path reserves more space, and UTF16 widening can swap arrays even without growth.
+ @Override
+ public void writeString(StringJsonWriter writer, QuotedValue value) {
+ assertSame(writer.getBuffer(), buffer);
+ writer.writeString(value.text);
+ buffer = writer.getBuffer();
+ }
+
+ @Override
+ public void writeUtf8(Utf8JsonWriter writer, QuotedValue value) {
+ assertSame(writer.getBuffer(), buffer);
+ writer.writeString(value.text);
+ buffer = writer.getBuffer();
+ }
+
+ @Override
+ public QuotedValue readLatin1(Latin1JsonReader reader) {
+ assertSame(reader.getStringDecodeBuffer(), buffer);
+ String value = reader.readString();
+ buffer = reader.getStringDecodeBuffer();
+ return new QuotedValue(value);
+ }
+
+ @Override
+ public QuotedValue readUtf16(Utf16JsonReader reader) {
+ assertSame(reader.getStringDecodeBuffer(), buffer);
+ String value = reader.readString();
+ buffer = reader.getStringDecodeBuffer();
+ return new QuotedValue(value);
+ }
+
+ @Override
+ public QuotedValue readUtf8(Utf8JsonReader reader) {
+ assertSame(reader.getStringDecodeBuffer(), buffer);
+ String value = reader.readString();
+ buffer = reader.getStringDecodeBuffer();
+ return new QuotedValue(value);
+ }
+ }
+
+ @Test
+ public void sharedBufferFailureCleanup() {
+ ForyJson json =
+ ForyJson.builder().withConcurrencyLevel(1).withBufferSizeLimitBytes(1024).build();
+ String malformed = "\"\\n" + repeat('a', 9000);
+ assertThrows(
+ ForyJsonException.class,
+ () -> json.fromJson(malformed.getBytes(StandardCharsets.UTF_8), String.class));
+ assertEquals(sharedBuffer(json).length, 1024);
+ assertBuffersDetached(json);
+ assertThrows(ForyJsonException.class, () -> json.fromJson(malformed, String.class));
+ assertThrows(
+ ForyJsonException.class,
+ () -> json.fromJson("\"\\n\u4e2d" + repeat('a', 9000), String.class));
+ assertThrows(ForyJsonException.class, () -> json.toJson(repeat('b', 9000) + "\uD800"));
+ assertThrows(ForyJsonException.class, () -> json.toJsonBytes(repeat('b', 9000) + "\uD800"));
+ assertThrows(
+ ForyJsonException.class,
+ () ->
+ json.writeJsonTo(
+ repeat('c', 9000),
+ new OutputStream() {
+ @Override
+ public void write(int value) throws IOException {
+ throw new IOException("Stream failure");
+ }
+ }));
+ assertThrows(NullPointerException.class, () -> json.fromJson((String) null, String.class));
+ assertThrows(
+ NullPointerException.class, () -> json.fromJson((String) null, TypeRef.of(String.class)));
+ assertThrows(NullPointerException.class, () -> json.fromJson((byte[]) null, String.class));
+ assertEquals(sharedBuffer(json).length, 1024);
+ assertEquals(json.fromJson(json.toJsonBytes("reused\n"), String.class), "reused\n");
+ assertBuffersDetached(json);
+ }
+
+ @Test
+ public void quotedTextBufferReuse() {
+ ForyJson json =
+ ForyJson.builder()
+ .withConcurrencyLevel(1)
+ .withBufferSizeLimitBytes(32768)
+ .registerCodec(
+ QuotedValue.class,
+ new AbstractJsonValueCodec<QuotedValue>() {
+ @Override
+ public void write(JsonWriter writer, QuotedValue value) {
+ writer.writeString(value.text);
+ }
+
+ @Override
+ public QuotedValue read(JsonReader reader) {
+ return new QuotedValue(reader.readQuotedText().toString());
+ }
+ })
+ .build();
+ for (String prefix : new String[] {"", "\u4e2d"}) {
+ String document = "\"" + prefix + repeat('a', 20000) + "\\n\"";
+ assertEquals(
+ json.fromJson(document, QuotedValue.class).text, prefix + repeat('a', 20000) + "\n");
+ assertEquals(sharedBuffer(json).length, 32768);
+ assertEquals(
+ json.fromJson(document.getBytes(StandardCharsets.UTF_8), QuotedValue.class).text,
+ prefix + repeat('a', 20000) + "\n");
+ byte[] retained = sharedBuffer(json);
+ assertEquals(retained.length, 32768);
+ assertEquals(json.toJson(new QuotedValue("small")), "\"small\"");
+ assertSame(sharedBuffer(json), retained);
+ assertBuffersDetached(json);
+ }
+ }
+
+ private static final class QuotedValue {
+ private final String text;
+
+ private QuotedValue(String text) {
+ this.text = text;
+ }
}
/**
@@ -911,10 +1059,8 @@ public class JsonStringTest extends ForyJsonTestModels {
() -> json.fromJson("\"" + Character.toString('\uD800') + "\"", String.class));
}
- private static int writerBufferLength(Object writer) throws Exception {
- Field field = writer.getClass().getDeclaredField("buffer");
- field.setAccessible(true);
- return ((byte[]) field.get(writer)).length;
+ private static byte[] sharedBuffer(ForyJson json) {
+ return (byte[]) currentStateField(json, "buffer");
}
private static int writerBufferLimit(Object writer) throws Exception {
@@ -923,10 +1069,13 @@ public class JsonStringTest extends ForyJsonTestModels {
return field.getInt(writer);
}
- private static int readerBufferLength(Object reader) throws Exception {
- Field field = reader.getClass().getDeclaredField("stringDecodeBuffer");
- field.setAccessible(true);
- return ((byte[]) field.get(reader)).length;
+ private static void assertBuffersDetached(ForyJson json) {
+ assertNull(((Utf8JsonWriter) currentStateField(json, "utf8Writer")).getBuffer());
+ assertNull(((StringJsonWriter) currentStateField(json, "stringWriter")).getBuffer());
+ assertNull(((Utf8JsonReader) currentStateField(json, "utf8Reader")).getStringDecodeBuffer());
+ assertNull(
+ ((Latin1JsonReader) currentStateField(json, "latin1Reader")).getStringDecodeBuffer());
+ assertNull(((Utf16JsonReader) currentStateField(json, "utf16Reader")).getStringDecodeBuffer());
}
private static String readUtf8String(ForyJson json, String input) {
@@ -948,7 +1097,7 @@ public class JsonStringTest extends ForyJsonTestModels {
private static Utf16JsonReader utf16Reader(String input) {
byte[] bytes = new byte[input.length() << 1];
StringSerializer.copyStringCharsToBytes(input, bytes);
- return newUtf16Reader().reset(input, bytes);
+ return newUtf16Reader().reset(input, bytes, new byte[1024]);
}
private static long packedNameMask(int length) {
diff --git a/java/fory-json/src/test/java/org/apache/fory/json/JsonTemporalTest.java b/java/fory-json/src/test/java/org/apache/fory/json/JsonTemporalTest.java
index 34f0810d8..eb4078b7d 100644
--- a/java/fory-json/src/test/java/org/apache/fory/json/JsonTemporalTest.java
+++ b/java/fory-json/src/test/java/org/apache/fory/json/JsonTemporalTest.java
@@ -241,7 +241,7 @@ public class JsonTemporalTest extends ForyJsonTestModels {
for (int month = 0; month <= 13; month++) {
for (int day = 0; day <= 32; day++) {
String token = String.format(Locale.ROOT, "\"--%02d-%02d\" 17", month, day);
- reader.reset(token.getBytes(StandardCharsets.US_ASCII));
+ reader.reset(token.getBytes(StandardCharsets.US_ASCII), reader.getStringDecodeBuffer());
MonthDay expected;
try {
expected = MonthDay.of(month, day);
@@ -263,13 +263,13 @@ public class JsonTemporalTest extends ForyJsonTestModels {
for (int offset = 0; offset < 8; offset++) {
byte[] bytes = new byte[offset + token.length + 8];
System.arraycopy(token, 0, bytes, offset, token.length);
- reader.reset(bytes, offset, token.length);
+ reader.reset(bytes, offset, token.length, reader.getStringDecodeBuffer());
assertEquals(reader.readMonthDay(), MonthDay.of(12, 31));
reader.expectNextToken(',');
assertEquals(reader.readInt(), 17);
reader.finish();
for (int length = 0; length < 9; length++) {
- reader.reset(bytes, offset, length);
+ reader.reset(bytes, offset, length, reader.getStringDecodeBuffer());
assertThrows(RuntimeException.class, reader::readMonthDay);
}
for (int lane : new int[] {3, 4, 6, 7}) {
@@ -279,7 +279,7 @@ public class JsonTemporalTest extends ForyJsonTestModels {
continue;
}
bytes[offset + lane] = (byte) value;
- reader.reset(bytes, offset, token.length);
+ reader.reset(bytes, offset, token.length, reader.getStringDecodeBuffer());
assertThrows(RuntimeException.class, reader::readMonthDay);
}
bytes[offset + lane] = saved;
@@ -312,7 +312,9 @@ public class JsonTemporalTest extends ForyJsonTestModels {
Utf8JsonReader reader = newUtf8Reader(new byte[0]);
for (int year = 0; year <= 9999; year++) {
YearMonth expected = YearMonth.of(year, 1 + year % 12);
- reader.reset(('"' + expected.toString() + "\" 17").getBytes(StandardCharsets.US_ASCII));
+ reader.reset(
+ ('"' + expected.toString() + "\" 17").getBytes(StandardCharsets.US_ASCII),
+ reader.getStringDecodeBuffer());
assertEquals(reader.readYearMonth(), expected);
assertEquals(reader.readInt(), 17);
reader.finish();
@@ -368,10 +370,10 @@ public class JsonTemporalTest extends ForyJsonTestModels {
System.arraycopy(token, 0, bytes, offset, token.length);
Utf8JsonReader reader = newUtf8Reader(bytes);
for (int length = 0; length < token.length; length++) {
- reader.reset(bytes, offset, length);
+ reader.reset(bytes, offset, length, reader.getStringDecodeBuffer());
assertThrows(RuntimeException.class, reader::readYearMonth);
}
- reader.reset(bytes, offset, token.length);
+ reader.reset(bytes, offset, token.length, reader.getStringDecodeBuffer());
assertEquals(reader.readYearMonth(), YearMonth.of(2024, 12));
reader.finish();
}
@@ -389,16 +391,16 @@ public class JsonTemporalTest extends ForyJsonTestModels {
byte[] bytes = new byte[offset + token.length + 8];
System.arraycopy(token, 0, bytes, offset, token.length);
for (int length = 0; length < token.length; length++) {
- reader.reset(bytes, offset, length);
+ reader.reset(bytes, offset, length, reader.getStringDecodeBuffer());
assertThrows(ForyJsonException.class, () -> reader.readYear());
}
- reader.reset(bytes, offset, token.length);
+ reader.reset(bytes, offset, token.length, reader.getStringDecodeBuffer());
assertEquals(reader.readYear(), Year.of(value));
reader.finish();
for (int digit = 1; digit <= 4; digit++) {
byte saved = bytes[offset + digit];
bytes[offset + digit] = 'x';
- reader.reset(bytes, offset, token.length);
+ reader.reset(bytes, offset, token.length, reader.getStringDecodeBuffer());
assertThrows(ForyJsonException.class, () -> reader.readYear());
bytes[offset + digit] = saved;
}
@@ -417,7 +419,7 @@ public class JsonTemporalTest extends ForyJsonTestModels {
for (int seconds = -64800; seconds <= 64800; seconds++) {
ZoneOffset expected = ZoneOffset.ofTotalSeconds(seconds);
byte[] token = ('"' + expected.getId() + '"').getBytes(StandardCharsets.UTF_8);
- reader.reset(token);
+ reader.reset(token, reader.getStringDecodeBuffer());
assertEquals(ScalarCodecs.ZoneOffsetCodec.INSTANCE.readUtf8(reader), expected);
reader.finish();
}
@@ -425,7 +427,9 @@ public class JsonTemporalTest extends ForyJsonTestModels {
ZoneOffset expected = ZoneOffset.of(text);
assertToken(ScalarCodecs.ZoneOffsetCodec.INSTANCE, text, expected);
byte[] token = ('"' + text + '"').getBytes(StandardCharsets.UTF_8);
- reader.reset((" \t\r\n\"" + text + "\",17").getBytes(StandardCharsets.US_ASCII));
+ reader.reset(
+ (" \t\r\n\"" + text + "\",17").getBytes(StandardCharsets.US_ASCII),
+ reader.getStringDecodeBuffer());
assertEquals(reader.readZoneOffset(), expected);
reader.expectNextToken(',');
assertEquals(reader.readInt(), 17);
@@ -434,10 +438,10 @@ public class JsonTemporalTest extends ForyJsonTestModels {
byte[] bytes = new byte[offset + token.length + 8];
System.arraycopy(token, 0, bytes, offset, token.length);
for (int length = 0; length < token.length; length++) {
- reader.reset(bytes, offset, length);
+ reader.reset(bytes, offset, length, reader.getStringDecodeBuffer());
assertThrows(RuntimeException.class, () -> reader.readZoneOffset());
}
- reader.reset(bytes, offset, token.length);
+ reader.reset(bytes, offset, token.length, reader.getStringDecodeBuffer());
assertEquals(reader.readZoneOffset(), expected);
reader.finish();
}
@@ -511,11 +515,11 @@ public class JsonTemporalTest extends ForyJsonTestModels {
byte[] bytes = new byte[offset + token.length + 8];
System.arraycopy(token, 0, bytes, offset, token.length);
Utf8JsonReader reader = newUtf8Reader(bytes);
- reader.reset(bytes, offset, token.length);
+ reader.reset(bytes, offset, token.length, reader.getStringDecodeBuffer());
assertEquals(reader.readOffsetTime(), expected);
reader.finish();
for (int length = 0; length < token.length; length++) {
- reader.reset(bytes, offset, length);
+ reader.reset(bytes, offset, length, reader.getStringDecodeBuffer());
assertThrows(RuntimeException.class, reader::readOffsetTime);
}
}
@@ -530,7 +534,7 @@ public class JsonTemporalTest extends ForyJsonTestModels {
for (int seconds = -64800; seconds <= 64800; seconds++) {
OffsetTime expected = OffsetTime.of(time, ZoneOffset.ofTotalSeconds(seconds));
byte[] token = ('"' + expected.toString() + "\",17").getBytes(StandardCharsets.US_ASCII);
- reader.reset(token);
+ reader.reset(token, reader.getStringDecodeBuffer());
assertEquals(reader.readOffsetTime(), expected);
reader.expectNextToken(',');
assertEquals(reader.readInt(), 17);
@@ -604,11 +608,11 @@ public class JsonTemporalTest extends ForyJsonTestModels {
byte saved = token[index];
for (int digit = 0; digit < 256; digit++) {
token[index] = (byte) digit;
- reader.reset(token);
+ reader.reset(token, reader.getStringDecodeBuffer());
// Preserve the unchanged text parser's grammar, including noncanonical offsets.
ZoneOffset expected = null;
try {
- latin1.reset(token);
+ latin1.reset(token, latin1.getStringDecodeBuffer());
expected = latin1.readZoneOffset();
latin1.finish();
} catch (RuntimeException e) {
@@ -640,7 +644,7 @@ public class JsonTemporalTest extends ForyJsonTestModels {
for (int second : new int[] {'0', '9', 0, 47, 58, 127, 128, 255}) {
token[4] = (byte) first;
token[5] = (byte) second;
- reader.reset(token);
+ reader.reset(token, reader.getStringDecodeBuffer());
if (first >= '0' && first <= '5' && second >= '0' && second <= '9') {
assertEquals(
reader.readIsoLocalTime(), LocalTime.of(1, (first - '0') * 10 + second - '0'));
@@ -671,7 +675,7 @@ public class JsonTemporalTest extends ForyJsonTestModels {
String token = '"' + timeText + '"';
// Trailing tokens provide eight readable bytes even for short fractional prefixes.
byte[] bytes = (token + ",123456789").getBytes(StandardCharsets.UTF_8);
- reader.reset(bytes);
+ reader.reset(bytes, reader.getStringDecodeBuffer());
assertEquals(reader.readIsoLocalTime(), expected);
reader.expectNextToken(',');
assertEquals(reader.readInt(), 123456789);
@@ -679,10 +683,10 @@ public class JsonTemporalTest extends ForyJsonTestModels {
for (int offset = 0; offset < 4; offset++) {
byte[] slice = new byte[offset + bytes.length];
System.arraycopy(bytes, 0, slice, offset, bytes.length);
- reader.reset(slice, offset, token.length());
+ reader.reset(slice, offset, token.length(), reader.getStringDecodeBuffer());
assertEquals(reader.readIsoLocalTime(), expected);
reader.finish();
- reader.reset(slice, offset, token.length() - 1);
+ reader.reset(slice, offset, token.length() - 1, reader.getStringDecodeBuffer());
assertThrows(RuntimeException.class, () -> reader.readIsoLocalTime());
}
assertToken(ScalarCodecs.LocalTimeCodec.INSTANCE, timeText, expected);
@@ -696,7 +700,7 @@ public class JsonTemporalTest extends ForyJsonTestModels {
for (int value : new int[] {0, 31, 47, 58, 127, 128, 192, 255}) {
byte[] bytes = "\"01:02:03.123456789\",123456789".getBytes(StandardCharsets.UTF_8);
bytes[10 + position] = (byte) value;
- reader.reset(bytes);
+ reader.reset(bytes, reader.getStringDecodeBuffer());
assertThrows(RuntimeException.class, () -> reader.readIsoLocalTime());
}
}
@@ -710,12 +714,15 @@ public class JsonTemporalTest extends ForyJsonTestModels {
LocalDate first = LocalDate.of(0, 3, 1);
for (int day = 0; day < 146097; day++) {
LocalDate expected = first.plusDays(day);
- reader.reset(('"' + expected.toString() + '"').getBytes(StandardCharsets.US_ASCII));
+ reader.reset(
+ ('"' + expected.toString() + '"').getBytes(StandardCharsets.US_ASCII),
+ reader.getStringDecodeBuffer());
assertEquals(reader.readIsoLocalDate(), expected);
reader.finish();
reader.reset(
('"' + expected.toString() + "T23:59:59.999999999\",17")
- .getBytes(StandardCharsets.US_ASCII));
+ .getBytes(StandardCharsets.US_ASCII),
+ reader.getStringDecodeBuffer());
assertEquals(reader.readIsoLocalDateTime(), expected.atTime(LocalTime.MAX));
reader.expectNextToken(',');
assertEquals(reader.readInt(), 17);
@@ -755,12 +762,12 @@ public class JsonTemporalTest extends ForyJsonTestModels {
byte[] bytes = new byte[start + token.length];
System.arraycopy(token, 0, bytes, start, token.length);
Utf8JsonReader reader = newUtf8Reader(bytes);
- reader.reset(bytes, start, token.length);
+ reader.reset(bytes, start, token.length, reader.getStringDecodeBuffer());
assertEquals(
withTime ? reader.readIsoLocalDateTime() : reader.readIsoLocalDate(), expected);
reader.finish();
for (int length = 0; length < token.length; length++) {
- reader.reset(bytes, start, length);
+ reader.reset(bytes, start, length, reader.getStringDecodeBuffer());
assertThrows(
RuntimeException.class,
() -> {
@@ -855,7 +862,7 @@ public class JsonTemporalTest extends ForyJsonTestModels {
LocalTime expected = LocalTime.of(hour, minute, second, nano);
byte[] bytes =
('"' + expected.toString() + "\",17").getBytes(StandardCharsets.US_ASCII);
- reader.reset(bytes);
+ reader.reset(bytes, reader.getStringDecodeBuffer());
assertEquals(reader.readIsoLocalTime(), expected);
reader.expectNextToken(',');
assertEquals(reader.readInt(), 17);
@@ -866,9 +873,12 @@ public class JsonTemporalTest extends ForyJsonTestModels {
}
for (String text :
new String[] {"24:00:00", "00:60:00", "00:00:60", "99:99:99", "23:59:59.9999999999"}) {
- reader.reset(('"' + text + '"').getBytes(StandardCharsets.US_ASCII));
+ reader.reset(
+ ('"' + text + '"').getBytes(StandardCharsets.US_ASCII), reader.getStringDecodeBuffer());
assertThrows(RuntimeException.class, reader::readIsoLocalTime);
- reader.reset("\"23:59:59.999999999\"".getBytes(StandardCharsets.US_ASCII));
+ reader.reset(
+ "\"23:59:59.999999999\"".getBytes(StandardCharsets.US_ASCII),
+ reader.getStringDecodeBuffer());
assertEquals(reader.readIsoLocalTime(), LocalTime.MAX);
reader.finish();
}
@@ -922,15 +932,15 @@ public class JsonTemporalTest extends ForyJsonTestModels {
byte[] input = new byte[offset + token.length + 8];
System.arraycopy(token, 0, input, offset, token.length);
for (int length = 0; length < token.length; length++) {
- reader.reset(input, offset, length);
+ reader.reset(input, offset, length, reader.getStringDecodeBuffer());
assertThrows(ForyJsonException.class, reader::readDuration);
}
- reader.reset(input, offset, token.length);
+ reader.reset(input, offset, token.length, reader.getStringDecodeBuffer());
assertEquals(reader.readDuration(), expected);
reader.finish();
}
byte[] adjacent = ('"' + text + "\" 17").getBytes(StandardCharsets.US_ASCII);
- reader.reset(adjacent);
+ reader.reset(adjacent, reader.getStringDecodeBuffer());
assertEquals(reader.readDuration(), expected);
assertEquals(reader.readInt(), 17);
}
@@ -1020,14 +1030,16 @@ public class JsonTemporalTest extends ForyJsonTestModels {
byte[] input = new byte[offset + token.length + 8];
System.arraycopy(token, 0, input, offset, token.length);
for (int length = 0; length < token.length; length++) {
- reader.reset(input, offset, length);
+ reader.reset(input, offset, length, reader.getStringDecodeBuffer());
assertThrows(ForyJsonException.class, reader::readPeriod);
}
- reader.reset(input, offset, token.length);
+ reader.reset(input, offset, token.length, reader.getStringDecodeBuffer());
assertEquals(reader.readPeriod(), expected);
reader.finish();
}
- reader.reset(('"' + text + "\" 17").getBytes(StandardCharsets.US_ASCII));
+ reader.reset(
+ ('"' + text + "\" 17").getBytes(StandardCharsets.US_ASCII),
+ reader.getStringDecodeBuffer());
assertEquals(reader.readPeriod(), expected);
assertEquals(reader.readInt(), 17);
}
@@ -1075,7 +1087,9 @@ public class JsonTemporalTest extends ForyJsonTestModels {
Utf8JsonReader reader = newUtf8Reader(new byte[0]);
for (int second = 0; second < 86400; second++) {
Instant expected = Instant.ofEpochSecond(second, second * 1001);
- reader.reset(('"' + expected.toString() + "\",17").getBytes(StandardCharsets.US_ASCII));
+ reader.reset(
+ ('"' + expected.toString() + "\",17").getBytes(StandardCharsets.US_ASCII),
+ reader.getStringDecodeBuffer());
assertEquals(reader.readIsoInstant(), expected);
reader.expectNextToken(',');
assertEquals(reader.readInt(), 17);
@@ -1098,7 +1112,9 @@ public class JsonTemporalTest extends ForyJsonTestModels {
(month + day) % 60,
nanos[(year + month + day) & 3])
.toInstant(ZoneOffset.UTC);
- reader.reset(('"' + expected.toString() + '"').getBytes(StandardCharsets.US_ASCII));
+ reader.reset(
+ ('"' + expected.toString() + '"').getBytes(StandardCharsets.US_ASCII),
+ reader.getStringDecodeBuffer());
assertEquals(reader.readIsoInstant(), expected);
reader.finish();
}
@@ -1113,10 +1129,10 @@ public class JsonTemporalTest extends ForyJsonTestModels {
byte[] bytes = new byte[offset + token.length + 8];
System.arraycopy(token, 0, bytes, offset, token.length);
for (int length = 0; length < token.length; length++) {
- reader.reset(bytes, offset, length);
+ reader.reset(bytes, offset, length, reader.getStringDecodeBuffer());
assertThrows(ForyJsonException.class, reader::readIsoInstant);
}
- reader.reset(bytes, offset, token.length);
+ reader.reset(bytes, offset, token.length, reader.getStringDecodeBuffer());
assertEquals(reader.readIsoInstant(), Instant.parse(value));
reader.finish();
}
@@ -1159,16 +1175,16 @@ public class JsonTemporalTest extends ForyJsonTestModels {
for (int offset = 0; offset < 8; offset++) {
byte[] bytes = new byte[offset + token.length];
System.arraycopy(token, 0, bytes, offset, token.length);
- reader.reset(bytes, offset, token.length);
+ reader.reset(bytes, offset, token.length, reader.getStringDecodeBuffer());
assertEquals(reader.readIsoInstant(), expected);
reader.expectNextToken(',');
assertEquals(reader.readInt(), 17);
reader.finish();
- reader.reset(bytes, offset, token.length - 3);
+ reader.reset(bytes, offset, token.length - 3, reader.getStringDecodeBuffer());
assertEquals(reader.readIsoInstant(), expected);
reader.finish();
for (int end = 0; end < token.length - 3; end++) {
- reader.reset(bytes, offset, end);
+ reader.reset(bytes, offset, end, reader.getStringDecodeBuffer());
assertThrows(RuntimeException.class, reader::readIsoInstant);
}
}
@@ -1178,7 +1194,7 @@ public class JsonTemporalTest extends ForyJsonTestModels {
for (int value = 0; value < 256; value++) {
exact[lane] = (byte) value;
Latin1JsonReader reference = newLatin1Reader(exact);
- reader.reset(exact);
+ reader.reset(exact, reader.getStringDecodeBuffer());
Instant parsed;
try {
parsed = reference.readIsoInstant();
@@ -1297,11 +1313,11 @@ public class JsonTemporalTest extends ForyJsonTestModels {
bytes[offset + token.length + 1] = '1';
bytes[offset + token.length + 2] = '7';
Utf8JsonReader reader = newUtf8Reader(bytes);
- reader.reset(bytes, offset, token.length + 3);
+ reader.reset(bytes, offset, token.length + 3, reader.getStringDecodeBuffer());
assertEquals(reader.readZonedDateTime(), expected);
reader.expect(',');
assertEquals(reader.readInt(), 17);
- reader.reset(bytes, offset, token.length - 2);
+ reader.reset(bytes, offset, token.length - 2, reader.getStringDecodeBuffer());
assertThrows(RuntimeException.class, reader::readZonedDateTime);
}
}
diff --git a/java/fory-json/src/test/java/org/apache/fory/json/JsonTestSupport.java b/java/fory-json/src/test/java/org/apache/fory/json/JsonTestSupport.java
index 9942614d5..aa08a0989 100644
--- a/java/fory-json/src/test/java/org/apache/fory/json/JsonTestSupport.java
+++ b/java/fory-json/src/test/java/org/apache/fory/json/JsonTestSupport.java
@@ -109,15 +109,15 @@ final class JsonTestSupport {
}
static Utf8JsonReader newUtf8Reader(byte[] input) {
- return new Utf8JsonReader(CONFIG, newResolver(), input);
+ return new Utf8JsonReader(CONFIG, newResolver(), input, new byte[1024]);
}
static Latin1JsonReader newLatin1Reader(byte[] input) {
- return new Latin1JsonReader(CONFIG, newResolver(), input);
+ return new Latin1JsonReader(CONFIG, newResolver(), input, new byte[1024]);
}
static Latin1JsonReader newLatin1Reader(String input) {
- return new Latin1JsonReader(CONFIG, newResolver(), input);
+ return new Latin1JsonReader(CONFIG, newResolver(), input, new byte[1024]);
}
static Utf16JsonReader newUtf16Reader() {
@@ -125,7 +125,7 @@ final class JsonTestSupport {
}
static Utf16JsonReader newUtf16Reader(String input) {
- return new Utf16JsonReader(CONFIG, newResolver(), input);
+ return new Utf16JsonReader(CONFIG, newResolver(), input, new byte[1024]);
}
@SuppressWarnings("unchecked")
diff --git a/java/fory-json/src/test/java/org/apache/fory/json/reader/ZoneIdCacheTest.java b/java/fory-json/src/test/java/org/apache/fory/json/reader/ZoneIdCacheTest.java
index b09526283..5b98f02c2 100644
--- a/java/fory-json/src/test/java/org/apache/fory/json/reader/ZoneIdCacheTest.java
+++ b/java/fory-json/src/test/java/org/apache/fory/json/reader/ZoneIdCacheTest.java
@@ -105,9 +105,10 @@ public class ZoneIdCacheTest {
byte[] bytes = new byte[start + length];
Arrays.fill(bytes, (byte) '!');
System.arraycopy(expected, 0, bytes, start, length);
- Utf8JsonReader utf8 = new Utf8JsonReader(CONFIG, new JsonTypeResolver(REGISTRY), bytes);
+ Utf8JsonReader utf8 =
+ new Utf8JsonReader(CONFIG, new JsonTypeResolver(REGISTRY), bytes, new byte[1024]);
Latin1JsonReader latin1 =
- new Latin1JsonReader(CONFIG, new JsonTypeResolver(REGISTRY), bytes);
+ new Latin1JsonReader(CONFIG, new JsonTypeResolver(REGISTRY), bytes, new byte[1024]);
for (JsonReader reader : new JsonReader[] {utf8, latin1}) {
assertTrue(reader.matchesZoneId(start, bytes.length, expected));
assertFalse(reader.matchesZoneId(start, bytes.length - 1, expected));
@@ -149,7 +150,10 @@ public class ZoneIdCacheTest {
private static ZoneId read(ZoneIdCache cache, String id, long hash) {
Utf8JsonReader reader =
new Utf8JsonReader(
- CONFIG, new JsonTypeResolver(REGISTRY), id.getBytes(StandardCharsets.UTF_8));
+ CONFIG,
+ new JsonTypeResolver(REGISTRY),
+ id.getBytes(StandardCharsets.UTF_8),
+ new byte[1024]);
return cache.get(reader, 0, id.length(), hash);
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org