You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@johnzon.apache.org by GitBox <gi...@apache.org> on 2022/04/16 19:45:51 UTC
[GitHub] [johnzon] dblevins opened a new pull request, #84: Show a chunk of json when mapping the object fails
dblevins opened a new pull request, #84:
URL: https://github.com/apache/johnzon/pull/84
This appears in a stacktrace like the following
```
javax.json.bind.JsonbException: Can't map JSON Object to class java.lang.String: {"rendered":"Privacy Policy"}
at org.apache.johnzon.jsonb.JohnzonJsonb.fromJson(JohnzonJsonb.java:89)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@johnzon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [johnzon] rmannibucau commented on a diff in pull request #84: Show a chunk of json when mapping the object fails
Posted by GitBox <gi...@apache.org>.
rmannibucau commented on code in PR #84:
URL: https://github.com/apache/johnzon/pull/84#discussion_r857320207
##########
johnzon-core/src/main/java/org/apache/johnzon/core/Snippet.java:
##########
@@ -0,0 +1,435 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.johnzon.core;
+
+import javax.json.JsonArray;
+import javax.json.JsonObject;
+import javax.json.JsonValue;
+import javax.json.stream.JsonGenerator;
+import javax.json.stream.JsonGeneratorFactory;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.UncheckedIOException;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.johnzon.core.JsonGeneratorFactoryImpl.GENERATOR_BUFFER_LENGTH;
+
+/**
+ * Constructs short snippets of serialized JSON text representations of
+ * JsonValue instances in a way that is ideal for error messages.
+ *
+ * Instances of Snippet are thread-safe, reusable and memory-safe. Snippet
+ * serializes only enough of the json to fill the desired snippet size and
+ * is therefore safe to use regardless of the size of the JsonValue.
+ */
+public class Snippet {
+
+ private final int max;
+ private final JsonGeneratorFactory generatorFactory;
+
+ /**
+ * This constructor should be used only in static or other scenarios were
+ * there is no JsonGeneratorFactory instance in scope.
+ *
+ * @param max the maximum length of the serialized json produced via of()
+ */
+ public Snippet(final int max) {
+ this(max, new JsonGeneratorFactoryImpl(new HashMap<String, Object>() {
+ {
+ this.put(GENERATOR_BUFFER_LENGTH, max);
+ }
+ }));
+ }
+
+ /**
+ * This is the preferred approach to using Snippet in any context where
+ * there is an existing JsonGeneratorFactory in scope.
+ *
+ * @param max the maximum length of the serialized json produced via of()
+ * @param generatorFactory the JsonGeneratorFactory created by the user
+ */
+ public Snippet(final int max, final JsonGeneratorFactory generatorFactory) {
+ this.max = max;
+ this.generatorFactory = generatorFactory;
+ }
+
+ /**
+ * Create a serialized json representation of the supplied
+ * JsonValue, truncating the value to the specified max length.
+ * Truncated text appears with a suffix of "..."
+ *
+ * This method is thread safe.
+ *
+ * @param value the JsonValue to be serialized as json text
+ * @return a potentially truncated json text
+ */
+ public String of(final JsonValue value) {
+ switch (value.getValueType()) {
+ case TRUE: return "true";
Review Comment:
still think we should handle primitve accordingly the configuration for consistency. One impact is when you have some configuration and do some pattern matching, if you configure a max size of 3 (dont ask me why but if we don't want that we should forbid size < 5 which is fine too) then you expect to be able to grok the output with this configuration IMHO so no particular case IMHO.
##########
johnzon-core/src/main/java/org/apache/johnzon/core/Snippet.java:
##########
@@ -0,0 +1,435 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.johnzon.core;
+
+import javax.json.JsonArray;
+import javax.json.JsonObject;
+import javax.json.JsonValue;
+import javax.json.stream.JsonGenerator;
+import javax.json.stream.JsonGeneratorFactory;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.UncheckedIOException;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.johnzon.core.JsonGeneratorFactoryImpl.GENERATOR_BUFFER_LENGTH;
+
+/**
+ * Constructs short snippets of serialized JSON text representations of
+ * JsonValue instances in a way that is ideal for error messages.
+ *
+ * Instances of Snippet are thread-safe, reusable and memory-safe. Snippet
+ * serializes only enough of the json to fill the desired snippet size and
+ * is therefore safe to use regardless of the size of the JsonValue.
+ */
+public class Snippet {
+
+ private final int max;
+ private final JsonGeneratorFactory generatorFactory;
+
+ /**
+ * This constructor should be used only in static or other scenarios were
+ * there is no JsonGeneratorFactory instance in scope.
+ *
+ * @param max the maximum length of the serialized json produced via of()
+ */
+ public Snippet(final int max) {
Review Comment:
I'd really drop this one since minimum would be to require to copy the config (except the buffer size maybe) from the mapper one which is likely equivalent to use the mapper one as we do in the other constructor.
Also note that the buffer size is not important since before using any output we should flush it as any "output stream" so this one should drop IMHO.
##########
johnzon-core/src/main/java/org/apache/johnzon/core/Snippet.java:
##########
@@ -0,0 +1,435 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.johnzon.core;
+
+import javax.json.JsonArray;
+import javax.json.JsonObject;
+import javax.json.JsonValue;
+import javax.json.stream.JsonGenerator;
+import javax.json.stream.JsonGeneratorFactory;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.UncheckedIOException;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.johnzon.core.JsonGeneratorFactoryImpl.GENERATOR_BUFFER_LENGTH;
+
+/**
+ * Constructs short snippets of serialized JSON text representations of
+ * JsonValue instances in a way that is ideal for error messages.
+ *
+ * Instances of Snippet are thread-safe, reusable and memory-safe. Snippet
+ * serializes only enough of the json to fill the desired snippet size and
+ * is therefore safe to use regardless of the size of the JsonValue.
+ */
+public class Snippet {
+
+ private final int max;
+ private final JsonGeneratorFactory generatorFactory;
+
+ /**
+ * This constructor should be used only in static or other scenarios were
+ * there is no JsonGeneratorFactory instance in scope.
+ *
+ * @param max the maximum length of the serialized json produced via of()
+ */
+ public Snippet(final int max) {
+ this(max, new JsonGeneratorFactoryImpl(new HashMap<String, Object>() {
+ {
+ this.put(GENERATOR_BUFFER_LENGTH, max);
+ }
+ }));
+ }
+
+ /**
+ * This is the preferred approach to using Snippet in any context where
+ * there is an existing JsonGeneratorFactory in scope.
+ *
+ * @param max the maximum length of the serialized json produced via of()
+ * @param generatorFactory the JsonGeneratorFactory created by the user
+ */
+ public Snippet(final int max, final JsonGeneratorFactory generatorFactory) {
+ this.max = max;
+ this.generatorFactory = generatorFactory;
+ }
+
+ /**
+ * Create a serialized json representation of the supplied
+ * JsonValue, truncating the value to the specified max length.
+ * Truncated text appears with a suffix of "..."
+ *
+ * This method is thread safe.
+ *
+ * @param value the JsonValue to be serialized as json text
+ * @return a potentially truncated json text
+ */
+ public String of(final JsonValue value) {
+ switch (value.getValueType()) {
+ case TRUE: return "true";
+ case FALSE: return "false";
+ case NULL: return "null";
+ default: {
+ try (final Buffer buffer = new Buffer()) {
+ buffer.write(value);
+ return buffer.get();
+ }
+ }
+ }
+ }
+
+ /**
+ * Create a serialized json representation of the supplied
+ * JsonValue, truncating the value to the specified max length.
+ * Truncated text appears with a suffix of "..."
+ *
+ * This method is thread safe.
+ *
+ * Avoid using this method in any context where there already
+ * is a JsonGeneratorFactory instance in scope. For those scenarios
+ * use the constructor that accepts a JsonGeneratorFactory instead.
+ *
+ * @param value the JsonValue to be serialized as json text
+ * @param max the maximum length of the serialized json text
+ * @return a potentially truncated json text
+ */
+ public static String of(final JsonValue value, final int max) {
+ return new Snippet(max).of(value);
+ }
+
+ /**
+ * There are several buffers involved in the creation of a json string.
+ * This class carefully manages them all.
+ *
+ * JsonGeneratorImpl with a 64k buffer (by default)
+ * ObjectStreamWriter with an 8k buffer
+ * SnippetOutputStream with a buffer of maxSnippetLength
+ *
+ * As we create json via calling the JsonGenerator it is critical we
+ * flush the work in progress all the way through these buffers and into
+ * the final SnippetOutputStream buffer.
+ *
+ * If we do not, we risk creating up to 64k of json when we may only
+ * need 50 bytes. We could potentially optimize this code so the
+ * buffer held by JsonGeneratorImpl is also the maxSnippetLength.
+ */
+ class Buffer implements Closeable {
+ private final JsonGenerator generator;
+ private final SnippetOutputStream snippet;
+
+ private Buffer() {
+ this.snippet = new SnippetOutputStream(max);
+ this.generator = generatorFactory.createGenerator(snippet);
+ }
+
+ private void write(final JsonValue value) {
+ if (snippet.terminate()) {
+ return;
+ }
+
+ switch (value.getValueType()) {
+ case ARRAY: {
+ write(value.asJsonArray());
+ break;
+ }
+ case OBJECT: {
+ write(value.asJsonObject());
+ break;
+ }
+ default: {
+ generator.write(value);
+ generator.flush();
+ }
+ }
+ }
+
+ private void write(final JsonArray array) {
+ if (snippet.terminate()) {
+ return;
+ }
+
+ if (array.isEmpty()) {
+ generator.write(array);
+ generator.flush();
+ return;
+ }
+
+ generator.writeStartArray();
+ generator.flush();
+ for (final JsonValue jsonValue : array) {
+ if (snippet.terminate()) {
+ break;
+ }
+ write(jsonValue);
+ }
+ generator.writeEnd();
+ generator.flush();
+ }
+
+ private void write(final JsonObject object) {
+ if (snippet.terminate()) {
+ return;
+ }
+
+ if (object.isEmpty()) {
+ generator.write(object);
+ generator.flush();
+ return;
+ }
+
+ generator.writeStartObject();
+ generator.flush();
+ for (final Map.Entry<String, JsonValue> entry : object.entrySet()) {
+ if (snippet.terminate()) {
+ break;
+ }
+ write(entry.getKey(), entry.getValue());
+ }
+ generator.writeEnd();
+ generator.flush();
+ }
+
+ private void write(final String name, final JsonValue value) {
+ if (snippet.terminate()) {
+ return;
+ }
+
+ switch (value.getValueType()) {
+ case ARRAY:
+ generator.writeStartArray(name);
+ generator.flush();
+ final JsonArray array = value.asJsonArray();
+ for (final JsonValue jsonValue : array) {
+ if (snippet.terminate()) {
+ break;
+ }
+ write(jsonValue);
+ }
+ generator.writeEnd();
+ generator.flush();
+
+ break;
+ case OBJECT:
+ generator.writeStartObject(name);
+ generator.flush();
+ final JsonObject object = value.asJsonObject();
+ for (final Map.Entry<String, JsonValue> keyval : object.entrySet()) {
+ if (snippet.terminate()) {
+ break;
+ }
+ write(keyval.getKey(), keyval.getValue());
+ }
+ generator.writeEnd();
+ generator.flush();
+
+ break;
+ default: {
+ generator.write(name, value);
+ generator.flush();
+ }
+ }
+ }
+
+ private String get() {
+ generator.flush();
+ return snippet.isTruncated() ? snippet.get() + "..." : snippet.get();
+ }
+
+ @Override
+ public void close() {
+ generator.close();
+ }
+ }
+
+ /**
+ * Specialized OutputStream with three internal states:
+ * Writing, Completed, Truncated.
+ *
+ * When there is still space left for more json, the
+ * state will be Writing
+ *
+ * If the last write brought is exactly to the end of
+ * the max length, the state will be Completed.
+ *
+ * If the last write brought us over the max length, the
+ * state will be Truncated.
+ */
+ static class SnippetOutputStream extends OutputStream {
+
+ private final ByteArrayOutputStream buffer;
+ private OutputStream mode;
+
+ public SnippetOutputStream(final int max) {
+ this.buffer = new ByteArrayOutputStream(Math.min(max, 8192));
+ this.mode = new Writing(max, buffer);
+ }
+
+ public String get() {
+ return buffer.toString();
+ }
+
+ /**
+ * Calling this method implies the need to continue
+ * writing and a question on if that is ok.
+ *
+ * It impacts internal state in the same way as
+ * calling a write method.
+ *
+ * @return true if no more writes are possible
+ */
+ public boolean terminate() {
+ if (mode instanceof Truncated) {
+ return true;
+ }
+
+ if (mode instanceof Completed) {
+ mode = new Truncated();
+ return true;
+ }
+
+ return false;
+ }
+
+ public boolean isTruncated() {
+ return mode instanceof Truncated;
+ }
+
+ @Override
+ public void write(final int b) throws IOException {
+ mode.write(b);
+ }
+
+ @Override
+ public void write(final byte[] b) throws IOException {
+ mode.write(b);
+ }
+
+ @Override
+ public void write(final byte[] b, final int off, final int len) throws IOException {
+ mode.write(b, off, len);
+ }
+
+ @Override
+ public void flush() throws IOException {
+ mode.flush();
+ }
+
+ @Override
+ public void close() throws IOException {
+ mode.close();
+ }
+
+ public void print(final String string) {
Review Comment:
unused? let's drop it?
side note: we shouldn't call getBytes() without the encoding anywhere there so since it is the single location doing it, dropping it solves it ;)
##########
johnzon-core/src/main/java/org/apache/johnzon/core/Snippet.java:
##########
@@ -0,0 +1,435 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.johnzon.core;
+
+import javax.json.JsonArray;
+import javax.json.JsonObject;
+import javax.json.JsonValue;
+import javax.json.stream.JsonGenerator;
+import javax.json.stream.JsonGeneratorFactory;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.UncheckedIOException;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.johnzon.core.JsonGeneratorFactoryImpl.GENERATOR_BUFFER_LENGTH;
+
+/**
+ * Constructs short snippets of serialized JSON text representations of
+ * JsonValue instances in a way that is ideal for error messages.
+ *
+ * Instances of Snippet are thread-safe, reusable and memory-safe. Snippet
+ * serializes only enough of the json to fill the desired snippet size and
+ * is therefore safe to use regardless of the size of the JsonValue.
+ */
+public class Snippet {
+
+ private final int max;
+ private final JsonGeneratorFactory generatorFactory;
+
+ /**
+ * This constructor should be used only in static or other scenarios were
+ * there is no JsonGeneratorFactory instance in scope.
+ *
+ * @param max the maximum length of the serialized json produced via of()
+ */
+ public Snippet(final int max) {
+ this(max, new JsonGeneratorFactoryImpl(new HashMap<String, Object>() {
+ {
+ this.put(GENERATOR_BUFFER_LENGTH, max);
+ }
+ }));
+ }
+
+ /**
+ * This is the preferred approach to using Snippet in any context where
+ * there is an existing JsonGeneratorFactory in scope.
+ *
+ * @param max the maximum length of the serialized json produced via of()
+ * @param generatorFactory the JsonGeneratorFactory created by the user
+ */
+ public Snippet(final int max, final JsonGeneratorFactory generatorFactory) {
+ this.max = max;
+ this.generatorFactory = generatorFactory;
+ }
+
+ /**
+ * Create a serialized json representation of the supplied
+ * JsonValue, truncating the value to the specified max length.
+ * Truncated text appears with a suffix of "..."
+ *
+ * This method is thread safe.
+ *
+ * @param value the JsonValue to be serialized as json text
+ * @return a potentially truncated json text
+ */
+ public String of(final JsonValue value) {
+ switch (value.getValueType()) {
+ case TRUE: return "true";
+ case FALSE: return "false";
+ case NULL: return "null";
+ default: {
+ try (final Buffer buffer = new Buffer()) {
+ buffer.write(value);
+ return buffer.get();
+ }
+ }
+ }
+ }
+
+ /**
+ * Create a serialized json representation of the supplied
+ * JsonValue, truncating the value to the specified max length.
+ * Truncated text appears with a suffix of "..."
+ *
+ * This method is thread safe.
+ *
+ * Avoid using this method in any context where there already
+ * is a JsonGeneratorFactory instance in scope. For those scenarios
+ * use the constructor that accepts a JsonGeneratorFactory instead.
+ *
+ * @param value the JsonValue to be serialized as json text
+ * @param max the maximum length of the serialized json text
+ * @return a potentially truncated json text
+ */
+ public static String of(final JsonValue value, final int max) {
+ return new Snippet(max).of(value);
+ }
+
+ /**
+ * There are several buffers involved in the creation of a json string.
+ * This class carefully manages them all.
+ *
+ * JsonGeneratorImpl with a 64k buffer (by default)
+ * ObjectStreamWriter with an 8k buffer
+ * SnippetOutputStream with a buffer of maxSnippetLength
+ *
+ * As we create json via calling the JsonGenerator it is critical we
+ * flush the work in progress all the way through these buffers and into
+ * the final SnippetOutputStream buffer.
+ *
+ * If we do not, we risk creating up to 64k of json when we may only
+ * need 50 bytes. We could potentially optimize this code so the
+ * buffer held by JsonGeneratorImpl is also the maxSnippetLength.
+ */
+ class Buffer implements Closeable {
+ private final JsonGenerator generator;
+ private final SnippetOutputStream snippet;
+
+ private Buffer() {
+ this.snippet = new SnippetOutputStream(max);
+ this.generator = generatorFactory.createGenerator(snippet);
+ }
+
+ private void write(final JsonValue value) {
+ if (snippet.terminate()) {
+ return;
+ }
+
+ switch (value.getValueType()) {
+ case ARRAY: {
+ write(value.asJsonArray());
+ break;
+ }
+ case OBJECT: {
+ write(value.asJsonObject());
+ break;
+ }
+ default: {
+ generator.write(value);
+ generator.flush();
+ }
+ }
+ }
+
+ private void write(final JsonArray array) {
+ if (snippet.terminate()) {
+ return;
+ }
+
+ if (array.isEmpty()) {
+ generator.write(array);
+ generator.flush();
+ return;
+ }
+
+ generator.writeStartArray();
+ generator.flush();
+ for (final JsonValue jsonValue : array) {
+ if (snippet.terminate()) {
+ break;
+ }
+ write(jsonValue);
+ }
+ generator.writeEnd();
+ generator.flush();
+ }
+
+ private void write(final JsonObject object) {
+ if (snippet.terminate()) {
+ return;
+ }
+
+ if (object.isEmpty()) {
+ generator.write(object);
+ generator.flush();
+ return;
+ }
+
+ generator.writeStartObject();
+ generator.flush();
+ for (final Map.Entry<String, JsonValue> entry : object.entrySet()) {
+ if (snippet.terminate()) {
+ break;
+ }
+ write(entry.getKey(), entry.getValue());
+ }
+ generator.writeEnd();
+ generator.flush();
+ }
+
+ private void write(final String name, final JsonValue value) {
+ if (snippet.terminate()) {
+ return;
+ }
+
+ switch (value.getValueType()) {
+ case ARRAY:
+ generator.writeStartArray(name);
+ generator.flush();
+ final JsonArray array = value.asJsonArray();
+ for (final JsonValue jsonValue : array) {
+ if (snippet.terminate()) {
+ break;
+ }
+ write(jsonValue);
+ }
+ generator.writeEnd();
+ generator.flush();
+
+ break;
+ case OBJECT:
+ generator.writeStartObject(name);
+ generator.flush();
+ final JsonObject object = value.asJsonObject();
+ for (final Map.Entry<String, JsonValue> keyval : object.entrySet()) {
+ if (snippet.terminate()) {
+ break;
+ }
+ write(keyval.getKey(), keyval.getValue());
+ }
+ generator.writeEnd();
+ generator.flush();
+
+ break;
+ default: {
+ generator.write(name, value);
+ generator.flush();
+ }
+ }
+ }
+
+ private String get() {
+ generator.flush();
+ return snippet.isTruncated() ? snippet.get() + "..." : snippet.get();
+ }
+
+ @Override
+ public void close() {
+ generator.close();
+ }
+ }
+
+ /**
+ * Specialized OutputStream with three internal states:
+ * Writing, Completed, Truncated.
+ *
+ * When there is still space left for more json, the
+ * state will be Writing
+ *
+ * If the last write brought is exactly to the end of
+ * the max length, the state will be Completed.
+ *
+ * If the last write brought us over the max length, the
+ * state will be Truncated.
+ */
+ static class SnippetOutputStream extends OutputStream {
+
+ private final ByteArrayOutputStream buffer;
+ private OutputStream mode;
+
+ public SnippetOutputStream(final int max) {
+ this.buffer = new ByteArrayOutputStream(Math.min(max, 8192));
+ this.mode = new Writing(max, buffer);
+ }
+
+ public String get() {
+ return buffer.toString();
+ }
+
+ /**
+ * Calling this method implies the need to continue
+ * writing and a question on if that is ok.
+ *
+ * It impacts internal state in the same way as
+ * calling a write method.
+ *
+ * @return true if no more writes are possible
+ */
+ public boolean terminate() {
Review Comment:
Like this rework, way more explicit!
##########
johnzon-core/src/main/java/org/apache/johnzon/core/Snippet.java:
##########
@@ -0,0 +1,435 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.johnzon.core;
+
+import javax.json.JsonArray;
+import javax.json.JsonObject;
+import javax.json.JsonValue;
+import javax.json.stream.JsonGenerator;
+import javax.json.stream.JsonGeneratorFactory;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.UncheckedIOException;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.johnzon.core.JsonGeneratorFactoryImpl.GENERATOR_BUFFER_LENGTH;
+
+/**
+ * Constructs short snippets of serialized JSON text representations of
+ * JsonValue instances in a way that is ideal for error messages.
+ *
+ * Instances of Snippet are thread-safe, reusable and memory-safe. Snippet
+ * serializes only enough of the json to fill the desired snippet size and
+ * is therefore safe to use regardless of the size of the JsonValue.
+ */
+public class Snippet {
+
+ private final int max;
+ private final JsonGeneratorFactory generatorFactory;
+
+ /**
+ * This constructor should be used only in static or other scenarios were
+ * there is no JsonGeneratorFactory instance in scope.
+ *
+ * @param max the maximum length of the serialized json produced via of()
+ */
+ public Snippet(final int max) {
+ this(max, new JsonGeneratorFactoryImpl(new HashMap<String, Object>() {
+ {
+ this.put(GENERATOR_BUFFER_LENGTH, max);
+ }
+ }));
+ }
+
+ /**
+ * This is the preferred approach to using Snippet in any context where
+ * there is an existing JsonGeneratorFactory in scope.
+ *
+ * @param max the maximum length of the serialized json produced via of()
+ * @param generatorFactory the JsonGeneratorFactory created by the user
+ */
+ public Snippet(final int max, final JsonGeneratorFactory generatorFactory) {
+ this.max = max;
+ this.generatorFactory = generatorFactory;
+ }
+
+ /**
+ * Create a serialized json representation of the supplied
+ * JsonValue, truncating the value to the specified max length.
+ * Truncated text appears with a suffix of "..."
+ *
+ * This method is thread safe.
+ *
+ * @param value the JsonValue to be serialized as json text
+ * @return a potentially truncated json text
+ */
+ public String of(final JsonValue value) {
+ switch (value.getValueType()) {
+ case TRUE: return "true";
+ case FALSE: return "false";
+ case NULL: return "null";
+ default: {
+ try (final Buffer buffer = new Buffer()) {
+ buffer.write(value);
+ return buffer.get();
+ }
+ }
+ }
+ }
+
+ /**
+ * Create a serialized json representation of the supplied
+ * JsonValue, truncating the value to the specified max length.
+ * Truncated text appears with a suffix of "..."
+ *
+ * This method is thread safe.
+ *
+ * Avoid using this method in any context where there already
+ * is a JsonGeneratorFactory instance in scope. For those scenarios
+ * use the constructor that accepts a JsonGeneratorFactory instead.
+ *
+ * @param value the JsonValue to be serialized as json text
+ * @param max the maximum length of the serialized json text
+ * @return a potentially truncated json text
+ */
+ public static String of(final JsonValue value, final int max) {
+ return new Snippet(max).of(value);
+ }
+
+ /**
+ * There are several buffers involved in the creation of a json string.
+ * This class carefully manages them all.
+ *
+ * JsonGeneratorImpl with a 64k buffer (by default)
+ * ObjectStreamWriter with an 8k buffer
+ * SnippetOutputStream with a buffer of maxSnippetLength
+ *
+ * As we create json via calling the JsonGenerator it is critical we
+ * flush the work in progress all the way through these buffers and into
+ * the final SnippetOutputStream buffer.
+ *
+ * If we do not, we risk creating up to 64k of json when we may only
+ * need 50 bytes. We could potentially optimize this code so the
+ * buffer held by JsonGeneratorImpl is also the maxSnippetLength.
+ */
+ class Buffer implements Closeable {
+ private final JsonGenerator generator;
+ private final SnippetOutputStream snippet;
+
+ private Buffer() {
+ this.snippet = new SnippetOutputStream(max);
+ this.generator = generatorFactory.createGenerator(snippet);
+ }
+
+ private void write(final JsonValue value) {
+ if (snippet.terminate()) {
+ return;
+ }
+
+ switch (value.getValueType()) {
+ case ARRAY: {
+ write(value.asJsonArray());
+ break;
+ }
+ case OBJECT: {
+ write(value.asJsonObject());
+ break;
+ }
+ default: {
+ generator.write(value);
+ generator.flush();
+ }
+ }
+ }
+
+ private void write(final JsonArray array) {
+ if (snippet.terminate()) {
+ return;
+ }
+
+ if (array.isEmpty()) {
+ generator.write(array);
+ generator.flush();
+ return;
+ }
+
+ generator.writeStartArray();
+ generator.flush();
Review Comment:
no need to flush *anywhere* (will not copy the comment on each line since it seems you got gluttonny ;))
##########
johnzon-core/src/main/java/org/apache/johnzon/core/Snippet.java:
##########
@@ -0,0 +1,435 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.johnzon.core;
+
+import javax.json.JsonArray;
+import javax.json.JsonObject;
+import javax.json.JsonValue;
+import javax.json.stream.JsonGenerator;
+import javax.json.stream.JsonGeneratorFactory;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.UncheckedIOException;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.johnzon.core.JsonGeneratorFactoryImpl.GENERATOR_BUFFER_LENGTH;
+
+/**
+ * Constructs short snippets of serialized JSON text representations of
+ * JsonValue instances in a way that is ideal for error messages.
+ *
+ * Instances of Snippet are thread-safe, reusable and memory-safe. Snippet
+ * serializes only enough of the json to fill the desired snippet size and
+ * is therefore safe to use regardless of the size of the JsonValue.
+ */
+public class Snippet {
+
+ private final int max;
+ private final JsonGeneratorFactory generatorFactory;
+
+ /**
+ * This constructor should be used only in static or other scenarios were
+ * there is no JsonGeneratorFactory instance in scope.
+ *
+ * @param max the maximum length of the serialized json produced via of()
+ */
+ public Snippet(final int max) {
+ this(max, new JsonGeneratorFactoryImpl(new HashMap<String, Object>() {
+ {
+ this.put(GENERATOR_BUFFER_LENGTH, max);
+ }
+ }));
+ }
+
+ /**
+ * This is the preferred approach to using Snippet in any context where
+ * there is an existing JsonGeneratorFactory in scope.
+ *
+ * @param max the maximum length of the serialized json produced via of()
+ * @param generatorFactory the JsonGeneratorFactory created by the user
+ */
+ public Snippet(final int max, final JsonGeneratorFactory generatorFactory) {
+ this.max = max;
+ this.generatorFactory = generatorFactory;
+ }
+
+ /**
+ * Create a serialized json representation of the supplied
+ * JsonValue, truncating the value to the specified max length.
+ * Truncated text appears with a suffix of "..."
+ *
+ * This method is thread safe.
+ *
+ * @param value the JsonValue to be serialized as json text
+ * @return a potentially truncated json text
+ */
+ public String of(final JsonValue value) {
+ switch (value.getValueType()) {
+ case TRUE: return "true";
+ case FALSE: return "false";
+ case NULL: return "null";
+ default: {
+ try (final Buffer buffer = new Buffer()) {
+ buffer.write(value);
+ return buffer.get();
+ }
+ }
+ }
+ }
+
+ /**
+ * Create a serialized json representation of the supplied
+ * JsonValue, truncating the value to the specified max length.
+ * Truncated text appears with a suffix of "..."
+ *
+ * This method is thread safe.
+ *
+ * Avoid using this method in any context where there already
+ * is a JsonGeneratorFactory instance in scope. For those scenarios
+ * use the constructor that accepts a JsonGeneratorFactory instead.
+ *
+ * @param value the JsonValue to be serialized as json text
+ * @param max the maximum length of the serialized json text
+ * @return a potentially truncated json text
+ */
+ public static String of(final JsonValue value, final int max) {
+ return new Snippet(max).of(value);
+ }
+
+ /**
+ * There are several buffers involved in the creation of a json string.
+ * This class carefully manages them all.
+ *
+ * JsonGeneratorImpl with a 64k buffer (by default)
+ * ObjectStreamWriter with an 8k buffer
+ * SnippetOutputStream with a buffer of maxSnippetLength
+ *
+ * As we create json via calling the JsonGenerator it is critical we
+ * flush the work in progress all the way through these buffers and into
+ * the final SnippetOutputStream buffer.
+ *
+ * If we do not, we risk creating up to 64k of json when we may only
+ * need 50 bytes. We could potentially optimize this code so the
+ * buffer held by JsonGeneratorImpl is also the maxSnippetLength.
+ */
+ class Buffer implements Closeable {
+ private final JsonGenerator generator;
+ private final SnippetOutputStream snippet;
+
+ private Buffer() {
+ this.snippet = new SnippetOutputStream(max);
+ this.generator = generatorFactory.createGenerator(snippet);
+ }
+
+ private void write(final JsonValue value) {
+ if (snippet.terminate()) {
+ return;
+ }
+
+ switch (value.getValueType()) {
+ case ARRAY: {
+ write(value.asJsonArray());
+ break;
+ }
+ case OBJECT: {
+ write(value.asJsonObject());
+ break;
+ }
+ default: {
+ generator.write(value);
+ generator.flush();
+ }
+ }
+ }
+
+ private void write(final JsonArray array) {
+ if (snippet.terminate()) {
+ return;
+ }
+
+ if (array.isEmpty()) {
+ generator.write(array);
+ generator.flush();
+ return;
+ }
+
+ generator.writeStartArray();
+ generator.flush();
+ for (final JsonValue jsonValue : array) {
+ if (snippet.terminate()) {
+ break;
+ }
+ write(jsonValue);
+ }
+ generator.writeEnd();
+ generator.flush();
+ }
+
+ private void write(final JsonObject object) {
+ if (snippet.terminate()) {
+ return;
+ }
+
+ if (object.isEmpty()) {
+ generator.write(object);
+ generator.flush();
+ return;
+ }
+
+ generator.writeStartObject();
+ generator.flush();
+ for (final Map.Entry<String, JsonValue> entry : object.entrySet()) {
+ if (snippet.terminate()) {
+ break;
+ }
+ write(entry.getKey(), entry.getValue());
+ }
+ generator.writeEnd();
+ generator.flush();
+ }
+
+ private void write(final String name, final JsonValue value) {
+ if (snippet.terminate()) {
+ return;
+ }
+
+ switch (value.getValueType()) {
+ case ARRAY:
+ generator.writeStartArray(name);
+ generator.flush();
+ final JsonArray array = value.asJsonArray();
+ for (final JsonValue jsonValue : array) {
+ if (snippet.terminate()) {
+ break;
+ }
+ write(jsonValue);
+ }
+ generator.writeEnd();
+ generator.flush();
+
+ break;
+ case OBJECT:
+ generator.writeStartObject(name);
+ generator.flush();
+ final JsonObject object = value.asJsonObject();
+ for (final Map.Entry<String, JsonValue> keyval : object.entrySet()) {
+ if (snippet.terminate()) {
+ break;
+ }
+ write(keyval.getKey(), keyval.getValue());
+ }
+ generator.writeEnd();
+ generator.flush();
+
+ break;
+ default: {
+ generator.write(name, value);
+ generator.flush();
+ }
+ }
+ }
+
+ private String get() {
Review Comment:
your flush issue comes from there, let's drop the flush and call get after the auto-close in https://github.com/apache/johnzon/pull/84/files#diff-f8878dc73ad8ecae625a2bf13a77b5718505c77f547c31cefc2454e42454138aR113 which should be written as:
```
public static String of(final JsonValue value, final int max) {
final Snippet snippet = new Snippet(max);
try (final Snippet s = snippet) { // use the style you want but the important part is to close before using the in mem stream
s.of(value)
}
return s.get();
}
```
side note: without that, the generator.close is not called and we leak as mentionned in early comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@johnzon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [johnzon] dblevins commented on a diff in pull request #84: Show a chunk of json when mapping the object fails
Posted by GitBox <gi...@apache.org>.
dblevins commented on code in PR #84:
URL: https://github.com/apache/johnzon/pull/84#discussion_r860575280
##########
johnzon-core/src/main/java/org/apache/johnzon/core/Snippet.java:
##########
@@ -228,172 +232,181 @@ private String get() {
public void close() {
generator.close();
}
- }
-
- /**
- * Specialized OutputStream with three internal states:
- * Writing, Completed, Truncated.
- *
- * When there is still space left for more json, the
- * state will be Writing
- *
- * If the last write brought is exactly to the end of
- * the max length, the state will be Completed.
- *
- * If the last write brought us over the max length, the
- * state will be Truncated.
- */
- static class SnippetOutputStream extends OutputStream {
-
- private final ByteArrayOutputStream buffer;
- private OutputStream mode;
-
- public SnippetOutputStream(final int max) {
- this.buffer = new ByteArrayOutputStream(Math.min(max, 8192));
- this.mode = new Writing(max, buffer);
- }
-
- public String get() {
- return buffer.toString();
- }
/**
- * Calling this method implies the need to continue
- * writing and a question on if that is ok.
+ * Specialized Writer with three internal states:
+ * Writing, Completed, Truncated.
*
- * It impacts internal state in the same way as
- * calling a write method.
+ * When there is still space left for more json, the
+ * state will be Writing
*
- * @return true if no more writes are possible
+ * If the last write brought is exactly to the end of
+ * the max length, the state will be Completed.
+ *
+ * If the last write brought us over the max length, the
+ * state will be Truncated.
*/
- public boolean terminate() {
- if (mode instanceof Truncated) {
- return true;
+ class SnippetWriter extends Writer implements Buffered {
+
+ private final ByteArrayOutputStream buffer;
+ private Mode mode;
+ private Supplier<Integer> bufferSize;
+
+ public SnippetWriter(final int max) {
+ final int size = Math.min(max, 8192);
+
+ this.buffer = new ByteArrayOutputStream(size);
+ this.mode = new Writing(max, new OutputStreamWriter(buffer));
+
+ /*
+ * The first time the buffer size is requested, disable flushing
+ * as we know our requested buffer size will be respected
+ */
+ this.bufferSize = () -> {
+ // disable flushing
+ flush = () -> {
+ };
+ // future calls can just return the size
+ bufferSize = () -> size;
+ return size;
+ };
}
- if (mode instanceof Completed) {
- mode = new Truncated();
- return true;
+ public String get() {
+ return buffer.toString();
}
- return false;
- }
-
- public boolean isTruncated() {
- return mode instanceof Truncated;
- }
-
- @Override
- public void write(final int b) throws IOException {
- mode.write(b);
- }
-
- @Override
- public void write(final byte[] b) throws IOException {
- mode.write(b);
- }
-
- @Override
- public void write(final byte[] b, final int off, final int len) throws IOException {
- mode.write(b, off, len);
- }
+ @Override
+ public int bufferSize() {
+ return bufferSize.get();
+ }
- @Override
- public void flush() throws IOException {
- mode.flush();
- }
+ /**
+ * Calling this method implies the need to continue
+ * writing and a question on if that is ok.
+ *
+ * It impacts internal state in the same way as
+ * calling a write method.
+ *
+ * @return true if no more writes are possible
+ */
+ public boolean terminate() {
+ if (mode instanceof Truncated) {
+ return true;
+ }
- @Override
- public void close() throws IOException {
- mode.close();
- }
+ if (mode instanceof Completed) {
+ mode = new Truncated();
+ return true;
+ }
- class Writing extends OutputStream {
- private final int max;
- private int count;
- private final OutputStream out;
+ return false;
+ }
- public Writing(final int max, final OutputStream out) {
- this.max = max;
- this.out = out;
+ public boolean isTruncated() {
+ return mode instanceof Truncated;
}
@Override
- public void write(final int b) throws IOException {
- if (++count < max) {
- out.write(b);
- } else {
- maxReached(new Truncated());
- }
+ public void write(final char[] cbuf, final int off, final int len) throws IOException {
+ mode.write(cbuf, off, len);
}
@Override
- public void write(final byte[] b) throws IOException {
- write(b, 0, b.length);
+ public void flush() throws IOException {
+ mode.flush();
}
@Override
- public void write(final byte[] b, final int off, final int len) throws IOException {
- final int remaining = max - count;
+ public void close() throws IOException {
+ mode.close();
+ }
- if (remaining <= 0) {
+ abstract class Mode extends Writer {
+ @Override
+ public void flush() throws IOException {
Review Comment:
Added! I'll make sure to do this any future PRs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@johnzon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [johnzon] dblevins commented on pull request #84: Show a chunk of json when mapping the object fails
Posted by GitBox <gi...@apache.org>.
dblevins commented on PR #84:
URL: https://github.com/apache/johnzon/pull/84#issuecomment-1101712945
When you say direct in mem value, do you mean there is already the related chunk of json already available in memory somewhere so it can be displayed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@johnzon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [johnzon] dblevins commented on a diff in pull request #84: Show a chunk of json when mapping the object fails
Posted by GitBox <gi...@apache.org>.
dblevins commented on code in PR #84:
URL: https://github.com/apache/johnzon/pull/84#discussion_r858239080
##########
johnzon-core/src/main/java/org/apache/johnzon/core/Snippet.java:
##########
@@ -0,0 +1,435 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.johnzon.core;
+
+import javax.json.JsonArray;
+import javax.json.JsonObject;
+import javax.json.JsonValue;
+import javax.json.stream.JsonGenerator;
+import javax.json.stream.JsonGeneratorFactory;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.UncheckedIOException;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.johnzon.core.JsonGeneratorFactoryImpl.GENERATOR_BUFFER_LENGTH;
+
+/**
+ * Constructs short snippets of serialized JSON text representations of
+ * JsonValue instances in a way that is ideal for error messages.
+ *
+ * Instances of Snippet are thread-safe, reusable and memory-safe. Snippet
+ * serializes only enough of the json to fill the desired snippet size and
+ * is therefore safe to use regardless of the size of the JsonValue.
+ */
+public class Snippet {
+
+ private final int max;
+ private final JsonGeneratorFactory generatorFactory;
+
+ /**
+ * This constructor should be used only in static or other scenarios were
+ * there is no JsonGeneratorFactory instance in scope.
+ *
+ * @param max the maximum length of the serialized json produced via of()
+ */
+ public Snippet(final int max) {
+ this(max, new JsonGeneratorFactoryImpl(new HashMap<String, Object>() {
+ {
+ this.put(GENERATOR_BUFFER_LENGTH, max);
+ }
+ }));
+ }
+
+ /**
+ * This is the preferred approach to using Snippet in any context where
+ * there is an existing JsonGeneratorFactory in scope.
+ *
+ * @param max the maximum length of the serialized json produced via of()
+ * @param generatorFactory the JsonGeneratorFactory created by the user
+ */
+ public Snippet(final int max, final JsonGeneratorFactory generatorFactory) {
+ this.max = max;
+ this.generatorFactory = generatorFactory;
+ }
+
+ /**
+ * Create a serialized json representation of the supplied
+ * JsonValue, truncating the value to the specified max length.
+ * Truncated text appears with a suffix of "..."
+ *
+ * This method is thread safe.
+ *
+ * @param value the JsonValue to be serialized as json text
+ * @return a potentially truncated json text
+ */
+ public String of(final JsonValue value) {
+ switch (value.getValueType()) {
+ case TRUE: return "true";
Review Comment:
Though not my preference, I implemented this in the spirit of compromise on the constructors/static method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@johnzon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [johnzon] dblevins commented on a diff in pull request #84: Show a chunk of json when mapping the object fails
Posted by GitBox <gi...@apache.org>.
dblevins commented on code in PR #84:
URL: https://github.com/apache/johnzon/pull/84#discussion_r860535420
##########
johnzon-core/src/main/java/org/apache/johnzon/core/Buffered.java:
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.johnzon.core;
+
+/**
+ * A <tt>Buffered</tt> is a source or destination of data that is buffered
+ * before writing or reading. The bufferSize method allows all participants
+ * in the underlying stream to align on this buffer size for optimization.
+ *
+ * This interface is designed in the spirit of {@code java.io.Flushable} and
+ * {@code java.io.Closeable}
+ *
+ * @since 1.2.17
+ */
+public interface Buffered {
Review Comment:
My first instinct was Bufferable as well. What swayed me away is the "able" interfaces are all describing an action someone can take on the object and if they don't take that action, the thing described is not done. If you don't serialize a `Serializable` it isn't serialized. If you don't close a `Closeable` it isn't closed. `Flushable` objects of course do flush eventually, but the method it has still triggers an action. So calling it `Bufferable` felt a bit off as there's no action for the user to do. It's buffered whether or not the user does anything and calling `bufferSize` does nothing. `Buffered` is a statement of fact and `bufferSize` returns that fact.
All that said, I'm not super married to it. That's just the logic that got me there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@johnzon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [johnzon] dblevins commented on pull request #84: Show a chunk of json when mapping the object fails
Posted by GitBox <gi...@apache.org>.
dblevins commented on PR #84:
URL: https://github.com/apache/johnzon/pull/84#issuecomment-1108749479
I'll review the comments in detail, but I'd like to offer a compromise. I'll make the change to "null", etc being truncated that I don't agree with in exchange for you allowing the Snippet(int) constructor and Snippet.of(value, int) static method to live. I use Johnzon in almost every project and I'd really like the ability to use this code elsewhere and have those methods.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@johnzon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [johnzon] dblevins commented on a diff in pull request #84: Show a chunk of json when mapping the object fails
Posted by GitBox <gi...@apache.org>.
dblevins commented on code in PR #84:
URL: https://github.com/apache/johnzon/pull/84#discussion_r858240095
##########
johnzon-core/src/main/java/org/apache/johnzon/core/Snippet.java:
##########
@@ -0,0 +1,435 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.johnzon.core;
+
+import javax.json.JsonArray;
+import javax.json.JsonObject;
+import javax.json.JsonValue;
+import javax.json.stream.JsonGenerator;
+import javax.json.stream.JsonGeneratorFactory;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.UncheckedIOException;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.johnzon.core.JsonGeneratorFactoryImpl.GENERATOR_BUFFER_LENGTH;
+
+/**
+ * Constructs short snippets of serialized JSON text representations of
+ * JsonValue instances in a way that is ideal for error messages.
+ *
+ * Instances of Snippet are thread-safe, reusable and memory-safe. Snippet
+ * serializes only enough of the json to fill the desired snippet size and
+ * is therefore safe to use regardless of the size of the JsonValue.
+ */
+public class Snippet {
+
+ private final int max;
+ private final JsonGeneratorFactory generatorFactory;
+
+ /**
+ * This constructor should be used only in static or other scenarios were
+ * there is no JsonGeneratorFactory instance in scope.
+ *
+ * @param max the maximum length of the serialized json produced via of()
+ */
+ public Snippet(final int max) {
+ this(max, new JsonGeneratorFactoryImpl(new HashMap<String, Object>() {
+ {
+ this.put(GENERATOR_BUFFER_LENGTH, max);
+ }
+ }));
+ }
+
+ /**
+ * This is the preferred approach to using Snippet in any context where
+ * there is an existing JsonGeneratorFactory in scope.
+ *
+ * @param max the maximum length of the serialized json produced via of()
+ * @param generatorFactory the JsonGeneratorFactory created by the user
+ */
+ public Snippet(final int max, final JsonGeneratorFactory generatorFactory) {
+ this.max = max;
+ this.generatorFactory = generatorFactory;
+ }
+
+ /**
+ * Create a serialized json representation of the supplied
+ * JsonValue, truncating the value to the specified max length.
+ * Truncated text appears with a suffix of "..."
+ *
+ * This method is thread safe.
+ *
+ * @param value the JsonValue to be serialized as json text
+ * @return a potentially truncated json text
+ */
+ public String of(final JsonValue value) {
+ switch (value.getValueType()) {
+ case TRUE: return "true";
+ case FALSE: return "false";
+ case NULL: return "null";
+ default: {
+ try (final Buffer buffer = new Buffer()) {
+ buffer.write(value);
+ return buffer.get();
+ }
+ }
+ }
+ }
+
+ /**
+ * Create a serialized json representation of the supplied
+ * JsonValue, truncating the value to the specified max length.
+ * Truncated text appears with a suffix of "..."
+ *
+ * This method is thread safe.
+ *
+ * Avoid using this method in any context where there already
+ * is a JsonGeneratorFactory instance in scope. For those scenarios
+ * use the constructor that accepts a JsonGeneratorFactory instead.
+ *
+ * @param value the JsonValue to be serialized as json text
+ * @param max the maximum length of the serialized json text
+ * @return a potentially truncated json text
+ */
+ public static String of(final JsonValue value, final int max) {
+ return new Snippet(max).of(value);
+ }
+
+ /**
+ * There are several buffers involved in the creation of a json string.
+ * This class carefully manages them all.
+ *
+ * JsonGeneratorImpl with a 64k buffer (by default)
+ * ObjectStreamWriter with an 8k buffer
+ * SnippetOutputStream with a buffer of maxSnippetLength
+ *
+ * As we create json via calling the JsonGenerator it is critical we
+ * flush the work in progress all the way through these buffers and into
+ * the final SnippetOutputStream buffer.
+ *
+ * If we do not, we risk creating up to 64k of json when we may only
+ * need 50 bytes. We could potentially optimize this code so the
+ * buffer held by JsonGeneratorImpl is also the maxSnippetLength.
+ */
+ class Buffer implements Closeable {
+ private final JsonGenerator generator;
+ private final SnippetOutputStream snippet;
+
+ private Buffer() {
+ this.snippet = new SnippetOutputStream(max);
+ this.generator = generatorFactory.createGenerator(snippet);
+ }
+
+ private void write(final JsonValue value) {
+ if (snippet.terminate()) {
+ return;
+ }
+
+ switch (value.getValueType()) {
+ case ARRAY: {
+ write(value.asJsonArray());
+ break;
+ }
+ case OBJECT: {
+ write(value.asJsonObject());
+ break;
+ }
+ default: {
+ generator.write(value);
+ generator.flush();
+ }
+ }
+ }
+
+ private void write(final JsonArray array) {
+ if (snippet.terminate()) {
+ return;
+ }
+
+ if (array.isEmpty()) {
+ generator.write(array);
+ generator.flush();
+ return;
+ }
+
+ generator.writeStartArray();
+ generator.flush();
Review Comment:
Marking as resolved based on explanations on the list
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@johnzon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [johnzon] rmannibucau commented on a diff in pull request #84: Show a chunk of json when mapping the object fails
Posted by GitBox <gi...@apache.org>.
rmannibucau commented on code in PR #84:
URL: https://github.com/apache/johnzon/pull/84#discussion_r860583242
##########
johnzon-core/src/main/java/org/apache/johnzon/core/Buffered.java:
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.johnzon.core;
+
+/**
+ * A <tt>Buffered</tt> is a source or destination of data that is buffered
+ * before writing or reading. The bufferSize method allows all participants
+ * in the underlying stream to align on this buffer size for optimization.
+ *
+ * This interface is designed in the spirit of {@code java.io.Flushable} and
+ * {@code java.io.Closeable}
+ *
+ * @since 1.2.17
+ */
+public interface Buffered {
Review Comment:
Oh I see where you coming from. I was more seeing it as:
johnzon: do you want to be buffered?
writer: yes, I need a custom buffer.
johnzon: Ok, what's your buffer requested size?
writer: It's 50.
johnzon: great, I'll write to you 50 bytes at a time
let me try to PR on top of your PR but this is not the critical point, was more curious ;)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@johnzon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [johnzon] dblevins commented on pull request #84: Show a chunk of json when mapping the object fails
Posted by GitBox <gi...@apache.org>.
dblevins commented on PR #84:
URL: https://github.com/apache/johnzon/pull/84#issuecomment-1111876449
Who hooo!! :) That feels great! :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@johnzon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [johnzon] dblevins commented on a diff in pull request #84: Show a chunk of json when mapping the object fails
Posted by GitBox <gi...@apache.org>.
dblevins commented on code in PR #84:
URL: https://github.com/apache/johnzon/pull/84#discussion_r860563179
##########
johnzon-core/src/main/java/org/apache/johnzon/core/Buffered.java:
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.johnzon.core;
+
+/**
+ * A <tt>Buffered</tt> is a source or destination of data that is buffered
+ * before writing or reading. The bufferSize method allows all participants
+ * in the underlying stream to align on this buffer size for optimization.
+ *
+ * This interface is designed in the spirit of {@code java.io.Flushable} and
+ * {@code java.io.Closeable}
+ *
+ * @since 1.2.17
+ */
+public interface Buffered {
Review Comment:
I personally like `Buffered` as it is telling Johnzon a fact, "I am buffered". There is a ByteArrayOutputStream inside the supplied SnippetWriter that Johnzon doesn't control. Johnzon can look at the writers and outputstreams it is given and use the interface to ask, "are you buffered?" and if they are Johnzon can optimize its own buffering to match their buffer size. So to me it makes sense as there's a conversation there:
johnzon: are you buffered?
writer: yes, I'm buffered.
johnzon: Ok, what's your buffer size?
writer: It's 50.
johnzon: great, I'll write to you 50 bytes at a time
That said, if you'd like to rename it after the merge I'm ok with that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@johnzon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [johnzon] dblevins commented on pull request #84: Show a chunk of json when mapping the object fails
Posted by GitBox <gi...@apache.org>.
dblevins commented on PR #84:
URL: https://github.com/apache/johnzon/pull/84#issuecomment-1111871092
@rmannibucau happy for you to push directly to this repo so I can feel happy seeing a purple "merged" symbol on this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@johnzon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [johnzon] dblevins commented on pull request #84: Show a chunk of json when mapping the object fails
Posted by GitBox <gi...@apache.org>.
dblevins commented on PR #84:
URL: https://github.com/apache/johnzon/pull/84#issuecomment-1107906383
You're right -- I missed the code for JohnzonBuilder and the wildcard builder. I went ahead and added it along with a test case to verify the `johnzon.snippetMaxLength` parameter works.
The supplied `JsonGeneratorFactory` is getting used. I suspect the feedback is for the second constructor that creates a JsonGeneratorFactory when one isn't available. I updated it to construct `JsonGeneratorFactoryImpl` directly and documented it is not the preferred constructor. Hopefully that addresses the feedback.
The `close` call was there due to buffering that would lead to an empty string. Those calls are documented as idempotent and safe to call more than once, so my thinking was "close is better than flush." I dug into that a bit more and found the buffering was causing more significant issues and hiding some bugs. The short version: the 64k buffer in JsonGenerator was causing the whole document to be serialized despite our `isComplete()` calls. The SnippetOutputStream would never see any bytes until the very end, so ultimately this code was no better than the `toString()` version I originally submitted.
I've significantly amped up the testing, reworked the code and all is looking good -- famous last words. Give it a good look and let me know if anything looks off.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@johnzon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [johnzon] dblevins commented on pull request #84: Show a chunk of json when mapping the object fails
Posted by GitBox <gi...@apache.org>.
dblevins commented on PR #84:
URL: https://github.com/apache/johnzon/pull/84#issuecomment-1105388280
I've updated the code to remove the casting. On formatting, I'd need some pointers on what should be updated as the checkstyle passed.
There is a call to `generator.close()` in the `get()` method in the Snippet class. If you're thinking of something else, let me know and I can adjust.
FYI, none of the existing `toString()` methods close their generators.
- https://github.com/apache/johnzon/blob/v1.2.16/johnzon-core/src/main/java/org/apache/johnzon/core/JsonObjectImpl.java#L145-L156
- https://github.com/apache/johnzon/blob/v1.2.16/johnzon-core/src/main/java/org/apache/johnzon/core/JsonArrayImpl.java#L165-L176
I can send a separate PR for that if you like.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@johnzon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [johnzon] dblevins commented on pull request #84: Show a chunk of json when mapping the object fails
Posted by GitBox <gi...@apache.org>.
dblevins commented on PR #84:
URL: https://github.com/apache/johnzon/pull/84#issuecomment-1107921537
Can you check the code again? The tests don't use the second constructor and close is called only once.
I'm not convinced trimming things like null to things like `n...` which is the same number of characters is a good idea. Do you have a scenario in mind where a user would actually find that more useful than seeing `null`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@johnzon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [johnzon] rmannibucau commented on a diff in pull request #84: Show a chunk of json when mapping the object fails
Posted by GitBox <gi...@apache.org>.
rmannibucau commented on code in PR #84:
URL: https://github.com/apache/johnzon/pull/84#discussion_r858952719
##########
johnzon-core/src/main/java/org/apache/johnzon/core/Snippet.java:
##########
@@ -228,172 +232,181 @@ private String get() {
public void close() {
generator.close();
}
- }
-
- /**
- * Specialized OutputStream with three internal states:
- * Writing, Completed, Truncated.
- *
- * When there is still space left for more json, the
- * state will be Writing
- *
- * If the last write brought is exactly to the end of
- * the max length, the state will be Completed.
- *
- * If the last write brought us over the max length, the
- * state will be Truncated.
- */
- static class SnippetOutputStream extends OutputStream {
-
- private final ByteArrayOutputStream buffer;
- private OutputStream mode;
-
- public SnippetOutputStream(final int max) {
- this.buffer = new ByteArrayOutputStream(Math.min(max, 8192));
- this.mode = new Writing(max, buffer);
- }
-
- public String get() {
- return buffer.toString();
- }
/**
- * Calling this method implies the need to continue
- * writing and a question on if that is ok.
+ * Specialized Writer with three internal states:
+ * Writing, Completed, Truncated.
*
- * It impacts internal state in the same way as
- * calling a write method.
+ * When there is still space left for more json, the
+ * state will be Writing
*
- * @return true if no more writes are possible
+ * If the last write brought is exactly to the end of
+ * the max length, the state will be Completed.
+ *
+ * If the last write brought us over the max length, the
+ * state will be Truncated.
*/
- public boolean terminate() {
- if (mode instanceof Truncated) {
- return true;
+ class SnippetWriter extends Writer implements Buffered {
+
+ private final ByteArrayOutputStream buffer;
+ private Mode mode;
+ private Supplier<Integer> bufferSize;
+
+ public SnippetWriter(final int max) {
+ final int size = Math.min(max, 8192);
+
+ this.buffer = new ByteArrayOutputStream(size);
+ this.mode = new Writing(max, new OutputStreamWriter(buffer));
+
+ /*
+ * The first time the buffer size is requested, disable flushing
+ * as we know our requested buffer size will be respected
+ */
+ this.bufferSize = () -> {
+ // disable flushing
+ flush = () -> {
+ };
+ // future calls can just return the size
+ bufferSize = () -> size;
+ return size;
+ };
}
- if (mode instanceof Completed) {
- mode = new Truncated();
- return true;
+ public String get() {
+ return buffer.toString();
}
- return false;
- }
-
- public boolean isTruncated() {
- return mode instanceof Truncated;
- }
-
- @Override
- public void write(final int b) throws IOException {
- mode.write(b);
- }
-
- @Override
- public void write(final byte[] b) throws IOException {
- mode.write(b);
- }
-
- @Override
- public void write(final byte[] b, final int off, final int len) throws IOException {
- mode.write(b, off, len);
- }
+ @Override
+ public int bufferSize() {
+ return bufferSize.get();
+ }
- @Override
- public void flush() throws IOException {
- mode.flush();
- }
+ /**
+ * Calling this method implies the need to continue
+ * writing and a question on if that is ok.
+ *
+ * It impacts internal state in the same way as
+ * calling a write method.
+ *
+ * @return true if no more writes are possible
+ */
+ public boolean terminate() {
+ if (mode instanceof Truncated) {
+ return true;
+ }
- @Override
- public void close() throws IOException {
- mode.close();
- }
+ if (mode instanceof Completed) {
+ mode = new Truncated();
+ return true;
+ }
- class Writing extends OutputStream {
- private final int max;
- private int count;
- private final OutputStream out;
+ return false;
+ }
- public Writing(final int max, final OutputStream out) {
- this.max = max;
- this.out = out;
+ public boolean isTruncated() {
+ return mode instanceof Truncated;
}
@Override
- public void write(final int b) throws IOException {
- if (++count < max) {
- out.write(b);
- } else {
- maxReached(new Truncated());
- }
+ public void write(final char[] cbuf, final int off, final int len) throws IOException {
+ mode.write(cbuf, off, len);
}
@Override
- public void write(final byte[] b) throws IOException {
- write(b, 0, b.length);
+ public void flush() throws IOException {
+ mode.flush();
}
@Override
- public void write(final byte[] b, final int off, final int len) throws IOException {
- final int remaining = max - count;
+ public void close() throws IOException {
+ mode.close();
+ }
- if (remaining <= 0) {
+ abstract class Mode extends Writer {
+ @Override
+ public void flush() throws IOException {
Review Comment:
for consistency can you just `// no-op` or something like that to not have an empty body please? (same for close/write)
##########
johnzon-core/src/main/java/org/apache/johnzon/core/Buffered.java:
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.johnzon.core;
+
+/**
+ * A <tt>Buffered</tt> is a source or destination of data that is buffered
+ * before writing or reading. The bufferSize method allows all participants
+ * in the underlying stream to align on this buffer size for optimization.
+ *
+ * This interface is designed in the spirit of {@code java.io.Flushable} and
+ * {@code java.io.Closeable}
+ *
+ * @since 1.2.17
+ */
+public interface Buffered {
Review Comment:
more a question than anything else: shouldnt it be `Bufferable` or something like that? (not blocking for me but asking before it gets merged)
##########
johnzon-mapper/src/main/java/org/apache/johnzon/mapper/MapperBuilder.java:
##########
@@ -236,7 +238,8 @@ public Mapper build() {
supportEnumContainerDeserialization,
typeLoader, discriminatorMapper, discriminator,
deserializationPredicate, serializationPredicate,
- enumConverterFactory),
+ enumConverterFactory,
+ new Snippet(snippetMaxLength, generatorFactory)),
Review Comment:
we must be able to run with any JSON-P impl so we likely need some indirection there (fine to not be optimum if we don't run with johnzon-core but a generic json-p impl)
##########
johnzon-core/src/main/java/org/apache/johnzon/core/Snippet.java:
##########
@@ -0,0 +1,413 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.johnzon.core;
+
+import javax.json.JsonArray;
+import javax.json.JsonObject;
+import javax.json.JsonValue;
+import javax.json.stream.JsonGenerator;
+import javax.json.stream.JsonGeneratorFactory;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.Writer;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.function.Supplier;
+
+import static org.apache.johnzon.core.JsonGeneratorFactoryImpl.GENERATOR_BUFFER_LENGTH;
+
+/**
+ * Constructs short snippets of serialized JSON text representations of
+ * JsonValue instances in a way that is ideal for error messages.
+ *
+ * Instances of Snippet are thread-safe, reusable and memory-safe. Snippet
+ * serializes only enough of the json to fill the desired snippet size and
+ * is therefore safe to use regardless of the size of the JsonValue.
+ */
+public class Snippet {
+
+ private final int max;
+ private final JsonGeneratorFactory generatorFactory;
+
+ /**
+ * This constructor should be used only in static or other scenarios were
+ * there is no JsonGeneratorFactory instance in scope.
+ *
+ * @param max the maximum length of the serialized json produced via of()
+ */
+ public Snippet(final int max) {
+ this(max, new JsonGeneratorFactoryImpl(new HashMap<String, Object>() {
+ {
+ this.put(GENERATOR_BUFFER_LENGTH, max);
+ }
+ }));
+ }
+
+ /**
+ * This is the preferred approach to using Snippet in any context where
+ * there is an existing JsonGeneratorFactory in scope.
+ *
+ * @param max the maximum length of the serialized json produced via of()
+ * @param generatorFactory the JsonGeneratorFactory created by the user
+ */
+ public Snippet(final int max, final JsonGeneratorFactory generatorFactory) {
+ this.max = max;
+ this.generatorFactory = generatorFactory;
+ }
+
+ /**
+ * Create a serialized json representation of the supplied
+ * JsonValue, truncating the value to the specified max length.
+ * Truncated text appears with a suffix of "..."
+ *
+ * This method is thread safe.
+ *
+ * @param value the JsonValue to be serialized as json text
+ * @return a potentially truncated json text
+ */
+ public String of(final JsonValue value) {
+ try (final Buffer buffer = new Buffer()) {
+ buffer.write(value);
+ return buffer.get();
+ }
+ }
+
+ /**
+ * Create a serialized json representation of the supplied
+ * JsonValue, truncating the value to the specified max length.
+ * Truncated text appears with a suffix of "..."
+ *
+ * This method is thread safe.
+ *
+ * Avoid using this method in any context where there already
+ * is a JsonGeneratorFactory instance in scope. For those scenarios
+ * use the constructor that accepts a JsonGeneratorFactory instead.
+ *
+ * @param value the JsonValue to be serialized as json text
+ * @param max the maximum length of the serialized json text
+ * @return a potentially truncated json text
+ */
+ public static String of(final JsonValue value, final int max) {
+ return new Snippet(max).of(value);
+ }
+
+ /**
+ * There are several buffers involved in the creation of a json string.
+ * This class carefully manages them all.
+ *
+ * JsonGeneratorImpl with a 64k buffer (by default)
+ * ObjectStreamWriter with an 8k buffer
+ * SnippetOutputStream with a buffer of maxSnippetLength
+ *
+ * As we create json via calling the JsonGenerator it is critical we
+ * flush the work in progress all the way through these buffers and into
+ * the final SnippetOutputStream buffer.
+ *
+ * If we do not, we risk creating up to 64k of json when we may only
+ * need 50 bytes. We could potentially optimize this code so the
+ * buffer held by JsonGeneratorImpl is also the maxSnippetLength.
+ */
+ class Buffer implements Closeable {
+ private final JsonGenerator generator;
+ private final SnippetWriter snippet;
+ private Runnable flush;
+
+ private Buffer() {
+ this.snippet = new SnippetWriter(max);
+ this.generator = generatorFactory.createGenerator(snippet);
+ this.flush = generator::flush;
+ }
+
+ private void write(final JsonValue value) {
+ if (terminate()) {
+ return;
+ }
+
+ switch (value.getValueType()) {
+ case ARRAY: {
+ write(value.asJsonArray());
+ break;
+ }
+ case OBJECT: {
+ write(value.asJsonObject());
+ break;
+ }
+ default: {
+ generator.write(value);
+ }
+ }
+ }
+
+ private void write(final JsonArray array) {
+ if (array.isEmpty()) {
+ generator.write(array);
+ return;
+ }
+
+ generator.writeStartArray();
+ for (final JsonValue jsonValue : array) {
+ if (terminate()) {
+ break;
+ }
+ write(jsonValue);
+ }
+ generator.writeEnd();
+ }
+
+ private void write(final JsonObject object) {
+ if (object.isEmpty()) {
+ generator.write(object);
+ return;
+ }
+
+ generator.writeStartObject();
+ for (final Map.Entry<String, JsonValue> entry : object.entrySet()) {
+ if (terminate()) {
+ break;
+ }
+ write(entry.getKey(), entry.getValue());
+ }
+ generator.writeEnd();
+ }
+
+ private void write(final String name, final JsonValue value) {
+ switch (value.getValueType()) {
+ case ARRAY:
+ generator.writeStartArray(name);
+ final JsonArray array = value.asJsonArray();
+ for (final JsonValue jsonValue : array) {
+ if (terminate()) {
+ break;
+ }
+ write(jsonValue);
+ }
+ generator.writeEnd();
+
+ break;
+ case OBJECT:
+ generator.writeStartObject(name);
+ final JsonObject object = value.asJsonObject();
+ for (final Map.Entry<String, JsonValue> keyval : object.entrySet()) {
+ if (terminate()) {
+ break;
+ }
+ write(keyval.getKey(), keyval.getValue());
+ }
+ generator.writeEnd();
+
+ break;
+ default: {
+ generator.write(name, value);
+ }
+ }
+ }
+
+ private boolean terminate() {
+ flush.run();
+ return snippet.terminate();
+ }
+
+ private String get() {
+ generator.flush();
+ return snippet.isTruncated() ? snippet.get() + "..." : snippet.get();
+ }
+
+ @Override
+ public void close() {
+ generator.close();
+ }
+
+ /**
+ * Specialized Writer with three internal states:
+ * Writing, Completed, Truncated.
+ *
+ * When there is still space left for more json, the
+ * state will be Writing
+ *
+ * If the last write brought is exactly to the end of
+ * the max length, the state will be Completed.
+ *
+ * If the last write brought us over the max length, the
+ * state will be Truncated.
+ */
+ class SnippetWriter extends Writer implements Buffered {
Review Comment:
think the move from stream to writer looses the encoding so snippet will be broken, no?
##########
johnzon-core/src/main/java/org/apache/johnzon/core/Snippet.java:
##########
@@ -0,0 +1,435 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.johnzon.core;
+
+import javax.json.JsonArray;
+import javax.json.JsonObject;
+import javax.json.JsonValue;
+import javax.json.stream.JsonGenerator;
+import javax.json.stream.JsonGeneratorFactory;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.UncheckedIOException;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.johnzon.core.JsonGeneratorFactoryImpl.GENERATOR_BUFFER_LENGTH;
+
+/**
+ * Constructs short snippets of serialized JSON text representations of
+ * JsonValue instances in a way that is ideal for error messages.
+ *
+ * Instances of Snippet are thread-safe, reusable and memory-safe. Snippet
+ * serializes only enough of the json to fill the desired snippet size and
+ * is therefore safe to use regardless of the size of the JsonValue.
+ */
+public class Snippet {
+
+ private final int max;
+ private final JsonGeneratorFactory generatorFactory;
+
+ /**
+ * This constructor should be used only in static or other scenarios were
+ * there is no JsonGeneratorFactory instance in scope.
+ *
+ * @param max the maximum length of the serialized json produced via of()
+ */
+ public Snippet(final int max) {
Review Comment:
Can you comment this constructor must not be used in johnzon project please but is only there for convenience for intergators and that using it disable several features of the project?
##########
johnzon-core/src/main/java/org/apache/johnzon/core/Snippet.java:
##########
@@ -0,0 +1,413 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.johnzon.core;
+
+import javax.json.JsonArray;
+import javax.json.JsonObject;
+import javax.json.JsonValue;
+import javax.json.stream.JsonGenerator;
+import javax.json.stream.JsonGeneratorFactory;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.Writer;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.function.Supplier;
+
+import static org.apache.johnzon.core.JsonGeneratorFactoryImpl.GENERATOR_BUFFER_LENGTH;
+
+/**
+ * Constructs short snippets of serialized JSON text representations of
+ * JsonValue instances in a way that is ideal for error messages.
+ *
+ * Instances of Snippet are thread-safe, reusable and memory-safe. Snippet
+ * serializes only enough of the json to fill the desired snippet size and
+ * is therefore safe to use regardless of the size of the JsonValue.
+ */
+public class Snippet {
+
+ private final int max;
+ private final JsonGeneratorFactory generatorFactory;
+
+ /**
+ * This constructor should be used only in static or other scenarios were
+ * there is no JsonGeneratorFactory instance in scope.
+ *
+ * @param max the maximum length of the serialized json produced via of()
+ */
+ public Snippet(final int max) {
+ this(max, new JsonGeneratorFactoryImpl(new HashMap<String, Object>() {
+ {
+ this.put(GENERATOR_BUFFER_LENGTH, max);
+ }
+ }));
+ }
+
+ /**
+ * This is the preferred approach to using Snippet in any context where
+ * there is an existing JsonGeneratorFactory in scope.
+ *
+ * @param max the maximum length of the serialized json produced via of()
+ * @param generatorFactory the JsonGeneratorFactory created by the user
+ */
+ public Snippet(final int max, final JsonGeneratorFactory generatorFactory) {
+ this.max = max;
+ this.generatorFactory = generatorFactory;
+ }
+
+ /**
+ * Create a serialized json representation of the supplied
+ * JsonValue, truncating the value to the specified max length.
+ * Truncated text appears with a suffix of "..."
+ *
+ * This method is thread safe.
+ *
+ * @param value the JsonValue to be serialized as json text
+ * @return a potentially truncated json text
+ */
+ public String of(final JsonValue value) {
+ try (final Buffer buffer = new Buffer()) {
+ buffer.write(value);
+ return buffer.get();
Review Comment:
was expecting that moving get() after the auto-close try block would auto-flush it, isn't it right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@johnzon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [johnzon] rmannibucau commented on pull request #84: Show a chunk of json when mapping the object fails
Posted by GitBox <gi...@apache.org>.
rmannibucau commented on PR #84:
URL: https://github.com/apache/johnzon/pull/84#issuecomment-1102151436
@dblevins guess we can passthrough the input (stream or reader or string) and if in memory we could just use it but I was more thinking about just giving some context iterating over the object and not serializing it as JSON but more a simple toString for a subpart, something in the spirit of `var buffer = new StringBuilder(50); var it = object.entrySet().iterator(); while (buffer.length() < 50 && iterator.hasNext()) { if (buffer.length() > 0) { buffer.append(", "); } var e = iterator.next(); buffer.append(e.getKey()).append(": ").append(e.getValue()); }`
It has the avantage to 1. not go through a real JSON serialization, 2. not serialize the full object when you only need one field (take the example of a json object of 1m with an url, here the url is sufficient to fill the 50 chars) so it will give some context to the error but also enables to keep performances fine when error are repeated (including attack cases ;)).
Hope it makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@johnzon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [johnzon] rmannibucau commented on pull request #84: Show a chunk of json when mapping the object fails
Posted by GitBox <gi...@apache.org>.
rmannibucau commented on PR #84:
URL: https://github.com/apache/johnzon/pull/84#issuecomment-1100743183
Can we try to not use toString which is slow but direct in mem value?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@johnzon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [johnzon] dblevins commented on pull request #84: Show a chunk of json when mapping the object fails
Posted by GitBox <gi...@apache.org>.
dblevins commented on PR #84:
URL: https://github.com/apache/johnzon/pull/84#issuecomment-1102871107
I like it! I'll properly open a JIRA for this and rework the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@johnzon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [johnzon] dblevins commented on pull request #84: Show a chunk of json when mapping the object fails
Posted by GitBox <gi...@apache.org>.
dblevins commented on PR #84:
URL: https://github.com/apache/johnzon/pull/84#issuecomment-1105784841
Ok, all that feedback should be addressed and I must say the result of having bounced this back and forth is really looking great!
Effectively, `Snippet.of(object, max)` is now `new Snippet(max).of(object)`. You can also pass the `JsonGeneratorFactory` in as a constructor. All the state that was in the main `Snippet` class has been moved into an inner-class called `Buffer` that is closable. Since our design is essentially trying to manage memory I wanted to 1) give the dangerous part a fitting name and 2) ensure all that code is in a class no one can reference. The `Buffer` class is only ever created in a try-with-resources, so we have 100% certainty no leaks can possibly happen.
Thank you so very much for the code that handled the internal plumbing. That's the kind of code so painful to figure out when you don't know the codebase / hard to describe even when you do. I greatly appreciate the assist and made it clear in the commit message you are the real author.
Such a truly fun collaboration!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@johnzon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [johnzon] rmannibucau commented on pull request #84: Show a chunk of json when mapping the object fails
Posted by GitBox <gi...@apache.org>.
rmannibucau commented on PR #84:
URL: https://github.com/apache/johnzon/pull/84#issuecomment-1108117862
Ok so the second constructor is never useful since there is always a generator factory in scope so let's drop it.
About null truncation the question is "is a snippet of size < 4 useful" but it can be configured and if so respected (size shouldnt be > its max including the ellipsis if we want to document properly and simply this config IMO).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@johnzon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [johnzon] dblevins commented on a diff in pull request #84: Show a chunk of json when mapping the object fails
Posted by GitBox <gi...@apache.org>.
dblevins commented on code in PR #84:
URL: https://github.com/apache/johnzon/pull/84#discussion_r860583842
##########
johnzon-core/src/main/java/org/apache/johnzon/core/Snippet.java:
##########
@@ -0,0 +1,413 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.johnzon.core;
+
+import javax.json.JsonArray;
+import javax.json.JsonObject;
+import javax.json.JsonValue;
+import javax.json.stream.JsonGenerator;
+import javax.json.stream.JsonGeneratorFactory;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.Writer;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.function.Supplier;
+
+import static org.apache.johnzon.core.JsonGeneratorFactoryImpl.GENERATOR_BUFFER_LENGTH;
+
+/**
+ * Constructs short snippets of serialized JSON text representations of
+ * JsonValue instances in a way that is ideal for error messages.
+ *
+ * Instances of Snippet are thread-safe, reusable and memory-safe. Snippet
+ * serializes only enough of the json to fill the desired snippet size and
+ * is therefore safe to use regardless of the size of the JsonValue.
+ */
+public class Snippet {
+
+ private final int max;
+ private final JsonGeneratorFactory generatorFactory;
+
+ /**
+ * This constructor should be used only in static or other scenarios were
+ * there is no JsonGeneratorFactory instance in scope.
+ *
+ * @param max the maximum length of the serialized json produced via of()
+ */
+ public Snippet(final int max) {
+ this(max, new JsonGeneratorFactoryImpl(new HashMap<String, Object>() {
+ {
+ this.put(GENERATOR_BUFFER_LENGTH, max);
+ }
+ }));
+ }
+
+ /**
+ * This is the preferred approach to using Snippet in any context where
+ * there is an existing JsonGeneratorFactory in scope.
+ *
+ * @param max the maximum length of the serialized json produced via of()
+ * @param generatorFactory the JsonGeneratorFactory created by the user
+ */
+ public Snippet(final int max, final JsonGeneratorFactory generatorFactory) {
+ this.max = max;
+ this.generatorFactory = generatorFactory;
+ }
+
+ /**
+ * Create a serialized json representation of the supplied
+ * JsonValue, truncating the value to the specified max length.
+ * Truncated text appears with a suffix of "..."
+ *
+ * This method is thread safe.
+ *
+ * @param value the JsonValue to be serialized as json text
+ * @return a potentially truncated json text
+ */
+ public String of(final JsonValue value) {
+ try (final Buffer buffer = new Buffer()) {
+ buffer.write(value);
+ return buffer.get();
+ }
+ }
+
+ /**
+ * Create a serialized json representation of the supplied
+ * JsonValue, truncating the value to the specified max length.
+ * Truncated text appears with a suffix of "..."
+ *
+ * This method is thread safe.
+ *
+ * Avoid using this method in any context where there already
+ * is a JsonGeneratorFactory instance in scope. For those scenarios
+ * use the constructor that accepts a JsonGeneratorFactory instead.
+ *
+ * @param value the JsonValue to be serialized as json text
+ * @param max the maximum length of the serialized json text
+ * @return a potentially truncated json text
+ */
+ public static String of(final JsonValue value, final int max) {
+ return new Snippet(max).of(value);
+ }
+
+ /**
+ * There are several buffers involved in the creation of a json string.
+ * This class carefully manages them all.
+ *
+ * JsonGeneratorImpl with a 64k buffer (by default)
+ * ObjectStreamWriter with an 8k buffer
+ * SnippetOutputStream with a buffer of maxSnippetLength
+ *
+ * As we create json via calling the JsonGenerator it is critical we
+ * flush the work in progress all the way through these buffers and into
+ * the final SnippetOutputStream buffer.
+ *
+ * If we do not, we risk creating up to 64k of json when we may only
+ * need 50 bytes. We could potentially optimize this code so the
+ * buffer held by JsonGeneratorImpl is also the maxSnippetLength.
+ */
+ class Buffer implements Closeable {
+ private final JsonGenerator generator;
+ private final SnippetWriter snippet;
+ private Runnable flush;
+
+ private Buffer() {
+ this.snippet = new SnippetWriter(max);
+ this.generator = generatorFactory.createGenerator(snippet);
+ this.flush = generator::flush;
+ }
+
+ private void write(final JsonValue value) {
+ if (terminate()) {
+ return;
+ }
+
+ switch (value.getValueType()) {
+ case ARRAY: {
+ write(value.asJsonArray());
+ break;
+ }
+ case OBJECT: {
+ write(value.asJsonObject());
+ break;
+ }
+ default: {
+ generator.write(value);
+ }
+ }
+ }
+
+ private void write(final JsonArray array) {
+ if (array.isEmpty()) {
+ generator.write(array);
+ return;
+ }
+
+ generator.writeStartArray();
+ for (final JsonValue jsonValue : array) {
+ if (terminate()) {
+ break;
+ }
+ write(jsonValue);
+ }
+ generator.writeEnd();
+ }
+
+ private void write(final JsonObject object) {
+ if (object.isEmpty()) {
+ generator.write(object);
+ return;
+ }
+
+ generator.writeStartObject();
+ for (final Map.Entry<String, JsonValue> entry : object.entrySet()) {
+ if (terminate()) {
+ break;
+ }
+ write(entry.getKey(), entry.getValue());
+ }
+ generator.writeEnd();
+ }
+
+ private void write(final String name, final JsonValue value) {
+ switch (value.getValueType()) {
+ case ARRAY:
+ generator.writeStartArray(name);
+ final JsonArray array = value.asJsonArray();
+ for (final JsonValue jsonValue : array) {
+ if (terminate()) {
+ break;
+ }
+ write(jsonValue);
+ }
+ generator.writeEnd();
+
+ break;
+ case OBJECT:
+ generator.writeStartObject(name);
+ final JsonObject object = value.asJsonObject();
+ for (final Map.Entry<String, JsonValue> keyval : object.entrySet()) {
+ if (terminate()) {
+ break;
+ }
+ write(keyval.getKey(), keyval.getValue());
+ }
+ generator.writeEnd();
+
+ break;
+ default: {
+ generator.write(name, value);
+ }
+ }
+ }
+
+ private boolean terminate() {
+ flush.run();
+ return snippet.terminate();
+ }
+
+ private String get() {
+ generator.flush();
+ return snippet.isTruncated() ? snippet.get() + "..." : snippet.get();
+ }
+
+ @Override
+ public void close() {
+ generator.close();
+ }
+
+ /**
+ * Specialized Writer with three internal states:
+ * Writing, Completed, Truncated.
+ *
+ * When there is still space left for more json, the
+ * state will be Writing
+ *
+ * If the last write brought is exactly to the end of
+ * the max length, the state will be Completed.
+ *
+ * If the last write brought us over the max length, the
+ * state will be Truncated.
+ */
+ class SnippetWriter extends Writer implements Buffered {
Review Comment:
Resolving as this was discussed on the list.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@johnzon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [johnzon] dblevins commented on a diff in pull request #84: Show a chunk of json when mapping the object fails
Posted by GitBox <gi...@apache.org>.
dblevins commented on code in PR #84:
URL: https://github.com/apache/johnzon/pull/84#discussion_r860583440
##########
johnzon-core/src/main/java/org/apache/johnzon/core/Snippet.java:
##########
@@ -0,0 +1,413 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.johnzon.core;
+
+import javax.json.JsonArray;
+import javax.json.JsonObject;
+import javax.json.JsonValue;
+import javax.json.stream.JsonGenerator;
+import javax.json.stream.JsonGeneratorFactory;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.Writer;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.function.Supplier;
+
+import static org.apache.johnzon.core.JsonGeneratorFactoryImpl.GENERATOR_BUFFER_LENGTH;
+
+/**
+ * Constructs short snippets of serialized JSON text representations of
+ * JsonValue instances in a way that is ideal for error messages.
+ *
+ * Instances of Snippet are thread-safe, reusable and memory-safe. Snippet
+ * serializes only enough of the json to fill the desired snippet size and
+ * is therefore safe to use regardless of the size of the JsonValue.
+ */
+public class Snippet {
+
+ private final int max;
+ private final JsonGeneratorFactory generatorFactory;
+
+ /**
+ * This constructor should be used only in static or other scenarios were
+ * there is no JsonGeneratorFactory instance in scope.
+ *
+ * @param max the maximum length of the serialized json produced via of()
+ */
+ public Snippet(final int max) {
+ this(max, new JsonGeneratorFactoryImpl(new HashMap<String, Object>() {
+ {
+ this.put(GENERATOR_BUFFER_LENGTH, max);
+ }
+ }));
+ }
+
+ /**
+ * This is the preferred approach to using Snippet in any context where
+ * there is an existing JsonGeneratorFactory in scope.
+ *
+ * @param max the maximum length of the serialized json produced via of()
+ * @param generatorFactory the JsonGeneratorFactory created by the user
+ */
+ public Snippet(final int max, final JsonGeneratorFactory generatorFactory) {
+ this.max = max;
+ this.generatorFactory = generatorFactory;
+ }
+
+ /**
+ * Create a serialized json representation of the supplied
+ * JsonValue, truncating the value to the specified max length.
+ * Truncated text appears with a suffix of "..."
+ *
+ * This method is thread safe.
+ *
+ * @param value the JsonValue to be serialized as json text
+ * @return a potentially truncated json text
+ */
+ public String of(final JsonValue value) {
+ try (final Buffer buffer = new Buffer()) {
+ buffer.write(value);
+ return buffer.get();
Review Comment:
I prefer to have the buffer instance constructed inside the try-with-resources as I like the assurance there can be 100% certain there can be no possible leak. If we don't like flush, we could call close which is documented to idempotent and safe to call more than once. Johnzon properly implements idempotent closes so it is safe. If a JSON-P impl doesn't properly implement close and fails, that'd be their bug IMO.
That said, I'm ok if you'd like to adjust it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@johnzon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [johnzon] dblevins commented on pull request #84: Show a chunk of json when mapping the object fails
Posted by GitBox <gi...@apache.org>.
dblevins commented on PR #84:
URL: https://github.com/apache/johnzon/pull/84#issuecomment-1111872838
@rmannibucau alternatively, you could merge it and then do a PR on top of master. Personally, I don't think anything in this PR is too dangerous to merge at this point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@johnzon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [johnzon] rmannibucau commented on pull request #84: Show a chunk of json when mapping the object fails
Posted by GitBox <gi...@apache.org>.
rmannibucau commented on PR #84:
URL: https://github.com/apache/johnzon/pull/84#issuecomment-1107911497
Second constructor is only used in test this is why i would drop it from main code since we will never want to allow its usage.
The max setting is also wrong due to the usage we do I think (this is not output buffer size but token one).
For close we must find a way to call it once and it should be fine with autocloseable so I suspect buffer was buggy.
Last thing is the bug for primitive we should fix since truncation should be respected there too, no passthrough IMHO ;).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@johnzon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [johnzon] dblevins commented on a diff in pull request #84: Show a chunk of json when mapping the object fails
Posted by GitBox <gi...@apache.org>.
dblevins commented on code in PR #84:
URL: https://github.com/apache/johnzon/pull/84#discussion_r858050906
##########
johnzon-core/src/main/java/org/apache/johnzon/core/Snippet.java:
##########
@@ -0,0 +1,435 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.johnzon.core;
+
+import javax.json.JsonArray;
+import javax.json.JsonObject;
+import javax.json.JsonValue;
+import javax.json.stream.JsonGenerator;
+import javax.json.stream.JsonGeneratorFactory;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.UncheckedIOException;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.johnzon.core.JsonGeneratorFactoryImpl.GENERATOR_BUFFER_LENGTH;
+
+/**
+ * Constructs short snippets of serialized JSON text representations of
+ * JsonValue instances in a way that is ideal for error messages.
+ *
+ * Instances of Snippet are thread-safe, reusable and memory-safe. Snippet
+ * serializes only enough of the json to fill the desired snippet size and
+ * is therefore safe to use regardless of the size of the JsonValue.
+ */
+public class Snippet {
+
+ private final int max;
+ private final JsonGeneratorFactory generatorFactory;
+
+ /**
+ * This constructor should be used only in static or other scenarios were
+ * there is no JsonGeneratorFactory instance in scope.
+ *
+ * @param max the maximum length of the serialized json produced via of()
+ */
+ public Snippet(final int max) {
+ this(max, new JsonGeneratorFactoryImpl(new HashMap<String, Object>() {
+ {
+ this.put(GENERATOR_BUFFER_LENGTH, max);
+ }
+ }));
+ }
+
+ /**
+ * This is the preferred approach to using Snippet in any context where
+ * there is an existing JsonGeneratorFactory in scope.
+ *
+ * @param max the maximum length of the serialized json produced via of()
+ * @param generatorFactory the JsonGeneratorFactory created by the user
+ */
+ public Snippet(final int max, final JsonGeneratorFactory generatorFactory) {
+ this.max = max;
+ this.generatorFactory = generatorFactory;
+ }
+
+ /**
+ * Create a serialized json representation of the supplied
+ * JsonValue, truncating the value to the specified max length.
+ * Truncated text appears with a suffix of "..."
+ *
+ * This method is thread safe.
+ *
+ * @param value the JsonValue to be serialized as json text
+ * @return a potentially truncated json text
+ */
+ public String of(final JsonValue value) {
+ switch (value.getValueType()) {
+ case TRUE: return "true";
+ case FALSE: return "false";
+ case NULL: return "null";
+ default: {
+ try (final Buffer buffer = new Buffer()) {
+ buffer.write(value);
+ return buffer.get();
+ }
+ }
+ }
+ }
+
+ /**
+ * Create a serialized json representation of the supplied
+ * JsonValue, truncating the value to the specified max length.
+ * Truncated text appears with a suffix of "..."
+ *
+ * This method is thread safe.
+ *
+ * Avoid using this method in any context where there already
+ * is a JsonGeneratorFactory instance in scope. For those scenarios
+ * use the constructor that accepts a JsonGeneratorFactory instead.
+ *
+ * @param value the JsonValue to be serialized as json text
+ * @param max the maximum length of the serialized json text
+ * @return a potentially truncated json text
+ */
+ public static String of(final JsonValue value, final int max) {
+ return new Snippet(max).of(value);
+ }
+
+ /**
+ * There are several buffers involved in the creation of a json string.
+ * This class carefully manages them all.
+ *
+ * JsonGeneratorImpl with a 64k buffer (by default)
+ * ObjectStreamWriter with an 8k buffer
+ * SnippetOutputStream with a buffer of maxSnippetLength
+ *
+ * As we create json via calling the JsonGenerator it is critical we
+ * flush the work in progress all the way through these buffers and into
+ * the final SnippetOutputStream buffer.
+ *
+ * If we do not, we risk creating up to 64k of json when we may only
+ * need 50 bytes. We could potentially optimize this code so the
+ * buffer held by JsonGeneratorImpl is also the maxSnippetLength.
+ */
+ class Buffer implements Closeable {
+ private final JsonGenerator generator;
+ private final SnippetOutputStream snippet;
+
+ private Buffer() {
+ this.snippet = new SnippetOutputStream(max);
+ this.generator = generatorFactory.createGenerator(snippet);
+ }
+
+ private void write(final JsonValue value) {
+ if (snippet.terminate()) {
+ return;
+ }
+
+ switch (value.getValueType()) {
+ case ARRAY: {
+ write(value.asJsonArray());
+ break;
+ }
+ case OBJECT: {
+ write(value.asJsonObject());
+ break;
+ }
+ default: {
+ generator.write(value);
+ generator.flush();
+ }
+ }
+ }
+
+ private void write(final JsonArray array) {
+ if (snippet.terminate()) {
+ return;
+ }
+
+ if (array.isEmpty()) {
+ generator.write(array);
+ generator.flush();
+ return;
+ }
+
+ generator.writeStartArray();
+ generator.flush();
+ for (final JsonValue jsonValue : array) {
+ if (snippet.terminate()) {
+ break;
+ }
+ write(jsonValue);
+ }
+ generator.writeEnd();
+ generator.flush();
+ }
+
+ private void write(final JsonObject object) {
+ if (snippet.terminate()) {
+ return;
+ }
+
+ if (object.isEmpty()) {
+ generator.write(object);
+ generator.flush();
+ return;
+ }
+
+ generator.writeStartObject();
+ generator.flush();
+ for (final Map.Entry<String, JsonValue> entry : object.entrySet()) {
+ if (snippet.terminate()) {
+ break;
+ }
+ write(entry.getKey(), entry.getValue());
+ }
+ generator.writeEnd();
+ generator.flush();
+ }
+
+ private void write(final String name, final JsonValue value) {
+ if (snippet.terminate()) {
+ return;
+ }
+
+ switch (value.getValueType()) {
+ case ARRAY:
+ generator.writeStartArray(name);
+ generator.flush();
+ final JsonArray array = value.asJsonArray();
+ for (final JsonValue jsonValue : array) {
+ if (snippet.terminate()) {
+ break;
+ }
+ write(jsonValue);
+ }
+ generator.writeEnd();
+ generator.flush();
+
+ break;
+ case OBJECT:
+ generator.writeStartObject(name);
+ generator.flush();
+ final JsonObject object = value.asJsonObject();
+ for (final Map.Entry<String, JsonValue> keyval : object.entrySet()) {
+ if (snippet.terminate()) {
+ break;
+ }
+ write(keyval.getKey(), keyval.getValue());
+ }
+ generator.writeEnd();
+ generator.flush();
+
+ break;
+ default: {
+ generator.write(name, value);
+ generator.flush();
+ }
+ }
+ }
+
+ private String get() {
+ generator.flush();
+ return snippet.isTruncated() ? snippet.get() + "..." : snippet.get();
+ }
+
+ @Override
+ public void close() {
+ generator.close();
+ }
+ }
+
+ /**
+ * Specialized OutputStream with three internal states:
+ * Writing, Completed, Truncated.
+ *
+ * When there is still space left for more json, the
+ * state will be Writing
+ *
+ * If the last write brought is exactly to the end of
+ * the max length, the state will be Completed.
+ *
+ * If the last write brought us over the max length, the
+ * state will be Truncated.
+ */
+ static class SnippetOutputStream extends OutputStream {
+
+ private final ByteArrayOutputStream buffer;
+ private OutputStream mode;
+
+ public SnippetOutputStream(final int max) {
+ this.buffer = new ByteArrayOutputStream(Math.min(max, 8192));
+ this.mode = new Writing(max, buffer);
+ }
+
+ public String get() {
+ return buffer.toString();
+ }
+
+ /**
+ * Calling this method implies the need to continue
+ * writing and a question on if that is ok.
+ *
+ * It impacts internal state in the same way as
+ * calling a write method.
+ *
+ * @return true if no more writes are possible
+ */
+ public boolean terminate() {
Review Comment:
Took a lot of renaming till I found something just right :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@johnzon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [johnzon] dblevins commented on a diff in pull request #84: Show a chunk of json when mapping the object fails
Posted by GitBox <gi...@apache.org>.
dblevins commented on code in PR #84:
URL: https://github.com/apache/johnzon/pull/84#discussion_r858242675
##########
johnzon-core/src/main/java/org/apache/johnzon/core/Snippet.java:
##########
@@ -0,0 +1,435 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.johnzon.core;
+
+import javax.json.JsonArray;
+import javax.json.JsonObject;
+import javax.json.JsonValue;
+import javax.json.stream.JsonGenerator;
+import javax.json.stream.JsonGeneratorFactory;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.UncheckedIOException;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.johnzon.core.JsonGeneratorFactoryImpl.GENERATOR_BUFFER_LENGTH;
+
+/**
+ * Constructs short snippets of serialized JSON text representations of
+ * JsonValue instances in a way that is ideal for error messages.
+ *
+ * Instances of Snippet are thread-safe, reusable and memory-safe. Snippet
+ * serializes only enough of the json to fill the desired snippet size and
+ * is therefore safe to use regardless of the size of the JsonValue.
+ */
+public class Snippet {
+
+ private final int max;
+ private final JsonGeneratorFactory generatorFactory;
+
+ /**
+ * This constructor should be used only in static or other scenarios were
+ * there is no JsonGeneratorFactory instance in scope.
+ *
+ * @param max the maximum length of the serialized json produced via of()
+ */
+ public Snippet(final int max) {
+ this(max, new JsonGeneratorFactoryImpl(new HashMap<String, Object>() {
+ {
+ this.put(GENERATOR_BUFFER_LENGTH, max);
+ }
+ }));
+ }
+
+ /**
+ * This is the preferred approach to using Snippet in any context where
+ * there is an existing JsonGeneratorFactory in scope.
+ *
+ * @param max the maximum length of the serialized json produced via of()
+ * @param generatorFactory the JsonGeneratorFactory created by the user
+ */
+ public Snippet(final int max, final JsonGeneratorFactory generatorFactory) {
+ this.max = max;
+ this.generatorFactory = generatorFactory;
+ }
+
+ /**
+ * Create a serialized json representation of the supplied
+ * JsonValue, truncating the value to the specified max length.
+ * Truncated text appears with a suffix of "..."
+ *
+ * This method is thread safe.
+ *
+ * @param value the JsonValue to be serialized as json text
+ * @return a potentially truncated json text
+ */
+ public String of(final JsonValue value) {
+ switch (value.getValueType()) {
+ case TRUE: return "true";
+ case FALSE: return "false";
+ case NULL: return "null";
+ default: {
+ try (final Buffer buffer = new Buffer()) {
+ buffer.write(value);
+ return buffer.get();
+ }
+ }
+ }
+ }
+
+ /**
+ * Create a serialized json representation of the supplied
+ * JsonValue, truncating the value to the specified max length.
+ * Truncated text appears with a suffix of "..."
+ *
+ * This method is thread safe.
+ *
+ * Avoid using this method in any context where there already
+ * is a JsonGeneratorFactory instance in scope. For those scenarios
+ * use the constructor that accepts a JsonGeneratorFactory instead.
+ *
+ * @param value the JsonValue to be serialized as json text
+ * @param max the maximum length of the serialized json text
+ * @return a potentially truncated json text
+ */
+ public static String of(final JsonValue value, final int max) {
+ return new Snippet(max).of(value);
+ }
+
+ /**
+ * There are several buffers involved in the creation of a json string.
+ * This class carefully manages them all.
+ *
+ * JsonGeneratorImpl with a 64k buffer (by default)
+ * ObjectStreamWriter with an 8k buffer
+ * SnippetOutputStream with a buffer of maxSnippetLength
+ *
+ * As we create json via calling the JsonGenerator it is critical we
+ * flush the work in progress all the way through these buffers and into
+ * the final SnippetOutputStream buffer.
+ *
+ * If we do not, we risk creating up to 64k of json when we may only
+ * need 50 bytes. We could potentially optimize this code so the
+ * buffer held by JsonGeneratorImpl is also the maxSnippetLength.
+ */
+ class Buffer implements Closeable {
+ private final JsonGenerator generator;
+ private final SnippetOutputStream snippet;
+
+ private Buffer() {
+ this.snippet = new SnippetOutputStream(max);
+ this.generator = generatorFactory.createGenerator(snippet);
+ }
+
+ private void write(final JsonValue value) {
+ if (snippet.terminate()) {
+ return;
+ }
+
+ switch (value.getValueType()) {
+ case ARRAY: {
+ write(value.asJsonArray());
+ break;
+ }
+ case OBJECT: {
+ write(value.asJsonObject());
+ break;
+ }
+ default: {
+ generator.write(value);
+ generator.flush();
+ }
+ }
+ }
+
+ private void write(final JsonArray array) {
+ if (snippet.terminate()) {
+ return;
+ }
+
+ if (array.isEmpty()) {
+ generator.write(array);
+ generator.flush();
+ return;
+ }
+
+ generator.writeStartArray();
+ generator.flush();
+ for (final JsonValue jsonValue : array) {
+ if (snippet.terminate()) {
+ break;
+ }
+ write(jsonValue);
+ }
+ generator.writeEnd();
+ generator.flush();
+ }
+
+ private void write(final JsonObject object) {
+ if (snippet.terminate()) {
+ return;
+ }
+
+ if (object.isEmpty()) {
+ generator.write(object);
+ generator.flush();
+ return;
+ }
+
+ generator.writeStartObject();
+ generator.flush();
+ for (final Map.Entry<String, JsonValue> entry : object.entrySet()) {
+ if (snippet.terminate()) {
+ break;
+ }
+ write(entry.getKey(), entry.getValue());
+ }
+ generator.writeEnd();
+ generator.flush();
+ }
+
+ private void write(final String name, final JsonValue value) {
+ if (snippet.terminate()) {
+ return;
+ }
+
+ switch (value.getValueType()) {
+ case ARRAY:
+ generator.writeStartArray(name);
+ generator.flush();
+ final JsonArray array = value.asJsonArray();
+ for (final JsonValue jsonValue : array) {
+ if (snippet.terminate()) {
+ break;
+ }
+ write(jsonValue);
+ }
+ generator.writeEnd();
+ generator.flush();
+
+ break;
+ case OBJECT:
+ generator.writeStartObject(name);
+ generator.flush();
+ final JsonObject object = value.asJsonObject();
+ for (final Map.Entry<String, JsonValue> keyval : object.entrySet()) {
+ if (snippet.terminate()) {
+ break;
+ }
+ write(keyval.getKey(), keyval.getValue());
+ }
+ generator.writeEnd();
+ generator.flush();
+
+ break;
+ default: {
+ generator.write(name, value);
+ generator.flush();
+ }
+ }
+ }
+
+ private String get() {
+ generator.flush();
+ return snippet.isTruncated() ? snippet.get() + "..." : snippet.get();
+ }
+
+ @Override
+ public void close() {
+ generator.close();
+ }
+ }
+
+ /**
+ * Specialized OutputStream with three internal states:
+ * Writing, Completed, Truncated.
+ *
+ * When there is still space left for more json, the
+ * state will be Writing
+ *
+ * If the last write brought is exactly to the end of
+ * the max length, the state will be Completed.
+ *
+ * If the last write brought us over the max length, the
+ * state will be Truncated.
+ */
+ static class SnippetOutputStream extends OutputStream {
+
+ private final ByteArrayOutputStream buffer;
+ private OutputStream mode;
+
+ public SnippetOutputStream(final int max) {
+ this.buffer = new ByteArrayOutputStream(Math.min(max, 8192));
+ this.mode = new Writing(max, buffer);
+ }
+
+ public String get() {
+ return buffer.toString();
+ }
+
+ /**
+ * Calling this method implies the need to continue
+ * writing and a question on if that is ok.
+ *
+ * It impacts internal state in the same way as
+ * calling a write method.
+ *
+ * @return true if no more writes are possible
+ */
+ public boolean terminate() {
+ if (mode instanceof Truncated) {
+ return true;
+ }
+
+ if (mode instanceof Completed) {
+ mode = new Truncated();
+ return true;
+ }
+
+ return false;
+ }
+
+ public boolean isTruncated() {
+ return mode instanceof Truncated;
+ }
+
+ @Override
+ public void write(final int b) throws IOException {
+ mode.write(b);
+ }
+
+ @Override
+ public void write(final byte[] b) throws IOException {
+ mode.write(b);
+ }
+
+ @Override
+ public void write(final byte[] b, final int off, final int len) throws IOException {
+ mode.write(b, off, len);
+ }
+
+ @Override
+ public void flush() throws IOException {
+ mode.flush();
+ }
+
+ @Override
+ public void close() throws IOException {
+ mode.close();
+ }
+
+ public void print(final String string) {
Review Comment:
Dropped! Was previously used for serializing the constants and forgot to clean that up. Good catch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@johnzon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [johnzon] dblevins commented on pull request #84: Show a chunk of json when mapping the object fails
Posted by GitBox <gi...@apache.org>.
dblevins commented on PR #84:
URL: https://github.com/apache/johnzon/pull/84#issuecomment-1111861106
Ok, I've made most the changes. I'm happy for there to be other changes and have provided access to my fork so you have the option to do that before anything is merged. I offer that in the spirit of speed as I only have a few more days to do all the message work and haven't even started yet. Feel free to make any change you need to feel confident merging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@johnzon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [johnzon] dblevins commented on a diff in pull request #84: Show a chunk of json when mapping the object fails
Posted by GitBox <gi...@apache.org>.
dblevins commented on code in PR #84:
URL: https://github.com/apache/johnzon/pull/84#discussion_r860593995
##########
johnzon-core/src/main/java/org/apache/johnzon/core/Buffered.java:
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.johnzon.core;
+
+/**
+ * A <tt>Buffered</tt> is a source or destination of data that is buffered
+ * before writing or reading. The bufferSize method allows all participants
+ * in the underlying stream to align on this buffer size for optimization.
+ *
+ * This interface is designed in the spirit of {@code java.io.Flushable} and
+ * {@code java.io.Closeable}
+ *
+ * @since 1.2.17
+ */
+public interface Buffered {
Review Comment:
Cool. As I said to me it makes sense. We could easily get a Writer that writes to a file and has a buffer in it optimized for the page size of disk. This interface would be a great way for them to tell that fact to Johnzon so it can participate in the optimization. Ideally, whatever interface name use use it is strongly tied to buffering and not generic and therefore has many other purposes in the future. If we did that, then it would get complicated for someone to say, "Yes, I am buffered, but I am not those other things."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@johnzon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [johnzon] rmannibucau commented on pull request #84: Show a chunk of json when mapping the object fails
Posted by GitBox <gi...@apache.org>.
rmannibucau commented on PR #84:
URL: https://github.com/apache/johnzon/pull/84#issuecomment-1111874919
ok let's do it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@johnzon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [johnzon] rmannibucau merged pull request #84: Show a chunk of json when mapping the object fails
Posted by GitBox <gi...@apache.org>.
rmannibucau merged PR #84:
URL: https://github.com/apache/johnzon/pull/84
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@johnzon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [johnzon] dblevins commented on a diff in pull request #84: Show a chunk of json when mapping the object fails
Posted by GitBox <gi...@apache.org>.
dblevins commented on code in PR #84:
URL: https://github.com/apache/johnzon/pull/84#discussion_r858241474
##########
johnzon-core/src/main/java/org/apache/johnzon/core/Snippet.java:
##########
@@ -0,0 +1,435 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.johnzon.core;
+
+import javax.json.JsonArray;
+import javax.json.JsonObject;
+import javax.json.JsonValue;
+import javax.json.stream.JsonGenerator;
+import javax.json.stream.JsonGeneratorFactory;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.UncheckedIOException;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.johnzon.core.JsonGeneratorFactoryImpl.GENERATOR_BUFFER_LENGTH;
+
+/**
+ * Constructs short snippets of serialized JSON text representations of
+ * JsonValue instances in a way that is ideal for error messages.
+ *
+ * Instances of Snippet are thread-safe, reusable and memory-safe. Snippet
+ * serializes only enough of the json to fill the desired snippet size and
+ * is therefore safe to use regardless of the size of the JsonValue.
+ */
+public class Snippet {
+
+ private final int max;
+ private final JsonGeneratorFactory generatorFactory;
+
+ /**
+ * This constructor should be used only in static or other scenarios were
+ * there is no JsonGeneratorFactory instance in scope.
+ *
+ * @param max the maximum length of the serialized json produced via of()
+ */
+ public Snippet(final int max) {
+ this(max, new JsonGeneratorFactoryImpl(new HashMap<String, Object>() {
+ {
+ this.put(GENERATOR_BUFFER_LENGTH, max);
+ }
+ }));
+ }
+
+ /**
+ * This is the preferred approach to using Snippet in any context where
+ * there is an existing JsonGeneratorFactory in scope.
+ *
+ * @param max the maximum length of the serialized json produced via of()
+ * @param generatorFactory the JsonGeneratorFactory created by the user
+ */
+ public Snippet(final int max, final JsonGeneratorFactory generatorFactory) {
+ this.max = max;
+ this.generatorFactory = generatorFactory;
+ }
+
+ /**
+ * Create a serialized json representation of the supplied
+ * JsonValue, truncating the value to the specified max length.
+ * Truncated text appears with a suffix of "..."
+ *
+ * This method is thread safe.
+ *
+ * @param value the JsonValue to be serialized as json text
+ * @return a potentially truncated json text
+ */
+ public String of(final JsonValue value) {
+ switch (value.getValueType()) {
+ case TRUE: return "true";
+ case FALSE: return "false";
+ case NULL: return "null";
+ default: {
+ try (final Buffer buffer = new Buffer()) {
+ buffer.write(value);
+ return buffer.get();
+ }
+ }
+ }
+ }
+
+ /**
+ * Create a serialized json representation of the supplied
+ * JsonValue, truncating the value to the specified max length.
+ * Truncated text appears with a suffix of "..."
+ *
+ * This method is thread safe.
+ *
+ * Avoid using this method in any context where there already
+ * is a JsonGeneratorFactory instance in scope. For those scenarios
+ * use the constructor that accepts a JsonGeneratorFactory instead.
+ *
+ * @param value the JsonValue to be serialized as json text
+ * @param max the maximum length of the serialized json text
+ * @return a potentially truncated json text
+ */
+ public static String of(final JsonValue value, final int max) {
+ return new Snippet(max).of(value);
+ }
+
+ /**
+ * There are several buffers involved in the creation of a json string.
+ * This class carefully manages them all.
+ *
+ * JsonGeneratorImpl with a 64k buffer (by default)
+ * ObjectStreamWriter with an 8k buffer
+ * SnippetOutputStream with a buffer of maxSnippetLength
+ *
+ * As we create json via calling the JsonGenerator it is critical we
+ * flush the work in progress all the way through these buffers and into
+ * the final SnippetOutputStream buffer.
+ *
+ * If we do not, we risk creating up to 64k of json when we may only
+ * need 50 bytes. We could potentially optimize this code so the
+ * buffer held by JsonGeneratorImpl is also the maxSnippetLength.
+ */
+ class Buffer implements Closeable {
+ private final JsonGenerator generator;
+ private final SnippetOutputStream snippet;
+
+ private Buffer() {
+ this.snippet = new SnippetOutputStream(max);
+ this.generator = generatorFactory.createGenerator(snippet);
+ }
+
+ private void write(final JsonValue value) {
+ if (snippet.terminate()) {
+ return;
+ }
+
+ switch (value.getValueType()) {
+ case ARRAY: {
+ write(value.asJsonArray());
+ break;
+ }
+ case OBJECT: {
+ write(value.asJsonObject());
+ break;
+ }
+ default: {
+ generator.write(value);
+ generator.flush();
+ }
+ }
+ }
+
+ private void write(final JsonArray array) {
+ if (snippet.terminate()) {
+ return;
+ }
+
+ if (array.isEmpty()) {
+ generator.write(array);
+ generator.flush();
+ return;
+ }
+
+ generator.writeStartArray();
+ generator.flush();
+ for (final JsonValue jsonValue : array) {
+ if (snippet.terminate()) {
+ break;
+ }
+ write(jsonValue);
+ }
+ generator.writeEnd();
+ generator.flush();
+ }
+
+ private void write(final JsonObject object) {
+ if (snippet.terminate()) {
+ return;
+ }
+
+ if (object.isEmpty()) {
+ generator.write(object);
+ generator.flush();
+ return;
+ }
+
+ generator.writeStartObject();
+ generator.flush();
+ for (final Map.Entry<String, JsonValue> entry : object.entrySet()) {
+ if (snippet.terminate()) {
+ break;
+ }
+ write(entry.getKey(), entry.getValue());
+ }
+ generator.writeEnd();
+ generator.flush();
+ }
+
+ private void write(final String name, final JsonValue value) {
+ if (snippet.terminate()) {
+ return;
+ }
+
+ switch (value.getValueType()) {
+ case ARRAY:
+ generator.writeStartArray(name);
+ generator.flush();
+ final JsonArray array = value.asJsonArray();
+ for (final JsonValue jsonValue : array) {
+ if (snippet.terminate()) {
+ break;
+ }
+ write(jsonValue);
+ }
+ generator.writeEnd();
+ generator.flush();
+
+ break;
+ case OBJECT:
+ generator.writeStartObject(name);
+ generator.flush();
+ final JsonObject object = value.asJsonObject();
+ for (final Map.Entry<String, JsonValue> keyval : object.entrySet()) {
+ if (snippet.terminate()) {
+ break;
+ }
+ write(keyval.getKey(), keyval.getValue());
+ }
+ generator.writeEnd();
+ generator.flush();
+
+ break;
+ default: {
+ generator.write(name, value);
+ generator.flush();
+ }
+ }
+ }
+
+ private String get() {
Review Comment:
Flush content covered on the lists.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@johnzon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [johnzon] rmannibucau commented on pull request #84: Show a chunk of json when mapping the object fails
Posted by GitBox <gi...@apache.org>.
rmannibucau commented on PR #84:
URL: https://github.com/apache/johnzon/pull/84#issuecomment-1108819082
@dblevins hmm, what I really dislike in this constructor is the fact I want to totally forbid it to be used inside the project itself for the reason mentionned earlier. I would be happy if you do a johnzon-snippet module exposing such a constructor and using `Json.createGeneratorFactory(<....>)` while it is not in mapper/jsonb modules. Alternative would be to ensure at build with a maven plugin this constructor is never used outside tests with a comment explaining this is not a reliable API for a generic purposes as the mapper/jsonb instance (agree it is simpler when you know you are UTF-8 for ex but it is generally not true from johnzon standpoint). Guess personally I would expose it in `johnzon-json-extras` a bit by default since it is already a generic module for such extensions not having other strong dependencies and it requires almost no work except having a new SnippetBis class delegating to Json and Snippet.
Hope it makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@johnzon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [johnzon] rmannibucau commented on pull request #84: Show a chunk of json when mapping the object fails
Posted by GitBox <gi...@apache.org>.
rmannibucau commented on PR #84:
URL: https://github.com/apache/johnzon/pull/84#issuecomment-1105504570
@dblevins
- about the formatting it is just a matter of not changing the indentation when no change were done ([MappingParserImpl.java](https://github.com/apache/johnzon/pull/84/files#diff-fb94af49bd5359016321309fc37e72958ebf59edc056b71223597162ccd6298e) for ex), we don't have a strong enforcement but "common" PR rule to not change what we don't modify applies when possible (not saying we are super clean but having a style enforcer is worse for contributor from my experience on other projects so we didn't set it up)
- on the new tests you should cose the parser ;) (same, it leaks mem/buffers)
> There is a call to generator.close() in the get() method in the Snippet class. If you're thinking of something else, let me know and I can adjust.
Sure, try/finally (or autoclose), no assumption it will be called...cause it can fail in between.
You should also not create the generator in the constructor (think primitive cases you hardcoded) but when hitting the root object probably.
Another related issue is, I think, we should reuse the json generator factory we already have with its config in the mapper (https://github.com/apache/johnzon/blob/c8e233e2f3c54037b8aa5d5d9bf165b550cc9641/johnzon-mapper/src/main/java/org/apache/johnzon/mapper/Mapper.java#L63) instead of using a banalized one (`Json`/`JsonProviderImpl`).
Long story short the idea would be something along this: https://gist.github.com/rmannibucau/bc400921f17dd12eb86491462071b0e9 .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@johnzon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [johnzon] dblevins commented on pull request #84: Show a chunk of json when mapping the object fails
Posted by GitBox <gi...@apache.org>.
dblevins commented on PR #84:
URL: https://github.com/apache/johnzon/pull/84#issuecomment-1105693384
Thanks for the clarification on the formatting -- I hadn't noticed that code was reformatted. Mixing reformatting and real changes is also on my no-no list :) I'll fix and again squash the commit so the reformatting is not in the history.
I'll fix the try/finally close and then take a closer look at the diff.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@johnzon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [johnzon] dblevins commented on pull request #84: Show a chunk of json when mapping the object fails
Posted by GitBox <gi...@apache.org>.
dblevins commented on PR #84:
URL: https://github.com/apache/johnzon/pull/84#issuecomment-1104586547
All right, I've gone ahead and made this change and squash committed over the previous attempt.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@johnzon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [johnzon] rmannibucau commented on pull request #84: Show a chunk of json when mapping the object fails
Posted by GitBox <gi...@apache.org>.
rmannibucau commented on PR #84:
URL: https://github.com/apache/johnzon/pull/84#issuecomment-1106029130
Hi David,
Looked quickly but seems several changes are lost or unintended like the double call to generator.close, the snippet creation using Json and not the configured JsonGeneratorFactory, the parser close call miss in tests or the fix in test/truncation behavior to respect max config.
Is it intended?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@johnzon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [johnzon] dblevins commented on a diff in pull request #84: Show a chunk of json when mapping the object fails
Posted by GitBox <gi...@apache.org>.
dblevins commented on code in PR #84:
URL: https://github.com/apache/johnzon/pull/84#discussion_r860577951
##########
johnzon-mapper/src/main/java/org/apache/johnzon/mapper/MapperBuilder.java:
##########
@@ -236,7 +238,8 @@ public Mapper build() {
supportEnumContainerDeserialization,
typeLoader, discriminatorMapper, discriminator,
deserializationPredicate, serializationPredicate,
- enumConverterFactory),
+ enumConverterFactory,
+ new Snippet(snippetMaxLength, generatorFactory)),
Review Comment:
I think this might have been a comment for the constructor as it referenced JsonGeneratorFactoryImpl. I updated it back to Json.createGeneratorFactory. If that wasn't what you meant, I sent an invite to the repo so you have access to do any adjustments you'd like.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@johnzon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [johnzon] rmannibucau commented on a diff in pull request #84: Show a chunk of json when mapping the object fails
Posted by GitBox <gi...@apache.org>.
rmannibucau commented on code in PR #84:
URL: https://github.com/apache/johnzon/pull/84#discussion_r860549485
##########
johnzon-core/src/main/java/org/apache/johnzon/core/Buffered.java:
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.johnzon.core;
+
+/**
+ * A <tt>Buffered</tt> is a source or destination of data that is buffered
+ * before writing or reading. The bufferSize method allows all participants
+ * in the underlying stream to align on this buffer size for optimization.
+ *
+ * This interface is designed in the spirit of {@code java.io.Flushable} and
+ * {@code java.io.Closeable}
+ *
+ * @since 1.2.17
+ */
+public interface Buffered {
Review Comment:
Hmm, guess the same reasoning got me asking the question, while not handled by johnzon it is not buffered.
What if we quit that area and call it `StreamDescriptor` which would be more OO (i am an instance of stream descriptor so I can get meta about the stream/writer).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@johnzon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [johnzon] rmannibucau commented on pull request #84: Show a chunk of json when mapping the object fails
Posted by GitBox <gi...@apache.org>.
rmannibucau commented on PR #84:
URL: https://github.com/apache/johnzon/pull/84#issuecomment-1111868567
@dblevins added some code on top of yours at https://github.com/apache/johnzon/pull/88 (sorry for the second PR but github messed up when I PR-ed on your repo not sure why)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@johnzon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [johnzon] dblevins commented on pull request #84: Show a chunk of json when mapping the object fails
Posted by GitBox <gi...@apache.org>.
dblevins commented on PR #84:
URL: https://github.com/apache/johnzon/pull/84#issuecomment-1111877095
Thank you, @rmannibucau !! I really enjoyed the help and collaboration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@johnzon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org