You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "xiangfu0 (via GitHub)" <gi...@apache.org> on 2026/08/12 05:28:37 UTC
[PR] Add experimental Fory JSON functions [pinot]
xiangfu0 opened a new pull request, #19223:
URL: https://github.com/apache/pinot/pull/19223
## What this PR does
Adds opt-in, experimental Apache Fory JSON support without changing the behavior or implementation selected by existing Pinot JSON functions.
- Adds `jsonExtractScalarFory(json, path, resultType[, default])` for query-time typed extraction.
- Adds `jsonPathStringFory`, `jsonPathLongFory`, and `jsonPathDoubleFory` for ingestion transforms.
- Uses a per-thread Fory streaming codec for eligible query-side simple paths.
- Uses a shared dynamic-tree parser plus direct path navigation for ingestion functions.
- Falls back to the existing Jackson/Jayway implementation for unsupported paths and values, precision-sensitive results, BYTES input, deeply nested or malformed documents, runtime/linkage failures, and other unsupported cases.
- Registers the functions with the parser, transform factory, planner fixtures, and UDF snapshot.
- Adds release LICENSE/NOTICE attribution for Fory and code embedded in its published fat JAR.
- Extends the JSON JMH coverage with end-to-end ValueBlock benchmarks.
All four public names are documented as experimental and may change or be removed while the integration is evaluated.
## Why
Fory's generated JSON parser benchmarks are promising, but Pinot's relevant workload is dynamic JSON path extraction rather than fixed-schema POJO binding. This PR provides an isolated surface for evaluating that workload in real Pinot query and ingestion paths while preserving the current implementation as the compatibility fallback.
## Local benchmark results
Exploratory late-field query throughput relative to `jsonExtractScalarFast`:
| JSON document | 1 thread | 8 threads |
| --- | ---: | ---: |
| 700 B | 1.11x | 1.60x |
| 8 KiB | 1.52x | 2.41x |
| 64 KiB | 3.53x | 4.72x |
These are local zero-fork JMH measurements and should be treated as directional. Fory's current released string-skip path trades throughput for allocation: approximately 1.65x, 12x, and 84x the allocation of Fast at 700 B, 8 KiB, and 64 KiB respectively. An upstream zero-allocation fast string-skip API would materially improve this tradeoff.
## Validation
- Fory/common suite: 64 tests, 0 failures.
- Core registry, ingestion evaluator, and typed transform suites: 266 tests, 0 failures.
- `QueryCompilationTest`: 229 tests, 0 failures.
- `QueryRunnerTest`: 135 tests, 0 failures.
- Spotless, license formatting, checkstyle, and license validation passed for:
- `pinot-common`
- `pinot-core`
- `pinot-integration-tests`
- `pinot-perf`
- `pinot-query-planner`
- `pinot-query-runtime`
- `git diff --check` is clean.
The real `JsonPathTest` integration cluster and full binary-distribution build were not run locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add experimental Fory JSON functions [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #19223:
URL: https://github.com/apache/pinot/pull/19223#discussion_r3769293921
##########
pinot-common/src/main/java/org/apache/pinot/common/function/scalar/JsonFunctions.java:
##########
@@ -129,6 +130,25 @@ private static Object fastJsonPath(Object object, String jsonPath, boolean useBi
return jsonPath(object, jsonPath);
}
+ /// Resolves a simple path over a JSON string with Fory's streaming reader. Complex paths, already-parsed inputs,
+ /// container results, and Fory extraction failures use the existing Jayway implementation so the experimental
+ /// functions below retain the current behavior outside Fory's supported envelope.
+ @Nullable
+ private static Object foryJsonPath(Object object, String jsonPath) {
+ SimpleJsonPath simpleJsonPath = SimpleJsonPath.compile(jsonPath);
+ if (!(object instanceof String) || simpleJsonPath == null) {
+ return jsonPath(object, jsonPath);
+ }
Review Comment:
`foryJsonPath()` compiles `SimpleJsonPath` even when the input is not a `String` (e.g., already-parsed Map/List input), then immediately falls back to Jayway. This adds unnecessary work on the fallback path and is inconsistent with `fastJsonPath()` which only compiles for String inputs.
##########
pinot-common/src/main/java/org/apache/pinot/common/function/ForyJsonPathExtractor.java:
##########
@@ -0,0 +1,439 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.function;
+
+import com.fasterxml.jackson.core.StreamReadConstraints;
+import java.util.concurrent.atomic.AtomicBoolean;
+import javax.annotation.Nullable;
+import org.apache.fory.json.ForyJson;
+import org.apache.fory.json.codec.JsonValueCodec;
+import org.apache.fory.json.reader.JsonReader;
+import org.apache.fory.json.reader.Latin1JsonReader;
+import org.apache.fory.json.reader.Utf16JsonReader;
+import org.apache.fory.json.reader.Utf8JsonReader;
+import org.apache.fory.json.writer.StringJsonWriter;
+import org.apache.fory.json.writer.Utf8JsonWriter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/// Shared Fory JSON parser for opt-in JSON-path implementations.
+///
+/// Streaming extraction uses one single-state parser per worker thread to avoid shared-pool contention. Initialization
+/// or runtime linkage failures permanently disable the optional path, allowing callers to fall back to
+/// Jackson/Jayway. Jackson's default scalar token limits are enforced while walking the document, and inputs outside
+/// Fory's safe nesting depth fall back to the reference parser.
+public final class ForyJsonPathExtractor {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ForyJsonPathExtractor.class);
+ private static final StreamReadConstraints JACKSON_CONSTRAINTS = StreamReadConstraints.defaults();
+ private static final ThreadLocal<PathContext> PATH_CONTEXT = ThreadLocal.withInitial(PathContext::new);
+ private static final AtomicBoolean UNAVAILABLE_WARNING_LOGGED = new AtomicBoolean();
+
+ private ForyJsonPathExtractor() {
+ }
+
+ private static final class Holder {
+ private static volatile boolean _streamingAvailable;
+ private static final ThreadLocal<ForyJson> STREAMING_PARSER = ThreadLocal.withInitial(() -> {
+ if (!_streamingAvailable) {
+ throw new IllegalStateException("Fory JSON is unavailable");
+ }
+ ForyJson parser = buildStreamingParser();
+ if (parser == null) {
+ _streamingAvailable = false;
+ throw new IllegalStateException("Fory JSON is unavailable");
+ }
+ return parser;
+ });
+
+ static {
+ ForyJson parser = buildStreamingParser();
+ _streamingAvailable = parser != null;
+ if (parser != null) {
+ STREAMING_PARSER.set(parser);
+ }
+ }
+
+ private Holder() {
+ }
+
+ @Nullable
+ private static ForyJson buildStreamingParser() {
+ try {
+ return ForyJson.builder().withCodegen(false).withAsyncCompilation(false).withConcurrencyLevel(1)
+ .registerCodec(PathResult.class, PathCodec.INSTANCE).build();
+ } catch (RuntimeException | LinkageError e) {
+ logUnavailable(e);
+ return null;
+ }
+ }
+ }
+
+ /// Returns whether the optional Fory runtime initialized successfully.
+ public static boolean isAvailable() {
+ return Holder._streamingAvailable;
+ }
+
+ /// Extracts a simple path with Fory's streaming reader without materializing the complete JSON tree.
+ ///
+ /// Unrelated values are still fully consumed so malformed input and duplicate-key last-wins behavior match the
+ /// reference parser. Jackson's nesting, field-name, string, and number limits are checked while scanning. Callers
+ /// should retry with the reference parser when this method throws.
+ @Nullable
+ public static Object extract(String json, SimpleJsonPath path) {
+ if ((JACKSON_CONSTRAINTS.hasMaxDocumentLength()
+ && json.length() > JACKSON_CONSTRAINTS.getMaxDocumentLength())
+ || (JACKSON_CONSTRAINTS.hasMaxTokenCount() && requiresJacksonFallback(json))) {
+ throw new IllegalArgumentException("JSON document requires Jackson constraint validation");
+ }
Review Comment:
`requiresJacksonFallback(json)` also checks max nesting depth / token sizes, but it is only invoked when `JACKSON_CONSTRAINTS.hasMaxTokenCount()` is true. That makes the nesting-depth and length checks inside `requiresJacksonFallback` effectively unreachable under the default Jackson constraints configuration, and contradicts the class comment about enforcing Jackson constraints while scanning.
##########
pinot-perf/src/main/java/org/apache/pinot/perf/BenchmarkJsonExtractScalarQuery.java:
##########
@@ -0,0 +1,218 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.perf;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import org.apache.pinot.common.request.context.ExpressionContext;
+import org.apache.pinot.common.request.context.LiteralContext;
+import org.apache.pinot.core.common.BlockValSet;
+import org.apache.pinot.core.operator.ColumnContext;
+import org.apache.pinot.core.operator.blocks.ValueBlock;
+import org.apache.pinot.core.operator.transform.TransformResultMetadata;
+import org.apache.pinot.core.operator.transform.function.BaseTransformFunction;
+import org.apache.pinot.core.operator.transform.function.JsonExtractScalarTransformFunction;
+import org.apache.pinot.core.operator.transform.function.LiteralTransformFunction;
+import org.apache.pinot.core.operator.transform.function.TransformFunction;
+import org.apache.pinot.spi.data.FieldSpec.DataType;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.OperationsPerInvocation;
+import org.openjdk.jmh.annotations.OutputTimeUnit;
+import org.openjdk.jmh.annotations.Param;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.Warmup;
+import org.openjdk.jmh.runner.Runner;
+import org.openjdk.jmh.runner.options.Options;
+import org.openjdk.jmh.runner.options.OptionsBuilder;
+
+
+/// Measures the actual query-side `jsonExtractScalar*` ValueBlock loop. Each invocation processes a 128-row block;
+/// JMH normalizes throughput and allocation to one row via [OperationsPerInvocation]. Input projection is represented
+/// by a pre-materialized String array so the comparison isolates JSON extraction, type coercion, and result writing.
+@BenchmarkMode(Mode.Throughput)
+@OutputTimeUnit(TimeUnit.SECONDS)
+@Fork(1)
+@Warmup(iterations = 3, time = 2)
+@Measurement(iterations = 5, time = 3)
+@State(Scope.Thread)
+public class BenchmarkJsonExtractScalarQuery {
+ private static final int BLOCK_ROWS = 128;
+ private static final TransformResultMetadata STRING_METADATA =
+ new TransformResultMetadata(DataType.STRING, true, false);
+ private static final String BASE_JSON = "{"
+ + "\"earlyMetric\":17,"
+ + "\"user\":{\"id\":\"u-19283\",\"country\":\"US\",\"tier\":\"gold\",\"age\":41},"
+ + "\"event\":{\"name\":\"checkout\",\"cart\":[{\"sku\":\"A1\",\"qty\":2,\"price\":19.99},"
+ + "{\"sku\":\"B7\",\"qty\":1,\"price\":129.0},{\"sku\":\"C3\",\"qty\":4,\"price\":3.5}],"
+ + "\"currency\":\"USD\",\"coupon\":null},"
+ + "\"device\":{\"os\":\"macOS\",\"version\":\"14.5\",\"browser\":\"Chrome\","
+ + "\"screen\":{\"width\":1728,\"height\":1117}},"
+ + "\"geo\":{\"city\":\"San Francisco\",\"region\":\"CA\",\"country\":\"US\","
+ + "\"coordinates\":[-122.4194,37.7749]},"
+ + "\"attributes\":{\"campaign\":\"summer-sale\",\"referrer\":\"search\",\"experiment\":\"checkout-v2\"},"
+ + "\"flags\":[\"returning\",\"subscribed\",\"beta\"],"
+ + "\"lateMetric\":19} ";
+
+ @Param({"early", "late"})
+ private String _fieldPosition;
+
+ @Param({"700", "8192", "65536"})
+ private int _documentBytes;
+
+ private ValueBlock _valueBlock;
+ private JsonExtractScalarTransformFunction _jayway;
+ private JsonExtractScalarTransformFunction _fast;
+ private JsonExtractScalarTransformFunction _firstMatch;
+ private JsonExtractScalarTransformFunction _fory;
+
+ @Setup
+ public void setUp() {
+ String json = buildJson(_documentBytes);
+ String[] jsonRows = new String[BLOCK_ROWS];
+ Arrays.fill(jsonRows, json);
+ TransformFunction input = new StringArrayTransformFunction(jsonRows);
+ String path = "early".equals(_fieldPosition) ? "$.earlyMetric" : "$.lateMetric";
+ List<TransformFunction> arguments = List.of(input, literal(path), literal("LONG"));
+
+ _valueBlock = new FixedValueBlock(BLOCK_ROWS);
+ _jayway = new JsonExtractScalarTransformFunction();
+ _fast = new JsonExtractScalarTransformFunction.Fast();
+ _firstMatch = new JsonExtractScalarTransformFunction.FirstMatch();
+ _fory = new JsonExtractScalarTransformFunction.Fory();
+ for (JsonExtractScalarTransformFunction function : List.of(_jayway, _fast, _firstMatch, _fory)) {
+ function.init(arguments, Map.<String, ColumnContext>of(), false);
+ long expected = "early".equals(_fieldPosition) ? 17L : 19L;
+ long[] values = function.transformToLongValuesSV(_valueBlock);
+ if (values.length < BLOCK_ROWS || values[0] != expected || values[BLOCK_ROWS - 1] != expected) {
+ throw new IllegalStateException(function.getName() + " produced an unexpected query result");
+ }
+ }
+ }
+
+ private static LiteralTransformFunction literal(String value) {
+ return new LiteralTransformFunction(new LiteralContext(DataType.STRING, value));
+ }
+
+ private static String buildJson(int targetBytes) {
+ if (BASE_JSON.length() >= targetBytes) {
+ return BASE_JSON;
+ }
+ String marker = "\"lateMetric\":";
+ int markerOffset = BASE_JSON.indexOf(marker);
+ String paddingPrefix = "\"padding\":\"";
+ String paddingSuffix = "\",";
+ int paddingLength = targetBytes - BASE_JSON.length() - paddingPrefix.length() - paddingSuffix.length();
+ return BASE_JSON.substring(0, markerOffset) + paddingPrefix + "x".repeat(paddingLength) + paddingSuffix
+ + BASE_JSON.substring(markerOffset);
Review Comment:
`buildJson(int targetBytes)` can compute a negative `paddingLength` when `targetBytes` is slightly larger than `BASE_JSON.length()` but smaller than `BASE_JSON.length() + paddingPrefix.length() + paddingSuffix.length()`, causing `"x".repeat(paddingLength)` to throw. `BenchmarkJsonPathExtraction#buildJson` already guards this case; this benchmark should do the same to keep the helper robust if the base JSON changes or params are adjusted.
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/JsonExtractScalarTransformFunction.java:
##########
@@ -72,6 +74,10 @@
/// keys to the first non-null occurrence and does not validate malformed content after the resolved value. Use it
/// only for well-formed, duplicate-free JSON. `Fast` scans the full root value and retains Jayway's last-key-wins
/// and malformed-document behavior; see [FastJsonPathExtractor] for one documented unaddressed-value edge case.
+/// `jsonExtractScalarFory` is experimental and must be selected explicitly. It accelerates simple paths over
+/// `STRING` input for scalar result types other than `STRING` and `BIG_DECIMAL`. `BYTES` input, complex paths,
+/// containers / array result types, precision-sensitive results, deeply nested documents, and Fory failures use
+/// Jayway. Its name, supported envelope, and implementation can change while the integration is evaluated.
Review Comment:
The Javadoc says `jsonExtractScalarFory` accelerates scalar result types other than `STRING` and `BIG_DECIMAL`, but the implementation enables the Fory extractor for `STRING` results as well (it only gates on `useBigDecimal`). This makes the doc misleading for callers trying to understand when the Fory path is actually used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add experimental Fory JSON functions [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #19223:
URL: https://github.com/apache/pinot/pull/19223#discussion_r3771082053
##########
pinot-common/src/main/java/org/apache/pinot/common/function/ForyJsonPathExtractor.java:
##########
@@ -0,0 +1,439 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.function;
+
+import com.fasterxml.jackson.core.StreamReadConstraints;
+import java.util.concurrent.atomic.AtomicBoolean;
+import javax.annotation.Nullable;
+import org.apache.fory.json.ForyJson;
+import org.apache.fory.json.codec.JsonValueCodec;
+import org.apache.fory.json.reader.JsonReader;
+import org.apache.fory.json.reader.Latin1JsonReader;
+import org.apache.fory.json.reader.Utf16JsonReader;
+import org.apache.fory.json.reader.Utf8JsonReader;
+import org.apache.fory.json.writer.StringJsonWriter;
+import org.apache.fory.json.writer.Utf8JsonWriter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/// Shared Fory JSON parser for opt-in JSON-path implementations.
+///
+/// Streaming extraction uses one single-state parser per worker thread to avoid shared-pool contention. Initialization
+/// or runtime linkage failures permanently disable the optional path, allowing callers to fall back to
+/// Jackson/Jayway. Jackson's default scalar token limits are enforced while walking the document, and inputs outside
+/// Fory's safe nesting depth fall back to the reference parser.
+public final class ForyJsonPathExtractor {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ForyJsonPathExtractor.class);
+ private static final StreamReadConstraints JACKSON_CONSTRAINTS = StreamReadConstraints.defaults();
+ private static final ThreadLocal<PathContext> PATH_CONTEXT = ThreadLocal.withInitial(PathContext::new);
+ private static final AtomicBoolean UNAVAILABLE_WARNING_LOGGED = new AtomicBoolean();
+
+ private ForyJsonPathExtractor() {
+ }
+
+ private static final class Holder {
+ private static volatile boolean _streamingAvailable;
+ private static final ThreadLocal<ForyJson> STREAMING_PARSER = ThreadLocal.withInitial(() -> {
+ if (!_streamingAvailable) {
+ throw new IllegalStateException("Fory JSON is unavailable");
+ }
+ ForyJson parser = buildStreamingParser();
+ if (parser == null) {
+ _streamingAvailable = false;
+ throw new IllegalStateException("Fory JSON is unavailable");
+ }
+ return parser;
+ });
+
+ static {
+ ForyJson parser = buildStreamingParser();
+ _streamingAvailable = parser != null;
+ if (parser != null) {
+ STREAMING_PARSER.set(parser);
+ }
+ }
+
+ private Holder() {
+ }
+
+ @Nullable
+ private static ForyJson buildStreamingParser() {
+ try {
+ return ForyJson.builder().withCodegen(false).withAsyncCompilation(false).withConcurrencyLevel(1)
+ .registerCodec(PathResult.class, PathCodec.INSTANCE).build();
+ } catch (RuntimeException | LinkageError e) {
+ logUnavailable(e);
+ return null;
+ }
+ }
+ }
+
+ /// Returns whether the optional Fory runtime initialized successfully.
+ public static boolean isAvailable() {
+ return Holder._streamingAvailable;
+ }
+
+ /// Extracts a simple path with Fory's streaming reader without materializing the complete JSON tree.
+ ///
+ /// Unrelated values are still fully consumed so malformed input and duplicate-key last-wins behavior match the
+ /// reference parser. Jackson's nesting, field-name, string, and number limits are checked while scanning. Callers
+ /// should retry with the reference parser when this method throws.
+ @Nullable
+ public static Object extract(String json, SimpleJsonPath path) {
+ if ((JACKSON_CONSTRAINTS.hasMaxDocumentLength()
+ && json.length() > JACKSON_CONSTRAINTS.getMaxDocumentLength())
+ || (JACKSON_CONSTRAINTS.hasMaxTokenCount() && requiresJacksonFallback(json))) {
+ throw new IllegalArgumentException("JSON document requires Jackson constraint validation");
+ }
Review Comment:
Fixed in f0f1036c99. The dead lexical prescan is removed. Document length is checked up front; nesting, field-name, string, and number limits are enforced during the streaming walk; configured maxTokenCount is counted in that same walk. The focused configured-token-limit child-JVM test passes.
##########
pinot-common/src/main/java/org/apache/pinot/common/function/scalar/JsonFunctions.java:
##########
@@ -129,6 +130,25 @@ private static Object fastJsonPath(Object object, String jsonPath, boolean useBi
return jsonPath(object, jsonPath);
}
+ /// Resolves a simple path over a JSON string with Fory's streaming reader. Complex paths, already-parsed inputs,
+ /// container results, and Fory extraction failures use the existing Jayway implementation so the experimental
+ /// functions below retain the current behavior outside Fory's supported envelope.
+ @Nullable
+ private static Object foryJsonPath(Object object, String jsonPath) {
+ SimpleJsonPath simpleJsonPath = SimpleJsonPath.compile(jsonPath);
+ if (!(object instanceof String) || simpleJsonPath == null) {
+ return jsonPath(object, jsonPath);
+ }
Review Comment:
Fixed in f0f1036c99. foryJsonPath now checks object instanceof String before compiling SimpleJsonPath, so Map/List and other non-String inputs go directly to the existing Jayway path without the extra compile.
##########
pinot-perf/src/main/java/org/apache/pinot/perf/BenchmarkJsonExtractScalarQuery.java:
##########
@@ -0,0 +1,218 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.perf;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import org.apache.pinot.common.request.context.ExpressionContext;
+import org.apache.pinot.common.request.context.LiteralContext;
+import org.apache.pinot.core.common.BlockValSet;
+import org.apache.pinot.core.operator.ColumnContext;
+import org.apache.pinot.core.operator.blocks.ValueBlock;
+import org.apache.pinot.core.operator.transform.TransformResultMetadata;
+import org.apache.pinot.core.operator.transform.function.BaseTransformFunction;
+import org.apache.pinot.core.operator.transform.function.JsonExtractScalarTransformFunction;
+import org.apache.pinot.core.operator.transform.function.LiteralTransformFunction;
+import org.apache.pinot.core.operator.transform.function.TransformFunction;
+import org.apache.pinot.spi.data.FieldSpec.DataType;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.OperationsPerInvocation;
+import org.openjdk.jmh.annotations.OutputTimeUnit;
+import org.openjdk.jmh.annotations.Param;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.Warmup;
+import org.openjdk.jmh.runner.Runner;
+import org.openjdk.jmh.runner.options.Options;
+import org.openjdk.jmh.runner.options.OptionsBuilder;
+
+
+/// Measures the actual query-side `jsonExtractScalar*` ValueBlock loop. Each invocation processes a 128-row block;
+/// JMH normalizes throughput and allocation to one row via [OperationsPerInvocation]. Input projection is represented
+/// by a pre-materialized String array so the comparison isolates JSON extraction, type coercion, and result writing.
+@BenchmarkMode(Mode.Throughput)
+@OutputTimeUnit(TimeUnit.SECONDS)
+@Fork(1)
+@Warmup(iterations = 3, time = 2)
+@Measurement(iterations = 5, time = 3)
+@State(Scope.Thread)
+public class BenchmarkJsonExtractScalarQuery {
+ private static final int BLOCK_ROWS = 128;
+ private static final TransformResultMetadata STRING_METADATA =
+ new TransformResultMetadata(DataType.STRING, true, false);
+ private static final String BASE_JSON = "{"
+ + "\"earlyMetric\":17,"
+ + "\"user\":{\"id\":\"u-19283\",\"country\":\"US\",\"tier\":\"gold\",\"age\":41},"
+ + "\"event\":{\"name\":\"checkout\",\"cart\":[{\"sku\":\"A1\",\"qty\":2,\"price\":19.99},"
+ + "{\"sku\":\"B7\",\"qty\":1,\"price\":129.0},{\"sku\":\"C3\",\"qty\":4,\"price\":3.5}],"
+ + "\"currency\":\"USD\",\"coupon\":null},"
+ + "\"device\":{\"os\":\"macOS\",\"version\":\"14.5\",\"browser\":\"Chrome\","
+ + "\"screen\":{\"width\":1728,\"height\":1117}},"
+ + "\"geo\":{\"city\":\"San Francisco\",\"region\":\"CA\",\"country\":\"US\","
+ + "\"coordinates\":[-122.4194,37.7749]},"
+ + "\"attributes\":{\"campaign\":\"summer-sale\",\"referrer\":\"search\",\"experiment\":\"checkout-v2\"},"
+ + "\"flags\":[\"returning\",\"subscribed\",\"beta\"],"
+ + "\"lateMetric\":19} ";
+
+ @Param({"early", "late"})
+ private String _fieldPosition;
+
+ @Param({"700", "8192", "65536"})
+ private int _documentBytes;
+
+ private ValueBlock _valueBlock;
+ private JsonExtractScalarTransformFunction _jayway;
+ private JsonExtractScalarTransformFunction _fast;
+ private JsonExtractScalarTransformFunction _firstMatch;
+ private JsonExtractScalarTransformFunction _fory;
+
+ @Setup
+ public void setUp() {
+ String json = buildJson(_documentBytes);
+ String[] jsonRows = new String[BLOCK_ROWS];
+ Arrays.fill(jsonRows, json);
+ TransformFunction input = new StringArrayTransformFunction(jsonRows);
+ String path = "early".equals(_fieldPosition) ? "$.earlyMetric" : "$.lateMetric";
+ List<TransformFunction> arguments = List.of(input, literal(path), literal("LONG"));
+
+ _valueBlock = new FixedValueBlock(BLOCK_ROWS);
+ _jayway = new JsonExtractScalarTransformFunction();
+ _fast = new JsonExtractScalarTransformFunction.Fast();
+ _firstMatch = new JsonExtractScalarTransformFunction.FirstMatch();
+ _fory = new JsonExtractScalarTransformFunction.Fory();
+ for (JsonExtractScalarTransformFunction function : List.of(_jayway, _fast, _firstMatch, _fory)) {
+ function.init(arguments, Map.<String, ColumnContext>of(), false);
+ long expected = "early".equals(_fieldPosition) ? 17L : 19L;
+ long[] values = function.transformToLongValuesSV(_valueBlock);
+ if (values.length < BLOCK_ROWS || values[0] != expected || values[BLOCK_ROWS - 1] != expected) {
+ throw new IllegalStateException(function.getName() + " produced an unexpected query result");
+ }
+ }
+ }
+
+ private static LiteralTransformFunction literal(String value) {
+ return new LiteralTransformFunction(new LiteralContext(DataType.STRING, value));
+ }
+
+ private static String buildJson(int targetBytes) {
+ if (BASE_JSON.length() >= targetBytes) {
+ return BASE_JSON;
+ }
+ String marker = "\"lateMetric\":";
+ int markerOffset = BASE_JSON.indexOf(marker);
+ String paddingPrefix = "\"padding\":\"";
+ String paddingSuffix = "\",";
+ int paddingLength = targetBytes - BASE_JSON.length() - paddingPrefix.length() - paddingSuffix.length();
+ return BASE_JSON.substring(0, markerOffset) + paddingPrefix + "x".repeat(paddingLength) + paddingSuffix
+ + BASE_JSON.substring(markerOffset);
Review Comment:
Fixed in f0f1036c99. buildJson now returns the base document when the prefix/suffix leave no positive padding length, avoiding a negative String.repeat argument.
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/JsonExtractScalarTransformFunction.java:
##########
@@ -72,6 +74,10 @@
/// keys to the first non-null occurrence and does not validate malformed content after the resolved value. Use it
/// only for well-formed, duplicate-free JSON. `Fast` scans the full root value and retains Jayway's last-key-wins
/// and malformed-document behavior; see [FastJsonPathExtractor] for one documented unaddressed-value edge case.
+/// `jsonExtractScalarFory` is experimental and must be selected explicitly. It accelerates simple paths over
+/// `STRING` input for scalar result types other than `STRING` and `BIG_DECIMAL`. `BYTES` input, complex paths,
+/// containers / array result types, precision-sensitive results, deeply nested documents, and Fory failures use
+/// Jayway. Its name, supported envelope, and implementation can change while the integration is evaluated.
Review Comment:
Resolved in f0f1036c99 with explicit initialization-time eligibility. Fory is eligible only for simple paths, STRING input, single-value output, and result types other than STRING/JSON/BIG_DECIMAL. The Javadoc now states this exact envelope, and the eligibility matrix test covers it.
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/JsonExtractScalarTransformFunction.java:
##########
@@ -802,7 +817,7 @@ private <T> IntFunction<T> getResultExtractor(ValueBlock valueBlock, ParseContex
if (_jsonFieldTransformFunction.getResultMetadata().getDataType() == DataType.BYTES) {
byte[][] jsonBytes = _jsonFieldTransformFunction.transformToBytesValuesSV(valueBlock);
IntFunction<T> jaywayExtractor = i -> parseContext.parseUtf8(jsonBytes[i]).read(_jsonPath);
- if (_simpleJsonPath == null) {
+ if (_simpleJsonPath == null || useBigDecimal || _extractionMode == ExtractionMode.FORY) {
Review Comment:
Fixed in f0f1036c99. The BYTES branch no longer gates Fast/FirstMatch on useBigDecimal; only Fory uses Jayway for BYTES. I added a non-vacuous regression using a precise addressed decimal plus a hostile unaddressed exponent that Jayway rejects but Fast/FirstMatch skip. The 246-case focused suite passes.
##########
pinot-common/src/main/java/org/apache/pinot/common/function/ForyJsonPathExtractor.java:
##########
@@ -0,0 +1,439 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.function;
+
+import com.fasterxml.jackson.core.StreamReadConstraints;
+import java.util.concurrent.atomic.AtomicBoolean;
+import javax.annotation.Nullable;
+import org.apache.fory.json.ForyJson;
+import org.apache.fory.json.codec.JsonValueCodec;
+import org.apache.fory.json.reader.JsonReader;
+import org.apache.fory.json.reader.Latin1JsonReader;
+import org.apache.fory.json.reader.Utf16JsonReader;
+import org.apache.fory.json.reader.Utf8JsonReader;
+import org.apache.fory.json.writer.StringJsonWriter;
+import org.apache.fory.json.writer.Utf8JsonWriter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/// Shared Fory JSON parser for opt-in JSON-path implementations.
+///
+/// Streaming extraction uses one single-state parser per worker thread to avoid shared-pool contention. Initialization
+/// or runtime linkage failures permanently disable the optional path, allowing callers to fall back to
+/// Jackson/Jayway. Jackson's default scalar token limits are enforced while walking the document, and inputs outside
+/// Fory's safe nesting depth fall back to the reference parser.
+public final class ForyJsonPathExtractor {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ForyJsonPathExtractor.class);
+ private static final StreamReadConstraints JACKSON_CONSTRAINTS = StreamReadConstraints.defaults();
+ private static final ThreadLocal<PathContext> PATH_CONTEXT = ThreadLocal.withInitial(PathContext::new);
+ private static final AtomicBoolean UNAVAILABLE_WARNING_LOGGED = new AtomicBoolean();
+
+ private ForyJsonPathExtractor() {
+ }
+
+ private static final class Holder {
+ private static volatile boolean _streamingAvailable;
+ private static final ThreadLocal<ForyJson> STREAMING_PARSER = ThreadLocal.withInitial(() -> {
+ if (!_streamingAvailable) {
+ throw new IllegalStateException("Fory JSON is unavailable");
+ }
+ ForyJson parser = buildStreamingParser();
+ if (parser == null) {
+ _streamingAvailable = false;
+ throw new IllegalStateException("Fory JSON is unavailable");
+ }
+ return parser;
+ });
+
+ static {
+ ForyJson parser = buildStreamingParser();
+ _streamingAvailable = parser != null;
+ if (parser != null) {
+ STREAMING_PARSER.set(parser);
+ }
+ }
+
+ private Holder() {
+ }
+
+ @Nullable
+ private static ForyJson buildStreamingParser() {
+ try {
+ return ForyJson.builder().withCodegen(false).withAsyncCompilation(false).withConcurrencyLevel(1)
+ .registerCodec(PathResult.class, PathCodec.INSTANCE).build();
+ } catch (RuntimeException | LinkageError e) {
+ logUnavailable(e);
+ return null;
+ }
+ }
+ }
+
+ /// Returns whether the optional Fory runtime initialized successfully.
+ public static boolean isAvailable() {
+ return Holder._streamingAvailable;
+ }
+
+ /// Extracts a simple path with Fory's streaming reader without materializing the complete JSON tree.
+ ///
+ /// Unrelated values are still fully consumed so malformed input and duplicate-key last-wins behavior match the
+ /// reference parser. Jackson's nesting, field-name, string, and number limits are checked while scanning. Callers
+ /// should retry with the reference parser when this method throws.
+ @Nullable
+ public static Object extract(String json, SimpleJsonPath path) {
+ if ((JACKSON_CONSTRAINTS.hasMaxDocumentLength()
+ && json.length() > JACKSON_CONSTRAINTS.getMaxDocumentLength())
+ || (JACKSON_CONSTRAINTS.hasMaxTokenCount() && requiresJacksonFallback(json))) {
+ throw new IllegalArgumentException("JSON document requires Jackson constraint validation");
+ }
+ if (!Holder._streamingAvailable) {
+ throw new IllegalStateException("Fory JSON is unavailable");
+ }
+ ForyJson parser = Holder.STREAMING_PARSER.get();
+ PathContext context = PATH_CONTEXT.get();
+ if (context._active) {
+ throw new IllegalStateException("Fory JSON path extraction is not reentrant");
+ }
+ context._active = true;
+ context._path = path;
+ context._result = null;
+ try {
+ parser.fromJson(json, PathResult.class);
+ return context._result;
+ } catch (LinkageError e) {
+ disable();
+ logUnavailable(e);
+ throw new IllegalStateException("Fory JSON became unavailable", e);
+ } finally {
+ context._path = null;
+ context._result = null;
+ context._active = false;
+ }
+ }
+
+ private static void disable() {
+ Holder._streamingAvailable = false;
+ Holder.STREAMING_PARSER.remove();
+ PATH_CONTEXT.remove();
+ }
+
+ private static void logUnavailable(Throwable cause) {
+ if (UNAVAILABLE_WARNING_LOGGED.compareAndSet(false, true)) {
+ LOGGER.warn("Experimental Fory JSON support is unavailable; falling back to Jackson/Jayway", cause);
+ }
+ }
+
+ private static Object readPath(JsonReader reader, SimpleJsonPath path, int depth) {
+ String key = path.getKey(depth);
+ if (key != null) {
+ return readObjectPath(reader, path, depth, key);
+ }
+ return readArrayPath(reader, path, depth, path.getIndex(depth));
+ }
+
+ @Nullable
+ private static Object readObjectPath(JsonReader reader, SimpleJsonPath path, int depth, String expectedKey) {
+ if (reader.peekToken() != '{') {
+ skipValue(reader);
+ return null;
+ }
+ reader.enterDepth();
+ try {
+ reader.expect('{');
+ if (reader.consume('}')) {
+ return null;
+ }
+ Object result = null;
+ boolean more;
+ do {
+ String fieldName = reader.readFieldName();
+ if (fieldName.length() > JACKSON_CONSTRAINTS.getMaxNameLength()) {
+ throw new IllegalArgumentException("JSON field name exceeds Jackson's configured limit");
+ }
+ reader.expect(':');
+ if (expectedKey.equals(fieldName)) {
+ result = depth + 1 == path.length() ? readScalar(reader) : readPath(reader, path, depth + 1);
+ } else {
+ skipValue(reader);
+ }
+ more = reader.consumeCommaOrEndObject();
+ } while (more);
+ return result;
+ } finally {
+ reader.exitDepth();
+ }
+ }
+
+ @Nullable
+ private static Object readArrayPath(JsonReader reader, SimpleJsonPath path, int depth, int expectedIndex) {
+ if (reader.peekToken() != '[') {
+ skipValue(reader);
+ return null;
+ }
+ reader.enterDepth();
+ try {
+ reader.expect('[');
+ if (reader.consume(']')) {
+ return null;
+ }
+ Object result = null;
+ int index = 0;
+ boolean more;
+ do {
+ if (index == expectedIndex) {
+ result = depth + 1 == path.length() ? readScalar(reader) : readPath(reader, path, depth + 1);
+ } else {
+ skipValue(reader);
+ }
+ index++;
+ more = reader.consumeCommaOrEndArray();
+ } while (more);
+ return result;
+ } finally {
+ reader.exitDepth();
+ }
+ }
+
+ @Nullable
+ private static Object readScalar(JsonReader reader) {
+ char token = reader.peekToken();
+ if (token == '"') {
+ String value = reader.readString();
+ if (value.length() > JACKSON_CONSTRAINTS.getMaxStringLength()) {
+ throw new IllegalArgumentException("JSON string exceeds Jackson's configured limit");
+ }
+ return value;
+ }
+ if (token == 't' || token == 'f') {
+ return reader.readBoolean();
+ }
+ if (token == 'n') {
+ reader.readNull();
+ return null;
+ }
+ if (token == '{' || token == '[') {
+ // Query scalar coercion has observable error/default behavior for containers. Let Jayway produce the exact
+ // reference value rather than materializing a Fory container on this uncommon path.
+ throw new IllegalArgumentException("Container result requires reference JSON parsing");
Review Comment:
Fixed in f0f1036c99. Container leaves are fully consumed and return a private identity sentinel instead of throwing per row; callers then use Jayway for exact container/coercion behavior. Duplicate container/scalar order and malformed-tail tests cover the sentinel path. At T32, object/array Fory allocation dropped from about 2.9/2.8 KiB to 2.2/1.8 KiB per op.
##########
pinot-common/src/main/java/org/apache/pinot/common/function/ForyJsonPathExtractor.java:
##########
@@ -0,0 +1,439 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.function;
+
+import com.fasterxml.jackson.core.StreamReadConstraints;
+import java.util.concurrent.atomic.AtomicBoolean;
+import javax.annotation.Nullable;
+import org.apache.fory.json.ForyJson;
+import org.apache.fory.json.codec.JsonValueCodec;
+import org.apache.fory.json.reader.JsonReader;
+import org.apache.fory.json.reader.Latin1JsonReader;
+import org.apache.fory.json.reader.Utf16JsonReader;
+import org.apache.fory.json.reader.Utf8JsonReader;
+import org.apache.fory.json.writer.StringJsonWriter;
+import org.apache.fory.json.writer.Utf8JsonWriter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/// Shared Fory JSON parser for opt-in JSON-path implementations.
+///
+/// Streaming extraction uses one single-state parser per worker thread to avoid shared-pool contention. Initialization
+/// or runtime linkage failures permanently disable the optional path, allowing callers to fall back to
+/// Jackson/Jayway. Jackson's default scalar token limits are enforced while walking the document, and inputs outside
+/// Fory's safe nesting depth fall back to the reference parser.
+public final class ForyJsonPathExtractor {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ForyJsonPathExtractor.class);
+ private static final StreamReadConstraints JACKSON_CONSTRAINTS = StreamReadConstraints.defaults();
+ private static final ThreadLocal<PathContext> PATH_CONTEXT = ThreadLocal.withInitial(PathContext::new);
+ private static final AtomicBoolean UNAVAILABLE_WARNING_LOGGED = new AtomicBoolean();
+
+ private ForyJsonPathExtractor() {
+ }
+
+ private static final class Holder {
+ private static volatile boolean _streamingAvailable;
+ private static final ThreadLocal<ForyJson> STREAMING_PARSER = ThreadLocal.withInitial(() -> {
Review Comment:
Fixed in f0f1036c99. The per-caller ThreadLocal<ForyJson> is replaced by one process-wide, thread-safe Fory facade with Fory's fixed default state pool; only the small path/result context remains thread-local and is cleared after each call. T32 ValueBlock Fory remains 1.15-1.32x Fast at 700 B and 3.62-4.13x at 64 KiB. The pool is bounded, though steady-state JMH does not measure its retained heap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add experimental Fory JSON functions [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on PR #19223:
URL: https://github.com/apache/pinot/pull/19223#issuecomment-5273911749
Addressed the latest review feedback in f0f1036c99.
Key changes:
- Replaced per-thread Fory parsers with one process-wide, fixed-size Fory pool; only the small path context remains thread-local.
- Replaced exception-driven container fallback with a sentinel and routes known Fory-ineligible transform types to Jayway once during initialization.
- Restored Fast and FirstMatch routing for BYTES plus BIG_DECIMAL.
- Aligned Fory depth with Jackson constraints, removed the dead lexical prescan, and counts configured token limits during the streaming walk.
- Made Fory optional and excluded it from the Pinot Common shaded JAR and default runtime dependency graph. Applications must add Fory to the application classpath to activate the experimental path.
- Added direct-path, sentinel, depth, token-limit, missing-runtime, concurrency, routing, and fallback benchmarks/tests.
Directional paired JMH rerun on JDK 25.0.4 with GC profiling. Scalar ratios are T1; production ValueBlock transform ratios are T32. Each cell is Fory/Fast and Fory/normal-Jayway:
| Surface | Type | 700 B | 8 KiB | 64 KiB |
|---|---:|---:|---:|---:|
| jsonPath scalar | STRING | 1.21x / 4.89x | 2.66x / 3.71x | 4.41x / 9.86x |
| jsonPath scalar | LONG | 1.24x / 3.44x | 2.46x / 3.25x | 3.70x / 9.47x |
| jsonPath scalar | DOUBLE | 1.35x / 3.54x | 3.17x / 4.60x | 4.68x / 10.94x |
| jsonExtractScalar ValueBlock | LONG | 1.15x / 1.70x | 2.24x / 2.71x | 3.62x / 9.97x |
| jsonExtractScalar ValueBlock | DOUBLE | 1.32x / 2.16x | 1.51x / 2.40x | 4.13x / 10.58x |
The new direct depth cases no longer fall back: depth-selected and depth-unrelated reached 13.9M and 17.1M ops/s at about 48 B/op. The old implementation measured 0.77M and 1.59M ops/s with about 16.0 KiB and 10.7 KiB/op.
Important caveat: Fory 1.6 still materializes skipped strings because it has no public allocation-free fast-skip API. Fory allocates roughly 1.1-1.3 KiB, 8.6-8.8 KiB, and 66 KiB per 700 B, 8 KiB, and 64 KiB row, versus roughly 0.7-0.96 KiB for Fast. This remains the reason to keep support experimental and optional. The one-fork T32 confidence intervals are wide, so these are directional results rather than statistically precise claims.
Validation:
- Fory common tests: 27/27 passed.
- JsonExtractScalarTransformFunctionTest: 246/246 passed.
- JsonPathTest integration suite: 48/48 passed.
- Spotless, license format/check, and checkstyle passed for all affected modules.
- Pinot Common shaded JAR contains no Fory classes; Pinot Core has no runtime Fory dependency.
- UdfTest did not reach its snapshot assertion because current main has an unrelated NotUdf reflection mismatch: NotUdf requests LogicalFunctions.not(boolean), while the source exposes not(Boolean). No unrelated UDF change is included here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add experimental Fory JSON functions [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 merged PR #19223:
URL: https://github.com/apache/pinot/pull/19223
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add experimental Fory JSON functions [pinot]
Posted by "yashmayya (via GitHub)" <gi...@apache.org>.
yashmayya commented on code in PR #19223:
URL: https://github.com/apache/pinot/pull/19223#discussion_r3770314522
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/JsonExtractScalarTransformFunction.java:
##########
@@ -802,7 +817,7 @@ private <T> IntFunction<T> getResultExtractor(ValueBlock valueBlock, ParseContex
if (_jsonFieldTransformFunction.getResultMetadata().getDataType() == DataType.BYTES) {
byte[][] jsonBytes = _jsonFieldTransformFunction.transformToBytesValuesSV(valueBlock);
IntFunction<T> jaywayExtractor = i -> parseContext.parseUtf8(jsonBytes[i]).read(_jsonPath);
- if (_simpleJsonPath == null) {
+ if (_simpleJsonPath == null || useBigDecimal || _extractionMode == ExtractionMode.FORY) {
Review Comment:
**Blocker.** This line changes the two existing functions, not only the new one.
Before this PR, BYTES input with a `BIG_DECIMAL` result type used `FastJsonPathExtractor`. The new `useBigDecimal` condition sends that case to Jayway instead.
The STRING branch at line 868 still passes `useBigDecimal` to the fast extractor. This difference between the two branches looks accidental.
To gate only Fory, write `_simpleJsonPath == null || _extractionMode == ExtractionMode.FORY`. If the change is intentional, please add a test and a note in the PR description.
##########
pinot-common/src/main/java/org/apache/pinot/common/function/ForyJsonPathExtractor.java:
##########
@@ -0,0 +1,439 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.function;
+
+import com.fasterxml.jackson.core.StreamReadConstraints;
+import java.util.concurrent.atomic.AtomicBoolean;
+import javax.annotation.Nullable;
+import org.apache.fory.json.ForyJson;
+import org.apache.fory.json.codec.JsonValueCodec;
+import org.apache.fory.json.reader.JsonReader;
+import org.apache.fory.json.reader.Latin1JsonReader;
+import org.apache.fory.json.reader.Utf16JsonReader;
+import org.apache.fory.json.reader.Utf8JsonReader;
+import org.apache.fory.json.writer.StringJsonWriter;
+import org.apache.fory.json.writer.Utf8JsonWriter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/// Shared Fory JSON parser for opt-in JSON-path implementations.
+///
+/// Streaming extraction uses one single-state parser per worker thread to avoid shared-pool contention. Initialization
+/// or runtime linkage failures permanently disable the optional path, allowing callers to fall back to
+/// Jackson/Jayway. Jackson's default scalar token limits are enforced while walking the document, and inputs outside
+/// Fory's safe nesting depth fall back to the reference parser.
+public final class ForyJsonPathExtractor {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ForyJsonPathExtractor.class);
+ private static final StreamReadConstraints JACKSON_CONSTRAINTS = StreamReadConstraints.defaults();
+ private static final ThreadLocal<PathContext> PATH_CONTEXT = ThreadLocal.withInitial(PathContext::new);
+ private static final AtomicBoolean UNAVAILABLE_WARNING_LOGGED = new AtomicBoolean();
+
+ private ForyJsonPathExtractor() {
+ }
+
+ private static final class Holder {
+ private static volatile boolean _streamingAvailable;
+ private static final ThreadLocal<ForyJson> STREAMING_PARSER = ThreadLocal.withInitial(() -> {
Review Comment:
**Blocker.** Each thread that calls this function keeps its own `ForyJson` instance forever.
I measured the live heap on your branch. 64 threads added 14.28 MB, which is about 220 KB for each thread. A control group of 64 idle threads added 0.53 MB.
This ThreadLocal is never removed. A Pinot server has many query threads, and ingestion adds one thread for each consuming partition. The total reaches tens of megabytes for an experimental function.
`ForyJson` is already thread-safe and pooled through `withConcurrencyLevel`. One shared instance, with a concurrency level that matches the thread pool, avoids this multiplication. The comment above says that a shared pool causes contention. The PR gives no measurement for that claim.
The static block at line 69 also calls `STREAMING_PARSER.set(parser)`. This pins one parser to the arbitrary thread that loads the class first, and that thread can be a Helix or ZooKeeper callback thread that never uses it.
##########
pinot-common/src/main/java/org/apache/pinot/common/function/ForyJsonPathExtractor.java:
##########
@@ -0,0 +1,439 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.function;
+
+import com.fasterxml.jackson.core.StreamReadConstraints;
+import java.util.concurrent.atomic.AtomicBoolean;
+import javax.annotation.Nullable;
+import org.apache.fory.json.ForyJson;
+import org.apache.fory.json.codec.JsonValueCodec;
+import org.apache.fory.json.reader.JsonReader;
+import org.apache.fory.json.reader.Latin1JsonReader;
+import org.apache.fory.json.reader.Utf16JsonReader;
+import org.apache.fory.json.reader.Utf8JsonReader;
+import org.apache.fory.json.writer.StringJsonWriter;
+import org.apache.fory.json.writer.Utf8JsonWriter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/// Shared Fory JSON parser for opt-in JSON-path implementations.
+///
+/// Streaming extraction uses one single-state parser per worker thread to avoid shared-pool contention. Initialization
+/// or runtime linkage failures permanently disable the optional path, allowing callers to fall back to
+/// Jackson/Jayway. Jackson's default scalar token limits are enforced while walking the document, and inputs outside
+/// Fory's safe nesting depth fall back to the reference parser.
+public final class ForyJsonPathExtractor {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ForyJsonPathExtractor.class);
+ private static final StreamReadConstraints JACKSON_CONSTRAINTS = StreamReadConstraints.defaults();
+ private static final ThreadLocal<PathContext> PATH_CONTEXT = ThreadLocal.withInitial(PathContext::new);
+ private static final AtomicBoolean UNAVAILABLE_WARNING_LOGGED = new AtomicBoolean();
+
+ private ForyJsonPathExtractor() {
+ }
+
+ private static final class Holder {
+ private static volatile boolean _streamingAvailable;
+ private static final ThreadLocal<ForyJson> STREAMING_PARSER = ThreadLocal.withInitial(() -> {
+ if (!_streamingAvailable) {
+ throw new IllegalStateException("Fory JSON is unavailable");
+ }
+ ForyJson parser = buildStreamingParser();
+ if (parser == null) {
+ _streamingAvailable = false;
+ throw new IllegalStateException("Fory JSON is unavailable");
+ }
+ return parser;
+ });
+
+ static {
+ ForyJson parser = buildStreamingParser();
+ _streamingAvailable = parser != null;
+ if (parser != null) {
+ STREAMING_PARSER.set(parser);
+ }
+ }
+
+ private Holder() {
+ }
+
+ @Nullable
+ private static ForyJson buildStreamingParser() {
+ try {
+ return ForyJson.builder().withCodegen(false).withAsyncCompilation(false).withConcurrencyLevel(1)
+ .registerCodec(PathResult.class, PathCodec.INSTANCE).build();
+ } catch (RuntimeException | LinkageError e) {
+ logUnavailable(e);
+ return null;
+ }
+ }
+ }
+
+ /// Returns whether the optional Fory runtime initialized successfully.
+ public static boolean isAvailable() {
+ return Holder._streamingAvailable;
+ }
+
+ /// Extracts a simple path with Fory's streaming reader without materializing the complete JSON tree.
+ ///
+ /// Unrelated values are still fully consumed so malformed input and duplicate-key last-wins behavior match the
+ /// reference parser. Jackson's nesting, field-name, string, and number limits are checked while scanning. Callers
+ /// should retry with the reference parser when this method throws.
+ @Nullable
+ public static Object extract(String json, SimpleJsonPath path) {
+ if ((JACKSON_CONSTRAINTS.hasMaxDocumentLength()
+ && json.length() > JACKSON_CONSTRAINTS.getMaxDocumentLength())
+ || (JACKSON_CONSTRAINTS.hasMaxTokenCount() && requiresJacksonFallback(json))) {
+ throw new IllegalArgumentException("JSON document requires Jackson constraint validation");
+ }
+ if (!Holder._streamingAvailable) {
+ throw new IllegalStateException("Fory JSON is unavailable");
+ }
+ ForyJson parser = Holder.STREAMING_PARSER.get();
+ PathContext context = PATH_CONTEXT.get();
+ if (context._active) {
+ throw new IllegalStateException("Fory JSON path extraction is not reentrant");
+ }
+ context._active = true;
+ context._path = path;
+ context._result = null;
+ try {
+ parser.fromJson(json, PathResult.class);
+ return context._result;
+ } catch (LinkageError e) {
+ disable();
+ logUnavailable(e);
+ throw new IllegalStateException("Fory JSON became unavailable", e);
+ } finally {
+ context._path = null;
+ context._result = null;
+ context._active = false;
+ }
+ }
+
+ private static void disable() {
+ Holder._streamingAvailable = false;
+ Holder.STREAMING_PARSER.remove();
+ PATH_CONTEXT.remove();
+ }
+
+ private static void logUnavailable(Throwable cause) {
+ if (UNAVAILABLE_WARNING_LOGGED.compareAndSet(false, true)) {
+ LOGGER.warn("Experimental Fory JSON support is unavailable; falling back to Jackson/Jayway", cause);
+ }
+ }
+
+ private static Object readPath(JsonReader reader, SimpleJsonPath path, int depth) {
+ String key = path.getKey(depth);
+ if (key != null) {
+ return readObjectPath(reader, path, depth, key);
+ }
+ return readArrayPath(reader, path, depth, path.getIndex(depth));
+ }
+
+ @Nullable
+ private static Object readObjectPath(JsonReader reader, SimpleJsonPath path, int depth, String expectedKey) {
+ if (reader.peekToken() != '{') {
+ skipValue(reader);
+ return null;
+ }
+ reader.enterDepth();
+ try {
+ reader.expect('{');
+ if (reader.consume('}')) {
+ return null;
+ }
+ Object result = null;
+ boolean more;
+ do {
+ String fieldName = reader.readFieldName();
+ if (fieldName.length() > JACKSON_CONSTRAINTS.getMaxNameLength()) {
+ throw new IllegalArgumentException("JSON field name exceeds Jackson's configured limit");
+ }
+ reader.expect(':');
+ if (expectedKey.equals(fieldName)) {
+ result = depth + 1 == path.length() ? readScalar(reader) : readPath(reader, path, depth + 1);
+ } else {
+ skipValue(reader);
+ }
+ more = reader.consumeCommaOrEndObject();
+ } while (more);
+ return result;
+ } finally {
+ reader.exitDepth();
+ }
+ }
+
+ @Nullable
+ private static Object readArrayPath(JsonReader reader, SimpleJsonPath path, int depth, int expectedIndex) {
+ if (reader.peekToken() != '[') {
+ skipValue(reader);
+ return null;
+ }
+ reader.enterDepth();
+ try {
+ reader.expect('[');
+ if (reader.consume(']')) {
+ return null;
+ }
+ Object result = null;
+ int index = 0;
+ boolean more;
+ do {
+ if (index == expectedIndex) {
+ result = depth + 1 == path.length() ? readScalar(reader) : readPath(reader, path, depth + 1);
+ } else {
+ skipValue(reader);
+ }
+ index++;
+ more = reader.consumeCommaOrEndArray();
+ } while (more);
+ return result;
+ } finally {
+ reader.exitDepth();
+ }
+ }
+
+ @Nullable
+ private static Object readScalar(JsonReader reader) {
+ char token = reader.peekToken();
+ if (token == '"') {
+ String value = reader.readString();
+ if (value.length() > JACKSON_CONSTRAINTS.getMaxStringLength()) {
+ throw new IllegalArgumentException("JSON string exceeds Jackson's configured limit");
+ }
+ return value;
+ }
+ if (token == 't' || token == 'f') {
+ return reader.readBoolean();
+ }
+ if (token == 'n') {
+ reader.readNull();
+ return null;
+ }
+ if (token == '{' || token == '[') {
+ // Query scalar coercion has observable error/default behavior for containers. Let Jayway produce the exact
+ // reference value rather than materializing a Fory container on this uncommon path.
+ throw new IllegalArgumentException("Container result requires reference JSON parsing");
+ }
+ int start = reader.position();
+ Number value = reader.readNumber();
+ if (reader.position() - start > JACKSON_CONSTRAINTS.getMaxNumberLength()) {
+ throw new IllegalArgumentException("JSON number exceeds Jackson's configured limit");
+ }
+ return value;
+ }
+
+ private static void skipValue(JsonReader reader) {
+ char token = reader.peekToken();
+ if (token == '{') {
+ skipObject(reader);
+ return;
+ }
+ if (token == '[') {
+ skipArray(reader);
+ return;
+ }
+ if (token == '"') {
+ // Fory 1.6's skipValue() computes an FNV hash over every character. Its string decoder uses packed scans and
+ // is substantially faster even when the decoded value is discarded. An upstream fast-skip API could remove
+ // this temporary allocation in a future Fory version.
+ String value = reader.readString();
+ if (value.length() > JACKSON_CONSTRAINTS.getMaxStringLength()) {
+ throw new IllegalArgumentException("JSON string exceeds Jackson's configured limit");
+ }
+ return;
+ }
+ int start = reader.position();
+ reader.skipValue();
+ int rawLength = reader.position() - start;
+ if (token != 't' && token != 'f' && token != 'n'
+ && rawLength > JACKSON_CONSTRAINTS.getMaxNumberLength()) {
+ throw new IllegalArgumentException("JSON number exceeds Jackson's configured limit");
+ }
+ }
+
+ private static void skipObject(JsonReader reader) {
+ reader.enterDepth();
+ try {
+ reader.expect('{');
+ if (reader.consume('}')) {
+ return;
+ }
+ boolean more;
+ do {
+ String fieldName = reader.readFieldName();
+ if (fieldName.length() > JACKSON_CONSTRAINTS.getMaxNameLength()) {
+ throw new IllegalArgumentException("JSON field name exceeds Jackson's configured limit");
+ }
+ reader.expect(':');
+ skipValue(reader);
+ more = reader.consumeCommaOrEndObject();
+ } while (more);
+ } finally {
+ reader.exitDepth();
+ }
+ }
+
+ private static void skipArray(JsonReader reader) {
+ reader.enterDepth();
+ try {
+ reader.expect('[');
+ if (reader.consume(']')) {
+ return;
+ }
+ boolean more;
+ do {
+ skipValue(reader);
+ more = reader.consumeCommaOrEndArray();
+ } while (more);
+ } finally {
+ reader.exitDepth();
+ }
+ }
+
+ private static final class PathContext {
+ private final PathResult _marker = new PathResult();
+ private boolean _active;
+ @Nullable
+ private SimpleJsonPath _path;
+ @Nullable
+ private Object _result;
+ }
+
+ private static final class PathResult {
+ }
+
+ private static final class PathCodec implements JsonValueCodec<PathResult> {
+ private static final PathCodec INSTANCE = new PathCodec();
+
+ @Override
+ public PathResult readLatin1(Latin1JsonReader reader) {
+ return read(reader);
+ }
+
+ @Override
+ public PathResult readUtf16(Utf16JsonReader reader) {
+ return read(reader);
+ }
+
+ @Override
+ public PathResult readUtf8(Utf8JsonReader reader) {
+ return read(reader);
+ }
+
+ private static PathResult read(JsonReader reader) {
+ PathContext context = PATH_CONTEXT.get();
+ SimpleJsonPath path = context._path;
+ if (!context._active || path == null) {
+ throw new IllegalStateException("Missing JSON path extraction context");
+ }
+ context._result = readPath(reader, path, 0);
+ return context._marker;
+ }
+
+ @Override
+ public void writeString(StringJsonWriter writer, PathResult value) {
+ throw new UnsupportedOperationException("PathResult is read-only");
+ }
+
+ @Override
+ public void writeUtf8(Utf8JsonWriter writer, PathResult value) {
+ throw new UnsupportedOperationException("PathResult is read-only");
+ }
+ }
+
+ private static boolean requiresJacksonFallback(String json) {
Review Comment:
This method never runs with the default Jackson configuration.
The caller at line 102 invokes it only when `hasMaxTokenCount()` is true. I printed the defaults on JDK 25.0.3 with Jackson 2.22.1:
```
hasMaxDocumentLength=false maxDocumentLength=-1
hasMaxTokenCount=false maxTokenCount=-1
maxNestingDepth=1000 maxNameLength=50000 maxStringLength=20000000 maxNumberLength=1000
```
Both parts of the guard at line 100 are always false. These 70 lines are dead code, and they explain the 50.9% patch coverage that Codecov reports for this file.
The class Javadoc also says that Jackson's nesting limit is enforced while the parser walks the document. It is not. Only Fory's limit of 20 applies.
Please delete this method, or read the constraints from the same `JsonFactory` that the Jayway path uses, so that the guard becomes reachable and testable.
##########
pinot-common/pom.xml:
##########
@@ -280,6 +280,10 @@
<groupId>com.jayway.jsonpath</groupId>
<artifactId>json-path</artifactId>
</dependency>
+ <dependency>
+ <groupId>org.apache.fory</groupId>
+ <artifactId>fory-json</artifactId>
Review Comment:
**Blocker.** This adds a hard dependency to `pinot-common`. Every Pinot module and every shaded jar then contains `fory-json` and `fory-core`, which are about 3.6 MB together.
The PR describes all four functions as experimental, and says that they can change or be removed.
`fory-core` also calls a terminally deprecated JDK method. The first call prints this on JDK 25.0.3:
```
WARNING: A terminally deprecated method in sun.misc.Unsafe has been called
WARNING: sun.misc.Unsafe::staticFieldOffset has been called by org.apache.fory.platform.internal._Lookup
WARNING: sun.misc.Unsafe::staticFieldOffset will be removed in a future release
```
Your fallback handles the future removal correctly, because `buildStreamingParser` catches `RuntimeException` and `LinkageError`. The warning still reaches the logs of every role.
Please consider a separate optional module for this dependency until the evaluation is complete. A module keeps the experiment out of the default distribution.
##########
pinot-common/src/main/java/org/apache/pinot/common/function/ForyJsonPathExtractor.java:
##########
@@ -0,0 +1,439 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.function;
+
+import com.fasterxml.jackson.core.StreamReadConstraints;
+import java.util.concurrent.atomic.AtomicBoolean;
+import javax.annotation.Nullable;
+import org.apache.fory.json.ForyJson;
+import org.apache.fory.json.codec.JsonValueCodec;
+import org.apache.fory.json.reader.JsonReader;
+import org.apache.fory.json.reader.Latin1JsonReader;
+import org.apache.fory.json.reader.Utf16JsonReader;
+import org.apache.fory.json.reader.Utf8JsonReader;
+import org.apache.fory.json.writer.StringJsonWriter;
+import org.apache.fory.json.writer.Utf8JsonWriter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/// Shared Fory JSON parser for opt-in JSON-path implementations.
+///
+/// Streaming extraction uses one single-state parser per worker thread to avoid shared-pool contention. Initialization
+/// or runtime linkage failures permanently disable the optional path, allowing callers to fall back to
+/// Jackson/Jayway. Jackson's default scalar token limits are enforced while walking the document, and inputs outside
+/// Fory's safe nesting depth fall back to the reference parser.
+public final class ForyJsonPathExtractor {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ForyJsonPathExtractor.class);
+ private static final StreamReadConstraints JACKSON_CONSTRAINTS = StreamReadConstraints.defaults();
+ private static final ThreadLocal<PathContext> PATH_CONTEXT = ThreadLocal.withInitial(PathContext::new);
+ private static final AtomicBoolean UNAVAILABLE_WARNING_LOGGED = new AtomicBoolean();
+
+ private ForyJsonPathExtractor() {
+ }
+
+ private static final class Holder {
+ private static volatile boolean _streamingAvailable;
+ private static final ThreadLocal<ForyJson> STREAMING_PARSER = ThreadLocal.withInitial(() -> {
+ if (!_streamingAvailable) {
+ throw new IllegalStateException("Fory JSON is unavailable");
+ }
+ ForyJson parser = buildStreamingParser();
+ if (parser == null) {
+ _streamingAvailable = false;
+ throw new IllegalStateException("Fory JSON is unavailable");
+ }
+ return parser;
+ });
+
+ static {
+ ForyJson parser = buildStreamingParser();
+ _streamingAvailable = parser != null;
+ if (parser != null) {
+ STREAMING_PARSER.set(parser);
+ }
+ }
+
+ private Holder() {
+ }
+
+ @Nullable
+ private static ForyJson buildStreamingParser() {
+ try {
+ return ForyJson.builder().withCodegen(false).withAsyncCompilation(false).withConcurrencyLevel(1)
+ .registerCodec(PathResult.class, PathCodec.INSTANCE).build();
+ } catch (RuntimeException | LinkageError e) {
+ logUnavailable(e);
+ return null;
+ }
+ }
+ }
+
+ /// Returns whether the optional Fory runtime initialized successfully.
+ public static boolean isAvailable() {
+ return Holder._streamingAvailable;
+ }
+
+ /// Extracts a simple path with Fory's streaming reader without materializing the complete JSON tree.
+ ///
+ /// Unrelated values are still fully consumed so malformed input and duplicate-key last-wins behavior match the
+ /// reference parser. Jackson's nesting, field-name, string, and number limits are checked while scanning. Callers
+ /// should retry with the reference parser when this method throws.
+ @Nullable
+ public static Object extract(String json, SimpleJsonPath path) {
+ if ((JACKSON_CONSTRAINTS.hasMaxDocumentLength()
+ && json.length() > JACKSON_CONSTRAINTS.getMaxDocumentLength())
+ || (JACKSON_CONSTRAINTS.hasMaxTokenCount() && requiresJacksonFallback(json))) {
+ throw new IllegalArgumentException("JSON document requires Jackson constraint validation");
+ }
+ if (!Holder._streamingAvailable) {
+ throw new IllegalStateException("Fory JSON is unavailable");
+ }
+ ForyJson parser = Holder.STREAMING_PARSER.get();
+ PathContext context = PATH_CONTEXT.get();
+ if (context._active) {
+ throw new IllegalStateException("Fory JSON path extraction is not reentrant");
+ }
+ context._active = true;
+ context._path = path;
+ context._result = null;
+ try {
+ parser.fromJson(json, PathResult.class);
+ return context._result;
+ } catch (LinkageError e) {
+ disable();
+ logUnavailable(e);
+ throw new IllegalStateException("Fory JSON became unavailable", e);
+ } finally {
+ context._path = null;
+ context._result = null;
+ context._active = false;
+ }
+ }
+
+ private static void disable() {
+ Holder._streamingAvailable = false;
+ Holder.STREAMING_PARSER.remove();
+ PATH_CONTEXT.remove();
+ }
+
+ private static void logUnavailable(Throwable cause) {
+ if (UNAVAILABLE_WARNING_LOGGED.compareAndSet(false, true)) {
+ LOGGER.warn("Experimental Fory JSON support is unavailable; falling back to Jackson/Jayway", cause);
+ }
+ }
+
+ private static Object readPath(JsonReader reader, SimpleJsonPath path, int depth) {
+ String key = path.getKey(depth);
+ if (key != null) {
+ return readObjectPath(reader, path, depth, key);
+ }
+ return readArrayPath(reader, path, depth, path.getIndex(depth));
+ }
+
+ @Nullable
+ private static Object readObjectPath(JsonReader reader, SimpleJsonPath path, int depth, String expectedKey) {
+ if (reader.peekToken() != '{') {
+ skipValue(reader);
+ return null;
+ }
+ reader.enterDepth();
+ try {
+ reader.expect('{');
+ if (reader.consume('}')) {
+ return null;
+ }
+ Object result = null;
+ boolean more;
+ do {
+ String fieldName = reader.readFieldName();
+ if (fieldName.length() > JACKSON_CONSTRAINTS.getMaxNameLength()) {
+ throw new IllegalArgumentException("JSON field name exceeds Jackson's configured limit");
+ }
+ reader.expect(':');
+ if (expectedKey.equals(fieldName)) {
+ result = depth + 1 == path.length() ? readScalar(reader) : readPath(reader, path, depth + 1);
+ } else {
+ skipValue(reader);
+ }
+ more = reader.consumeCommaOrEndObject();
+ } while (more);
+ return result;
+ } finally {
+ reader.exitDepth();
+ }
+ }
+
+ @Nullable
+ private static Object readArrayPath(JsonReader reader, SimpleJsonPath path, int depth, int expectedIndex) {
+ if (reader.peekToken() != '[') {
+ skipValue(reader);
+ return null;
+ }
+ reader.enterDepth();
+ try {
+ reader.expect('[');
+ if (reader.consume(']')) {
+ return null;
+ }
+ Object result = null;
+ int index = 0;
+ boolean more;
+ do {
+ if (index == expectedIndex) {
+ result = depth + 1 == path.length() ? readScalar(reader) : readPath(reader, path, depth + 1);
+ } else {
+ skipValue(reader);
+ }
+ index++;
+ more = reader.consumeCommaOrEndArray();
+ } while (more);
+ return result;
+ } finally {
+ reader.exitDepth();
+ }
+ }
+
+ @Nullable
+ private static Object readScalar(JsonReader reader) {
+ char token = reader.peekToken();
+ if (token == '"') {
+ String value = reader.readString();
+ if (value.length() > JACKSON_CONSTRAINTS.getMaxStringLength()) {
+ throw new IllegalArgumentException("JSON string exceeds Jackson's configured limit");
+ }
+ return value;
+ }
+ if (token == 't' || token == 'f') {
+ return reader.readBoolean();
+ }
+ if (token == 'n') {
+ reader.readNull();
+ return null;
+ }
+ if (token == '{' || token == '[') {
+ // Query scalar coercion has observable error/default behavior for containers. Let Jayway produce the exact
+ // reference value rather than materializing a Fory container on this uncommon path.
+ throw new IllegalArgumentException("Container result requires reference JSON parsing");
+ }
+ int start = reader.position();
+ Number value = reader.readNumber();
+ if (reader.position() - start > JACKSON_CONSTRAINTS.getMaxNumberLength()) {
+ throw new IllegalArgumentException("JSON number exceeds Jackson's configured limit");
+ }
+ return value;
+ }
+
+ private static void skipValue(JsonReader reader) {
+ char token = reader.peekToken();
+ if (token == '{') {
+ skipObject(reader);
+ return;
+ }
+ if (token == '[') {
+ skipArray(reader);
+ return;
+ }
+ if (token == '"') {
+ // Fory 1.6's skipValue() computes an FNV hash over every character. Its string decoder uses packed scans and
+ // is substantially faster even when the decoded value is discarded. An upstream fast-skip API could remove
+ // this temporary allocation in a future Fory version.
+ String value = reader.readString();
+ if (value.length() > JACKSON_CONSTRAINTS.getMaxStringLength()) {
+ throw new IllegalArgumentException("JSON string exceeds Jackson's configured limit");
+ }
+ return;
+ }
+ int start = reader.position();
+ reader.skipValue();
+ int rawLength = reader.position() - start;
+ if (token != 't' && token != 'f' && token != 'n'
+ && rawLength > JACKSON_CONSTRAINTS.getMaxNumberLength()) {
+ throw new IllegalArgumentException("JSON number exceeds Jackson's configured limit");
+ }
+ }
+
+ private static void skipObject(JsonReader reader) {
+ reader.enterDepth();
Review Comment:
`skipObject` and `skipArray` also increase the depth counter. So a deep subtree that the path never enters still disables the fast path for the whole row.
`ForyJson.DEFAULT_MAX_DEPTH` is 20, and the builder at line 79 does not change it. Jayway allows 1000 levels.
I verified this on your branch. The document `{"a":1,"b":<21 nested objects>}` with path `$.a` throws `ForyJsonException: JSON max depth 20 exceeded at JSON position 106`.
A limit of 20 is low for real JSON, and one deep field anywhere in the document removes the benefit for every row. Please raise the limit with the builder, or write the number 20 in the class Javadoc.
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/JsonExtractScalarTransformFunction.java:
##########
@@ -72,6 +74,10 @@
/// keys to the first non-null occurrence and does not validate malformed content after the resolved value. Use it
/// only for well-formed, duplicate-free JSON. `Fast` scans the full root value and retains Jayway's last-key-wins
/// and malformed-document behavior; see [FastJsonPathExtractor] for one documented unaddressed-value edge case.
+/// `jsonExtractScalarFory` is experimental and must be selected explicitly. It accelerates simple paths over
Review Comment:
This paragraph does not match the code. `getResultExtractor` gates only `useBigDecimal`. `STRING` and `JSON` result types both use Fory.
For a `STRING` result type the fast path works and is faster, so the text is too modest there. For a `JSON` result type the value is usually an object or an array, so Fory throws for each row and Jayway parses the document a second time.
##########
pinot-common/src/main/java/org/apache/pinot/common/function/ForyJsonPathExtractor.java:
##########
@@ -0,0 +1,439 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.function;
+
+import com.fasterxml.jackson.core.StreamReadConstraints;
+import java.util.concurrent.atomic.AtomicBoolean;
+import javax.annotation.Nullable;
+import org.apache.fory.json.ForyJson;
+import org.apache.fory.json.codec.JsonValueCodec;
+import org.apache.fory.json.reader.JsonReader;
+import org.apache.fory.json.reader.Latin1JsonReader;
+import org.apache.fory.json.reader.Utf16JsonReader;
+import org.apache.fory.json.reader.Utf8JsonReader;
+import org.apache.fory.json.writer.StringJsonWriter;
+import org.apache.fory.json.writer.Utf8JsonWriter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/// Shared Fory JSON parser for opt-in JSON-path implementations.
+///
+/// Streaming extraction uses one single-state parser per worker thread to avoid shared-pool contention. Initialization
+/// or runtime linkage failures permanently disable the optional path, allowing callers to fall back to
+/// Jackson/Jayway. Jackson's default scalar token limits are enforced while walking the document, and inputs outside
+/// Fory's safe nesting depth fall back to the reference parser.
+public final class ForyJsonPathExtractor {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ForyJsonPathExtractor.class);
+ private static final StreamReadConstraints JACKSON_CONSTRAINTS = StreamReadConstraints.defaults();
+ private static final ThreadLocal<PathContext> PATH_CONTEXT = ThreadLocal.withInitial(PathContext::new);
+ private static final AtomicBoolean UNAVAILABLE_WARNING_LOGGED = new AtomicBoolean();
+
+ private ForyJsonPathExtractor() {
+ }
+
+ private static final class Holder {
+ private static volatile boolean _streamingAvailable;
+ private static final ThreadLocal<ForyJson> STREAMING_PARSER = ThreadLocal.withInitial(() -> {
+ if (!_streamingAvailable) {
+ throw new IllegalStateException("Fory JSON is unavailable");
+ }
+ ForyJson parser = buildStreamingParser();
+ if (parser == null) {
+ _streamingAvailable = false;
+ throw new IllegalStateException("Fory JSON is unavailable");
+ }
+ return parser;
+ });
+
+ static {
+ ForyJson parser = buildStreamingParser();
+ _streamingAvailable = parser != null;
+ if (parser != null) {
+ STREAMING_PARSER.set(parser);
+ }
+ }
+
+ private Holder() {
+ }
+
+ @Nullable
+ private static ForyJson buildStreamingParser() {
+ try {
+ return ForyJson.builder().withCodegen(false).withAsyncCompilation(false).withConcurrencyLevel(1)
+ .registerCodec(PathResult.class, PathCodec.INSTANCE).build();
+ } catch (RuntimeException | LinkageError e) {
+ logUnavailable(e);
+ return null;
+ }
+ }
+ }
+
+ /// Returns whether the optional Fory runtime initialized successfully.
+ public static boolean isAvailable() {
+ return Holder._streamingAvailable;
+ }
+
+ /// Extracts a simple path with Fory's streaming reader without materializing the complete JSON tree.
+ ///
+ /// Unrelated values are still fully consumed so malformed input and duplicate-key last-wins behavior match the
+ /// reference parser. Jackson's nesting, field-name, string, and number limits are checked while scanning. Callers
+ /// should retry with the reference parser when this method throws.
+ @Nullable
+ public static Object extract(String json, SimpleJsonPath path) {
+ if ((JACKSON_CONSTRAINTS.hasMaxDocumentLength()
+ && json.length() > JACKSON_CONSTRAINTS.getMaxDocumentLength())
+ || (JACKSON_CONSTRAINTS.hasMaxTokenCount() && requiresJacksonFallback(json))) {
+ throw new IllegalArgumentException("JSON document requires Jackson constraint validation");
+ }
+ if (!Holder._streamingAvailable) {
+ throw new IllegalStateException("Fory JSON is unavailable");
+ }
+ ForyJson parser = Holder.STREAMING_PARSER.get();
+ PathContext context = PATH_CONTEXT.get();
+ if (context._active) {
+ throw new IllegalStateException("Fory JSON path extraction is not reentrant");
+ }
+ context._active = true;
+ context._path = path;
+ context._result = null;
+ try {
+ parser.fromJson(json, PathResult.class);
+ return context._result;
+ } catch (LinkageError e) {
+ disable();
+ logUnavailable(e);
+ throw new IllegalStateException("Fory JSON became unavailable", e);
+ } finally {
+ context._path = null;
+ context._result = null;
+ context._active = false;
+ }
+ }
+
+ private static void disable() {
+ Holder._streamingAvailable = false;
+ Holder.STREAMING_PARSER.remove();
+ PATH_CONTEXT.remove();
+ }
+
+ private static void logUnavailable(Throwable cause) {
+ if (UNAVAILABLE_WARNING_LOGGED.compareAndSet(false, true)) {
+ LOGGER.warn("Experimental Fory JSON support is unavailable; falling back to Jackson/Jayway", cause);
+ }
+ }
+
+ private static Object readPath(JsonReader reader, SimpleJsonPath path, int depth) {
+ String key = path.getKey(depth);
+ if (key != null) {
+ return readObjectPath(reader, path, depth, key);
+ }
+ return readArrayPath(reader, path, depth, path.getIndex(depth));
+ }
+
+ @Nullable
+ private static Object readObjectPath(JsonReader reader, SimpleJsonPath path, int depth, String expectedKey) {
+ if (reader.peekToken() != '{') {
+ skipValue(reader);
+ return null;
+ }
+ reader.enterDepth();
+ try {
+ reader.expect('{');
+ if (reader.consume('}')) {
+ return null;
+ }
+ Object result = null;
+ boolean more;
+ do {
+ String fieldName = reader.readFieldName();
+ if (fieldName.length() > JACKSON_CONSTRAINTS.getMaxNameLength()) {
+ throw new IllegalArgumentException("JSON field name exceeds Jackson's configured limit");
+ }
+ reader.expect(':');
+ if (expectedKey.equals(fieldName)) {
+ result = depth + 1 == path.length() ? readScalar(reader) : readPath(reader, path, depth + 1);
+ } else {
+ skipValue(reader);
+ }
+ more = reader.consumeCommaOrEndObject();
+ } while (more);
+ return result;
+ } finally {
+ reader.exitDepth();
+ }
+ }
+
+ @Nullable
+ private static Object readArrayPath(JsonReader reader, SimpleJsonPath path, int depth, int expectedIndex) {
+ if (reader.peekToken() != '[') {
+ skipValue(reader);
+ return null;
+ }
+ reader.enterDepth();
+ try {
+ reader.expect('[');
+ if (reader.consume(']')) {
+ return null;
+ }
+ Object result = null;
+ int index = 0;
+ boolean more;
+ do {
+ if (index == expectedIndex) {
+ result = depth + 1 == path.length() ? readScalar(reader) : readPath(reader, path, depth + 1);
+ } else {
+ skipValue(reader);
+ }
+ index++;
+ more = reader.consumeCommaOrEndArray();
+ } while (more);
+ return result;
+ } finally {
+ reader.exitDepth();
+ }
+ }
+
+ @Nullable
+ private static Object readScalar(JsonReader reader) {
+ char token = reader.peekToken();
+ if (token == '"') {
+ String value = reader.readString();
+ if (value.length() > JACKSON_CONSTRAINTS.getMaxStringLength()) {
+ throw new IllegalArgumentException("JSON string exceeds Jackson's configured limit");
+ }
+ return value;
+ }
+ if (token == 't' || token == 'f') {
+ return reader.readBoolean();
+ }
+ if (token == 'n') {
+ reader.readNull();
+ return null;
+ }
+ if (token == '{' || token == '[') {
+ // Query scalar coercion has observable error/default behavior for containers. Let Jayway produce the exact
+ // reference value rather than materializing a Fory container on this uncommon path.
+ throw new IllegalArgumentException("Container result requires reference JSON parsing");
Review Comment:
**Blocker.** The fallback throws a new exception for each row. The Fory function is then slower than the function that it replaces.
I measured this on your branch with JDK 25.0.3, 200,000 iterations for each case, best of two runs after warmup:
| document | path | Fory | Jayway | result |
| --- | --- | ---: | ---: | --- |
| `{"a":"v","b":"w"}` | `$.a` | 0.06 us/op | 0.35 us/op | 5.8x faster |
| `{"a":{"b":1},"b":"w"}` | `$.a` | 1.92 us/op | 0.51 us/op | **3.8x slower** |
| `{"a":[1,2,3],"b":"w"}` | `$.a` | 1.63 us/op | 0.48 us/op | **3.4x slower** |
| 25-level nested document | `$.k` x25 | 5.24 us/op | 2.19 us/op | **2.4x slower** |
The cost has two parts. `new IllegalArgumentException(...)` fills in a stack trace, and then Jayway parses the same document a second time.
Two changes remove most of this cost. First, return a sentinel value from `readScalar` instead of a new exception, because a container result is normal control flow and not an error. Second, route `JSON` result types to Jayway once in `init()`, because `_dataType` is known there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add experimental Fory JSON functions [pinot]
Posted by "yashmayya (via GitHub)" <gi...@apache.org>.
yashmayya commented on code in PR #19223:
URL: https://github.com/apache/pinot/pull/19223#discussion_r3771202463
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/JsonExtractScalarTransformFunction.java:
##########
@@ -241,6 +260,15 @@ public void init(List<TransformFunction> arguments, Map<String, ColumnContext> c
}
}
_resultMetadata = new TransformResultMetadata(_dataType, isSingleValue, false);
+ DataType inputDataType = firstArgument.getResultMetadata().getDataType();
+ _foryEligible = _extractionMode == ExtractionMode.FORY && _simpleJsonPath != null
+ && inputDataType == DataType.STRING && isSingleValue && _dataType != DataType.STRING
+ && _dataType != DataType.JSON && _dataType != DataType.BIG_DECIMAL;
Review Comment:
`_dataType != DataType.STRING` removes the most common case from the fast path.
The scalar function `jsonPathStringFory` still sends string values through Fory. So the same extraction is accelerated during ingestion, but not during a query.
My differential run over 83 documents found no difference between `jsonPathStringFory` and `jsonPathString`, and that function does use Fory for strings. My timing run shows the largest win on exactly this shape: a string leaf in a 25-level document measured 0.33 us/op against 2.09 us/op for Jayway.
What is the reason to exclude `STRING` here? If a `toString` coercion case breaks, one test can show it, and then the exclusion is clearly correct. If there is no such case, this line gives up the main benefit of the change.
##########
pinot-common/src/main/java/org/apache/pinot/common/function/ForyJsonPathExtractor.java:
##########
@@ -0,0 +1,388 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.function;
+
+import com.fasterxml.jackson.core.StreamReadConstraints;
+import java.util.concurrent.atomic.AtomicBoolean;
+import javax.annotation.Nullable;
+import org.apache.fory.json.ForyJson;
+import org.apache.fory.json.codec.JsonValueCodec;
+import org.apache.fory.json.reader.JsonReader;
+import org.apache.fory.json.reader.Latin1JsonReader;
+import org.apache.fory.json.reader.Utf16JsonReader;
+import org.apache.fory.json.reader.Utf8JsonReader;
+import org.apache.fory.json.writer.StringJsonWriter;
+import org.apache.fory.json.writer.Utf8JsonWriter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/// Shared Fory JSON parser for opt-in JSON-path implementations.
+///
+/// Streaming extraction uses one bounded parser pool shared by all worker threads. Initialization or runtime linkage
+/// failures permanently disable the optional path, allowing callers to fall back to Jackson/Jayway. Jackson's
+/// document, token, nesting, field-name, string, and number constraints are enforced while walking the document.
+/// The Fory runtime is an optional dependency; applications must add it to the application classpath to enable this
+/// experimental path.
+public final class ForyJsonPathExtractor {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ForyJsonPathExtractor.class);
+ private static final StreamReadConstraints JACKSON_CONSTRAINTS = StreamReadConstraints.defaults();
+ private static final ThreadLocal<PathContext> PATH_CONTEXT = ThreadLocal.withInitial(PathContext::new);
+ private static final AtomicBoolean UNAVAILABLE_WARNING_LOGGED = new AtomicBoolean();
+ private static final Object FALLBACK_REQUIRED = new Object();
+
+ private ForyJsonPathExtractor() {
+ }
+
+ private static final class Holder {
+ @Nullable
+ private static final ForyJson STREAMING_PARSER = buildStreamingParser();
+ private static volatile boolean _streamingAvailable = STREAMING_PARSER != null;
+
+ private Holder() {
+ }
+
+ @Nullable
+ private static ForyJson buildStreamingParser() {
+ try {
+ return ForyJson.builder().withCodegen(false).withAsyncCompilation(false)
+ .maxDepth(JACKSON_CONSTRAINTS.getMaxNestingDepth())
+ .registerCodec(PathResult.class, PathCodec.INSTANCE).build();
+ } catch (RuntimeException | LinkageError e) {
+ logUnavailable(e);
+ return null;
+ }
+ }
+ }
+
+ /// Returns whether the optional Fory runtime initialized successfully.
+ public static boolean isAvailable() {
+ return Holder._streamingAvailable;
+ }
+
+ /// Returns whether an extracted value requires the reference parser for exact container/coercion semantics.
+ public static boolean isFallbackRequired(@Nullable Object value) {
Review Comment:
The sentinel works. I ran 36 documents through both call paths, including duplicate keys in each order, containers under a nested path, malformed tails, and depths from 5 to 1001. There were no leaks and no differences against Jayway.
The API shape stays fragile. `extract` returns an `Object` that can be a real value, `null`, or a private marker. A caller that forgets `isFallbackRequired` writes `java.lang.Object@1a2b3c` into user data, and no test catches it.
A small result holder, or a second method such as `extractOrNull` plus `extractContainerFlag`, removes that risk. This is not urgent, because there are only two call sites today.
##########
pinot-common/pom.xml:
##########
@@ -280,6 +293,11 @@
<groupId>com.jayway.jsonpath</groupId>
<artifactId>json-path</artifactId>
</dependency>
+ <dependency>
+ <groupId>org.apache.fory</groupId>
+ <artifactId>fory-json</artifactId>
+ <optional>true</optional>
Review Comment:
This is the right call for an experiment, and I verified that it works. The shaded jar contains no Fory classes, and an optional dependency never reaches `pinot-distribution`.
It also means that a standard Pinot build cannot run the fast path. All four functions return the Jayway result, and the operator sees one WARN line.
Two requests.
First, name the exact artifact where a user can find it, for example `org.apache.fory:fory-json:1.6.0`. The Javadoc says only "add it to the application classpath", which does not tell the user what to add.
Second, please state this in the PR description, above the benchmark tables. A reader can expect those numbers from a standard build today.
Note that `pinot-integration-tests` declares Fory in test scope. So `JsonPathTest` asserts the accelerated path in a configuration that the distribution does not ship. The child JVM tests cover the shipped configuration, so the gap is small, but it is worth knowing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add experimental Fory JSON functions [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #19223:
URL: https://github.com/apache/pinot/pull/19223#discussion_r3771083743
##########
pinot-common/src/main/java/org/apache/pinot/common/function/ForyJsonPathExtractor.java:
##########
@@ -0,0 +1,439 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.function;
+
+import com.fasterxml.jackson.core.StreamReadConstraints;
+import java.util.concurrent.atomic.AtomicBoolean;
+import javax.annotation.Nullable;
+import org.apache.fory.json.ForyJson;
+import org.apache.fory.json.codec.JsonValueCodec;
+import org.apache.fory.json.reader.JsonReader;
+import org.apache.fory.json.reader.Latin1JsonReader;
+import org.apache.fory.json.reader.Utf16JsonReader;
+import org.apache.fory.json.reader.Utf8JsonReader;
+import org.apache.fory.json.writer.StringJsonWriter;
+import org.apache.fory.json.writer.Utf8JsonWriter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/// Shared Fory JSON parser for opt-in JSON-path implementations.
+///
+/// Streaming extraction uses one single-state parser per worker thread to avoid shared-pool contention. Initialization
+/// or runtime linkage failures permanently disable the optional path, allowing callers to fall back to
+/// Jackson/Jayway. Jackson's default scalar token limits are enforced while walking the document, and inputs outside
+/// Fory's safe nesting depth fall back to the reference parser.
+public final class ForyJsonPathExtractor {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ForyJsonPathExtractor.class);
+ private static final StreamReadConstraints JACKSON_CONSTRAINTS = StreamReadConstraints.defaults();
+ private static final ThreadLocal<PathContext> PATH_CONTEXT = ThreadLocal.withInitial(PathContext::new);
+ private static final AtomicBoolean UNAVAILABLE_WARNING_LOGGED = new AtomicBoolean();
+
+ private ForyJsonPathExtractor() {
+ }
+
+ private static final class Holder {
+ private static volatile boolean _streamingAvailable;
+ private static final ThreadLocal<ForyJson> STREAMING_PARSER = ThreadLocal.withInitial(() -> {
+ if (!_streamingAvailable) {
+ throw new IllegalStateException("Fory JSON is unavailable");
+ }
+ ForyJson parser = buildStreamingParser();
+ if (parser == null) {
+ _streamingAvailable = false;
+ throw new IllegalStateException("Fory JSON is unavailable");
+ }
+ return parser;
+ });
+
+ static {
+ ForyJson parser = buildStreamingParser();
+ _streamingAvailable = parser != null;
+ if (parser != null) {
+ STREAMING_PARSER.set(parser);
+ }
+ }
+
+ private Holder() {
+ }
+
+ @Nullable
+ private static ForyJson buildStreamingParser() {
+ try {
+ return ForyJson.builder().withCodegen(false).withAsyncCompilation(false).withConcurrencyLevel(1)
+ .registerCodec(PathResult.class, PathCodec.INSTANCE).build();
+ } catch (RuntimeException | LinkageError e) {
+ logUnavailable(e);
+ return null;
+ }
+ }
+ }
+
+ /// Returns whether the optional Fory runtime initialized successfully.
+ public static boolean isAvailable() {
+ return Holder._streamingAvailable;
+ }
+
+ /// Extracts a simple path with Fory's streaming reader without materializing the complete JSON tree.
+ ///
+ /// Unrelated values are still fully consumed so malformed input and duplicate-key last-wins behavior match the
+ /// reference parser. Jackson's nesting, field-name, string, and number limits are checked while scanning. Callers
+ /// should retry with the reference parser when this method throws.
+ @Nullable
+ public static Object extract(String json, SimpleJsonPath path) {
+ if ((JACKSON_CONSTRAINTS.hasMaxDocumentLength()
+ && json.length() > JACKSON_CONSTRAINTS.getMaxDocumentLength())
+ || (JACKSON_CONSTRAINTS.hasMaxTokenCount() && requiresJacksonFallback(json))) {
+ throw new IllegalArgumentException("JSON document requires Jackson constraint validation");
+ }
+ if (!Holder._streamingAvailable) {
+ throw new IllegalStateException("Fory JSON is unavailable");
+ }
+ ForyJson parser = Holder.STREAMING_PARSER.get();
+ PathContext context = PATH_CONTEXT.get();
+ if (context._active) {
+ throw new IllegalStateException("Fory JSON path extraction is not reentrant");
+ }
+ context._active = true;
+ context._path = path;
+ context._result = null;
+ try {
+ parser.fromJson(json, PathResult.class);
+ return context._result;
+ } catch (LinkageError e) {
+ disable();
+ logUnavailable(e);
+ throw new IllegalStateException("Fory JSON became unavailable", e);
+ } finally {
+ context._path = null;
+ context._result = null;
+ context._active = false;
+ }
+ }
+
+ private static void disable() {
+ Holder._streamingAvailable = false;
+ Holder.STREAMING_PARSER.remove();
+ PATH_CONTEXT.remove();
+ }
+
+ private static void logUnavailable(Throwable cause) {
+ if (UNAVAILABLE_WARNING_LOGGED.compareAndSet(false, true)) {
+ LOGGER.warn("Experimental Fory JSON support is unavailable; falling back to Jackson/Jayway", cause);
+ }
+ }
+
+ private static Object readPath(JsonReader reader, SimpleJsonPath path, int depth) {
+ String key = path.getKey(depth);
+ if (key != null) {
+ return readObjectPath(reader, path, depth, key);
+ }
+ return readArrayPath(reader, path, depth, path.getIndex(depth));
+ }
+
+ @Nullable
+ private static Object readObjectPath(JsonReader reader, SimpleJsonPath path, int depth, String expectedKey) {
+ if (reader.peekToken() != '{') {
+ skipValue(reader);
+ return null;
+ }
+ reader.enterDepth();
+ try {
+ reader.expect('{');
+ if (reader.consume('}')) {
+ return null;
+ }
+ Object result = null;
+ boolean more;
+ do {
+ String fieldName = reader.readFieldName();
+ if (fieldName.length() > JACKSON_CONSTRAINTS.getMaxNameLength()) {
+ throw new IllegalArgumentException("JSON field name exceeds Jackson's configured limit");
+ }
+ reader.expect(':');
+ if (expectedKey.equals(fieldName)) {
+ result = depth + 1 == path.length() ? readScalar(reader) : readPath(reader, path, depth + 1);
+ } else {
+ skipValue(reader);
+ }
+ more = reader.consumeCommaOrEndObject();
+ } while (more);
+ return result;
+ } finally {
+ reader.exitDepth();
+ }
+ }
+
+ @Nullable
+ private static Object readArrayPath(JsonReader reader, SimpleJsonPath path, int depth, int expectedIndex) {
+ if (reader.peekToken() != '[') {
+ skipValue(reader);
+ return null;
+ }
+ reader.enterDepth();
+ try {
+ reader.expect('[');
+ if (reader.consume(']')) {
+ return null;
+ }
+ Object result = null;
+ int index = 0;
+ boolean more;
+ do {
+ if (index == expectedIndex) {
+ result = depth + 1 == path.length() ? readScalar(reader) : readPath(reader, path, depth + 1);
+ } else {
+ skipValue(reader);
+ }
+ index++;
+ more = reader.consumeCommaOrEndArray();
+ } while (more);
+ return result;
+ } finally {
+ reader.exitDepth();
+ }
+ }
+
+ @Nullable
+ private static Object readScalar(JsonReader reader) {
+ char token = reader.peekToken();
+ if (token == '"') {
+ String value = reader.readString();
+ if (value.length() > JACKSON_CONSTRAINTS.getMaxStringLength()) {
+ throw new IllegalArgumentException("JSON string exceeds Jackson's configured limit");
+ }
+ return value;
+ }
+ if (token == 't' || token == 'f') {
+ return reader.readBoolean();
+ }
+ if (token == 'n') {
+ reader.readNull();
+ return null;
+ }
+ if (token == '{' || token == '[') {
+ // Query scalar coercion has observable error/default behavior for containers. Let Jayway produce the exact
+ // reference value rather than materializing a Fory container on this uncommon path.
+ throw new IllegalArgumentException("Container result requires reference JSON parsing");
+ }
+ int start = reader.position();
+ Number value = reader.readNumber();
+ if (reader.position() - start > JACKSON_CONSTRAINTS.getMaxNumberLength()) {
+ throw new IllegalArgumentException("JSON number exceeds Jackson's configured limit");
+ }
+ return value;
+ }
+
+ private static void skipValue(JsonReader reader) {
+ char token = reader.peekToken();
+ if (token == '{') {
+ skipObject(reader);
+ return;
+ }
+ if (token == '[') {
+ skipArray(reader);
+ return;
+ }
+ if (token == '"') {
+ // Fory 1.6's skipValue() computes an FNV hash over every character. Its string decoder uses packed scans and
+ // is substantially faster even when the decoded value is discarded. An upstream fast-skip API could remove
+ // this temporary allocation in a future Fory version.
+ String value = reader.readString();
+ if (value.length() > JACKSON_CONSTRAINTS.getMaxStringLength()) {
+ throw new IllegalArgumentException("JSON string exceeds Jackson's configured limit");
+ }
+ return;
+ }
+ int start = reader.position();
+ reader.skipValue();
+ int rawLength = reader.position() - start;
+ if (token != 't' && token != 'f' && token != 'n'
+ && rawLength > JACKSON_CONSTRAINTS.getMaxNumberLength()) {
+ throw new IllegalArgumentException("JSON number exceeds Jackson's configured limit");
+ }
+ }
+
+ private static void skipObject(JsonReader reader) {
+ reader.enterDepth();
Review Comment:
Fixed in f0f1036c99. The builder now uses Jackson's configured max nesting depth rather than Fory's default 20. Direct-extractor tests cover selected and unrelated depth-25 documents plus the 1000/1001 boundary. The new direct-path JMH reports 13.9M/17.1M ops/s and about 48 B/op for selected/unrelated deep cases at T32.
##########
pinot-common/src/main/java/org/apache/pinot/common/function/ForyJsonPathExtractor.java:
##########
@@ -0,0 +1,439 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.function;
+
+import com.fasterxml.jackson.core.StreamReadConstraints;
+import java.util.concurrent.atomic.AtomicBoolean;
+import javax.annotation.Nullable;
+import org.apache.fory.json.ForyJson;
+import org.apache.fory.json.codec.JsonValueCodec;
+import org.apache.fory.json.reader.JsonReader;
+import org.apache.fory.json.reader.Latin1JsonReader;
+import org.apache.fory.json.reader.Utf16JsonReader;
+import org.apache.fory.json.reader.Utf8JsonReader;
+import org.apache.fory.json.writer.StringJsonWriter;
+import org.apache.fory.json.writer.Utf8JsonWriter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/// Shared Fory JSON parser for opt-in JSON-path implementations.
+///
+/// Streaming extraction uses one single-state parser per worker thread to avoid shared-pool contention. Initialization
+/// or runtime linkage failures permanently disable the optional path, allowing callers to fall back to
+/// Jackson/Jayway. Jackson's default scalar token limits are enforced while walking the document, and inputs outside
+/// Fory's safe nesting depth fall back to the reference parser.
+public final class ForyJsonPathExtractor {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ForyJsonPathExtractor.class);
+ private static final StreamReadConstraints JACKSON_CONSTRAINTS = StreamReadConstraints.defaults();
+ private static final ThreadLocal<PathContext> PATH_CONTEXT = ThreadLocal.withInitial(PathContext::new);
+ private static final AtomicBoolean UNAVAILABLE_WARNING_LOGGED = new AtomicBoolean();
+
+ private ForyJsonPathExtractor() {
+ }
+
+ private static final class Holder {
+ private static volatile boolean _streamingAvailable;
+ private static final ThreadLocal<ForyJson> STREAMING_PARSER = ThreadLocal.withInitial(() -> {
+ if (!_streamingAvailable) {
+ throw new IllegalStateException("Fory JSON is unavailable");
+ }
+ ForyJson parser = buildStreamingParser();
+ if (parser == null) {
+ _streamingAvailable = false;
+ throw new IllegalStateException("Fory JSON is unavailable");
+ }
+ return parser;
+ });
+
+ static {
+ ForyJson parser = buildStreamingParser();
+ _streamingAvailable = parser != null;
+ if (parser != null) {
+ STREAMING_PARSER.set(parser);
+ }
+ }
+
+ private Holder() {
+ }
+
+ @Nullable
+ private static ForyJson buildStreamingParser() {
+ try {
+ return ForyJson.builder().withCodegen(false).withAsyncCompilation(false).withConcurrencyLevel(1)
+ .registerCodec(PathResult.class, PathCodec.INSTANCE).build();
+ } catch (RuntimeException | LinkageError e) {
+ logUnavailable(e);
+ return null;
+ }
+ }
+ }
+
+ /// Returns whether the optional Fory runtime initialized successfully.
+ public static boolean isAvailable() {
+ return Holder._streamingAvailable;
+ }
+
+ /// Extracts a simple path with Fory's streaming reader without materializing the complete JSON tree.
+ ///
+ /// Unrelated values are still fully consumed so malformed input and duplicate-key last-wins behavior match the
+ /// reference parser. Jackson's nesting, field-name, string, and number limits are checked while scanning. Callers
+ /// should retry with the reference parser when this method throws.
+ @Nullable
+ public static Object extract(String json, SimpleJsonPath path) {
+ if ((JACKSON_CONSTRAINTS.hasMaxDocumentLength()
+ && json.length() > JACKSON_CONSTRAINTS.getMaxDocumentLength())
+ || (JACKSON_CONSTRAINTS.hasMaxTokenCount() && requiresJacksonFallback(json))) {
+ throw new IllegalArgumentException("JSON document requires Jackson constraint validation");
+ }
+ if (!Holder._streamingAvailable) {
+ throw new IllegalStateException("Fory JSON is unavailable");
+ }
+ ForyJson parser = Holder.STREAMING_PARSER.get();
+ PathContext context = PATH_CONTEXT.get();
+ if (context._active) {
+ throw new IllegalStateException("Fory JSON path extraction is not reentrant");
+ }
+ context._active = true;
+ context._path = path;
+ context._result = null;
+ try {
+ parser.fromJson(json, PathResult.class);
+ return context._result;
+ } catch (LinkageError e) {
+ disable();
+ logUnavailable(e);
+ throw new IllegalStateException("Fory JSON became unavailable", e);
+ } finally {
+ context._path = null;
+ context._result = null;
+ context._active = false;
+ }
+ }
+
+ private static void disable() {
+ Holder._streamingAvailable = false;
+ Holder.STREAMING_PARSER.remove();
+ PATH_CONTEXT.remove();
+ }
+
+ private static void logUnavailable(Throwable cause) {
+ if (UNAVAILABLE_WARNING_LOGGED.compareAndSet(false, true)) {
+ LOGGER.warn("Experimental Fory JSON support is unavailable; falling back to Jackson/Jayway", cause);
+ }
+ }
+
+ private static Object readPath(JsonReader reader, SimpleJsonPath path, int depth) {
+ String key = path.getKey(depth);
+ if (key != null) {
+ return readObjectPath(reader, path, depth, key);
+ }
+ return readArrayPath(reader, path, depth, path.getIndex(depth));
+ }
+
+ @Nullable
+ private static Object readObjectPath(JsonReader reader, SimpleJsonPath path, int depth, String expectedKey) {
+ if (reader.peekToken() != '{') {
+ skipValue(reader);
+ return null;
+ }
+ reader.enterDepth();
+ try {
+ reader.expect('{');
+ if (reader.consume('}')) {
+ return null;
+ }
+ Object result = null;
+ boolean more;
+ do {
+ String fieldName = reader.readFieldName();
+ if (fieldName.length() > JACKSON_CONSTRAINTS.getMaxNameLength()) {
+ throw new IllegalArgumentException("JSON field name exceeds Jackson's configured limit");
+ }
+ reader.expect(':');
+ if (expectedKey.equals(fieldName)) {
+ result = depth + 1 == path.length() ? readScalar(reader) : readPath(reader, path, depth + 1);
+ } else {
+ skipValue(reader);
+ }
+ more = reader.consumeCommaOrEndObject();
+ } while (more);
+ return result;
+ } finally {
+ reader.exitDepth();
+ }
+ }
+
+ @Nullable
+ private static Object readArrayPath(JsonReader reader, SimpleJsonPath path, int depth, int expectedIndex) {
+ if (reader.peekToken() != '[') {
+ skipValue(reader);
+ return null;
+ }
+ reader.enterDepth();
+ try {
+ reader.expect('[');
+ if (reader.consume(']')) {
+ return null;
+ }
+ Object result = null;
+ int index = 0;
+ boolean more;
+ do {
+ if (index == expectedIndex) {
+ result = depth + 1 == path.length() ? readScalar(reader) : readPath(reader, path, depth + 1);
+ } else {
+ skipValue(reader);
+ }
+ index++;
+ more = reader.consumeCommaOrEndArray();
+ } while (more);
+ return result;
+ } finally {
+ reader.exitDepth();
+ }
+ }
+
+ @Nullable
+ private static Object readScalar(JsonReader reader) {
+ char token = reader.peekToken();
+ if (token == '"') {
+ String value = reader.readString();
+ if (value.length() > JACKSON_CONSTRAINTS.getMaxStringLength()) {
+ throw new IllegalArgumentException("JSON string exceeds Jackson's configured limit");
+ }
+ return value;
+ }
+ if (token == 't' || token == 'f') {
+ return reader.readBoolean();
+ }
+ if (token == 'n') {
+ reader.readNull();
+ return null;
+ }
+ if (token == '{' || token == '[') {
+ // Query scalar coercion has observable error/default behavior for containers. Let Jayway produce the exact
+ // reference value rather than materializing a Fory container on this uncommon path.
+ throw new IllegalArgumentException("Container result requires reference JSON parsing");
+ }
+ int start = reader.position();
+ Number value = reader.readNumber();
+ if (reader.position() - start > JACKSON_CONSTRAINTS.getMaxNumberLength()) {
+ throw new IllegalArgumentException("JSON number exceeds Jackson's configured limit");
+ }
+ return value;
+ }
+
+ private static void skipValue(JsonReader reader) {
+ char token = reader.peekToken();
+ if (token == '{') {
+ skipObject(reader);
+ return;
+ }
+ if (token == '[') {
+ skipArray(reader);
+ return;
+ }
+ if (token == '"') {
+ // Fory 1.6's skipValue() computes an FNV hash over every character. Its string decoder uses packed scans and
+ // is substantially faster even when the decoded value is discarded. An upstream fast-skip API could remove
+ // this temporary allocation in a future Fory version.
+ String value = reader.readString();
+ if (value.length() > JACKSON_CONSTRAINTS.getMaxStringLength()) {
+ throw new IllegalArgumentException("JSON string exceeds Jackson's configured limit");
+ }
+ return;
+ }
+ int start = reader.position();
+ reader.skipValue();
+ int rawLength = reader.position() - start;
+ if (token != 't' && token != 'f' && token != 'n'
+ && rawLength > JACKSON_CONSTRAINTS.getMaxNumberLength()) {
+ throw new IllegalArgumentException("JSON number exceeds Jackson's configured limit");
+ }
+ }
+
+ private static void skipObject(JsonReader reader) {
+ reader.enterDepth();
+ try {
+ reader.expect('{');
+ if (reader.consume('}')) {
+ return;
+ }
+ boolean more;
+ do {
+ String fieldName = reader.readFieldName();
+ if (fieldName.length() > JACKSON_CONSTRAINTS.getMaxNameLength()) {
+ throw new IllegalArgumentException("JSON field name exceeds Jackson's configured limit");
+ }
+ reader.expect(':');
+ skipValue(reader);
+ more = reader.consumeCommaOrEndObject();
+ } while (more);
+ } finally {
+ reader.exitDepth();
+ }
+ }
+
+ private static void skipArray(JsonReader reader) {
+ reader.enterDepth();
+ try {
+ reader.expect('[');
+ if (reader.consume(']')) {
+ return;
+ }
+ boolean more;
+ do {
+ skipValue(reader);
+ more = reader.consumeCommaOrEndArray();
+ } while (more);
+ } finally {
+ reader.exitDepth();
+ }
+ }
+
+ private static final class PathContext {
+ private final PathResult _marker = new PathResult();
+ private boolean _active;
+ @Nullable
+ private SimpleJsonPath _path;
+ @Nullable
+ private Object _result;
+ }
+
+ private static final class PathResult {
+ }
+
+ private static final class PathCodec implements JsonValueCodec<PathResult> {
+ private static final PathCodec INSTANCE = new PathCodec();
+
+ @Override
+ public PathResult readLatin1(Latin1JsonReader reader) {
+ return read(reader);
+ }
+
+ @Override
+ public PathResult readUtf16(Utf16JsonReader reader) {
+ return read(reader);
+ }
+
+ @Override
+ public PathResult readUtf8(Utf8JsonReader reader) {
+ return read(reader);
+ }
+
+ private static PathResult read(JsonReader reader) {
+ PathContext context = PATH_CONTEXT.get();
+ SimpleJsonPath path = context._path;
+ if (!context._active || path == null) {
+ throw new IllegalStateException("Missing JSON path extraction context");
+ }
+ context._result = readPath(reader, path, 0);
+ return context._marker;
+ }
+
+ @Override
+ public void writeString(StringJsonWriter writer, PathResult value) {
+ throw new UnsupportedOperationException("PathResult is read-only");
+ }
+
+ @Override
+ public void writeUtf8(Utf8JsonWriter writer, PathResult value) {
+ throw new UnsupportedOperationException("PathResult is read-only");
+ }
+ }
+
+ private static boolean requiresJacksonFallback(String json) {
Review Comment:
Fixed in f0f1036c99. requiresJacksonFallback and its approximate scanner are deleted. Configured token counting is performed as the streaming parser consumes starts/ends, field names, and scalar values; the default-disabled setting is JIT-cheap and an overridden-limit child-JVM test verifies behavior.
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/JsonExtractScalarTransformFunction.java:
##########
@@ -72,6 +74,10 @@
/// keys to the first non-null occurrence and does not validate malformed content after the resolved value. Use it
/// only for well-formed, duplicate-free JSON. `Fast` scans the full root value and retains Jayway's last-key-wins
/// and malformed-document behavior; see [FastJsonPathExtractor] for one documented unaddressed-value edge case.
+/// `jsonExtractScalarFory` is experimental and must be selected explicitly. It accelerates simple paths over
Review Comment:
Resolved in f0f1036c99 with a single initialization-time _foryEligible decision used by all extraction call paths, including null bitmap handling. STRING, JSON, BIG_DECIMAL, every MV result, BYTES input, and complex paths route directly to Jayway. Tests assert the eligibility matrix, and the Javadoc documents the same envelope.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add experimental Fory JSON functions [pinot]
Posted by "yashmayya (via GitHub)" <gi...@apache.org>.
yashmayya commented on code in PR #19223:
URL: https://github.com/apache/pinot/pull/19223#discussion_r3770495041
##########
pinot-common/pom.xml:
##########
@@ -280,6 +280,10 @@
<groupId>com.jayway.jsonpath</groupId>
<artifactId>json-path</artifactId>
</dependency>
+ <dependency>
+ <groupId>org.apache.fory</groupId>
+ <artifactId>fory-json</artifactId>
Review Comment:
On second thoughts, this seems overkill and I'm going to mark the comment as resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add experimental Fory JSON functions [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on PR #19223:
URL: https://github.com/apache/pinot/pull/19223#issuecomment-5277876949
Documentation follow-up: https://github.com/pinot-contrib/pinot-docs/pull/981 (merged).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add experimental Fory JSON functions [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on PR #19223:
URL: https://github.com/apache/pinot/pull/19223#issuecomment-5272178376
## Counterpart audit and expanded benchmark matrix
I audited the public JSON function surface before adding APIs. No production Fory counterpart was missing:
- `jsonPathStringFast(Object, String, String)` has `jsonPathStringFory(Object, String, String)`.
- `jsonPathLongFast(Object, String, long)` has `jsonPathLongFory(Object, String, long)`.
- `jsonPathDoubleFast(Object, String, double)` has `jsonPathDoubleFory(Object, String, double)`.
- `jsonExtractScalar`, `jsonExtractScalarFast`, and `jsonExtractScalarFory` all support the same 3- and 4-argument transform signatures and are registered through the enum, transform factory, parser, and planner.
The normal two-argument scalar overloads do not have Fast overloads either, so omitting Fory two-argument overloads is intentional parity. Generic `jsonPath`, array, and exists functions also have no Fast variants, so they are not Fory gaps.
This follow-up adds:
- production-wrapper JMH trios for String, Long, and Double: normal/Jayway, Fast, and Fory;
- query `ValueBlock` trios for LONG and DOUBLE, with STRING available explicitly as a documented Jayway-fallback characterization;
- early/late, 700 B/8 KiB/64 KiB, hit/missing, typed-default fixtures;
- setup assertions that directly execute Fory, reject unavailable Fory, and then verify wrapper parity, preventing a silent Jayway fallback from being published as Fory performance;
- missing Fast registry/evaluator coverage and corrected Fory result-type documentation.
### Local JMH results
JDK 25.0.4, JMH 1.37, one thread, one fork, late-field hit, `5 x 2s` warmup, `5 x 2s` measurement, GC profiler. Throughput is in thousands of operations/rows per second. These are local one-fork measurements; the large documents intentionally contain one large skipped string and therefore stress Fory 1.6.0's current skip-string allocation behavior.
#### Ingestion scalar functions
| Type | JSON | Normal | Fast | Fory | Fory / Fast |
| --- | ---: | ---: | ---: | ---: | ---: |
| STRING | 700 B | 372.6k | 671.7k | 1,141.4k | 1.70x |
| STRING | 8 KiB | 153.2k | 177.4k | 564.0k | 3.18x |
| STRING | 64 KiB | 10.7k | 26.4k | 127.7k | 4.83x |
| LONG | 700 B | 338.9k | 689.2k | 1,029.5k | 1.49x |
| LONG | 8 KiB | 136.0k | 180.0k | 594.3k | 3.30x |
| LONG | 64 KiB | 10.5k | 26.4k | 94.1k | 3.57x |
| DOUBLE | 700 B | 192.0k | 625.8k | 821.0k | 1.31x |
| DOUBLE | 8 KiB | 153.9k | 177.6k | 548.7k | 3.09x |
| DOUBLE | 64 KiB | 10.8k | 27.0k | 118.8k | 4.39x |
#### Query `jsonExtractScalar*` ValueBlock path
JMH normalizes each 128-row invocation to one row.
| Result type | JSON | Normal | Fast | Fory | Fory / Fast |
| --- | ---: | ---: | ---: | ---: | ---: |
| LONG | 700 B | 316.7k | 730.5k | 760.4k | 1.04x |
| LONG | 8 KiB | 126.8k | 351.5k | 471.5k | 1.34x |
| LONG | 64 KiB | 10.8k | 26.7k | 109.5k | 4.11x |
| DOUBLE | 700 B | 307.1k | 632.4k | 779.0k | 1.23x |
| DOUBLE | 8 KiB | 138.6k | 326.8k | 509.9k | 1.56x |
| DOUBLE | 64 KiB | 10.7k | 26.8k | 109.3k | 4.08x |
`jsonExtractScalarFory(..., 'STRING')` intentionally uses Jayway for precision/serialization parity, so it is not reported as Fory acceleration. The same is true for JSON, BIG_DECIMAL, arrays/containers, BYTES input, and complex paths.
### Allocation caveat
Fory scalar allocation was approximately 1.1-1.2 KiB at 700 B, 8.6-8.7 KiB at 8 KiB, and 66 KiB at 64 KiB, versus roughly 0.7-1.0 KiB for Fast. Fory 1.6.0 materializes large irrelevant strings because its public allocation-free skip path hashes every character and is substantially slower. Throughput is favorable here, but an upstream fast allocation-free string-skip API remains the main requirement before recommending broad production adoption.
### Validation
- Focused common/core suites: 817 tests, 0 failures or errors.
- Full `pinot-perf -am package -DskipTests`: 64 modules, build success.
- Spotless, Checkstyle, license format, and license check passed for `pinot-common`, `pinot-core`, and `pinot-perf`.
- `git diff --check` clean.
Commit: `62c2db4b670153709edc03eb98bf70fcb53b5880`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add experimental Fory JSON functions [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #19223:
URL: https://github.com/apache/pinot/pull/19223#discussion_r3771659742
##########
pinot-common/pom.xml:
##########
@@ -280,6 +293,11 @@
<groupId>com.jayway.jsonpath</groupId>
<artifactId>json-path</artifactId>
</dependency>
+ <dependency>
+ <groupId>org.apache.fory</groupId>
+ <artifactId>fory-json</artifactId>
+ <optional>true</optional>
Review Comment:
Addressed in 760d96a56e. The extractor, scalar-function, and transform Javadocs now name the exact optional runtime coordinate `org.apache.fory:fory-json:1.6.0` (with transitive `fory-core`). I also added an IMPORTANT runtime-availability section above the benchmark tables in the PR description: standard builds/distributions do not ship Fory and therefore use the Jackson/Jayway fallback, while the benchmarks and `pinot-integration-tests` explicitly include Fory.
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/JsonExtractScalarTransformFunction.java:
##########
@@ -241,6 +260,15 @@ public void init(List<TransformFunction> arguments, Map<String, ColumnContext> c
}
}
_resultMetadata = new TransformResultMetadata(_dataType, isSingleValue, false);
+ DataType inputDataType = firstArgument.getResultMetadata().getDataType();
+ _foryEligible = _extractionMode == ExtractionMode.FORY && _simpleJsonPath != null
+ && inputDataType == DataType.STRING && isSingleValue && _dataType != DataType.STRING
+ && _dataType != DataType.JSON && _dataType != DataType.BIG_DECIMAL;
Review Comment:
The exclusion is intentional because query `jsonExtractScalar(..., 'STRING')` has a stronger precision contract than the ingestion `jsonPathString` helper. The query path uses `JSON_PARSER_CONTEXT_WITH_BIG_DECIMAL` so a non-string numeric leaf such as `{"v":12345678901234567890.123456789}` must be returned as the exact string `12345678901234567890.123456789`; `testExtractStringPreservesNumericPrecision` covers this for all variants. Fory 1.6 materializes fractional/exponent numbers as `Double`, so enabling it for the generic STRING result would round that value. Ingestion `jsonPathString` uses the normal non-BigDecimal Jayway context, which is why its Fory counterpart has a different eligible envelope. Commit 760d96a56e adds this rationale next to the eligibility decision. A future token-aware path could accelerate string leaves while falling back before fractional-number coercion, but I kept the experimental path conservative here.
##########
pinot-common/src/main/java/org/apache/pinot/common/function/ForyJsonPathExtractor.java:
##########
@@ -0,0 +1,388 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.function;
+
+import com.fasterxml.jackson.core.StreamReadConstraints;
+import java.util.concurrent.atomic.AtomicBoolean;
+import javax.annotation.Nullable;
+import org.apache.fory.json.ForyJson;
+import org.apache.fory.json.codec.JsonValueCodec;
+import org.apache.fory.json.reader.JsonReader;
+import org.apache.fory.json.reader.Latin1JsonReader;
+import org.apache.fory.json.reader.Utf16JsonReader;
+import org.apache.fory.json.reader.Utf8JsonReader;
+import org.apache.fory.json.writer.StringJsonWriter;
+import org.apache.fory.json.writer.Utf8JsonWriter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/// Shared Fory JSON parser for opt-in JSON-path implementations.
+///
+/// Streaming extraction uses one bounded parser pool shared by all worker threads. Initialization or runtime linkage
+/// failures permanently disable the optional path, allowing callers to fall back to Jackson/Jayway. Jackson's
+/// document, token, nesting, field-name, string, and number constraints are enforced while walking the document.
+/// The Fory runtime is an optional dependency; applications must add it to the application classpath to enable this
+/// experimental path.
+public final class ForyJsonPathExtractor {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ForyJsonPathExtractor.class);
+ private static final StreamReadConstraints JACKSON_CONSTRAINTS = StreamReadConstraints.defaults();
+ private static final ThreadLocal<PathContext> PATH_CONTEXT = ThreadLocal.withInitial(PathContext::new);
+ private static final AtomicBoolean UNAVAILABLE_WARNING_LOGGED = new AtomicBoolean();
+ private static final Object FALLBACK_REQUIRED = new Object();
+
+ private ForyJsonPathExtractor() {
+ }
+
+ private static final class Holder {
+ @Nullable
+ private static final ForyJson STREAMING_PARSER = buildStreamingParser();
+ private static volatile boolean _streamingAvailable = STREAMING_PARSER != null;
+
+ private Holder() {
+ }
+
+ @Nullable
+ private static ForyJson buildStreamingParser() {
+ try {
+ return ForyJson.builder().withCodegen(false).withAsyncCompilation(false)
+ .maxDepth(JACKSON_CONSTRAINTS.getMaxNestingDepth())
+ .registerCodec(PathResult.class, PathCodec.INSTANCE).build();
+ } catch (RuntimeException | LinkageError e) {
+ logUnavailable(e);
+ return null;
+ }
+ }
+ }
+
+ /// Returns whether the optional Fory runtime initialized successfully.
+ public static boolean isAvailable() {
+ return Holder._streamingAvailable;
+ }
+
+ /// Returns whether an extracted value requires the reference parser for exact container/coercion semantics.
+ public static boolean isFallbackRequired(@Nullable Object value) {
Review Comment:
Agreed that a typed result holder would be safer if this API grows. I am keeping the private identity sentinel in this experimental patch because there are only two production callers, both immediately check `isFallbackRequired`, and direct tests cover duplicate ordering, container fallback, malformed tails, and cross-thread cleanup. I will treat a result-holder API as the follow-up before exposing additional callers rather than expanding this PR further.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add experimental Fory JSON functions [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19223:
URL: https://github.com/apache/pinot/pull/19223#issuecomment-5263047385
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/19223?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `66.89895%` with `95 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 57.70%. Comparing base ([`4fa3bac`](https://app.codecov.io/gh/apache/pinot/commit/4fa3bac0093b6940a8099c34c83862e8c0c88e34?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`b62ccdf`](https://app.codecov.io/gh/apache/pinot/commit/b62ccdfd807f1c150e376f4835e9bb762ef313e1?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/pinot/pull/19223?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...e/pinot/common/function/ForyJsonPathExtractor.java](https://app.codecov.io/gh/apache/pinot/pull/19223?src=pr&el=tree&filepath=pinot-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcommon%2Ffunction%2FForyJsonPathExtractor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vRm9yeUpzb25QYXRoRXh0cmFjdG9yLmphdmE=) | 64.63% | [51 Missing and 36 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/19223?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...he/pinot/common/function/scalar/JsonFunctions.java](https://app.codecov.io/gh/apache/pinot/pull/19223?src=pr&el=tree&filepath=pinot-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcommon%2Ffunction%2Fscalar%2FJsonFunctions.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vc2NhbGFyL0pzb25GdW5jdGlvbnMuamF2YQ==) | 75.00% | [3 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/19223?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...m/function/JsonExtractScalarTransformFunction.java](https://app.codecov.io/gh/apache/pinot/pull/19223?src=pr&el=tree&filepath=pinot-core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcore%2Foperator%2Ftransform%2Ffunction%2FJsonExtractScalarTransformFunction.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vSnNvbkV4dHJhY3RTY2FsYXJUcmFuc2Zvcm1GdW5jdGlvbi5qYXZh) | 83.33% | [0 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/19223?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (4fa3bac) and HEAD (b62ccdf). Click for more details.
>
> <details><summary>HEAD has 24 uploads less than BASE</summary>
>
>| Flag | BASE (4fa3bac) | HEAD (b62ccdf) |
>|------|------|------|
>|lane-a|2|0|
>|integration|4|0|
>|java-25|6|1|
>|integration2|2|0|
>|temurin|6|1|
>|lane-b|2|0|
>|unittests|2|1|
>|integration1|2|0|
>|unittests2|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19223 +/- ##
============================================
- Coverage 66.94% 57.70% -9.24%
+ Complexity 1423 1 -1422
============================================
Files 3452 2656 -796
Lines 218534 159106 -59428
Branches 34741 26118 -8623
============================================
- Hits 146288 91807 -54481
+ Misses 60559 59486 -1073
+ Partials 11687 7813 -3874
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/19223/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [integration](https://app.codecov.io/gh/apache/pinot/pull/19223/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/19223/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/19223/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-25](https://app.codecov.io/gh/apache/pinot/pull/19223/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.70% <66.89%> (-9.24%)` | :arrow_down: |
| [lane-a](https://app.codecov.io/gh/apache/pinot/pull/19223/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [lane-b](https://app.codecov.io/gh/apache/pinot/pull/19223/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/19223/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.70% <66.89%> (-9.24%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/19223/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.70% <66.89%> (-9.24%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/19223/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.70% <66.89%> (+0.01%)` | :arrow_up: |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/19223/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/pinot/pull/19223?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org