You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "xiangfu0 (via GitHub)" <gi...@apache.org> on 2026/07/27 12:25:45 UTC
[PR] Add end-to-end Parquet VARIANT support [pinot]
xiangfu0 opened a new pull request, #19101:
URL: https://github.com/apache/pinot/pull/19101
## Summary
Add an end-to-end `VARIANT` user journey for Apache Pinot:
- define a single-value `VARIANT` dimension that is retained in Pinot's raw forward index
- ingest top-level, non-repeated Apache Parquet `VARIANT(1)` values, including unshredded and shredded layouts
- materialize frequently queried paths during ingestion while retaining the original Variant value
- expose Spark-style parse, extraction, type, null, and JSON-rendering functions in both query engines
- carry Variant through Pinot's schema, data block, DDL, JDBC, and response layers
- add a packaged quickstart and an integration suite covering table creation, Parquet ingestion, and queries
Raw Variant values are deliberately not orderable, comparable, or generally aggregatable. Pinot allows `COUNT(raw_variant)` but rejects raw comparison, grouping, distinct, ordering, join-key, set-operation, and other aggregate uses with guidance to extract a typed scalar first.
The persisted format, null semantics, ownership boundaries, compatibility
contract, and activation/rollback gates are recorded in
[`pinot-spi/VARIANT_DESIGN.md`](https://github.com/xiangfu0/pinot/blob/xiangfu0/variant-e2e/pinot-spi/VARIANT_DESIGN.md).
## Try it
Build Pinot and launch the dedicated batch quickstart:
```shell
./mvnw clean install -DskipTests -Pbin-dist -Pbuild-shaded-jar
build/bin/quick-start-variant-batch.sh
```
The quickstart:
1. registers the `variantEvents` schema and offline table
2. ingests the committed Parquet `VARIANT(1)` fixture
3. uploads a five-row segment
4. runs the representative queries below
```sql
SET enableNullHandling=true;
SELECT eventType, COUNT(*)
FROM variantEvents
GROUP BY eventType
ORDER BY eventType;
SELECT
eventId,
variant_get(payload, '$.user.id', 'STRING') AS userId,
variant_get(payload, '$.amount', 'DOUBLE') AS amount
FROM variantEvents
WHERE eventType = 'checkout'
ORDER BY eventId;
SELECT eventId, variantToJson(payload)
FROM variantEvents
ORDER BY eventId;
```
The complete sample is under
`pinot-tools/src/main/resources/examples/batch/variantEvents`.
## Supported scope
- top-level, non-repeated Parquet `VARIANT(1)` columns
- a single-value Pinot `VARIANT` dimension stored without a dictionary
- raw forward-index retention plus optional ingestion-time scalar materialization
- storage null handling enabled by the schema or table
- query-time typed extraction and JSON rendering with `SET enableNullHandling=true`
- both single-stage and multi-stage query execution
Nested/repeated Variant columns, streaming ingestion, quoted path keys, and mixed-version queries over active Variant tables are outside this initial verified scope.
The automatic Parquet reader selection keeps the existing Avro-metadata
precedence. Files that contain Avro metadata must explicitly select the native
Parquet reader to ingest a Variant column.
## Production safety
- centralizes Pinot's `PVAR` framing and validation in `VariantEnvelope`, freezes the version-1 bytes with a golden test, and assigns a new protobuf wire value without renumbering existing types
- reports an actionable full-upgrade error when an unknown query-wire type is received
- rejects schemas/tables without effective storage null handling and rejects VARIANT SQL functions when query null handling is disabled
- rejects unsupported index configurations and unsafe raw Variant query operations, including window partition/order keys and non-`COUNT` raw aggregates
- preserves historical Parquet reader selection, uses locale-independent type canonicalization, and converts Parquet dates to UTC epoch-day timestamps
- initializes Parquet converters atomically, publishes replacement readers transactionally, caches immutable Variant schema indexes, preserves progress when a malformed row is skipped, and tests terminal malformed-row behavior
- validates Parquet decimal precision, scale, byte bounds, and hostile exponents before allocation
- writes array-backed, direct, and read-only Parquet `Binary` values into the final `PVAR` envelope without a full-payload intermediate copy and without changing source buffer positions or limits
- compiles query paths and target types once, reuses allocation-free cursors, vectorizes existence checks, and shares cache/null lifecycle across the single-stage Variant functions
- parses literal `parseJson`/`tryParseJson` inputs once per query in both engines and specializes ingestion-time Variant functions with constant paths/types into compiled, cursor-reusing evaluators
- keeps tolerant extraction mismatch and numeric overflow on a non-throwing path, while preserving strict conversion behavior, and avoids `BigInteger` allocation for JSON integers that fit primitive ranges
- restores the Java client's legacy textual JSON-null contract while distinguishing SQL null, encoded Variant null, and the Variant string `"null"` across Arrow, JSON, HTTP JDBC, and gRPC JDBC results
- keeps only `parquet-variant` in `pinot-common`; Parquet column/schema dependencies remain isolated to the input plugin
- adds approximately 124 KiB to the shaded common artifact, with no Parquet column/schema classes
- covers raw forward-index min/max behavior and direct-buffer inputs with regressions
- documents the activation contract: upgrade the complete Pinot fleet and external ingestion jobs before registering a Variant schema, and do not roll back while a Variant table is active
`japicmp` reports no binary-incompatible changes for `pinot-spi` or
`pinot-segment-spi`. The published Java 11 artifacts retain classfile major
version 55.
### Compatibility and rollout
Existing schemas, segments, and data types keep their current behavior. Older
components reject an unknown Variant wire type deterministically, but they
cannot operate on an active Variant table. Upgrade controllers, brokers,
servers, clients, and external ingestion jobs/plugins before creating a Variant
schema; do not activate Variant during a rolling mixed-version window.
## Verification
- focused wire, utility, transform, validation, planner, runtime, client, DDL, and Parquet reader suites
- 19 `VariantTypeTest` integration scenarios across both query engines
- 63-module integration-test reactor build
- 83-module clean shaded binary distribution build on JDK 25 (`BUILD SUCCESS`, 11m24s)
- Java 11 compatibility scan: classfile major version 55 for SPI, segment SPI, common, time-series SPI, Java client, and JDBC client
- packaged quickstart: table creation, five-row Parquet ingestion, materialized-field grouping, typed nested extraction, JSON rendering, SQL/Variant null semantics, and clean shutdown
- local JMH smoke: compiled reusable extraction is 90.953 ns/op with 0.001 B/op versus 92.328 ns/op and 112.001 B/op for the object-returning path
- local JMH smoke: unshredded Parquet conversion is 27.682, 29.470, and 30.850 ns/op for array-backed, direct, and read-only buffers respectively; all allocate the same 328 B/op final result
- spotless, checkstyle, license formatting, and license checks across all 16 affected modules
- `git diff --check`
## Review notes
This is a design and implementation draft, not a merge-ready ownership
approval. It changes public SPI (`FieldSpec.DataType.VARIANT`,
`VariantEnvelope`, protobuf/DataSchema mappings) and requires explicit
compatibility sign-off before merge:
- [ ] SPI, schema, and query-wire owners
- [ ] Parquet and input-format plugin owners
- [ ] single-stage and multi-stage query-engine owners
- [ ] Java, JDBC, Arrow, JSON, and gRPC client/response owners
The draft is kept end-to-end so those owners can review one complete activation
contract: the public type, its only supported ingestion path, the raw-value
safety guards, and the executable acceptance test. No partial subset is safe to
activate: the public type without ingestion is unusable, ingestion without all
query guards exposes byte-layout semantics, and any subset without mixed-version
gates is unsafe to roll out. The design document records the dependency order
and the rule that VARIANT remains unactivatable until all safety pieces land.
If maintainers prefer smaller rollback units after design review, this draft can
be converted into that documented dependency-ordered stack: foundation and wire
contract; segment and Parquet ingestion; query functions and engines; clients
and DDL; then quickstart, integration, and benchmarks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add end-to-end Parquet VARIANT support [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #19101:
URL: https://github.com/apache/pinot/pull/19101#discussion_r3742924231
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/SortOperator.java:
##########
@@ -73,6 +74,11 @@ public SortOperator(OpChainExecutionContext context, MultiStageOperator input, S
// - There is no collation
// - Input is already sorted
List<RelFieldCollation> collations = node.getCollations();
+ for (RelFieldCollation collation : collations) {
+ Preconditions.checkArgument(
+ _dataSchema.getColumnDataType(collation.getFieldIndex()).supportsOrdering(),
+ "ORDER BY does not support raw VARIANT values; extract a typed path with variantGet first");
Review Comment:
Fixed in xiangfu0/pinot#236. The ORDER BY / sort / window / set-op / aggregate rejections now name the actual unsupported type (e.g. "ORDER BY does not support OBJECT values") and keep the raw-VARIANT wording plus `variantGet` guidance only for VARIANT. Applied to `SortOperator`, `SortedMailboxReceiveOperator`, `OrderByComparatorFactory`, and `VariantTypeValidationVisitor`.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java:
##########
@@ -1448,6 +1456,10 @@ static void validatePartialUpsertStrategies(TableConfig tableConfig, Schema sche
FieldSpec fieldSpec = schema.getFieldSpecFor(column);
Preconditions.checkState(fieldSpec != null, "Merger cannot be applied to non-existing column: %s", column);
+ if (fieldSpec.getDataType() == DataType.VARIANT) {
+ Preconditions.checkState(columnStrategy == UpsertConfig.Strategy.OVERWRITE,
+ "VARIANT column supports only OVERWRITE partial-upsert strategy: %s", column);
Review Comment:
Fixed in xiangfu0/pinot#236. `validatePartialUpsertStrategies` now validates the *effective* strategy for every VARIANT column up front: it uses the explicit entry if listed, otherwise `defaultPartialUpsertStrategy`, and requires OVERWRITE. It also rejects a custom `partialUpsertMergerClass` on VARIANT columns (which cannot be validated against the OVERWRITE-only contract). Added 3 tests covering unlisted-column default, valid default, and custom-merger rejection.
##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/validation/VariantTypeValidationVisitor.java:
##########
@@ -0,0 +1,185 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.query.planner.validation;
+
+import java.util.List;
+import org.apache.calcite.rel.RelFieldCollation;
+import org.apache.pinot.common.utils.DataSchema;
+import org.apache.pinot.common.utils.VariantUtils;
+import org.apache.pinot.query.planner.logical.RexExpression;
+import org.apache.pinot.query.planner.plannode.AggregateNode;
+import org.apache.pinot.query.planner.plannode.JoinNode;
+import org.apache.pinot.query.planner.plannode.PlanNode;
+import org.apache.pinot.query.planner.plannode.PlanNodeVisitor;
+import org.apache.pinot.query.planner.plannode.SetOpNode;
+import org.apache.pinot.query.planner.plannode.SortNode;
+import org.apache.pinot.query.planner.plannode.WindowNode;
+import org.apache.pinot.spi.exception.QueryErrorCode;
+import org.apache.pinot.spi.exception.QueryException;
+
+
+/// Rejects operations that would otherwise assign physical byte ordering, equality, or hashing semantics to a raw
+/// VARIANT value. The visitor has no mutable state and is thread-safe, so callers may share {@link #INSTANCE}.
+public final class VariantTypeValidationVisitor extends PlanNodeVisitor.DepthFirstVisitor<Void, Void> {
+ public static final VariantTypeValidationVisitor INSTANCE = new VariantTypeValidationVisitor();
+
+ private VariantTypeValidationVisitor() {
+ }
+
+ @Override
+ protected boolean traverseStageBoundary() {
+ return false;
+ }
+
+ @Override
+ public Void visitAggregate(AggregateNode node, Void context) {
+ List<PlanNode> inputs = node.getInputs();
+ if (inputs.size() == 1) {
+ validateAggregateInputs(node, inputs.get(0).getDataSchema());
+ }
+ return super.visitAggregate(node, context);
+ }
+
+ /// Validates aggregate operands against their logical input schema.
+ ///
+ /// <p>This method is also invoked by the runtime as a defensive check for plans that did not pass through the
+ /// current broker planner.
+ public static void validateAggregateInputs(AggregateNode node, DataSchema inputSchema) {
Review Comment:
Fixed in xiangfu0/pinot#236. `validateAggregateInputs(AggregateNode, ...)` now also validates `AggregateNode.getGroupKeys()` (equality+hashing), so the planner gate rejects `GROUP BY raw_variant` consistently with sort/join/set-op/window (runtime already rejected it). Added `testRejectsRawVariantGroupByKey`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add end-to-end Parquet VARIANT support [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #19101:
URL: https://github.com/apache/pinot/pull/19101#discussion_r3920903302
##########
docs/design/VARIANT.md:
##########
@@ -0,0 +1,344 @@
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+# Apache Pinot VARIANT — Design Document
Review Comment:
Moved the spec to `docs/design/VARIANT.md`, updated the quickstart and PR links, and corrected section 8 to say 22/23 are already assigned to UUID/UUID_ARRAY.
##########
pinot-common/src/main/proto/expressions.proto:
##########
@@ -46,6 +46,7 @@ enum ColumnDataType {
BIG_DECIMAL_ARRAY = 21;
UUID = 22;
UUID_ARRAY = 23;
+ VARIANT = 24;
Review Comment:
Kept wire value 24 and the frozen legacy-proto `UNRECOGNIZED` assertion. The PR body now calls this out as an explicit independent sign-off requirement, and I requested reviews from you and `raghavyadav01`. I am leaving this thread open for that sign-off rather than self-resolving it.
##########
pinot-spi/src/main/java/org/apache/pinot/spi/utils/VariantEnvelope.java:
##########
@@ -0,0 +1,243 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.spi.utils;
+
+import java.nio.ByteBuffer;
+import java.util.Objects;
+import javax.annotation.Nullable;
+
+
+/// Pinot-owned framing for the two buffers that make up a Parquet Variant value.
+///
+/// The version-1 wire format is:
+/// ```
+/// 0 4 bytes ASCII magic "PVAR"
+/// 4 1 byte envelope version (1)
+/// 5 1 byte flags (0)
+/// 6 2 bytes reserved (0)
+/// 8 4 bytes metadata length, unsigned range restricted to Java array sizes
+/// 12 4 bytes value length, unsigned range restricted to Java array sizes
+/// 16 M bytes Parquet Variant metadata
+/// 16 + M V bytes Parquet Variant value
+/// ```
+///
+/// An empty byte array is deliberately not an envelope. Pinot reserves it as the default null value for a
+/// `VARIANT` field, allowing the null-value vector to distinguish SQL null from an encoded Variant null.
+///
+/// This class validates only Pinot's stable outer framing. Producers and consumers remain responsible for validating
+/// the Parquet Variant metadata and value payloads.
+public final class VariantEnvelope {
+ public static final int HEADER_SIZE = 16;
+ public static final byte VERSION = 1;
+ public static final byte FLAGS = 0;
+
+ private static final int MAGIC = 0x50564152; // ASCII "PVAR"
+
+ private VariantEnvelope() {
+ }
+
+ /// Encodes the remaining bytes of the supplied metadata and value buffers without changing their positions or
+ /// limits.
+ ///
+ /// Array-backed buffers are copied directly from their backing arrays. Other buffers, including direct and
+ /// read-only buffers, are read through independent duplicate views.
+ public static byte[] encode(ByteBuffer metadata, ByteBuffer value) {
+ Objects.requireNonNull(metadata, "metadata must not be null");
+ Objects.requireNonNull(value, "value must not be null");
+
+ int metadataLength = metadata.remaining();
+ int valueLength = value.remaining();
+ byte[] envelope = allocate(metadataLength, valueLength);
+ copyRemaining(metadata, envelope, HEADER_SIZE);
+ copyRemaining(value, envelope, HEADER_SIZE + metadataLength);
+ return envelope;
+ }
+
+ /// Encodes slices of the supplied arrays without allocating intermediate buffer views.
+ public static byte[] encode(byte[] metadata, int metadataOffset, int metadataLength, byte[] value, int valueOffset,
+ int valueLength) {
+ Objects.requireNonNull(metadata, "metadata must not be null");
+ Objects.requireNonNull(value, "value must not be null");
+ requireRange(metadata, metadataOffset, metadataLength, "metadata");
+ requireRange(value, valueOffset, valueLength, "value");
+
+ byte[] envelope = allocate(metadataLength, valueLength);
+ System.arraycopy(metadata, metadataOffset, envelope, HEADER_SIZE, metadataLength);
+ System.arraycopy(value, valueOffset, envelope, HEADER_SIZE + metadataLength, valueLength);
+ return envelope;
+ }
+
+ /// Decodes and validates an envelope, returning zero-copy, read-only views over its metadata and value buffers.
+ ///
+ /// The returned views alias `envelope`; this method does not copy either payload. The decoded object and any views
+ /// obtained from it keep the backing array alive, so the caller does not need to retain a separate reference to
+ /// `envelope`. Mutations made to the input array after this method returns are visible through the views and can
+ /// corrupt the decoded payload. Callers must therefore treat the input array as immutable for as long as the decoded
+ /// object or any returned view may be used.
+ ///
+ /// The decoded holder is safe for concurrent reads when the aliased input array is not mutated. Each accessor returns
+ /// a read-only view with independent position and limit, so cursor movement by one reader does not affect another
+ /// reader.
+ public static Decoded decode(byte[] envelope) {
Review Comment:
Renamed the public factory to `allocateUnfilled` and aligned the class/spec wording to big-endian signed int restricted to >= 0. Golden and validation tests cover the unfilled/header contract.
##########
pinot-common/src/main/java/org/apache/pinot/common/utils/VariantUtils.java:
##########
@@ -0,0 +1,2060 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils;
+
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.sql.Timestamp;
+import java.time.Instant;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.LocalTime;
+import java.time.ZoneOffset;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Locale;
+import java.util.Objects;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.Nullable;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.commons.io.output.StringBuilderWriter;
+import org.apache.parquet.variant.Variant;
+import org.apache.parquet.variant.VariantArrayBuilder;
+import org.apache.parquet.variant.VariantBuilder;
+import org.apache.parquet.variant.VariantObjectBuilder;
+import org.apache.pinot.spi.data.FieldSpec.DataType;
+import org.apache.pinot.spi.utils.ByteArray;
+import org.apache.pinot.spi.utils.UuidUtils;
+import org.apache.pinot.spi.utils.VariantEnvelope;
+
+
+/// Query-side operations for Pinot {@code VARIANT} values.
+///
+/// <p>The utility navigates the Parquet Variant binary representation directly. It never materializes a JSON tree.
+/// Instances are not required, and stateless convenience methods are thread-safe. Overloads that accept a
+/// caller-provided {@link ReusableResult} require that result to be thread-confined and not shared by concurrent calls.
+/// An empty byte array is Pinot's SQL-null placeholder and is never decoded as an envelope.
+public final class VariantUtils {
+ public static final String RAW_VARIANT_REQUIRES_NULL_HANDLING_ERROR =
+ "Raw VARIANT projection requires query null handling to be enabled; set enableNullHandling=true";
+
+ private static final JsonFactory JSON_FACTORY = new JsonFactory();
+ private static final BigDecimal MIN_INT_DECIMAL = BigDecimal.valueOf(Integer.MIN_VALUE);
+ private static final BigDecimal MAX_INT_DECIMAL = BigDecimal.valueOf(Integer.MAX_VALUE);
+ private static final BigDecimal MIN_LONG_DECIMAL = BigDecimal.valueOf(Long.MIN_VALUE);
+ private static final BigDecimal MAX_LONG_DECIMAL = BigDecimal.valueOf(Long.MAX_VALUE);
+ private static final int MAX_JSON_NESTING_DEPTH = 100;
+ private static final int MAX_VARIANT_DECIMAL_PRECISION = 38;
+ private static final int MAX_VARIANT_DECIMAL_SCALE = 38;
+ private static final int MAX_VARIANT_DECIMAL_BYTES = 16;
+ private static final long MICROS_PER_SECOND = TimeUnit.SECONDS.toMicros(1);
+ private static final long NANOS_PER_MICRO = TimeUnit.MICROSECONDS.toNanos(1);
+ private static final long NANOS_PER_DAY = TimeUnit.DAYS.toNanos(1);
+ private static final int VARIANT_BASIC_TYPE_MASK = 0x03;
+ private static final int VARIANT_PRIMITIVE_TYPE_MASK = 0x3F;
+ private static final int VARIANT_PRIMITIVE = 0;
+ private static final int VARIANT_SHORT_STRING = 1;
+ private static final int VARIANT_OBJECT = 2;
+ private static final int VARIANT_ARRAY = 3;
+ private static final int VARIANT_NULL = 0;
+ private static final int VARIANT_TRUE = 1;
+ private static final int VARIANT_FALSE = 2;
+ private static final int VARIANT_INT8 = 3;
+ private static final int VARIANT_INT16 = 4;
+ private static final int VARIANT_INT32 = 5;
+ private static final int VARIANT_INT64 = 6;
+ private static final int VARIANT_DOUBLE = 7;
+ private static final int VARIANT_DECIMAL4 = 8;
+ private static final int VARIANT_DECIMAL8 = 9;
+ private static final int VARIANT_DECIMAL16 = 10;
+ private static final int VARIANT_DATE = 11;
+ private static final int VARIANT_TIMESTAMP_TZ = 12;
+ private static final int VARIANT_TIMESTAMP_NTZ = 13;
+ private static final int VARIANT_FLOAT = 14;
+ private static final int VARIANT_BINARY = 15;
+ private static final int VARIANT_LONG_STRING = 16;
+ private static final int VARIANT_TIME = 17;
+ private static final int VARIANT_TIMESTAMP_NANOS_TZ = 18;
+ private static final int VARIANT_TIMESTAMP_NANOS_NTZ = 19;
+ private static final int VARIANT_UUID = 20;
+ private static final int VARIANT_METADATA_VERSION_MASK = 0x0F;
+ private static final int VARIANT_METADATA_VERSION = 1;
+ private static final int OBJECT_BINARY_SEARCH_THRESHOLD = 32;
+ private static final int INVALID_UTF8_COMPARISON = Integer.MIN_VALUE;
+ private static final VariantPath ROOT_PATH = new VariantPath(new PathElement[0]);
+
+ private VariantUtils() {
+ }
+
+ /// Returns whether a final result containing raw VARIANT values requires query null handling. Without a null bitmap,
+ /// Pinot's reserved empty-byte SQL-null placeholder cannot be distinguished from a logical Variant value.
+ public static boolean requiresNullHandlingForRawVariantResult(DataSchema resultSchema,
+ boolean nullHandlingEnabled) {
+ if (nullHandlingEnabled) {
+ return false;
+ }
+ for (DataSchema.ColumnDataType dataType : resultSchema.getColumnDataTypes()) {
+ if (dataType == DataSchema.ColumnDataType.VARIANT) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /// Statically supported result types for {@code variantGet} and {@code tryVariantGet}.
+ public enum ResultType {
+ BOOLEAN(DataType.BOOLEAN, SqlTypeName.BOOLEAN),
+ INT(DataType.INT, SqlTypeName.INTEGER),
+ LONG(DataType.LONG, SqlTypeName.BIGINT),
+ FLOAT(DataType.FLOAT, SqlTypeName.REAL),
+ DOUBLE(DataType.DOUBLE, SqlTypeName.DOUBLE),
+ BIG_DECIMAL(DataType.BIG_DECIMAL, SqlTypeName.DECIMAL),
+ STRING(DataType.STRING, SqlTypeName.VARCHAR),
+ BYTES(DataType.BYTES, SqlTypeName.VARBINARY),
+ UUID(DataType.UUID, SqlTypeName.UUID),
+ TIMESTAMP(DataType.TIMESTAMP, SqlTypeName.TIMESTAMP),
+ VARIANT(DataType.VARIANT, SqlTypeName.VARIANT),
+ JSON(DataType.JSON, SqlTypeName.VARCHAR);
+
+ private final DataType _dataType;
+ private final SqlTypeName _sqlTypeName;
+
+ ResultType(DataType dataType, SqlTypeName sqlTypeName) {
+ _dataType = dataType;
+ _sqlTypeName = sqlTypeName;
+ }
+
+ public DataType getDataType() {
+ return _dataType;
+ }
+
+ public SqlTypeName getSqlTypeName() {
+ return _sqlTypeName;
+ }
+ }
+
+ /// An immutable, pre-parsed Variant path. The v1 grammar supports {@code $}, dot-separated object fields, and
+ /// non-negative array subscripts.
+ public static final class VariantPath {
+ private final PathElement[] _elements;
+
+ private VariantPath(PathElement[] elements) {
+ _elements = elements;
+ }
+ }
+
+ /// Reusable, unboxed destination for vectorized Variant extraction.
+ ///
+ /// <p>Only the getter corresponding to the requested {@link ResultType} is defined after a successful extraction.
+ /// The instance is mutable and not thread-safe; callers should retain one per transform-function instance. Every
+ /// extraction may replace its state. Each successful byte-valued extraction installs a newly materialized array.
+ /// Values returned as {@code byte[]} or as a {@link ByteArray} may be retained after this result is reused, but they
+ /// are read-only by contract and must be copied before mutation.
+ public static final class ReusableResult {
+ private final Cursor _cursor = new Cursor();
+ private int _intValue;
+ private long _longValue;
+ private float _floatValue;
+ private double _doubleValue;
+ private BigDecimal _bigDecimalValue;
+ private String _stringValue;
+ private byte[] _bytesValue;
+
+ public int getIntValue() {
+ return _intValue;
+ }
+
+ public long getLongValue() {
+ return _longValue;
+ }
+
+ public float getFloatValue() {
+ return _floatValue;
+ }
+
+ public double getDoubleValue() {
+ return _doubleValue;
+ }
+
+ public BigDecimal getBigDecimalValue() {
+ return _bigDecimalValue;
+ }
+
+ public String getStringValue() {
+ return _stringValue;
+ }
+
+ /// Returns the extracted BYTES, VARIANT, or direct 16-byte UUID representation.
+ ///
+ /// <p>The returned array is replaced, but not mutated, by the next byte-valued extraction. It may be retained after
+ /// this result is reused, but must be treated as immutable and copied before mutation.
+ public byte[] getBytesValue() {
+ return _bytesValue;
+ }
+
+ public UUID getUuidValue() {
+ return UuidUtils.toUUID(_bytesValue);
+ }
+
+ /// Materializes the extracted value in the external representation used by scalar functions and ingestion.
+ ///
+ /// <p>For BYTES and VARIANT, the returned {@code byte[]} may be retained after this result is reused. It must be
+ /// treated as immutable and copied before mutation.
+ public Object toExternalValue(ResultType resultType) {
+ switch (resultType) {
+ case BOOLEAN:
+ return _intValue != 0;
+ case INT:
+ return _intValue;
+ case LONG:
+ return _longValue;
+ case FLOAT:
+ return _floatValue;
+ case DOUBLE:
+ return _doubleValue;
+ case BIG_DECIMAL:
+ return _bigDecimalValue;
+ case STRING:
+ case JSON:
+ return _stringValue;
+ case BYTES:
+ case VARIANT:
+ return _bytesValue;
+ case UUID:
+ return UuidUtils.toUUID(_bytesValue);
+ case TIMESTAMP:
+ return new Timestamp(_longValue);
+ default:
+ throw new IllegalStateException("Unhandled Variant target type: " + resultType);
+ }
+ }
+
+ /// Materializes the extracted value in {@link DataSchema}'s internal representation.
+ ///
+ /// <p>TIMESTAMP remains epoch milliseconds and UUID wraps the directly copied 16-byte value, avoiding an
+ /// external-object round trip in the multi-stage engine. For BYTES, UUID, and VARIANT, the returned
+ /// {@link ByteArray} wraps a newly materialized array that may be retained after this result is reused. Neither the
+ /// wrapper nor its array may be mutated; callers must copy the array before mutation.
+ public Object toInternalValue(ResultType resultType) {
+ switch (resultType) {
+ case BOOLEAN:
+ return _intValue;
+ case INT:
+ return _intValue;
+ case LONG:
+ case TIMESTAMP:
+ return _longValue;
+ case FLOAT:
+ return _floatValue;
+ case DOUBLE:
+ return _doubleValue;
+ case BIG_DECIMAL:
+ return _bigDecimalValue;
+ case STRING:
+ case JSON:
+ return _stringValue;
+ case BYTES:
+ case UUID:
+ case VARIANT:
+ return new ByteArray(_bytesValue);
+ default:
+ throw new IllegalStateException("Unhandled Variant target type: " + resultType);
+ }
+ }
+ }
+
+ /// Parses a target type literal once for reuse by a transform function.
+ public static ResultType parseResultType(String targetType) {
+ if (targetType == null) {
+ throw new IllegalArgumentException("Variant target type must not be null");
+ }
+ try {
+ return ResultType.valueOf(targetType.trim().toUpperCase(Locale.ROOT));
+ } catch (IllegalArgumentException e) {
+ throw new IllegalArgumentException("Unsupported Variant target type: " + targetType, e);
+ }
+ }
+
+ /// Compiles a v1 Variant path.
+ public static VariantPath compilePath(String path) {
+ if (path == null || path.isEmpty() || path.charAt(0) != '$') {
+ throw new IllegalArgumentException("Variant path must start with '$': " + path);
+ }
+ List<PathElement> elements = new ArrayList<>();
+ int index = 1;
+ while (index < path.length()) {
+ char current = path.charAt(index);
+ if (current == '.') {
+ int fieldStart = ++index;
+ while (index < path.length()) {
+ char next = path.charAt(index);
+ if (next == '.' || next == '[') {
+ break;
+ }
+ index++;
+ }
+ if (fieldStart == index) {
+ throw new IllegalArgumentException("Variant path contains an empty field: " + path);
+ }
+ elements.add(PathElement.forField(path.substring(fieldStart, index)));
+ } else if (current == '[') {
+ int subscriptStart = ++index;
+ while (index < path.length() && Character.isDigit(path.charAt(index))) {
+ index++;
+ }
+ if (subscriptStart == index || index >= path.length() || path.charAt(index) != ']') {
+ throw new IllegalArgumentException("Invalid Variant array subscript in path: " + path);
+ }
+ try {
+ elements.add(PathElement.forIndex(Integer.parseInt(path.substring(subscriptStart, index))));
+ } catch (NumberFormatException e) {
+ throw new IllegalArgumentException("Variant array subscript is too large in path: " + path, e);
+ }
+ index++;
+ } else {
+ throw new IllegalArgumentException("Unexpected character at offset " + index + " in Variant path: " + path);
+ }
+ }
+ return new VariantPath(elements.toArray(new PathElement[0]));
+ }
+
+ /// Extracts a Variant value. A missing path or SQL null returns Java null; a Variant null remains an encoded Variant
+ /// value.
+ @Nullable
+ public static byte[] variantGet(@Nullable byte[] envelope, String path) {
+ return (byte[]) variantGet(envelope, compilePath(path), ResultType.VARIANT);
+ }
+
+ /// Strictly extracts and converts a value. A missing path or SQL null returns Java null. A Variant null remains
+ /// encoded when the target type is {@link ResultType#VARIANT}, and returns Java null for other target types. An
+ /// incompatible non-null value throws.
+ @Nullable
+ public static Object variantGet(@Nullable byte[] envelope, String path, String targetType) {
+ return variantGet(envelope, compilePath(path), parseResultType(targetType));
+ }
+
+ /// Strictly extracts using pre-parsed path and type values.
+ @Nullable
+ public static Object variantGet(@Nullable byte[] envelope, VariantPath path, ResultType targetType) {
+ ReusableResult result = new ReusableResult();
+ return extractInto(envelope, path, targetType, result) ? result.toExternalValue(targetType) : null;
+ }
+
+ /// Strictly extracts into a reusable, unboxed result.
+ ///
+ /// @return {@code false} for SQL null, a missing path, or Variant null converted to a non-Variant target
+ public static boolean extractInto(@Nullable byte[] envelope, VariantPath path, ResultType targetType,
+ ReusableResult result) {
+ Objects.requireNonNull(result, "result must not be null");
+ if (isSqlNull(envelope)) {
+ return false;
+ }
+ Objects.requireNonNull(path, "path must not be null");
+ Objects.requireNonNull(targetType, "targetType must not be null");
+ Cursor cursor = result._cursor;
+ if (!cursor.navigate(envelope, path)) {
+ return false;
+ }
+ if (cursor.getType() == Variant.Type.NULL && targetType != ResultType.VARIANT) {
+ return false;
+ }
+ convert(cursor, targetType, result);
+ return true;
+ }
+
+ /// Tolerant Variant extraction. Malformed input returns Java null.
+ @Nullable
+ public static byte[] tryVariantGet(@Nullable byte[] envelope, String path) {
+ return (byte[]) tryVariantGet(envelope, compilePath(path), ResultType.VARIANT);
+ }
+
+ /// Tolerant typed extraction. Malformed input and incompatible types return Java null.
+ @Nullable
+ public static Object tryVariantGet(@Nullable byte[] envelope, String path, String targetType) {
+ try {
+ return tryVariantGet(envelope, compilePath(path), parseResultType(targetType));
+ } catch (RuntimeException e) {
+ return null;
+ }
+ }
+
+ /// Tolerant extraction using pre-parsed path and type values.
+ @Nullable
+ public static Object tryVariantGet(@Nullable byte[] envelope, VariantPath path, ResultType targetType) {
+ try {
+ ReusableResult result = new ReusableResult();
+ return tryExtractInto(envelope, path, targetType, result) ? result.toExternalValue(targetType) : null;
+ } catch (RuntimeException e) {
+ return null;
+ }
+ }
+
+ /// Tolerantly extracts into a reusable, unboxed result.
+ ///
+ /// @return {@code false} for SQL null, missing paths, Variant null converted to a non-Variant target,
+ /// malformed input, or an incompatible conversion
+ public static boolean tryExtractInto(@Nullable byte[] envelope, VariantPath path, ResultType targetType,
+ ReusableResult result) {
+ Objects.requireNonNull(result, "result must not be null");
+ if (isSqlNull(envelope)) {
+ return false;
+ }
+ Objects.requireNonNull(path, "path must not be null");
+ Objects.requireNonNull(targetType, "targetType must not be null");
+ Cursor cursor = result._cursor;
+ try {
+ if (!cursor.navigate(envelope, path)) {
+ return false;
+ }
+ if (cursor.getType() == Variant.Type.NULL && targetType != ResultType.VARIANT) {
+ return false;
+ }
+ return tryConvert(cursor, targetType, result);
+ } catch (IllegalArgumentException | IllegalStateException | UnsupportedOperationException
+ | IndexOutOfBoundsException e) {
+ // Cursor operations use these exceptions only for malformed or unsupported Variant encodings.
+ return false;
+ }
+ }
+
+ /// Returns whether the path is present. A present Variant null counts as present.
+ @Nullable
+ public static Boolean variantExists(@Nullable byte[] envelope, String path) {
+ return variantExists(envelope, compilePath(path));
+ }
+
+ /// Returns whether a compiled path is present. A present Variant null counts as present.
+ @Nullable
+ public static Boolean variantExists(@Nullable byte[] envelope, VariantPath path) {
+ return variantExists(envelope, path, new ReusableResult());
+ }
+
+ /// Allocation-free compiled-path form of {@link #variantExists(byte[], VariantPath)} when the caller retains the
+ /// supplied result between rows.
+ @Nullable
+ public static Boolean variantExists(@Nullable byte[] envelope, VariantPath path, ReusableResult result) {
+ Objects.requireNonNull(result, "result must not be null");
+ if (isSqlNull(envelope)) {
+ return null;
+ }
+ return result._cursor.navigate(envelope, Objects.requireNonNull(path, "path must not be null"));
+ }
+
+ /// Returns whether the root value is a Variant null. SQL null is not a Variant null.
+ public static boolean isVariantNull(@Nullable byte[] envelope) {
+ return isVariantNull(envelope, ROOT_PATH, new ReusableResult());
+ }
+
+ /// Returns whether a present value at the path is a Variant null. SQL null and missing paths return false.
+ public static boolean isVariantNull(@Nullable byte[] envelope, String path) {
+ return isVariantNull(envelope, compilePath(path));
+ }
+
+ /// Returns whether a present value at a compiled path is a Variant null. SQL null and missing paths return false.
+ public static boolean isVariantNull(@Nullable byte[] envelope, VariantPath path) {
+ return isVariantNull(envelope, path, new ReusableResult());
+ }
+
+ /// Allocation-free compiled-path form of {@link #isVariantNull(byte[], VariantPath)} when the caller retains the
+ /// supplied result between rows.
+ public static boolean isVariantNull(@Nullable byte[] envelope, VariantPath path, ReusableResult result) {
+ Objects.requireNonNull(result, "result must not be null");
+ if (isSqlNull(envelope)) {
+ return false;
+ }
+ Cursor cursor = result._cursor;
+ return cursor.navigate(envelope, Objects.requireNonNull(path, "path must not be null"))
+ && cursor.getType() == Variant.Type.NULL;
+ }
+
+ /// Returns the Variant type name at the root, or Java null for SQL null.
+ @Nullable
+ public static String variantTypeOf(@Nullable byte[] envelope) {
+ return variantTypeOf(envelope, ROOT_PATH, new ReusableResult());
+ }
+
+ /// Returns the Variant type name at a path, or Java null for SQL null or a missing path.
+ @Nullable
+ public static String variantTypeOf(@Nullable byte[] envelope, String path) {
+ return variantTypeOf(envelope, compilePath(path));
+ }
+
+ /// Returns the Variant type name at a compiled path, or Java null for SQL null or a missing path.
+ @Nullable
+ public static String variantTypeOf(@Nullable byte[] envelope, VariantPath path) {
+ return variantTypeOf(envelope, path, new ReusableResult());
+ }
+
+ /// Allocation-free compiled-path form of {@link #variantTypeOf(byte[], VariantPath)} when the caller retains the
+ /// supplied result between rows.
+ @Nullable
+ public static String variantTypeOf(@Nullable byte[] envelope, VariantPath path, ReusableResult result) {
+ Objects.requireNonNull(result, "result must not be null");
+ if (isSqlNull(envelope)) {
+ return null;
+ }
+ Cursor cursor = result._cursor;
+ return cursor.navigate(envelope, Objects.requireNonNull(path, "path must not be null"))
+ ? typeName(cursor.getType()) : null;
+ }
+
+ /// Renders the Variant value as canonical JSON text without constructing a JSON tree.
+ @Nullable
+ public static String variantToJson(@Nullable byte[] envelope) {
+ return variantToJson(envelope, new ReusableResult());
+ }
+
+ /// Allocation-reduced form of [#variantToJson(byte[])] when the caller retains the supplied result between rows.
+ @Nullable
+ public static String variantToJson(@Nullable byte[] envelope, ReusableResult result) {
+ Objects.requireNonNull(result, "result must not be null");
+ if (isSqlNull(envelope)) {
+ return null;
+ }
+ Cursor cursor = result._cursor;
+ cursor.navigate(envelope, ROOT_PATH);
+ return variantToJson(cursor.asVariant());
+ }
+
+ /// Parses JSON text into a Pinot Variant envelope without constructing a JSON tree.
+ @Nullable
+ public static byte[] parseJsonToVariant(@Nullable String json) {
+ if (json == null) {
+ return null;
+ }
+ try (JsonParser parser = JSON_FACTORY.createParser(json)) {
+ JsonToken token = parser.nextToken();
+ if (token == null) {
+ throw new IllegalArgumentException("Cannot parse empty text as Variant");
+ }
+ VariantBuilder builder = new VariantBuilder();
+ appendJsonValue(parser, token, builder, 0);
+ if (parser.nextToken() != null) {
+ throw new IllegalArgumentException("Unexpected trailing token after Variant JSON value");
+ }
+ Variant variant = builder.build();
+ return VariantEnvelope.encode(variant.getMetadataBuffer(), variant.getValueBuffer());
+ } catch (IOException | RuntimeException e) {
+ throw new IllegalArgumentException("Cannot parse JSON as Variant", e);
+ }
+ }
+
+ /// Tolerant JSON parser. Malformed or unsupported input returns Java null.
+ @Nullable
+ public static byte[] tryParseJsonToVariant(@Nullable String json) {
+ try {
+ return parseJsonToVariant(json);
+ } catch (RuntimeException e) {
+ return null;
+ }
+ }
+
+ private static boolean isSqlNull(@Nullable byte[] envelope) {
+ return envelope == null || envelope.length == 0;
+ }
+
+ private static void convert(Cursor value, ResultType targetType, ReusableResult result) {
+ switch (targetType) {
+ case BOOLEAN:
+ requireType(value, Variant.Type.BOOLEAN, targetType);
+ result._intValue = value.getBoolean() ? 1 : 0;
+ break;
+ case INT:
+ result._intValue = toInt(value, targetType);
+ break;
+ case LONG:
+ result._longValue = toLong(value, targetType);
+ break;
+ case FLOAT:
+ result._floatValue = toFloat(value, targetType);
+ break;
+ case DOUBLE:
+ result._doubleValue = toDouble(value, targetType);
+ break;
+ case BIG_DECIMAL:
+ result._bigDecimalValue = toBigDecimal(value, targetType);
+ break;
+ case STRING:
+ requireType(value, Variant.Type.STRING, targetType);
+ result._stringValue = value.getString();
+ break;
+ case BYTES:
+ requireType(value, Variant.Type.BINARY, targetType);
+ result._bytesValue = value.getBinary();
+ break;
+ case UUID:
+ requireType(value, Variant.Type.UUID, targetType);
+ result._bytesValue = value.getUuidBytes();
+ break;
+ case TIMESTAMP:
+ result._longValue = toTimestampMillis(value, targetType);
+ break;
+ case VARIANT:
+ result._bytesValue = value.copyEnvelope();
+ break;
+ case JSON:
+ result._stringValue = variantToJson(value.asVariant());
+ break;
+ default:
+ throw new IllegalStateException("Unhandled Variant target type: " + targetType);
+ }
+ }
+
+ private static boolean tryConvert(Cursor value, ResultType targetType, ReusableResult result) {
+ Variant.Type valueType = value.getType();
+ switch (targetType) {
+ case BOOLEAN:
+ if (valueType != Variant.Type.BOOLEAN) {
+ return false;
+ }
+ result._intValue = value.getBoolean() ? 1 : 0;
+ return true;
+ case INT:
+ return tryConvertToInt(value, valueType, result);
+ case LONG:
+ return tryConvertToLong(value, valueType, result);
+ case FLOAT:
+ switch (valueType) {
+ case BYTE:
+ case SHORT:
+ case INT:
+ case LONG:
+ result._floatValue = value.getInteger();
+ return true;
+ case FLOAT:
+ result._floatValue = value.getFloat();
+ return true;
+ case DOUBLE:
+ result._floatValue = (float) value.getDouble();
+ return true;
+ case DECIMAL4:
+ case DECIMAL8:
+ case DECIMAL16:
+ result._floatValue = value.getDecimal().floatValue();
+ return true;
+ default:
+ return false;
+ }
+ case DOUBLE:
+ switch (valueType) {
+ case BYTE:
+ case SHORT:
+ case INT:
+ case LONG:
+ result._doubleValue = value.getInteger();
+ return true;
+ case FLOAT:
+ result._doubleValue = value.getFloat();
+ return true;
+ case DOUBLE:
+ result._doubleValue = value.getDouble();
+ return true;
+ case DECIMAL4:
+ case DECIMAL8:
+ case DECIMAL16:
+ result._doubleValue = value.getDecimal().doubleValue();
+ return true;
+ default:
+ return false;
+ }
+ case BIG_DECIMAL:
+ switch (valueType) {
+ case BYTE:
+ case SHORT:
+ case INT:
+ case LONG:
+ result._bigDecimalValue = BigDecimal.valueOf(value.getInteger());
+ return true;
+ case FLOAT:
+ float floatValue = value.getFloat();
+ if (!Float.isFinite(floatValue)) {
+ return false;
+ }
+ result._bigDecimalValue = BigDecimal.valueOf(floatValue);
+ return true;
+ case DOUBLE:
+ double doubleValue = value.getDouble();
+ if (!Double.isFinite(doubleValue)) {
+ return false;
+ }
+ result._bigDecimalValue = BigDecimal.valueOf(doubleValue);
+ return true;
+ case DECIMAL4:
+ case DECIMAL8:
+ case DECIMAL16:
+ result._bigDecimalValue = value.getDecimal();
+ return true;
+ default:
+ return false;
+ }
+ case STRING:
+ if (valueType != Variant.Type.STRING) {
+ return false;
+ }
+ result._stringValue = value.getString();
+ return true;
+ case BYTES:
+ if (valueType != Variant.Type.BINARY) {
+ return false;
+ }
+ result._bytesValue = value.getBinary();
+ return true;
+ case UUID:
+ if (valueType != Variant.Type.UUID) {
+ return false;
+ }
+ result._bytesValue = value.getUuidBytes();
+ return true;
+ case TIMESTAMP:
+ switch (valueType) {
+ case DATE:
+ result._longValue = value.getInteger() * TimeUnit.DAYS.toMillis(1);
+ return true;
+ case TIMESTAMP_TZ:
+ case TIMESTAMP_NTZ:
+ result._longValue = Math.floorDiv(value.getInteger(), TimeUnit.MILLISECONDS.toMicros(1));
+ return true;
+ case TIMESTAMP_NANOS_TZ:
+ case TIMESTAMP_NANOS_NTZ:
+ result._longValue = Math.floorDiv(value.getInteger(), TimeUnit.MILLISECONDS.toNanos(1));
+ return true;
+ default:
+ return false;
+ }
+ case VARIANT:
+ result._bytesValue = value.copyEnvelope();
+ return true;
+ case JSON:
+ result._stringValue = variantToJson(value.asVariant());
+ return true;
+ default:
+ throw new AssertionError("Unhandled Variant target type: " + targetType);
+ }
+ }
+
+ private static boolean tryConvertToInt(Cursor value, Variant.Type valueType, ReusableResult result) {
+ switch (valueType) {
+ case BYTE:
+ case SHORT:
+ case INT:
+ result._intValue = (int) value.getInteger();
+ return true;
+ case LONG:
+ long longValue = value.getInteger();
+ if (longValue < Integer.MIN_VALUE || longValue > Integer.MAX_VALUE) {
+ return false;
+ }
+ result._intValue = (int) longValue;
+ return true;
+ case DECIMAL4:
+ case DECIMAL8:
+ case DECIMAL16:
+ BigDecimal decimalValue = value.getDecimal();
+ if (!isIntegralInRange(decimalValue, MIN_INT_DECIMAL, MAX_INT_DECIMAL)) {
+ return false;
+ }
+ result._intValue = decimalValue.intValue();
+ return true;
+ default:
+ return false;
+ }
+ }
+
+ private static boolean tryConvertToLong(Cursor value, Variant.Type valueType, ReusableResult result) {
+ switch (valueType) {
+ case BYTE:
+ case SHORT:
+ case INT:
+ case LONG:
+ result._longValue = value.getInteger();
+ return true;
+ case DECIMAL4:
+ case DECIMAL8:
+ case DECIMAL16:
+ BigDecimal decimalValue = value.getDecimal();
+ if (!isIntegralInRange(decimalValue, MIN_LONG_DECIMAL, MAX_LONG_DECIMAL)) {
+ return false;
+ }
+ result._longValue = decimalValue.longValue();
+ return true;
+ default:
+ return false;
+ }
+ }
+
+ private static boolean isIntegralInRange(BigDecimal value, BigDecimal minimum, BigDecimal maximum) {
+ return value.compareTo(minimum) >= 0 && value.compareTo(maximum) <= 0
+ && (value.scale() <= 0 || value.stripTrailingZeros().scale() <= 0);
+ }
+
+ private static int toInt(Cursor value, ResultType targetType) {
+ switch (value.getType()) {
+ case BYTE:
+ case SHORT:
+ case INT:
+ return (int) value.getInteger();
+ case LONG:
+ return Math.toIntExact(value.getInteger());
+ case DECIMAL4:
+ case DECIMAL8:
+ case DECIMAL16:
+ return value.getDecimal().intValueExact();
+ default:
+ throw typeMismatch(value, targetType);
+ }
+ }
+
+ private static long toLong(Cursor value, ResultType targetType) {
+ switch (value.getType()) {
+ case BYTE:
+ case SHORT:
+ case INT:
+ case LONG:
+ return value.getInteger();
+ case DECIMAL4:
+ case DECIMAL8:
+ case DECIMAL16:
+ return value.getDecimal().longValueExact();
+ default:
+ throw typeMismatch(value, targetType);
+ }
+ }
+
+ private static float toFloat(Cursor value, ResultType targetType) {
+ switch (value.getType()) {
+ case BYTE:
+ case SHORT:
+ case INT:
+ case LONG:
+ return value.getInteger();
+ case FLOAT:
+ return value.getFloat();
+ case DOUBLE:
+ return (float) value.getDouble();
+ case DECIMAL4:
+ case DECIMAL8:
+ case DECIMAL16:
+ return value.getDecimal().floatValue();
+ default:
+ throw typeMismatch(value, targetType);
+ }
+ }
+
+ private static double toDouble(Cursor value, ResultType targetType) {
+ switch (value.getType()) {
+ case BYTE:
+ case SHORT:
+ case INT:
+ case LONG:
+ return value.getInteger();
+ case FLOAT:
+ return value.getFloat();
+ case DOUBLE:
+ return value.getDouble();
+ case DECIMAL4:
+ case DECIMAL8:
+ case DECIMAL16:
+ return value.getDecimal().doubleValue();
+ default:
+ throw typeMismatch(value, targetType);
+ }
+ }
+
+ private static BigDecimal toBigDecimal(Cursor value, ResultType targetType) {
+ switch (value.getType()) {
+ case BYTE:
+ case SHORT:
+ case INT:
+ case LONG:
+ return BigDecimal.valueOf(value.getInteger());
+ case FLOAT:
+ return BigDecimal.valueOf(value.getFloat());
+ case DOUBLE:
+ return BigDecimal.valueOf(value.getDouble());
+ case DECIMAL4:
+ case DECIMAL8:
+ case DECIMAL16:
+ return value.getDecimal();
+ default:
+ throw typeMismatch(value, targetType);
+ }
+ }
+
+ private static long toTimestampMillis(Cursor value, ResultType targetType) {
+ switch (value.getType()) {
+ case DATE:
+ return Math.multiplyExact(value.getInteger(), TimeUnit.DAYS.toMillis(1));
+ case TIMESTAMP_TZ:
+ case TIMESTAMP_NTZ:
+ return Math.floorDiv(value.getInteger(), TimeUnit.MILLISECONDS.toMicros(1));
+ case TIMESTAMP_NANOS_TZ:
+ case TIMESTAMP_NANOS_NTZ:
+ return Math.floorDiv(value.getInteger(), TimeUnit.MILLISECONDS.toNanos(1));
+ default:
+ throw typeMismatch(value, targetType);
+ }
+ }
+
+ private static void requireType(Cursor value, Variant.Type expected, ResultType targetType) {
+ if (value.getType() != expected) {
+ throw typeMismatch(value, targetType);
+ }
+ }
+
+ private static IllegalArgumentException typeMismatch(Cursor value, ResultType targetType) {
+ return new IllegalArgumentException(
+ "Cannot convert Variant " + typeName(value.getType()) + " to " + targetType.name());
+ }
+
+ private static String typeName(Variant.Type type) {
+ switch (type) {
+ case DECIMAL4:
+ case DECIMAL8:
+ case DECIMAL16:
+ return "DECIMAL";
+ default:
+ return type.name();
+ }
+ }
+
+ private static String variantToJson(Variant variant) {
+ try {
+ // StringBuilder-backed writer: StringWriter wraps a synchronized StringBuffer and would pay a monitor
+ // acquisition per append on this per-row rendering path.
+ StringBuilderWriter writer = new StringBuilderWriter();
+ try (JsonGenerator generator = JSON_FACTORY.createGenerator(writer)) {
+ writeJsonValue(generator, variant);
+ }
+ return writer.toString();
+ } catch (IOException e) {
+ throw new IllegalStateException("Cannot render Variant as JSON", e);
+ }
+ }
+
+
+ private static void writeJsonValue(JsonGenerator generator, Variant variant)
+ throws IOException {
+ switch (variant.getType()) {
+ case OBJECT:
+ generator.writeStartObject();
+ for (int i = 0; i < variant.numObjectElements(); i++) {
+ Variant.ObjectField field = variant.getFieldAtIndex(i);
+ generator.writeFieldName(field.key);
+ writeJsonValue(generator, field.value);
+ }
+ generator.writeEndObject();
+ break;
+ case ARRAY:
+ generator.writeStartArray();
+ for (int i = 0; i < variant.numArrayElements(); i++) {
+ writeJsonValue(generator, variant.getElementAtIndex(i));
+ }
+ generator.writeEndArray();
+ break;
+ case NULL:
+ generator.writeNull();
+ break;
+ case BOOLEAN:
+ generator.writeBoolean(variant.getBoolean());
+ break;
+ case BYTE:
+ generator.writeNumber(variant.getByte());
+ break;
+ case SHORT:
+ generator.writeNumber(variant.getShort());
+ break;
+ case INT:
+ generator.writeNumber(variant.getInt());
+ break;
+ case LONG:
+ generator.writeNumber(variant.getLong());
+ break;
+ case FLOAT:
+ generator.writeNumber(variant.getFloat());
+ break;
+ case DOUBLE:
+ generator.writeNumber(variant.getDouble());
+ break;
+ case DECIMAL4:
+ case DECIMAL8:
+ case DECIMAL16:
+ generator.writeNumber(variant.getDecimal());
+ break;
+ case STRING:
+ generator.writeString(variant.getString());
+ break;
+ case BINARY:
+ generator.writeBinary(toBytes(variant.getBinary()));
+ break;
+ case UUID:
+ generator.writeString(variant.getUUID().toString());
+ break;
+ case DATE:
+ generator.writeString(LocalDate.ofEpochDay(variant.getInt()).toString());
+ break;
+ case TIMESTAMP_TZ:
+ generator.writeString(instantFromMicros(variant.getLong()).toString());
+ break;
+ case TIMESTAMP_NTZ:
+ generator.writeString(LocalDateTime.ofInstant(instantFromMicros(variant.getLong()), ZoneOffset.UTC).toString());
+ break;
+ case TIMESTAMP_NANOS_TZ:
+ generator.writeString(instantFromNanos(variant.getLong()).toString());
+ break;
+ case TIMESTAMP_NANOS_NTZ:
+ generator.writeString(LocalDateTime.ofInstant(instantFromNanos(variant.getLong()), ZoneOffset.UTC).toString());
+ break;
+ case TIME:
+ generator.writeString(LocalTime.ofNanoOfDay(Math.floorMod(variant.getLong() * NANOS_PER_MICRO, NANOS_PER_DAY))
+ .toString());
+ break;
+ default:
+ throw new IllegalStateException("Unsupported Variant type: " + variant.getType());
+ }
+ }
+
+ private static Instant instantFromMicros(long micros) {
+ long seconds = Math.floorDiv(micros, MICROS_PER_SECOND);
+ long nanos = Math.floorMod(micros, MICROS_PER_SECOND) * NANOS_PER_MICRO;
+ return Instant.ofEpochSecond(seconds, nanos);
+ }
+
+ private static Instant instantFromNanos(long nanos) {
+ return Instant.ofEpochSecond(Math.floorDiv(nanos, TimeUnit.SECONDS.toNanos(1)),
+ Math.floorMod(nanos, TimeUnit.SECONDS.toNanos(1)));
+ }
+
+ private static byte[] toBytes(ByteBuffer buffer) {
+ ByteBuffer view = buffer.slice();
+ byte[] bytes = new byte[view.remaining()];
+ view.get(bytes);
+ return bytes;
+ }
+
+ private static void appendJsonValue(JsonParser parser, JsonToken token, VariantBuilder builder, int depth)
+ throws IOException {
+ if (depth > MAX_JSON_NESTING_DEPTH) {
+ throw new IllegalArgumentException("Variant JSON exceeds maximum nesting depth " + MAX_JSON_NESTING_DEPTH);
+ }
+ switch (token) {
+ case START_OBJECT:
+ VariantObjectBuilder objectBuilder = builder.startObject();
+ while (parser.nextToken() != JsonToken.END_OBJECT) {
+ if (parser.currentToken() != JsonToken.FIELD_NAME) {
+ throw new IllegalArgumentException("Expected a JSON object field name");
+ }
+ objectBuilder.appendKey(parser.currentName());
+ JsonToken fieldValue = parser.nextToken();
+ if (fieldValue == null) {
+ throw new IllegalArgumentException("Unexpected end of JSON object");
+ }
+ appendJsonValue(parser, fieldValue, objectBuilder, depth + 1);
+ }
+ builder.endObject();
+ break;
+ case START_ARRAY:
+ VariantArrayBuilder arrayBuilder = builder.startArray();
+ while (true) {
+ JsonToken element = parser.nextToken();
+ if (element == JsonToken.END_ARRAY) {
+ break;
+ }
+ if (element == null) {
+ throw new IllegalArgumentException("Unexpected end of JSON array");
+ }
+ appendJsonValue(parser, element, arrayBuilder, depth + 1);
+ }
+ builder.endArray();
+ break;
+ case VALUE_NULL:
+ builder.appendNull();
+ break;
+ case VALUE_TRUE:
+ builder.appendBoolean(true);
+ break;
+ case VALUE_FALSE:
+ builder.appendBoolean(false);
+ break;
+ case VALUE_STRING:
+ builder.appendString(parser.getText());
+ break;
+ case VALUE_NUMBER_INT:
+ appendInteger(parser, builder);
+ break;
+ case VALUE_NUMBER_FLOAT:
+ appendDecimal(parser.getDecimalValue(), builder);
+ break;
+ default:
+ throw new IllegalArgumentException("Unsupported JSON token for Variant: " + token);
+ }
+ }
+
+ private static void appendInteger(JsonParser parser, VariantBuilder builder)
+ throws IOException {
+ switch (parser.getNumberType()) {
+ case INT:
+ builder.appendInt(parser.getIntValue());
+ break;
+ case LONG:
+ builder.appendLong(parser.getLongValue());
+ break;
+ case BIG_INTEGER:
+ appendBigInteger(parser.getBigIntegerValue(), builder);
+ break;
+ default:
+ throw new IllegalArgumentException("Unsupported JSON integer representation: " + parser.getNumberType());
+ }
+ }
+
+ private static void appendBigInteger(BigInteger value, VariantBuilder builder) {
+ if (value.bitLength() < Integer.SIZE) {
+ builder.appendInt(value.intValue());
+ } else if (value.bitLength() < Long.SIZE) {
+ builder.appendLong(value.longValue());
+ } else {
+ appendDecimal(new BigDecimal(value), builder);
+ }
+ }
+
+ private static void appendDecimal(BigDecimal value, VariantBuilder builder) {
+ BigDecimal normalized = value;
+ if (normalized.scale() < 0) {
+ // Parquet Variant stores scale as an unsigned byte. Expand exponent notation exactly instead of allowing a
+ // negative scale to wrap during encoding.
+ long expandedPrecision = (long) normalized.precision() - normalized.scale();
+ if (normalized.signum() != 0 && expandedPrecision > MAX_VARIANT_DECIMAL_PRECISION) {
+ throw unsupportedVariantDecimal(value);
+ }
+ normalized = normalized.signum() == 0 ? BigDecimal.ZERO : normalized.setScale(0);
+ } else if (normalized.scale() > MAX_VARIANT_DECIMAL_SCALE) {
+ // Accept values whose excessive lexical scale consists only of insignificant trailing zeros.
+ normalized = normalized.stripTrailingZeros();
+ if (normalized.scale() < 0) {
+ normalized = normalized.setScale(0);
+ }
+ }
+ byte[] unscaledBytes = normalized.unscaledValue().toByteArray();
+ if (normalized.scale() > MAX_VARIANT_DECIMAL_SCALE
+ || normalized.precision() > MAX_VARIANT_DECIMAL_PRECISION
+ || unscaledBytes.length > MAX_VARIANT_DECIMAL_BYTES) {
+ throw unsupportedVariantDecimal(value);
+ }
+ builder.appendDecimal(normalized);
+ }
+
+ private static IllegalArgumentException unsupportedVariantDecimal(BigDecimal value) {
+ return new IllegalArgumentException(
+ "JSON decimal exceeds Parquet Variant decimal(38) encoding: precision=" + value.precision()
+ + ", scale=" + value.scale());
+ }
+
+ /// Mutable zero-copy view over one selected value in a Pinot envelope.
+ ///
+ /// <p>The constants and layouts used here mirror Parquet Variant encoding version 1. Keeping this cursor on
+ /// {@link ReusableResult} avoids allocating envelope views, Variant wrappers, and navigation wrappers for every row.
+ private static final class Cursor {
Review Comment:
Added an explicit pinned sync policy in `VariantUtils` and a reflection test comparing the cursor masks/tags and every `Variant.Type` with parquet-variant 1.18.0. Unknown metadata versions and primitive tags now fail closed with actionable strict errors and tolerant nulls. I deliberately did not fall back through an old decoder for an unknown encoding because that risks silent misinterpretation.
##########
pinot-common/src/main/java/org/apache/pinot/common/utils/VariantUtils.java:
##########
@@ -0,0 +1,2060 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils;
+
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.sql.Timestamp;
+import java.time.Instant;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.LocalTime;
+import java.time.ZoneOffset;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Locale;
+import java.util.Objects;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.Nullable;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.commons.io.output.StringBuilderWriter;
+import org.apache.parquet.variant.Variant;
+import org.apache.parquet.variant.VariantArrayBuilder;
+import org.apache.parquet.variant.VariantBuilder;
+import org.apache.parquet.variant.VariantObjectBuilder;
+import org.apache.pinot.spi.data.FieldSpec.DataType;
+import org.apache.pinot.spi.utils.ByteArray;
+import org.apache.pinot.spi.utils.UuidUtils;
+import org.apache.pinot.spi.utils.VariantEnvelope;
+
+
+/// Query-side operations for Pinot {@code VARIANT} values.
+///
+/// <p>The utility navigates the Parquet Variant binary representation directly. It never materializes a JSON tree.
+/// Instances are not required, and stateless convenience methods are thread-safe. Overloads that accept a
+/// caller-provided {@link ReusableResult} require that result to be thread-confined and not shared by concurrent calls.
+/// An empty byte array is Pinot's SQL-null placeholder and is never decoded as an envelope.
+public final class VariantUtils {
+ public static final String RAW_VARIANT_REQUIRES_NULL_HANDLING_ERROR =
+ "Raw VARIANT projection requires query null handling to be enabled; set enableNullHandling=true";
+
+ private static final JsonFactory JSON_FACTORY = new JsonFactory();
+ private static final BigDecimal MIN_INT_DECIMAL = BigDecimal.valueOf(Integer.MIN_VALUE);
+ private static final BigDecimal MAX_INT_DECIMAL = BigDecimal.valueOf(Integer.MAX_VALUE);
+ private static final BigDecimal MIN_LONG_DECIMAL = BigDecimal.valueOf(Long.MIN_VALUE);
+ private static final BigDecimal MAX_LONG_DECIMAL = BigDecimal.valueOf(Long.MAX_VALUE);
+ private static final int MAX_JSON_NESTING_DEPTH = 100;
+ private static final int MAX_VARIANT_DECIMAL_PRECISION = 38;
+ private static final int MAX_VARIANT_DECIMAL_SCALE = 38;
+ private static final int MAX_VARIANT_DECIMAL_BYTES = 16;
+ private static final long MICROS_PER_SECOND = TimeUnit.SECONDS.toMicros(1);
+ private static final long NANOS_PER_MICRO = TimeUnit.MICROSECONDS.toNanos(1);
+ private static final long NANOS_PER_DAY = TimeUnit.DAYS.toNanos(1);
+ private static final int VARIANT_BASIC_TYPE_MASK = 0x03;
+ private static final int VARIANT_PRIMITIVE_TYPE_MASK = 0x3F;
+ private static final int VARIANT_PRIMITIVE = 0;
+ private static final int VARIANT_SHORT_STRING = 1;
+ private static final int VARIANT_OBJECT = 2;
+ private static final int VARIANT_ARRAY = 3;
+ private static final int VARIANT_NULL = 0;
+ private static final int VARIANT_TRUE = 1;
+ private static final int VARIANT_FALSE = 2;
+ private static final int VARIANT_INT8 = 3;
+ private static final int VARIANT_INT16 = 4;
+ private static final int VARIANT_INT32 = 5;
+ private static final int VARIANT_INT64 = 6;
+ private static final int VARIANT_DOUBLE = 7;
+ private static final int VARIANT_DECIMAL4 = 8;
+ private static final int VARIANT_DECIMAL8 = 9;
+ private static final int VARIANT_DECIMAL16 = 10;
+ private static final int VARIANT_DATE = 11;
+ private static final int VARIANT_TIMESTAMP_TZ = 12;
+ private static final int VARIANT_TIMESTAMP_NTZ = 13;
+ private static final int VARIANT_FLOAT = 14;
+ private static final int VARIANT_BINARY = 15;
+ private static final int VARIANT_LONG_STRING = 16;
+ private static final int VARIANT_TIME = 17;
+ private static final int VARIANT_TIMESTAMP_NANOS_TZ = 18;
+ private static final int VARIANT_TIMESTAMP_NANOS_NTZ = 19;
+ private static final int VARIANT_UUID = 20;
+ private static final int VARIANT_METADATA_VERSION_MASK = 0x0F;
+ private static final int VARIANT_METADATA_VERSION = 1;
+ private static final int OBJECT_BINARY_SEARCH_THRESHOLD = 32;
+ private static final int INVALID_UTF8_COMPARISON = Integer.MIN_VALUE;
+ private static final VariantPath ROOT_PATH = new VariantPath(new PathElement[0]);
+
+ private VariantUtils() {
+ }
+
+ /// Returns whether a final result containing raw VARIANT values requires query null handling. Without a null bitmap,
+ /// Pinot's reserved empty-byte SQL-null placeholder cannot be distinguished from a logical Variant value.
+ public static boolean requiresNullHandlingForRawVariantResult(DataSchema resultSchema,
+ boolean nullHandlingEnabled) {
+ if (nullHandlingEnabled) {
+ return false;
+ }
+ for (DataSchema.ColumnDataType dataType : resultSchema.getColumnDataTypes()) {
+ if (dataType == DataSchema.ColumnDataType.VARIANT) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /// Statically supported result types for {@code variantGet} and {@code tryVariantGet}.
+ public enum ResultType {
+ BOOLEAN(DataType.BOOLEAN, SqlTypeName.BOOLEAN),
+ INT(DataType.INT, SqlTypeName.INTEGER),
+ LONG(DataType.LONG, SqlTypeName.BIGINT),
+ FLOAT(DataType.FLOAT, SqlTypeName.REAL),
+ DOUBLE(DataType.DOUBLE, SqlTypeName.DOUBLE),
+ BIG_DECIMAL(DataType.BIG_DECIMAL, SqlTypeName.DECIMAL),
+ STRING(DataType.STRING, SqlTypeName.VARCHAR),
+ BYTES(DataType.BYTES, SqlTypeName.VARBINARY),
+ UUID(DataType.UUID, SqlTypeName.UUID),
+ TIMESTAMP(DataType.TIMESTAMP, SqlTypeName.TIMESTAMP),
+ VARIANT(DataType.VARIANT, SqlTypeName.VARIANT),
+ JSON(DataType.JSON, SqlTypeName.VARCHAR);
+
+ private final DataType _dataType;
+ private final SqlTypeName _sqlTypeName;
+
+ ResultType(DataType dataType, SqlTypeName sqlTypeName) {
+ _dataType = dataType;
+ _sqlTypeName = sqlTypeName;
+ }
+
+ public DataType getDataType() {
+ return _dataType;
+ }
+
+ public SqlTypeName getSqlTypeName() {
+ return _sqlTypeName;
+ }
+ }
+
+ /// An immutable, pre-parsed Variant path. The v1 grammar supports {@code $}, dot-separated object fields, and
+ /// non-negative array subscripts.
+ public static final class VariantPath {
+ private final PathElement[] _elements;
+
+ private VariantPath(PathElement[] elements) {
+ _elements = elements;
+ }
+ }
+
+ /// Reusable, unboxed destination for vectorized Variant extraction.
+ ///
+ /// <p>Only the getter corresponding to the requested {@link ResultType} is defined after a successful extraction.
+ /// The instance is mutable and not thread-safe; callers should retain one per transform-function instance. Every
+ /// extraction may replace its state. Each successful byte-valued extraction installs a newly materialized array.
+ /// Values returned as {@code byte[]} or as a {@link ByteArray} may be retained after this result is reused, but they
+ /// are read-only by contract and must be copied before mutation.
+ public static final class ReusableResult {
+ private final Cursor _cursor = new Cursor();
+ private int _intValue;
+ private long _longValue;
+ private float _floatValue;
+ private double _doubleValue;
+ private BigDecimal _bigDecimalValue;
+ private String _stringValue;
+ private byte[] _bytesValue;
+
+ public int getIntValue() {
+ return _intValue;
+ }
+
+ public long getLongValue() {
+ return _longValue;
+ }
+
+ public float getFloatValue() {
+ return _floatValue;
+ }
+
+ public double getDoubleValue() {
+ return _doubleValue;
+ }
+
+ public BigDecimal getBigDecimalValue() {
+ return _bigDecimalValue;
+ }
+
+ public String getStringValue() {
+ return _stringValue;
+ }
+
+ /// Returns the extracted BYTES, VARIANT, or direct 16-byte UUID representation.
+ ///
+ /// <p>The returned array is replaced, but not mutated, by the next byte-valued extraction. It may be retained after
+ /// this result is reused, but must be treated as immutable and copied before mutation.
+ public byte[] getBytesValue() {
+ return _bytesValue;
+ }
+
+ public UUID getUuidValue() {
+ return UuidUtils.toUUID(_bytesValue);
+ }
+
+ /// Materializes the extracted value in the external representation used by scalar functions and ingestion.
+ ///
+ /// <p>For BYTES and VARIANT, the returned {@code byte[]} may be retained after this result is reused. It must be
+ /// treated as immutable and copied before mutation.
+ public Object toExternalValue(ResultType resultType) {
+ switch (resultType) {
+ case BOOLEAN:
+ return _intValue != 0;
+ case INT:
+ return _intValue;
+ case LONG:
+ return _longValue;
+ case FLOAT:
+ return _floatValue;
+ case DOUBLE:
+ return _doubleValue;
+ case BIG_DECIMAL:
+ return _bigDecimalValue;
+ case STRING:
+ case JSON:
+ return _stringValue;
+ case BYTES:
+ case VARIANT:
+ return _bytesValue;
+ case UUID:
+ return UuidUtils.toUUID(_bytesValue);
+ case TIMESTAMP:
+ return new Timestamp(_longValue);
+ default:
+ throw new IllegalStateException("Unhandled Variant target type: " + resultType);
+ }
+ }
+
+ /// Materializes the extracted value in {@link DataSchema}'s internal representation.
+ ///
+ /// <p>TIMESTAMP remains epoch milliseconds and UUID wraps the directly copied 16-byte value, avoiding an
+ /// external-object round trip in the multi-stage engine. For BYTES, UUID, and VARIANT, the returned
+ /// {@link ByteArray} wraps a newly materialized array that may be retained after this result is reused. Neither the
+ /// wrapper nor its array may be mutated; callers must copy the array before mutation.
+ public Object toInternalValue(ResultType resultType) {
+ switch (resultType) {
+ case BOOLEAN:
+ return _intValue;
+ case INT:
+ return _intValue;
+ case LONG:
+ case TIMESTAMP:
+ return _longValue;
+ case FLOAT:
+ return _floatValue;
+ case DOUBLE:
+ return _doubleValue;
+ case BIG_DECIMAL:
+ return _bigDecimalValue;
+ case STRING:
+ case JSON:
+ return _stringValue;
+ case BYTES:
+ case UUID:
+ case VARIANT:
+ return new ByteArray(_bytesValue);
+ default:
+ throw new IllegalStateException("Unhandled Variant target type: " + resultType);
+ }
+ }
+ }
+
+ /// Parses a target type literal once for reuse by a transform function.
+ public static ResultType parseResultType(String targetType) {
+ if (targetType == null) {
+ throw new IllegalArgumentException("Variant target type must not be null");
+ }
+ try {
+ return ResultType.valueOf(targetType.trim().toUpperCase(Locale.ROOT));
+ } catch (IllegalArgumentException e) {
+ throw new IllegalArgumentException("Unsupported Variant target type: " + targetType, e);
+ }
+ }
+
+ /// Compiles a v1 Variant path.
+ public static VariantPath compilePath(String path) {
+ if (path == null || path.isEmpty() || path.charAt(0) != '$') {
+ throw new IllegalArgumentException("Variant path must start with '$': " + path);
+ }
+ List<PathElement> elements = new ArrayList<>();
+ int index = 1;
+ while (index < path.length()) {
+ char current = path.charAt(index);
+ if (current == '.') {
+ int fieldStart = ++index;
+ while (index < path.length()) {
+ char next = path.charAt(index);
+ if (next == '.' || next == '[') {
+ break;
+ }
+ index++;
+ }
+ if (fieldStart == index) {
+ throw new IllegalArgumentException("Variant path contains an empty field: " + path);
+ }
+ elements.add(PathElement.forField(path.substring(fieldStart, index)));
+ } else if (current == '[') {
+ int subscriptStart = ++index;
+ while (index < path.length() && Character.isDigit(path.charAt(index))) {
+ index++;
+ }
+ if (subscriptStart == index || index >= path.length() || path.charAt(index) != ']') {
+ throw new IllegalArgumentException("Invalid Variant array subscript in path: " + path);
+ }
+ try {
+ elements.add(PathElement.forIndex(Integer.parseInt(path.substring(subscriptStart, index))));
+ } catch (NumberFormatException e) {
+ throw new IllegalArgumentException("Variant array subscript is too large in path: " + path, e);
+ }
+ index++;
+ } else {
+ throw new IllegalArgumentException("Unexpected character at offset " + index + " in Variant path: " + path);
+ }
+ }
+ return new VariantPath(elements.toArray(new PathElement[0]));
+ }
+
+ /// Extracts a Variant value. A missing path or SQL null returns Java null; a Variant null remains an encoded Variant
+ /// value.
+ @Nullable
+ public static byte[] variantGet(@Nullable byte[] envelope, String path) {
+ return (byte[]) variantGet(envelope, compilePath(path), ResultType.VARIANT);
+ }
+
+ /// Strictly extracts and converts a value. A missing path or SQL null returns Java null. A Variant null remains
+ /// encoded when the target type is {@link ResultType#VARIANT}, and returns Java null for other target types. An
+ /// incompatible non-null value throws.
+ @Nullable
+ public static Object variantGet(@Nullable byte[] envelope, String path, String targetType) {
+ return variantGet(envelope, compilePath(path), parseResultType(targetType));
+ }
+
+ /// Strictly extracts using pre-parsed path and type values.
+ @Nullable
+ public static Object variantGet(@Nullable byte[] envelope, VariantPath path, ResultType targetType) {
+ ReusableResult result = new ReusableResult();
+ return extractInto(envelope, path, targetType, result) ? result.toExternalValue(targetType) : null;
+ }
+
+ /// Strictly extracts into a reusable, unboxed result.
+ ///
+ /// @return {@code false} for SQL null, a missing path, or Variant null converted to a non-Variant target
+ public static boolean extractInto(@Nullable byte[] envelope, VariantPath path, ResultType targetType,
+ ReusableResult result) {
+ Objects.requireNonNull(result, "result must not be null");
+ if (isSqlNull(envelope)) {
+ return false;
+ }
+ Objects.requireNonNull(path, "path must not be null");
+ Objects.requireNonNull(targetType, "targetType must not be null");
+ Cursor cursor = result._cursor;
+ if (!cursor.navigate(envelope, path)) {
+ return false;
+ }
+ if (cursor.getType() == Variant.Type.NULL && targetType != ResultType.VARIANT) {
+ return false;
+ }
+ convert(cursor, targetType, result);
+ return true;
+ }
+
+ /// Tolerant Variant extraction. Malformed input returns Java null.
+ @Nullable
+ public static byte[] tryVariantGet(@Nullable byte[] envelope, String path) {
+ return (byte[]) tryVariantGet(envelope, compilePath(path), ResultType.VARIANT);
+ }
+
+ /// Tolerant typed extraction. Malformed input and incompatible types return Java null.
+ @Nullable
+ public static Object tryVariantGet(@Nullable byte[] envelope, String path, String targetType) {
+ try {
+ return tryVariantGet(envelope, compilePath(path), parseResultType(targetType));
+ } catch (RuntimeException e) {
+ return null;
+ }
+ }
+
+ /// Tolerant extraction using pre-parsed path and type values.
+ @Nullable
+ public static Object tryVariantGet(@Nullable byte[] envelope, VariantPath path, ResultType targetType) {
+ try {
+ ReusableResult result = new ReusableResult();
+ return tryExtractInto(envelope, path, targetType, result) ? result.toExternalValue(targetType) : null;
+ } catch (RuntimeException e) {
+ return null;
+ }
+ }
+
+ /// Tolerantly extracts into a reusable, unboxed result.
+ ///
+ /// @return {@code false} for SQL null, missing paths, Variant null converted to a non-Variant target,
+ /// malformed input, or an incompatible conversion
+ public static boolean tryExtractInto(@Nullable byte[] envelope, VariantPath path, ResultType targetType,
+ ReusableResult result) {
+ Objects.requireNonNull(result, "result must not be null");
+ if (isSqlNull(envelope)) {
+ return false;
+ }
+ Objects.requireNonNull(path, "path must not be null");
+ Objects.requireNonNull(targetType, "targetType must not be null");
+ Cursor cursor = result._cursor;
+ try {
+ if (!cursor.navigate(envelope, path)) {
+ return false;
+ }
+ if (cursor.getType() == Variant.Type.NULL && targetType != ResultType.VARIANT) {
+ return false;
+ }
+ return tryConvert(cursor, targetType, result);
+ } catch (IllegalArgumentException | IllegalStateException | UnsupportedOperationException
+ | IndexOutOfBoundsException e) {
+ // Cursor operations use these exceptions only for malformed or unsupported Variant encodings.
+ return false;
+ }
+ }
+
+ /// Returns whether the path is present. A present Variant null counts as present.
+ @Nullable
+ public static Boolean variantExists(@Nullable byte[] envelope, String path) {
+ return variantExists(envelope, compilePath(path));
+ }
+
+ /// Returns whether a compiled path is present. A present Variant null counts as present.
+ @Nullable
+ public static Boolean variantExists(@Nullable byte[] envelope, VariantPath path) {
+ return variantExists(envelope, path, new ReusableResult());
+ }
+
+ /// Allocation-free compiled-path form of {@link #variantExists(byte[], VariantPath)} when the caller retains the
+ /// supplied result between rows.
+ @Nullable
+ public static Boolean variantExists(@Nullable byte[] envelope, VariantPath path, ReusableResult result) {
+ Objects.requireNonNull(result, "result must not be null");
+ if (isSqlNull(envelope)) {
+ return null;
+ }
+ return result._cursor.navigate(envelope, Objects.requireNonNull(path, "path must not be null"));
+ }
+
+ /// Returns whether the root value is a Variant null. SQL null is not a Variant null.
+ public static boolean isVariantNull(@Nullable byte[] envelope) {
+ return isVariantNull(envelope, ROOT_PATH, new ReusableResult());
+ }
+
+ /// Returns whether a present value at the path is a Variant null. SQL null and missing paths return false.
+ public static boolean isVariantNull(@Nullable byte[] envelope, String path) {
+ return isVariantNull(envelope, compilePath(path));
+ }
+
+ /// Returns whether a present value at a compiled path is a Variant null. SQL null and missing paths return false.
+ public static boolean isVariantNull(@Nullable byte[] envelope, VariantPath path) {
+ return isVariantNull(envelope, path, new ReusableResult());
+ }
+
+ /// Allocation-free compiled-path form of {@link #isVariantNull(byte[], VariantPath)} when the caller retains the
+ /// supplied result between rows.
+ public static boolean isVariantNull(@Nullable byte[] envelope, VariantPath path, ReusableResult result) {
+ Objects.requireNonNull(result, "result must not be null");
+ if (isSqlNull(envelope)) {
+ return false;
+ }
+ Cursor cursor = result._cursor;
+ return cursor.navigate(envelope, Objects.requireNonNull(path, "path must not be null"))
+ && cursor.getType() == Variant.Type.NULL;
+ }
+
+ /// Returns the Variant type name at the root, or Java null for SQL null.
+ @Nullable
+ public static String variantTypeOf(@Nullable byte[] envelope) {
+ return variantTypeOf(envelope, ROOT_PATH, new ReusableResult());
+ }
+
+ /// Returns the Variant type name at a path, or Java null for SQL null or a missing path.
+ @Nullable
+ public static String variantTypeOf(@Nullable byte[] envelope, String path) {
+ return variantTypeOf(envelope, compilePath(path));
+ }
+
+ /// Returns the Variant type name at a compiled path, or Java null for SQL null or a missing path.
+ @Nullable
+ public static String variantTypeOf(@Nullable byte[] envelope, VariantPath path) {
+ return variantTypeOf(envelope, path, new ReusableResult());
+ }
+
+ /// Allocation-free compiled-path form of {@link #variantTypeOf(byte[], VariantPath)} when the caller retains the
+ /// supplied result between rows.
+ @Nullable
+ public static String variantTypeOf(@Nullable byte[] envelope, VariantPath path, ReusableResult result) {
+ Objects.requireNonNull(result, "result must not be null");
+ if (isSqlNull(envelope)) {
+ return null;
+ }
+ Cursor cursor = result._cursor;
+ return cursor.navigate(envelope, Objects.requireNonNull(path, "path must not be null"))
+ ? typeName(cursor.getType()) : null;
+ }
+
+ /// Renders the Variant value as canonical JSON text without constructing a JSON tree.
+ @Nullable
+ public static String variantToJson(@Nullable byte[] envelope) {
+ return variantToJson(envelope, new ReusableResult());
+ }
+
+ /// Allocation-reduced form of [#variantToJson(byte[])] when the caller retains the supplied result between rows.
+ @Nullable
+ public static String variantToJson(@Nullable byte[] envelope, ReusableResult result) {
+ Objects.requireNonNull(result, "result must not be null");
+ if (isSqlNull(envelope)) {
+ return null;
+ }
+ Cursor cursor = result._cursor;
+ cursor.navigate(envelope, ROOT_PATH);
+ return variantToJson(cursor.asVariant());
+ }
+
+ /// Parses JSON text into a Pinot Variant envelope without constructing a JSON tree.
+ @Nullable
+ public static byte[] parseJsonToVariant(@Nullable String json) {
+ if (json == null) {
+ return null;
+ }
+ try (JsonParser parser = JSON_FACTORY.createParser(json)) {
+ JsonToken token = parser.nextToken();
+ if (token == null) {
+ throw new IllegalArgumentException("Cannot parse empty text as Variant");
+ }
+ VariantBuilder builder = new VariantBuilder();
+ appendJsonValue(parser, token, builder, 0);
+ if (parser.nextToken() != null) {
+ throw new IllegalArgumentException("Unexpected trailing token after Variant JSON value");
+ }
+ Variant variant = builder.build();
+ return VariantEnvelope.encode(variant.getMetadataBuffer(), variant.getValueBuffer());
+ } catch (IOException | RuntimeException e) {
+ throw new IllegalArgumentException("Cannot parse JSON as Variant", e);
+ }
+ }
+
+ /// Tolerant JSON parser. Malformed or unsupported input returns Java null.
+ @Nullable
+ public static byte[] tryParseJsonToVariant(@Nullable String json) {
+ try {
+ return parseJsonToVariant(json);
+ } catch (RuntimeException e) {
+ return null;
+ }
+ }
+
+ private static boolean isSqlNull(@Nullable byte[] envelope) {
+ return envelope == null || envelope.length == 0;
+ }
+
+ private static void convert(Cursor value, ResultType targetType, ReusableResult result) {
+ switch (targetType) {
+ case BOOLEAN:
+ requireType(value, Variant.Type.BOOLEAN, targetType);
+ result._intValue = value.getBoolean() ? 1 : 0;
+ break;
+ case INT:
+ result._intValue = toInt(value, targetType);
+ break;
+ case LONG:
+ result._longValue = toLong(value, targetType);
+ break;
+ case FLOAT:
+ result._floatValue = toFloat(value, targetType);
+ break;
+ case DOUBLE:
+ result._doubleValue = toDouble(value, targetType);
+ break;
+ case BIG_DECIMAL:
+ result._bigDecimalValue = toBigDecimal(value, targetType);
+ break;
+ case STRING:
+ requireType(value, Variant.Type.STRING, targetType);
+ result._stringValue = value.getString();
+ break;
+ case BYTES:
+ requireType(value, Variant.Type.BINARY, targetType);
+ result._bytesValue = value.getBinary();
+ break;
+ case UUID:
+ requireType(value, Variant.Type.UUID, targetType);
+ result._bytesValue = value.getUuidBytes();
+ break;
+ case TIMESTAMP:
+ result._longValue = toTimestampMillis(value, targetType);
+ break;
+ case VARIANT:
+ result._bytesValue = value.copyEnvelope();
+ break;
+ case JSON:
+ result._stringValue = variantToJson(value.asVariant());
+ break;
+ default:
+ throw new IllegalStateException("Unhandled Variant target type: " + targetType);
+ }
+ }
+
+ private static boolean tryConvert(Cursor value, ResultType targetType, ReusableResult result) {
+ Variant.Type valueType = value.getType();
+ switch (targetType) {
+ case BOOLEAN:
+ if (valueType != Variant.Type.BOOLEAN) {
+ return false;
+ }
+ result._intValue = value.getBoolean() ? 1 : 0;
+ return true;
+ case INT:
+ return tryConvertToInt(value, valueType, result);
+ case LONG:
+ return tryConvertToLong(value, valueType, result);
+ case FLOAT:
+ switch (valueType) {
+ case BYTE:
+ case SHORT:
+ case INT:
+ case LONG:
+ result._floatValue = value.getInteger();
+ return true;
+ case FLOAT:
+ result._floatValue = value.getFloat();
+ return true;
+ case DOUBLE:
+ result._floatValue = (float) value.getDouble();
+ return true;
+ case DECIMAL4:
+ case DECIMAL8:
+ case DECIMAL16:
+ result._floatValue = value.getDecimal().floatValue();
+ return true;
+ default:
+ return false;
+ }
+ case DOUBLE:
+ switch (valueType) {
+ case BYTE:
+ case SHORT:
+ case INT:
+ case LONG:
+ result._doubleValue = value.getInteger();
+ return true;
+ case FLOAT:
+ result._doubleValue = value.getFloat();
+ return true;
+ case DOUBLE:
+ result._doubleValue = value.getDouble();
+ return true;
+ case DECIMAL4:
+ case DECIMAL8:
+ case DECIMAL16:
+ result._doubleValue = value.getDecimal().doubleValue();
+ return true;
+ default:
+ return false;
+ }
+ case BIG_DECIMAL:
+ switch (valueType) {
+ case BYTE:
+ case SHORT:
+ case INT:
+ case LONG:
+ result._bigDecimalValue = BigDecimal.valueOf(value.getInteger());
+ return true;
+ case FLOAT:
+ float floatValue = value.getFloat();
+ if (!Float.isFinite(floatValue)) {
+ return false;
+ }
+ result._bigDecimalValue = BigDecimal.valueOf(floatValue);
+ return true;
+ case DOUBLE:
+ double doubleValue = value.getDouble();
+ if (!Double.isFinite(doubleValue)) {
+ return false;
+ }
+ result._bigDecimalValue = BigDecimal.valueOf(doubleValue);
+ return true;
+ case DECIMAL4:
+ case DECIMAL8:
+ case DECIMAL16:
+ result._bigDecimalValue = value.getDecimal();
+ return true;
+ default:
+ return false;
+ }
+ case STRING:
+ if (valueType != Variant.Type.STRING) {
+ return false;
+ }
+ result._stringValue = value.getString();
+ return true;
+ case BYTES:
+ if (valueType != Variant.Type.BINARY) {
+ return false;
+ }
+ result._bytesValue = value.getBinary();
+ return true;
+ case UUID:
+ if (valueType != Variant.Type.UUID) {
+ return false;
+ }
+ result._bytesValue = value.getUuidBytes();
+ return true;
+ case TIMESTAMP:
+ switch (valueType) {
+ case DATE:
+ result._longValue = value.getInteger() * TimeUnit.DAYS.toMillis(1);
Review Comment:
The tolerant path now uses `Math.multiplyExact` too. One correction to the reproduction: Parquet DATE is signed int32, and `Integer.MAX_VALUE * 86,400,000` still fits in a long, so a legal DATE cannot overflow. The max-int32 boundary now returns the exact millis in both strict and tolerant paths; malformed wider payloads are rejected.
##########
pinot-common/src/test/java/org/apache/pinot/common/utils/VariantUtilsTest.java:
##########
@@ -0,0 +1,1153 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils;
+
+import java.math.BigDecimal;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.sql.Timestamp;
+import java.util.Arrays;
+import java.util.EnumSet;
+import java.util.List;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Consumer;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.parquet.variant.Variant;
+import org.apache.parquet.variant.VariantBuilder;
+import org.apache.parquet.variant.VariantObjectBuilder;
+import org.apache.pinot.common.utils.DataSchema.ColumnDataType;
+import org.apache.pinot.common.utils.VariantUtils.ResultType;
+import org.apache.pinot.common.utils.VariantUtils.ReusableResult;
+import org.apache.pinot.common.utils.VariantUtils.VariantPath;
+import org.apache.pinot.spi.data.FieldSpec.DataType;
+import org.apache.pinot.spi.utils.ByteArray;
+import org.apache.pinot.spi.utils.UuidUtils;
+import org.apache.pinot.spi.utils.VariantEnvelope;
+import org.testng.annotations.DataProvider;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertNull;
+import static org.testng.Assert.assertSame;
+import static org.testng.Assert.assertThrows;
+import static org.testng.Assert.assertTrue;
+
+
+public class VariantUtilsTest {
+ @Test
+ public void testRawVariantResultRequiresNullHandling() {
+ DataSchema variantSchema =
+ new DataSchema(new String[]{"payload"}, new ColumnDataType[]{ColumnDataType.VARIANT});
+ DataSchema typedSchema =
+ new DataSchema(new String[]{"eventType"}, new ColumnDataType[]{ColumnDataType.STRING});
+
+ assertTrue(VariantUtils.requiresNullHandlingForRawVariantResult(variantSchema, false));
+ assertFalse(VariantUtils.requiresNullHandlingForRawVariantResult(variantSchema, true));
+ assertFalse(VariantUtils.requiresNullHandlingForRawVariantResult(typedSchema, false));
+ }
+
+ @Test
+ public void testResultTypeContract() {
+ assertEquals(ResultType.BOOLEAN.getDataType(), DataType.BOOLEAN);
+ assertEquals(ResultType.BOOLEAN.getSqlTypeName(), SqlTypeName.BOOLEAN);
+ assertEquals(ResultType.INT.getDataType(), DataType.INT);
+ assertEquals(ResultType.INT.getSqlTypeName(), SqlTypeName.INTEGER);
+ assertEquals(ResultType.LONG.getDataType(), DataType.LONG);
+ assertEquals(ResultType.LONG.getSqlTypeName(), SqlTypeName.BIGINT);
+ assertEquals(ResultType.FLOAT.getDataType(), DataType.FLOAT);
+ assertEquals(ResultType.FLOAT.getSqlTypeName(), SqlTypeName.REAL);
+ assertEquals(ResultType.DOUBLE.getDataType(), DataType.DOUBLE);
+ assertEquals(ResultType.DOUBLE.getSqlTypeName(), SqlTypeName.DOUBLE);
+ assertEquals(ResultType.BIG_DECIMAL.getDataType(), DataType.BIG_DECIMAL);
+ assertEquals(ResultType.BIG_DECIMAL.getSqlTypeName(), SqlTypeName.DECIMAL);
+ assertEquals(ResultType.STRING.getDataType(), DataType.STRING);
+ assertEquals(ResultType.STRING.getSqlTypeName(), SqlTypeName.VARCHAR);
+ assertEquals(ResultType.BYTES.getDataType(), DataType.BYTES);
+ assertEquals(ResultType.BYTES.getSqlTypeName(), SqlTypeName.VARBINARY);
+ assertEquals(ResultType.UUID.getDataType(), DataType.UUID);
+ assertEquals(ResultType.UUID.getSqlTypeName(), SqlTypeName.UUID);
+ assertEquals(ResultType.TIMESTAMP.getDataType(), DataType.TIMESTAMP);
+ assertEquals(ResultType.TIMESTAMP.getSqlTypeName(), SqlTypeName.TIMESTAMP);
+ assertEquals(ResultType.VARIANT.getDataType(), DataType.VARIANT);
+ assertEquals(ResultType.VARIANT.getSqlTypeName(), SqlTypeName.VARIANT);
+ assertEquals(ResultType.JSON.getDataType(), DataType.JSON);
+ assertEquals(ResultType.JSON.getSqlTypeName(), SqlTypeName.VARCHAR);
+ }
+
+ @Test
+ public void testDirectBinaryPathExtractionAndPredicates() {
+ byte[] variant = VariantUtils.parseJsonToVariant(
+ "{\"eventType\":\"click\",\"items\":[{\"price\":12.5},null],\"active\":true}");
+
+ assertEquals(VariantUtils.variantGet(variant, "$.eventType", "STRING"), "click");
+ assertEquals((double) VariantUtils.variantGet(variant, "$.items[0].price", "DOUBLE"), 12.5);
+ assertEquals(VariantUtils.variantGet(variant, "$.active", "BOOLEAN"), true);
+ assertTrue(VariantUtils.variantExists(variant, "$.items[1]"));
+ assertFalse(VariantUtils.variantExists(variant, "$.missing"));
+ assertTrue(VariantUtils.isVariantNull(variant, "$.items[1]"));
+ assertFalse(VariantUtils.isVariantNull(variant, "$.missing"));
+ assertEquals(VariantUtils.variantTypeOf(variant, "$.items[0]"), "OBJECT");
+ assertEquals(VariantUtils.variantTypeOf(variant, "$.items[0].price"), "DECIMAL");
+ }
+
+ @Test
+ public void testStrictAndTolerantExtraction() {
+ byte[] variant = VariantUtils.parseJsonToVariant("{\"eventType\":\"click\",\"score\":\"not-a-number\"}");
+
+ assertNull(VariantUtils.variantGet(variant, "$.missing", "STRING"));
+ assertThrows(IllegalArgumentException.class, () -> VariantUtils.variantGet(variant, "$.score", "DOUBLE"));
+ assertNull(VariantUtils.tryVariantGet(variant, "$.missing", "STRING"));
+ assertNull(VariantUtils.tryVariantGet(variant, "$.score", "DOUBLE"));
+ }
+
+ @Test
+ public void testReusableResultStrictAndTolerantExtraction() {
+ byte[] variant = VariantUtils.parseJsonToVariant("{\"value\":7,\"null\":null,\"text\":\"x\"}");
+ VariantPath valuePath = VariantUtils.compilePath("$.value");
+ ReusableResult result = new ReusableResult();
+
+ assertTrue(VariantUtils.extractInto(variant, valuePath, ResultType.INT, result));
+ assertEquals(result.getIntValue(), 7);
+ assertFalse(VariantUtils.extractInto(variant, VariantUtils.compilePath("$.missing"), ResultType.INT, result));
+ assertFalse(VariantUtils.extractInto(variant, VariantUtils.compilePath("$.null"), ResultType.INT, result));
+ assertThrows(IllegalArgumentException.class,
+ () -> VariantUtils.extractInto(variant, VariantUtils.compilePath("$.text"), ResultType.DOUBLE, result));
+ assertFalse(
+ VariantUtils.tryExtractInto(variant, VariantUtils.compilePath("$.text"), ResultType.DOUBLE, result));
+ assertFalse(VariantUtils.tryExtractInto(new byte[]{1}, valuePath, ResultType.INT, result));
+ assertThrows(NullPointerException.class, () -> VariantUtils.extractInto(
+ variant, valuePath, ResultType.INT, null));
+ assertThrows(NullPointerException.class, () -> VariantUtils.tryExtractInto(
+ variant, valuePath, ResultType.INT, null));
+ }
+
+ @Test
+ public void testReusableTolerantHeterogeneousMismatchesAndNumericRange() {
+ byte[][] rows = {
+ VariantUtils.parseJsonToVariant("{\"value\":\"not-an-int\"}"),
+ VariantUtils.parseJsonToVariant("{\"value\":true}"),
+ VariantUtils.parseJsonToVariant("{\"value\":{}}"),
+ VariantUtils.parseJsonToVariant("{\"value\":[]}"),
+ VariantUtils.parseJsonToVariant("{\"value\":2147483648}"),
+ VariantUtils.parseJsonToVariant("{\"value\":-2147483649}"),
+ VariantUtils.parseJsonToVariant("{\"value\":1.5}"),
+ VariantUtils.parseJsonToVariant("{\"value\":9223372036854775808}"),
+ VariantUtils.parseJsonToVariant("{\"value\":-9223372036854775809}")
+ };
+ ResultType[] resultTypes = {
+ ResultType.INT,
+ ResultType.INT,
+ ResultType.INT,
+ ResultType.INT,
+ ResultType.INT,
+ ResultType.INT,
+ ResultType.INT,
+ ResultType.LONG,
+ ResultType.LONG
+ };
+ VariantPath path = VariantUtils.compilePath("$.value");
+ ReusableResult result = new ReusableResult();
+
+ for (int i = 0; i < rows.length; i++) {
+ assertFalse(VariantUtils.tryExtractInto(rows[i], path, resultTypes[i], result),
+ "Expected tolerant conversion to reject row " + i);
+ }
+
+ assertThrows(IllegalArgumentException.class,
+ () -> VariantUtils.extractInto(rows[0], path, ResultType.INT, result));
+ assertThrows(ArithmeticException.class,
+ () -> VariantUtils.extractInto(rows[4], path, ResultType.INT, result));
+ assertThrows(ArithmeticException.class,
+ () -> VariantUtils.extractInto(rows[7], path, ResultType.LONG, result));
+
+ byte[] valid = VariantUtils.parseJsonToVariant("{\"value\":17}");
+ assertTrue(VariantUtils.tryExtractInto(valid, path, ResultType.INT, result));
+ assertEquals(result.getIntValue(), 17);
+ }
+
+ @Test
+ public void testReusableResultParityForEveryResultType() {
+ VariantBuilder builder = new VariantBuilder();
+ builder.appendBoolean(true);
+ assertReusableParity(encode(builder), ResultType.BOOLEAN);
+
+ builder = new VariantBuilder();
+ builder.appendInt(-17);
+ assertReusableParity(encode(builder), ResultType.INT);
+
+ builder = new VariantBuilder();
+ builder.appendLong(9_876_543_210L);
+ assertReusableParity(encode(builder), ResultType.LONG);
+
+ builder = new VariantBuilder();
+ builder.appendFloat(1.25F);
+ assertReusableParity(encode(builder), ResultType.FLOAT);
+
+ builder = new VariantBuilder();
+ builder.appendDouble(-123.5D);
+ assertReusableParity(encode(builder), ResultType.DOUBLE);
+
+ builder = new VariantBuilder();
+ builder.appendDecimal(new BigDecimal("12345678901234567890.1234"));
+ assertReusableParity(encode(builder), ResultType.BIG_DECIMAL);
+
+ builder = new VariantBuilder();
+ builder.appendString("a UTF-8 value \uD83D\uDE00");
+ assertReusableParity(encode(builder), ResultType.STRING);
+
+ builder = new VariantBuilder();
+ builder.appendBinary(ByteBuffer.wrap(new byte[]{0, 1, -1, 42}));
+ assertReusableParity(encode(builder), ResultType.BYTES);
+
+ builder = new VariantBuilder();
+ builder.appendUUID(UUID.fromString("00112233-4455-6677-8899-aabbccddeeff"));
+ assertReusableParity(encode(builder), ResultType.UUID);
+
+ builder = new VariantBuilder();
+ builder.appendTimestampNanosTz(-1_234_567_890L);
+ assertReusableParity(encode(builder), ResultType.TIMESTAMP);
+
+ byte[] nested = VariantUtils.parseJsonToVariant("{\"payload\":{\"count\":7}}");
+ assertReusableParity(nested, VariantUtils.compilePath("$.payload"), ResultType.VARIANT);
+ assertReusableParity(nested, VariantUtils.compilePath("$.payload"), ResultType.JSON);
+ }
+
+ @Test
+ public void testReusableNumericAndTemporalEncodingParity() {
+ VariantBuilder builder = new VariantBuilder();
+ builder.appendByte((byte) -8);
+ byte[] byteValue = encode(builder);
+ assertReusableParity(byteValue, ResultType.INT);
+ assertReusableParity(byteValue, ResultType.LONG);
+ assertReusableParity(byteValue, ResultType.FLOAT);
+ assertReusableParity(byteValue, ResultType.DOUBLE);
+ assertReusableParity(byteValue, ResultType.BIG_DECIMAL);
+
+ builder = new VariantBuilder();
+ builder.appendShort((short) 32_000);
+ assertReusableParity(encode(builder), ResultType.INT);
+
+ builder = new VariantBuilder();
+ builder.appendDecimal(new BigDecimal("123.45"));
+ assertReusableParity(encode(builder), ResultType.BIG_DECIMAL);
+
+ builder = new VariantBuilder();
+ builder.appendDecimal(new BigDecimal("123.00"));
+ byte[] integralDecimal = encode(builder);
+ assertReusableParity(integralDecimal, ResultType.INT);
+ assertReusableParity(integralDecimal, ResultType.LONG);
+
+ builder = new VariantBuilder();
+ builder.appendDecimal(new BigDecimal("1234567890123.45"));
+ assertReusableParity(encode(builder), ResultType.BIG_DECIMAL);
+
+ builder = new VariantBuilder();
+ builder.appendString("x".repeat(128));
+ assertReusableParity(encode(builder), ResultType.STRING);
+
+ builder = new VariantBuilder();
+ builder.appendDate(-1);
+ byte[] dateValue = encode(builder);
+ assertReusableParity(dateValue, ResultType.TIMESTAMP);
Review Comment:
Added `testDateToTimestampInt32BoundaryDoesNotOverflow`, covering `variantGet`, `tryVariantGet`, and reusable strict/tolerant extraction at `Integer.MAX_VALUE`. It asserts exact success because the full legal int32 DATE range fits in long milliseconds.
##########
pinot-spi/src/main/java/org/apache/pinot/spi/data/FieldSpec.java:
##########
@@ -909,6 +991,9 @@ public int hashCode(Object value) {
/// return -1 if value1 is less than value2
/// return 1 if value1 is greater than value2
public int compare(Object value1, Object value2) {
+ if (!supportsOrdering()) {
+ throw new UnsupportedOperationException(this + " does not support ordering");
Review Comment:
Narrowed the throwing branches to `this == VARIANT`. STRUCT/MAP/OPEN_STRUCT/LIST/UNKNOWN retain their previous exception behavior, and the PR body explicitly records that non-Variant behavior is preserved.
##########
pinot-spi/src/main/java/org/apache/pinot/spi/data/FieldSpec.java:
##########
@@ -880,6 +952,10 @@ public Object convert(String value) {
return value;
case BYTES:
return BytesUtils.toBytes(value);
+ case VARIANT:
+ byte[] envelope = BytesUtils.toBytes(value);
+ VariantEnvelope.decode(envelope);
Review Comment:
Special-cased the zero-length Variant SQL-null sentinel in both conversion paths. An explicit empty hex/default now round-trips, while non-empty values are still envelope-validated; schema and FieldSpec regressions cover it.
##########
pinot-spi/src/main/java/org/apache/pinot/spi/data/FieldSpec.java:
##########
@@ -684,6 +691,9 @@ protected void appendDefaultNullValue(ObjectNode jsonNode) {
case BYTES:
jsonNode.put(key, BytesUtils.toHexString((byte[]) _defaultNullValue));
break;
+ case VARIANT:
Review Comment:
Collapsed BYTES and VARIANT into the shared serialization switch arm. The conversion arms remain separate because only VARIANT validates non-empty envelopes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add end-to-end Parquet VARIANT support [pinot]
Posted by "raghavyadav01 (via GitHub)" <gi...@apache.org>.
raghavyadav01 commented on code in PR #19101:
URL: https://github.com/apache/pinot/pull/19101#discussion_r4328340409
##########
pinot-plugins/pinot-input-format/pinot-parquet/src/main/java/org/apache/pinot/plugin/inputformat/parquet/ParquetVariantConverter.java:
##########
@@ -0,0 +1,324 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.inputformat.parquet;
+
+import java.nio.ByteBuffer;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+import org.apache.parquet.example.data.Group;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.Converter;
+import org.apache.parquet.io.api.GroupConverter;
+import org.apache.parquet.io.api.PrimitiveConverter;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+import org.apache.parquet.variant.ImmutableMetadata;
+import org.apache.parquet.variant.Variant;
+import org.apache.parquet.variant.VariantBuilder;
+import org.apache.parquet.variant.VariantConverters;
+import org.apache.pinot.spi.utils.VariantEnvelope;
+
+
+/// Converts a top-level Parquet `VARIANT(1)` group into Pinot's self-describing VARIANT byte envelope.
+///
+/// Detection is based only on [LogicalTypeAnnotation.VariantLogicalTypeAnnotation], never on field names.
+/// The parquet-java [VariantConverters] implementation performs the logical reconstruction so unshredded,
+/// shredded, and partially shredded encodings all produce complete Variant metadata and value buffers.
+///
+/// Instances retain parquet-java's converter tree and are intentionally scoped to one record reader because the
+/// converter and builder holder are mutable and not thread-safe.
+final class ParquetVariantConverter {
+ private static final byte SUPPORTED_SPEC_VERSION = 1;
+
+ private final GroupType _variantType;
+ private final int _metadataIndex;
+ private final int _valueIndex;
+ private final int _typedValueIndex;
+ private final BuilderHolder _holder = new BuilderHolder();
+ private final GroupConverter _converter;
+
+ private ParquetVariantConverter(GroupType variantType) {
+ _variantType = variantType;
+ _metadataIndex = variantType.getFieldIndex("metadata");
+ _valueIndex = variantType.containsField("value") ? variantType.getFieldIndex("value") : -1;
+ _typedValueIndex = variantType.containsField("typed_value") ? variantType.getFieldIndex("typed_value") : -1;
+ _converter = VariantConverters.newVariantConverter(variantType, _holder::setMetadata, _holder::build);
+ }
+
+ /// Validates every VARIANT annotation in the file and returns the supported top-level field names.
+ ///
+ /// Pinot currently supports only non-repeated, top-level `VARIANT(1)` values. Failing during reader
+ /// initialization avoids silently surfacing an unsupported nested/repeated Variant as an ordinary struct.
+ static Set<String> validateAndGetTopLevelVariantFields(GroupType schema) {
+ return validateAndGetTopLevelVariantFields(schema, fieldName -> true);
+ }
+
+ private static Set<String> validateAndGetTopLevelVariantFields(GroupType schema, Predicate<String> fieldSelector) {
+ Set<String> variantFields = new HashSet<>();
+ for (Type field : schema.getFields()) {
+ if (!fieldSelector.test(field.getName())) {
+ continue;
+ }
+ if (isVariant(field)) {
+ validateTopLevelVariant(field);
+ variantFields.add(field.getName());
+ } else if (!field.isPrimitive()) {
+ rejectNestedVariants(field.asGroupType(), field.getName());
+ }
+ }
+ return Set.copyOf(variantFields);
+ }
+
+ /// Validates the file schema and builds an index-aligned reusable converter tree for each top-level VARIANT column.
+ static ParquetVariantConverter[] createTopLevelVariantConverters(GroupType schema) {
+ Set<String> variantFields = validateAndGetTopLevelVariantFields(schema);
+ return buildTopLevelVariantConverters(schema, variantFields);
+ }
+
+ /// Validates selected top-level fields and builds index-aligned converters for selected VARIANT columns.
+ static ParquetVariantConverter[] createTopLevelVariantConverters(GroupType schema, Predicate<String> fieldSelector) {
+ Set<String> variantFields = validateAndGetTopLevelVariantFields(schema, fieldSelector);
+ return buildTopLevelVariantConverters(schema, variantFields);
+ }
+
+ private static ParquetVariantConverter[] buildTopLevelVariantConverters(GroupType schema,
+ Set<String> variantFields) {
+ ParquetVariantConverter[] variantConverters = new ParquetVariantConverter[schema.getFieldCount()];
+ for (int fieldIndex = 0; fieldIndex < schema.getFieldCount(); fieldIndex++) {
+ Type field = schema.getType(fieldIndex);
+ if (variantFields.contains(field.getName())) {
+ variantConverters[fieldIndex] = new ParquetVariantConverter(field.asGroupType());
+ }
+ }
+ return variantConverters;
+ }
+
+ static boolean isVariant(Type type) {
+ return type.getLogicalTypeAnnotation() instanceof LogicalTypeAnnotation.VariantLogicalTypeAnnotation;
+ }
+
+ byte[] convert(Group group) {
+ if (_valueIndex >= 0 && group.getFieldRepetitionCount(_valueIndex) == 1
+ && (_typedValueIndex < 0 || group.getFieldRepetitionCount(_typedValueIndex) == 0)) {
+ return pack(group.getBinary(_metadataIndex, 0), group.getBinary(_valueIndex, 0));
+ }
+ _holder.reset();
+ feedVariantGroup(group, _converter, _variantType, _metadataIndex);
+ Variant variant = _holder.finish();
+ return VariantEnvelope.encode(variant.getMetadataBuffer(), variant.getValueBuffer());
+ }
+
+ static byte[] pack(ByteBuffer metadata, ByteBuffer value) {
+ return VariantEnvelope.encode(metadata, value);
+ }
+
+ private byte[] pack(Binary metadata, Binary value) {
+ // Binary.writeTo(OutputStream) materializes the full payload for a direct or read-only ByteBuffer-backed Binary.
+ // Consume the independent views returned by toByteBuffer() so both payloads go straight into the final envelope,
+ // without intermediate byte arrays or another duplicate view.
+ int metadataLength = metadata.length();
+ int valueLength = value.length();
+ byte[] envelope = VariantEnvelope.allocateUnfilled(metadataLength, valueLength);
+ copyInto(metadata, envelope, VariantEnvelope.HEADER_SIZE, metadataLength, "metadata");
+ copyInto(value, envelope, VariantEnvelope.HEADER_SIZE + metadataLength, valueLength, "value");
+ return envelope;
+ }
+
+ private static void copyInto(Binary source, byte[] target, int targetOffset, int expectedLength, String payload) {
+ ByteBuffer sourceView = source.toByteBuffer();
+ if (sourceView.remaining() != expectedLength) {
+ throw new IllegalStateException(
+ "Parquet VARIANT " + payload + " length changed while copying: expected " + expectedLength + " but found "
+ + sourceView.remaining());
+ }
+ if (sourceView.hasArray()) {
+ System.arraycopy(sourceView.array(), sourceView.arrayOffset() + sourceView.position(), target, targetOffset,
+ expectedLength);
+ } else {
+ sourceView.get(target, targetOffset, expectedLength);
+ }
+ }
+
+ private static void validateTopLevelVariant(Type field) {
+ if (!isVariant(field) || field.isPrimitive()) {
+ throw new IllegalArgumentException("Parquet VARIANT must be a group: " + field);
+ }
+ LogicalTypeAnnotation.VariantLogicalTypeAnnotation annotation =
+ (LogicalTypeAnnotation.VariantLogicalTypeAnnotation) field.getLogicalTypeAnnotation();
+ if (annotation.getSpecVersion() != SUPPORTED_SPEC_VERSION) {
+ throw new UnsupportedOperationException(
+ "Unsupported Parquet VARIANT spec version: " + annotation.getSpecVersion());
+ }
+ if (field.isRepetition(Type.Repetition.REPEATED)) {
+ throw new UnsupportedOperationException("Repeated Parquet VARIANT is not supported: " + field.getName());
+ }
+
+ GroupType group = field.asGroupType();
+ for (Type child : group.getFields()) {
+ String name = child.getName();
+ if (!"metadata".equals(name) && !"value".equals(name) && !"typed_value".equals(name)) {
+ throw new IllegalArgumentException("Invalid Parquet VARIANT field: " + child);
Review Comment:
What happens on a spec-illegal file, e.g. a key present in both `typed_value` and the residual `value`? Does it fail the row, fail the file, or pick one silently?
##########
docs/design/VARIANT.md:
##########
@@ -0,0 +1,355 @@
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+# Apache Pinot VARIANT — Design Document
+
+Status: draft for community and component-owner review.
+
+## 1. Motivation and goals
+
+Apache Parquet `VARIANT(1)` provides a portable binary representation for
+semi-structured values. Pinot should ingest that representation without converting
+every row through a JSON tree, retain the complete value for late-bound queries, and
+let users materialize frequently queried paths into ordinary Pinot columns.
+
+This design has five goals:
+
+1. Define a first-class Pinot `VARIANT` logical type with an explicit persisted and
+ query-wire contract.
+2. Ingest top-level, non-repeated Parquet `VARIANT(1)` values, including unshredded,
+ fully shredded, and partially shredded layouts.
+3. Provide Spark-style parse, extraction, existence, type, null, and JSON-rendering
+ functions in both Pinot query engines.
+4. Fail explicitly when a raw Variant value reaches an operation whose semantics are
+ undefined, while allowing users to extract a typed scalar for that operation.
+5. Ship an executable table-definition, ingestion, and query story with integration
+ coverage.
+
+### 1.1 Non-goals
+
+The initial contract does not include:
+
+- nested or repeated Parquet Variant columns;
+- Pinot multi-value `VARIANT` columns;
+- streaming-format Variant ingestion;
+- quoted object keys in Variant paths;
+- semantic equality, hashing, or ordering of raw Variant values;
+- mixed-version execution for queries that contain a Variant type; or
+- automatic rollback after a Variant table has been activated.
+
+## 2. User model
+
+A Variant column is declared as a single-value dimension:
+
+```json
+{
+ "name": "payload",
+ "dataType": "VARIANT"
+}
+```
+
+The table stores `payload` in a raw forward index without a dictionary. Storage null
+handling must be enabled. A common layout retains the source value while materializing
+an indexed scalar during ingestion:
+
+```json
+{
+ "columnName": "eventType",
+ "transformFunction": "variant_get(payload, '$.eventType', 'STRING')"
+}
+```
+
+Late-bound paths remain queryable:
+
+```sql
+SET enableNullHandling=true;
+
+SELECT
+ eventId,
+ variant_get(payload, '$.user.id', 'STRING') AS userId,
+ variant_get(payload, '$.amount', 'DOUBLE') AS amount
+FROM variantEvents
+WHERE eventType = 'checkout';
+```
+
+## 3. Data flow and ownership
+
+```text
+Parquet VARIANT(1)
+ |
+ | pinot-parquet: validate shape and reconstruct shredded values
+ v
+metadata ByteBuffer + value ByteBuffer
+ |
+ | VariantEnvelope: add stable Pinot PVAR framing
+ v
+single-value raw BYTES forward index in a Pinot segment
+ |
+ | DataSchema/protobuf: carry logical VARIANT, not generic BYTES
+ v
+single-stage or multi-stage Variant functions
+ |
+ +--> typed scalar for filtering/grouping/joining/aggregation
+ |
+ +--> canonical JSON text for response clients
+```
+
+Ownership is deliberately split by module:
+
+- `pinot-spi` owns the public logical type, capability contract, and stable `PVAR`
+ envelope.
+- `pinot-segment-local` owns schema, table, index, null, and segment-metadata
+ validation.
+- `pinot-parquet` owns Parquet logical-type validation and reconstruction.
+- `pinot-common`, `pinot-core`, `pinot-query-planner`, and `pinot-query-runtime` own
+ function semantics and defensive query guards.
+- client, response, and DDL modules carry the logical type without redefining its
+ semantics.
+
+`pinot-common` depends on `parquet-variant` and its required `parquet-common` and
+`parquet-column` codec closure; the unused `parquet-jackson` artifact is excluded.
+Hadoop and record-reader integrations remain isolated to the input-format plugin.
+
+## 4. Persisted `PVAR` envelope
+
+Pinot segments need one byte sequence that contains both buffers in a Parquet Variant
+value. `VariantEnvelope` owns that framing. Version 1 is:
+
+```text
+offset size field
+0 4 ASCII magic "PVAR"
+4 1 envelope version (1)
+5 1 flags (0)
+6 2 reserved (0)
+8 4 metadata length, big-endian signed int restricted to >= 0
+12 4 value length, big-endian signed int restricted to >= 0
+16 M Parquet Variant metadata bytes
+16 + M V Parquet Variant value bytes
+```
+
+The complete envelope must fit in a Java byte array. Decoders reject unknown versions,
+non-zero flags or reserved bytes, negative lengths, bad magic, and length mismatches.
+Metadata and value bytes are copied without interpretation by the envelope layer; the
+Parquet Variant codec validates their internal representation.
+
+Version 1 is immutable and protected by a golden-byte test. A future incompatible
+framing change must use a new envelope version and retain an explicit reader for every
+supported old version. Unknown versions fail closed; they must never fall back to
+generic `BYTES`.
+
+`decode(byte[])` returns zero-copy read-only views that alias the input array. The input
+must therefore remain immutable for the lifetime of those views. Array, direct, and
+read-only source buffers are supported without changing their positions. Parquet
+ingestion copies payload bytes directly into the final envelope and must not allocate a
+full intermediate payload array.
+
+## 5. Logical type and segment contract
+
+`FieldSpec.DataType.VARIANT` is logically distinct from `BYTES` but uses variable-width
+bytes as its physical stored type.
+
+The initial schema and table constraints are:
+
+- dimension field only;
+- single-value only;
+- the reserved empty-byte default null value only;
+- no truncating or default-substitution max-length policy;
+- storage null handling enabled at schema or table level;
+- enabled raw forward index;
+- no dictionary or secondary index on the raw Variant column;
+- not a primary key, partition column, sorted column, upsert comparison column,
+ metrics-aggregation key, or star-tree dimension;
+- partial upsert may use only the `OVERWRITE` strategy; and
+- minion merge tasks (`MergeRollupTask`, `RealtimeToOfflineSegmentsTask`) may use only the `CONCAT` merge type,
+ because rollup and dedup group rows by raw stored bytes.
+
+Segment creation validates every non-null value as a complete `PVAR` envelope. Raw
+Variant columns do not publish logical minimum or maximum values and are never marked
+sorted, including single-row segments.
+
+### 5.1 Capability policy
+
+The Parquet representation is not a canonical semantic encoding: equivalent logical
+objects can use different metadata dictionaries or shredded layouts. Byte equality,
+byte hashing, and lexicographic byte order would therefore expose storage-layout
+accidents as SQL semantics.
+
+The logical type reports:
+
+| Capability on raw value | Supported |
+|---|---|
+| equality / `IN` / pattern predicates | no |
+| hashing / `GROUP BY` / `DISTINCT` | no |
+| ordering / min-max / sort / ASOF match key | no |
+| join, lookup, primary, or partition key | no |
+| direct value-consuming aggregation | no |
+| `CAST` from a raw value | no |
+| non-distinct `COUNT(raw_variant)` | yes |
+
+Planner validation provides the primary error. Runtime and single-stage guards remain
+in place for mixed plans or future planner changes. Error messages direct users to
+`variant_get` a typed path first.
+
+## 6. Parquet ingestion
+
+Reader initialization scans the file schema before publishing reader state.
+
+- A supported field is a top-level, non-repeated group annotated `VARIANT(1)`.
+- `metadata` is required binary; at least one of `value` or `typed_value` is present.
+- The parquet-java `VariantConverters` tree reconstructs fully or partially shredded
+ values.
+- The unshredded path retains the exact metadata and value buffers and adds only the
+ `PVAR` envelope.
+- Converter plans and field indexes are immutable and created once per reader.
+- A failed reinitialization leaves the previous reader usable. A failure while closing
+ the old reader does not invalidate the successfully initialized replacement.
+
+Automatic reader selection preserves existing Avro-metadata precedence for backward
+compatibility. A Parquet file with Avro metadata must explicitly select the native
+reader to ingest Variant.
+
+Malformed rows follow the existing record-reader skip policy and do not corrupt reader
+progress. Unsupported Variant spec versions and unsupported nested/repeated shapes fail
+during initialization rather than appearing as ordinary structs.
+
+## 7. Query functions
+
+Both query engines expose:
+
+- `variant_get(value, path[, targetType])`
+- `try_variant_get(value, path[, targetType])`
+- `variant_exists(value, path)`
+- `is_variant_null(value[, path])`
+- `variant_typeof(value[, path])`
+- `variant_to_json(value)`
+- `parse_json(text)` / `parse_json_to_variant(text)`
+- `try_parse_json(text)` / `try_parse_json_to_variant(text)`
+
+Function lookup remains case-insensitive under Pinot's existing registration rules.
+The v1 path grammar supports root `$`, dot-separated object fields, and non-negative
+array indexes. Supported extraction targets are `BOOLEAN`, `INT`, `LONG`, `FLOAT`,
+`DOUBLE`, `BIG_DECIMAL`, `STRING`, `BYTES`, `UUID`, `TIMESTAMP`, `VARIANT`, and `JSON`.
Review Comment:
How do Variant primitives map to Pinot types in `variant_get` and materialization? Mainly decimals (BIG_DECIMAL vs DOUBLE), timestamp and timestamp_ntz (micros into Pinot's millis loses precision), time, uuid and binary.
##########
docs/design/VARIANT.md:
##########
@@ -0,0 +1,355 @@
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+# Apache Pinot VARIANT — Design Document
+
+Status: draft for community and component-owner review.
+
+## 1. Motivation and goals
+
+Apache Parquet `VARIANT(1)` provides a portable binary representation for
+semi-structured values. Pinot should ingest that representation without converting
+every row through a JSON tree, retain the complete value for late-bound queries, and
+let users materialize frequently queried paths into ordinary Pinot columns.
+
+This design has five goals:
+
+1. Define a first-class Pinot `VARIANT` logical type with an explicit persisted and
+ query-wire contract.
+2. Ingest top-level, non-repeated Parquet `VARIANT(1)` values, including unshredded,
+ fully shredded, and partially shredded layouts.
+3. Provide Spark-style parse, extraction, existence, type, null, and JSON-rendering
+ functions in both Pinot query engines.
+4. Fail explicitly when a raw Variant value reaches an operation whose semantics are
+ undefined, while allowing users to extract a typed scalar for that operation.
+5. Ship an executable table-definition, ingestion, and query story with integration
+ coverage.
+
+### 1.1 Non-goals
+
+The initial contract does not include:
+
+- nested or repeated Parquet Variant columns;
+- Pinot multi-value `VARIANT` columns;
+- streaming-format Variant ingestion;
+- quoted object keys in Variant paths;
+- semantic equality, hashing, or ordering of raw Variant values;
+- mixed-version execution for queries that contain a Variant type; or
+- automatic rollback after a Variant table has been activated.
+
+## 2. User model
Review Comment:
External Tables already read Parquet VARIANT as OPEN_STRUCT, where each shredded key becomes its own typed, indexable child. This adds a second mapping as an opaque VARIANT value. Which should a Parquet VARIANT column map to by default, and can a table switch between them later?
##########
pinot-plugins/pinot-input-format/pinot-parquet/src/main/java/org/apache/pinot/plugin/inputformat/parquet/ParquetVariantConverter.java:
##########
@@ -0,0 +1,324 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.inputformat.parquet;
+
+import java.nio.ByteBuffer;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+import org.apache.parquet.example.data.Group;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.Converter;
+import org.apache.parquet.io.api.GroupConverter;
+import org.apache.parquet.io.api.PrimitiveConverter;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+import org.apache.parquet.variant.ImmutableMetadata;
+import org.apache.parquet.variant.Variant;
+import org.apache.parquet.variant.VariantBuilder;
+import org.apache.parquet.variant.VariantConverters;
+import org.apache.pinot.spi.utils.VariantEnvelope;
+
+
+/// Converts a top-level Parquet `VARIANT(1)` group into Pinot's self-describing VARIANT byte envelope.
+///
+/// Detection is based only on [LogicalTypeAnnotation.VariantLogicalTypeAnnotation], never on field names.
+/// The parquet-java [VariantConverters] implementation performs the logical reconstruction so unshredded,
Review Comment:
Going through `VariantConverters` is a good call. Do the tests cover the shapes that bit us on External Tables: element fields shredded further, arrays inside arrays, element groups with `value` but no `typed_value`, and residual `value` at intermediate levels? Our random generator passed 4000 shapes and still missed 8 real bugs because it never produced those.
##########
docs/design/VARIANT.md:
##########
@@ -0,0 +1,355 @@
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+# Apache Pinot VARIANT — Design Document
+
+Status: draft for community and component-owner review.
+
+## 1. Motivation and goals
+
+Apache Parquet `VARIANT(1)` provides a portable binary representation for
+semi-structured values. Pinot should ingest that representation without converting
+every row through a JSON tree, retain the complete value for late-bound queries, and
+let users materialize frequently queried paths into ordinary Pinot columns.
+
+This design has five goals:
+
+1. Define a first-class Pinot `VARIANT` logical type with an explicit persisted and
+ query-wire contract.
+2. Ingest top-level, non-repeated Parquet `VARIANT(1)` values, including unshredded,
+ fully shredded, and partially shredded layouts.
+3. Provide Spark-style parse, extraction, existence, type, null, and JSON-rendering
+ functions in both Pinot query engines.
+4. Fail explicitly when a raw Variant value reaches an operation whose semantics are
+ undefined, while allowing users to extract a typed scalar for that operation.
+5. Ship an executable table-definition, ingestion, and query story with integration
+ coverage.
+
+### 1.1 Non-goals
+
+The initial contract does not include:
+
+- nested or repeated Parquet Variant columns;
+- Pinot multi-value `VARIANT` columns;
+- streaming-format Variant ingestion;
+- quoted object keys in Variant paths;
+- semantic equality, hashing, or ordering of raw Variant values;
+- mixed-version execution for queries that contain a Variant type; or
+- automatic rollback after a Variant table has been activated.
+
+## 2. User model
+
+A Variant column is declared as a single-value dimension:
+
+```json
+{
+ "name": "payload",
+ "dataType": "VARIANT"
+}
+```
+
+The table stores `payload` in a raw forward index without a dictionary. Storage null
+handling must be enabled. A common layout retains the source value while materializing
+an indexed scalar during ingestion:
+
+```json
+{
+ "columnName": "eventType",
+ "transformFunction": "variant_get(payload, '$.eventType', 'STRING')"
+}
+```
+
+Late-bound paths remain queryable:
+
+```sql
+SET enableNullHandling=true;
+
+SELECT
+ eventId,
+ variant_get(payload, '$.user.id', 'STRING') AS userId,
+ variant_get(payload, '$.amount', 'DOUBLE') AS amount
+FROM variantEvents
+WHERE eventType = 'checkout';
+```
+
+## 3. Data flow and ownership
+
+```text
+Parquet VARIANT(1)
+ |
+ | pinot-parquet: validate shape and reconstruct shredded values
+ v
+metadata ByteBuffer + value ByteBuffer
+ |
+ | VariantEnvelope: add stable Pinot PVAR framing
+ v
+single-value raw BYTES forward index in a Pinot segment
+ |
+ | DataSchema/protobuf: carry logical VARIANT, not generic BYTES
+ v
+single-stage or multi-stage Variant functions
+ |
+ +--> typed scalar for filtering/grouping/joining/aggregation
+ |
+ +--> canonical JSON text for response clients
+```
+
+Ownership is deliberately split by module:
+
+- `pinot-spi` owns the public logical type, capability contract, and stable `PVAR`
+ envelope.
+- `pinot-segment-local` owns schema, table, index, null, and segment-metadata
+ validation.
+- `pinot-parquet` owns Parquet logical-type validation and reconstruction.
+- `pinot-common`, `pinot-core`, `pinot-query-planner`, and `pinot-query-runtime` own
+ function semantics and defensive query guards.
+- client, response, and DDL modules carry the logical type without redefining its
+ semantics.
+
+`pinot-common` depends on `parquet-variant` and its required `parquet-common` and
+`parquet-column` codec closure; the unused `parquet-jackson` artifact is excluded.
+Hadoop and record-reader integrations remain isolated to the input-format plugin.
+
+## 4. Persisted `PVAR` envelope
+
+Pinot segments need one byte sequence that contains both buffers in a Parquet Variant
+value. `VariantEnvelope` owns that framing. Version 1 is:
+
+```text
+offset size field
+0 4 ASCII magic "PVAR"
+4 1 envelope version (1)
+5 1 flags (0)
+6 2 reserved (0)
+8 4 metadata length, big-endian signed int restricted to >= 0
+12 4 value length, big-endian signed int restricted to >= 0
+16 M Parquet Variant metadata bytes
+16 + M V Parquet Variant value bytes
+```
+
+The complete envelope must fit in a Java byte array. Decoders reject unknown versions,
+non-zero flags or reserved bytes, negative lengths, bad magic, and length mismatches.
+Metadata and value bytes are copied without interpretation by the envelope layer; the
+Parquet Variant codec validates their internal representation.
+
+Version 1 is immutable and protected by a golden-byte test. A future incompatible
+framing change must use a new envelope version and retain an explicit reader for every
+supported old version. Unknown versions fail closed; they must never fall back to
+generic `BYTES`.
+
+`decode(byte[])` returns zero-copy read-only views that alias the input array. The input
+must therefore remain immutable for the lifetime of those views. Array, direct, and
+read-only source buffers are supported without changing their positions. Parquet
+ingestion copies payload bytes directly into the final envelope and must not allocate a
+full intermediate payload array.
+
+## 5. Logical type and segment contract
+
+`FieldSpec.DataType.VARIANT` is logically distinct from `BYTES` but uses variable-width
+bytes as its physical stored type.
+
+The initial schema and table constraints are:
+
+- dimension field only;
+- single-value only;
+- the reserved empty-byte default null value only;
+- no truncating or default-substitution max-length policy;
+- storage null handling enabled at schema or table level;
+- enabled raw forward index;
+- no dictionary or secondary index on the raw Variant column;
+- not a primary key, partition column, sorted column, upsert comparison column,
+ metrics-aggregation key, or star-tree dimension;
+- partial upsert may use only the `OVERWRITE` strategy; and
+- minion merge tasks (`MergeRollupTask`, `RealtimeToOfflineSegmentsTask`) may use only the `CONCAT` merge type,
+ because rollup and dedup group rows by raw stored bytes.
+
+Segment creation validates every non-null value as a complete `PVAR` envelope. Raw
+Variant columns do not publish logical minimum or maximum values and are never marked
+sorted, including single-row segments.
+
+### 5.1 Capability policy
+
+The Parquet representation is not a canonical semantic encoding: equivalent logical
+objects can use different metadata dictionaries or shredded layouts. Byte equality,
+byte hashing, and lexicographic byte order would therefore expose storage-layout
+accidents as SQL semantics.
+
+The logical type reports:
+
+| Capability on raw value | Supported |
+|---|---|
+| equality / `IN` / pattern predicates | no |
+| hashing / `GROUP BY` / `DISTINCT` | no |
+| ordering / min-max / sort / ASOF match key | no |
+| join, lookup, primary, or partition key | no |
+| direct value-consuming aggregation | no |
+| `CAST` from a raw value | no |
+| non-distinct `COUNT(raw_variant)` | yes |
+
+Planner validation provides the primary error. Runtime and single-stage guards remain
+in place for mixed plans or future planner changes. Error messages direct users to
+`variant_get` a typed path first.
+
+## 6. Parquet ingestion
+
+Reader initialization scans the file schema before publishing reader state.
+
+- A supported field is a top-level, non-repeated group annotated `VARIANT(1)`.
+- `metadata` is required binary; at least one of `value` or `typed_value` is present.
+- The parquet-java `VariantConverters` tree reconstructs fully or partially shredded
+ values.
+- The unshredded path retains the exact metadata and value buffers and adds only the
+ `PVAR` envelope.
+- Converter plans and field indexes are immutable and created once per reader.
+- A failed reinitialization leaves the previous reader usable. A failure while closing
+ the old reader does not invalidate the successfully initialized replacement.
+
+Automatic reader selection preserves existing Avro-metadata precedence for backward
+compatibility. A Parquet file with Avro metadata must explicitly select the native
+reader to ingest Variant.
+
+Malformed rows follow the existing record-reader skip policy and do not corrupt reader
+progress. Unsupported Variant spec versions and unsupported nested/repeated shapes fail
+during initialization rather than appearing as ordinary structs.
+
+## 7. Query functions
+
+Both query engines expose:
+
+- `variant_get(value, path[, targetType])`
+- `try_variant_get(value, path[, targetType])`
+- `variant_exists(value, path)`
+- `is_variant_null(value[, path])`
+- `variant_typeof(value[, path])`
+- `variant_to_json(value)`
+- `parse_json(text)` / `parse_json_to_variant(text)`
+- `try_parse_json(text)` / `try_parse_json_to_variant(text)`
+
+Function lookup remains case-insensitive under Pinot's existing registration rules.
+The v1 path grammar supports root `$`, dot-separated object fields, and non-negative
+array indexes. Supported extraction targets are `BOOLEAN`, `INT`, `LONG`, `FLOAT`,
+`DOUBLE`, `BIG_DECIMAL`, `STRING`, `BYTES`, `UUID`, `TIMESTAMP`, `VARIANT`, and `JSON`.
+
+Neither engine constant-folds the JSON-to-Variant functions: a folded result would
+degrade to a plain `BYTES` literal and erase the logical type together with its guards.
+
+Strict functions reject malformed encodings, paths, incompatible conversions, numeric
+overflow, and malformed JSON. `try_` functions return SQL null for those failures.
+Literal paths, target types, and JSON inputs are compiled or parsed once per query.
+Vectorized execution reuses thread-confined cursors and unboxed result holders.
+
+The query cursor directly decodes the Parquet Variant binary layout to avoid per-row
+codec objects and buffer allocation. Its type tags, length rules, metadata version,
+and wide-object threshold must stay synchronized with the pinned `parquet-variant`
+dependency. Tests compare the cursor constants with parquet-java's codec constants,
+exercise every `Variant.Type`, and reject unrecognized metadata versions and primitive
+tags. A parquet-variant upgrade that adds an encoding must update the cursor and these
+tests in the same change; unknown encodings fail closed rather than returning a
+partially decoded or incorrectly typed result.
+
+JSON parsing is bounded to 100 nesting levels. Parquet Variant decimal encoding is
+bounded to 38 digits of precision and a scale of at most 38 after exact normalization.
+Exponent bounds are checked before expansion so hostile inputs cannot request
+pathological intermediate allocations.
+
+### 7.1 Null states
+
+Storage and query null handling are required because four states must remain distinct:
+
+| State | Stored bytes / null vector | `variant_exists` | `is_variant_null` | `variant_typeof` | scalar `variant_get` |
+|---|---|---:|---:|---|---|
+| SQL null | empty bytes, null bit set | SQL null | false | SQL null | SQL null |
+| Variant null | non-empty `PVAR`, null bit clear | true | true | `NULL` | SQL null |
Review Comment:
In parquet-java a Variant null and an unknown primitive type id both decode to null, which we couldn't tell apart. The PR says unknown tags fail explicitly. Does that also hold on ingestion, given reconstruction is parquet-java's?
##########
docs/design/VARIANT.md:
##########
@@ -0,0 +1,355 @@
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+# Apache Pinot VARIANT — Design Document
+
+Status: draft for community and component-owner review.
+
+## 1. Motivation and goals
+
+Apache Parquet `VARIANT(1)` provides a portable binary representation for
+semi-structured values. Pinot should ingest that representation without converting
+every row through a JSON tree, retain the complete value for late-bound queries, and
+let users materialize frequently queried paths into ordinary Pinot columns.
+
+This design has five goals:
+
+1. Define a first-class Pinot `VARIANT` logical type with an explicit persisted and
+ query-wire contract.
+2. Ingest top-level, non-repeated Parquet `VARIANT(1)` values, including unshredded,
+ fully shredded, and partially shredded layouts.
+3. Provide Spark-style parse, extraction, existence, type, null, and JSON-rendering
+ functions in both Pinot query engines.
+4. Fail explicitly when a raw Variant value reaches an operation whose semantics are
+ undefined, while allowing users to extract a typed scalar for that operation.
+5. Ship an executable table-definition, ingestion, and query story with integration
+ coverage.
+
+### 1.1 Non-goals
+
+The initial contract does not include:
+
+- nested or repeated Parquet Variant columns;
+- Pinot multi-value `VARIANT` columns;
+- streaming-format Variant ingestion;
+- quoted object keys in Variant paths;
+- semantic equality, hashing, or ordering of raw Variant values;
+- mixed-version execution for queries that contain a Variant type; or
+- automatic rollback after a Variant table has been activated.
+
+## 2. User model
+
+A Variant column is declared as a single-value dimension:
+
+```json
+{
+ "name": "payload",
+ "dataType": "VARIANT"
+}
+```
+
+The table stores `payload` in a raw forward index without a dictionary. Storage null
+handling must be enabled. A common layout retains the source value while materializing
+an indexed scalar during ingestion:
+
+```json
+{
+ "columnName": "eventType",
+ "transformFunction": "variant_get(payload, '$.eventType', 'STRING')"
+}
+```
+
+Late-bound paths remain queryable:
+
+```sql
+SET enableNullHandling=true;
+
+SELECT
+ eventId,
+ variant_get(payload, '$.user.id', 'STRING') AS userId,
+ variant_get(payload, '$.amount', 'DOUBLE') AS amount
+FROM variantEvents
+WHERE eventType = 'checkout';
+```
+
+## 3. Data flow and ownership
+
+```text
+Parquet VARIANT(1)
+ |
+ | pinot-parquet: validate shape and reconstruct shredded values
+ v
+metadata ByteBuffer + value ByteBuffer
+ |
+ | VariantEnvelope: add stable Pinot PVAR framing
+ v
+single-value raw BYTES forward index in a Pinot segment
+ |
+ | DataSchema/protobuf: carry logical VARIANT, not generic BYTES
+ v
+single-stage or multi-stage Variant functions
+ |
+ +--> typed scalar for filtering/grouping/joining/aggregation
+ |
+ +--> canonical JSON text for response clients
+```
+
+Ownership is deliberately split by module:
+
+- `pinot-spi` owns the public logical type, capability contract, and stable `PVAR`
+ envelope.
+- `pinot-segment-local` owns schema, table, index, null, and segment-metadata
+ validation.
+- `pinot-parquet` owns Parquet logical-type validation and reconstruction.
+- `pinot-common`, `pinot-core`, `pinot-query-planner`, and `pinot-query-runtime` own
+ function semantics and defensive query guards.
+- client, response, and DDL modules carry the logical type without redefining its
+ semantics.
+
+`pinot-common` depends on `parquet-variant` and its required `parquet-common` and
+`parquet-column` codec closure; the unused `parquet-jackson` artifact is excluded.
+Hadoop and record-reader integrations remain isolated to the input-format plugin.
+
+## 4. Persisted `PVAR` envelope
+
+Pinot segments need one byte sequence that contains both buffers in a Parquet Variant
+value. `VariantEnvelope` owns that framing. Version 1 is:
+
+```text
+offset size field
+0 4 ASCII magic "PVAR"
+4 1 envelope version (1)
+5 1 flags (0)
+6 2 reserved (0)
+8 4 metadata length, big-endian signed int restricted to >= 0
+12 4 value length, big-endian signed int restricted to >= 0
+16 M Parquet Variant metadata bytes
+16 + M V Parquet Variant value bytes
+```
+
+The complete envelope must fit in a Java byte array. Decoders reject unknown versions,
+non-zero flags or reserved bytes, negative lengths, bad magic, and length mismatches.
+Metadata and value bytes are copied without interpretation by the envelope layer; the
+Parquet Variant codec validates their internal representation.
+
+Version 1 is immutable and protected by a golden-byte test. A future incompatible
+framing change must use a new envelope version and retain an explicit reader for every
+supported old version. Unknown versions fail closed; they must never fall back to
+generic `BYTES`.
+
+`decode(byte[])` returns zero-copy read-only views that alias the input array. The input
+must therefore remain immutable for the lifetime of those views. Array, direct, and
+read-only source buffers are supported without changing their positions. Parquet
+ingestion copies payload bytes directly into the final envelope and must not allocate a
+full intermediate payload array.
+
+## 5. Logical type and segment contract
+
+`FieldSpec.DataType.VARIANT` is logically distinct from `BYTES` but uses variable-width
+bytes as its physical stored type.
+
+The initial schema and table constraints are:
+
+- dimension field only;
+- single-value only;
+- the reserved empty-byte default null value only;
+- no truncating or default-substitution max-length policy;
+- storage null handling enabled at schema or table level;
+- enabled raw forward index;
+- no dictionary or secondary index on the raw Variant column;
Review Comment:
With no path index and the JSON index rejected, every filter on a non-materialized path decodes every row. Is a per-path index planned, or is materialization the long-term answer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add end-to-end Parquet VARIANT support [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #19101:
URL: https://github.com/apache/pinot/pull/19101#discussion_r3742923641
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/operands/FilterOperand.java:
##########
@@ -193,7 +196,10 @@ public Predicate(List<RexExpression> operands, DataSchema dataSchema, IntPredica
ColumnDataType lhsType = _lhs.getResultType();
ColumnDataType rhsType = _rhs.getResultType();
- if (lhsType == rhsType) {
+ Preconditions.checkArgument((lhsType == ColumnDataType.UNKNOWN || lhsType.supportsOrdering())
+ && (rhsType == ColumnDataType.UNKNOWN || rhsType.supportsOrdering()),
+ "Raw VARIANT values do not support comparison; extract a typed path with variantGet first");
Review Comment:
Fixed in xiangfu0/pinot#236. The comparison/IN/DISTINCT-FROM guards now reject only raw `VARIANT` (`!= ColumnDataType.VARIANT`) instead of any non-orderable/non-equatable type, so OBJECT/array/MAP comparisons keep their prior behavior. Applied to both engines: `FilterOperand` (Predicate, In), `TransformOperandFactory` (distinct-from), and the single-stage `BinaryOperatorTransformFunction` / `InTransformFunction`, which had the same guard. Added `FilterOperandTest.testNonVariantOpaqueTypesAreNotRejectedAsVariant`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add end-to-end Parquet VARIANT support [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #19101:
URL: https://github.com/apache/pinot/pull/19101#discussion_r3920906239
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/operands/FilterOperand.java:
##########
@@ -193,7 +197,12 @@ public Predicate(List<RexExpression> operands, DataSchema dataSchema, IntPredica
ColumnDataType lhsType = _lhs.getResultType();
ColumnDataType rhsType = _rhs.getResultType();
- if (lhsType == rhsType) {
+ // Reject raw VARIANT operands only; other non-orderable types (OBJECT, arrays, MAP) keep their existing
+ // best-effort comparison behavior. VARIANT is opaque because its PVAR byte encoding is not a canonical
+ // semantic ordering, so a comparison must extract a typed scalar first.
+ Preconditions.checkArgument(lhsType != ColumnDataType.VARIANT && rhsType != ColumnDataType.VARIANT,
+ "Raw VARIANT values do not support comparison; extract a typed path with variantGet first");
+ if (lhsType == ColumnDataType.UNKNOWN || rhsType == ColumnDataType.UNKNOWN || lhsType == rhsType) {
Review Comment:
Kept the SQL-null short-circuit in both engines and added an explicit compatibility note in the PR description. The existing left/right NULL-literal regressions remain part of verification.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/ColumnarValueNormalizer.java:
##########
@@ -28,21 +28,21 @@
/// (`buildColumnar`) path.
///
/// The row-major build runs each record through a `TransformPipeline` whose
-/// `NullValueTransformer` substitutes [FieldSpec#getDefaultNullValue()] for `null`
-/// and whose `DataTypeTransformer` coerces every value to the column's stored type (e.g.
+/// `DataTypeTransformer` coerces every non-null value to the column's stored type (e.g.
/// `Boolean` → `Integer` for a `BOOLEAN` column stored as `INT`,
-/// `Timestamp` → `Long` for `TIMESTAMP`). The column-major driver deliberately runs
-/// with no transform pipeline, so a non-segment source (e.g. Arrow) delivers values in the source's
+/// `Timestamp` → `Long` for `TIMESTAMP`) and whose `NullValueTransformer` substitutes
+/// [FieldSpec#getDefaultNullValue()] for values that remain `null`. The column-major driver deliberately runs with
+/// no transform pipeline, so a non-segment source (e.g. Arrow) delivers values in the source's
/// logical type with raw `null`s — which the typed collectors / index creators do not accept.
///
/// This helper applies the equivalent of those two transformers to one value, in the same order:
///
-/// 1. `NullValueTransformer`: a `null` value becomes the column default — the scalar
-/// default for single-value columns, or a one-element `Object[]` of that scalar for
-/// multi-value columns (matching `NullValueTransformerUtils.getDefaultNullValue`).
-/// 2. `DataTypeTransformer`: [DataTypeTransformerUtils#transformValue] standardizes the
+/// 1. `DataTypeTransformer`: [DataTypeTransformerUtils#transformValue] standardizes the
Review Comment:
Documented the columnar normalization ordering change and its shared-path impact in the PR description. The implementation now matches the row pipeline.
##########
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/creator/impl/ColumnarValueNormalizerTest.java:
##########
@@ -0,0 +1,43 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl;
+
+import org.apache.pinot.spi.data.DimensionFieldSpec;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.FieldSpec.DataType;
+import org.apache.pinot.spi.utils.PinotDataType;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertSame;
+
+
+public class ColumnarValueNormalizerTest {
+ private static final String COLUMN = "payload";
+
+ @Test
+ public void testNullVariantReturnsDefaultSentinelWithoutDecoding() {
Review Comment:
Expanded `ColumnarValueNormalizerTest` to pin BOOLEAN as `Integer 0`, TIMESTAMP as `Long 0`, a one-element MV default array, and the Variant sentinel.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java:
##########
@@ -1970,7 +2060,7 @@ private static void validateStarTreeIndexConfigs(List<StarTreeIndexConfig> starT
List<String> dimensionsSplitOrder = starTreeIndexConfig.getDimensionsSplitOrder();
assert CollectionUtils.isNotEmpty(dimensionsSplitOrder);
for (String dimension : dimensionsSplitOrder) {
- if (timestampIndexColumns.contains(dimension)) {
+ if (timestampIndexColumns.contains(dimension) && schema.getFieldSpecFor(dimension) == null) {
Review Comment:
Reworked this to preserve the unconditional timestamp-derived-name skip, avoiding schema-provenance-dependent validation. A schema-declared collision is checked only to reject VARIANT; declared non-Variant and generated-column cases stay valid. Both paths are tested, and the change is isolated in `07d3dea1da`.
##########
pinot-common/pom.xml:
##########
@@ -200,6 +200,18 @@
<groupId>org.apache.pinot</groupId>
<artifactId>pinot-timeseries-spi</artifactId>
</dependency>
+ <dependency>
+ <groupId>org.apache.parquet</groupId>
+ <artifactId>parquet-variant</artifactId>
Review Comment:
Removed the unsound parquet-column exclusion, excluded unused parquet-jackson, and added the requested global-shade / apache/pinot#18459 note. `mvn -pl pinot-common dependency:tree -Dincludes=org.apache.parquet` now shows parquet-variant -> parquet-common/format-structures plus parquet-column/encoding, with no parquet-jackson.
##########
pinot-query-planner/src/main/java/org/apache/pinot/calcite/sql/fun/PinotOperatorTable.java:
##########
@@ -470,6 +472,11 @@ public void lookupOperatorOverloads(SqlIdentifier opName, @Nullable SqlFunctionC
if (!opName.isSimple()) {
return;
}
+ if (!_nullHandlingEnabled && TransformFunctionType.requiresNullHandling(opName.getSimple())) {
Review Comment:
The operator table now canonicalizes once, returns on map miss, and consults a constructor-cached empty-or-canonical gated-name set only for Pinot operator hits. The user error is a `QueryException` with `QUERY_VALIDATION`, matching broker/result validation.
##########
pinot-common/src/main/java/org/apache/pinot/common/function/TransformFunctionType.java:
##########
@@ -117,6 +122,17 @@ public enum TransformFunctionType {
JSON_EXTRACT_KEY("jsonExtractKey", ReturnTypes.TO_ARRAY,
OperandTypes.family(
List.of(SqlTypeFamily.CHARACTER, SqlTypeFamily.CHARACTER, SqlTypeFamily.CHARACTER), i -> i > 1)),
+ VARIANT_GET("variantGet", TransformFunctionType::variantGetReturnTypeInference, variantGetOperandTypeChecker()),
+ TRY_VARIANT_GET("tryVariantGet", TransformFunctionType::variantGetReturnTypeInference,
+ variantGetOperandTypeChecker()),
+ VARIANT_EXISTS("variantExists", ReturnTypes.BOOLEAN_NULLABLE, variantPathOperandTypeChecker()),
+ IS_VARIANT_NULL("isVariantNull", ReturnTypes.BOOLEAN, optionalVariantPathOperandTypeChecker()),
+ VARIANT_TYPE_OF("variantTypeOf", ReturnTypes.VARCHAR_2000_NULLABLE, optionalVariantPathOperandTypeChecker()),
+ VARIANT_TO_JSON("variantToJson", ReturnTypes.VARCHAR_2000_NULLABLE, OperandTypes.ANY),
+ PARSE_JSON_TO_VARIANT("parseJson", TransformFunctionType::nullableVariantReturnTypeInference,
Review Comment:
Made `parseJsonToVariant` / `tryParseJsonToVariant` the primary names and retained Spark-compatible `parseJson` aliases. `variantToJson` now uses a VARIANT operand checker, so non-Variant arguments fail during planning.
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/HashJoinOperator.java:
##########
@@ -79,18 +99,55 @@ public HashJoinOperator(OpChainExecutionContext context, MultiStageOperator left
_nullKeyRightRows = needUnmatchedRightRows() ? new ArrayList<>() : null;
}
- /// Constructor that takes the schema for NonEquiEvaluator as an argument
+ /// Constructor that takes the schema for NonEquiEvaluator as an argument.
+ ///
+ /// <p>For SEMI and ANTI joins whose node does not carry its inputs, the result schema contains only left columns, so
+ /// this legacy constructor cannot validate the right key's logical type. New callers that need right-side VARIANT
+ /// validation must use the overload that accepts {@code rightSchema}.
public HashJoinOperator(OpChainExecutionContext context, MultiStageOperator leftInput, DataSchema leftSchema,
MultiStageOperator rightInput, JoinNode node, DataSchema nonEquiEvaluationSchema) {
+ this(context, leftInput, leftSchema, rightInput, tryInferRightSchema(leftSchema, node), node,
+ nonEquiEvaluationSchema, false);
+ }
+
+ /// Constructor that takes the schema for NonEquiEvaluator as an argument
+ public HashJoinOperator(OpChainExecutionContext context, MultiStageOperator leftInput, DataSchema leftSchema,
+ MultiStageOperator rightInput, DataSchema rightSchema, JoinNode node, DataSchema nonEquiEvaluationSchema) {
+ this(context, leftInput, leftSchema, rightInput, rightSchema, node, nonEquiEvaluationSchema, true);
+ }
+
+ private HashJoinOperator(OpChainExecutionContext context, MultiStageOperator leftInput, DataSchema leftSchema,
+ MultiStageOperator rightInput, @Nullable DataSchema rightSchema, JoinNode node,
+ DataSchema nonEquiEvaluationSchema, boolean rightSchemaRequired) {
super(context, leftInput, leftSchema, rightInput, node, nonEquiEvaluationSchema);
List<Integer> leftKeys = node.getLeftKeys();
Preconditions.checkState(!leftKeys.isEmpty(), "Hash join operator requires join keys");
+ Preconditions.checkArgument(!rightSchemaRequired || rightSchema != null, "Right input schema must not be null");
+ JoinKeyTypeValidator.validate(node, leftSchema, rightSchema);
_leftKeySelector = KeySelectorFactory.getKeySelector(leftKeys);
_rightKeySelector = KeySelectorFactory.getKeySelector(node.getRightKeys());
_rightTable = createLookupTable(leftKeys, leftSchema);
_matchedRightRows = needUnmatchedRightRows() ? new HashMap<>() : null;
}
+ @Nullable
+ private static DataSchema tryInferRightSchema(DataSchema leftSchema, JoinNode node) {
Review Comment:
Deprecated both legacy constructors and made the null-right-schema path still validate left keys. All in-tree construction now passes the right schema explicitly, including SEMI/ANTI joins, and tests cover right-side raw VARIANT rejection. The limitation is disclosed in the PR body.
##########
pinot-core/src/main/java/org/apache/pinot/core/plan/DistinctPlanNode.java:
##########
@@ -54,6 +55,18 @@ public DistinctPlanNode(SegmentContext segmentContext, QueryContext queryContext
@Override
public Operator<DistinctResultsBlock> run() {
List<ExpressionContext> expressions = _queryContext.getSelectExpressions();
+ for (ExpressionContext expression : expressions) {
+ String column = expression.getIdentifier();
+ if (column != null) {
+ DataType dataType = _indexSegment.getDataSource(column, _queryContext.getSchema())
Review Comment:
Added a one-shot cached `QueryContext.hasVariantColumns()` flag. DISTINCT and aggregation identifier validation skip all per-expression data-source lookups when the query schema has no VARIANT; focused tests cover both fast-skip and validation paths.
##########
compatibility-verifier/compCheck.sh:
##########
@@ -496,14 +498,24 @@ setupControllerVariables
setupBrokerVariables
setupServerVariables
-export JAVA_OPTS="-DControllerPort=${CONTROLLER_PORT} -DBrokerQueryPort=${BROKER_QUERY_PORT} -DServerAdminPort=${SERVER_ADMIN_PORT}"
-
mkdir ${PID_DIR}
mkdir ${LOG_DIR}
oldTargetDir="$workingDir"/oldTargetDir
newTargetDir="$workingDir"/newTargetDir
+oldServerSupportsVariant=false
+oldExpressionsProto="${oldTargetDir}/pinot-common/src/main/proto/expressions.proto"
+if [ -f "${oldExpressionsProto}" ] \
+ && grep -Eq '^[[:space:]]*VARIANT[[:space:]]*=[[:space:]]*24[[:space:]]*;' "${oldExpressionsProto}"; then
Review Comment:
Replaced source-text grep with capability detection against the built old `pinot-common` artifact via `FileContainsOp`; inability to inspect is a hard failure rather than a silent skip. Also documented intentional generation reuse for the old-broker/new-server phase.
##########
pinot-plugins/pinot-input-format/pinot-parquet/src/main/java/org/apache/pinot/plugin/inputformat/parquet/ParquetNativeRecordReader.java:
##########
@@ -122,9 +159,11 @@ public GenericRow next(GenericRow reuse)
} catch (Exception e) {
Review Comment:
Moved the full reader init/rewind/close robustness changes into standalone commit `55f1aa6d37` for backportability. `useAvroParquetRecordReader()` now retains explicit selected-reader state before init and after close, with a dedicated regression test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add end-to-end Parquet VARIANT support [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #19101:
URL: https://github.com/apache/pinot/pull/19101#discussion_r3742852804
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/SortOperator.java:
##########
@@ -73,6 +74,11 @@ public SortOperator(OpChainExecutionContext context, MultiStageOperator input, S
// - There is no collation
// - Input is already sorted
List<RelFieldCollation> collations = node.getCollations();
+ for (RelFieldCollation collation : collations) {
+ Preconditions.checkArgument(
+ _dataSchema.getColumnDataType(collation.getFieldIndex()).supportsOrdering(),
+ "ORDER BY does not support raw VARIANT values; extract a typed path with variantGet first");
Review Comment:
**Misleading message (Low).** `supportsOrdering()` is false for VARIANT **and** arrays/OBJECT/MAP/STRUCT/LIST, so this rejects e.g. `ORDER BY array_col` with a VARIANT-specific message. The rejection is correct; only the text is wrong for non-VARIANT types. Same hard-coded-VARIANT-message-on-a-generic-`supportsOrdering()`-gate appears in `SortedMailboxReceiveOperator`, `OrderByComparatorFactory`, and `VariantTypeValidationVisitor#visitSort`/`visitWindow`. Consider naming the actual unsupported type.
##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/validation/VariantTypeValidationVisitor.java:
##########
@@ -0,0 +1,185 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.query.planner.validation;
+
+import java.util.List;
+import org.apache.calcite.rel.RelFieldCollation;
+import org.apache.pinot.common.utils.DataSchema;
+import org.apache.pinot.common.utils.VariantUtils;
+import org.apache.pinot.query.planner.logical.RexExpression;
+import org.apache.pinot.query.planner.plannode.AggregateNode;
+import org.apache.pinot.query.planner.plannode.JoinNode;
+import org.apache.pinot.query.planner.plannode.PlanNode;
+import org.apache.pinot.query.planner.plannode.PlanNodeVisitor;
+import org.apache.pinot.query.planner.plannode.SetOpNode;
+import org.apache.pinot.query.planner.plannode.SortNode;
+import org.apache.pinot.query.planner.plannode.WindowNode;
+import org.apache.pinot.spi.exception.QueryErrorCode;
+import org.apache.pinot.spi.exception.QueryException;
+
+
+/// Rejects operations that would otherwise assign physical byte ordering, equality, or hashing semantics to a raw
+/// VARIANT value. The visitor has no mutable state and is thread-safe, so callers may share {@link #INSTANCE}.
+public final class VariantTypeValidationVisitor extends PlanNodeVisitor.DepthFirstVisitor<Void, Void> {
+ public static final VariantTypeValidationVisitor INSTANCE = new VariantTypeValidationVisitor();
+
+ private VariantTypeValidationVisitor() {
+ }
+
+ @Override
+ protected boolean traverseStageBoundary() {
+ return false;
+ }
+
+ @Override
+ public Void visitAggregate(AggregateNode node, Void context) {
+ List<PlanNode> inputs = node.getInputs();
+ if (inputs.size() == 1) {
+ validateAggregateInputs(node, inputs.get(0).getDataSchema());
+ }
+ return super.visitAggregate(node, context);
+ }
+
+ /// Validates aggregate operands against their logical input schema.
+ ///
+ /// <p>This method is also invoked by the runtime as a defensive check for plans that did not pass through the
+ /// current broker planner.
+ public static void validateAggregateInputs(AggregateNode node, DataSchema inputSchema) {
Review Comment:
**Consistency (Low, not exploitable).** `validateAggregateInputs` iterates only the agg-call operands; it never checks `AggregateNode.getGroupKeys()`, so the planner gate doesn't reject `GROUP BY raw_variant`. This is *not* a hole — `MultistageGroupByExecutor` and the single-stage GroupBy operators reject it at runtime via `supportsEquality()/supportsHashing()`. But since every other operation (sort, set-op, join, window partition) is validated in this visitor, adding the group-key check here would make the planner the consistent authoritative gate.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java:
##########
@@ -1448,6 +1456,10 @@ static void validatePartialUpsertStrategies(TableConfig tableConfig, Schema sche
FieldSpec fieldSpec = schema.getFieldSpecFor(column);
Preconditions.checkState(fieldSpec != null, "Merger cannot be applied to non-existing column: %s", column);
+ if (fieldSpec.getDataType() == DataType.VARIANT) {
+ Preconditions.checkState(columnStrategy == UpsertConfig.Strategy.OVERWRITE,
+ "VARIANT column supports only OVERWRITE partial-upsert strategy: %s", column);
Review Comment:
**Validation gap (Medium).** This VARIANT `OVERWRITE`-only check only runs for columns explicitly present in `partialUpsertStrategies`. A VARIANT column **not listed** here falls back to `upsertConfig.getDefaultPartialUpsertStrategy()` at merge time — see `PartialUpsertColumnarMerger#merge` (`_column2Mergers.getOrDefault(column, _defaultColumnValueMerger)`), where the default merger is built from `getDefaultPartialUpsertStrategy()` and is user-settable to `INCREMENT`/`APPEND`/`UNION`/`IGNORE`.
Failing scenario: a partial-upsert table with an unlisted VARIANT dimension and `defaultPartialUpsertStrategy: INCREMENT`. Validation passes, then at runtime `IncrementMerger` is applied to the byte envelope → failure/corruption; `IGNORE` would silently retain stale variant values. This violates the documented "partial upsert only OVERWRITE" contract.
Suggest: when a VARIANT column exists under active partial upsert, also require the effective default strategy to be OVERWRITE. (Also note the custom `partialUpsertMergerClass` path applies no VARIANT guard — likely acceptable as user code, but worth a doc note.)
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/operands/FilterOperand.java:
##########
@@ -193,7 +196,10 @@ public Predicate(List<RexExpression> operands, DataSchema dataSchema, IntPredica
ColumnDataType lhsType = _lhs.getResultType();
ColumnDataType rhsType = _rhs.getResultType();
- if (lhsType == rhsType) {
+ Preconditions.checkArgument((lhsType == ColumnDataType.UNKNOWN || lhsType.supportsOrdering())
+ && (rhsType == ColumnDataType.UNKNOWN || rhsType.supportsOrdering()),
+ "Raw VARIANT values do not support comparison; extract a typed path with variantGet first");
Review Comment:
**Behavior regression + misleading message (Medium).** This guard requires both operands to be `UNKNOWN || supportsOrdering()`, but `ColumnDataType.supportsOrdering()` is false for **OBJECT, arrays, and MAP** — not just VARIANT. The class Javadoc just above ("if either side is null or OBJECT, we best-effort cast data into the other side's data type") documents OBJECT as an intentionally-supported case, and `OBJECT = OBJECT` previously took the equal-types branch and ran. After this change such a comparison throws `"Raw VARIANT values do not support comparison"` — both a narrowing of the existing contract and a wrong error for non-VARIANT types.
Suggest gating specifically on the VARIANT case (or restoring the OBJECT/UNKNOWN best-effort path) and making the message name the actual offending type. Same pattern applies to the `In` guard at line 120 (`supportsEquality()`, also false for OBJECT/arrays).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add end-to-end Parquet VARIANT support [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #19101:
URL: https://github.com/apache/pinot/pull/19101#discussion_r4366442882
##########
docs/design/VARIANT.md:
##########
@@ -0,0 +1,355 @@
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+# Apache Pinot VARIANT — Design Document
+
+Status: draft for community and component-owner review.
+
+## 1. Motivation and goals
+
+Apache Parquet `VARIANT(1)` provides a portable binary representation for
+semi-structured values. Pinot should ingest that representation without converting
+every row through a JSON tree, retain the complete value for late-bound queries, and
+let users materialize frequently queried paths into ordinary Pinot columns.
+
+This design has five goals:
+
+1. Define a first-class Pinot `VARIANT` logical type with an explicit persisted and
+ query-wire contract.
+2. Ingest top-level, non-repeated Parquet `VARIANT(1)` values, including unshredded,
+ fully shredded, and partially shredded layouts.
+3. Provide Spark-style parse, extraction, existence, type, null, and JSON-rendering
+ functions in both Pinot query engines.
+4. Fail explicitly when a raw Variant value reaches an operation whose semantics are
+ undefined, while allowing users to extract a typed scalar for that operation.
+5. Ship an executable table-definition, ingestion, and query story with integration
+ coverage.
+
+### 1.1 Non-goals
+
+The initial contract does not include:
+
+- nested or repeated Parquet Variant columns;
+- Pinot multi-value `VARIANT` columns;
+- streaming-format Variant ingestion;
+- quoted object keys in Variant paths;
+- semantic equality, hashing, or ordering of raw Variant values;
+- mixed-version execution for queries that contain a Variant type; or
+- automatic rollback after a Variant table has been activated.
+
+## 2. User model
+
+A Variant column is declared as a single-value dimension:
+
+```json
+{
+ "name": "payload",
+ "dataType": "VARIANT"
+}
+```
+
+The table stores `payload` in a raw forward index without a dictionary. Storage null
+handling must be enabled. A common layout retains the source value while materializing
+an indexed scalar during ingestion:
+
+```json
+{
+ "columnName": "eventType",
+ "transformFunction": "variant_get(payload, '$.eventType', 'STRING')"
+}
+```
+
+Late-bound paths remain queryable:
+
+```sql
+SET enableNullHandling=true;
+
+SELECT
+ eventId,
+ variant_get(payload, '$.user.id', 'STRING') AS userId,
+ variant_get(payload, '$.amount', 'DOUBLE') AS amount
+FROM variantEvents
+WHERE eventType = 'checkout';
+```
+
+## 3. Data flow and ownership
+
+```text
+Parquet VARIANT(1)
+ |
+ | pinot-parquet: validate shape and reconstruct shredded values
+ v
+metadata ByteBuffer + value ByteBuffer
+ |
+ | VariantEnvelope: add stable Pinot PVAR framing
+ v
+single-value raw BYTES forward index in a Pinot segment
+ |
+ | DataSchema/protobuf: carry logical VARIANT, not generic BYTES
+ v
+single-stage or multi-stage Variant functions
+ |
+ +--> typed scalar for filtering/grouping/joining/aggregation
+ |
+ +--> canonical JSON text for response clients
+```
+
+Ownership is deliberately split by module:
+
+- `pinot-spi` owns the public logical type, capability contract, and stable `PVAR`
+ envelope.
+- `pinot-segment-local` owns schema, table, index, null, and segment-metadata
+ validation.
+- `pinot-parquet` owns Parquet logical-type validation and reconstruction.
+- `pinot-common`, `pinot-core`, `pinot-query-planner`, and `pinot-query-runtime` own
+ function semantics and defensive query guards.
+- client, response, and DDL modules carry the logical type without redefining its
+ semantics.
+
+`pinot-common` depends on `parquet-variant` and its required `parquet-common` and
+`parquet-column` codec closure; the unused `parquet-jackson` artifact is excluded.
+Hadoop and record-reader integrations remain isolated to the input-format plugin.
+
+## 4. Persisted `PVAR` envelope
+
+Pinot segments need one byte sequence that contains both buffers in a Parquet Variant
+value. `VariantEnvelope` owns that framing. Version 1 is:
+
+```text
+offset size field
+0 4 ASCII magic "PVAR"
+4 1 envelope version (1)
+5 1 flags (0)
+6 2 reserved (0)
+8 4 metadata length, big-endian signed int restricted to >= 0
+12 4 value length, big-endian signed int restricted to >= 0
+16 M Parquet Variant metadata bytes
+16 + M V Parquet Variant value bytes
+```
+
+The complete envelope must fit in a Java byte array. Decoders reject unknown versions,
+non-zero flags or reserved bytes, negative lengths, bad magic, and length mismatches.
+Metadata and value bytes are copied without interpretation by the envelope layer; the
+Parquet Variant codec validates their internal representation.
+
+Version 1 is immutable and protected by a golden-byte test. A future incompatible
+framing change must use a new envelope version and retain an explicit reader for every
+supported old version. Unknown versions fail closed; they must never fall back to
+generic `BYTES`.
+
+`decode(byte[])` returns zero-copy read-only views that alias the input array. The input
+must therefore remain immutable for the lifetime of those views. Array, direct, and
+read-only source buffers are supported without changing their positions. Parquet
+ingestion copies payload bytes directly into the final envelope and must not allocate a
+full intermediate payload array.
+
+## 5. Logical type and segment contract
+
+`FieldSpec.DataType.VARIANT` is logically distinct from `BYTES` but uses variable-width
+bytes as its physical stored type.
+
+The initial schema and table constraints are:
+
+- dimension field only;
+- single-value only;
+- the reserved empty-byte default null value only;
+- no truncating or default-substitution max-length policy;
+- storage null handling enabled at schema or table level;
+- enabled raw forward index;
+- no dictionary or secondary index on the raw Variant column;
+- not a primary key, partition column, sorted column, upsert comparison column,
+ metrics-aggregation key, or star-tree dimension;
+- partial upsert may use only the `OVERWRITE` strategy; and
+- minion merge tasks (`MergeRollupTask`, `RealtimeToOfflineSegmentsTask`) may use only the `CONCAT` merge type,
+ because rollup and dedup group rows by raw stored bytes.
+
+Segment creation validates every non-null value as a complete `PVAR` envelope. Raw
+Variant columns do not publish logical minimum or maximum values and are never marked
+sorted, including single-row segments.
+
+### 5.1 Capability policy
+
+The Parquet representation is not a canonical semantic encoding: equivalent logical
+objects can use different metadata dictionaries or shredded layouts. Byte equality,
+byte hashing, and lexicographic byte order would therefore expose storage-layout
+accidents as SQL semantics.
+
+The logical type reports:
+
+| Capability on raw value | Supported |
+|---|---|
+| equality / `IN` / pattern predicates | no |
+| hashing / `GROUP BY` / `DISTINCT` | no |
+| ordering / min-max / sort / ASOF match key | no |
+| join, lookup, primary, or partition key | no |
+| direct value-consuming aggregation | no |
+| `CAST` from a raw value | no |
+| non-distinct `COUNT(raw_variant)` | yes |
+
+Planner validation provides the primary error. Runtime and single-stage guards remain
+in place for mixed plans or future planner changes. Error messages direct users to
+`variant_get` a typed path first.
+
+## 6. Parquet ingestion
+
+Reader initialization scans the file schema before publishing reader state.
+
+- A supported field is a top-level, non-repeated group annotated `VARIANT(1)`.
+- `metadata` is required binary; at least one of `value` or `typed_value` is present.
+- The parquet-java `VariantConverters` tree reconstructs fully or partially shredded
+ values.
+- The unshredded path retains the exact metadata and value buffers and adds only the
+ `PVAR` envelope.
+- Converter plans and field indexes are immutable and created once per reader.
+- A failed reinitialization leaves the previous reader usable. A failure while closing
+ the old reader does not invalidate the successfully initialized replacement.
+
+Automatic reader selection preserves existing Avro-metadata precedence for backward
+compatibility. A Parquet file with Avro metadata must explicitly select the native
+reader to ingest Variant.
+
+Malformed rows follow the existing record-reader skip policy and do not corrupt reader
+progress. Unsupported Variant spec versions and unsupported nested/repeated shapes fail
+during initialization rather than appearing as ordinary structs.
+
+## 7. Query functions
+
+Both query engines expose:
+
+- `variant_get(value, path[, targetType])`
+- `try_variant_get(value, path[, targetType])`
+- `variant_exists(value, path)`
+- `is_variant_null(value[, path])`
+- `variant_typeof(value[, path])`
+- `variant_to_json(value)`
+- `parse_json(text)` / `parse_json_to_variant(text)`
+- `try_parse_json(text)` / `try_parse_json_to_variant(text)`
+
+Function lookup remains case-insensitive under Pinot's existing registration rules.
+The v1 path grammar supports root `$`, dot-separated object fields, and non-negative
+array indexes. Supported extraction targets are `BOOLEAN`, `INT`, `LONG`, `FLOAT`,
+`DOUBLE`, `BIG_DECIMAL`, `STRING`, `BYTES`, `UUID`, `TIMESTAMP`, `VARIANT`, and `JSON`.
+
+Neither engine constant-folds the JSON-to-Variant functions: a folded result would
+degrade to a plain `BYTES` literal and erase the logical type together with its guards.
+
+Strict functions reject malformed encodings, paths, incompatible conversions, numeric
+overflow, and malformed JSON. `try_` functions return SQL null for those failures.
+Literal paths, target types, and JSON inputs are compiled or parsed once per query.
+Vectorized execution reuses thread-confined cursors and unboxed result holders.
+
+The query cursor directly decodes the Parquet Variant binary layout to avoid per-row
+codec objects and buffer allocation. Its type tags, length rules, metadata version,
+and wide-object threshold must stay synchronized with the pinned `parquet-variant`
+dependency. Tests compare the cursor constants with parquet-java's codec constants,
+exercise every `Variant.Type`, and reject unrecognized metadata versions and primitive
+tags. A parquet-variant upgrade that adds an encoding must update the cursor and these
+tests in the same change; unknown encodings fail closed rather than returning a
+partially decoded or incorrectly typed result.
+
+JSON parsing is bounded to 100 nesting levels. Parquet Variant decimal encoding is
+bounded to 38 digits of precision and a scale of at most 38 after exact normalization.
+Exponent bounds are checked before expansion so hostile inputs cannot request
+pathological intermediate allocations.
+
+### 7.1 Null states
+
+Storage and query null handling are required because four states must remain distinct:
+
+| State | Stored bytes / null vector | `variant_exists` | `is_variant_null` | `variant_typeof` | scalar `variant_get` |
+|---|---|---:|---:|---|---|
+| SQL null | empty bytes, null bit set | SQL null | false | SQL null | SQL null |
+| Variant null | non-empty `PVAR`, null bit clear | true | true | `NULL` | SQL null |
Review Comment:
Clarified §6.1 in cb407e5149: ingestion is not an eager recursive validator—opaque raw values/residual subtrees can retain unknown tags until strict decoding, while reconstruction that actually decodes an unknown primitive fails the row.
The pinned parquet-java 1.18.1 decoder throws for unknown primitive tags; it does not convert them to Variant null. The passing real-file regression verifies raw-byte preservation, strict decode/reconstruction failures, reader recovery, and genuine Variant null.
We keep the raw fast path; `try_` query functions intentionally return SQL null on decoding errors.
##########
docs/design/VARIANT.md:
##########
@@ -0,0 +1,355 @@
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+# Apache Pinot VARIANT — Design Document
+
+Status: draft for community and component-owner review.
+
+## 1. Motivation and goals
+
+Apache Parquet `VARIANT(1)` provides a portable binary representation for
+semi-structured values. Pinot should ingest that representation without converting
+every row through a JSON tree, retain the complete value for late-bound queries, and
+let users materialize frequently queried paths into ordinary Pinot columns.
+
+This design has five goals:
+
+1. Define a first-class Pinot `VARIANT` logical type with an explicit persisted and
+ query-wire contract.
+2. Ingest top-level, non-repeated Parquet `VARIANT(1)` values, including unshredded,
+ fully shredded, and partially shredded layouts.
+3. Provide Spark-style parse, extraction, existence, type, null, and JSON-rendering
+ functions in both Pinot query engines.
+4. Fail explicitly when a raw Variant value reaches an operation whose semantics are
+ undefined, while allowing users to extract a typed scalar for that operation.
+5. Ship an executable table-definition, ingestion, and query story with integration
+ coverage.
+
+### 1.1 Non-goals
+
+The initial contract does not include:
+
+- nested or repeated Parquet Variant columns;
+- Pinot multi-value `VARIANT` columns;
+- streaming-format Variant ingestion;
+- quoted object keys in Variant paths;
+- semantic equality, hashing, or ordering of raw Variant values;
+- mixed-version execution for queries that contain a Variant type; or
+- automatic rollback after a Variant table has been activated.
+
+## 2. User model
+
+A Variant column is declared as a single-value dimension:
+
+```json
+{
+ "name": "payload",
+ "dataType": "VARIANT"
+}
+```
+
+The table stores `payload` in a raw forward index without a dictionary. Storage null
+handling must be enabled. A common layout retains the source value while materializing
+an indexed scalar during ingestion:
+
+```json
+{
+ "columnName": "eventType",
+ "transformFunction": "variant_get(payload, '$.eventType', 'STRING')"
+}
+```
+
+Late-bound paths remain queryable:
+
+```sql
+SET enableNullHandling=true;
+
+SELECT
+ eventId,
+ variant_get(payload, '$.user.id', 'STRING') AS userId,
+ variant_get(payload, '$.amount', 'DOUBLE') AS amount
+FROM variantEvents
+WHERE eventType = 'checkout';
+```
+
+## 3. Data flow and ownership
+
+```text
+Parquet VARIANT(1)
+ |
+ | pinot-parquet: validate shape and reconstruct shredded values
+ v
+metadata ByteBuffer + value ByteBuffer
+ |
+ | VariantEnvelope: add stable Pinot PVAR framing
+ v
+single-value raw BYTES forward index in a Pinot segment
+ |
+ | DataSchema/protobuf: carry logical VARIANT, not generic BYTES
+ v
+single-stage or multi-stage Variant functions
+ |
+ +--> typed scalar for filtering/grouping/joining/aggregation
+ |
+ +--> canonical JSON text for response clients
+```
+
+Ownership is deliberately split by module:
+
+- `pinot-spi` owns the public logical type, capability contract, and stable `PVAR`
+ envelope.
+- `pinot-segment-local` owns schema, table, index, null, and segment-metadata
+ validation.
+- `pinot-parquet` owns Parquet logical-type validation and reconstruction.
+- `pinot-common`, `pinot-core`, `pinot-query-planner`, and `pinot-query-runtime` own
+ function semantics and defensive query guards.
+- client, response, and DDL modules carry the logical type without redefining its
+ semantics.
+
+`pinot-common` depends on `parquet-variant` and its required `parquet-common` and
+`parquet-column` codec closure; the unused `parquet-jackson` artifact is excluded.
+Hadoop and record-reader integrations remain isolated to the input-format plugin.
+
+## 4. Persisted `PVAR` envelope
+
+Pinot segments need one byte sequence that contains both buffers in a Parquet Variant
+value. `VariantEnvelope` owns that framing. Version 1 is:
+
+```text
+offset size field
+0 4 ASCII magic "PVAR"
+4 1 envelope version (1)
+5 1 flags (0)
+6 2 reserved (0)
+8 4 metadata length, big-endian signed int restricted to >= 0
+12 4 value length, big-endian signed int restricted to >= 0
+16 M Parquet Variant metadata bytes
+16 + M V Parquet Variant value bytes
+```
+
+The complete envelope must fit in a Java byte array. Decoders reject unknown versions,
+non-zero flags or reserved bytes, negative lengths, bad magic, and length mismatches.
+Metadata and value bytes are copied without interpretation by the envelope layer; the
+Parquet Variant codec validates their internal representation.
+
+Version 1 is immutable and protected by a golden-byte test. A future incompatible
+framing change must use a new envelope version and retain an explicit reader for every
+supported old version. Unknown versions fail closed; they must never fall back to
+generic `BYTES`.
+
+`decode(byte[])` returns zero-copy read-only views that alias the input array. The input
+must therefore remain immutable for the lifetime of those views. Array, direct, and
+read-only source buffers are supported without changing their positions. Parquet
+ingestion copies payload bytes directly into the final envelope and must not allocate a
+full intermediate payload array.
+
+## 5. Logical type and segment contract
+
+`FieldSpec.DataType.VARIANT` is logically distinct from `BYTES` but uses variable-width
+bytes as its physical stored type.
+
+The initial schema and table constraints are:
+
+- dimension field only;
+- single-value only;
+- the reserved empty-byte default null value only;
+- no truncating or default-substitution max-length policy;
+- storage null handling enabled at schema or table level;
+- enabled raw forward index;
+- no dictionary or secondary index on the raw Variant column;
Review Comment:
Documented the v1 boundary in §5 (cb407e5149): no per-path VARIANT index or raw-column JSON index is included; supported indexed access uses explicit typed scalar materialization plus an ordinary Pinot index.
Unmaterialized paths are decoded for candidate rows reaching the expression, which other predicates/pruning can reduce; a future path index needs a separate design, not a roadmap promise from this PR.
Historical segments require explicit backfill through rebuild/reingestion, or a separately validated supported derived-column reload route.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add end-to-end Parquet VARIANT support [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #19101:
URL: https://github.com/apache/pinot/pull/19101#discussion_r3914270630
##########
pinot-spi/VARIANT_DESIGN.md:
##########
@@ -0,0 +1,344 @@
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+# Apache Pinot VARIANT — Design Document
Review Comment:
Read this first: it is the real spec for the PR, and it is unusually good. Sections 5.1 (capability policy) and 7.1 (the four null states) are what everything else derives from.
Two housekeeping points. (1) The doc lives in pinot-spi/ root, which is a source module, not a docs location - it will be shipped inside the pinot-spi source tree and will not be found by anyone reading pinot-docs. Move it to pinot-docs, or to a top-level design/ or docs/design/ folder. (2) Section 8 says numbers 22 and 23 'remain reserved for the separately allocated UUID contract'; they are not reserved, they are already assigned to UUID and UUID_ARRAY. That sentence is in the paragraph about permanent wire allocations, so it should be exact.
##########
pinot-spi/src/main/java/org/apache/pinot/spi/utils/VariantEnvelope.java:
##########
@@ -0,0 +1,243 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.spi.utils;
+
+import java.nio.ByteBuffer;
+import java.util.Objects;
+import javax.annotation.Nullable;
+
+
+/// Pinot-owned framing for the two buffers that make up a Parquet Variant value.
+///
+/// The version-1 wire format is:
+/// ```
+/// 0 4 bytes ASCII magic "PVAR"
+/// 4 1 byte envelope version (1)
+/// 5 1 byte flags (0)
+/// 6 2 bytes reserved (0)
+/// 8 4 bytes metadata length, unsigned range restricted to Java array sizes
+/// 12 4 bytes value length, unsigned range restricted to Java array sizes
+/// 16 M bytes Parquet Variant metadata
+/// 16 + M V bytes Parquet Variant value
+/// ```
+///
+/// An empty byte array is deliberately not an envelope. Pinot reserves it as the default null value for a
+/// `VARIANT` field, allowing the null-value vector to distinguish SQL null from an encoded Variant null.
+///
+/// This class validates only Pinot's stable outer framing. Producers and consumers remain responsible for validating
+/// the Parquet Variant metadata and value payloads.
+public final class VariantEnvelope {
+ public static final int HEADER_SIZE = 16;
+ public static final byte VERSION = 1;
+ public static final byte FLAGS = 0;
+
+ private static final int MAGIC = 0x50564152; // ASCII "PVAR"
+
+ private VariantEnvelope() {
+ }
+
+ /// Encodes the remaining bytes of the supplied metadata and value buffers without changing their positions or
+ /// limits.
+ ///
+ /// Array-backed buffers are copied directly from their backing arrays. Other buffers, including direct and
+ /// read-only buffers, are read through independent duplicate views.
+ public static byte[] encode(ByteBuffer metadata, ByteBuffer value) {
+ Objects.requireNonNull(metadata, "metadata must not be null");
+ Objects.requireNonNull(value, "value must not be null");
+
+ int metadataLength = metadata.remaining();
+ int valueLength = value.remaining();
+ byte[] envelope = allocate(metadataLength, valueLength);
+ copyRemaining(metadata, envelope, HEADER_SIZE);
+ copyRemaining(value, envelope, HEADER_SIZE + metadataLength);
+ return envelope;
+ }
+
+ /// Encodes slices of the supplied arrays without allocating intermediate buffer views.
+ public static byte[] encode(byte[] metadata, int metadataOffset, int metadataLength, byte[] value, int valueOffset,
+ int valueLength) {
+ Objects.requireNonNull(metadata, "metadata must not be null");
+ Objects.requireNonNull(value, "value must not be null");
+ requireRange(metadata, metadataOffset, metadataLength, "metadata");
+ requireRange(value, valueOffset, valueLength, "value");
+
+ byte[] envelope = allocate(metadataLength, valueLength);
+ System.arraycopy(metadata, metadataOffset, envelope, HEADER_SIZE, metadataLength);
+ System.arraycopy(value, valueOffset, envelope, HEADER_SIZE + metadataLength, valueLength);
+ return envelope;
+ }
+
+ /// Decodes and validates an envelope, returning zero-copy, read-only views over its metadata and value buffers.
+ ///
+ /// The returned views alias `envelope`; this method does not copy either payload. The decoded object and any views
+ /// obtained from it keep the backing array alive, so the caller does not need to retain a separate reference to
+ /// `envelope`. Mutations made to the input array after this method returns are visible through the views and can
+ /// corrupt the decoded payload. Callers must therefore treat the input array as immutable for as long as the decoded
+ /// object or any returned view may be used.
+ ///
+ /// The decoded holder is safe for concurrent reads when the aliased input array is not mutated. Each accessor returns
+ /// a read-only view with independent position and limit, so cursor movement by one reader does not affect another
+ /// reader.
+ public static Decoded decode(byte[] envelope) {
Review Comment:
The framing layer is the strongest part of the change: magic, version, flags, reserved bytes, negative lengths and length agreement are all validated, and decode() aliases the input without copying.
Two small notes. (1) allocate() is public and returns an array with a valid header but an uninitialised payload, so isEnvelope() returns true for it. The javadoc says callers must fill both regions before publishing, but a public factory that can hand out a 'valid' envelope full of zeros is easy to misuse - consider package-private, or a name that says so (allocateUnfilled). (2) The class javadoc describes the two length fields as 'unsigned range restricted to Java array sizes' while VARIANT_DESIGN.md section 4 describes them as 'big-endian signed int restricted to >= 0'. Same behaviour, two descriptions of a format that is explicitly frozen - pick one wording and use it in both places.
##########
pinot-common/src/main/proto/expressions.proto:
##########
@@ -46,6 +46,7 @@ enum ColumnDataType {
BIG_DECIMAL_ARRAY = 21;
UUID = 22;
UUID_ARRAY = 23;
+ VARIANT = 24;
Review Comment:
Permanent wire number 24 - the one line in this PR that can never be taken back. Worth an explicit sign-off on the number alone, separate from the rest of the review.
Everything else here is revertible. An enum number that has shipped in a release is not: any future reuse of 24 silently misinterprets in-flight plans between mixed-version nodes. The frozen legacy_expressions.proto test that asserts an old peer reads 24 as UNRECOGNIZED is the right guard and is well done.
##########
pinot-common/src/test/java/org/apache/pinot/common/utils/VariantUtilsTest.java:
##########
@@ -0,0 +1,1153 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils;
+
+import java.math.BigDecimal;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.sql.Timestamp;
+import java.util.Arrays;
+import java.util.EnumSet;
+import java.util.List;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Consumer;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.parquet.variant.Variant;
+import org.apache.parquet.variant.VariantBuilder;
+import org.apache.parquet.variant.VariantObjectBuilder;
+import org.apache.pinot.common.utils.DataSchema.ColumnDataType;
+import org.apache.pinot.common.utils.VariantUtils.ResultType;
+import org.apache.pinot.common.utils.VariantUtils.ReusableResult;
+import org.apache.pinot.common.utils.VariantUtils.VariantPath;
+import org.apache.pinot.spi.data.FieldSpec.DataType;
+import org.apache.pinot.spi.utils.ByteArray;
+import org.apache.pinot.spi.utils.UuidUtils;
+import org.apache.pinot.spi.utils.VariantEnvelope;
+import org.testng.annotations.DataProvider;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertNull;
+import static org.testng.Assert.assertSame;
+import static org.testng.Assert.assertThrows;
+import static org.testng.Assert.assertTrue;
+
+
+public class VariantUtilsTest {
+ @Test
+ public void testRawVariantResultRequiresNullHandling() {
+ DataSchema variantSchema =
+ new DataSchema(new String[]{"payload"}, new ColumnDataType[]{ColumnDataType.VARIANT});
+ DataSchema typedSchema =
+ new DataSchema(new String[]{"eventType"}, new ColumnDataType[]{ColumnDataType.STRING});
+
+ assertTrue(VariantUtils.requiresNullHandlingForRawVariantResult(variantSchema, false));
+ assertFalse(VariantUtils.requiresNullHandlingForRawVariantResult(variantSchema, true));
+ assertFalse(VariantUtils.requiresNullHandlingForRawVariantResult(typedSchema, false));
+ }
+
+ @Test
+ public void testResultTypeContract() {
+ assertEquals(ResultType.BOOLEAN.getDataType(), DataType.BOOLEAN);
+ assertEquals(ResultType.BOOLEAN.getSqlTypeName(), SqlTypeName.BOOLEAN);
+ assertEquals(ResultType.INT.getDataType(), DataType.INT);
+ assertEquals(ResultType.INT.getSqlTypeName(), SqlTypeName.INTEGER);
+ assertEquals(ResultType.LONG.getDataType(), DataType.LONG);
+ assertEquals(ResultType.LONG.getSqlTypeName(), SqlTypeName.BIGINT);
+ assertEquals(ResultType.FLOAT.getDataType(), DataType.FLOAT);
+ assertEquals(ResultType.FLOAT.getSqlTypeName(), SqlTypeName.REAL);
+ assertEquals(ResultType.DOUBLE.getDataType(), DataType.DOUBLE);
+ assertEquals(ResultType.DOUBLE.getSqlTypeName(), SqlTypeName.DOUBLE);
+ assertEquals(ResultType.BIG_DECIMAL.getDataType(), DataType.BIG_DECIMAL);
+ assertEquals(ResultType.BIG_DECIMAL.getSqlTypeName(), SqlTypeName.DECIMAL);
+ assertEquals(ResultType.STRING.getDataType(), DataType.STRING);
+ assertEquals(ResultType.STRING.getSqlTypeName(), SqlTypeName.VARCHAR);
+ assertEquals(ResultType.BYTES.getDataType(), DataType.BYTES);
+ assertEquals(ResultType.BYTES.getSqlTypeName(), SqlTypeName.VARBINARY);
+ assertEquals(ResultType.UUID.getDataType(), DataType.UUID);
+ assertEquals(ResultType.UUID.getSqlTypeName(), SqlTypeName.UUID);
+ assertEquals(ResultType.TIMESTAMP.getDataType(), DataType.TIMESTAMP);
+ assertEquals(ResultType.TIMESTAMP.getSqlTypeName(), SqlTypeName.TIMESTAMP);
+ assertEquals(ResultType.VARIANT.getDataType(), DataType.VARIANT);
+ assertEquals(ResultType.VARIANT.getSqlTypeName(), SqlTypeName.VARIANT);
+ assertEquals(ResultType.JSON.getDataType(), DataType.JSON);
+ assertEquals(ResultType.JSON.getSqlTypeName(), SqlTypeName.VARCHAR);
+ }
+
+ @Test
+ public void testDirectBinaryPathExtractionAndPredicates() {
+ byte[] variant = VariantUtils.parseJsonToVariant(
+ "{\"eventType\":\"click\",\"items\":[{\"price\":12.5},null],\"active\":true}");
+
+ assertEquals(VariantUtils.variantGet(variant, "$.eventType", "STRING"), "click");
+ assertEquals((double) VariantUtils.variantGet(variant, "$.items[0].price", "DOUBLE"), 12.5);
+ assertEquals(VariantUtils.variantGet(variant, "$.active", "BOOLEAN"), true);
+ assertTrue(VariantUtils.variantExists(variant, "$.items[1]"));
+ assertFalse(VariantUtils.variantExists(variant, "$.missing"));
+ assertTrue(VariantUtils.isVariantNull(variant, "$.items[1]"));
+ assertFalse(VariantUtils.isVariantNull(variant, "$.missing"));
+ assertEquals(VariantUtils.variantTypeOf(variant, "$.items[0]"), "OBJECT");
+ assertEquals(VariantUtils.variantTypeOf(variant, "$.items[0].price"), "DECIMAL");
+ }
+
+ @Test
+ public void testStrictAndTolerantExtraction() {
+ byte[] variant = VariantUtils.parseJsonToVariant("{\"eventType\":\"click\",\"score\":\"not-a-number\"}");
+
+ assertNull(VariantUtils.variantGet(variant, "$.missing", "STRING"));
+ assertThrows(IllegalArgumentException.class, () -> VariantUtils.variantGet(variant, "$.score", "DOUBLE"));
+ assertNull(VariantUtils.tryVariantGet(variant, "$.missing", "STRING"));
+ assertNull(VariantUtils.tryVariantGet(variant, "$.score", "DOUBLE"));
+ }
+
+ @Test
+ public void testReusableResultStrictAndTolerantExtraction() {
+ byte[] variant = VariantUtils.parseJsonToVariant("{\"value\":7,\"null\":null,\"text\":\"x\"}");
+ VariantPath valuePath = VariantUtils.compilePath("$.value");
+ ReusableResult result = new ReusableResult();
+
+ assertTrue(VariantUtils.extractInto(variant, valuePath, ResultType.INT, result));
+ assertEquals(result.getIntValue(), 7);
+ assertFalse(VariantUtils.extractInto(variant, VariantUtils.compilePath("$.missing"), ResultType.INT, result));
+ assertFalse(VariantUtils.extractInto(variant, VariantUtils.compilePath("$.null"), ResultType.INT, result));
+ assertThrows(IllegalArgumentException.class,
+ () -> VariantUtils.extractInto(variant, VariantUtils.compilePath("$.text"), ResultType.DOUBLE, result));
+ assertFalse(
+ VariantUtils.tryExtractInto(variant, VariantUtils.compilePath("$.text"), ResultType.DOUBLE, result));
+ assertFalse(VariantUtils.tryExtractInto(new byte[]{1}, valuePath, ResultType.INT, result));
+ assertThrows(NullPointerException.class, () -> VariantUtils.extractInto(
+ variant, valuePath, ResultType.INT, null));
+ assertThrows(NullPointerException.class, () -> VariantUtils.tryExtractInto(
+ variant, valuePath, ResultType.INT, null));
+ }
+
+ @Test
+ public void testReusableTolerantHeterogeneousMismatchesAndNumericRange() {
+ byte[][] rows = {
+ VariantUtils.parseJsonToVariant("{\"value\":\"not-an-int\"}"),
+ VariantUtils.parseJsonToVariant("{\"value\":true}"),
+ VariantUtils.parseJsonToVariant("{\"value\":{}}"),
+ VariantUtils.parseJsonToVariant("{\"value\":[]}"),
+ VariantUtils.parseJsonToVariant("{\"value\":2147483648}"),
+ VariantUtils.parseJsonToVariant("{\"value\":-2147483649}"),
+ VariantUtils.parseJsonToVariant("{\"value\":1.5}"),
+ VariantUtils.parseJsonToVariant("{\"value\":9223372036854775808}"),
+ VariantUtils.parseJsonToVariant("{\"value\":-9223372036854775809}")
+ };
+ ResultType[] resultTypes = {
+ ResultType.INT,
+ ResultType.INT,
+ ResultType.INT,
+ ResultType.INT,
+ ResultType.INT,
+ ResultType.INT,
+ ResultType.INT,
+ ResultType.LONG,
+ ResultType.LONG
+ };
+ VariantPath path = VariantUtils.compilePath("$.value");
+ ReusableResult result = new ReusableResult();
+
+ for (int i = 0; i < rows.length; i++) {
+ assertFalse(VariantUtils.tryExtractInto(rows[i], path, resultTypes[i], result),
+ "Expected tolerant conversion to reject row " + i);
+ }
+
+ assertThrows(IllegalArgumentException.class,
+ () -> VariantUtils.extractInto(rows[0], path, ResultType.INT, result));
+ assertThrows(ArithmeticException.class,
+ () -> VariantUtils.extractInto(rows[4], path, ResultType.INT, result));
+ assertThrows(ArithmeticException.class,
+ () -> VariantUtils.extractInto(rows[7], path, ResultType.LONG, result));
+
+ byte[] valid = VariantUtils.parseJsonToVariant("{\"value\":17}");
+ assertTrue(VariantUtils.tryExtractInto(valid, path, ResultType.INT, result));
+ assertEquals(result.getIntValue(), 17);
+ }
+
+ @Test
+ public void testReusableResultParityForEveryResultType() {
+ VariantBuilder builder = new VariantBuilder();
+ builder.appendBoolean(true);
+ assertReusableParity(encode(builder), ResultType.BOOLEAN);
+
+ builder = new VariantBuilder();
+ builder.appendInt(-17);
+ assertReusableParity(encode(builder), ResultType.INT);
+
+ builder = new VariantBuilder();
+ builder.appendLong(9_876_543_210L);
+ assertReusableParity(encode(builder), ResultType.LONG);
+
+ builder = new VariantBuilder();
+ builder.appendFloat(1.25F);
+ assertReusableParity(encode(builder), ResultType.FLOAT);
+
+ builder = new VariantBuilder();
+ builder.appendDouble(-123.5D);
+ assertReusableParity(encode(builder), ResultType.DOUBLE);
+
+ builder = new VariantBuilder();
+ builder.appendDecimal(new BigDecimal("12345678901234567890.1234"));
+ assertReusableParity(encode(builder), ResultType.BIG_DECIMAL);
+
+ builder = new VariantBuilder();
+ builder.appendString("a UTF-8 value \uD83D\uDE00");
+ assertReusableParity(encode(builder), ResultType.STRING);
+
+ builder = new VariantBuilder();
+ builder.appendBinary(ByteBuffer.wrap(new byte[]{0, 1, -1, 42}));
+ assertReusableParity(encode(builder), ResultType.BYTES);
+
+ builder = new VariantBuilder();
+ builder.appendUUID(UUID.fromString("00112233-4455-6677-8899-aabbccddeeff"));
+ assertReusableParity(encode(builder), ResultType.UUID);
+
+ builder = new VariantBuilder();
+ builder.appendTimestampNanosTz(-1_234_567_890L);
+ assertReusableParity(encode(builder), ResultType.TIMESTAMP);
+
+ byte[] nested = VariantUtils.parseJsonToVariant("{\"payload\":{\"count\":7}}");
+ assertReusableParity(nested, VariantUtils.compilePath("$.payload"), ResultType.VARIANT);
+ assertReusableParity(nested, VariantUtils.compilePath("$.payload"), ResultType.JSON);
+ }
+
+ @Test
+ public void testReusableNumericAndTemporalEncodingParity() {
+ VariantBuilder builder = new VariantBuilder();
+ builder.appendByte((byte) -8);
+ byte[] byteValue = encode(builder);
+ assertReusableParity(byteValue, ResultType.INT);
+ assertReusableParity(byteValue, ResultType.LONG);
+ assertReusableParity(byteValue, ResultType.FLOAT);
+ assertReusableParity(byteValue, ResultType.DOUBLE);
+ assertReusableParity(byteValue, ResultType.BIG_DECIMAL);
+
+ builder = new VariantBuilder();
+ builder.appendShort((short) 32_000);
+ assertReusableParity(encode(builder), ResultType.INT);
+
+ builder = new VariantBuilder();
+ builder.appendDecimal(new BigDecimal("123.45"));
+ assertReusableParity(encode(builder), ResultType.BIG_DECIMAL);
+
+ builder = new VariantBuilder();
+ builder.appendDecimal(new BigDecimal("123.00"));
+ byte[] integralDecimal = encode(builder);
+ assertReusableParity(integralDecimal, ResultType.INT);
+ assertReusableParity(integralDecimal, ResultType.LONG);
+
+ builder = new VariantBuilder();
+ builder.appendDecimal(new BigDecimal("1234567890123.45"));
+ assertReusableParity(encode(builder), ResultType.BIG_DECIMAL);
+
+ builder = new VariantBuilder();
+ builder.appendString("x".repeat(128));
+ assertReusableParity(encode(builder), ResultType.STRING);
+
+ builder = new VariantBuilder();
+ builder.appendDate(-1);
+ byte[] dateValue = encode(builder);
+ assertReusableParity(dateValue, ResultType.TIMESTAMP);
Review Comment:
Test gap for the DATE -> TIMESTAMP overflow above: this suite covers DATE conversion parity but never an epoch-day value large enough to overflow the millis multiply.
assertReusableParity() compares strict and tolerant paths for well-behaved values, so it cannot catch the case where strict throws and tolerant returns a wrapped long - the two paths genuinely disagree and the harness never asks them to. Please add a case with epochDay near Integer.MAX_VALUE asserting variantGet throws and tryVariantGet returns null. The suite is otherwise excellent - 72 cases covering every physical encoding, the wide-object threshold, and the parquet-java-vs-Arrow-RS key ordering divergence is real diligence.
##########
pinot-spi/src/main/java/org/apache/pinot/spi/data/FieldSpec.java:
##########
@@ -880,6 +952,10 @@ public Object convert(String value) {
return value;
case BYTES:
return BytesUtils.toBytes(value);
+ case VARIANT:
+ byte[] envelope = BytesUtils.toBytes(value);
+ VariantEnvelope.decode(envelope);
Review Comment:
The only legal default null value for a VARIANT column cannot be expressed in schema JSON. convert("") -> BytesUtils.toBytes("") = empty array -> VariantEnvelope.decode(empty) throws, and the user sees an opaque 'Cannot convert value' message.
SchemaUtils.validateVariantFieldSpec requires the default to equal DEFAULT_DIMENSION_NULL_VALUE_OF_BYTES - i.e. exactly the empty array that convert() refuses to produce. So a schema that spells out its (correct, and only allowed) default is rejected, while the identical schema that omits the field is accepted. Special-case the empty array here - it is the documented SQL-null sentinel, deliberately not an envelope, and the class javadoc already says so - or at minimum make the error message say 'VARIANT default null value must be omitted'.
##########
pinot-common/src/main/java/org/apache/pinot/common/utils/DataSchema.java:
##########
@@ -465,6 +473,56 @@ public ColumnDataType getStoredType() {
return _storedColumnDataType;
}
+ public boolean supportsEquality() {
Review Comment:
This is the mechanism behind my next comment: every capability predicate here returns false for arrays and OBJECT (and MAP for pattern matching), not just for VARIANT. Everything that consults them therefore widens beyond VARIANT.
The delegation to toCapabilityDataType() is clean and the array/OBJECT short-circuits are defensible on their own terms - hashing a Java array by identity was never correct. The issue is only that the new callers are query-path guards that previously admitted these types. See TypeCapabilityValidationVisitor.visitSetOp.
##########
pinot-core/src/main/java/org/apache/pinot/core/query/utils/OrderByComparatorFactory.java:
##########
@@ -51,6 +52,12 @@ public static Comparator<Object[]> getComparator(List<OrderByExpressionContext>
throw new BadQueryRequestException("MV expression: " + orderByExpressions.get(i)
+ " should not be included in the ORDER-BY clause");
}
+ FieldSpec.DataType dataType = orderByColumnContexts[i].getDataType();
+ if (!dataType.supportsOrdering()) {
Review Comment:
Same widening on the single-stage side, and here UNKNOWN is the reachable one: DataType.UNKNOWN.supportsOrdering() is false, so any ORDER BY over a NULL-typed expression now fails with BadQueryRequestException.
A literal null expression gets DataType.UNKNOWN from its ColumnContext, so a generated 'ORDER BY NULL' (not exotic in BI-tool SQL) is the concrete risk. Please verify it still works and add a test either way. If UNKNOWN should stay permissive - and I think it should, since ordering nulls is well defined - gate this on == DataType.VARIANT rather than on !supportsOrdering(), or add UNKNOWN to supportsOrdering()'s true set. The same question applies to PredicateEvaluatorProvider, where RANGE now requires supportsOrdering() and would reject an UNKNOWN-typed data source.
##########
pinot-core/src/test/java/org/apache/pinot/core/query/utils/OrderByComparatorFactoryTest.java:
##########
@@ -104,4 +109,32 @@ public void testTwoNullsCompareNextColumn() {
assertEquals(extractColumn(_rows, COLUMN2_INDEX), Arrays.asList(1, 2, 3));
}
+
+ @Test
+ public void testRejectsRawVariant() {
Review Comment:
Test gap that mirrors the risk exactly: the new tests cover VARIANT and MAP, but MAP can barely reach an ORDER BY in practice while UNKNOWN can (ORDER BY on a null literal) - and UNKNOWN is untested.
Please add a third case with DataType.UNKNOWN asserting the intended behaviour. If the answer is 'UNKNOWN must still be orderable', that test will fail today and shows the guard needs narrowing to VARIANT.
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/operands/FilterOperand.java:
##########
@@ -193,7 +197,12 @@ public Predicate(List<RexExpression> operands, DataSchema dataSchema, IntPredica
ColumnDataType lhsType = _lhs.getResultType();
ColumnDataType rhsType = _rhs.getResultType();
- if (lhsType == rhsType) {
+ // Reject raw VARIANT operands only; other non-orderable types (OBJECT, arrays, MAP) keep their existing
+ // best-effort comparison behavior. VARIANT is opaque because its PVAR byte encoding is not a canonical
+ // semantic ordering, so a comparison must extract a typed scalar first.
+ Preconditions.checkArgument(lhsType != ColumnDataType.VARIANT && rhsType != ColumnDataType.VARIANT,
+ "Raw VARIANT values do not support comparison; extract a typed path with variantGet first");
+ if (lhsType == ColumnDataType.UNKNOWN || rhsType == ColumnDataType.UNKNOWN || lhsType == rhsType) {
Review Comment:
Unrelated to VARIANT but semantically real: this short-circuits comparison casting whenever either side is UNKNOWN, changing NULL-literal comparison behaviour for all types. BinaryOperatorTransformFunction:116 does the equivalent on the single-stage side via _alwaysNull.
Both look like genuine fixes - previously 'NULL = intCol' computed a common cast type and could fail on an UNKNOWN stored type - and both are tested (FilterOperandTest.testComparisonWithNullLiteral, BinaryOperatorTransformFunctionTest.testLeftNullRightLiteral), which is good. The only gap is disclosure: these are SQL null-comparison semantics changes to shared code, landing inside a VARIANT feature PR under a commit message ('Preserve SQL null comparison semantics') that reads like a no-op restoration. Please call them out in the PR description so the release notes pick them up.
##########
pinot-common/src/main/java/org/apache/pinot/common/function/TransformFunctionType.java:
##########
@@ -117,6 +122,17 @@ public enum TransformFunctionType {
JSON_EXTRACT_KEY("jsonExtractKey", ReturnTypes.TO_ARRAY,
OperandTypes.family(
List.of(SqlTypeFamily.CHARACTER, SqlTypeFamily.CHARACTER, SqlTypeFamily.CHARACTER), i -> i > 1)),
+ VARIANT_GET("variantGet", TransformFunctionType::variantGetReturnTypeInference, variantGetOperandTypeChecker()),
+ TRY_VARIANT_GET("tryVariantGet", TransformFunctionType::variantGetReturnTypeInference,
+ variantGetOperandTypeChecker()),
+ VARIANT_EXISTS("variantExists", ReturnTypes.BOOLEAN_NULLABLE, variantPathOperandTypeChecker()),
+ IS_VARIANT_NULL("isVariantNull", ReturnTypes.BOOLEAN, optionalVariantPathOperandTypeChecker()),
+ VARIANT_TYPE_OF("variantTypeOf", ReturnTypes.VARCHAR_2000_NULLABLE, optionalVariantPathOperandTypeChecker()),
+ VARIANT_TO_JSON("variantToJson", ReturnTypes.VARCHAR_2000_NULLABLE, OperandTypes.ANY),
+ PARSE_JSON_TO_VARIANT("parseJson", TransformFunctionType::nullableVariantReturnTypeInference,
Review Comment:
Naming: parseJson is a very generic name to claim for a function that returns VARIANT rather than Pinot's JSON type, and it sits directly among jsonExtractScalar / jsonExtractKey / jsonFormat, which all mean Pinot JSON.
Nothing named parseJson exists today so there is no conflict, and the Spark precedent (parse_json -> variant) is a fair argument. But a user reading the function list will reasonably expect parseJson to produce the JSON type, and the alias parseJsonToVariant is the self-documenting spelling. Suggest making parseJsonToVariant (and tryParseJsonToVariant) the primary names and parseJson / tryParseJson the aliases, so the generic name is not the one people find first and the return type is obvious at the call site. Also, unrelated but on this block: VARIANT_TO_JSON uses OperandTypes.ANY, so variantToJson(intCol) passes planning and only fails at runtime - worth a real operand checker.
##########
pinot-common/pom.xml:
##########
@@ -200,6 +200,18 @@
<groupId>org.apache.pinot</groupId>
<artifactId>pinot-timeseries-spi</artifactId>
</dependency>
+ <dependency>
+ <groupId>org.apache.parquet</groupId>
+ <artifactId>parquet-variant</artifactId>
Review Comment:
The parquet-column exclusion is unsound: VariantBuilder, the class parseJsonToVariant() instantiates, has a member referencing org.apache.parquet.io.api.Binary, which lives in the excluded artifact. Drop the exclusion - and note that the decoder half of the library is provably clean, so a narrower boundary is available if the 3.3 MB matters.
Verified against parquet-variant 1.18.0. Of the classes Pinot touches, exactly one is 'dirty': VariantBuilder has `void appendAsString(org.apache.parquet.io.api.Binary)`, and Binary ships in parquet-column, which this exclusion removes. Everything on the decode side - Variant, Variant$Type, VariantUtil, Metadata, ImmutableMetadata, MetadataBuilder - and also VariantObjectBuilder and VariantArrayBuilder have zero references outside parquet-common. It works today only because the JVM resolves method descriptors lazily and appendJsonValue() calls only appendString/appendInt/appendLong/appendDecimal/appendBoolean/appendNull/startObject/startArray; it would become a runtime NoClassDefFoundError, with no build-time signal, if anything reflected over VariantBuilder's declared methods or a future parquet release routed a called method through appendAsString.
Recommendation: drop the exclusion. You cannot exclude an artifact you compile against, and a lazily-unresolved descriptor is not a safety property to rely on. The cost to be aware of is that this is exactly what the exclusion was for - parquet-column is ~3.3 MB and pulls parquet-generator at compile scope, so pinot-common's closure grows by roughly that (worth a `mvn dependency:tree -pl pinot-common` in the PR to state the real number).
If that cost is unwelcome, there is a narrower boundary that keeps the original intent AND is sound, because the dirty/clean split falls almost exactly where the exclusion was aiming: only the JSON-to-Variant construction path (parseJsonToVariant / tryParseJsonToVariant, i.e. the VariantBuilder hierarchy) needs parquet-column. Move just that pair out of pinot-common - its consumers are ingestion (InbuiltFunctionEvaluator) and the MSE literal-folding path (LiteralParseJsonOperand), neither of which is pinot-common-only - and pinot-common retains only parquet-common-clean classes, at which point the exclusion becomes correct rather than accidental.
Two things to fix regardless of which route you take. (a) parquet-jackson is declared at runtime scope and is NOT excluded - a 2.4 MB shaded-Jackson jar in pinot-common's closure for no benefit, since Pinot uses its own Jackson. One-line win. (b) The reason a pinot-common dependency hurts disproportionately today is the global shade, which as apache/pinot#18459 argues buys us no real isolation in return. Worth a comment right here recording that, e.g. 'Dependencies added to pinot-common are amplified by the global shade; see apache/pinot#18459 (Replace global shading with classloader-realm isolation for plugins), which removes it - pushing that through reduces the blast radius of additions like this one.' That keeps the tradeoff visible without blocking this PR on an unrelated build change.
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/HashJoinOperator.java:
##########
@@ -79,18 +99,55 @@ public HashJoinOperator(OpChainExecutionContext context, MultiStageOperator left
_nullKeyRightRows = needUnmatchedRightRows() ? new ArrayList<>() : null;
}
- /// Constructor that takes the schema for NonEquiEvaluator as an argument
+ /// Constructor that takes the schema for NonEquiEvaluator as an argument.
+ ///
+ /// <p>For SEMI and ANTI joins whose node does not carry its inputs, the result schema contains only left columns, so
+ /// this legacy constructor cannot validate the right key's logical type. New callers that need right-side VARIANT
+ /// validation must use the overload that accepts {@code rightSchema}.
public HashJoinOperator(OpChainExecutionContext context, MultiStageOperator leftInput, DataSchema leftSchema,
MultiStageOperator rightInput, JoinNode node, DataSchema nonEquiEvaluationSchema) {
+ this(context, leftInput, leftSchema, rightInput, tryInferRightSchema(leftSchema, node), node,
+ nonEquiEvaluationSchema, false);
+ }
+
+ /// Constructor that takes the schema for NonEquiEvaluator as an argument
+ public HashJoinOperator(OpChainExecutionContext context, MultiStageOperator leftInput, DataSchema leftSchema,
+ MultiStageOperator rightInput, DataSchema rightSchema, JoinNode node, DataSchema nonEquiEvaluationSchema) {
+ this(context, leftInput, leftSchema, rightInput, rightSchema, node, nonEquiEvaluationSchema, true);
+ }
+
+ private HashJoinOperator(OpChainExecutionContext context, MultiStageOperator leftInput, DataSchema leftSchema,
+ MultiStageOperator rightInput, @Nullable DataSchema rightSchema, JoinNode node,
+ DataSchema nonEquiEvaluationSchema, boolean rightSchemaRequired) {
super(context, leftInput, leftSchema, rightInput, node, nonEquiEvaluationSchema);
List<Integer> leftKeys = node.getLeftKeys();
Preconditions.checkState(!leftKeys.isEmpty(), "Hash join operator requires join keys");
+ Preconditions.checkArgument(!rightSchemaRequired || rightSchema != null, "Right input schema must not be null");
+ JoinKeyTypeValidator.validate(node, leftSchema, rightSchema);
_leftKeySelector = KeySelectorFactory.getKeySelector(leftKeys);
_rightKeySelector = KeySelectorFactory.getKeySelector(node.getRightKeys());
_rightTable = createLookupTable(leftKeys, leftSchema);
_matchedRightRows = needUnmatchedRightRows() ? new HashMap<>() : null;
}
+ @Nullable
+ private static DataSchema tryInferRightSchema(DataSchema leftSchema, JoinNode node) {
Review Comment:
tryInferRightSchema reconstructs the right schema by slicing the result schema, and returns null for SEMI/ANTI joins - where JoinKeyTypeValidator.validate then validates nothing. The result is a guard that looks present but silently does not apply on exactly the paths the javadoc admits it cannot cover.
The javadoc is honest, which I appreciate, but the outcome is four overlapping constructors where two enforce the contract and two do not, and a raw VARIANT right-hand key in a SEMI or ANTI join reaches the lookup table with no check. Since the planner-level TypeCapabilityValidationVisitor should normally catch it first, this is defence-in-depth rather than the only line - but then the defence has a documented hole. Two options: deprecate the two legacy constructors and fix the in-repo call sites to pass rightSchema explicitly (they appear to be the only callers), or make the null-rightSchema case still validate the left keys and log rather than skip entirely. Either way, worth a line in the PR description since the javadoc is currently the only record of the limitation.
##########
compatibility-verifier/compCheck.sh:
##########
@@ -496,14 +498,24 @@ setupControllerVariables
setupBrokerVariables
setupServerVariables
-export JAVA_OPTS="-DControllerPort=${CONTROLLER_PORT} -DBrokerQueryPort=${BROKER_QUERY_PORT} -DServerAdminPort=${SERVER_ADMIN_PORT}"
-
mkdir ${PID_DIR}
mkdir ${LOG_DIR}
oldTargetDir="$workingDir"/oldTargetDir
newTargetDir="$workingDir"/newTargetDir
+oldServerSupportsVariant=false
+oldExpressionsProto="${oldTargetDir}/pinot-common/src/main/proto/expressions.proto"
+if [ -f "${oldExpressionsProto}" ] \
+ && grep -Eq '^[[:space:]]*VARIANT[[:space:]]*=[[:space:]]*24[[:space:]]*;' "${oldExpressionsProto}"; then
Review Comment:
This decides whether to run the Variant wire assertions by grepping the OLD checkout's expressions.proto source text for the literal pattern `VARIANT = 24;`. If that grep ever stops matching for a reason unrelated to capability, the suite quietly skips the Variant checks instead of failing.
To spell out the mechanism: the compat suite builds two Pinot trees (old and new) and needs to know whether the OLD side understands the VARIANT wire type, because the expected outcome differs - an old server must reject the type deterministically, a new one must serve it. This script answers that question by reading the old tree's .proto file as text and setting -Dpinot.compat.oldServerSupportsVariant, which the new runIfSystemProperty gate on BaseOp then uses to enable or skip individual ops.
The fragility is that the answer is derived from source formatting rather than from the built artifact. Reformat the enum, move it to another file, add a comment between the name and the number, renumber it, or wrap the line differently, and the grep returns false. False means 'old side does not support VARIANT', which means the ops gated on the true branch are skipped and the run still reports success - so the failure mode is a green build with the Variant compatibility coverage silently switched off, which is the worst possible outcome for a suite whose entire job is proving the mixed-version contract. It also means the gate silently rots the first time someone touches that proto file for an unrelated reason.
Two better options: derive the capability from the built old artifact (reflect over the generated ColumnDataType enum in the old jar and check for a VARIANT constant - that is the actual thing you care about, and it cannot disagree with reality), or keep the grep but fail loudly if the file exists and the enum block cannot be parsed at all, so 'I could not tell' is distinguishable from 'the answer is no'.
Small separate note on this file: the new old-broker-new-servers phase at line 621 passes "$genNum" without incrementing it, unlike every other phase. That is presumably deliberate so the old broker queries the generation that already exists - but it should be commented, because it will silently collide the day that yaml gains a segmentOp or tableOp.
##########
pinot-common/src/main/java/org/apache/pinot/common/utils/VariantUtils.java:
##########
@@ -0,0 +1,2060 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils;
+
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.sql.Timestamp;
+import java.time.Instant;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.LocalTime;
+import java.time.ZoneOffset;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Locale;
+import java.util.Objects;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.Nullable;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.commons.io.output.StringBuilderWriter;
+import org.apache.parquet.variant.Variant;
+import org.apache.parquet.variant.VariantArrayBuilder;
+import org.apache.parquet.variant.VariantBuilder;
+import org.apache.parquet.variant.VariantObjectBuilder;
+import org.apache.pinot.spi.data.FieldSpec.DataType;
+import org.apache.pinot.spi.utils.ByteArray;
+import org.apache.pinot.spi.utils.UuidUtils;
+import org.apache.pinot.spi.utils.VariantEnvelope;
+
+
+/// Query-side operations for Pinot {@code VARIANT} values.
+///
+/// <p>The utility navigates the Parquet Variant binary representation directly. It never materializes a JSON tree.
+/// Instances are not required, and stateless convenience methods are thread-safe. Overloads that accept a
+/// caller-provided {@link ReusableResult} require that result to be thread-confined and not shared by concurrent calls.
+/// An empty byte array is Pinot's SQL-null placeholder and is never decoded as an envelope.
+public final class VariantUtils {
+ public static final String RAW_VARIANT_REQUIRES_NULL_HANDLING_ERROR =
+ "Raw VARIANT projection requires query null handling to be enabled; set enableNullHandling=true";
+
+ private static final JsonFactory JSON_FACTORY = new JsonFactory();
+ private static final BigDecimal MIN_INT_DECIMAL = BigDecimal.valueOf(Integer.MIN_VALUE);
+ private static final BigDecimal MAX_INT_DECIMAL = BigDecimal.valueOf(Integer.MAX_VALUE);
+ private static final BigDecimal MIN_LONG_DECIMAL = BigDecimal.valueOf(Long.MIN_VALUE);
+ private static final BigDecimal MAX_LONG_DECIMAL = BigDecimal.valueOf(Long.MAX_VALUE);
+ private static final int MAX_JSON_NESTING_DEPTH = 100;
+ private static final int MAX_VARIANT_DECIMAL_PRECISION = 38;
+ private static final int MAX_VARIANT_DECIMAL_SCALE = 38;
+ private static final int MAX_VARIANT_DECIMAL_BYTES = 16;
+ private static final long MICROS_PER_SECOND = TimeUnit.SECONDS.toMicros(1);
+ private static final long NANOS_PER_MICRO = TimeUnit.MICROSECONDS.toNanos(1);
+ private static final long NANOS_PER_DAY = TimeUnit.DAYS.toNanos(1);
+ private static final int VARIANT_BASIC_TYPE_MASK = 0x03;
+ private static final int VARIANT_PRIMITIVE_TYPE_MASK = 0x3F;
+ private static final int VARIANT_PRIMITIVE = 0;
+ private static final int VARIANT_SHORT_STRING = 1;
+ private static final int VARIANT_OBJECT = 2;
+ private static final int VARIANT_ARRAY = 3;
+ private static final int VARIANT_NULL = 0;
+ private static final int VARIANT_TRUE = 1;
+ private static final int VARIANT_FALSE = 2;
+ private static final int VARIANT_INT8 = 3;
+ private static final int VARIANT_INT16 = 4;
+ private static final int VARIANT_INT32 = 5;
+ private static final int VARIANT_INT64 = 6;
+ private static final int VARIANT_DOUBLE = 7;
+ private static final int VARIANT_DECIMAL4 = 8;
+ private static final int VARIANT_DECIMAL8 = 9;
+ private static final int VARIANT_DECIMAL16 = 10;
+ private static final int VARIANT_DATE = 11;
+ private static final int VARIANT_TIMESTAMP_TZ = 12;
+ private static final int VARIANT_TIMESTAMP_NTZ = 13;
+ private static final int VARIANT_FLOAT = 14;
+ private static final int VARIANT_BINARY = 15;
+ private static final int VARIANT_LONG_STRING = 16;
+ private static final int VARIANT_TIME = 17;
+ private static final int VARIANT_TIMESTAMP_NANOS_TZ = 18;
+ private static final int VARIANT_TIMESTAMP_NANOS_NTZ = 19;
+ private static final int VARIANT_UUID = 20;
+ private static final int VARIANT_METADATA_VERSION_MASK = 0x0F;
+ private static final int VARIANT_METADATA_VERSION = 1;
+ private static final int OBJECT_BINARY_SEARCH_THRESHOLD = 32;
+ private static final int INVALID_UTF8_COMPARISON = Integer.MIN_VALUE;
+ private static final VariantPath ROOT_PATH = new VariantPath(new PathElement[0]);
+
+ private VariantUtils() {
+ }
+
+ /// Returns whether a final result containing raw VARIANT values requires query null handling. Without a null bitmap,
+ /// Pinot's reserved empty-byte SQL-null placeholder cannot be distinguished from a logical Variant value.
+ public static boolean requiresNullHandlingForRawVariantResult(DataSchema resultSchema,
+ boolean nullHandlingEnabled) {
+ if (nullHandlingEnabled) {
+ return false;
+ }
+ for (DataSchema.ColumnDataType dataType : resultSchema.getColumnDataTypes()) {
+ if (dataType == DataSchema.ColumnDataType.VARIANT) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /// Statically supported result types for {@code variantGet} and {@code tryVariantGet}.
+ public enum ResultType {
+ BOOLEAN(DataType.BOOLEAN, SqlTypeName.BOOLEAN),
+ INT(DataType.INT, SqlTypeName.INTEGER),
+ LONG(DataType.LONG, SqlTypeName.BIGINT),
+ FLOAT(DataType.FLOAT, SqlTypeName.REAL),
+ DOUBLE(DataType.DOUBLE, SqlTypeName.DOUBLE),
+ BIG_DECIMAL(DataType.BIG_DECIMAL, SqlTypeName.DECIMAL),
+ STRING(DataType.STRING, SqlTypeName.VARCHAR),
+ BYTES(DataType.BYTES, SqlTypeName.VARBINARY),
+ UUID(DataType.UUID, SqlTypeName.UUID),
+ TIMESTAMP(DataType.TIMESTAMP, SqlTypeName.TIMESTAMP),
+ VARIANT(DataType.VARIANT, SqlTypeName.VARIANT),
+ JSON(DataType.JSON, SqlTypeName.VARCHAR);
+
+ private final DataType _dataType;
+ private final SqlTypeName _sqlTypeName;
+
+ ResultType(DataType dataType, SqlTypeName sqlTypeName) {
+ _dataType = dataType;
+ _sqlTypeName = sqlTypeName;
+ }
+
+ public DataType getDataType() {
+ return _dataType;
+ }
+
+ public SqlTypeName getSqlTypeName() {
+ return _sqlTypeName;
+ }
+ }
+
+ /// An immutable, pre-parsed Variant path. The v1 grammar supports {@code $}, dot-separated object fields, and
+ /// non-negative array subscripts.
+ public static final class VariantPath {
+ private final PathElement[] _elements;
+
+ private VariantPath(PathElement[] elements) {
+ _elements = elements;
+ }
+ }
+
+ /// Reusable, unboxed destination for vectorized Variant extraction.
+ ///
+ /// <p>Only the getter corresponding to the requested {@link ResultType} is defined after a successful extraction.
+ /// The instance is mutable and not thread-safe; callers should retain one per transform-function instance. Every
+ /// extraction may replace its state. Each successful byte-valued extraction installs a newly materialized array.
+ /// Values returned as {@code byte[]} or as a {@link ByteArray} may be retained after this result is reused, but they
+ /// are read-only by contract and must be copied before mutation.
+ public static final class ReusableResult {
+ private final Cursor _cursor = new Cursor();
+ private int _intValue;
+ private long _longValue;
+ private float _floatValue;
+ private double _doubleValue;
+ private BigDecimal _bigDecimalValue;
+ private String _stringValue;
+ private byte[] _bytesValue;
+
+ public int getIntValue() {
+ return _intValue;
+ }
+
+ public long getLongValue() {
+ return _longValue;
+ }
+
+ public float getFloatValue() {
+ return _floatValue;
+ }
+
+ public double getDoubleValue() {
+ return _doubleValue;
+ }
+
+ public BigDecimal getBigDecimalValue() {
+ return _bigDecimalValue;
+ }
+
+ public String getStringValue() {
+ return _stringValue;
+ }
+
+ /// Returns the extracted BYTES, VARIANT, or direct 16-byte UUID representation.
+ ///
+ /// <p>The returned array is replaced, but not mutated, by the next byte-valued extraction. It may be retained after
+ /// this result is reused, but must be treated as immutable and copied before mutation.
+ public byte[] getBytesValue() {
+ return _bytesValue;
+ }
+
+ public UUID getUuidValue() {
+ return UuidUtils.toUUID(_bytesValue);
+ }
+
+ /// Materializes the extracted value in the external representation used by scalar functions and ingestion.
+ ///
+ /// <p>For BYTES and VARIANT, the returned {@code byte[]} may be retained after this result is reused. It must be
+ /// treated as immutable and copied before mutation.
+ public Object toExternalValue(ResultType resultType) {
+ switch (resultType) {
+ case BOOLEAN:
+ return _intValue != 0;
+ case INT:
+ return _intValue;
+ case LONG:
+ return _longValue;
+ case FLOAT:
+ return _floatValue;
+ case DOUBLE:
+ return _doubleValue;
+ case BIG_DECIMAL:
+ return _bigDecimalValue;
+ case STRING:
+ case JSON:
+ return _stringValue;
+ case BYTES:
+ case VARIANT:
+ return _bytesValue;
+ case UUID:
+ return UuidUtils.toUUID(_bytesValue);
+ case TIMESTAMP:
+ return new Timestamp(_longValue);
+ default:
+ throw new IllegalStateException("Unhandled Variant target type: " + resultType);
+ }
+ }
+
+ /// Materializes the extracted value in {@link DataSchema}'s internal representation.
+ ///
+ /// <p>TIMESTAMP remains epoch milliseconds and UUID wraps the directly copied 16-byte value, avoiding an
+ /// external-object round trip in the multi-stage engine. For BYTES, UUID, and VARIANT, the returned
+ /// {@link ByteArray} wraps a newly materialized array that may be retained after this result is reused. Neither the
+ /// wrapper nor its array may be mutated; callers must copy the array before mutation.
+ public Object toInternalValue(ResultType resultType) {
+ switch (resultType) {
+ case BOOLEAN:
+ return _intValue;
+ case INT:
+ return _intValue;
+ case LONG:
+ case TIMESTAMP:
+ return _longValue;
+ case FLOAT:
+ return _floatValue;
+ case DOUBLE:
+ return _doubleValue;
+ case BIG_DECIMAL:
+ return _bigDecimalValue;
+ case STRING:
+ case JSON:
+ return _stringValue;
+ case BYTES:
+ case UUID:
+ case VARIANT:
+ return new ByteArray(_bytesValue);
+ default:
+ throw new IllegalStateException("Unhandled Variant target type: " + resultType);
+ }
+ }
+ }
+
+ /// Parses a target type literal once for reuse by a transform function.
+ public static ResultType parseResultType(String targetType) {
+ if (targetType == null) {
+ throw new IllegalArgumentException("Variant target type must not be null");
+ }
+ try {
+ return ResultType.valueOf(targetType.trim().toUpperCase(Locale.ROOT));
+ } catch (IllegalArgumentException e) {
+ throw new IllegalArgumentException("Unsupported Variant target type: " + targetType, e);
+ }
+ }
+
+ /// Compiles a v1 Variant path.
+ public static VariantPath compilePath(String path) {
+ if (path == null || path.isEmpty() || path.charAt(0) != '$') {
+ throw new IllegalArgumentException("Variant path must start with '$': " + path);
+ }
+ List<PathElement> elements = new ArrayList<>();
+ int index = 1;
+ while (index < path.length()) {
+ char current = path.charAt(index);
+ if (current == '.') {
+ int fieldStart = ++index;
+ while (index < path.length()) {
+ char next = path.charAt(index);
+ if (next == '.' || next == '[') {
+ break;
+ }
+ index++;
+ }
+ if (fieldStart == index) {
+ throw new IllegalArgumentException("Variant path contains an empty field: " + path);
+ }
+ elements.add(PathElement.forField(path.substring(fieldStart, index)));
+ } else if (current == '[') {
+ int subscriptStart = ++index;
+ while (index < path.length() && Character.isDigit(path.charAt(index))) {
+ index++;
+ }
+ if (subscriptStart == index || index >= path.length() || path.charAt(index) != ']') {
+ throw new IllegalArgumentException("Invalid Variant array subscript in path: " + path);
+ }
+ try {
+ elements.add(PathElement.forIndex(Integer.parseInt(path.substring(subscriptStart, index))));
+ } catch (NumberFormatException e) {
+ throw new IllegalArgumentException("Variant array subscript is too large in path: " + path, e);
+ }
+ index++;
+ } else {
+ throw new IllegalArgumentException("Unexpected character at offset " + index + " in Variant path: " + path);
+ }
+ }
+ return new VariantPath(elements.toArray(new PathElement[0]));
+ }
+
+ /// Extracts a Variant value. A missing path or SQL null returns Java null; a Variant null remains an encoded Variant
+ /// value.
+ @Nullable
+ public static byte[] variantGet(@Nullable byte[] envelope, String path) {
+ return (byte[]) variantGet(envelope, compilePath(path), ResultType.VARIANT);
+ }
+
+ /// Strictly extracts and converts a value. A missing path or SQL null returns Java null. A Variant null remains
+ /// encoded when the target type is {@link ResultType#VARIANT}, and returns Java null for other target types. An
+ /// incompatible non-null value throws.
+ @Nullable
+ public static Object variantGet(@Nullable byte[] envelope, String path, String targetType) {
+ return variantGet(envelope, compilePath(path), parseResultType(targetType));
+ }
+
+ /// Strictly extracts using pre-parsed path and type values.
+ @Nullable
+ public static Object variantGet(@Nullable byte[] envelope, VariantPath path, ResultType targetType) {
+ ReusableResult result = new ReusableResult();
+ return extractInto(envelope, path, targetType, result) ? result.toExternalValue(targetType) : null;
+ }
+
+ /// Strictly extracts into a reusable, unboxed result.
+ ///
+ /// @return {@code false} for SQL null, a missing path, or Variant null converted to a non-Variant target
+ public static boolean extractInto(@Nullable byte[] envelope, VariantPath path, ResultType targetType,
+ ReusableResult result) {
+ Objects.requireNonNull(result, "result must not be null");
+ if (isSqlNull(envelope)) {
+ return false;
+ }
+ Objects.requireNonNull(path, "path must not be null");
+ Objects.requireNonNull(targetType, "targetType must not be null");
+ Cursor cursor = result._cursor;
+ if (!cursor.navigate(envelope, path)) {
+ return false;
+ }
+ if (cursor.getType() == Variant.Type.NULL && targetType != ResultType.VARIANT) {
+ return false;
+ }
+ convert(cursor, targetType, result);
+ return true;
+ }
+
+ /// Tolerant Variant extraction. Malformed input returns Java null.
+ @Nullable
+ public static byte[] tryVariantGet(@Nullable byte[] envelope, String path) {
+ return (byte[]) tryVariantGet(envelope, compilePath(path), ResultType.VARIANT);
+ }
+
+ /// Tolerant typed extraction. Malformed input and incompatible types return Java null.
+ @Nullable
+ public static Object tryVariantGet(@Nullable byte[] envelope, String path, String targetType) {
+ try {
+ return tryVariantGet(envelope, compilePath(path), parseResultType(targetType));
+ } catch (RuntimeException e) {
+ return null;
+ }
+ }
+
+ /// Tolerant extraction using pre-parsed path and type values.
+ @Nullable
+ public static Object tryVariantGet(@Nullable byte[] envelope, VariantPath path, ResultType targetType) {
+ try {
+ ReusableResult result = new ReusableResult();
+ return tryExtractInto(envelope, path, targetType, result) ? result.toExternalValue(targetType) : null;
+ } catch (RuntimeException e) {
+ return null;
+ }
+ }
+
+ /// Tolerantly extracts into a reusable, unboxed result.
+ ///
+ /// @return {@code false} for SQL null, missing paths, Variant null converted to a non-Variant target,
+ /// malformed input, or an incompatible conversion
+ public static boolean tryExtractInto(@Nullable byte[] envelope, VariantPath path, ResultType targetType,
+ ReusableResult result) {
+ Objects.requireNonNull(result, "result must not be null");
+ if (isSqlNull(envelope)) {
+ return false;
+ }
+ Objects.requireNonNull(path, "path must not be null");
+ Objects.requireNonNull(targetType, "targetType must not be null");
+ Cursor cursor = result._cursor;
+ try {
+ if (!cursor.navigate(envelope, path)) {
+ return false;
+ }
+ if (cursor.getType() == Variant.Type.NULL && targetType != ResultType.VARIANT) {
+ return false;
+ }
+ return tryConvert(cursor, targetType, result);
+ } catch (IllegalArgumentException | IllegalStateException | UnsupportedOperationException
+ | IndexOutOfBoundsException e) {
+ // Cursor operations use these exceptions only for malformed or unsupported Variant encodings.
+ return false;
+ }
+ }
+
+ /// Returns whether the path is present. A present Variant null counts as present.
+ @Nullable
+ public static Boolean variantExists(@Nullable byte[] envelope, String path) {
+ return variantExists(envelope, compilePath(path));
+ }
+
+ /// Returns whether a compiled path is present. A present Variant null counts as present.
+ @Nullable
+ public static Boolean variantExists(@Nullable byte[] envelope, VariantPath path) {
+ return variantExists(envelope, path, new ReusableResult());
+ }
+
+ /// Allocation-free compiled-path form of {@link #variantExists(byte[], VariantPath)} when the caller retains the
+ /// supplied result between rows.
+ @Nullable
+ public static Boolean variantExists(@Nullable byte[] envelope, VariantPath path, ReusableResult result) {
+ Objects.requireNonNull(result, "result must not be null");
+ if (isSqlNull(envelope)) {
+ return null;
+ }
+ return result._cursor.navigate(envelope, Objects.requireNonNull(path, "path must not be null"));
+ }
+
+ /// Returns whether the root value is a Variant null. SQL null is not a Variant null.
+ public static boolean isVariantNull(@Nullable byte[] envelope) {
+ return isVariantNull(envelope, ROOT_PATH, new ReusableResult());
+ }
+
+ /// Returns whether a present value at the path is a Variant null. SQL null and missing paths return false.
+ public static boolean isVariantNull(@Nullable byte[] envelope, String path) {
+ return isVariantNull(envelope, compilePath(path));
+ }
+
+ /// Returns whether a present value at a compiled path is a Variant null. SQL null and missing paths return false.
+ public static boolean isVariantNull(@Nullable byte[] envelope, VariantPath path) {
+ return isVariantNull(envelope, path, new ReusableResult());
+ }
+
+ /// Allocation-free compiled-path form of {@link #isVariantNull(byte[], VariantPath)} when the caller retains the
+ /// supplied result between rows.
+ public static boolean isVariantNull(@Nullable byte[] envelope, VariantPath path, ReusableResult result) {
+ Objects.requireNonNull(result, "result must not be null");
+ if (isSqlNull(envelope)) {
+ return false;
+ }
+ Cursor cursor = result._cursor;
+ return cursor.navigate(envelope, Objects.requireNonNull(path, "path must not be null"))
+ && cursor.getType() == Variant.Type.NULL;
+ }
+
+ /// Returns the Variant type name at the root, or Java null for SQL null.
+ @Nullable
+ public static String variantTypeOf(@Nullable byte[] envelope) {
+ return variantTypeOf(envelope, ROOT_PATH, new ReusableResult());
+ }
+
+ /// Returns the Variant type name at a path, or Java null for SQL null or a missing path.
+ @Nullable
+ public static String variantTypeOf(@Nullable byte[] envelope, String path) {
+ return variantTypeOf(envelope, compilePath(path));
+ }
+
+ /// Returns the Variant type name at a compiled path, or Java null for SQL null or a missing path.
+ @Nullable
+ public static String variantTypeOf(@Nullable byte[] envelope, VariantPath path) {
+ return variantTypeOf(envelope, path, new ReusableResult());
+ }
+
+ /// Allocation-free compiled-path form of {@link #variantTypeOf(byte[], VariantPath)} when the caller retains the
+ /// supplied result between rows.
+ @Nullable
+ public static String variantTypeOf(@Nullable byte[] envelope, VariantPath path, ReusableResult result) {
+ Objects.requireNonNull(result, "result must not be null");
+ if (isSqlNull(envelope)) {
+ return null;
+ }
+ Cursor cursor = result._cursor;
+ return cursor.navigate(envelope, Objects.requireNonNull(path, "path must not be null"))
+ ? typeName(cursor.getType()) : null;
+ }
+
+ /// Renders the Variant value as canonical JSON text without constructing a JSON tree.
+ @Nullable
+ public static String variantToJson(@Nullable byte[] envelope) {
+ return variantToJson(envelope, new ReusableResult());
+ }
+
+ /// Allocation-reduced form of [#variantToJson(byte[])] when the caller retains the supplied result between rows.
+ @Nullable
+ public static String variantToJson(@Nullable byte[] envelope, ReusableResult result) {
+ Objects.requireNonNull(result, "result must not be null");
+ if (isSqlNull(envelope)) {
+ return null;
+ }
+ Cursor cursor = result._cursor;
+ cursor.navigate(envelope, ROOT_PATH);
+ return variantToJson(cursor.asVariant());
+ }
+
+ /// Parses JSON text into a Pinot Variant envelope without constructing a JSON tree.
+ @Nullable
+ public static byte[] parseJsonToVariant(@Nullable String json) {
+ if (json == null) {
+ return null;
+ }
+ try (JsonParser parser = JSON_FACTORY.createParser(json)) {
+ JsonToken token = parser.nextToken();
+ if (token == null) {
+ throw new IllegalArgumentException("Cannot parse empty text as Variant");
+ }
+ VariantBuilder builder = new VariantBuilder();
+ appendJsonValue(parser, token, builder, 0);
+ if (parser.nextToken() != null) {
+ throw new IllegalArgumentException("Unexpected trailing token after Variant JSON value");
+ }
+ Variant variant = builder.build();
+ return VariantEnvelope.encode(variant.getMetadataBuffer(), variant.getValueBuffer());
+ } catch (IOException | RuntimeException e) {
+ throw new IllegalArgumentException("Cannot parse JSON as Variant", e);
+ }
+ }
+
+ /// Tolerant JSON parser. Malformed or unsupported input returns Java null.
+ @Nullable
+ public static byte[] tryParseJsonToVariant(@Nullable String json) {
+ try {
+ return parseJsonToVariant(json);
+ } catch (RuntimeException e) {
+ return null;
+ }
+ }
+
+ private static boolean isSqlNull(@Nullable byte[] envelope) {
+ return envelope == null || envelope.length == 0;
+ }
+
+ private static void convert(Cursor value, ResultType targetType, ReusableResult result) {
+ switch (targetType) {
+ case BOOLEAN:
+ requireType(value, Variant.Type.BOOLEAN, targetType);
+ result._intValue = value.getBoolean() ? 1 : 0;
+ break;
+ case INT:
+ result._intValue = toInt(value, targetType);
+ break;
+ case LONG:
+ result._longValue = toLong(value, targetType);
+ break;
+ case FLOAT:
+ result._floatValue = toFloat(value, targetType);
+ break;
+ case DOUBLE:
+ result._doubleValue = toDouble(value, targetType);
+ break;
+ case BIG_DECIMAL:
+ result._bigDecimalValue = toBigDecimal(value, targetType);
+ break;
+ case STRING:
+ requireType(value, Variant.Type.STRING, targetType);
+ result._stringValue = value.getString();
+ break;
+ case BYTES:
+ requireType(value, Variant.Type.BINARY, targetType);
+ result._bytesValue = value.getBinary();
+ break;
+ case UUID:
+ requireType(value, Variant.Type.UUID, targetType);
+ result._bytesValue = value.getUuidBytes();
+ break;
+ case TIMESTAMP:
+ result._longValue = toTimestampMillis(value, targetType);
+ break;
+ case VARIANT:
+ result._bytesValue = value.copyEnvelope();
+ break;
+ case JSON:
+ result._stringValue = variantToJson(value.asVariant());
+ break;
+ default:
+ throw new IllegalStateException("Unhandled Variant target type: " + targetType);
+ }
+ }
+
+ private static boolean tryConvert(Cursor value, ResultType targetType, ReusableResult result) {
+ Variant.Type valueType = value.getType();
+ switch (targetType) {
+ case BOOLEAN:
+ if (valueType != Variant.Type.BOOLEAN) {
+ return false;
+ }
+ result._intValue = value.getBoolean() ? 1 : 0;
+ return true;
+ case INT:
+ return tryConvertToInt(value, valueType, result);
+ case LONG:
+ return tryConvertToLong(value, valueType, result);
+ case FLOAT:
+ switch (valueType) {
+ case BYTE:
+ case SHORT:
+ case INT:
+ case LONG:
+ result._floatValue = value.getInteger();
+ return true;
+ case FLOAT:
+ result._floatValue = value.getFloat();
+ return true;
+ case DOUBLE:
+ result._floatValue = (float) value.getDouble();
+ return true;
+ case DECIMAL4:
+ case DECIMAL8:
+ case DECIMAL16:
+ result._floatValue = value.getDecimal().floatValue();
+ return true;
+ default:
+ return false;
+ }
+ case DOUBLE:
+ switch (valueType) {
+ case BYTE:
+ case SHORT:
+ case INT:
+ case LONG:
+ result._doubleValue = value.getInteger();
+ return true;
+ case FLOAT:
+ result._doubleValue = value.getFloat();
+ return true;
+ case DOUBLE:
+ result._doubleValue = value.getDouble();
+ return true;
+ case DECIMAL4:
+ case DECIMAL8:
+ case DECIMAL16:
+ result._doubleValue = value.getDecimal().doubleValue();
+ return true;
+ default:
+ return false;
+ }
+ case BIG_DECIMAL:
+ switch (valueType) {
+ case BYTE:
+ case SHORT:
+ case INT:
+ case LONG:
+ result._bigDecimalValue = BigDecimal.valueOf(value.getInteger());
+ return true;
+ case FLOAT:
+ float floatValue = value.getFloat();
+ if (!Float.isFinite(floatValue)) {
+ return false;
+ }
+ result._bigDecimalValue = BigDecimal.valueOf(floatValue);
+ return true;
+ case DOUBLE:
+ double doubleValue = value.getDouble();
+ if (!Double.isFinite(doubleValue)) {
+ return false;
+ }
+ result._bigDecimalValue = BigDecimal.valueOf(doubleValue);
+ return true;
+ case DECIMAL4:
+ case DECIMAL8:
+ case DECIMAL16:
+ result._bigDecimalValue = value.getDecimal();
+ return true;
+ default:
+ return false;
+ }
+ case STRING:
+ if (valueType != Variant.Type.STRING) {
+ return false;
+ }
+ result._stringValue = value.getString();
+ return true;
+ case BYTES:
+ if (valueType != Variant.Type.BINARY) {
+ return false;
+ }
+ result._bytesValue = value.getBinary();
+ return true;
+ case UUID:
+ if (valueType != Variant.Type.UUID) {
+ return false;
+ }
+ result._bytesValue = value.getUuidBytes();
+ return true;
+ case TIMESTAMP:
+ switch (valueType) {
+ case DATE:
+ result._longValue = value.getInteger() * TimeUnit.DAYS.toMillis(1);
Review Comment:
Correctness: strict/tolerant asymmetry on DATE -> TIMESTAMP. toTimestampMillis() uses Math.multiplyExact, but this tryConvert path uses a plain multiply - so try_variant_get(v, '$.d', 'TIMESTAMP') silently returns a wrapped garbage timestamp where the strict form correctly throws.
The try_ contract is 'return SQL null on failure', not 'return a wrong answer quietly' - a silently wrapped value is worse than an error because nothing downstream can detect it. The fix is Math.multiplyExact wrapped so overflow returns false, which is exactly the shape tryConvertToInt already uses for LONG out of range a few lines above. An epoch-day value only has to exceed ~106 million to overflow, which a hostile or corrupt producer can trivially emit.
##########
pinot-common/src/main/java/org/apache/pinot/common/utils/VariantUtils.java:
##########
@@ -0,0 +1,2060 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils;
+
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.sql.Timestamp;
+import java.time.Instant;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.LocalTime;
+import java.time.ZoneOffset;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Locale;
+import java.util.Objects;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.Nullable;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.commons.io.output.StringBuilderWriter;
+import org.apache.parquet.variant.Variant;
+import org.apache.parquet.variant.VariantArrayBuilder;
+import org.apache.parquet.variant.VariantBuilder;
+import org.apache.parquet.variant.VariantObjectBuilder;
+import org.apache.pinot.spi.data.FieldSpec.DataType;
+import org.apache.pinot.spi.utils.ByteArray;
+import org.apache.pinot.spi.utils.UuidUtils;
+import org.apache.pinot.spi.utils.VariantEnvelope;
+
+
+/// Query-side operations for Pinot {@code VARIANT} values.
+///
+/// <p>The utility navigates the Parquet Variant binary representation directly. It never materializes a JSON tree.
+/// Instances are not required, and stateless convenience methods are thread-safe. Overloads that accept a
+/// caller-provided {@link ReusableResult} require that result to be thread-confined and not shared by concurrent calls.
+/// An empty byte array is Pinot's SQL-null placeholder and is never decoded as an envelope.
+public final class VariantUtils {
+ public static final String RAW_VARIANT_REQUIRES_NULL_HANDLING_ERROR =
+ "Raw VARIANT projection requires query null handling to be enabled; set enableNullHandling=true";
+
+ private static final JsonFactory JSON_FACTORY = new JsonFactory();
+ private static final BigDecimal MIN_INT_DECIMAL = BigDecimal.valueOf(Integer.MIN_VALUE);
+ private static final BigDecimal MAX_INT_DECIMAL = BigDecimal.valueOf(Integer.MAX_VALUE);
+ private static final BigDecimal MIN_LONG_DECIMAL = BigDecimal.valueOf(Long.MIN_VALUE);
+ private static final BigDecimal MAX_LONG_DECIMAL = BigDecimal.valueOf(Long.MAX_VALUE);
+ private static final int MAX_JSON_NESTING_DEPTH = 100;
+ private static final int MAX_VARIANT_DECIMAL_PRECISION = 38;
+ private static final int MAX_VARIANT_DECIMAL_SCALE = 38;
+ private static final int MAX_VARIANT_DECIMAL_BYTES = 16;
+ private static final long MICROS_PER_SECOND = TimeUnit.SECONDS.toMicros(1);
+ private static final long NANOS_PER_MICRO = TimeUnit.MICROSECONDS.toNanos(1);
+ private static final long NANOS_PER_DAY = TimeUnit.DAYS.toNanos(1);
+ private static final int VARIANT_BASIC_TYPE_MASK = 0x03;
+ private static final int VARIANT_PRIMITIVE_TYPE_MASK = 0x3F;
+ private static final int VARIANT_PRIMITIVE = 0;
+ private static final int VARIANT_SHORT_STRING = 1;
+ private static final int VARIANT_OBJECT = 2;
+ private static final int VARIANT_ARRAY = 3;
+ private static final int VARIANT_NULL = 0;
+ private static final int VARIANT_TRUE = 1;
+ private static final int VARIANT_FALSE = 2;
+ private static final int VARIANT_INT8 = 3;
+ private static final int VARIANT_INT16 = 4;
+ private static final int VARIANT_INT32 = 5;
+ private static final int VARIANT_INT64 = 6;
+ private static final int VARIANT_DOUBLE = 7;
+ private static final int VARIANT_DECIMAL4 = 8;
+ private static final int VARIANT_DECIMAL8 = 9;
+ private static final int VARIANT_DECIMAL16 = 10;
+ private static final int VARIANT_DATE = 11;
+ private static final int VARIANT_TIMESTAMP_TZ = 12;
+ private static final int VARIANT_TIMESTAMP_NTZ = 13;
+ private static final int VARIANT_FLOAT = 14;
+ private static final int VARIANT_BINARY = 15;
+ private static final int VARIANT_LONG_STRING = 16;
+ private static final int VARIANT_TIME = 17;
+ private static final int VARIANT_TIMESTAMP_NANOS_TZ = 18;
+ private static final int VARIANT_TIMESTAMP_NANOS_NTZ = 19;
+ private static final int VARIANT_UUID = 20;
+ private static final int VARIANT_METADATA_VERSION_MASK = 0x0F;
+ private static final int VARIANT_METADATA_VERSION = 1;
+ private static final int OBJECT_BINARY_SEARCH_THRESHOLD = 32;
+ private static final int INVALID_UTF8_COMPARISON = Integer.MIN_VALUE;
+ private static final VariantPath ROOT_PATH = new VariantPath(new PathElement[0]);
+
+ private VariantUtils() {
+ }
+
+ /// Returns whether a final result containing raw VARIANT values requires query null handling. Without a null bitmap,
+ /// Pinot's reserved empty-byte SQL-null placeholder cannot be distinguished from a logical Variant value.
+ public static boolean requiresNullHandlingForRawVariantResult(DataSchema resultSchema,
+ boolean nullHandlingEnabled) {
+ if (nullHandlingEnabled) {
+ return false;
+ }
+ for (DataSchema.ColumnDataType dataType : resultSchema.getColumnDataTypes()) {
+ if (dataType == DataSchema.ColumnDataType.VARIANT) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /// Statically supported result types for {@code variantGet} and {@code tryVariantGet}.
+ public enum ResultType {
+ BOOLEAN(DataType.BOOLEAN, SqlTypeName.BOOLEAN),
+ INT(DataType.INT, SqlTypeName.INTEGER),
+ LONG(DataType.LONG, SqlTypeName.BIGINT),
+ FLOAT(DataType.FLOAT, SqlTypeName.REAL),
+ DOUBLE(DataType.DOUBLE, SqlTypeName.DOUBLE),
+ BIG_DECIMAL(DataType.BIG_DECIMAL, SqlTypeName.DECIMAL),
+ STRING(DataType.STRING, SqlTypeName.VARCHAR),
+ BYTES(DataType.BYTES, SqlTypeName.VARBINARY),
+ UUID(DataType.UUID, SqlTypeName.UUID),
+ TIMESTAMP(DataType.TIMESTAMP, SqlTypeName.TIMESTAMP),
+ VARIANT(DataType.VARIANT, SqlTypeName.VARIANT),
+ JSON(DataType.JSON, SqlTypeName.VARCHAR);
+
+ private final DataType _dataType;
+ private final SqlTypeName _sqlTypeName;
+
+ ResultType(DataType dataType, SqlTypeName sqlTypeName) {
+ _dataType = dataType;
+ _sqlTypeName = sqlTypeName;
+ }
+
+ public DataType getDataType() {
+ return _dataType;
+ }
+
+ public SqlTypeName getSqlTypeName() {
+ return _sqlTypeName;
+ }
+ }
+
+ /// An immutable, pre-parsed Variant path. The v1 grammar supports {@code $}, dot-separated object fields, and
+ /// non-negative array subscripts.
+ public static final class VariantPath {
+ private final PathElement[] _elements;
+
+ private VariantPath(PathElement[] elements) {
+ _elements = elements;
+ }
+ }
+
+ /// Reusable, unboxed destination for vectorized Variant extraction.
+ ///
+ /// <p>Only the getter corresponding to the requested {@link ResultType} is defined after a successful extraction.
+ /// The instance is mutable and not thread-safe; callers should retain one per transform-function instance. Every
+ /// extraction may replace its state. Each successful byte-valued extraction installs a newly materialized array.
+ /// Values returned as {@code byte[]} or as a {@link ByteArray} may be retained after this result is reused, but they
+ /// are read-only by contract and must be copied before mutation.
+ public static final class ReusableResult {
+ private final Cursor _cursor = new Cursor();
+ private int _intValue;
+ private long _longValue;
+ private float _floatValue;
+ private double _doubleValue;
+ private BigDecimal _bigDecimalValue;
+ private String _stringValue;
+ private byte[] _bytesValue;
+
+ public int getIntValue() {
+ return _intValue;
+ }
+
+ public long getLongValue() {
+ return _longValue;
+ }
+
+ public float getFloatValue() {
+ return _floatValue;
+ }
+
+ public double getDoubleValue() {
+ return _doubleValue;
+ }
+
+ public BigDecimal getBigDecimalValue() {
+ return _bigDecimalValue;
+ }
+
+ public String getStringValue() {
+ return _stringValue;
+ }
+
+ /// Returns the extracted BYTES, VARIANT, or direct 16-byte UUID representation.
+ ///
+ /// <p>The returned array is replaced, but not mutated, by the next byte-valued extraction. It may be retained after
+ /// this result is reused, but must be treated as immutable and copied before mutation.
+ public byte[] getBytesValue() {
+ return _bytesValue;
+ }
+
+ public UUID getUuidValue() {
+ return UuidUtils.toUUID(_bytesValue);
+ }
+
+ /// Materializes the extracted value in the external representation used by scalar functions and ingestion.
+ ///
+ /// <p>For BYTES and VARIANT, the returned {@code byte[]} may be retained after this result is reused. It must be
+ /// treated as immutable and copied before mutation.
+ public Object toExternalValue(ResultType resultType) {
+ switch (resultType) {
+ case BOOLEAN:
+ return _intValue != 0;
+ case INT:
+ return _intValue;
+ case LONG:
+ return _longValue;
+ case FLOAT:
+ return _floatValue;
+ case DOUBLE:
+ return _doubleValue;
+ case BIG_DECIMAL:
+ return _bigDecimalValue;
+ case STRING:
+ case JSON:
+ return _stringValue;
+ case BYTES:
+ case VARIANT:
+ return _bytesValue;
+ case UUID:
+ return UuidUtils.toUUID(_bytesValue);
+ case TIMESTAMP:
+ return new Timestamp(_longValue);
+ default:
+ throw new IllegalStateException("Unhandled Variant target type: " + resultType);
+ }
+ }
+
+ /// Materializes the extracted value in {@link DataSchema}'s internal representation.
+ ///
+ /// <p>TIMESTAMP remains epoch milliseconds and UUID wraps the directly copied 16-byte value, avoiding an
+ /// external-object round trip in the multi-stage engine. For BYTES, UUID, and VARIANT, the returned
+ /// {@link ByteArray} wraps a newly materialized array that may be retained after this result is reused. Neither the
+ /// wrapper nor its array may be mutated; callers must copy the array before mutation.
+ public Object toInternalValue(ResultType resultType) {
+ switch (resultType) {
+ case BOOLEAN:
+ return _intValue;
+ case INT:
+ return _intValue;
+ case LONG:
+ case TIMESTAMP:
+ return _longValue;
+ case FLOAT:
+ return _floatValue;
+ case DOUBLE:
+ return _doubleValue;
+ case BIG_DECIMAL:
+ return _bigDecimalValue;
+ case STRING:
+ case JSON:
+ return _stringValue;
+ case BYTES:
+ case UUID:
+ case VARIANT:
+ return new ByteArray(_bytesValue);
+ default:
+ throw new IllegalStateException("Unhandled Variant target type: " + resultType);
+ }
+ }
+ }
+
+ /// Parses a target type literal once for reuse by a transform function.
+ public static ResultType parseResultType(String targetType) {
+ if (targetType == null) {
+ throw new IllegalArgumentException("Variant target type must not be null");
+ }
+ try {
+ return ResultType.valueOf(targetType.trim().toUpperCase(Locale.ROOT));
+ } catch (IllegalArgumentException e) {
+ throw new IllegalArgumentException("Unsupported Variant target type: " + targetType, e);
+ }
+ }
+
+ /// Compiles a v1 Variant path.
+ public static VariantPath compilePath(String path) {
+ if (path == null || path.isEmpty() || path.charAt(0) != '$') {
+ throw new IllegalArgumentException("Variant path must start with '$': " + path);
+ }
+ List<PathElement> elements = new ArrayList<>();
+ int index = 1;
+ while (index < path.length()) {
+ char current = path.charAt(index);
+ if (current == '.') {
+ int fieldStart = ++index;
+ while (index < path.length()) {
+ char next = path.charAt(index);
+ if (next == '.' || next == '[') {
+ break;
+ }
+ index++;
+ }
+ if (fieldStart == index) {
+ throw new IllegalArgumentException("Variant path contains an empty field: " + path);
+ }
+ elements.add(PathElement.forField(path.substring(fieldStart, index)));
+ } else if (current == '[') {
+ int subscriptStart = ++index;
+ while (index < path.length() && Character.isDigit(path.charAt(index))) {
+ index++;
+ }
+ if (subscriptStart == index || index >= path.length() || path.charAt(index) != ']') {
+ throw new IllegalArgumentException("Invalid Variant array subscript in path: " + path);
+ }
+ try {
+ elements.add(PathElement.forIndex(Integer.parseInt(path.substring(subscriptStart, index))));
+ } catch (NumberFormatException e) {
+ throw new IllegalArgumentException("Variant array subscript is too large in path: " + path, e);
+ }
+ index++;
+ } else {
+ throw new IllegalArgumentException("Unexpected character at offset " + index + " in Variant path: " + path);
+ }
+ }
+ return new VariantPath(elements.toArray(new PathElement[0]));
+ }
+
+ /// Extracts a Variant value. A missing path or SQL null returns Java null; a Variant null remains an encoded Variant
+ /// value.
+ @Nullable
+ public static byte[] variantGet(@Nullable byte[] envelope, String path) {
+ return (byte[]) variantGet(envelope, compilePath(path), ResultType.VARIANT);
+ }
+
+ /// Strictly extracts and converts a value. A missing path or SQL null returns Java null. A Variant null remains
+ /// encoded when the target type is {@link ResultType#VARIANT}, and returns Java null for other target types. An
+ /// incompatible non-null value throws.
+ @Nullable
+ public static Object variantGet(@Nullable byte[] envelope, String path, String targetType) {
+ return variantGet(envelope, compilePath(path), parseResultType(targetType));
+ }
+
+ /// Strictly extracts using pre-parsed path and type values.
+ @Nullable
+ public static Object variantGet(@Nullable byte[] envelope, VariantPath path, ResultType targetType) {
+ ReusableResult result = new ReusableResult();
+ return extractInto(envelope, path, targetType, result) ? result.toExternalValue(targetType) : null;
+ }
+
+ /// Strictly extracts into a reusable, unboxed result.
+ ///
+ /// @return {@code false} for SQL null, a missing path, or Variant null converted to a non-Variant target
+ public static boolean extractInto(@Nullable byte[] envelope, VariantPath path, ResultType targetType,
+ ReusableResult result) {
+ Objects.requireNonNull(result, "result must not be null");
+ if (isSqlNull(envelope)) {
+ return false;
+ }
+ Objects.requireNonNull(path, "path must not be null");
+ Objects.requireNonNull(targetType, "targetType must not be null");
+ Cursor cursor = result._cursor;
+ if (!cursor.navigate(envelope, path)) {
+ return false;
+ }
+ if (cursor.getType() == Variant.Type.NULL && targetType != ResultType.VARIANT) {
+ return false;
+ }
+ convert(cursor, targetType, result);
+ return true;
+ }
+
+ /// Tolerant Variant extraction. Malformed input returns Java null.
+ @Nullable
+ public static byte[] tryVariantGet(@Nullable byte[] envelope, String path) {
+ return (byte[]) tryVariantGet(envelope, compilePath(path), ResultType.VARIANT);
+ }
+
+ /// Tolerant typed extraction. Malformed input and incompatible types return Java null.
+ @Nullable
+ public static Object tryVariantGet(@Nullable byte[] envelope, String path, String targetType) {
+ try {
+ return tryVariantGet(envelope, compilePath(path), parseResultType(targetType));
+ } catch (RuntimeException e) {
+ return null;
+ }
+ }
+
+ /// Tolerant extraction using pre-parsed path and type values.
+ @Nullable
+ public static Object tryVariantGet(@Nullable byte[] envelope, VariantPath path, ResultType targetType) {
+ try {
+ ReusableResult result = new ReusableResult();
+ return tryExtractInto(envelope, path, targetType, result) ? result.toExternalValue(targetType) : null;
+ } catch (RuntimeException e) {
+ return null;
+ }
+ }
+
+ /// Tolerantly extracts into a reusable, unboxed result.
+ ///
+ /// @return {@code false} for SQL null, missing paths, Variant null converted to a non-Variant target,
+ /// malformed input, or an incompatible conversion
+ public static boolean tryExtractInto(@Nullable byte[] envelope, VariantPath path, ResultType targetType,
+ ReusableResult result) {
+ Objects.requireNonNull(result, "result must not be null");
+ if (isSqlNull(envelope)) {
+ return false;
+ }
+ Objects.requireNonNull(path, "path must not be null");
+ Objects.requireNonNull(targetType, "targetType must not be null");
+ Cursor cursor = result._cursor;
+ try {
+ if (!cursor.navigate(envelope, path)) {
+ return false;
+ }
+ if (cursor.getType() == Variant.Type.NULL && targetType != ResultType.VARIANT) {
+ return false;
+ }
+ return tryConvert(cursor, targetType, result);
+ } catch (IllegalArgumentException | IllegalStateException | UnsupportedOperationException
+ | IndexOutOfBoundsException e) {
+ // Cursor operations use these exceptions only for malformed or unsupported Variant encodings.
+ return false;
+ }
+ }
+
+ /// Returns whether the path is present. A present Variant null counts as present.
+ @Nullable
+ public static Boolean variantExists(@Nullable byte[] envelope, String path) {
+ return variantExists(envelope, compilePath(path));
+ }
+
+ /// Returns whether a compiled path is present. A present Variant null counts as present.
+ @Nullable
+ public static Boolean variantExists(@Nullable byte[] envelope, VariantPath path) {
+ return variantExists(envelope, path, new ReusableResult());
+ }
+
+ /// Allocation-free compiled-path form of {@link #variantExists(byte[], VariantPath)} when the caller retains the
+ /// supplied result between rows.
+ @Nullable
+ public static Boolean variantExists(@Nullable byte[] envelope, VariantPath path, ReusableResult result) {
+ Objects.requireNonNull(result, "result must not be null");
+ if (isSqlNull(envelope)) {
+ return null;
+ }
+ return result._cursor.navigate(envelope, Objects.requireNonNull(path, "path must not be null"));
+ }
+
+ /// Returns whether the root value is a Variant null. SQL null is not a Variant null.
+ public static boolean isVariantNull(@Nullable byte[] envelope) {
+ return isVariantNull(envelope, ROOT_PATH, new ReusableResult());
+ }
+
+ /// Returns whether a present value at the path is a Variant null. SQL null and missing paths return false.
+ public static boolean isVariantNull(@Nullable byte[] envelope, String path) {
+ return isVariantNull(envelope, compilePath(path));
+ }
+
+ /// Returns whether a present value at a compiled path is a Variant null. SQL null and missing paths return false.
+ public static boolean isVariantNull(@Nullable byte[] envelope, VariantPath path) {
+ return isVariantNull(envelope, path, new ReusableResult());
+ }
+
+ /// Allocation-free compiled-path form of {@link #isVariantNull(byte[], VariantPath)} when the caller retains the
+ /// supplied result between rows.
+ public static boolean isVariantNull(@Nullable byte[] envelope, VariantPath path, ReusableResult result) {
+ Objects.requireNonNull(result, "result must not be null");
+ if (isSqlNull(envelope)) {
+ return false;
+ }
+ Cursor cursor = result._cursor;
+ return cursor.navigate(envelope, Objects.requireNonNull(path, "path must not be null"))
+ && cursor.getType() == Variant.Type.NULL;
+ }
+
+ /// Returns the Variant type name at the root, or Java null for SQL null.
+ @Nullable
+ public static String variantTypeOf(@Nullable byte[] envelope) {
+ return variantTypeOf(envelope, ROOT_PATH, new ReusableResult());
+ }
+
+ /// Returns the Variant type name at a path, or Java null for SQL null or a missing path.
+ @Nullable
+ public static String variantTypeOf(@Nullable byte[] envelope, String path) {
+ return variantTypeOf(envelope, compilePath(path));
+ }
+
+ /// Returns the Variant type name at a compiled path, or Java null for SQL null or a missing path.
+ @Nullable
+ public static String variantTypeOf(@Nullable byte[] envelope, VariantPath path) {
+ return variantTypeOf(envelope, path, new ReusableResult());
+ }
+
+ /// Allocation-free compiled-path form of {@link #variantTypeOf(byte[], VariantPath)} when the caller retains the
+ /// supplied result between rows.
+ @Nullable
+ public static String variantTypeOf(@Nullable byte[] envelope, VariantPath path, ReusableResult result) {
+ Objects.requireNonNull(result, "result must not be null");
+ if (isSqlNull(envelope)) {
+ return null;
+ }
+ Cursor cursor = result._cursor;
+ return cursor.navigate(envelope, Objects.requireNonNull(path, "path must not be null"))
+ ? typeName(cursor.getType()) : null;
+ }
+
+ /// Renders the Variant value as canonical JSON text without constructing a JSON tree.
+ @Nullable
+ public static String variantToJson(@Nullable byte[] envelope) {
+ return variantToJson(envelope, new ReusableResult());
+ }
+
+ /// Allocation-reduced form of [#variantToJson(byte[])] when the caller retains the supplied result between rows.
+ @Nullable
+ public static String variantToJson(@Nullable byte[] envelope, ReusableResult result) {
+ Objects.requireNonNull(result, "result must not be null");
+ if (isSqlNull(envelope)) {
+ return null;
+ }
+ Cursor cursor = result._cursor;
+ cursor.navigate(envelope, ROOT_PATH);
+ return variantToJson(cursor.asVariant());
+ }
+
+ /// Parses JSON text into a Pinot Variant envelope without constructing a JSON tree.
+ @Nullable
+ public static byte[] parseJsonToVariant(@Nullable String json) {
+ if (json == null) {
+ return null;
+ }
+ try (JsonParser parser = JSON_FACTORY.createParser(json)) {
+ JsonToken token = parser.nextToken();
+ if (token == null) {
+ throw new IllegalArgumentException("Cannot parse empty text as Variant");
+ }
+ VariantBuilder builder = new VariantBuilder();
+ appendJsonValue(parser, token, builder, 0);
+ if (parser.nextToken() != null) {
+ throw new IllegalArgumentException("Unexpected trailing token after Variant JSON value");
+ }
+ Variant variant = builder.build();
+ return VariantEnvelope.encode(variant.getMetadataBuffer(), variant.getValueBuffer());
+ } catch (IOException | RuntimeException e) {
+ throw new IllegalArgumentException("Cannot parse JSON as Variant", e);
+ }
+ }
+
+ /// Tolerant JSON parser. Malformed or unsupported input returns Java null.
+ @Nullable
+ public static byte[] tryParseJsonToVariant(@Nullable String json) {
+ try {
+ return parseJsonToVariant(json);
+ } catch (RuntimeException e) {
+ return null;
+ }
+ }
+
+ private static boolean isSqlNull(@Nullable byte[] envelope) {
+ return envelope == null || envelope.length == 0;
+ }
+
+ private static void convert(Cursor value, ResultType targetType, ReusableResult result) {
+ switch (targetType) {
+ case BOOLEAN:
+ requireType(value, Variant.Type.BOOLEAN, targetType);
+ result._intValue = value.getBoolean() ? 1 : 0;
+ break;
+ case INT:
+ result._intValue = toInt(value, targetType);
+ break;
+ case LONG:
+ result._longValue = toLong(value, targetType);
+ break;
+ case FLOAT:
+ result._floatValue = toFloat(value, targetType);
+ break;
+ case DOUBLE:
+ result._doubleValue = toDouble(value, targetType);
+ break;
+ case BIG_DECIMAL:
+ result._bigDecimalValue = toBigDecimal(value, targetType);
+ break;
+ case STRING:
+ requireType(value, Variant.Type.STRING, targetType);
+ result._stringValue = value.getString();
+ break;
+ case BYTES:
+ requireType(value, Variant.Type.BINARY, targetType);
+ result._bytesValue = value.getBinary();
+ break;
+ case UUID:
+ requireType(value, Variant.Type.UUID, targetType);
+ result._bytesValue = value.getUuidBytes();
+ break;
+ case TIMESTAMP:
+ result._longValue = toTimestampMillis(value, targetType);
+ break;
+ case VARIANT:
+ result._bytesValue = value.copyEnvelope();
+ break;
+ case JSON:
+ result._stringValue = variantToJson(value.asVariant());
+ break;
+ default:
+ throw new IllegalStateException("Unhandled Variant target type: " + targetType);
+ }
+ }
+
+ private static boolean tryConvert(Cursor value, ResultType targetType, ReusableResult result) {
+ Variant.Type valueType = value.getType();
+ switch (targetType) {
+ case BOOLEAN:
+ if (valueType != Variant.Type.BOOLEAN) {
+ return false;
+ }
+ result._intValue = value.getBoolean() ? 1 : 0;
+ return true;
+ case INT:
+ return tryConvertToInt(value, valueType, result);
+ case LONG:
+ return tryConvertToLong(value, valueType, result);
+ case FLOAT:
+ switch (valueType) {
+ case BYTE:
+ case SHORT:
+ case INT:
+ case LONG:
+ result._floatValue = value.getInteger();
+ return true;
+ case FLOAT:
+ result._floatValue = value.getFloat();
+ return true;
+ case DOUBLE:
+ result._floatValue = (float) value.getDouble();
+ return true;
+ case DECIMAL4:
+ case DECIMAL8:
+ case DECIMAL16:
+ result._floatValue = value.getDecimal().floatValue();
+ return true;
+ default:
+ return false;
+ }
+ case DOUBLE:
+ switch (valueType) {
+ case BYTE:
+ case SHORT:
+ case INT:
+ case LONG:
+ result._doubleValue = value.getInteger();
+ return true;
+ case FLOAT:
+ result._doubleValue = value.getFloat();
+ return true;
+ case DOUBLE:
+ result._doubleValue = value.getDouble();
+ return true;
+ case DECIMAL4:
+ case DECIMAL8:
+ case DECIMAL16:
+ result._doubleValue = value.getDecimal().doubleValue();
+ return true;
+ default:
+ return false;
+ }
+ case BIG_DECIMAL:
+ switch (valueType) {
+ case BYTE:
+ case SHORT:
+ case INT:
+ case LONG:
+ result._bigDecimalValue = BigDecimal.valueOf(value.getInteger());
+ return true;
+ case FLOAT:
+ float floatValue = value.getFloat();
+ if (!Float.isFinite(floatValue)) {
+ return false;
+ }
+ result._bigDecimalValue = BigDecimal.valueOf(floatValue);
+ return true;
+ case DOUBLE:
+ double doubleValue = value.getDouble();
+ if (!Double.isFinite(doubleValue)) {
+ return false;
+ }
+ result._bigDecimalValue = BigDecimal.valueOf(doubleValue);
+ return true;
+ case DECIMAL4:
+ case DECIMAL8:
+ case DECIMAL16:
+ result._bigDecimalValue = value.getDecimal();
+ return true;
+ default:
+ return false;
+ }
+ case STRING:
+ if (valueType != Variant.Type.STRING) {
+ return false;
+ }
+ result._stringValue = value.getString();
+ return true;
+ case BYTES:
+ if (valueType != Variant.Type.BINARY) {
+ return false;
+ }
+ result._bytesValue = value.getBinary();
+ return true;
+ case UUID:
+ if (valueType != Variant.Type.UUID) {
+ return false;
+ }
+ result._bytesValue = value.getUuidBytes();
+ return true;
+ case TIMESTAMP:
+ switch (valueType) {
+ case DATE:
+ result._longValue = value.getInteger() * TimeUnit.DAYS.toMillis(1);
+ return true;
+ case TIMESTAMP_TZ:
+ case TIMESTAMP_NTZ:
+ result._longValue = Math.floorDiv(value.getInteger(), TimeUnit.MILLISECONDS.toMicros(1));
+ return true;
+ case TIMESTAMP_NANOS_TZ:
+ case TIMESTAMP_NANOS_NTZ:
+ result._longValue = Math.floorDiv(value.getInteger(), TimeUnit.MILLISECONDS.toNanos(1));
+ return true;
+ default:
+ return false;
+ }
+ case VARIANT:
+ result._bytesValue = value.copyEnvelope();
+ return true;
+ case JSON:
+ result._stringValue = variantToJson(value.asVariant());
+ return true;
+ default:
+ throw new AssertionError("Unhandled Variant target type: " + targetType);
+ }
+ }
+
+ private static boolean tryConvertToInt(Cursor value, Variant.Type valueType, ReusableResult result) {
+ switch (valueType) {
+ case BYTE:
+ case SHORT:
+ case INT:
+ result._intValue = (int) value.getInteger();
+ return true;
+ case LONG:
+ long longValue = value.getInteger();
+ if (longValue < Integer.MIN_VALUE || longValue > Integer.MAX_VALUE) {
+ return false;
+ }
+ result._intValue = (int) longValue;
+ return true;
+ case DECIMAL4:
+ case DECIMAL8:
+ case DECIMAL16:
+ BigDecimal decimalValue = value.getDecimal();
+ if (!isIntegralInRange(decimalValue, MIN_INT_DECIMAL, MAX_INT_DECIMAL)) {
+ return false;
+ }
+ result._intValue = decimalValue.intValue();
+ return true;
+ default:
+ return false;
+ }
+ }
+
+ private static boolean tryConvertToLong(Cursor value, Variant.Type valueType, ReusableResult result) {
+ switch (valueType) {
+ case BYTE:
+ case SHORT:
+ case INT:
+ case LONG:
+ result._longValue = value.getInteger();
+ return true;
+ case DECIMAL4:
+ case DECIMAL8:
+ case DECIMAL16:
+ BigDecimal decimalValue = value.getDecimal();
+ if (!isIntegralInRange(decimalValue, MIN_LONG_DECIMAL, MAX_LONG_DECIMAL)) {
+ return false;
+ }
+ result._longValue = decimalValue.longValue();
+ return true;
+ default:
+ return false;
+ }
+ }
+
+ private static boolean isIntegralInRange(BigDecimal value, BigDecimal minimum, BigDecimal maximum) {
+ return value.compareTo(minimum) >= 0 && value.compareTo(maximum) <= 0
+ && (value.scale() <= 0 || value.stripTrailingZeros().scale() <= 0);
+ }
+
+ private static int toInt(Cursor value, ResultType targetType) {
+ switch (value.getType()) {
+ case BYTE:
+ case SHORT:
+ case INT:
+ return (int) value.getInteger();
+ case LONG:
+ return Math.toIntExact(value.getInteger());
+ case DECIMAL4:
+ case DECIMAL8:
+ case DECIMAL16:
+ return value.getDecimal().intValueExact();
+ default:
+ throw typeMismatch(value, targetType);
+ }
+ }
+
+ private static long toLong(Cursor value, ResultType targetType) {
+ switch (value.getType()) {
+ case BYTE:
+ case SHORT:
+ case INT:
+ case LONG:
+ return value.getInteger();
+ case DECIMAL4:
+ case DECIMAL8:
+ case DECIMAL16:
+ return value.getDecimal().longValueExact();
+ default:
+ throw typeMismatch(value, targetType);
+ }
+ }
+
+ private static float toFloat(Cursor value, ResultType targetType) {
+ switch (value.getType()) {
+ case BYTE:
+ case SHORT:
+ case INT:
+ case LONG:
+ return value.getInteger();
+ case FLOAT:
+ return value.getFloat();
+ case DOUBLE:
+ return (float) value.getDouble();
+ case DECIMAL4:
+ case DECIMAL8:
+ case DECIMAL16:
+ return value.getDecimal().floatValue();
+ default:
+ throw typeMismatch(value, targetType);
+ }
+ }
+
+ private static double toDouble(Cursor value, ResultType targetType) {
+ switch (value.getType()) {
+ case BYTE:
+ case SHORT:
+ case INT:
+ case LONG:
+ return value.getInteger();
+ case FLOAT:
+ return value.getFloat();
+ case DOUBLE:
+ return value.getDouble();
+ case DECIMAL4:
+ case DECIMAL8:
+ case DECIMAL16:
+ return value.getDecimal().doubleValue();
+ default:
+ throw typeMismatch(value, targetType);
+ }
+ }
+
+ private static BigDecimal toBigDecimal(Cursor value, ResultType targetType) {
+ switch (value.getType()) {
+ case BYTE:
+ case SHORT:
+ case INT:
+ case LONG:
+ return BigDecimal.valueOf(value.getInteger());
+ case FLOAT:
+ return BigDecimal.valueOf(value.getFloat());
+ case DOUBLE:
+ return BigDecimal.valueOf(value.getDouble());
+ case DECIMAL4:
+ case DECIMAL8:
+ case DECIMAL16:
+ return value.getDecimal();
+ default:
+ throw typeMismatch(value, targetType);
+ }
+ }
+
+ private static long toTimestampMillis(Cursor value, ResultType targetType) {
+ switch (value.getType()) {
+ case DATE:
+ return Math.multiplyExact(value.getInteger(), TimeUnit.DAYS.toMillis(1));
+ case TIMESTAMP_TZ:
+ case TIMESTAMP_NTZ:
+ return Math.floorDiv(value.getInteger(), TimeUnit.MILLISECONDS.toMicros(1));
+ case TIMESTAMP_NANOS_TZ:
+ case TIMESTAMP_NANOS_NTZ:
+ return Math.floorDiv(value.getInteger(), TimeUnit.MILLISECONDS.toNanos(1));
+ default:
+ throw typeMismatch(value, targetType);
+ }
+ }
+
+ private static void requireType(Cursor value, Variant.Type expected, ResultType targetType) {
+ if (value.getType() != expected) {
+ throw typeMismatch(value, targetType);
+ }
+ }
+
+ private static IllegalArgumentException typeMismatch(Cursor value, ResultType targetType) {
+ return new IllegalArgumentException(
+ "Cannot convert Variant " + typeName(value.getType()) + " to " + targetType.name());
+ }
+
+ private static String typeName(Variant.Type type) {
+ switch (type) {
+ case DECIMAL4:
+ case DECIMAL8:
+ case DECIMAL16:
+ return "DECIMAL";
+ default:
+ return type.name();
+ }
+ }
+
+ private static String variantToJson(Variant variant) {
+ try {
+ // StringBuilder-backed writer: StringWriter wraps a synchronized StringBuffer and would pay a monitor
+ // acquisition per append on this per-row rendering path.
+ StringBuilderWriter writer = new StringBuilderWriter();
+ try (JsonGenerator generator = JSON_FACTORY.createGenerator(writer)) {
+ writeJsonValue(generator, variant);
+ }
+ return writer.toString();
+ } catch (IOException e) {
+ throw new IllegalStateException("Cannot render Variant as JSON", e);
+ }
+ }
+
+
+ private static void writeJsonValue(JsonGenerator generator, Variant variant)
+ throws IOException {
+ switch (variant.getType()) {
+ case OBJECT:
+ generator.writeStartObject();
+ for (int i = 0; i < variant.numObjectElements(); i++) {
+ Variant.ObjectField field = variant.getFieldAtIndex(i);
+ generator.writeFieldName(field.key);
+ writeJsonValue(generator, field.value);
+ }
+ generator.writeEndObject();
+ break;
+ case ARRAY:
+ generator.writeStartArray();
+ for (int i = 0; i < variant.numArrayElements(); i++) {
+ writeJsonValue(generator, variant.getElementAtIndex(i));
+ }
+ generator.writeEndArray();
+ break;
+ case NULL:
+ generator.writeNull();
+ break;
+ case BOOLEAN:
+ generator.writeBoolean(variant.getBoolean());
+ break;
+ case BYTE:
+ generator.writeNumber(variant.getByte());
+ break;
+ case SHORT:
+ generator.writeNumber(variant.getShort());
+ break;
+ case INT:
+ generator.writeNumber(variant.getInt());
+ break;
+ case LONG:
+ generator.writeNumber(variant.getLong());
+ break;
+ case FLOAT:
+ generator.writeNumber(variant.getFloat());
+ break;
+ case DOUBLE:
+ generator.writeNumber(variant.getDouble());
+ break;
+ case DECIMAL4:
+ case DECIMAL8:
+ case DECIMAL16:
+ generator.writeNumber(variant.getDecimal());
+ break;
+ case STRING:
+ generator.writeString(variant.getString());
+ break;
+ case BINARY:
+ generator.writeBinary(toBytes(variant.getBinary()));
+ break;
+ case UUID:
+ generator.writeString(variant.getUUID().toString());
+ break;
+ case DATE:
+ generator.writeString(LocalDate.ofEpochDay(variant.getInt()).toString());
+ break;
+ case TIMESTAMP_TZ:
+ generator.writeString(instantFromMicros(variant.getLong()).toString());
+ break;
+ case TIMESTAMP_NTZ:
+ generator.writeString(LocalDateTime.ofInstant(instantFromMicros(variant.getLong()), ZoneOffset.UTC).toString());
+ break;
+ case TIMESTAMP_NANOS_TZ:
+ generator.writeString(instantFromNanos(variant.getLong()).toString());
+ break;
+ case TIMESTAMP_NANOS_NTZ:
+ generator.writeString(LocalDateTime.ofInstant(instantFromNanos(variant.getLong()), ZoneOffset.UTC).toString());
+ break;
+ case TIME:
+ generator.writeString(LocalTime.ofNanoOfDay(Math.floorMod(variant.getLong() * NANOS_PER_MICRO, NANOS_PER_DAY))
+ .toString());
+ break;
+ default:
+ throw new IllegalStateException("Unsupported Variant type: " + variant.getType());
+ }
+ }
+
+ private static Instant instantFromMicros(long micros) {
+ long seconds = Math.floorDiv(micros, MICROS_PER_SECOND);
+ long nanos = Math.floorMod(micros, MICROS_PER_SECOND) * NANOS_PER_MICRO;
+ return Instant.ofEpochSecond(seconds, nanos);
+ }
+
+ private static Instant instantFromNanos(long nanos) {
+ return Instant.ofEpochSecond(Math.floorDiv(nanos, TimeUnit.SECONDS.toNanos(1)),
+ Math.floorMod(nanos, TimeUnit.SECONDS.toNanos(1)));
+ }
+
+ private static byte[] toBytes(ByteBuffer buffer) {
+ ByteBuffer view = buffer.slice();
+ byte[] bytes = new byte[view.remaining()];
+ view.get(bytes);
+ return bytes;
+ }
+
+ private static void appendJsonValue(JsonParser parser, JsonToken token, VariantBuilder builder, int depth)
+ throws IOException {
+ if (depth > MAX_JSON_NESTING_DEPTH) {
+ throw new IllegalArgumentException("Variant JSON exceeds maximum nesting depth " + MAX_JSON_NESTING_DEPTH);
+ }
+ switch (token) {
+ case START_OBJECT:
+ VariantObjectBuilder objectBuilder = builder.startObject();
+ while (parser.nextToken() != JsonToken.END_OBJECT) {
+ if (parser.currentToken() != JsonToken.FIELD_NAME) {
+ throw new IllegalArgumentException("Expected a JSON object field name");
+ }
+ objectBuilder.appendKey(parser.currentName());
+ JsonToken fieldValue = parser.nextToken();
+ if (fieldValue == null) {
+ throw new IllegalArgumentException("Unexpected end of JSON object");
+ }
+ appendJsonValue(parser, fieldValue, objectBuilder, depth + 1);
+ }
+ builder.endObject();
+ break;
+ case START_ARRAY:
+ VariantArrayBuilder arrayBuilder = builder.startArray();
+ while (true) {
+ JsonToken element = parser.nextToken();
+ if (element == JsonToken.END_ARRAY) {
+ break;
+ }
+ if (element == null) {
+ throw new IllegalArgumentException("Unexpected end of JSON array");
+ }
+ appendJsonValue(parser, element, arrayBuilder, depth + 1);
+ }
+ builder.endArray();
+ break;
+ case VALUE_NULL:
+ builder.appendNull();
+ break;
+ case VALUE_TRUE:
+ builder.appendBoolean(true);
+ break;
+ case VALUE_FALSE:
+ builder.appendBoolean(false);
+ break;
+ case VALUE_STRING:
+ builder.appendString(parser.getText());
+ break;
+ case VALUE_NUMBER_INT:
+ appendInteger(parser, builder);
+ break;
+ case VALUE_NUMBER_FLOAT:
+ appendDecimal(parser.getDecimalValue(), builder);
+ break;
+ default:
+ throw new IllegalArgumentException("Unsupported JSON token for Variant: " + token);
+ }
+ }
+
+ private static void appendInteger(JsonParser parser, VariantBuilder builder)
+ throws IOException {
+ switch (parser.getNumberType()) {
+ case INT:
+ builder.appendInt(parser.getIntValue());
+ break;
+ case LONG:
+ builder.appendLong(parser.getLongValue());
+ break;
+ case BIG_INTEGER:
+ appendBigInteger(parser.getBigIntegerValue(), builder);
+ break;
+ default:
+ throw new IllegalArgumentException("Unsupported JSON integer representation: " + parser.getNumberType());
+ }
+ }
+
+ private static void appendBigInteger(BigInteger value, VariantBuilder builder) {
+ if (value.bitLength() < Integer.SIZE) {
+ builder.appendInt(value.intValue());
+ } else if (value.bitLength() < Long.SIZE) {
+ builder.appendLong(value.longValue());
+ } else {
+ appendDecimal(new BigDecimal(value), builder);
+ }
+ }
+
+ private static void appendDecimal(BigDecimal value, VariantBuilder builder) {
+ BigDecimal normalized = value;
+ if (normalized.scale() < 0) {
+ // Parquet Variant stores scale as an unsigned byte. Expand exponent notation exactly instead of allowing a
+ // negative scale to wrap during encoding.
+ long expandedPrecision = (long) normalized.precision() - normalized.scale();
+ if (normalized.signum() != 0 && expandedPrecision > MAX_VARIANT_DECIMAL_PRECISION) {
+ throw unsupportedVariantDecimal(value);
+ }
+ normalized = normalized.signum() == 0 ? BigDecimal.ZERO : normalized.setScale(0);
+ } else if (normalized.scale() > MAX_VARIANT_DECIMAL_SCALE) {
+ // Accept values whose excessive lexical scale consists only of insignificant trailing zeros.
+ normalized = normalized.stripTrailingZeros();
+ if (normalized.scale() < 0) {
+ normalized = normalized.setScale(0);
+ }
+ }
+ byte[] unscaledBytes = normalized.unscaledValue().toByteArray();
+ if (normalized.scale() > MAX_VARIANT_DECIMAL_SCALE
+ || normalized.precision() > MAX_VARIANT_DECIMAL_PRECISION
+ || unscaledBytes.length > MAX_VARIANT_DECIMAL_BYTES) {
+ throw unsupportedVariantDecimal(value);
+ }
+ builder.appendDecimal(normalized);
+ }
+
+ private static IllegalArgumentException unsupportedVariantDecimal(BigDecimal value) {
+ return new IllegalArgumentException(
+ "JSON decimal exceeds Parquet Variant decimal(38) encoding: precision=" + value.precision()
+ + ", scale=" + value.scale());
+ }
+
+ /// Mutable zero-copy view over one selected value in a Pinot envelope.
+ ///
+ /// <p>The constants and layouts used here mirror Parquet Variant encoding version 1. Keeping this cursor on
+ /// {@link ReusableResult} avoids allocating envelope views, Variant wrappers, and navigation wrappers for every row.
+ private static final class Cursor {
Review Comment:
Design question worth settling before merge: Cursor is a ~700-line hand-rolled reimplementation of the Parquet Variant binary decoder that Pinot already has on the classpath (org.apache.parquet.variant.Variant), duplicating every type tag, offset layout and metadata-dictionary rule.
The performance motivation is real - zero-copy navigation, no per-row Variant/ByteBuffer allocation, and the cross-row dictionary/entry-index memo is genuinely clever - and asVariant() proves the two can coexist. But this is now a second implementation of a spec Pinot does not own. Concrete failure mode: when parquet-variant gains a primitive type tag or a v2 encoding, the Parquet converters accept the file at ingestion while this cursor throws UnsupportedOperationException from getType()/encodedValueLength() at query time. So the data lands and only breaks when someone queries it. Please state the sync policy explicitly - a test that asserts this constant table matches parquet-variant's VariantUtil constants, or a hard-pinned parquet version with a comment - and consider falling back to the library decoder for tags Cursor does not recognise rather than throwing.
##########
pinot-spi/src/main/java/org/apache/pinot/spi/data/FieldSpec.java:
##########
@@ -909,6 +991,9 @@ public int hashCode(Object value) {
/// return -1 if value1 is less than value2
/// return 1 if value1 is greater than value2
public int compare(Object value1, Object value2) {
+ if (!supportsOrdering()) {
+ throw new UnsupportedOperationException(this + " does not support ordering");
Review Comment:
Worth calling out in the PR body even if it stays as-is: compare() now throws UnsupportedOperationException for STRUCT / MAP / OPEN_STRUCT / LIST / UNKNOWN, which previously reached the default branch and threw IllegalStateException. equals() and hashCode() gained a new throwing branch on the same public SPI enum.
The fail-closed design is right for VARIANT; the incidental part is that five pre-existing types changed their exception type. It only bites a caller doing catch (IllegalStateException) - unlikely in-tree, possible in a plugin or a downstream fork. If you keep it, mention it in the PR description so it reaches the release notes; if you would rather have zero blast radius, throw only for VARIANT here and let the other five fall through to the old default branch as before.
##########
pinot-spi/src/main/java/org/apache/pinot/spi/data/FieldSpec.java:
##########
@@ -684,6 +691,9 @@ protected void appendDefaultNullValue(ObjectNode jsonNode) {
case BYTES:
jsonNode.put(key, BytesUtils.toHexString((byte[]) _defaultNullValue));
break;
+ case VARIANT:
Review Comment:
Nit: this VARIANT arm is byte-identical to the BYTES arm three lines above. Just fall through (case BYTES: case VARIANT:) as the file already does elsewhere for the BYTES/VARIANT pair.
Same in convert() and convertInternal(), where the VARIANT arms differ from BYTES only by the added VariantEnvelope.decode() validation - those are worth keeping separate. This one is pure duplication and will drift the first time the BYTES hex encoding changes.
##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/validation/TypeCapabilityValidationVisitor.java:
##########
@@ -0,0 +1,199 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.query.planner.validation;
+
+import java.util.List;
+import org.apache.calcite.rel.RelFieldCollation;
+import org.apache.pinot.common.utils.DataSchema;
+import org.apache.pinot.common.utils.VariantUtils;
+import org.apache.pinot.query.planner.logical.RexExpression;
+import org.apache.pinot.query.planner.plannode.AggregateNode;
+import org.apache.pinot.query.planner.plannode.JoinNode;
+import org.apache.pinot.query.planner.plannode.PlanNode;
+import org.apache.pinot.query.planner.plannode.PlanNodeVisitor;
+import org.apache.pinot.query.planner.plannode.SetOpNode;
+import org.apache.pinot.query.planner.plannode.SortNode;
+import org.apache.pinot.query.planner.plannode.WindowNode;
+import org.apache.pinot.spi.exception.QueryErrorCode;
+import org.apache.pinot.spi.exception.QueryException;
+
+
+/// Validates that each logical input type supports the capabilities required by its operation, including equality,
+/// hashing, ordering, aggregation, and lossless result projection. The visitor has no mutable state and is thread-safe,
+/// so callers may share {@link #INSTANCE}.
+public final class TypeCapabilityValidationVisitor extends PlanNodeVisitor.DepthFirstVisitor<Void, Void> {
+ public static final TypeCapabilityValidationVisitor INSTANCE = new TypeCapabilityValidationVisitor();
+
+ private TypeCapabilityValidationVisitor() {
+ }
+
+ @Override
+ protected boolean traverseStageBoundary() {
+ return false;
+ }
+
+ @Override
+ public Void visitAggregate(AggregateNode node, Void context) {
+ List<PlanNode> inputs = node.getInputs();
+ if (inputs.size() == 1) {
+ validateAggregateInputs(node, inputs.get(0).getDataSchema());
+ }
+ return super.visitAggregate(node, context);
+ }
+
+ /// Validates aggregate operands against their logical input schema.
+ ///
+ /// <p>This method is also invoked by the runtime as a defensive check for plans that did not pass through the
+ /// current broker planner.
+ public static void validateAggregateInputs(AggregateNode node, DataSchema inputSchema) {
+ for (int key : node.getGroupKeys()) {
+ DataSchema.ColumnDataType dataType = inputSchema.getColumnDataType(key);
+ if (!dataType.supportsEquality() || !dataType.supportsHashing()) {
+ throw unsupported("GROUP BY", dataType);
+ }
+ }
+ validateAggregateInputs(node.getAggCalls(), inputSchema);
+ }
+
+ /// Validates aggregate or window-function operands against their logical input schema.
+ public static void validateAggregateInputs(List<RexExpression.FunctionCall> aggCalls, DataSchema inputSchema) {
+ for (RexExpression.FunctionCall aggCall : aggCalls) {
+ if (isRawVariantIndependent(aggCall)) {
+ continue;
+ }
+ for (RexExpression operand : aggCall.getFunctionOperands()) {
+ DataSchema.ColumnDataType dataType = getLogicalType(operand, inputSchema);
+ if (!dataType.supportsDirectAggregation()) {
+ throw unsupported("Aggregate function " + aggCall.getFunctionName(), dataType);
+ }
+ }
+ }
+ }
+
+ /// Rejects a raw VARIANT result when query null handling is disabled. Without the null bitmap, the reserved empty
+ /// byte placeholder cannot participate in normal disabled-null semantics while also remaining distinguishable from
+ /// an encoded Variant null.
+ public static void validateResultSchema(DataSchema resultSchema, boolean nullHandlingEnabled) {
+ if (VariantUtils.requiresNullHandlingForRawVariantResult(resultSchema, nullHandlingEnabled)) {
+ throw new QueryException(QueryErrorCode.QUERY_PLANNING,
+ VariantUtils.RAW_VARIANT_REQUIRES_NULL_HANDLING_ERROR);
+ }
+ }
+
+ @Override
+ public Void visitSort(SortNode node, Void context) {
+ DataSchema dataSchema = node.getDataSchema();
+ for (RelFieldCollation collation : node.getCollations()) {
+ int fieldIndex = collation.getFieldIndex();
+ DataSchema.ColumnDataType dataType = dataSchema.getColumnDataType(fieldIndex);
+ if (!dataType.supportsOrdering()) {
+ throw unsupported("ORDER BY", dataType);
+ }
+ }
+ return super.visitSort(node, context);
+ }
+
+ @Override
+ public Void visitSetOp(SetOpNode node, Void context) {
Review Comment:
Biggest scope concern in the PR. This visitor runs on EVERY multi-stage plan, and the capability predicates are false for arrays and OBJECT as well as VARIANT - so these guards reject non-VARIANT queries that plan successfully today.
Concretely: INTERSECT / EXCEPT / UNION DISTINCT over any array or OBJECT column becomes a QUERY_PLANNING error, and SortOperator adds a matching runtime Preconditions check for ORDER BY. The unsupported() helper explicitly formats messages for 'non-VARIANT opaque types (OBJECT, arrays, MAP)', and OrderByComparatorFactoryTest adds a MAP rejection test, so this is clearly deliberate - but it is a user-visible behaviour change that the PR description does not mention. The backward-incompat label is already applied; what is missing is a sentence in the PR body so it reaches the release notes, plus the test coverage in my note on TypeCapabilityValidationVisitorTest. Note also that GROUP BY on an MV column still produces the friendlier 'Use ARRAY_TO_MV()' message only because runValidations() happens to run ArrayToMvValidationVisitor before this one - that ordering is now load-bearing and undocumented; a comment in PinotDispatchPlanner.runValidations would help.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/ColumnarValueNormalizer.java:
##########
@@ -28,21 +28,21 @@
/// (`buildColumnar`) path.
///
/// The row-major build runs each record through a `TransformPipeline` whose
-/// `NullValueTransformer` substitutes [FieldSpec#getDefaultNullValue()] for `null`
-/// and whose `DataTypeTransformer` coerces every value to the column's stored type (e.g.
+/// `DataTypeTransformer` coerces every non-null value to the column's stored type (e.g.
/// `Boolean` → `Integer` for a `BOOLEAN` column stored as `INT`,
-/// `Timestamp` → `Long` for `TIMESTAMP`). The column-major driver deliberately runs
-/// with no transform pipeline, so a non-segment source (e.g. Arrow) delivers values in the source's
+/// `Timestamp` → `Long` for `TIMESTAMP`) and whose `NullValueTransformer` substitutes
+/// [FieldSpec#getDefaultNullValue()] for values that remain `null`. The column-major driver deliberately runs with
+/// no transform pipeline, so a non-segment source (e.g. Arrow) delivers values in the source's
/// logical type with raw `null`s — which the typed collectors / index creators do not accept.
///
/// This helper applies the equivalent of those two transformers to one value, in the same order:
///
-/// 1. `NullValueTransformer`: a `null` value becomes the column default — the scalar
-/// default for single-value columns, or a one-element `Object[]` of that scalar for
-/// multi-value columns (matching `NullValueTransformerUtils.getDefaultNullValue`).
-/// 2. `DataTypeTransformer`: [DataTypeTransformerUtils#transformValue] standardizes the
+/// 1. `DataTypeTransformer`: [DataTypeTransformerUtils#transformValue] standardizes the
Review Comment:
This reorder is correct - I checked RecordTransformerUtils.getTransformers() and the real pipeline is DataTypeTransformer then NullValueTransformer, so the previous doc and code had it backwards - but it changes the columnar build path for every data type, not just VARIANT.
Net effect: the FieldSpec default null value no longer passes through DataTypeTransformerUtils.transformValue. That happens to be safe as written (BOOLEAN default is Integer 0 and TIMESTAMP default is Long 0 - already in stored form), but the safety is incidental rather than enforced: a future type whose FieldSpec default is not already stored-form would now reach the collectors unconverted. Worth a line in the PR body, and see my test note on ColumnarValueNormalizerTest.
##########
pinot-query-planner/src/test/java/org/apache/pinot/query/planner/validation/TypeCapabilityValidationVisitorTest.java:
##########
@@ -0,0 +1,276 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.query.planner.validation;
+
+import java.util.List;
+import org.apache.calcite.rel.RelFieldCollation;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.pinot.common.utils.DataSchema;
+import org.apache.pinot.query.planner.logical.RexExpression;
+import org.apache.pinot.query.planner.plannode.AggregateNode;
+import org.apache.pinot.query.planner.plannode.JoinNode;
+import org.apache.pinot.query.planner.plannode.PlanNode;
+import org.apache.pinot.query.planner.plannode.SetOpNode;
+import org.apache.pinot.query.planner.plannode.SortNode;
+import org.apache.pinot.query.planner.plannode.ValueNode;
+import org.apache.pinot.query.planner.plannode.WindowNode;
+import org.apache.pinot.spi.exception.QueryException;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+
+public class TypeCapabilityValidationVisitorTest {
+ private static final DataSchema VARIANT_SCHEMA =
+ new DataSchema(new String[]{"payload"}, new DataSchema.ColumnDataType[]{DataSchema.ColumnDataType.VARIANT});
+ private static final DataSchema TYPED_EXTRACTION_SCHEMA =
+ new DataSchema(new String[]{"typedPayload"}, new DataSchema.ColumnDataType[]{DataSchema.ColumnDataType.STRING});
+
+ @Test
+ public void testRejectsVariantOrderBy() {
+ SortNode sortNode = new SortNode(0, VARIANT_SCHEMA, PlanNode.NodeHint.EMPTY, List.of(),
+ List.of(new RelFieldCollation(0)), 10, 0);
+
+ QueryException exception =
+ Assert.expectThrows(QueryException.class, () -> sortNode.visit(TypeCapabilityValidationVisitor.INSTANCE, null));
+ Assert.assertTrue(exception.getMessage().contains("ORDER BY"));
+ }
+
+ @Test
+ public void testNamesUnsupportedNonVariantOrderByType() {
Review Comment:
Test gap: the guards now reject arrays, but this suite has zero array coverage. The only non-VARIANT case is this single OBJECT ORDER BY test.
Since the widening to non-VARIANT types is intentional, it should be pinned. Please add cases for INT_ARRAY / STRING_ARRAY through visitSetOp (INTERSECT / EXCEPT / UNION DISTINCT), visitSort, visitAggregate group keys and visitWindow partition keys - asserting whatever the intended behaviour is. Right now a future change that relaxes or tightens array handling would not fail a single test, and the only signal that arrays are affected at all is a comment in the unsupported() helper.
##########
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/creator/impl/ColumnarValueNormalizerTest.java:
##########
@@ -0,0 +1,43 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl;
+
+import org.apache.pinot.spi.data.DimensionFieldSpec;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.FieldSpec.DataType;
+import org.apache.pinot.spi.utils.PinotDataType;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertSame;
+
+
+public class ColumnarValueNormalizerTest {
+ private static final String COLUMN = "payload";
+
+ @Test
+ public void testNullVariantReturnsDefaultSentinelWithoutDecoding() {
Review Comment:
Test gap: this new suite has exactly one test, and it only covers the VARIANT sentinel. The transformer reorder it accompanies affects every type on the columnar path, with nothing pinning the result.
Please add cases asserting the normalized default for BOOLEAN (expect Integer 0, not Boolean), TIMESTAMP (expect Long, not java.sql.Timestamp) and a multi-value column (expect a one-element Object[] of the scalar default) - i.e. the invariants that make the reorder safe. Those three assertions are what turn 'happens to work' into 'guaranteed to work'.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java:
##########
@@ -1970,7 +2060,7 @@ private static void validateStarTreeIndexConfigs(List<StarTreeIndexConfig> starT
List<String> dimensionsSplitOrder = starTreeIndexConfig.getDimensionsSplitOrder();
assert CollectionUtils.isNotEmpty(dimensionsSplitOrder);
for (String dimension : dimensionsSplitOrder) {
- if (timestampIndexColumns.contains(dimension)) {
+ if (timestampIndexColumns.contains(dimension) && schema.getFieldSpecFor(dimension) == null) {
Review Comment:
This is the change in the PR most likely to break existing users, and it has nothing to do with VARIANT. Adding `&& schema.getFieldSpecFor(dimension) == null` means TIMESTAMP-index derived columns that ARE declared in the schema now go through full star-tree validation instead of being skipped - at three call sites in this method.
To spell out the mechanism: previously any name in timestampIndexColumns was waved through all three loops unconditionally. Now it is only skipped when the schema has no field spec for it, so whenever the field spec IS present the column must satisfy every star-tree check - existence, not MAP, not VARIANT, single-value, and the dictionary-encoding/cardinality rules further down.
Whether the field spec is present at validation time is not obvious, and that is the problem. TimestampIndexUtils.applyTimestampIndex() mutates the Schema object in place to add the derived columns, and it is invoked from SegmentGeneratorConfig, IndexLoadingConfig, SegmentProcessorConfig and HelixInstanceDataManager. So presence depends on whether the Schema instance reaching TableConfigUtils.validate() has already been through one of those paths - which is very plausibly why the original code skipped unconditionally rather than reasoning about it. Flipping that to a conditional makes validation outcome depend on Schema-instance provenance, and table-config validation failures surface on controller restart and on any subsequent config PUT, so if it does trigger it can block unrelated operational changes on a live table.
I am not arguing against the intent - the updated javadoc's reasoning is sound, a user column colliding with a derived name should not bypass checks. But this is three one-line changes with no test, in a 15k-line diff about something else. Please add: (a) a test with a schema that declares $ts$DAY plus a star-tree config referencing it, asserting the intended outcome; (b) a clearly-labelled separate commit; (c) a note in the PR description, since this can change validation results for tables that contain no VARIANT column at all.
##########
pinot-query-planner/src/main/java/org/apache/pinot/calcite/sql/fun/PinotOperatorTable.java:
##########
@@ -470,6 +472,11 @@ public void lookupOperatorOverloads(SqlIdentifier opName, @Nullable SqlFunctionC
if (!opName.isSimple()) {
return;
}
+ if (!_nullHandlingEnabled && TransformFunctionType.requiresNullHandling(opName.getSimple())) {
Review Comment:
Agreed on throwing a QueryException instead of IllegalStateException. Concrete optimization below: reuse the canonicalName this method already computes, and gate on the map hit so the check stops running for every operator in every query.
Two separate problems, both fixable in this one method.
(1) WRONG LAYER FOR THE ERROR - agreed this should be a QueryException. lookupOperatorOverloads is Calcite's name-resolution hook; an IllegalStateException thrown from there is not mapped by Pinot's error handling, so a user who forgot enableNullHandling=true gets an internal planner failure with a Calcite stack trace instead of a clean error, and the actionable 'set enableNullHandling=true' text gets buried. It is also an odd contract for a lookup method, which Calcite may call speculatively while resolving a name.
While changing it, please also pick ONE error code for this condition. Right now the same user mistake reports differently depending on the path: TypeCapabilityValidationVisitor.validateResultSchema uses QUERY_PLANNING, while SelectionResultsBlock.getDataTable and BaseSingleStageBrokerRequestHandler.validateRawVariantResult both use QUERY_VALIDATION - for the identical 'raw VARIANT needs null handling' message. Clients keying on error codes will see two codes for one mistake. My preference is QUERY_VALIDATION everywhere (it is a query-option problem, not a planning failure), but consistency matters more than which one.
(2) HOW TO OPTIMIZE - the method already canonicalizes the name two lines below, so the check is doing that work twice, and it does it for every operator name in every query rather than only for names that resolve to a Pinot operator. Reorder so the existing canonicalName is reused and the gate runs only on a map hit:
String canonicalName = FunctionRegistry.canonicalize(opName.getSimple());
List<SqlOperator> operators = _operatorMap.get(canonicalName);
if (operators == null) {
return;
}
if (!_nullHandlingGatedNames.isEmpty() && _nullHandlingGatedNames.contains(canonicalName)) {
throw new QueryException(QueryErrorCode.QUERY_VALIDATION, ...);
}
operatorList.addAll(operators);
where _nullHandlingGatedNames is a field set in the constructor: nullHandlingEnabled ? Set.of() : TransformFunctionType.getNullHandlingRequiredCanonicalNames(). That gives you three things at once - zero extra String allocations (canonicalName is already computed and already canonical, so no second canonicalize call is needed), zero hash lookups on the common path (the isEmpty() check short-circuits whenever null handling is on), and the gate only fires for names that actually are Pinot operators. It also means TransformFunctionType can expose the pre-canonicalized set directly instead of canonicalizing per call; keep the existing requiresNullHandling(String) for any caller that has a raw name.
##########
pinot-core/src/main/java/org/apache/pinot/core/plan/DistinctPlanNode.java:
##########
@@ -54,6 +55,18 @@ public DistinctPlanNode(SegmentContext segmentContext, QueryContext queryContext
@Override
public Operator<DistinctResultsBlock> run() {
List<ExpressionContext> expressions = _queryContext.getSelectExpressions();
+ for (ExpressionContext expression : expressions) {
+ String column = expression.getIdentifier();
+ if (column != null) {
+ DataType dataType = _indexSegment.getDataSource(column, _queryContext.getSchema())
Review Comment:
Perf: this adds an _indexSegment.getDataSource(column, schema) call per select expression per segment on every DISTINCT query, ahead of the dictionary fast path - paid by every table whether or not a VARIANT column exists anywhere.
AggregationFunctionUtils.validateRawVariantIdentifierInputs does the same on the aggregation path, and buildAggregationInfoWithStarTree now calls it before even attempting the star-tree route. Each call is cheap, but these are two hot per-segment planning paths and the cost is unconditional. A one-shot check on the QueryContext schema for 'any VARIANT column' (which is already at hand here) would let both loops be skipped entirely for the overwhelming majority of tables. Worth a JMH or at least a fan-out sanity check on a high-segment-count table before merge.
##########
pinot-plugins/pinot-input-format/pinot-parquet/src/main/java/org/apache/pinot/plugin/inputformat/parquet/ParquetNativeRecordReader.java:
##########
@@ -122,9 +159,11 @@ public GenericRow next(GenericRow reuse)
} catch (Exception e) {
Review Comment:
Good catch moving _currentPageIdx++ ahead of extraction so a continueOnError caller cannot re-read the same row or leave hasNext() stuck at EOF. Two smaller notes on this file and its siblings.
(1) The init/rewind/close rewrites across all four readers - publish only the fully initialised replacement, suppress close exceptions into the primary, null out state on close - are a genuine robustness improvement, and independent of VARIANT. Worth their own clearly-labelled commit so they can be backported without the feature. (2) ParquetRecordReader.useAvroParquetRecordReader() now derives from the delegate instance, so it returns false before init() and after close(), where before it defaulted to true and retained its value post-close. In-repo callers are tests only, but it is a public method on a plugin class, so a downstream caller could notice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add end-to-end Parquet VARIANT support [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #19101:
URL: https://github.com/apache/pinot/pull/19101#discussion_r3781402954
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/MultistageGroupByExecutor.java:
##########
@@ -83,6 +83,13 @@ public MultistageGroupByExecutor(int[] groupKeyIds, AggregationFunction[] aggFun
_aggType = aggType;
_leafReturnFinalResult = leafReturnFinalResult;
_resultSchema = resultSchema;
+ for (int i = 0; i < groupKeyIds.length; i++) {
+ ColumnDataType dataType = resultSchema.getColumnDataType(i);
+ if (!dataType.supportsEquality() || !dataType.supportsHashing()) {
+ throw new IllegalArgumentException(
+ "Raw VARIANT values do not support GROUP BY; extract a typed path with variantGet first");
+ }
+ }
Review Comment:
Addressed on current head ae60e3de73. The executor now receives the actual logical input schema and validates inputSchema.getColumnDataType(groupKeyIds[i]); grouping-set plans use RepeatOperator’s expanded schema. AggregateOperatorTest and the upstream GroupingSetsQueriesTest reproduction cover the expanded/non-prefix key shape and pass.
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/SortOperator.java:
##########
@@ -73,6 +74,11 @@ public SortOperator(OpChainExecutionContext context, MultiStageOperator input, S
// - There is no collation
// - Input is already sorted
List<RelFieldCollation> collations = node.getCollations();
+ for (RelFieldCollation collation : collations) {
+ Preconditions.checkArgument(
+ _dataSchema.getColumnDataType(collation.getFieldIndex()).supportsOrdering(),
+ "ORDER BY does not support raw VARIANT values; extract a typed path with variantGet first");
Review Comment:
Follow-up on current head ae60e3de73: added explicit non-VARIANT opaque-type diagnostics for SortOperator, SortedMailboxReceiveOperator, OrderByComparatorFactory, the planner capability visitor, and set operators. These assert the actual OBJECT/MAP type while preserving VARIANT-specific variantGet guidance.
##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/validation/TypeCapabilityValidationVisitor.java:
##########
@@ -0,0 +1,185 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.query.planner.validation;
+
+import java.util.List;
+import org.apache.calcite.rel.RelFieldCollation;
+import org.apache.pinot.common.utils.DataSchema;
+import org.apache.pinot.common.utils.VariantUtils;
+import org.apache.pinot.query.planner.logical.RexExpression;
+import org.apache.pinot.query.planner.plannode.AggregateNode;
+import org.apache.pinot.query.planner.plannode.JoinNode;
+import org.apache.pinot.query.planner.plannode.PlanNode;
+import org.apache.pinot.query.planner.plannode.PlanNodeVisitor;
+import org.apache.pinot.query.planner.plannode.SetOpNode;
+import org.apache.pinot.query.planner.plannode.SortNode;
+import org.apache.pinot.query.planner.plannode.WindowNode;
+import org.apache.pinot.spi.exception.QueryErrorCode;
+import org.apache.pinot.spi.exception.QueryException;
+
+
+/// Rejects operations that would otherwise assign physical byte ordering, equality, or hashing semantics to a raw
+/// VARIANT value. The visitor has no mutable state and is thread-safe, so callers may share {@link #INSTANCE}.
+public final class VariantTypeValidationVisitor extends PlanNodeVisitor.DepthFirstVisitor<Void, Void> {
+ public static final VariantTypeValidationVisitor INSTANCE = new VariantTypeValidationVisitor();
+
+ private VariantTypeValidationVisitor() {
+ }
+
+ @Override
+ protected boolean traverseStageBoundary() {
+ return false;
+ }
+
+ @Override
+ public Void visitAggregate(AggregateNode node, Void context) {
+ List<PlanNode> inputs = node.getInputs();
+ if (inputs.size() == 1) {
+ validateAggregateInputs(node, inputs.get(0).getDataSchema());
+ }
+ return super.visitAggregate(node, context);
+ }
+
+ /// Validates aggregate operands against their logical input schema.
+ ///
+ /// <p>This method is also invoked by the runtime as a defensive check for plans that did not pass through the
+ /// current broker planner.
+ public static void validateAggregateInputs(AggregateNode node, DataSchema inputSchema) {
Review Comment:
Follow-up on current head ae60e3de73: planner group-key validation remains in TypeCapabilityValidationVisitor (renamed to reflect its generic role), and runtime validation now uses the exact group-key IDs against the actual expanded input schema. The grouping-set regression is covered in both AggregateOperatorTest and the integration suite.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add end-to-end Parquet VARIANT support [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #19101:
URL: https://github.com/apache/pinot/pull/19101#discussion_r3920905680
##########
pinot-spi/src/main/java/org/apache/pinot/spi/data/FieldSpec.java:
##########
@@ -684,6 +691,9 @@ protected void appendDefaultNullValue(ObjectNode jsonNode) {
case BYTES:
jsonNode.put(key, BytesUtils.toHexString((byte[]) _defaultNullValue));
break;
+ case VARIANT:
Review Comment:
Collapsed BYTES and VARIANT into the shared serialization switch arm. The conversion arms remain separate because only VARIANT validates non-empty envelopes.
##########
pinot-common/src/main/java/org/apache/pinot/common/utils/DataSchema.java:
##########
@@ -465,6 +473,56 @@ public ColumnDataType getStoredType() {
return _storedColumnDataType;
}
+ public boolean supportsEquality() {
Review Comment:
Kept the capability predicates as generic metadata but scoped every new enforcement caller specifically to raw VARIANT. The generic capability visitor was replaced with `RawVariantValidationVisitor`, so arrays/OBJECT/MAP are not newly rejected.
##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/validation/RawVariantValidationVisitor.java:
##########
@@ -0,0 +1,199 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.query.planner.validation;
+
+import java.util.List;
+import org.apache.calcite.rel.RelFieldCollation;
+import org.apache.pinot.common.utils.DataSchema;
+import org.apache.pinot.common.utils.VariantUtils;
+import org.apache.pinot.query.planner.logical.RexExpression;
+import org.apache.pinot.query.planner.plannode.AggregateNode;
+import org.apache.pinot.query.planner.plannode.JoinNode;
+import org.apache.pinot.query.planner.plannode.PlanNode;
+import org.apache.pinot.query.planner.plannode.PlanNodeVisitor;
+import org.apache.pinot.query.planner.plannode.SetOpNode;
+import org.apache.pinot.query.planner.plannode.SortNode;
+import org.apache.pinot.query.planner.plannode.WindowNode;
+import org.apache.pinot.spi.exception.QueryErrorCode;
+import org.apache.pinot.spi.exception.QueryException;
+
+
+/// Validates that each logical input type supports the capabilities required by its operation, including equality,
+/// hashing, ordering, aggregation, and lossless result projection. The visitor has no mutable state and is thread-safe,
+/// so callers may share {@link #INSTANCE}.
+public final class TypeCapabilityValidationVisitor extends PlanNodeVisitor.DepthFirstVisitor<Void, Void> {
+ public static final TypeCapabilityValidationVisitor INSTANCE = new TypeCapabilityValidationVisitor();
+
+ private TypeCapabilityValidationVisitor() {
+ }
+
+ @Override
+ protected boolean traverseStageBoundary() {
+ return false;
+ }
+
+ @Override
+ public Void visitAggregate(AggregateNode node, Void context) {
+ List<PlanNode> inputs = node.getInputs();
+ if (inputs.size() == 1) {
+ validateAggregateInputs(node, inputs.get(0).getDataSchema());
+ }
+ return super.visitAggregate(node, context);
+ }
+
+ /// Validates aggregate operands against their logical input schema.
+ ///
+ /// <p>This method is also invoked by the runtime as a defensive check for plans that did not pass through the
+ /// current broker planner.
+ public static void validateAggregateInputs(AggregateNode node, DataSchema inputSchema) {
+ for (int key : node.getGroupKeys()) {
+ DataSchema.ColumnDataType dataType = inputSchema.getColumnDataType(key);
+ if (!dataType.supportsEquality() || !dataType.supportsHashing()) {
+ throw unsupported("GROUP BY", dataType);
+ }
+ }
+ validateAggregateInputs(node.getAggCalls(), inputSchema);
+ }
+
+ /// Validates aggregate or window-function operands against their logical input schema.
+ public static void validateAggregateInputs(List<RexExpression.FunctionCall> aggCalls, DataSchema inputSchema) {
+ for (RexExpression.FunctionCall aggCall : aggCalls) {
+ if (isRawVariantIndependent(aggCall)) {
+ continue;
+ }
+ for (RexExpression operand : aggCall.getFunctionOperands()) {
+ DataSchema.ColumnDataType dataType = getLogicalType(operand, inputSchema);
+ if (!dataType.supportsDirectAggregation()) {
+ throw unsupported("Aggregate function " + aggCall.getFunctionName(), dataType);
+ }
+ }
+ }
+ }
+
+ /// Rejects a raw VARIANT result when query null handling is disabled. Without the null bitmap, the reserved empty
+ /// byte placeholder cannot participate in normal disabled-null semantics while also remaining distinguishable from
+ /// an encoded Variant null.
+ public static void validateResultSchema(DataSchema resultSchema, boolean nullHandlingEnabled) {
+ if (VariantUtils.requiresNullHandlingForRawVariantResult(resultSchema, nullHandlingEnabled)) {
+ throw new QueryException(QueryErrorCode.QUERY_PLANNING,
+ VariantUtils.RAW_VARIANT_REQUIRES_NULL_HANDLING_ERROR);
+ }
+ }
+
+ @Override
+ public Void visitSort(SortNode node, Void context) {
+ DataSchema dataSchema = node.getDataSchema();
+ for (RelFieldCollation collation : node.getCollations()) {
+ int fieldIndex = collation.getFieldIndex();
+ DataSchema.ColumnDataType dataType = dataSchema.getColumnDataType(fieldIndex);
+ if (!dataType.supportsOrdering()) {
+ throw unsupported("ORDER BY", dataType);
+ }
+ }
+ return super.visitSort(node, context);
+ }
+
+ @Override
+ public Void visitSetOp(SetOpNode node, Void context) {
Review Comment:
Replaced the generic visitor with `RawVariantValidationVisitor`; it checks only raw VARIANT. Existing array/OBJECT/MAP set/sort/group/window behavior is retained, so the `ArrayToMvValidationVisitor` ordering is no longer load-bearing for this feature.
##########
pinot-query-planner/src/test/java/org/apache/pinot/query/planner/validation/TypeCapabilityValidationVisitorTest.java:
##########
@@ -0,0 +1,276 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.query.planner.validation;
+
+import java.util.List;
+import org.apache.calcite.rel.RelFieldCollation;
+import org.apache.calcite.rel.core.JoinRelType;
+import org.apache.pinot.common.utils.DataSchema;
+import org.apache.pinot.query.planner.logical.RexExpression;
+import org.apache.pinot.query.planner.plannode.AggregateNode;
+import org.apache.pinot.query.planner.plannode.JoinNode;
+import org.apache.pinot.query.planner.plannode.PlanNode;
+import org.apache.pinot.query.planner.plannode.SetOpNode;
+import org.apache.pinot.query.planner.plannode.SortNode;
+import org.apache.pinot.query.planner.plannode.ValueNode;
+import org.apache.pinot.query.planner.plannode.WindowNode;
+import org.apache.pinot.spi.exception.QueryException;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+
+public class TypeCapabilityValidationVisitorTest {
+ private static final DataSchema VARIANT_SCHEMA =
+ new DataSchema(new String[]{"payload"}, new DataSchema.ColumnDataType[]{DataSchema.ColumnDataType.VARIANT});
+ private static final DataSchema TYPED_EXTRACTION_SCHEMA =
+ new DataSchema(new String[]{"typedPayload"}, new DataSchema.ColumnDataType[]{DataSchema.ColumnDataType.STRING});
+
+ @Test
+ public void testRejectsVariantOrderBy() {
+ SortNode sortNode = new SortNode(0, VARIANT_SCHEMA, PlanNode.NodeHint.EMPTY, List.of(),
+ List.of(new RelFieldCollation(0)), 10, 0);
+
+ QueryException exception =
+ Assert.expectThrows(QueryException.class, () -> sortNode.visit(TypeCapabilityValidationVisitor.INSTANCE, null));
+ Assert.assertTrue(exception.getMessage().contains("ORDER BY"));
+ }
+
+ @Test
+ public void testNamesUnsupportedNonVariantOrderByType() {
Review Comment:
Removed the generic capability test suite with the generic visitor. The replacement suite validates raw VARIANT only; because the proposed array widening was reverted, there is no new array-rejection contract to pin.
##########
pinot-core/src/main/java/org/apache/pinot/core/query/utils/OrderByComparatorFactory.java:
##########
@@ -51,6 +52,12 @@ public static Comparator<Object[]> getComparator(List<OrderByExpressionContext>
throw new BadQueryRequestException("MV expression: " + orderByExpressions.get(i)
+ " should not be included in the ORDER-BY clause");
}
+ FieldSpec.DataType dataType = orderByColumnContexts[i].getDataType();
+ if (!dataType.supportsOrdering()) {
Review Comment:
Narrowed the single-stage ORDER BY and predicate guards to raw VARIANT. `DataType.UNKNOWN` remains allowed for `ORDER BY NULL`, and RANGE no longer rejects UNKNOWN.
##########
pinot-core/src/test/java/org/apache/pinot/core/query/utils/OrderByComparatorFactoryTest.java:
##########
@@ -104,4 +109,32 @@ public void testTwoNullsCompareNextColumn() {
assertEquals(extractColumn(_rows, COLUMN2_INDEX), Arrays.asList(1, 2, 3));
}
+
+ @Test
+ public void testRejectsRawVariant() {
Review Comment:
Added `testAllowsUnknownForOrderByNull`, which constructs an UNKNOWN column context and verifies comparator creation succeeds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add end-to-end Parquet VARIANT support [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on PR #19101:
URL: https://github.com/apache/pinot/pull/19101#issuecomment-6394643619
Split into a stack of 8 reviewable PRs. Each one is based on the previous branch, so its diff is just that layer:
1. #20027 — [VARIANT 1/8] Add VARIANT logical type foundation (pinot-spi)
2. #20028 — [VARIANT 2/8] Add Variant codec, scalar functions and VARIANT column type (pinot-common)
3. #20029 — [VARIANT 3/8] VARIANT segment storage and schema/table-config validation
4. #20030 — [VARIANT 4/8] Read Parquet VARIANT columns during ingestion
5. #20031 — [VARIANT 5/8] VARIANT single-stage query functions and opacity guards
6. #20032 — [VARIANT 6/8] VARIANT multi-stage query support and opacity guards
7. #20033 — [VARIANT 7/8] VARIANT response encoding and Java/JDBC clients
8. #20034 — [VARIANT 8/8] VARIANT compatibility suites, integration test and quickstart
I'll keep this PR open as the umbrella for the full feature description. Review is easier on the stack.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add end-to-end Parquet VARIANT support [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 closed pull request #19101: Add end-to-end Parquet VARIANT support
URL: https://github.com/apache/pinot/pull/19101
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add end-to-end Parquet VARIANT support [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19101:
URL: https://github.com/apache/pinot/pull/19101#issuecomment-5092003346
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/19101?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 0.00%. Comparing base ([`81b82b2`](https://app.codecov.io/gh/apache/pinot/commit/81b82b2685049e3e1025bef5c15c86ec3f65724b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`35a69ac`](https://app.codecov.io/gh/apache/pinot/commit/35a69acf8917b6cceae7b84b558c38be06d5fa9a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
> :exclamation: There is a different number of reports uploaded between BASE (81b82b2) and HEAD (35a69ac). Click for more details.
>
> <details><summary>HEAD has 16 uploads less than BASE</summary>
>
>| Flag | BASE (81b82b2) | HEAD (35a69ac) |
>|------|------|------|
>|unittests1|1|0|
>|unittests|2|0|
>|java-25|5|1|
>|temurin|5|1|
>|unittests2|1|0|
>|integration|3|1|
>|integration1|1|0|
>|custom-integration1|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19101 +/- ##
=============================================
- Coverage 65.50% 0.00% -65.51%
=============================================
Files 3430 3 -3427
Lines 218000 6 -217994
Branches 34642 0 -34642
=============================================
- Hits 142799 0 -142799
+ Misses 63641 6 -63635
+ Partials 11560 0 -11560
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/19101/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/19101/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/19101/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-100.00%)` | :arrow_down: |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/19101/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/19101/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (ø)` | |
| [java-25](https://app.codecov.io/gh/apache/pinot/pull/19101/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-65.51%)` | :arrow_down: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/19101/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-65.51%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/19101/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/19101/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/19101/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/pinot/pull/19101?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add end-to-end Parquet VARIANT support [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on PR #19101:
URL: https://github.com/apache/pinot/pull/19101#issuecomment-5230725515
Follow-up on my review: all four findings are now fixed and merged into this PR (head is now `8b38248f`).
| Severity | Finding | Fix |
|---|---|---|
| Medium | Comparison / `IN` / `IS DISTINCT FROM` guards rejected any non-orderable/non-equatable type (OBJECT, arrays, MAP) with a VARIANT-specific error, narrowing existing behavior | Guards now reject **only** raw `VARIANT`, in both engines: `FilterOperand`, `TransformOperandFactory`, `BinaryOperatorTransformFunction`, `InTransformFunction` |
| Medium | VARIANT OVERWRITE-only partial-upsert check only covered listed columns; unlisted columns use `defaultPartialUpsertStrategy` (and a custom `partialUpsertMergerClass`), which could silently apply INCREMENT/APPEND/UNION/IGNORE to a `PVAR` envelope | `TableConfigUtils` now validates the effective strategy for every VARIANT column and rejects a custom merger on VARIANT columns |
| Low | Opacity errors said "raw VARIANT" for any non-orderable type | Messages now name the actual type; raw-VARIANT wording + `variantGet` guidance kept only for VARIANT (`VariantTypeValidationVisitor`, `SortOperator`, `SortedMailboxReceiveOperator`, `OrderByComparatorFactory`) |
| Low | Planner did not validate `GROUP BY` keys (runtime did) | `VariantTypeValidationVisitor.validateAggregateInputs` now validates `AggregateNode` group keys |
Regression tests were added for each. spotless/checkstyle/license are clean on all four affected modules, and every affected unit-test class passes locally on JDK 25 (`VariantTableConfigValidationTest`, `FilterOperandTest`, `VariantTypeValidationVisitorTest`, and the pinot-core comparison/`IN`/`ORDER BY` suites).
On the current CI run, the Linter and all VARIANT-exercising tests pass. The three red checks are pre-existing flakes unrelated to these changes and to VARIANT: `FilteredAggregationsTest.testFilterVsCase` (Unit Set 1, same as before this PR) and `testGeneratedQueries`/`testConcurrentQueries` random-query flakes in the two integration sets (no `VariantType*` failures). A re-run should clear them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add end-to-end Parquet VARIANT support [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #19101:
URL: https://github.com/apache/pinot/pull/19101#discussion_r3686893867
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/MultistageGroupByExecutor.java:
##########
@@ -83,6 +83,13 @@ public MultistageGroupByExecutor(int[] groupKeyIds, AggregationFunction[] aggFun
_aggType = aggType;
_leafReturnFinalResult = leafReturnFinalResult;
_resultSchema = resultSchema;
+ for (int i = 0; i < groupKeyIds.length; i++) {
+ ColumnDataType dataType = resultSchema.getColumnDataType(i);
+ if (!dataType.supportsEquality() || !dataType.supportsHashing()) {
+ throw new IllegalArgumentException(
+ "Raw VARIANT values do not support GROUP BY; extract a typed path with variantGet first");
+ }
+ }
Review Comment:
The raw-VARIANT GROUP BY validation is checking the first N columns in the schema (`getColumnDataType(i)`), but the executor later uses `_groupKeyIds` to select the actual group-by key columns from each row. If the group keys are not exactly `[0..N-1]`, this can (a) miss rejecting a VARIANT group key, or (b) incorrectly reject a query where a non-key column is VARIANT.
Validate the data types at the actual key indices instead of assuming keys occupy the first columns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add end-to-end Parquet VARIANT support [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #19101:
URL: https://github.com/apache/pinot/pull/19101#discussion_r4366442436
##########
docs/design/VARIANT.md:
##########
@@ -0,0 +1,355 @@
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+# Apache Pinot VARIANT — Design Document
+
+Status: draft for community and component-owner review.
+
+## 1. Motivation and goals
+
+Apache Parquet `VARIANT(1)` provides a portable binary representation for
+semi-structured values. Pinot should ingest that representation without converting
+every row through a JSON tree, retain the complete value for late-bound queries, and
+let users materialize frequently queried paths into ordinary Pinot columns.
+
+This design has five goals:
+
+1. Define a first-class Pinot `VARIANT` logical type with an explicit persisted and
+ query-wire contract.
+2. Ingest top-level, non-repeated Parquet `VARIANT(1)` values, including unshredded,
+ fully shredded, and partially shredded layouts.
+3. Provide Spark-style parse, extraction, existence, type, null, and JSON-rendering
+ functions in both Pinot query engines.
+4. Fail explicitly when a raw Variant value reaches an operation whose semantics are
+ undefined, while allowing users to extract a typed scalar for that operation.
+5. Ship an executable table-definition, ingestion, and query story with integration
+ coverage.
+
+### 1.1 Non-goals
+
+The initial contract does not include:
+
+- nested or repeated Parquet Variant columns;
+- Pinot multi-value `VARIANT` columns;
+- streaming-format Variant ingestion;
+- quoted object keys in Variant paths;
+- semantic equality, hashing, or ordering of raw Variant values;
+- mixed-version execution for queries that contain a Variant type; or
+- automatic rollback after a Variant table has been activated.
+
+## 2. User model
Review Comment:
Documented in `docs/design/VARIANT.md` §2.1 in cb407e5149: this PR leaves the existing External Tables OPEN_STRUCT mapping unchanged; native ingestion uses an explicitly declared VARIANT destination, not a global default.
For another destination, use `variantToJson(source)` for JSON or `jsonStringToMap(variantToJson(source))` for OPEN_STRUCT; the passing reader test creates and reads a segment with both mappings, and the docs explain the native type-tag loss through JSON.
Changing an existing column's type is not an in-place toggle: use a new column/table and rebuild/reingest before switching queries.
##########
docs/design/VARIANT.md:
##########
@@ -0,0 +1,355 @@
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+# Apache Pinot VARIANT — Design Document
+
+Status: draft for community and component-owner review.
+
+## 1. Motivation and goals
+
+Apache Parquet `VARIANT(1)` provides a portable binary representation for
+semi-structured values. Pinot should ingest that representation without converting
+every row through a JSON tree, retain the complete value for late-bound queries, and
+let users materialize frequently queried paths into ordinary Pinot columns.
+
+This design has five goals:
+
+1. Define a first-class Pinot `VARIANT` logical type with an explicit persisted and
+ query-wire contract.
+2. Ingest top-level, non-repeated Parquet `VARIANT(1)` values, including unshredded,
+ fully shredded, and partially shredded layouts.
+3. Provide Spark-style parse, extraction, existence, type, null, and JSON-rendering
+ functions in both Pinot query engines.
+4. Fail explicitly when a raw Variant value reaches an operation whose semantics are
+ undefined, while allowing users to extract a typed scalar for that operation.
+5. Ship an executable table-definition, ingestion, and query story with integration
+ coverage.
+
+### 1.1 Non-goals
+
+The initial contract does not include:
+
+- nested or repeated Parquet Variant columns;
+- Pinot multi-value `VARIANT` columns;
+- streaming-format Variant ingestion;
+- quoted object keys in Variant paths;
+- semantic equality, hashing, or ordering of raw Variant values;
+- mixed-version execution for queries that contain a Variant type; or
+- automatic rollback after a Variant table has been activated.
+
+## 2. User model
+
+A Variant column is declared as a single-value dimension:
+
+```json
+{
+ "name": "payload",
+ "dataType": "VARIANT"
+}
+```
+
+The table stores `payload` in a raw forward index without a dictionary. Storage null
+handling must be enabled. A common layout retains the source value while materializing
+an indexed scalar during ingestion:
+
+```json
+{
+ "columnName": "eventType",
+ "transformFunction": "variant_get(payload, '$.eventType', 'STRING')"
+}
+```
+
+Late-bound paths remain queryable:
+
+```sql
+SET enableNullHandling=true;
+
+SELECT
+ eventId,
+ variant_get(payload, '$.user.id', 'STRING') AS userId,
+ variant_get(payload, '$.amount', 'DOUBLE') AS amount
+FROM variantEvents
+WHERE eventType = 'checkout';
+```
+
+## 3. Data flow and ownership
+
+```text
+Parquet VARIANT(1)
+ |
+ | pinot-parquet: validate shape and reconstruct shredded values
+ v
+metadata ByteBuffer + value ByteBuffer
+ |
+ | VariantEnvelope: add stable Pinot PVAR framing
+ v
+single-value raw BYTES forward index in a Pinot segment
+ |
+ | DataSchema/protobuf: carry logical VARIANT, not generic BYTES
+ v
+single-stage or multi-stage Variant functions
+ |
+ +--> typed scalar for filtering/grouping/joining/aggregation
+ |
+ +--> canonical JSON text for response clients
+```
+
+Ownership is deliberately split by module:
+
+- `pinot-spi` owns the public logical type, capability contract, and stable `PVAR`
+ envelope.
+- `pinot-segment-local` owns schema, table, index, null, and segment-metadata
+ validation.
+- `pinot-parquet` owns Parquet logical-type validation and reconstruction.
+- `pinot-common`, `pinot-core`, `pinot-query-planner`, and `pinot-query-runtime` own
+ function semantics and defensive query guards.
+- client, response, and DDL modules carry the logical type without redefining its
+ semantics.
+
+`pinot-common` depends on `parquet-variant` and its required `parquet-common` and
+`parquet-column` codec closure; the unused `parquet-jackson` artifact is excluded.
+Hadoop and record-reader integrations remain isolated to the input-format plugin.
+
+## 4. Persisted `PVAR` envelope
+
+Pinot segments need one byte sequence that contains both buffers in a Parquet Variant
+value. `VariantEnvelope` owns that framing. Version 1 is:
+
+```text
+offset size field
+0 4 ASCII magic "PVAR"
+4 1 envelope version (1)
+5 1 flags (0)
+6 2 reserved (0)
+8 4 metadata length, big-endian signed int restricted to >= 0
+12 4 value length, big-endian signed int restricted to >= 0
+16 M Parquet Variant metadata bytes
+16 + M V Parquet Variant value bytes
+```
+
+The complete envelope must fit in a Java byte array. Decoders reject unknown versions,
+non-zero flags or reserved bytes, negative lengths, bad magic, and length mismatches.
+Metadata and value bytes are copied without interpretation by the envelope layer; the
+Parquet Variant codec validates their internal representation.
+
+Version 1 is immutable and protected by a golden-byte test. A future incompatible
+framing change must use a new envelope version and retain an explicit reader for every
+supported old version. Unknown versions fail closed; they must never fall back to
+generic `BYTES`.
+
+`decode(byte[])` returns zero-copy read-only views that alias the input array. The input
+must therefore remain immutable for the lifetime of those views. Array, direct, and
+read-only source buffers are supported without changing their positions. Parquet
+ingestion copies payload bytes directly into the final envelope and must not allocate a
+full intermediate payload array.
+
+## 5. Logical type and segment contract
+
+`FieldSpec.DataType.VARIANT` is logically distinct from `BYTES` but uses variable-width
+bytes as its physical stored type.
+
+The initial schema and table constraints are:
+
+- dimension field only;
+- single-value only;
+- the reserved empty-byte default null value only;
+- no truncating or default-substitution max-length policy;
+- storage null handling enabled at schema or table level;
+- enabled raw forward index;
+- no dictionary or secondary index on the raw Variant column;
+- not a primary key, partition column, sorted column, upsert comparison column,
+ metrics-aggregation key, or star-tree dimension;
+- partial upsert may use only the `OVERWRITE` strategy; and
+- minion merge tasks (`MergeRollupTask`, `RealtimeToOfflineSegmentsTask`) may use only the `CONCAT` merge type,
+ because rollup and dedup group rows by raw stored bytes.
+
+Segment creation validates every non-null value as a complete `PVAR` envelope. Raw
+Variant columns do not publish logical minimum or maximum values and are never marked
+sorted, including single-row segments.
+
+### 5.1 Capability policy
+
+The Parquet representation is not a canonical semantic encoding: equivalent logical
+objects can use different metadata dictionaries or shredded layouts. Byte equality,
+byte hashing, and lexicographic byte order would therefore expose storage-layout
+accidents as SQL semantics.
+
+The logical type reports:
+
+| Capability on raw value | Supported |
+|---|---|
+| equality / `IN` / pattern predicates | no |
+| hashing / `GROUP BY` / `DISTINCT` | no |
+| ordering / min-max / sort / ASOF match key | no |
+| join, lookup, primary, or partition key | no |
+| direct value-consuming aggregation | no |
+| `CAST` from a raw value | no |
+| non-distinct `COUNT(raw_variant)` | yes |
+
+Planner validation provides the primary error. Runtime and single-stage guards remain
+in place for mixed plans or future planner changes. Error messages direct users to
+`variant_get` a typed path first.
+
+## 6. Parquet ingestion
+
+Reader initialization scans the file schema before publishing reader state.
+
+- A supported field is a top-level, non-repeated group annotated `VARIANT(1)`.
+- `metadata` is required binary; at least one of `value` or `typed_value` is present.
+- The parquet-java `VariantConverters` tree reconstructs fully or partially shredded
+ values.
+- The unshredded path retains the exact metadata and value buffers and adds only the
+ `PVAR` envelope.
+- Converter plans and field indexes are immutable and created once per reader.
+- A failed reinitialization leaves the previous reader usable. A failure while closing
+ the old reader does not invalidate the successfully initialized replacement.
+
+Automatic reader selection preserves existing Avro-metadata precedence for backward
+compatibility. A Parquet file with Avro metadata must explicitly select the native
+reader to ingest Variant.
+
+Malformed rows follow the existing record-reader skip policy and do not corrupt reader
+progress. Unsupported Variant spec versions and unsupported nested/repeated shapes fail
+during initialization rather than appearing as ordinary structs.
+
+## 7. Query functions
+
+Both query engines expose:
+
+- `variant_get(value, path[, targetType])`
+- `try_variant_get(value, path[, targetType])`
+- `variant_exists(value, path)`
+- `is_variant_null(value[, path])`
+- `variant_typeof(value[, path])`
+- `variant_to_json(value)`
+- `parse_json(text)` / `parse_json_to_variant(text)`
+- `try_parse_json(text)` / `try_parse_json_to_variant(text)`
+
+Function lookup remains case-insensitive under Pinot's existing registration rules.
+The v1 path grammar supports root `$`, dot-separated object fields, and non-negative
+array indexes. Supported extraction targets are `BOOLEAN`, `INT`, `LONG`, `FLOAT`,
+`DOUBLE`, `BIG_DECIMAL`, `STRING`, `BYTES`, `UUID`, `TIMESTAMP`, `VARIANT`, and `JSON`.
Review Comment:
Added the conversion/precision table in `docs/design/VARIANT.md` §7.1 (cb407e5149): extraction without a target retains VARIANT; DECIMAL → BIG_DECIMAL is exact, FLOAT/DOUBLE round, and INT/LONG require an exact integral value in range.
TIMESTAMP floors micros/nanos to milliseconds and loses sub-millisecond precision and TZ/NTZ distinction without a session-zone adjustment; temporal values cannot use LONG/STRING extraction as a workaround, and TIME has no typed target.
UUID → UUID and binary → BYTES preserve bytes; keep VARIANT for native-type fidelity, or render JSON for inspection. Ingestion transforms use these same conversions and should match the destination type.
##########
pinot-plugins/pinot-input-format/pinot-parquet/src/main/java/org/apache/pinot/plugin/inputformat/parquet/ParquetVariantConverter.java:
##########
@@ -0,0 +1,324 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.inputformat.parquet;
+
+import java.nio.ByteBuffer;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+import org.apache.parquet.example.data.Group;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.Converter;
+import org.apache.parquet.io.api.GroupConverter;
+import org.apache.parquet.io.api.PrimitiveConverter;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+import org.apache.parquet.variant.ImmutableMetadata;
+import org.apache.parquet.variant.Variant;
+import org.apache.parquet.variant.VariantBuilder;
+import org.apache.parquet.variant.VariantConverters;
+import org.apache.pinot.spi.utils.VariantEnvelope;
+
+
+/// Converts a top-level Parquet `VARIANT(1)` group into Pinot's self-describing VARIANT byte envelope.
+///
+/// Detection is based only on [LogicalTypeAnnotation.VariantLogicalTypeAnnotation], never on field names.
+/// The parquet-java [VariantConverters] implementation performs the logical reconstruction so unshredded,
Review Comment:
Added `testNestedShreddedArraysAndIntermediateResidualObjects` in cb407e5149 with a real Parquet file covering all four shapes: further-shredded object elements, arrays within arrays, value-only elements, and intermediate residual objects.
It includes empty arrays/null elements and actual object keys named `value` and `typed_value`; replay now distinguishes those keys from structural typed-value containers.
All 23 `ParquetVariantRecordReaderTest` cases passed in the 143-test focused reactor.
##########
pinot-plugins/pinot-input-format/pinot-parquet/src/main/java/org/apache/pinot/plugin/inputformat/parquet/ParquetVariantConverter.java:
##########
@@ -0,0 +1,324 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.inputformat.parquet;
+
+import java.nio.ByteBuffer;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+import org.apache.parquet.example.data.Group;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.Converter;
+import org.apache.parquet.io.api.GroupConverter;
+import org.apache.parquet.io.api.PrimitiveConverter;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+import org.apache.parquet.variant.ImmutableMetadata;
+import org.apache.parquet.variant.Variant;
+import org.apache.parquet.variant.VariantBuilder;
+import org.apache.parquet.variant.VariantConverters;
+import org.apache.pinot.spi.utils.VariantEnvelope;
+
+
+/// Converts a top-level Parquet `VARIANT(1)` group into Pinot's self-describing VARIANT byte envelope.
+///
+/// Detection is based only on [LogicalTypeAnnotation.VariantLogicalTypeAnnotation], never on field names.
+/// The parquet-java [VariantConverters] implementation performs the logical reconstruction so unshredded,
+/// shredded, and partially shredded encodings all produce complete Variant metadata and value buffers.
+///
+/// Instances retain parquet-java's converter tree and are intentionally scoped to one record reader because the
+/// converter and builder holder are mutable and not thread-safe.
+final class ParquetVariantConverter {
+ private static final byte SUPPORTED_SPEC_VERSION = 1;
+
+ private final GroupType _variantType;
+ private final int _metadataIndex;
+ private final int _valueIndex;
+ private final int _typedValueIndex;
+ private final BuilderHolder _holder = new BuilderHolder();
+ private final GroupConverter _converter;
+
+ private ParquetVariantConverter(GroupType variantType) {
+ _variantType = variantType;
+ _metadataIndex = variantType.getFieldIndex("metadata");
+ _valueIndex = variantType.containsField("value") ? variantType.getFieldIndex("value") : -1;
+ _typedValueIndex = variantType.containsField("typed_value") ? variantType.getFieldIndex("typed_value") : -1;
+ _converter = VariantConverters.newVariantConverter(variantType, _holder::setMetadata, _holder::build);
+ }
+
+ /// Validates every VARIANT annotation in the file and returns the supported top-level field names.
+ ///
+ /// Pinot currently supports only non-repeated, top-level `VARIANT(1)` values. Failing during reader
+ /// initialization avoids silently surfacing an unsupported nested/repeated Variant as an ordinary struct.
+ static Set<String> validateAndGetTopLevelVariantFields(GroupType schema) {
+ return validateAndGetTopLevelVariantFields(schema, fieldName -> true);
+ }
+
+ private static Set<String> validateAndGetTopLevelVariantFields(GroupType schema, Predicate<String> fieldSelector) {
+ Set<String> variantFields = new HashSet<>();
+ for (Type field : schema.getFields()) {
+ if (!fieldSelector.test(field.getName())) {
+ continue;
+ }
+ if (isVariant(field)) {
+ validateTopLevelVariant(field);
+ variantFields.add(field.getName());
+ } else if (!field.isPrimitive()) {
+ rejectNestedVariants(field.asGroupType(), field.getName());
+ }
+ }
+ return Set.copyOf(variantFields);
+ }
+
+ /// Validates the file schema and builds an index-aligned reusable converter tree for each top-level VARIANT column.
+ static ParquetVariantConverter[] createTopLevelVariantConverters(GroupType schema) {
+ Set<String> variantFields = validateAndGetTopLevelVariantFields(schema);
+ return buildTopLevelVariantConverters(schema, variantFields);
+ }
+
+ /// Validates selected top-level fields and builds index-aligned converters for selected VARIANT columns.
+ static ParquetVariantConverter[] createTopLevelVariantConverters(GroupType schema, Predicate<String> fieldSelector) {
+ Set<String> variantFields = validateAndGetTopLevelVariantFields(schema, fieldSelector);
+ return buildTopLevelVariantConverters(schema, variantFields);
+ }
+
+ private static ParquetVariantConverter[] buildTopLevelVariantConverters(GroupType schema,
+ Set<String> variantFields) {
+ ParquetVariantConverter[] variantConverters = new ParquetVariantConverter[schema.getFieldCount()];
+ for (int fieldIndex = 0; fieldIndex < schema.getFieldCount(); fieldIndex++) {
+ Type field = schema.getType(fieldIndex);
+ if (variantFields.contains(field.getName())) {
+ variantConverters[fieldIndex] = new ParquetVariantConverter(field.asGroupType());
+ }
+ }
+ return variantConverters;
+ }
+
+ static boolean isVariant(Type type) {
+ return type.getLogicalTypeAnnotation() instanceof LogicalTypeAnnotation.VariantLogicalTypeAnnotation;
+ }
+
+ byte[] convert(Group group) {
+ if (_valueIndex >= 0 && group.getFieldRepetitionCount(_valueIndex) == 1
+ && (_typedValueIndex < 0 || group.getFieldRepetitionCount(_typedValueIndex) == 0)) {
+ return pack(group.getBinary(_metadataIndex, 0), group.getBinary(_valueIndex, 0));
+ }
+ _holder.reset();
+ feedVariantGroup(group, _converter, _variantType, _metadataIndex);
+ Variant variant = _holder.finish();
+ return VariantEnvelope.encode(variant.getMetadataBuffer(), variant.getValueBuffer());
+ }
+
+ static byte[] pack(ByteBuffer metadata, ByteBuffer value) {
+ return VariantEnvelope.encode(metadata, value);
+ }
+
+ private byte[] pack(Binary metadata, Binary value) {
+ // Binary.writeTo(OutputStream) materializes the full payload for a direct or read-only ByteBuffer-backed Binary.
+ // Consume the independent views returned by toByteBuffer() so both payloads go straight into the final envelope,
+ // without intermediate byte arrays or another duplicate view.
+ int metadataLength = metadata.length();
+ int valueLength = value.length();
+ byte[] envelope = VariantEnvelope.allocateUnfilled(metadataLength, valueLength);
+ copyInto(metadata, envelope, VariantEnvelope.HEADER_SIZE, metadataLength, "metadata");
+ copyInto(value, envelope, VariantEnvelope.HEADER_SIZE + metadataLength, valueLength, "value");
+ return envelope;
+ }
+
+ private static void copyInto(Binary source, byte[] target, int targetOffset, int expectedLength, String payload) {
+ ByteBuffer sourceView = source.toByteBuffer();
+ if (sourceView.remaining() != expectedLength) {
+ throw new IllegalStateException(
+ "Parquet VARIANT " + payload + " length changed while copying: expected " + expectedLength + " but found "
+ + sourceView.remaining());
+ }
+ if (sourceView.hasArray()) {
+ System.arraycopy(sourceView.array(), sourceView.arrayOffset() + sourceView.position(), target, targetOffset,
+ expectedLength);
+ } else {
+ sourceView.get(target, targetOffset, expectedLength);
+ }
+ }
+
+ private static void validateTopLevelVariant(Type field) {
+ if (!isVariant(field) || field.isPrimitive()) {
+ throw new IllegalArgumentException("Parquet VARIANT must be a group: " + field);
+ }
+ LogicalTypeAnnotation.VariantLogicalTypeAnnotation annotation =
+ (LogicalTypeAnnotation.VariantLogicalTypeAnnotation) field.getLogicalTypeAnnotation();
+ if (annotation.getSpecVersion() != SUPPORTED_SPEC_VERSION) {
+ throw new UnsupportedOperationException(
+ "Unsupported Parquet VARIANT spec version: " + annotation.getSpecVersion());
+ }
+ if (field.isRepetition(Type.Repetition.REPEATED)) {
+ throw new UnsupportedOperationException("Repeated Parquet VARIANT is not supported: " + field.getName());
+ }
+
+ GroupType group = field.asGroupType();
+ for (Type child : group.getFields()) {
+ String name = child.getName();
+ if (!"metadata".equals(name) && !"value".equals(name) && !"typed_value".equals(name)) {
+ throw new IllegalArgumentException("Invalid Parquet VARIANT field: " + child);
Review Comment:
Fixed in cb407e5149: when a shredded typed-value object is present, reject residual keys declared in its shredded schema, including nested objects and absent child values, instead of inheriting parquet-java's silent preference; the raw path with an entirely absent typed parent remains unchanged.
Corrected the old positive fixture containing illegal overlap and added `testDuplicateShreddedKeysFailRowAndReaderRecovers`, which passed for root/nested overlaps and recovery to subsequent valid rows.
This is a row extraction error after physical advancement; the ingestion caller's continue-on-error policy chooses stop versus skip, not value substitution or null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org