You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "xiangfu0 (via GitHub)" <gi...@apache.org> on 2026/10/25 06:58:50 UTC

[PR] [VARIANT 1/8] Add VARIANT logical type foundation (pinot-spi) [pinot]

xiangfu0 opened a new pull request, #20027:
URL: https://github.com/apache/pinot/pull/20027

   ## What
   Adds the `VARIANT` logical type to `pinot-spi`. VARIANT is stored as variable-width `BYTES` holding a Pinot-owned `PVAR` envelope around the Parquet Variant metadata and value buffers.
   
   ## Changes
   - `FieldSpec.DataType.VARIANT` (stored type `BYTES`, appended last so existing ordinals don't shift), with capability methods `supportsEquality/Hashing/Ordering/MinMax/DirectAggregation/PatternMatching`. `equals`, `hashCode` and `compare` throw for VARIANT so a raw value can never be compared by accident.
   - `VariantEnvelope`: the versioned `PVAR` framing, validation and encode/decode, frozen with golden-byte tests.
   - `Schema` accepts VARIANT only for dimension fields and tracks `hasVariantColumn()`. `FieldSpec` compares the default-null sentinel structurally, not through value equality.
   - `PinotDataType.VARIANT`.
   - `docs/design/VARIANT.md`: the design for the whole stack (format, null semantics, ownership boundaries, rollout contract).
   - Two small type-enumeration follow-ups that would otherwise fail as soon as the enum constant exists: `TableIndexingTest` skips VARIANT, and the multi-stage `TypeFactory` maps it to Calcite `SqlTypeName.VARIANT`.
   
   ## Depends on
   Nothing; this is the bottom of the stack.
   
   ## About this PR
   This is **part 1 of 8** splitting #19101 (end-to-end Parquet VARIANT support) into layered PRs. #19101 stays open as the umbrella with the full feature description; [`docs/design/VARIANT.md`](https://github.com/apache/pinot/blob/xiangfu0/variant-split/01-spi-foundation/docs/design/VARIANT.md) (added in 1/8) is the design reference.
   
   The base branch is the previous PR in the stack, so the diff below is exactly this layer. Each layer compiles and passes its own tests on top of the one below it, and the top of the stack has the same tree as #19101.
   
   > VARIANT is not usable end to end until the whole stack has landed. Raw-value opacity guards arrive in 5/8 (single-stage) and 6/8 (multi-stage), so please don't create VARIANT columns on a build that has only part of the stack. The stack should land within one release.
   
   ### Full stack (merge bottom → top)
   _filled in once all PRs are open_
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [VARIANT 1/8] Add VARIANT logical type foundation (pinot-spi) [pinot]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #20027:
URL: https://github.com/apache/pinot/pull/20027#issuecomment-6395091516

   ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/20027?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :white_check_mark: All modified and coverable lines are covered by tests.
   :white_check_mark: Project coverage is 0.00%. Comparing base ([`083f293`](https://app.codecov.io/gh/apache/pinot/commit/083f293738090b43e9f4b9425654c9810acfbda5?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`de2b399`](https://app.codecov.io/gh/apache/pinot/commit/de2b399c4fbeb80b22e87659ff23c068aff87ab6?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 4 commits behind head on master.
   > :exclamation:  There is a different number of reports uploaded between BASE (083f293) and HEAD (de2b399). Click for more details.
   > 
   > <details><summary>HEAD has 18 uploads less than BASE</summary>
   >
   >| Flag | BASE (083f293) | HEAD (de2b399) |
   >|------|------|------|
   >|unittests1|1|0|
   >|unittests|2|0|
   >|java-25|6|2|
   >|temurin|6|2|
   >|lane-a|2|1|
   >|integration|4|2|
   >|integration1|2|0|
   >|lane-b|2|1|
   >|unittests2|1|0|
   ></details>
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@              Coverage Diff              @@
   ##             master   #20027       +/-   ##
   =============================================
   - Coverage     69.82%    0.00%   -69.83%     
   =============================================
     Files          3558        3     -3555     
     Lines        235843        6   -235837     
     Branches      37848        0    -37848     
   =============================================
   - Hits         164678        0   -164678     
   + Misses        58464        6    -58458     
   + Partials      12701        0    -12701     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/pinot/pull/20027/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [integration](https://app.codecov.io/gh/apache/pinot/pull/20027/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-100.00%)` | :arrow_down: |
   | [integration1](https://app.codecov.io/gh/apache/pinot/pull/20027/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration2](https://app.codecov.io/gh/apache/pinot/pull/20027/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (ø)` | |
   | [java-25](https://app.codecov.io/gh/apache/pinot/pull/20027/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-69.83%)` | :arrow_down: |
   | [lane-a](https://app.codecov.io/gh/apache/pinot/pull/20027/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-100.00%)` | :arrow_down: |
   | [lane-b](https://app.codecov.io/gh/apache/pinot/pull/20027/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (ø)` | |
   | [temurin](https://app.codecov.io/gh/apache/pinot/pull/20027/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-69.83%)` | :arrow_down: |
   | [unittests](https://app.codecov.io/gh/apache/pinot/pull/20027/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/20027/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/20027/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   </details>
   
   [:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/pinot/pull/20027?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [VARIANT 1/8] Add VARIANT logical type foundation (pinot-spi) [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #20027:
URL: https://github.com/apache/pinot/pull/20027#discussion_r4371629662


##########
pinot-spi/src/main/java/org/apache/pinot/spi/data/FieldSpec.java:
##########
@@ -880,6 +950,12 @@ public Object convert(String value) {
             return value;
           case BYTES:
             return BytesUtils.toBytes(value);
+          case VARIANT:
+            byte[] envelope = BytesUtils.toBytes(value);
+            if (envelope.length != 0) {
+              VariantEnvelope.decode(envelope);

Review Comment:
   At the SPI layer a non-empty user-supplied `defaultNullValue` for VARIANT is accepted here (decoded as a valid PVAR envelope), while the design restricts VARIANT to the empty-byte null sentinel. That constraint lands in the segment-local validation PR (3/8), so this is fine as-is — just flagging that SPI is intentionally permissive.



##########
pinot-spi/src/main/java/org/apache/pinot/spi/data/FieldSpec.java:
##########
@@ -880,6 +950,12 @@ public Object convert(String value) {
             return value;
           case BYTES:
             return BytesUtils.toBytes(value);
+          case VARIANT:
+            byte[] envelope = BytesUtils.toBytes(value);
+            if (envelope.length != 0) {
+              VariantEnvelope.decode(envelope);

Review Comment:
   At the SPI layer a non-empty user-supplied `defaultNullValue` for VARIANT is accepted here (decoded as a valid PVAR envelope), while the design restricts VARIANT to the empty-byte null sentinel. That constraint lands in the segment-local validation PR (3/8), so this is fine as-is — just flagging that SPI is intentionally permissive.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [VARIANT 1/8] Add VARIANT logical type foundation (pinot-spi) [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #20027:
URL: https://github.com/apache/pinot/pull/20027#discussion_r4369977429


##########
pinot-spi/src/main/java/org/apache/pinot/spi/data/FieldSpec.java:
##########
@@ -880,6 +950,12 @@ public Object convert(String value) {
             return value;
           case BYTES:
             return BytesUtils.toBytes(value);
+          case VARIANT:
+            byte[] envelope = BytesUtils.toBytes(value);
+            if (envelope.length != 0) {
+              VariantEnvelope.decode(envelope);

Review Comment:
   Minor: `convert`/`convertInternal` validate by calling `VariantEnvelope.decode(envelope)` and discarding the result, which allocates a `Decoded` plus two `ByteBuffer` slices just to throw them away. `PinotDataType.VARIANT` already uses the lighter `validateAndGetMetadataLength` for the same check. Consider using the same here so both validation paths behave and allocate identically.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org