You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "xiangfu0 (via GitHub)" <gi...@apache.org> on 2026/08/19 05:39:00 UTC
[PR] Add T64 and GORILLA packing transform codecs [pinot]
xiangfu0 opened a new pull request, #19306:
URL: https://github.com/apache/pinot/pull/19306
Stacked on #19305 (`xiangfu0/codex/codec-stack/03-delta-codecs`). Part of the split of #18229.
## Summary
Adds the two **packing transform** codecs to the package-private codec runtime in
`pinot-segment-local` `org.apache.pinot.segment.local.io.codec`:
- **`T64CodecDefinition`** — frame-of-reference (FOR) bit-packing over fixed 64-value blocks for
SV INT/LONG. Each block stores `[baseline][bitWidth][bit-packed deltas]` inside a
`[flag][count]` frame. Partial last blocks still emit full 64-slot packed payloads (zero-filled
tail) so payload size scales with block count.
- **`GorillaCodecDefinition`** — Gorilla-style XOR compression (Facebook VLDB 2015) for SV
INT/LONG: verbatim first value, then per-value `0` bit for repeats or
`1 + window metadata + meaningful bits`, with window reuse between consecutive XOR deltas.
Both are `CodecKind.TRANSFORM` stages with `preservesTypedValueLayout() == false`: they consume a
typed value array and emit a headered byte frame, so the existing `CodecPipelineValidator`
structural rule automatically enforces that they must be the **last transform** (only compression
stages may follow). No validator changes were needed — the rule generalizes from the synthetic
`PACKING` handler introduced in PR 02.
Both are registered in `CodecRegistry.DEFAULT` (names `T64` / `GORILLA` are frozen on-disk
contract strings). Both codecs are corrupt-segment defensive: header flag/count validation runs
before any count-based short-circuit, bit widths and window `(leading, width)` pairs are
range-checked, and `decodeInto` enforces dst-capacity bounds.
## Tests
- `T64CodecDefinitionTest` — round-trips for empty/single/all-equal/partial-block/max-range
inputs; a parameterized sweep over every bitWidth 1..32 (INT) and 1..64 (LONG); an on-disk
byte-layout pin test; encoded-size-within-`maxEncodedSize` bound checks; corrupt-segment
defenses (bad flag, bad bitWidth, negative count, frame/column type mismatch).
- `GorillaCodecDefinitionTest` — round-trips for repeat/window-reuse/explicit-window transitions,
boundary values, timestamp/counter-like sequences; corrupt-segment defenses including
window-reuse-before-explicit.
- `CodecPipelineValidatorTest` — real-codec packing-rule scenarios through
`CodecRegistry.DEFAULT`: `DELTA,T64,LZ4` / `DELTADELTA,GORILLA,ZSTD(3)` / `DELTA,T64` accepted;
compression-after-packing (`T64,ZSTD(3)`, `GORILLA,SNAPPY`, `T64,LZ4,GZIP`) accepted;
`T64,DELTA` and `T64,GORILLA` rejected; type restrictions and no-arg enforcement;
`preservesTypedValueLayout()` contract pinned to `false` for both.
- `CodecRegistryTest` — `T64`/`GORILLA` now resolve from the default registry (case-insensitive).
- `TableConfigUtilsTest` — a mis-ordered `T64,DELTA,LZ4` codecSpec is still gate-rejected at
config time with `codecSpec is not supported yet for column: intCol`.
## Deliberately excluded (carried by later PRs)
- **V7 raw forward index format** (writer/reader that actually executes codec pipelines against
segment chunks) — next PR in the stack (05, from local `codec-split/07-v7-format`).
- **Opening the feature gate** (config/creator/reader/reload surfaces accepting codecSpec,
reload change-detection via canonical specs) — final PR (from local `codec-split/08-reload-enable`).
- Any public SPI exposure: both codecs are package-private inside `io.codec`; the only
cross-package entry point remains `CodecPipelineExecutor`.
## Why master stays safe with only this merged
The codecSpec feature gate is untouched and still closed: every config/creator/reader/reload
surface continues to throw `codecSpec is not supported yet for column: %s`, so no production
path can reach these codecs. The new classes are package-private additions to `io.codec` that are
only reachable through `CodecPipelineExecutor`, which nothing in production instantiates yet.
No on-disk format, wire protocol, or public API changes.
## Verification
```
./mvnw -q -T 1C install -DskipTests -Ppinot-fastdev -pl pinot-segment-local -am
./mvnw -q test -Ppinot-fastdev -pl pinot-segment-local \
-Dtest=T64CodecDefinitionTest,GorillaCodecDefinitionTest,CodecPipelineValidatorTest,CodecRegistryTest,DeltaCodecRoundTripTest,CodecPipelineExecutorTest \
-Dsurefire.failIfNoSpecifiedTests=false
./mvnw -q test -Ppinot-fastdev -pl pinot-segment-local -Dtest=TableConfigUtilsTest \
-Dsurefire.failIfNoSpecifiedTests=false
./mvnw spotless:apply license:format -pl pinot-segment-local
./mvnw checkstyle:check license:check -pl pinot-segment-local
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add T64 and GORILLA packing transform codecs [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 merged PR #19306:
URL: https://github.com/apache/pinot/pull/19306
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add T64 and GORILLA packing transform codecs [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19306:
URL: https://github.com/apache/pinot/pull/19306#issuecomment-5405176402
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/19306?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `405 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 57.48%. Comparing base ([`b81db76`](https://app.codecov.io/gh/apache/pinot/commit/b81db7680fbbd31b8819fcab0fe149e353c40a86?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`8eecfc9`](https://app.codecov.io/gh/apache/pinot/commit/8eecfc9e67b33dc8aed2a035446c5cd07e37e78d?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/pinot/pull/19306?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...segment/local/io/codec/GorillaCodecDefinition.java](https://app.codecov.io/gh/apache/pinot/pull/19306?src=pr&el=tree&filepath=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Fio%2Fcodec%2FGorillaCodecDefinition.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9pby9jb2RlYy9Hb3JpbGxhQ29kZWNEZWZpbml0aW9uLmphdmE=) | 0.00% | [213 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/19306?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...not/segment/local/io/codec/T64CodecDefinition.java](https://app.codecov.io/gh/apache/pinot/pull/19306?src=pr&el=tree&filepath=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Fio%2Fcodec%2FT64CodecDefinition.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9pby9jb2RlYy9UNjRDb2RlY0RlZmluaXRpb24uamF2YQ==) | 0.00% | [190 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/19306?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...he/pinot/segment/local/io/codec/CodecRegistry.java](https://app.codecov.io/gh/apache/pinot/pull/19306?src=pr&el=tree&filepath=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Fio%2Fcodec%2FCodecRegistry.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9pby9jb2RlYy9Db2RlY1JlZ2lzdHJ5LmphdmE=) | 0.00% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/19306?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (b81db76) and HEAD (8eecfc9). Click for more details.
>
> <details><summary>HEAD has 4 uploads less than BASE</summary>
>
>| Flag | BASE (b81db76) | HEAD (8eecfc9) |
>|------|------|------|
>|unittests|2|1|
>|java-25|6|5|
>|temurin|6|5|
>|unittests2|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## xiangfu0/codex/codec-stack/03-delta-codecs #19306 +/- ##
================================================================================
- Coverage 67.21% 57.48% -9.73%
+ Complexity 1424 7 -1417
================================================================================
Files 3482 2684 -798
Lines 223280 163583 -59697
Branches 35166 26518 -8648
================================================================================
- Hits 150080 94038 -56042
- Misses 61255 61566 +311
+ Partials 11945 7979 -3966
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/19306/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [integration](https://app.codecov.io/gh/apache/pinot/pull/19306/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `100.00% <ø> (ø)` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/19306/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `100.00% <ø> (ø)` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/19306/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (ø)` | |
| [java-25](https://app.codecov.io/gh/apache/pinot/pull/19306/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.48% <0.00%> (-9.73%)` | :arrow_down: |
| [lane-a](https://app.codecov.io/gh/apache/pinot/pull/19306/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `100.00% <ø> (ø)` | |
| [lane-b](https://app.codecov.io/gh/apache/pinot/pull/19306/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (ø)` | |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/19306/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.48% <0.00%> (-9.73%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/19306/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.48% <0.00%> (-9.73%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/19306/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.48% <0.00%> (-0.16%)` | :arrow_down: |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/19306/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/pinot/pull/19306?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org