You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "xiangfu0 (via GitHub)" <gi...@apache.org> on 2026/08/07 05:50:50 UTC
[PR] Expose common segment metadata as built-in virtual columns [pinot]
xiangfu0 opened a new pull request, #19179:
URL: https://github.com/apache/pinot/pull/19179
## Summary
Adds five built-in virtual columns that expose common segment metadata to queries, alongside the existing `$docId` / `$hostName` / `$segmentName` / `$partitionId`.
| Column | Type | Value |
|---|---|---|
| `$creationTime` | LONG | segment creation time, epoch millis |
| `$startTimeMs` | LONG | segment time-range start, normalized to epoch millis |
| `$endTimeMs` | LONG | segment time-range end, epoch millis |
| `$totalDocs` | INT | documents stored in the segment (documents indexed so far while CONSUMING) |
| `$segmentCrc` | STRING | segment CRC |
```sql
SELECT $segmentName, $segmentCrc, $totalDocs, $creationTime
FROM myTable GROUP BY 1, 2, 3, 4
```
This makes a number of operational questions answerable directly in SQL — spotting replicas of a segment whose CRC has diverged, finding segments created before a rollout, or looking at per-segment document skew — without walking the segment metadata REST API.
Like the existing built-ins, these are excluded from `SELECT *` and only materialize when named explicitly.
## Behavior
Each column is a constant single-value column within a segment. Values are read from `SegmentMetadata` every time the column is built rather than baked into the field spec, so mutable segments — which rebuild their virtual data sources on every access — always observe current metadata.
**Metadata that genuinely does not exist yet reads as SQL `NULL`, not a sentinel.** A CONSUMING segment has no time range and no CRC until it is committed, and a table without a time column never has a time range. This required teaching the virtual column path to carry a null value vector: `VirtualColumnIndexContainer` now serves `StandardIndexes.nullValueVector()`, and `VirtualColumnProvider` gained a `buildNullValueVector` hook that defaults to "no nulls", so existing providers are unaffected. Without it the placeholder in the forward index would be indistinguishable from a real value once null handling is enabled — e.g. `MIN($startTimeMs)` returning `Long.MIN_VALUE` on a consuming segment, and `IS NOT NULL` matching every row.
`$totalDocs` counts the documents physically stored in the segment, so for an upsert table it also includes documents that have been replaced and are no longer returned by queries.
## Naming
Only the time-range columns carry the `Ms` suffix. `SegmentMetadata#getStartTime()`/`#getEndTime()` return values in the time column's own unit, so unsuffixed `$startTime`/`$endTime` would be ambiguous about what they return; a creation time is always epoch millis throughout Pinot, so `$creationTime` is not. The rationale is recorded next to the constants. These strings are a permanent SQL contract, so this is worth a look during review.
## Supporting changes
- **New `BuiltInVirtualColumns` in `pinot-spi`** is the single source of each column's name, data type, and single-value/multi-value shape. The broker side (`TableCache#addBuiltInVirtualColumns`) and the server side (`VirtualColumnProviderFactory#addBuiltInVirtualColumnsToSegmentSchema`) now both build their field specs from it, so the two can no longer disagree on a type. Previously these specs were declared twice by hand, and a divergence would mean the broker declaring one type while the server produced another.
- **`DefaultNullValueVirtualColumnProvider`** grew an overridable `getValue(context)` so it can back any per-segment constant column, plus a type check that names the offending column and provider instead of throwing a bare `ClassCastException` from inside segment loading. Behavior is unchanged for existing callers.
- **Fixes a pre-existing bug in `SchemaInfo`**, which computed `getDimensionFieldSpecs().size() - 3` with a comment naming three virtual columns. That has been off by one since `$partitionId` was added, and these five columns would have made `GET /schemas/info` over-report user dimension counts by six in the controller UI. It now excludes built-in virtual columns by name, with a test pinning the invariant against the real `addToSchema` path.
## Cost
Each column adds a constant dictionary, readers, and column metadata per immutable segment — roughly 1–2 KB per segment across the five, so on the order of 35 MB on a server holding 20k segments. There is no config gate, matching the existing four built-ins.
## Testing
- New `SegmentMetadataVirtualColumnProviderTest` — values, column metadata, and the NULL path for both a missing `SegmentMetadata` and a real consuming-segment `SegmentMetadataImpl`, across the several representations of an unset creation time.
- `LoaderTest` now iterates the full built-in set and asserts both the value path and the NULL path on a really-loaded segment.
- `MutableSegmentImplTest` gained a test pinning the CONSUMING-segment shape (NULL time range/CRC/creation time, `$totalDocs` tracking documents indexed).
- `OfflineClusterIntegrationTest.testSegmentMetadataVirtualColumns` covers both query engines: data schema, per-segment `$totalDocs` summing to the table row count, millisecond normalization checked against `DaysSinceEpoch * 86_400_000`, and filters on `$creationTime` and `$segmentCrc`.
- Full `clean test` across `pinot-spi` → `pinot-common` → `pinot-segment-local` → `pinot-core` → `pinot-query-planner`, plus the integration tests above. All green.
## Docs
The virtual-columns page in `pinot-docs` needs a matching update for these five columns and the NULL-on-unavailable behavior.
## Labels
`feature`, `release-notes`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Expose common segment metadata as built-in virtual columns [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19179:
URL: https://github.com/apache/pinot/pull/19179#issuecomment-5213552356
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/19179?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `66.92308%` with `43 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 57.15%. Comparing base ([`d3604a5`](https://app.codecov.io/gh/apache/pinot/commit/d3604a585db46dcf75691887b9e9d3c7f91721d9?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`bad5922`](https://app.codecov.io/gh/apache/pinot/commit/bad59225d14138a5aa907df22c3b6b62f0087a23?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 1 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/pinot/pull/19179?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [.../column/DefaultNullValueVirtualColumnProvider.java](https://app.codecov.io/gh/apache/pinot/pull/19179?src=pr&el=tree&filepath=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Fsegment%2Findex%2Fcolumn%2FDefaultNullValueVirtualColumnProvider.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2NvbHVtbi9EZWZhdWx0TnVsbFZhbHVlVmlydHVhbENvbHVtblByb3ZpZGVyLmphdmE=) | 42.55% | [25 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/19179?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...lumn/BaseSegmentMetadataVirtualColumnProvider.java](https://app.codecov.io/gh/apache/pinot/pull/19179?src=pr&el=tree&filepath=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Fsegment%2Fvirtualcolumn%2FBaseSegmentMetadataVirtualColumnProvider.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L3ZpcnR1YWxjb2x1bW4vQmFzZVNlZ21lbnRNZXRhZGF0YVZpcnR1YWxDb2x1bW5Qcm92aWRlci5qYXZh) | 71.42% | [2 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/19179?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...nt/virtualcolumn/VirtualColumnProviderFactory.java](https://app.codecov.io/gh/apache/pinot/pull/19179?src=pr&el=tree&filepath=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Fsegment%2Fvirtualcolumn%2FVirtualColumnProviderFactory.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L3ZpcnR1YWxjb2x1bW4vVmlydHVhbENvbHVtblByb3ZpZGVyRmFjdG9yeS5qYXZh) | 81.81% | [2 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/19179?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ent/virtualcolumn/VirtualColumnIndexContainer.java](https://app.codecov.io/gh/apache/pinot/pull/19179?src=pr&el=tree&filepath=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Fsegment%2Fvirtualcolumn%2FVirtualColumnIndexContainer.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L3ZpcnR1YWxjb2x1bW4vVmlydHVhbENvbHVtbkluZGV4Q29udGFpbmVyLmphdmE=) | 50.00% | [2 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/pinot/pull/19179?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/pinot/spi/data/BuiltInVirtualColumns.java](https://app.codecov.io/gh/apache/pinot/pull/19179?src=pr&el=tree&filepath=pinot-spi%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fspi%2Fdata%2FBuiltInVirtualColumns.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvZGF0YS9CdWlsdEluVmlydHVhbENvbHVtbnMuamF2YQ==) | 80.00% | [2 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/pinot/pull/19179?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...pache/pinot/common/config/provider/TableCache.java](https://app.codecov.io/gh/apache/pinot/pull/19179?src=pr&el=tree&filepath=pinot-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcommon%2Fconfig%2Fprovider%2FTableCache.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vY29uZmlnL3Byb3ZpZGVyL1RhYmxlQ2FjaGUuamF2YQ==) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/19179?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...virtualcolumn/SegmentCrcVirtualColumnProvider.java](https://app.codecov.io/gh/apache/pinot/pull/19179?src=pr&el=tree&filepath=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Fsegment%2Fvirtualcolumn%2FSegmentCrcVirtualColumnProvider.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L3ZpcnR1YWxjb2x1bW4vU2VnbWVudENyY1ZpcnR1YWxDb2x1bW5Qcm92aWRlci5qYXZh) | 75.00% | [0 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/pinot/pull/19179?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (d3604a5) and HEAD (bad5922). Click for more details.
>
> <details><summary>HEAD has 16 uploads less than BASE</summary>
>
>| Flag | BASE (d3604a5) | HEAD (bad5922) |
>|------|------|------|
>|unittests|2|1|
>|java-25|5|1|
>|temurin|5|1|
>|unittests2|1|0|
>|integration|3|0|
>|integration1|1|0|
>|integration2|1|0|
>|custom-integration1|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19179 +/- ##
============================================
- Coverage 66.62% 57.15% -9.47%
+ Complexity 1423 1 -1422
============================================
Files 3443 2653 -790
Lines 218577 158995 -59582
Branches 34792 26124 -8668
============================================
- Hits 145624 90880 -54744
+ Misses 61218 60335 -883
+ Partials 11735 7780 -3955
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/19179/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/19179/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/19179/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/19179/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/19179/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-25](https://app.codecov.io/gh/apache/pinot/pull/19179/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.15% <66.92%> (-9.47%)` | :arrow_down: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/19179/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.15% <66.92%> (-9.47%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/19179/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.15% <66.92%> (-9.47%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/19179/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.15% <66.92%> (+0.03%)` | :arrow_up: |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/19179/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/pinot/pull/19179?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Expose common segment metadata as built-in virtual columns [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on PR #19179:
URL: https://github.com/apache/pinot/pull/19179#issuecomment-5237693043
Documentation follow-up: https://github.com/pinot-contrib/pinot-docs/pull/971 (merged)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Expose common segment metadata as built-in virtual columns [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 merged PR #19179:
URL: https://github.com/apache/pinot/pull/19179
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org