You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "Jackie-Jiang (via GitHub)" <gi...@apache.org> on 2026/07/24 00:09:42 UTC
[PR] Support backfilling null value vector from the default null value [pinot]
Jackie-Jiang opened a new pull request, #19072:
URL: https://github.com/apache/pinot/pull/19072
## Summary
Adds an **opt-in, per-column** reload-time backfill that reconstructs a column's null value vector for segments that were created before null handling was enabled, by treating every stored value equal to the column's default null value as null. This lets operators enable null handling on a table and have `IS NULL` / null-aware queries work on already-ingested data without re-ingesting.
The reconstruction mirrors how ingestion records nulls:
- single-value: a doc is null when its stored value equals the column's default null value;
- multi-value: a doc is null when its entry is a single-element array holding the default null value.
### Opt-in and safety
The backfill is **lossy** — a genuine value that happens to equal the default null value is also marked null — so it is opt-in per column, via the null value vector index config:
```json
"fieldConfigList": [
{ "name": "status", "indexes": { "null": { "backfill": true } } }
]
```
- A new `NullValueVectorConfig` carries the `backfill` flag; its `enabled` state is still derived from null handling (column-based `isNullable` or the table-level `nullHandlingEnabled`), and the two are merged in `NullValueIndexType`'s deserializer.
- Enable it only for columns whose default null value is a non-occurring sentinel (e.g. a dimension's `MIN_VALUE`), not for metric/boolean columns whose default is a common value like `0`.
- `MAP`/complex types are rejected at table-config validation time (`NullValueIndexType.validate`), since the default (empty map) is an ordinary value and `OPEN_STRUCT`-backed maps have no single scannable parent forward index. A `TODO` is left to revisit if complex-type null handling matures.
### `nonNull` column-metadata flag
A new `nonNull` column-metadata property records the "null handling enabled, but no null values" case. A bitmap file is written only when a column actually has nulls (unchanged from today); the no-null case is recorded via this flag instead. This keeps behavior **symmetric between segment creation and backfill** — no empty bitmap files — and makes the backfill **idempotent** (a flagged column is not re-scanned on subsequent reloads) while distinguishing a null-handled-but-empty column from one that was never null-handled.
### Backward / mixed-version compatibility
- The metadata flag defaults to `false` and is additive; older segments are unaffected.
- The feature is off unless a column explicitly opts in.
- Rolling-upgrade note: once a column opts in (writes `indexes.null` to the table config), a node still running the pre-change code will fail to resolve that table's index configs (the old null-vector deserializer treats the config as an exclusive alternative and every column already gets a derived config). Enable `backfill` only after the whole cluster is upgraded; a rollback with the config still set will fail on that table.
### Tests
- `NullValueVectorHandlerTest` — backfill across SV/MV and dict/raw scalar types, selectivity (an un-opted column holding the sentinel is left untouched), the creation-time `nonNull` flag write, and idempotency on reload.
- `NullValueIndexTypeTest` — config resolution (`enabled` derived from null handling) and `validate` rejecting `MAP` opt-in.
- `TableConfigUtilsTest` — end-to-end validation: scalar opt-in passes, `MAP` opt-in is rejected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Support backfilling null value vector from the default null value [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on PR #19072:
URL: https://github.com/apache/pinot/pull/19072#issuecomment-5075835967
Pushed an update that substantially expands test coverage (the earlier patch-coverage gap was mostly the per-type comparison branches in `NullValueVectorHandler`) and addresses two issues found while reviewing:
**Coverage added**
- **Every data type**, single- and multi-value: all 11 dimension types (`INT`, `LONG`, `FLOAT`, `DOUBLE`, `BIG_DECIMAL`, `BOOLEAN`, `TIMESTAMP`, `STRING`, `JSON`, `BYTES`, `UUID`) plus all 6 metric types, whose default null values differ (`0`/`0.0` instead of `MIN_VALUE`/`-Infinity`). This covers the logical types that map onto another stored type (`BOOLEAN`→`INT`, `TIMESTAMP`→`LONG`, `JSON`→`STRING`, `UUID`→`BYTES`) and their round-trip through the segment metadata's `DEFAULT_NULL_VALUE`.
- **Schema-configured (non-default) default null values**, which resolve via a different path (`DataType.convert`) than the built-in constants. Includes a `BIG_DECIMAL` case whose configured scale (`-1.50`) differs from the stored scale (`-1.5`, trailing zeros stripped at ingestion) and therefore only matches because the comparison uses `compareTo` rather than `equals`.
- **The creation path** for the new `nonNull` column-metadata flag, **idempotency** across reloads, a column **missing from the segment**, and **config serde** (both the config itself and a full `TableConfig` round-trip).
**Fixes from this round**
- The time column is the one column whose stored null value is not always its default: ingestion substitutes the current time when the configured default falls outside the valid time range, while the metadata records the field spec's default. A backfill scan would never match, and would then record the column as containing no nulls. `TableConfigUtils` now rejects a backfill opt-in on the time column in exactly that case (an explicit in-range default is stored as-is and remains supported), sharing the range check with ingestion so the two cannot drift. Covered end to end for both branches.
- `NullValueVectorHandler` now reuses `NullValueVectorCreator` rather than serializing its own bitmap, so backfilled and freshly created vectors are written by the same code path — including the run-length encoding that `RoaringBitmapWriter` applies by default. No change to the serialized format.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Support backfilling null value vector from the default null value [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #19072:
URL: https://github.com/apache/pinot/pull/19072#discussion_r3648894675
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/nullvalue/NullValueIndexType.java:
##########
@@ -110,20 +116,53 @@ protected IndexReaderFactory<NullValueVectorReader> createReaderFactory() {
return ReaderFactory.INSTANCE;
}
+ @Override
+ public void validate(FieldIndexConfigs indexConfigs, FieldSpec fieldSpec, TableConfig tableConfig) {
+ if (indexConfigs.getConfig(this).isBackfill()) {
+ // Backfill reconstructs nulls by comparing each stored value against the column's default null value, which is
+ // only meaningful for scalar stored types. MAP (and other complex types) are not supported because:
+ // - the default null value for a MAP is an empty map — an ordinary value rather than a rare sentinel — so
+ // treating every empty map as null would be far too lossy to be safe; and
+ // - an OPEN_STRUCT-backed MAP is materialized into child columns with no single scannable parent forward
+ // index, so there is nothing coherent to scan for the parent column.
+ // TODO: Revisit MAP/complex backfill if complex-type null handling matures and a safe (non-occurring) sentinel
+ // default null value becomes available.
Review Comment:
Keeping the TODO deliberately. It sits directly under the paragraph explaining *why* MAP/complex types are unsupported (the default null value for a MAP is the empty map — an ordinary value rather than a rare sentinel — and an OPEN_STRUCT-backed MAP has no single scannable parent forward index), so it reads as a marker on documented design rationale rather than an untracked task. Un-linked `TODO`s of exactly this shape are the existing convention in this module, e.g. `SegmentPreProcessor` ("TODO: Reduce segment metadata reload...") and `SegmentMetadataUtils` ("TODO: Revisit if we can save the overhead..."). Happy to swap in an issue link if a committer would prefer one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Support backfilling null value vector from the default null value [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on PR #19072:
URL: https://github.com/apache/pinot/pull/19072#issuecomment-5082511202
Documentation follow-up: https://github.com/pinot-contrib/pinot-docs/pull/939
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Support backfilling null value vector from the default null value [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19072:
URL: https://github.com/apache/pinot/pull/19072#issuecomment-5065054123
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/19072?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `20.77295%` with `164 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 56.80%. Comparing base ([`706b772`](https://app.codecov.io/gh/apache/pinot/commit/706b772334e67740187ab7642c73342dbe9e28c2?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`1453ddb`](https://app.codecov.io/gh/apache/pinot/commit/1453ddbcfd07a72947a3be048152bd3ee2ffbac7?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/pinot/pull/19072?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...egment/index/nullvalue/NullValueVectorHandler.java](https://app.codecov.io/gh/apache/pinot/pull/19072?src=pr&el=tree&filepath=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Fsegment%2Findex%2Fnullvalue%2FNullValueVectorHandler.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L251bGx2YWx1ZS9OdWxsVmFsdWVWZWN0b3JIYW5kbGVyLmphdmE=) | 5.12% | [146 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/19072?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...al/segment/index/nullvalue/NullValueIndexType.java](https://app.codecov.io/gh/apache/pinot/pull/19072?src=pr&el=tree&filepath=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Fsegment%2Findex%2Fnullvalue%2FNullValueIndexType.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L251bGx2YWx1ZS9OdWxsVmFsdWVJbmRleFR5cGUuamF2YQ==) | 62.50% | [7 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/19072?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../pinot/spi/config/table/NullValueVectorConfig.java](https://app.codecov.io/gh/apache/pinot/pull/19072?src=pr&el=tree&filepath=pinot-spi%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fspi%2Fconfig%2Ftable%2FNullValueVectorConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvY29uZmlnL3RhYmxlL051bGxWYWx1ZVZlY3RvckNvbmZpZy5qYXZh) | 40.00% | [6 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/19072?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...a/org/apache/pinot/segment/spi/ColumnMetadata.java](https://app.codecov.io/gh/apache/pinot/pull/19072?src=pr&el=tree&filepath=pinot-segment-spi%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Fspi%2FColumnMetadata.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL0NvbHVtbk1ldGFkYXRhLmphdmE=) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/19072?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (706b772) and HEAD (1453ddb). Click for more details.
>
> <details><summary>HEAD has 16 uploads less than BASE</summary>
>
>| Flag | BASE (706b772) | HEAD (1453ddb) |
>|------|------|------|
>|unittests|2|1|
>|java-25|5|1|
>|temurin|5|1|
>|integration|3|0|
>|integration1|1|0|
>|unittests2|1|0|
>|custom-integration1|1|0|
>|integration2|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19072 +/- ##
============================================
- Coverage 65.43% 56.80% -8.63%
+ Complexity 1421 1 -1420
============================================
Files 3425 2630 -795
Lines 216173 156675 -59498
Branches 34239 25627 -8612
============================================
- Hits 141449 89003 -52446
+ Misses 63340 60066 -3274
+ Partials 11384 7606 -3778
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/19072/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/19072/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/19072/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/19072/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/19072/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-25](https://app.codecov.io/gh/apache/pinot/pull/19072/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.80% <20.77%> (-8.63%)` | :arrow_down: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/19072/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.80% <20.77%> (-8.63%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/19072/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.80% <20.77%> (-8.63%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/19072/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.80% <20.77%> (-0.06%)` | :arrow_down: |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/19072/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/pinot/pull/19072?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Support backfilling null value vector from the default null value [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #19072:
URL: https://github.com/apache/pinot/pull/19072#discussion_r3648895506
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/StandardIndexes.java:
##########
@@ -97,8 +98,8 @@ public static IndexType<ForwardIndexConfig, ForwardIndexReader, ForwardIndexCrea
IndexService.getInstance().get(DICTIONARY_ID);
}
- public static IndexType<IndexConfig, NullValueVectorReader, ?> nullValueVector() {
- return (IndexType<IndexConfig, NullValueVectorReader, ?>)
+ public static IndexType<NullValueVectorConfig, NullValueVectorReader, ?> nullValueVector() {
+ return (IndexType<NullValueVectorConfig, NullValueVectorReader, ?>)
IndexService.getInstance().get(NULL_VALUE_VECTOR_ID);
}
Review Comment:
Good catch on the generic-invariance point in principle, but I checked and it is a no-op here, so I'd like to keep the narrowed type.
- **No source break in practice.** Grepping both `apache/pinot` and our downstream repo for `IndexType<IndexConfig, NullValueVectorReader, ...>` returns zero hits. Every one of the ~40 `StandardIndexes.nullValueVector()` call sites passes it to a wildcard-accepting API (`hasIndexFor`, `getIndex`, `newIndexFor`, `columnsWithIndexEnabled`, `FieldIndexConfigs.Builder#with`, Mockito `eq(...)`, set membership) or to `FieldIndexConfigs#getConfig`, which infers `C` from the index type. The only site that named the type explicitly was this index's own `ReaderFactory`, updated in this PR.
- **It aligns the accessor with every sibling.** `bloomFilter()` → `BloomFilterConfig`, `json()` → `JsonIndexConfig`, `forward()` → `ForwardIndexConfig`, `h3()` → `H3IndexConfig`, and so on. `nullValueVector()` returned the generic `IndexConfig` only because it had no dedicated config class; now that it does, this brings it into line rather than out of it.
- **The suggested alternative is worse.** Keeping `IndexConfig` and casting at call sites would make the declared type contradict the actually-registered `AbstractIndexType<NullValueVectorConfig, ...>` and force an unchecked cast in the handler to read `backfill`.
On documenting it: the PR description carries a rolling-upgrade note, since the more consequential compatibility point is that once a column opts in via `indexes.null`, a node still running the pre-change code fails to resolve that table's index configs (the old deserializer treats `indexes.null` as an exclusive alternative to the always-present derived config).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Support backfilling null value vector from the default null value [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #19072:
URL: https://github.com/apache/pinot/pull/19072#discussion_r3642169387
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/StandardIndexes.java:
##########
@@ -97,8 +98,8 @@ public static IndexType<ForwardIndexConfig, ForwardIndexReader, ForwardIndexCrea
IndexService.getInstance().get(DICTIONARY_ID);
}
- public static IndexType<IndexConfig, NullValueVectorReader, ?> nullValueVector() {
- return (IndexType<IndexConfig, NullValueVectorReader, ?>)
+ public static IndexType<NullValueVectorConfig, NullValueVectorReader, ?> nullValueVector() {
+ return (IndexType<NullValueVectorConfig, NullValueVectorReader, ?>)
IndexService.getInstance().get(NULL_VALUE_VECTOR_ID);
}
Review Comment:
Changing this helper’s generic return type from `IndexConfig` to `NullValueVectorConfig` is source-incompatible for any downstream code that typed the result as `IndexType<IndexConfig, ...>` (generic invariance), even though it’s likely binary-compatible due to type erasure. If `pinot-segment-spi` is treated as an external surface, consider whether you want to preserve source compatibility (e.g., keep the old signature and cast at call sites, or introduce a separate typed accessor) and document the change accordingly.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/nullvalue/NullValueIndexType.java:
##########
@@ -110,20 +116,53 @@ protected IndexReaderFactory<NullValueVectorReader> createReaderFactory() {
return ReaderFactory.INSTANCE;
}
+ @Override
+ public void validate(FieldIndexConfigs indexConfigs, FieldSpec fieldSpec, TableConfig tableConfig) {
+ if (indexConfigs.getConfig(this).isBackfill()) {
+ // Backfill reconstructs nulls by comparing each stored value against the column's default null value, which is
+ // only meaningful for scalar stored types. MAP (and other complex types) are not supported because:
+ // - the default null value for a MAP is an empty map — an ordinary value rather than a rare sentinel — so
+ // treating every empty map as null would be far too lossy to be safe; and
+ // - an OPEN_STRUCT-backed MAP is materialized into child columns with no single scannable parent forward
+ // index, so there is nothing coherent to scan for the parent column.
+ // TODO: Revisit MAP/complex backfill if complex-type null handling matures and a safe (non-occurring) sentinel
+ // default null value becomes available.
Review Comment:
Avoid leaving an untracked TODO in production code. Either remove it or replace it with a concrete link to an issue/design doc so it doesn’t become stale (Pinot’s agent guidelines call out TODO hygiene).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Support backfilling null value vector from the default null value [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang merged PR #19072:
URL: https://github.com/apache/pinot/pull/19072
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org