You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by xi...@apache.org on 2026/05/07 20:41:51 UTC
(pinot) branch master updated: Auto-create dictionary when secondary indexes need it (inverted, FST, range) (#17269)
This is an automated email from the ASF dual-hosted git repository.
xiangfu0 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pinot.git
The following commit(s) were added to refs/heads/master by this push:
new df39617239f Auto-create dictionary when secondary indexes need it (inverted, FST, range) (#17269)
df39617239f is described below
commit df39617239f2ab14172f4f66bdd99d9d27bcd035
Author: Xiang Fu <xi...@apache.org>
AuthorDate: Thu May 7 13:41:44 2026 -0700
Auto-create dictionary when secondary indexes need it (inverted, FST, range) (#17269)
* Auto-create dictionary when required by secondary indexes (inverted, FST, range)
When a column is configured with RAW forward index encoding but a secondary index
(inverted, FST, IFST, range) requires a dictionary, automatically create a shared
standalone dictionary alongside the raw forward index so the secondary index can
function. This unlocks "RAW forward + dict + secondary index" as a supported
column shape and supports queries that mix dict-based and raw-value predicate
evaluators on the same column.
Feature
- ForwardIndexConfig: add forwardIndexEncoding (DICTIONARY|RAW) carried on the
config so callers can request RAW forward independently of dictionary presence.
- IndexCreationContext: surface the ForwardIndexConfig via withForwardIndexConfig;
ForwardIndexCreatorFactory reads encoding from there.
- IndexType.requiresDictionary SPI hook + DictionaryIndexConfig.isDictionaryRequired
/ getIndexTypesWithDictionaryRequired drive auto-create decisions; Iterable
overload decouples from IndexService.getInstance() for tests/plugins.
- Inverted / FST / IFST / Sorted index types implement requiresDictionary() = true.
- ForwardIndexHandler: ENABLE_DICTIONARY operation creates the shared dictionary
on a RAW forward column (createDictionaryForRawForwardIndex); on dict toggle
removes range/inverted/FST so they get rebuilt against the new dict.
- InvertedIndexHandler / RangeIndexHandler: build dict-id-based indexes from raw
forward data via DictionaryBasedIndexBuilder (extracted shared helper).
- Segment metadata: new FORWARD_INDEX_ENCODING property; ColumnMetadataImpl
fromPropertiesConfiguration falls back to hasDictionary for old segments.
- BaseSegmentCreator: write FORWARD_INDEX_ENCODING per column, skip dictionary
for ComplexFieldSpec (MAP) columns.
- TableConfigUtils.validateExplicitDictionaryForRawForwardIndex: hard-fail if a
dict-requiring index is configured on a RAW column without an explicit
dictionary in FieldConfig.indexes.
- FieldIndexConfigsUtil: derive forwardIndexEncoding from FieldConfig encodingType
/ noDictionaryColumns; fail-fast if user-supplied encoding disagrees with the
derived value (only when explicitly set, not the default).
- Removed RawValueBitmapInvertedIndexCreator/Reader and
RawValueInvertedIndexFilterOperator (the embedded-dictionary inverted format
is replaced by the shared standalone dictionary + standard bitmap inverted).
- SegmentPreProcessor: pre-pass invalidates legacy raw-value embedded-dict
inverted indexes so InvertedIndexHandler rebuilds them in the dict-id format.
Query path
- PredicateEvaluatorProvider.getDictionaryUsableForFiltering: per-predicate-type
dict-drop. RAW forward + dict columns drop the dictionary when the predicate
type's only available filter operator is scan, so each predicate gets the
correct evaluator (dict-based for inverted/sorted/exact-range; raw-value for
scan) — works correctly when one query mixes both on the same column.
- FilterPlanNode REGEXP_LIKE path uses the dataSource-aware overload.
- DataFetcher: support SV and MV columns with raw forward + shared dictionary
(readDictIdsFromRawValues / readDictIdsFromRawValuesMV); dict-id paths now
branch on _useDictionary instead of conflating with hasDictionary.
- ForwardIndexHandler.disableDictionaryAndCreateRawForwardIndex supports
removing dict on a column that previously had it.
- InvertedIndexAndDictionaryBasedForwardIndexCreator: split _dictionaryEnabled
into _dictionaryPresent and _dictionaryBasedForwardIndex with documented
semantics.
FilePerIndexDirectory cleanup
- Drop the redundant getReadableFilesFor logic; getFilesFor (driven by
ForwardIndexType.getFileExtension which uses ForwardIndexEncoding) is the
single source of truth and matches the write path.
Backward compatibility
- ColumnMetadata: deprecated INDEX_NOT_FOUND alias for binary-compat.
- MutableIndexContext: deprecated 11-arg constructor bridge.
- BitSlicedRangeIndexReader continues to compute _max from cardinality when
metadata.hasDictionary() is true; ForwardIndexHandler removes the range index
on dict toggle so the rebuild always matches the metadata invariant.
- IndexCreationContext: drop redundant withForwardIndexEncoding setter on the
Builder; encoding now flows only via withForwardIndexConfig.
getForwardIndexConfig() documented as Nullable; consumers fall back to
ForwardIndexConfig.getDefault().
- ForwardIndexConfig: add DEFAULT_FORWARD_INDEX_ENCODING constant for callers
that need to distinguish "user did not specify" from "explicit DICTIONARY".
Tests
- IndexCombinationValidationTest (47): exhaustive raw + dict + secondary-index
validation matrix.
- ColumnMetadataImplTest (5): FORWARD_INDEX_ENCODING fallback for old segments
(key absent), explicit RAW + hasDictionary=true, explicit DICTIONARY,
case-insensitive parsing.
- IndexCreationContextTest: encoding flows through ForwardIndexConfig.
- ForwardIndexHandlerTest: dict toggle ↔ range index rebuild (both directions),
handler ordering contract (ForwardIndexHandler before InvertedIndexHandler).
- InvertedIndexHandlerTest: legacy raw-value embedded-dict format detection,
E2E SegmentPreProcessor invalidation + rebuild.
- PredicateEvaluatorProviderTest (14): per-predicate-type dict-drop across
EQ / NOT_EQ / IN / NOT_IN / RANGE / REGEXP_LIKE × {sorted, inverted, range,
none}, including skipIndexes paths and forward-index-disabled fallback.
- CrcUtilsTest: updated CRC values for FORWARD_INDEX_ENCODING metadata change.
- RawForwardIndexInvertedIndexTest: integration test for raw + dict + inverted.
- RawForwardIndexWithDictionaryTest (27): integration test verifying query
result equivalence between dict-encoded baseline and raw + dict columns
across SV/MV, EQ/RANGE/IN/REGEXP_LIKE, with/without skipIndexes, mixed
predicates on the same column.
Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
* Address remaining MAJOR review findings on shared-dict feature
* DataFetcher.readDictIdsFromRawValuesMV: lazy-init reusable per-type
value buffer (_rawMVValueBuffer) sized to maxNumValuesPerMVEntry to
avoid per-batch allocations under the shared-dict + RAW MV workload.
* InvertedIndexHandler: clarify that the legacy-format detector reads
big-endian (not little-endian); document the disambiguation against
the modern BitmapInvertedIndexWriter offset table.
* ForwardIndexHandler.createDictionaryForRawForwardIndex: use
try-with-resources for the SegmentDictionaryCreator, derive
dictionaryCardinality from the unique-values set actually written to
disk, guard cardinality > 0 before computing BITS_PER_ELEMENT, and
delete the in-progress marker only after secondary-index cleanup so
recovery on JVM crash is atomic.
* InvertedIndexType.requiresDictionary: drop stale comment referencing
the deleted RawValueBitmapInvertedIndex* path.
* DictionaryBasedIndexBuilder: document why per-row dictIds allocation
is bound by the addXxxMV(values, dictIds) SPI contract.
Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
* Add v3-format legacy fixture + white-box on-disk migration assertions
* Add a v3-format legacy fixture by running the existing v1 fixture
through Pinot 1.5.0's SegmentV1V2ToV3FormatConverter — packs the
legacy embedded-dict bytes into columns.psf with index_map entries.
The legacy bytes are byte-identical to the v1 fixture; the
SingleFileIndexDirectory magic-marker prefix is added by the writer
and stripped by getIndexFor on read, so the format detector sees the
same bytes either way.
* Parameterize the integration test with a TestNG DataProvider so all
assertions run against both v1 and v3 fixtures (in two separate
tables loaded into the same cluster).
* Add testSegmentMigratedOnDisk: open the server's local segment
directory after load and explicitly assert the on-disk migration
outcome — column metadata flips to hasDictionary=true, dictionary +
inverted indexes are registered, the inverted-index buffer is no
longer in the legacy embedded-dict format, and the forward index
remains present (raw forward is preserved).
Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
* Drop FilterOperatorUtilsTest cases for reverted RawValueInvertedIndex path
The two tests `testLeafFilterOperatorFallsBackToScanForNonDictionaryPredicateEvaluator`
and `testLeafFilterOperatorUsesStandardInvertedIndexWhenDictionaryBased` were
added alongside an earlier `FilterOperatorUtils` change that branched on
`predicateEvaluator.isDictionaryBased()`. That production change was reverted
because, in the shared-dict design, a column with an inverted index always has
a dictionary, so the predicate evaluator is always dictionary-based and the
non-dict branch is unreachable. The tests assert behavior of code that no
longer exists; remove them and the imports they brought in.
Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
* Range index always requires dictionary; inline ENABLE_DICTIONARY helper
* RangeIndexType.requiresDictionary now returns true. Range indexes are
built over dictionary IDs (cardinality-bounded BitSliced); a range
index on a no-dictionary column is no longer a valid configuration.
Users with raw forward + range must opt in to a shared standalone
dictionary via fieldConfig.indexes.dictionary, the same pattern as
inverted/FST/IFST.
* RangeIndexHandler: collapse handleDictionaryBasedColumn /
handleNonDictionaryBasedColumn into a single createRangeIndexForColumn
that fail-fasts if no dictionary is present (matching
InvertedIndexHandler's pattern). The raw-value branch is dead under
the new contract.
* ForwardIndexHandler.shouldDisableDictionary now consults
DictionaryIndexConfig.requiresDictionary against the new config
instead of hard-coding inverted+FST. This automatically extends the
refusal to range and any future dictionary-requiring index — a user
trying to disable the dictionary while range/inverted/FST is still
configured gets refused with a clear log message naming the offending
indexes.
* Inline isTargetForwardIndexDictionaryEncoded into its single call
site at the ENABLE_DICTIONARY case and remove the helper.
* Update ForwardIndexHandlerTest's two range tests to match the new
contract: enable-dict path now adds dict + range together, disable-
dict path removes dict + range together.
* Update SegmentPreProcessorTest's testDisableDictAndOtherIndexes
SV/MV: TEST 3 (SV) and TEST 1 (MV) now first observe that disabling
dict while keeping range is refused, then remove range from config
and observe both get dropped.
* Delete SegmentGeneratorConfigTest — its coverage is subsumed by the
later loader/handler implementation tests.
Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
* Simplify overlapping tests and align range validation with new contract
* Delete RawForwardIndexInvertedIndexTest (196 lines) — fully subsumed
by RawForwardIndexWithDictionaryTest's RAW_DICT_INV_DIMENSION column,
which covers metadata + EQ/IN/REGEXP_LIKE/inequality across both
query engines plus several skip-index paths.
* Trim InvertedIndexHandlerTest down to just the format-detector unit
tests (130 lines removed). The deleted E2E preprocessor-rebuild test
is fully covered by LegacyRawValueInvertedIndexMigrationIntegration
Test, which exercises the same flow against real v1 AND v3 fixtures
with both black-box query checks and white-box on-disk assertions.
* RangeIndexType.validate now requires a dictionary unconditionally.
Drop the old "numeric columns can have raw range without dict"
exception so the validation matches the new RangeIndexType
.requiresDictionary == true contract end-to-end (config-time
validation, segment-build-time dictionary preservation, and
reload-time fail-fast all agree).
* Update the now-stale numeric-range tests in
IndexCombinationValidationTest: rename Passes → Fails for
testRawWithRangeIndex{Numeric,Float}ColumnNoDict, and add a positive
testRawWithRangeIndexNumericColumnWithExplicitDictPasses showing the
shared-dict opt-in path. Update testMixedColumnsAllValidPasses to
drop the floatCol raw+range case which is no longer valid.
* Update TableConfigUtilsTest.testValidateFieldConfig: the existing
"RAW intCol + range with FORWARD_INDEX_DISABLED" case now expects a
dictionary-related rejection instead of being permitted.
Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
* Fix shared-dict + RAW forward path in segment creator and update tests
* ForwardIndexCreator default add/addXxx methods routed by `dictId >= 0`
/ `dictIds != null`. Under the shared-dict design that breaks: a RAW
forward column with a shared dictionary now produces real dictIds
from the dict creator but the forward creator must still be the RAW
variant. The dictId-based path then calls putDictId() on a raw
creator, which throws UnsupportedOperationException. Switch every
default method to route on `isDictionaryEncoded()` (the forward
creator's own self-description) so the dictId is ignored when the
forward index is RAW.
* TableIndexingTest range_index case now opts in to a shared standalone
dictionary for RAW columns by adding `indexes.dictionary: {}` to
the FieldConfig (mirroring fst_index). Update TableIndexingTest.csv
expected outcomes accordingly:
- raw + range on numeric/string/json/big_decimal → success (was
success-on-numeric-only before the dictionary-required change)
- raw + range on map → "Cannot create range index on MAP column"
- raw + range on bytes → "Caught exception while reading data"
(still unsupported, different layer)
- raw + fst on string/json → success (dict opt-in path)
- raw + fst on non-string sv → "of stored type other than STRING"
- raw + fst on mv → "Cannot create FST index on multi-value column"
- raw + fst on map → "of stored type other than STRING"
* ForwardIndexHandlerReloadQueriesTest: column9 (RAW forward + range
index) now opts in to a shared dictionary via FieldConfig
`indexes.dictionary: {}`. Removed column9 from `noDictionaryColumns`
to avoid the "configuration declared in two different ways" conflict.
Update assertion to expect dict on column9 since the explicit opt-in
materializes the shared standalone dictionary while the forward
index stays RAW.
Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
* Reject range index on BYTES upfront in RangeIndexType.validate
Range index needs an order over the values; BYTES has no defined
comparison semantics under range filters and previously failed at
ingestion time with a generic "Caught exception while reading data"
error. Surface the rejection at table-config validation so the user
gets a clear, actionable message ("Cannot create range index on BYTES
column: %s") at config time, mirroring the existing MAP rejection.
Update TableIndexingTest.csv expected outcomes for raw_sv_BYTES,
raw_mv_BYTES, dict_sv_BYTES, dict_mv_BYTES range_index rows to expect
the new validation error.
Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
* Move legacy RawInvertedIndex cleanup to its own file; revert range-requires-dict
* Extract all legacy raw-value inverted-index detection and removal
logic out of InvertedIndexHandler and SegmentPreProcessor into a
dedicated LegacyRawValueInvertedIndexCleanup class. This makes the
one-shot deletion (after Pinot 1.7) easy to spot and audit:
delete the file, remove the call site in SegmentPreProcessor.process,
delete the unit tests in InvertedIndexHandlerTest, and delete the
integration test + fixtures. Sunset checklist documented in the
class javadoc.
* Revert RangeIndexType.requiresDictionary back to false. Range index
supports both shapes — dict-encoded (range over dict IDs) for any
sortable type, and raw (range over values) for numeric types — and
the upstream behavior was correct as-is.
* Revert the supporting changes:
- RangeIndexHandler: restore handleDictionaryBasedColumn /
handleNonDictionaryBasedColumn split with the raw-value SV/MV
add* loops for INT/LONG/FLOAT/DOUBLE.
- ForwardIndexHandler.shouldDisableDictionary: drop the
DictionaryIndexConfig.requiresDictionary check; restore the
explicit inverted+FST refusal that allows dictionary disable when
only range is configured.
- SegmentPreProcessorTest: restore the original "disable dict on
column with range" scenario (was: dict refused → both kept; now:
range gets rebuilt as raw-value-based).
- IndexCombinationValidationTest: restore the original raw + range
on numeric columns "passes" expectations.
- TableConfigUtilsTest.testValidateFieldConfig: restore the original
"raw intCol + range with FORWARD_INDEX_DISABLED" allowed case.
- ForwardIndexHandlerReloadQueriesTest: revert column9 back to the
original noDictionaryColumns + rangeIndexColumns scenario (the
range index gets rebuilt as raw-value-based after dict disable).
- TableIndexingTest range_index case: drop the explicit dict opt-in
that was added when range required dict.
- TableIndexingTest.csv: restore from upstream baseline; only keep
the FST CSV updates (the test always opts dict in for raw + FST,
so the validation order shifts to "type other than STRING" /
"multi-value" instead of "without dictionary").
* Restore SegmentGeneratorConfigTest.java that was deleted earlier in
this PR — that change was out of scope.
Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
* Inline single-use methods to reduce unnecessary code structure
Address review feedback: collapse single-use private helpers back into
their call sites.
* ForwardIndexHandler.handleDisableDictionary — inlined into the
DISABLE_DICTIONARY case body. Three sub-cases (forward-disabled,
dict-encoded forward, raw forward) sit directly in the switch.
* InvertedIndexHandler.shouldRebuildInvertedIndex (both Reader and
Writer overloads) — inlined into the two call sites in
needUpdateIndices and updateIndices. Each call now invokes
LegacyRawValueInvertedIndexCleanup.isLegacyRawValueInvertedIndexFormat
directly.
* Operation enum: revert single-line cosmetic split.
Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
* Add mixed-predicate tests for RAW + dict + inverted index column
Each sub-case routes one predicate through the inverted-index dict path
and another through the raw-scan path on the same physical column, then
asserts an explicit non-zero expected count to defuse vacuous-pass risk.
Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
* DataFetcher cleanup; route shared-dict + RAW columns to raw-value path
Addresses Jackie-Jiang's PR comments on DataFetcher: drop the redundant
isDictionaryEncoded()-and-dictionary-non-null check, rename _useDictionary
to _dictionaryEncoded, remove hot-path Preconditions guards, and stop the
implicit per-row dictionary-lookup fallback in readDictIds/readDictIdsMV.
The two now-unused private helpers are deleted.
Add an explicit (non-default) BlockValSet#isDictionaryEncoded() and
ColumnContext#isDictionaryEncoded() so every block-value-set impl has to
declare whether dict-id reads are cheap. ProjectionBlockValSet consults
the underlying forward index encoding; the materialized-in-memory impls
return false when there is no dictionary.
Operators that previously gated on "dictionary != null" alone — group-by
key generators, distinct executors, IdentifierTransformFunction, and
every aggregation function that picks a dict-id-based or raw-value-based
path (DistinctCount[Bitmap/HLL/HLL+/SmartHLL/SmartHLL+/ULL/SmartULL/CPC/
OffHeap], BaseDistinctAggregate, BaseDistinctCountSmartSketch,
SegmentPartitionedDistinctCount, AnyValue, Mode) — now require the
forward index itself to be dict-encoded, so a column with a shared
standalone dictionary on a RAW forward index falls back to the raw-value
path instead of triggering per-row dictionary lookups.
Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
* Restore readDictIdsFromRawValues as opt-in DataFetcher API
Per @Jackie-Jiang's PR comment: keep the per-row dictionary-lookup
helpers for callers that explicitly need exact dict ids on a shared-dict
+ RAW column (e.g. DistinctCountBitmap on non-INT types where the
raw-value fallback is approximate). Surface them as
DataFetcher#fetchDictIdsFromRawValues (SV + MV) with Javadoc flagging
the O(numDocs * Dictionary#indexOf) cost. The implicit fallback from
readDictIds / readDictIdsMV remains gone — those still throw on a RAW
forward index, so callers must opt in deliberately.
No production caller is wired yet; the API is reserved for follow-up
work where the operator's correctness depends on dict-id semantics.
Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
* Update DataFetcherTest for the new dict-id contract on RAW forward index
testFetchDictIdsFromRawForwardIndexWithSharedDictionary previously called
DataFetcher#fetchDictIds and expected the implicit per-row dictionary
lookup. With the new contract, fetchDictIds throws on a RAW forward
index and callers must opt in via fetchDictIdsFromRawValues. The test
now asserts both: the throw (1) and the explicit-path dict ids (2).
Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
* Revert "Update DataFetcherTest for the new dict-id contract on RAW forward index"
This reverts commit 7e66df43fccb39e5188696d973e2926aec66111f.
* Revert "Restore readDictIdsFromRawValues as opt-in DataFetcher API"
This reverts commit c8290ec478a33d346ff685246c255cc76e3ca6de.
* Revert "DataFetcher cleanup; route shared-dict + RAW columns to raw-value path"
This reverts commit 1b45308cc69b5351ec9cd15fda3622e7f3596d52.
* DataFetcher: split readDictIds and readDictIdsFromRawValues read paths
Per @Jackie-Jiang's PR comment: readDictIds should focus on reading dict
ids directly from a dictionary-encoded forward index;
readDictIdsFromRawValues handles the explicit per-row dictionary lookup
when the forward index is RAW but a (shared) dictionary exists. Each
ColumnValueReader method now does exactly one thing.
The dispatch between the two paths moves up to the public
DataFetcher#fetchDictIds entry point so existing callers are unaffected.
A new public DataFetcher#fetchDictIdsFromRawValues (SV + MV) lets future
callers opt explicitly into the per-row dictionary lookup when they have
already verified the column is RAW + shared-dict.
Also rename the ColumnValueReader field _useDictionary to
_dictionaryEncoded to match the boolean's actual meaning, and remove the
hot-path Preconditions guards in readDictIdsFromRawValues* — the caller
contract documents the precondition and the underlying readers will
throw if invariants are violated.
Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
* DataFetcher.fetchDictIds: drop auto-fallback to readDictIdsFromRawValues
fetchDictIds (SV + MV) now reads dict ids directly from the forward
index with no implicit fallback. On a RAW + shared-dict column the
underlying ForwardIndexReader#readDictIds throws
UnsupportedOperationException — callers that genuinely need dict ids on
such a column must call the new public fetchDictIdsFromRawValues
(SV + MV) explicitly. This makes the per-row Dictionary#indexOf cost a
deliberate caller decision instead of a hidden one.
Update DataFetcherTest to assert the new contract: fetchDictIds throws
on a RAW reader, and fetchDictIdsFromRawValues returns the dict ids
produced by per-row dictionary lookups.
Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
* DefaultGroupByExecutor: route shared-dict + RAW columns to no-dict GROUP BY
The previous gate (columnContext.getDictionary() == null) routed
shared-dict + RAW columns to DictionaryBasedGroupKeyGenerator, which
calls BlockValSet#getDictionaryIdsSV → DataFetcher#fetchDictIds. With
fetchDictIds no longer falling back to per-row dictionary lookups, that
path now throws.
Extend the gate to also require the underlying forward index to be
dictionary-encoded so columns with a shared standalone dictionary on a
RAW forward index take the NoDictionarySingleColumnGroupKeyGenerator
path. Other operators (DistinctExecutorFactory, aggregation functions,
NoDictionaryMultiColumnGroupKeyGenerator, IdentifierTransformFunction)
will be migrated to the same gate in a follow-up PR.
Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
* Simplify ForwardIndexHandler.computeOperations; auto-toggle dict by index need
Refactor computeOperations into a per-column helper that decomposes into
four orthogonal questions: forward-index transition, dictionary
transition, compression-only change, and cross-cutting guards (sorted
columns, range-index format compatibility, enable-forward prerequisites,
enable-dictionary needs forward).
The dictionary decision is now a single rule: desiredDict = newIsDict ||
DictionaryIndexConfig.requiresDictionary(fieldSpec, newConf). The "force
dict on if any index requires it" path means adding an inverted index
to a raw column now auto-creates the dictionary (and the inverted index
gets built against it by InvertedIndexHandler in the same reload),
where the pre-PR behavior silently skipped this combo and left the
inverted-index request orphaned. Removing that index later auto-removes
the dictionary again.
shouldDisableDictionary now consults DictionaryIndexConfig.requiresDictionary
to refuse dict removal whenever any enabled index in the new config
still needs it (covers inverted, FST, IFST, and any future
dict-requiring index type). The on-disk inverted/FST hasIndex check was
removed — InvertedIndexHandler runs after ForwardIndexHandler and
removes orphaned indexes, so refusing dict removal based on transient
on-disk state was overly conservative.
getStatsCollector gains a requireUniqueValues parameter so the
dict-creation paths skip the no-dict-optimized collector and use the
type-specific collector that tracks unique values. Without this, the
auto-create-dict path NPEs when ClusterConfigForTable's optimized
no-dict collector is in effect.
Tests:
- testRangeIndexRebuiltOnDictionaryToggle: new SegmentPreProcessorTest
case asserting the range index is rebuilt with the right format every
time the dictionary state toggles, covering both the auto path
(driven by inverted-index add/remove) and the explicit toggle path
(noDictionaryColumns add/remove).
- testIfNeedProcess: updated to expect ENABLE_DICTIONARY when adding
inverted to a raw column on v3 (the new auto path); v1 still skips.
- testComputeOperationDisableForwardIndex TEST13: updated to expect
no operation queued when inverted is in the new config (dictionary
must stay).
Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
* ForwardIndexHandler: track encoding (DICT/RAW/null) and split ENABLE_FORWARD_INDEX
State representation:
Replace the existingHasFwd / newIsFwd booleans with FieldConfig.EncodingType
values where null means "forward index disabled / not on disk", DICTIONARY
means dict-encoded forward, and RAW means raw forward. This makes the state
space explicit and lets the compression-only branch test
existingFwdEncoding == FieldConfig.EncodingType.RAW directly instead of
calling isForwardIndexDictionaryEncoded(column).
Operation split:
ENABLE_FORWARD_INDEX is split into ENABLE_DICT_FORWARD_INDEX and
ENABLE_RAW_FORWARD_INDEX. computeColumnOperations picks the right variant
based on newFwdEncoding so the intent is explicit at the operation level.
Both variants flow through createForwardIndexIfNeeded (which already reads
the target encoding from the new config), but the post-rebuild assertions
in updateIndices are now type-specific: ENABLE_DICT_FORWARD_INDEX requires
the dictionary to remain after rebuild, ENABLE_RAW_FORWARD_INDEX requires
it to be absent.
Updated test cases in ForwardIndexHandlerTest accordingly: TESTs that
explicitly set FieldConfig.EncodingType.RAW for a forward-index-disabled
column now expect ENABLE_RAW_FORWARD_INDEX, while default-dict cases keep
ENABLE_DICT_FORWARD_INDEX.
Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
* ForwardIndexHandler: support DICT⇄RAW encoding flip with dict kept
Fix the wrong assertion in ENABLE_RAW_FORWARD_INDEX (dict + inverted +
raw forward is a valid post-rebuild state when an inverted index is
enabled — the assertion that the dictionary must be absent contradicted
the auto-keep-dictionary-when-required-by-index logic from the previous
commits) and extend the operation set to actually perform the encoding
flip when the dict transition step won't.
The forward-index transition path now has three sub-cases:
- Existing forward, new disables it → DISABLE_FORWARD_INDEX.
- Existing disabled, new re-enables it → ENABLE_DICT/RAW_FORWARD_INDEX,
reusing createForwardIndexIfNeeded.
- Forward stays on but encoding flips DICT⇄RAW → queued only when the
dict transition won't already cover the conversion (i.e. existingHasDict
== desiredDict). The DICT→RAW + drop-dict and RAW→DICT + create-dict
combos still fall out of DISABLE_DICTIONARY / ENABLE_DICTIONARY
unchanged, so the encoding-flip op is reserved for the dict-stays cases:
- DICT→RAW with dict kept (e.g. inverted index added to a dict-encoded
column without explicitly enabling dict)
- RAW→DICT with shared dict already present
For the DICT→RAW + dict kept case, add convertDictForwardToRawKeepingDictionary:
the rewrite reuses the existing rewriteDictToRawForwardIndex helper but
keeps the dictionary on disk and does not call removeDictRelatedIndexes
(secondary indexes against unchanged dict ids stay valid).
Tests:
- testFlipDictForwardToRawForwardKeepingDictionaryForInvertedIndex:
end-to-end test asserting the encoding flip on a dict-encoded INT column
with an inverted index leaves the column as dict + inverted + raw forward.
- testComputeOperationDisableDictionary TEST3: previously asserted no-op
for "disable dict + add inverted on dict-encoded column" (the pre-PR
silent-skip behavior). Updated to expect the correct op
ENABLE_RAW_FORWARD_INDEX, which performs the encoding flip while the
inverted-index requirement keeps the dictionary.
Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
* DataFetcher: drop the (shared) dictionary on RAW forward in addDataSource
Revert all the DataFetcher / DataFetcherTest churn from earlier commits
(API split, _dictionaryEncoded rename, Preconditions on hot paths, etc.)
back to master and replace it with the minimal change that solves the
shared-dict + RAW correctness issue: when forwardIndexReader.isDictionaryEncoded()
is false, pass null instead of dataSource.getDictionary() to
ColumnValueReader.
ColumnValueReader's existing per-method branches all gate on
_dictionary != null, so dropping the dictionary at construction makes
every value-read method take the raw-value path uniformly without
touching the read methods themselves. Callers that need dict ids on a
RAW + shared-dict column still propagate UnsupportedOperationException
from the underlying RAW reader's readDictIds default — that surfaces as
a clear failure for any operator that hasn't been migrated to gate on
forward-index encoding (DefaultGroupByExecutor was migrated in an earlier
commit; the remaining migrations live in the BlockValSet follow-up
branch).
Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
* ColumnContext: drop the dictionary on RAW forward at the source
Centralize the "no dictionary path on RAW forward index" rule in
ColumnContext.fromDataSource. Every consumer of columnContext.getDictionary()
now naturally routes shared-dict + RAW columns to the raw-value path
without needing per-call check:
- IdentifierTransformFunction: simplified — its inline check
(forwardIndex.isDictionaryEncoded() ? columnContext.getDictionary() : null)
is now redundant because ColumnContext returns null directly.
- DefaultGroupByExecutor: revert the extra forward-index-encoding check
added in an earlier commit; the master gate
(columnContext.getDictionary() == null) is correct again now that
ColumnContext drops the dictionary at the source.
- DictionaryBasedGroupKeyGenerator, DistinctExecutorFactory,
NoDictionaryMultiColumnGroupKeyGenerator: no change needed — they
already gate on columnContext.getDictionary() and now see null for
shared-dict + RAW.
Callers that legitimately need the underlying dictionary (e.g.
DictionaryBasedDistinctOperator iterating dict values directly) can
still get it via columnContext.getDataSource().getDictionary().
This is the same pattern applied earlier to DataFetcher.addDataSource;
moving it to ColumnContext makes the rule the single source of truth
for the operator/transform-function tree.
Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
* Revert ColumnContext dict-drop; gate dict-id consumers per call site
Per @xiangfu0's review: ColumnContext should expose both the data source
and the dictionary so upper functions can decide for themselves. Some
operators (e.g. DictionaryBasedDistinctOperator) iterate the dictionary
directly and don't care about forward-index encoding — they would lose
the dictionary if it were dropped at the source.
Restore ColumnContext.fromDataSource to master behavior. Apply the
"forward must be dict-encoded for dict-id reads" rule at each call site
that does dict-id reads:
- IdentifierTransformFunction: drop dict for shared-dict + RAW so
transformToDictIdsSV/MV doesn't get advertised.
- DefaultGroupByExecutor: route shared-dict + RAW columns to the
no-dict GROUP BY path so DictionaryBasedGroupKeyGenerator's dict-id
reads don't fire.
Both checks are null-safe against DataSource without a forward index
(test mock case).
Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
* Test: filterMv predicate evaluation on shared-dict + RAW MV column
Add INT_MV_DICT_RAW_COLUMN to BaseTransformFunctionTest, configured as
EncodingType.RAW with both an explicit dictionary and inverted index in
its FieldConfig — the canonical shared-dict + RAW shape introduced by
this PR. The values mirror INT_MV_COLUMN so callers can compare results
against the dict-encoded baseline.
testFilterMvOnSharedDictRawForwardColumn in FilterMvTransformFunctionTest
exercises every predicate in the existing IntPredicate matrix (EQ, NEQ,
RANGE, IN, NOT_IN, BETWEEN, AND, OR, NOT) and asserts:
- The data source actually has a RAW forward index plus a dictionary
on disk (sanity: the configuration produced the intended shape).
- The TransformFunction reports getDictionary() == null and
TransformResultMetadata.hasDictionary() == false. This is the signal
that drives FilterMvPredicateEvaluator -> PredicateEvaluatorProvider
to construct a raw-value matching evaluator (matchesInt etc.) instead
of a matchesDictId evaluator.
- transformToIntValuesMV produces results that match the expected
matcher applied to _intMVValues, confirming the predicate evaluator
built without a dictionary still returns correct rows.
Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
* Test: filterMv on dict + RAW + inverted, plus DataSource-aware evaluator overload
Split the existing INT_MV_DICT_RAW_COLUMN coverage into two distinct on-disk
shapes and exercise filterMv against both:
- INT_MV_DICT_RAW_COLUMN: RAW forward + explicit shared dictionary, no
secondary index.
- INT_MV_DICT_RAW_INV_COLUMN (new): RAW forward + shared dictionary + inverted
index — covers the case the user wanted documented in tests.
Both columns hold the same values as INT_MV_COLUMN. The new
testFilterMvOnSharedDictRawForwardWithInvertedColumn parameterizes over
the existing IntPredicate matrix and asserts the filterMv result is
identical for all three columns (dict-encoded baseline, RAW + dict, RAW +
dict + inverted) for every predicate. The inverted index is irrelevant
to filterMv's per-value evaluation but having it on disk must not change
the result.
Also add a DataSource-aware overload to FilterMvPredicateEvaluator so the
evaluator can pull the dictionary only when the underlying forward index
is dictionary-encoded:
public static FilterMvPredicateEvaluator forPredicate(String predicate, DataSource dataSource)
FilterMvTransformFunction now uses this overload when the inner argument
is a direct column reference (IdentifierTransformFunction); for transform
arguments it keeps the dictionary-only path. The control flow is fixed
to use an else branch so the DataSource-aware evaluator isn't overwritten
by the legacy call.
Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
* PredicateEvaluatorProvider: collapse to a single public entry point
Per @xiangfu0's review: each external caller should evaluate whether it
has a DataSource. Consolidate PredicateEvaluatorProvider's public surface
to one method:
public static PredicateEvaluator getPredicateEvaluator(Predicate predicate,
@Nullable DataSource dataSource, DataType dataType, @Nullable QueryContext queryContext)
When the caller has a column DataSource, pass it so the gating logic in
getDictionaryUsableForFiltering can pick between dict-based and raw-value
evaluation for shared-dict + RAW columns. When the caller doesn't (post-
reduction matchers, intermediate-result aggregators, computed transforms),
pass null and the evaluator is built from raw values using the supplied
data type.
The two dictionary-based getPredicateEvaluator overloads (3-arg and 4-arg)
are now private — they remain as the internal dispatch implementation but
are no longer part of the public API.
Migrations:
- FilterPlanNode: pass dataSource.getDataSourceMetadata().getDataType() through.
- BinaryOperatorTransformFunction: drill into IdentifierTransformFunction
for the column's DataSource; for non-Identifier transforms pass null.
- ExpressionFilterOperator: same Identifier-or-null pattern.
- PredicateRowMatcher, DistinctCountThetaSketchAggregationFunction: pass
null DataSource (post-reduction / intermediate-result paths have no
column to point at).
- PredicateEvaluatorProviderTest: pass FieldSpec.DataType.STRING through.
FilterMvPredicateEvaluator deliberately bypasses PredicateEvaluatorProvider
now: filterMv evaluates per-value at transform time, so the filter-plan-time
gating PredicateEvaluatorProvider applies (which would route shared-dict +
RAW columns through the dict-based evaluator) is the wrong policy at this
layer. FilterMvPredicateEvaluator builds its evaluator directly via the
Equals/NotEquals/In/NotIn/Range/RegexpLike factories based solely on
whether a dictionary is supplied — IdentifierTransformFunction already
returns null for shared-dict + RAW columns, so the existing forPredicate
(predicate, dataType, dictionary) signature is the right call surface.
Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
* Address Jackie-Jiang's open review comments
PredicateEvaluatorProvider:
- For RANGE on RAW forward, drop the dictionary unless the range index is
exact (Jackie: "we need to ensure the predicate can solely be resolved
with range index. Mixing dictionary encoded range index and raw forward
index scan will break"). Non-exact (legacy) range readers fall back to
ScanBasedFilterOperator for partial matches; that scan applies the
predicate evaluator on raw forward values, which would break a dict-based
evaluator.
- Remove the sortedAvailable check inside the RAW-forward branch
(Jackie: "this is dead code. Sorted index is run-length dictionary
encoded forward index"). A sorted forward index is dict-encoded by
definition, so it can never coexist with a RAW forward in this branch.
- Make getDictionaryUsableForFiltering package-private and annotate it
with @VisibleForTesting (Jackie's minor suggestion).
- Update the public method's docstring to clarify caller responsibilities:
leaf-filter callers may pass DataSource freely; transform-layer callers
must only pass DataSource when the inner transform's getDictionary() is
non-null (Jackie: "double check all callers of this, make sure it is
not using forward index").
BinaryOperatorTransformFunction: pass leftDataSource only when the inner
transform exposes a dictionary, mirroring IdentifierTransformFunction's
"forward index is dict-encoded" contract. For RAW (including shared-dict
+ RAW) the inner transform returns null from getDictionary(), so the
predicate evaluator is built against raw values and applySV(value) on
the raw forward output stays consistent.
ExpressionFilterOperator: always pass null DataSource. The inner is
always a function (FilterPlanNode dispatches direct column refs through
the leaf-filter path) and ExpressionScanDocIdIterator scans the transform
output applying applySV(value) on raw values, so a dict-based evaluator
would never be safe here.
DataFetcher: add Preconditions check that dictionary != null when the
forward index claims to be dictionary-encoded (Jackie: "is it possible
that forward index is dictionary encoded by dictionary doesn't exist?").
Defensive against an impossible state — fails fast at DataFetcher
construction rather than silently producing a ColumnValueReader that
will NPE later in the read path.
BaseSegmentCreator.createDictionaryForColumn: when an index requires a
dictionary, create one regardless of the user's explicit dictionary
setting (Jackie: "should we always create dictionary when any index
requires it?"). Previously, explicitly disabling the dictionary while
enabling an index that requires it would lead to segment creation
failure (e.g. inverted index can't be built without a dict) or silent
index loss. Now we log a warning and proceed.
DefaultGroupByExecutor: add a comment explaining that
ColumnContext.getDataSource() can be null for computed (non-Identifier)
transforms (Jackie: "could datasource ever be null?"); in that case
getDictionary() == null already routes those columns onto the no-dict
GROUP BY path via the first condition.
Tests: drop two tests that exercised an impossible RAW + sorted scenario
and add a new test for the non-exact range index case.
Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
* Drop redundant null guards: trust segment-loader invariants
DataFetcher.addDataSource: drop the Preconditions check that asserted a
dictionary exists when forward index claims to be dictionary-encoded.
The segment loader sets isDictionaryEncoded() in lockstep with the
dictionary file's presence — the check was over-protecting an invariant
that's already guaranteed.
IdentifierTransformFunction.forwardIndexIsDictEncoded: drop the
ColumnContext.getDataSource() null guard. IdentifierTransformFunction is
only constructed via TransformFunctionFactory with ColumnContexts built
from ColumnContext.fromDataSource(...), where the DataSource is always
non-null. The remaining `forwardIndex != null` check is the real one —
it guards forward-index-disabled columns.
Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
* Auto-enable dictionary at config time for FST/IFST/INVERTED, dedup factory dispatch
Address Jackie-Jiang's remaining review comments:
DictionaryIndexType: when a FieldConfig with encoding=RAW also configures
an index that always requires a dictionary (FST, IFST, INVERTED), do NOT
mark the dictionary as DISABLED in the deserialized config. The dict
falls through to its default-enabled state, validation passes, and the
runtime auto-creation paths (BaseSegmentCreator.createDictionaryForColumn
/ ForwardIndexHandler) build a shared-dict + RAW forward index.
This closes the loop on Jackie's "Dictionary should be generated on FST
without explicit configuration" comment — users no longer need to inject
a dictionary entry alongside FST/IFST/INVERTED on a RAW column.
PredicateEvaluatorProvider:
- Replace the now-private factory-dispatch overload with a public
buildEvaluator(predicate, dictionary, dataType, queryContext) entry
point. This is for callers whose value stream is statically known
(e.g. per-value transform evaluation in filterMv); filter-plan-time
callers still go through getPredicateEvaluator(predicate, DataSource,
...) which applies gating.
FilterMvPredicateEvaluator: drop the duplicated buildPerValueEvaluator
switch — it was a copy of the factory dispatch. Now delegates to
PredicateEvaluatorProvider.buildEvaluator.
IdentifierTransformFunction: revert the encoding-based dictionary drop.
The Identifier always exposes the column dictionary if one exists; it is
the consumer's responsibility to additionally check forward-index
encoding when deciding whether dict-id reads will be cheap. This matches
the principle that callers should make decisions with all available
information.
FilterMvTransformFunction: now performs its own forward-encoding gate
via columnContextMap to decide whether to use the dict-id matching path
or fall back to per-value matching. Also rebuilds the result metadata so
hasDictionary() reflects the gated dictionary actually used here, rather
than the inner Identifier's (which always reports the underlying dict).
BinaryOperatorTransformFunction: same gate inline — only pass DataSource
to the predicate evaluator when the LHS Identifier wraps a dict-encoded
forward index.
ForwardIndexHandler: inline the unused two-arg getStatsCollector
overload at its single call site.
TableConfigUtilsTest: update two assertions that previously expected
"Cannot create inverted index ... without dictionary" failures for
encoding=RAW + INVERTED. With auto-enable that combination now
validates and produces a shared-dict + RAW forward index.
Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
* Revert TableIndexingTest; remove redundant getStatsCollector boolean parameter
TableIndexingTest.java: the FST encoding=RAW dictionary workaround,
indexes.put→set rewrites, and getErrorMessage null-safety changes were
all unrelated to this PR's scope. Revert the file to match master.
The DictionaryIndexType.fromFieldConfigs auto-enable now handles the
FST + encoding=RAW case at config time, so the test no longer needs
the manual dictionary block.
ForwardIndexHandler.getStatsCollector: drop the requireUniqueValues
boolean parameter. The flag was redundant — when the auto-dict-creation
path runs (the new code paths that flip a column from no-dict to dict),
_fieldIndexConfigs already carries the new config with dict enabled, so
hasIndex(column, dictionary) returns true and the no-dict optimization
is already skipped, producing a per-type collector that tracks unique
values. Restoring the original two-arg signature.
Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
* Unified PredicateEvaluator entry; PredicateEvaluator handles dict-vs-raw internally
PredicateEvaluatorProvider:
- Single public entry: getPredicateEvaluator(predicate, dataSource, dictionary, dataType, queryContext).
When dataSource is non-null the gating in getDictionaryUsableForFiltering derives the dictionary;
when null the dictionary parameter is used directly. buildEvaluator (factory dispatch) is now private.
FilterMvPredicateEvaluator:
- Accept (predicate, dataType, dictionary, dataSource). Drop the dictionary internally if the inner
forward index is RAW — filterMv evaluates per-value and forward.getDictIdMV would need expensive
Dictionary#indexOf for RAW. Expose isDictionaryBased() so callers can sync their result metadata.
- forPredicate now delegates to the unified PredicateEvaluatorProvider; no more duplicated factory
dispatch.
FilterMvTransformFunction:
- Drop the dictionaryUsableForFilterMv helper; let FilterMvPredicateEvaluator do the gate. Pass the
inner Identifier's DataSource through. Build resultMetadata from the gated dictionary so downstream
consumers (ExpressionScanDocIdIterator) take the right value-stream path.
DictionaryIndexType.hasIndexRequiringDictionary:
- Replace the hardcoded FST/IFST/INVERTED list with an iteration over IndexService.getInstance()
.getAllIndexes(), checking IndexType.requiresDictionary against the IndexType's default config and
honoring the FieldConfig's per-index "disabled":true flag in the indexes JsonNode. Generalizes to
any IndexType (including future plugins) that declares requiresDictionary.
TableIndexingTest: drop the now-redundant manual dictionary block in the FST + RAW case. The
DictionaryIndexType auto-enable now handles it.
Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
* Revert TableIndexingTest.java to master — keep this file out of the PR
Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
* Fix CI unit test regressions from rebase: ExpressionFilterOperator dict, getStatsCollector flag
Two regressions surfaced in GitHub Actions Pinot Unit Test Sets:
1) NullHandlingEnabledQueriesTest.testExpressionFilterOperatorNotFilterOnMultiValue:
ExpressionFilterOperator was passing a null dictionary to the unified
getPredicateEvaluator. ExpressionScanDocIdIterator follows the inner
transform's resultMetadata.hasDictionary() to choose between the dict-id
and raw-value paths, so when the inner transform reports hasDictionary=true
it feeds dict ids into a raw-value evaluator — wrong results. Hand the
transform's getDictionary() through directly so the evaluator matches the
value stream.
2) SegmentPreProcessorTest.testRangeIndexRebuiltOnDictionaryToggle and
testIfNeedProcess[v3]: NPE in createDictionaryForRawForwardIndex because
getStatsCollector returned a NoDictColumnStatisticsCollector (no unique
values) when _fieldIndexConfigs still reported dict=disabled. This happens
on the auto-toggle path: legacy invertedIndexColumns triggers
ENABLE_DICTIONARY in computeOperations, but the deserialized
FieldIndexConfigs hasn't been mutated to reflect the auto-derived dict
requirement. Restore the requireUniqueValues parameter on getStatsCollector
and pass true at the two dict-building call sites; this forces a per-type
collector that tracks unique values regardless of the stale config flag.
Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
* Fix predicate evaluator provider
* Fix RAW-forward and non-exact range correctness gaps
FilterMvTransformFunction: gate the dictionary on the inner forward-index
encoding. IdentifierTransformFunction surfaces the column's dictionary
unchanged, but the dict-id matching path requires forward.getDictIdMV() to
serve dict ids cheaply — RAW forward indexes throw
UnsupportedOperationException from that method, so for shared-dict + RAW
columns we drop the dictionary internally and the predicate evaluator falls
back to per-value raw matching.
RangeIndexBasedFilterOperator.canEvaluate: also check that the predicate
evaluator's encoding matches the range index's encoding. RangeIndexCreator
and BitSlicedRangeIndexCreator both switch on hasDictionary at build/rebuild
time — the on-disk index stores dict IDs whenever a dictionary exists. Pairing
that index with a raw-value evaluator (e.g. our gating drops the dict on
shared-dict + RAW with a non-exact range index) silently returns wrong matches:
raw values would be compared against dict IDs by RangeIndexBasedFilterOperator.
Fall through to ScanBasedFilterOperator instead so raw values are scanned
against the raw forward index.
Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
* Fix TransformFunction
* Reject legacy range index v1 on RAW + dict; rebuild range index on version change
RangeIndexType.validate: reject the combination of RAW forward index +
dictionary + range version 1 (legacy RangeIndexCreator). The on-disk v1
range index over dict IDs is non-exact — query-time partial matches fall
back to ScanBasedFilterOperator, which on a RAW forward column would have
to apply a dict-based evaluator against raw values, silently producing
wrong results. Force the BitSliced range index (version 2, exact) for this
combination so the failure mode is config validation rather than wrong
query answers.
RangeIndexHandler.needUpdateIndices / updateIndices: also detect when the
on-disk range index version differs from the configured version. v1 and v2
have incompatible on-disk layouts and serve different query semantics
(non-exact vs exact), so a version change requires a full rebuild. Read the
version from the index buffer's first int (matching RangeIndexType.read's
dispatch) and rebuild when it doesn't match. The buffer is owned by
SegmentDirectory — don't close it from the handler, since the mmap region
is shared and closing it would crash other readers.
Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <no...@anthropic.com>
Co-authored-by: Xiaotian (Jackie) Jiang <ja...@gmail.com>
---
.../org/apache/pinot/core/common/DataFetcher.java | 7 +-
.../filter/RangeIndexBasedFilterOperator.java | 16 +-
.../predicate/PredicateEvaluatorProvider.java | 87 ++-
.../function/ArrayLiteralTransformFunction.java | 6 -
.../transform/function/BaseTransformFunction.java | 5 -
.../function/BinaryOperatorTransformFunction.java | 2 +-
.../function/FilterMvPredicateEvaluator.java | 5 +-
.../function/FilterMvTransformFunction.java | 29 +-
.../function/GenerateArrayTransformFunction.java | 7 -
.../function/IdentifierTransformFunction.java | 10 +-
.../transform/function/ItemTransformFunction.java | 82 +--
.../function/LiteralTransformFunction.java | 6 -
.../transform/function/TransformFunction.java | 4 +-
.../org/apache/pinot/core/plan/FilterPlanNode.java | 6 +-
...istinctCountThetaSketchAggregationFunction.java | 3 +-
.../groupby/DefaultGroupByExecutor.java | 11 +-
.../query/reduce/filter/PredicateRowMatcher.java | 2 +-
.../predicate/PredicateEvaluatorProviderTest.java | 260 ++++++++
.../function/BaseTransformFunctionTest.java | 32 +
.../DateTimeConversionTransformFunctionTest.java | 7 -
.../function/FilterMvTransformFunctionTest.java | 111 ++++
.../ForwardIndexHandlerReloadQueriesTest.java | 132 +++-
.../src/test/resources/TableIndexingTest.csv | 46 +-
.../custom/RawForwardIndexWithDictionaryTest.java | 601 +++++++++++++++++++
...ValueInvertedIndexMigrationIntegrationTest.java | 311 ++++++++++
.../legacy/legacyRawInverted_OFFLINE_0.tar.gz | Bin 0 -> 2815 bytes
.../legacy/legacyRawInverted_v3_OFFLINE_0.tar.gz | Bin 0 -> 2885 bytes
.../segment/creator/impl/BaseSegmentCreator.java | 13 +
.../index/dictionary/DictionaryIndexType.java | 111 +++-
.../segment/index/inverted/InvertedIndexType.java | 6 +-
.../segment/index/loader/BaseIndexHandler.java | 5 +-
.../segment/index/loader/ForwardIndexHandler.java | 530 +++++++++++-----
...IndexAndDictionaryBasedForwardIndexCreator.java | 55 +-
.../segment/index/loader/SegmentPreProcessor.java | 19 +-
.../invertedindex/DictionaryBasedIndexBuilder.java | 198 ++++++
.../loader/invertedindex/InvertedIndexHandler.java | 73 ++-
.../LegacyRawValueInvertedIndexCleanup.java | 137 +++++
.../loader/invertedindex/RangeIndexHandler.java | 61 +-
.../local/segment/index/range/RangeIndexType.java | 12 +
.../segment/readers/PinotSegmentColumnReader.java | 28 +-
.../index/dictionary/DictionaryIndexTypeTest.java | 29 +
.../index/loader/ForwardIndexHandlerTest.java | 503 ++++++++++++++--
.../index/loader/SegmentPreProcessorTest.java | 144 ++++-
.../invertedindex/InvertedIndexHandlerTest.java | 80 +++
.../segment/local/segment/readers/LazyRowTest.java | 2 +
.../utils/IndexCombinationValidationTest.java | 665 +++++++++++++++++++++
.../segment/local/utils/TableConfigUtilsTest.java | 81 ++-
.../segment/spi/index/DictionaryIndexConfig.java | 2 +-
.../spi/index/creator/ForwardIndexCreator.java | 30 +-
.../spi/index/metadata/ColumnMetadataImplTest.java | 118 ++++
...thubComplexTypeEvents_offline_table_config.json | 12 +
51 files changed, 4224 insertions(+), 478 deletions(-)
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/common/DataFetcher.java b/pinot-core/src/main/java/org/apache/pinot/core/common/DataFetcher.java
index 16a74918c6b..60dfc92d01c 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/common/DataFetcher.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/common/DataFetcher.java
@@ -80,7 +80,12 @@ public class DataFetcher implements AutoCloseable {
ForwardIndexReader<?> forwardIndexReader = dataSource.getForwardIndex();
Preconditions.checkState(forwardIndexReader != null,
"Forward index disabled for column: %s, cannot create DataFetcher!", column);
- ColumnValueReader columnValueReader = new ColumnValueReader(forwardIndexReader, dataSource.getDictionary());
+ // A RAW forward index cannot serve dict ids cheaply even when a (shared) dictionary is on disk —
+ // looking up dict ids would require a per-row Dictionary#indexOf call. Drop the dictionary here so
+ // ColumnValueReader takes the raw-value paths uniformly; callers that genuinely need dict ids on a
+ // RAW + shared-dict column must read raw values and consult the dictionary directly.
+ Dictionary dictionary = forwardIndexReader.isDictionaryEncoded() ? dataSource.getDictionary() : null;
+ ColumnValueReader columnValueReader = new ColumnValueReader(forwardIndexReader, dictionary);
_columnValueReaderMap.put(column, columnValueReader);
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/operator/filter/RangeIndexBasedFilterOperator.java b/pinot-core/src/main/java/org/apache/pinot/core/operator/filter/RangeIndexBasedFilterOperator.java
index a5f1f849a37..7c8a807dd81 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/operator/filter/RangeIndexBasedFilterOperator.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/operator/filter/RangeIndexBasedFilterOperator.java
@@ -56,10 +56,20 @@ public class RangeIndexBasedFilterOperator extends BaseColumnFilterOperator {
private final FieldSpec.DataType _parameterType;
static boolean canEvaluate(PredicateEvaluator predicateEvaluator, DataSource dataSource) {
- Predicate.Type type = predicateEvaluator.getPredicateType();
RangeIndexReader<?> rangeIndex = dataSource.getRangeIndex();
- return rangeIndex != null && (type == Predicate.Type.RANGE || (type == Predicate.Type.EQ
- && dataSource.getRangeIndex().isExact()));
+ if (rangeIndex == null) {
+ return false;
+ }
+ // Range-index format mirrors the column's dictionary state at index build/rebuild time: if the
+ // dictionary exists the range stores dict IDs (RangeIndexCreator/BitSlicedRangeIndexCreator both
+ // switch on hasDictionary), so the evaluator must also be dict-based to compare against the index.
+ // Dropping into this operator with a raw-value evaluator on a dict-built range silently returns
+ // wrong matches (raw values compared against dict IDs); fall through to scan instead.
+ if (dataSource.getDictionary() != null && !predicateEvaluator.isDictionaryBased()) {
+ return false;
+ }
+ Predicate.Type type = predicateEvaluator.getPredicateType();
+ return type == Predicate.Type.RANGE || (type == Predicate.Type.EQ && rangeIndex.isExact());
}
@SuppressWarnings("unchecked")
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/operator/filter/predicate/PredicateEvaluatorProvider.java b/pinot-core/src/main/java/org/apache/pinot/core/operator/filter/predicate/PredicateEvaluatorProvider.java
index 033cb68b537..9a4662d99bb 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/operator/filter/predicate/PredicateEvaluatorProvider.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/operator/filter/predicate/PredicateEvaluatorProvider.java
@@ -18,6 +18,7 @@
*/
package org.apache.pinot.core.operator.filter.predicate;
+import com.google.common.annotations.VisibleForTesting;
import javax.annotation.Nullable;
import org.apache.pinot.common.request.context.predicate.EqPredicate;
import org.apache.pinot.common.request.context.predicate.InPredicate;
@@ -29,6 +30,9 @@ import org.apache.pinot.common.request.context.predicate.RegexpLikePredicate;
import org.apache.pinot.core.query.request.context.QueryContext;
import org.apache.pinot.segment.spi.datasource.DataSource;
import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.index.reader.ForwardIndexReader;
+import org.apache.pinot.segment.spi.index.reader.RangeIndexReader;
+import org.apache.pinot.spi.config.table.FieldConfig;
import org.apache.pinot.spi.data.FieldSpec.DataType;
import org.apache.pinot.spi.exception.BadQueryRequestException;
@@ -37,13 +41,29 @@ public class PredicateEvaluatorProvider {
private PredicateEvaluatorProvider() {
}
- public static PredicateEvaluator getPredicateEvaluator(Predicate predicate, @Nullable Dictionary dictionary,
- DataType dataType) {
- return getPredicateEvaluator(predicate, dictionary, dataType, null);
+ /// Builds a [PredicateEvaluator] for a leaf filter on the column backed by `dataSource`. The dictionary is derived
+ /// via [#getDictionaryUsableForFiltering], which keeps it only when a dict-consuming filter operator (inverted /
+ /// exact range) will actually run for this predicate type on the column's forward-index encoding. The data type is
+ /// taken from the data-source metadata.
+ public static PredicateEvaluator getPredicateEvaluator(Predicate predicate, DataSource dataSource,
+ QueryContext queryContext) {
+ Dictionary dictionary = getDictionaryUsableForFiltering(dataSource, queryContext, predicate);
+ DataType dataType = dataSource.getDataSourceMetadata().getDataType();
+ return buildEvaluator(predicate, dictionary, dataType, queryContext);
}
+ /// Builds a [PredicateEvaluator] when the value source and `dictionary` are already in sync by construction: when
+ /// `dictionary` is non-null the source produces dict ids decodable by that dictionary; when `dictionary` is null
+ /// the source produces raw values. No gating logic runs — the dictionary (if any) is taken as-is, so the caller is
+ /// responsible for the match.
+ // TODO: Always pass in query context
public static PredicateEvaluator getPredicateEvaluator(Predicate predicate, @Nullable Dictionary dictionary,
DataType dataType, @Nullable QueryContext queryContext) {
+ return buildEvaluator(predicate, dictionary, dataType, queryContext);
+ }
+
+ private static PredicateEvaluator buildEvaluator(Predicate predicate, @Nullable Dictionary dictionary,
+ DataType dataType, @Nullable QueryContext queryContext) {
try {
if (dictionary != null) {
// dictionary based predicate evaluators
@@ -95,9 +115,62 @@ public class PredicateEvaluatorProvider {
}
}
- public static PredicateEvaluator getPredicateEvaluator(Predicate predicate, DataSource dataSource,
- QueryContext queryContext) {
- return getPredicateEvaluator(predicate, dataSource.getDictionary(),
- dataSource.getDataSourceMetadata().getDataType(), queryContext);
+ /// Returns the column dictionary if the planner can actually use it for filtering this specific predicate, otherwise
+ /// `null`.
+ ///
+ /// When the forward index is RAW, scan-based filtering reads raw values rather than dict IDs, so a dict-based
+ /// predicate evaluator (which only implements `applySV(int dictId)`) cannot be applied during scan. The
+ /// dictionary is only useful when the planner will pick a dict-consuming filter operator that resolves the
+ /// predicate fully (no scan fallback). The decision is per-predicate-type because each predicate type uses a
+ /// different subset of dict-consuming operators (see `FilterOperatorUtils.getLeafFilterOperator`):
+ ///
+ /// - `RANGE`: only the range-index path can consume the dictionary on a RAW column. (Sorted index implies a
+ /// dict-encoded forward index, so it cannot apply here.) Furthermore, the range index must be exact —
+ /// non-exact (legacy) range readers fall back to `ScanBasedFilterOperator` for partial matches, which feeds
+ /// raw forward-index values into the predicate evaluator and breaks if the evaluator is dict-based.
+ /// - `REGEXP_LIKE / NOT_EQ / IN / NOT_IN`: only the inverted-index path consumes the dictionary on a RAW
+ /// column. FST/IFST handle their own evaluators upstream of this method.
+ /// - `EQ`: inverted index, plus exact range index (see `RangeIndexBasedFilterOperator#canEvaluate`).
+ ///
+ /// If the forward index itself is missing (explicitly disabled), scan is impossible and dict-based eval is the
+ /// only option, so the dictionary is preserved regardless of predicate type.
+ @VisibleForTesting
+ @Nullable
+ static Dictionary getDictionaryUsableForFiltering(DataSource dataSource, @Nullable QueryContext queryContext,
+ Predicate predicate) {
+ Dictionary dictionary = dataSource.getDictionary();
+ if (dictionary == null) {
+ return null;
+ }
+ ForwardIndexReader<?> forwardIndex = dataSource.getForwardIndex();
+ if (forwardIndex == null || forwardIndex.isDictionaryEncoded()) {
+ return dictionary;
+ }
+ // RAW forward index: keep the dictionary only if a dict-consuming filter operator is available AND enabled for
+ // this predicate type. NOTE: a sorted forward index is run-length dict-encoded by definition, so the sorted
+ // path is unreachable from this RAW branch — only inverted and (exact) range remain.
+ boolean invertedAvailable = dataSource.getInvertedIndex() != null
+ && isIndexAllowedForQuery(queryContext, dataSource, FieldConfig.IndexType.INVERTED);
+ RangeIndexReader<?> rangeIndex = dataSource.getRangeIndex();
+ boolean exactRangeAvailable = rangeIndex != null && rangeIndex.isExact()
+ && isIndexAllowedForQuery(queryContext, dataSource, FieldConfig.IndexType.RANGE);
+ switch (predicate.getType()) {
+ case RANGE:
+ return exactRangeAvailable ? dictionary : null;
+ case EQ:
+ return (invertedAvailable || exactRangeAvailable) ? dictionary : null;
+ case REGEXP_LIKE:
+ case NOT_EQ:
+ case IN:
+ case NOT_IN:
+ return invertedAvailable ? dictionary : null;
+ default:
+ return dictionary;
+ }
+ }
+
+ private static boolean isIndexAllowedForQuery(@Nullable QueryContext queryContext, DataSource dataSource,
+ FieldConfig.IndexType indexType) {
+ return queryContext == null || queryContext.isIndexUseAllowed(dataSource, indexType);
}
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/ArrayLiteralTransformFunction.java b/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/ArrayLiteralTransformFunction.java
index 64878aa218e..c76977b6c9c 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/ArrayLiteralTransformFunction.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/ArrayLiteralTransformFunction.java
@@ -28,7 +28,6 @@ import org.apache.pinot.common.request.context.LiteralContext;
import org.apache.pinot.core.operator.ColumnContext;
import org.apache.pinot.core.operator.blocks.ValueBlock;
import org.apache.pinot.core.operator.transform.TransformResultMetadata;
-import org.apache.pinot.segment.spi.index.reader.Dictionary;
import org.apache.pinot.spi.data.FieldSpec.DataType;
import org.roaringbitmap.RoaringBitmap;
@@ -221,11 +220,6 @@ public class ArrayLiteralTransformFunction implements TransformFunction {
return new TransformResultMetadata(_dataType, false, false);
}
- @Override
- public Dictionary getDictionary() {
- return null;
- }
-
@Override
public int[] transformToDictIdsSV(ValueBlock valueBlock) {
throw new UnsupportedOperationException();
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/BaseTransformFunction.java b/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/BaseTransformFunction.java
index 3b82a98bd3b..80dc59ea8db 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/BaseTransformFunction.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/BaseTransformFunction.java
@@ -121,11 +121,6 @@ public abstract class BaseTransformFunction implements TransformFunction {
_nullHandlingEnabled = nullHandlingEnabled;
}
- @Override
- public Dictionary getDictionary() {
- return null;
- }
-
@Override
public int[] transformToDictIdsSV(ValueBlock valueBlock) {
throw new UnsupportedOperationException();
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/BinaryOperatorTransformFunction.java b/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/BinaryOperatorTransformFunction.java
index 52809a0f375..4a9aff8e591 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/BinaryOperatorTransformFunction.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/BinaryOperatorTransformFunction.java
@@ -143,7 +143,7 @@ public abstract class BinaryOperatorTransformFunction extends BaseTransformFunct
if (predicate == null) {
return null;
}
- return PredicateEvaluatorProvider.getPredicateEvaluator(predicate, leftDictionary, leftDataType);
+ return PredicateEvaluatorProvider.getPredicateEvaluator(predicate, leftDictionary, leftDataType, null);
}
/**
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/FilterMvPredicateEvaluator.java b/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/FilterMvPredicateEvaluator.java
index 66ffa299065..058c2d104f9 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/FilterMvPredicateEvaluator.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/FilterMvPredicateEvaluator.java
@@ -23,6 +23,7 @@ import java.util.EnumSet;
import java.util.List;
import javax.annotation.Nullable;
import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.Strings;
import org.apache.pinot.common.request.Expression;
import org.apache.pinot.common.request.context.ExpressionContext;
import org.apache.pinot.common.request.context.FilterContext;
@@ -75,7 +76,7 @@ public final class FilterMvPredicateEvaluator {
throw new IllegalArgumentException(
"filterMv only supports predicates on placeholder '" + VALUE_PLACEHOLDER + "' without functions");
}
- if (!StringUtils.equalsIgnoreCase(lhs.getIdentifier(), VALUE_PLACEHOLDER)) {
+ if (!Strings.CI.equals(lhs.getIdentifier(), VALUE_PLACEHOLDER)) {
throw new IllegalArgumentException(
"filterMv predicate must reference placeholder '" + VALUE_PLACEHOLDER + "', but found '"
+ lhs.getIdentifier() + "'");
@@ -106,7 +107,7 @@ public final class FilterMvPredicateEvaluator {
"filterMv does not support predicate type: " + predicate.getType());
}
PredicateEvaluator evaluator =
- PredicateEvaluatorProvider.getPredicateEvaluator(predicate, dictionary, dataType);
+ PredicateEvaluatorProvider.getPredicateEvaluator(predicate, dictionary, dataType, null);
if (evaluator.isAlwaysTrue()) {
return EvalNode.constant(true);
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/FilterMvTransformFunction.java b/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/FilterMvTransformFunction.java
index a1ceaee2cba..3b7bff155a1 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/FilterMvTransformFunction.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/FilterMvTransformFunction.java
@@ -24,7 +24,9 @@ import javax.annotation.Nullable;
import org.apache.pinot.core.operator.ColumnContext;
import org.apache.pinot.core.operator.blocks.ValueBlock;
import org.apache.pinot.core.operator.transform.TransformResultMetadata;
+import org.apache.pinot.segment.spi.datasource.DataSource;
import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.index.reader.ForwardIndexReader;
import org.apache.pinot.spi.data.FieldSpec.DataType;
@@ -63,9 +65,8 @@ public class FilterMvTransformFunction extends BaseTransformFunction {
"The first argument of filterMv transform function must be a multi-valued column or a transform function");
}
_mainTransformFunction = firstArgument;
- _resultMetadata = _mainTransformFunction.getResultMetadata();
- _dictionary = _mainTransformFunction.getDictionary();
- _dataType = _resultMetadata.getDataType();
+ TransformResultMetadata innerMetadata = _mainTransformFunction.getResultMetadata();
+ _dataType = innerMetadata.getDataType();
TransformFunction predicateArgument = arguments.get(1);
if (!(predicateArgument instanceof LiteralTransformFunction) || !predicateArgument.getResultMetadata()
@@ -74,7 +75,26 @@ public class FilterMvTransformFunction extends BaseTransformFunction {
"The second argument of filterMv transform function must be a single-valued string literal");
}
String predicate = ((LiteralTransformFunction) predicateArgument).getStringLiteral();
+ // IdentifierTransformFunction always exposes the column dictionary if one exists, but the dict-id
+ // matching path requires forward.getDictIdMV() to actually serve dict ids cheaply. RAW forward indexes
+ // throw UnsupportedOperationException from getDictIdMV — so when the inner Identifier wraps a RAW
+ // forward column (with or without a shared dictionary), drop the dictionary here so filterMv falls
+ // back to per-value raw matching.
+ _dictionary = dictionaryUsableForFilterMv(firstArgument, columnContextMap);
_predicateEvaluator = FilterMvPredicateEvaluator.forPredicate(predicate, _dataType, _dictionary);
+ _resultMetadata = new TransformResultMetadata(_dataType, innerMetadata.isSingleValue(), _dictionary != null);
+ }
+
+ @Nullable
+ private static Dictionary dictionaryUsableForFilterMv(TransformFunction inner,
+ Map<String, ColumnContext> columnContextMap) {
+ Dictionary dictionary = inner.getDictionary();
+ if (dictionary == null || !(inner instanceof IdentifierTransformFunction)) {
+ return dictionary;
+ }
+ DataSource dataSource = columnContextMap.get(((IdentifierTransformFunction) inner).getColumnName()).getDataSource();
+ ForwardIndexReader<?> forwardIndex = dataSource.getForwardIndex();
+ return (forwardIndex != null && forwardIndex.isDictionaryEncoded()) ? dictionary : null;
}
@Override
@@ -90,9 +110,6 @@ public class FilterMvTransformFunction extends BaseTransformFunction {
@Override
public int[][] transformToDictIdsMV(ValueBlock valueBlock) {
- if (_dictionary == null) {
- return super.transformToDictIdsMV(valueBlock);
- }
int length = valueBlock.getNumDocs();
if (_dictIdsMV == null || _dictIdsMV.length < length) {
_dictIdsMV = new int[length][];
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/GenerateArrayTransformFunction.java b/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/GenerateArrayTransformFunction.java
index df7e42a1b15..393432b875e 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/GenerateArrayTransformFunction.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/GenerateArrayTransformFunction.java
@@ -28,7 +28,6 @@ import org.apache.pinot.common.request.context.ExpressionContext;
import org.apache.pinot.core.operator.ColumnContext;
import org.apache.pinot.core.operator.blocks.ValueBlock;
import org.apache.pinot.core.operator.transform.TransformResultMetadata;
-import org.apache.pinot.segment.spi.index.reader.Dictionary;
import org.apache.pinot.spi.data.FieldSpec.DataType;
import org.roaringbitmap.RoaringBitmap;
@@ -191,12 +190,6 @@ public class GenerateArrayTransformFunction implements TransformFunction {
return new TransformResultMetadata(_dataType, false, false);
}
- @Nullable
- @Override
- public Dictionary getDictionary() {
- return null;
- }
-
@Override
public int[] transformToDictIdsSV(ValueBlock valueBlock) {
throw new UnsupportedOperationException();
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/IdentifierTransformFunction.java b/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/IdentifierTransformFunction.java
index 48700c7cb0c..1d918499494 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/IdentifierTransformFunction.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/IdentifierTransformFunction.java
@@ -25,7 +25,9 @@ import javax.annotation.Nullable;
import org.apache.pinot.core.operator.ColumnContext;
import org.apache.pinot.core.operator.blocks.ValueBlock;
import org.apache.pinot.core.operator.transform.TransformResultMetadata;
+import org.apache.pinot.segment.spi.datasource.DataSource;
import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.index.reader.ForwardIndexReader;
import org.roaringbitmap.RoaringBitmap;
@@ -40,7 +42,13 @@ public class IdentifierTransformFunction implements TransformFunction {
public IdentifierTransformFunction(String columnName, ColumnContext columnContext) {
_columnName = columnName;
- _dictionary = columnContext.getDictionary();
+ DataSource dataSource = columnContext.getDataSource();
+ assert dataSource != null;
+ // Only expose the dictionary when the forward index is dict-encoded. A column can have a dictionary alongside
+ // a RAW forward index (e.g. dict + inverted/range), in which case transformToDictIdsSV would fail because
+ // BlockValueSet.getDictionaryIdsSV requires a dict-encoded forward index.
+ ForwardIndexReader<?> forwardIndex = dataSource.getForwardIndex();
+ _dictionary = forwardIndex != null && forwardIndex.isDictionaryEncoded() ? dataSource.getDictionary() : null;
_resultMetadata =
new TransformResultMetadata(columnContext.getDataType(), columnContext.isSingleValue(), _dictionary != null);
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/ItemTransformFunction.java b/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/ItemTransformFunction.java
index cdb5c12c9ca..c88ca0ed154 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/ItemTransformFunction.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/ItemTransformFunction.java
@@ -25,9 +25,10 @@ import org.apache.pinot.core.operator.ColumnContext;
import org.apache.pinot.core.operator.blocks.ValueBlock;
import org.apache.pinot.core.operator.transform.TransformResultMetadata;
import org.apache.pinot.segment.spi.datasource.DataSource;
+import org.apache.pinot.segment.spi.datasource.DataSourceMetadata;
import org.apache.pinot.segment.spi.datasource.MapDataSource;
import org.apache.pinot.segment.spi.index.reader.Dictionary;
-import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.segment.spi.index.reader.ForwardIndexReader;
/**
@@ -35,56 +36,40 @@ import org.apache.pinot.spi.data.FieldSpec;
*/
public class ItemTransformFunction extends BaseTransformFunction {
public static final String FUNCTION_NAME = "item";
- String _column;
- String _key;
- String[] _keyPath;
- TransformFunction _mapValue;
- TransformFunction _keyValue;
- Dictionary _keyDictionary;
+
+ private String[] _keyPath;
+ private Dictionary _dictionary;
private TransformResultMetadata _resultMetadata;
@Override
public void init(List<TransformFunction> arguments, Map<String, ColumnContext> columnContextMap) {
super.init(arguments, columnContextMap);
- // Should be exactly 2 arguments (map value expression and key expression
- if (arguments.size() != 2) {
- throw new IllegalArgumentException("Exactly 1 argument is required for Vector transform function");
- }
-
- // Check if the second operand (the key) is a string literal, if it is then we can directly construct the
- // MapDataSource which will pre-compute the Key ID.
-
- _mapValue = arguments.get(0);
- Preconditions.checkArgument(_mapValue instanceof IdentifierTransformFunction, "Map Item: Left operand"
- + "must be an identifier");
- _column = ((IdentifierTransformFunction) _mapValue).getColumnName();
- if (_column == null) {
- throw new IllegalArgumentException("Map Item: left operand resolved to a null column name");
- }
-
- _keyValue = arguments.get(1);
- Preconditions.checkArgument(_keyValue instanceof LiteralTransformFunction, "Map Item: Right operand"
- + "must be a literal");
- _key = ((LiteralTransformFunction) arguments.get(1)).getStringLiteral();
- Preconditions.checkArgument(_key != null, "Map Item: Right operand"
- + "must be a string literal");
- _keyPath = new String[]{_column, _key};
-
- // The metadata about the values that this operation will resolve to is determined by the type of the data
- // under the key, not by the Map column. So we need to look up the Key's Metadata.
- DataSource dataSource = columnContextMap.get(_column).getDataSource();
-
- if (dataSource instanceof MapDataSource) {
- MapDataSource mapDS = (MapDataSource) dataSource;
- DataSource keyDS = mapDS.getKeyDataSource(_key);
- FieldSpec.DataType keyType = keyDS.getDataSourceMetadata().getDataType().getStoredType();
- _keyDictionary = keyDS.getDictionary();
- _resultMetadata =
- new TransformResultMetadata(keyType, keyDS.getDataSourceMetadata().isSingleValue(),
- _keyDictionary != null);
- } else {
- throw new RuntimeException("The left operand for a MAP ITEM operation must resolve to a Map Data Source");
- }
+ Preconditions.checkArgument(arguments.size() == 2, "Expected exactly 2 arguments, got: %s", arguments.size());
+
+ TransformFunction mapValue = arguments.get(0);
+ Preconditions.checkArgument(mapValue instanceof IdentifierTransformFunction,
+ "Map Item: Left operand must be an identifier");
+ String column = ((IdentifierTransformFunction) mapValue).getColumnName();
+
+ TransformFunction keyValue = arguments.get(1);
+ Preconditions.checkArgument(keyValue instanceof LiteralTransformFunction,
+ "Map Item: Right operand must be a literal");
+ String key = ((LiteralTransformFunction) arguments.get(1)).getStringLiteral();
+ _keyPath = new String[]{column, key};
+
+ DataSource dataSource = columnContextMap.get(column).getDataSource();
+ Preconditions.checkState(dataSource instanceof MapDataSource, "Column: %s must be a MAP column", column);
+ MapDataSource mapDataSource = (MapDataSource) dataSource;
+ DataSource valueDataSource = mapDataSource.getKeyDataSource(key);
+ // Only expose the dictionary when the forward index is dict-encoded. A column can have a dictionary alongside
+ // a RAW forward index (e.g. dict + inverted/range), in which case transformToDictIdsSV would fail because
+ // BlockValueSet.getDictionaryIdsSV requires a dict-encoded forward index.
+ ForwardIndexReader<?> forwardIndex = valueDataSource.getForwardIndex();
+ _dictionary = forwardIndex != null && forwardIndex.isDictionaryEncoded() ? valueDataSource.getDictionary() : null;
+ DataSourceMetadata valueDataSourceMetadata = valueDataSource.getDataSourceMetadata();
+ _resultMetadata =
+ new TransformResultMetadata(valueDataSourceMetadata.getDataType(), valueDataSourceMetadata.isSingleValue(),
+ _dictionary != null);
}
@Override
@@ -94,13 +79,12 @@ public class ItemTransformFunction extends BaseTransformFunction {
@Override
public TransformResultMetadata getResultMetadata() {
- return new TransformResultMetadata(_resultMetadata.getDataType().getStoredType(), true,
- _resultMetadata.hasDictionary());
+ return _resultMetadata;
}
@Override
public Dictionary getDictionary() {
- return _keyDictionary;
+ return _dictionary;
}
@Override
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/LiteralTransformFunction.java b/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/LiteralTransformFunction.java
index 9700a09b170..9d6b2930324 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/LiteralTransformFunction.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/LiteralTransformFunction.java
@@ -27,7 +27,6 @@ import org.apache.pinot.common.request.context.LiteralContext;
import org.apache.pinot.core.operator.ColumnContext;
import org.apache.pinot.core.operator.blocks.ValueBlock;
import org.apache.pinot.core.operator.transform.TransformResultMetadata;
-import org.apache.pinot.segment.spi.index.reader.Dictionary;
import org.roaringbitmap.RoaringBitmap;
@@ -110,11 +109,6 @@ public class LiteralTransformFunction implements TransformFunction {
return new TransformResultMetadata(_literalContext.getType(), true, false);
}
- @Override
- public Dictionary getDictionary() {
- return null;
- }
-
@Override
public int[] transformToDictIdsSV(ValueBlock valueBlock) {
throw new UnsupportedOperationException();
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/TransformFunction.java b/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/TransformFunction.java
index 3a0a5720f2b..87987fd8228 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/TransformFunction.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/TransformFunction.java
@@ -77,7 +77,9 @@ public interface TransformFunction {
* Returns the dictionary for the transform result if the result is dictionary-encoded, or {@code null} if not.
*/
@Nullable
- Dictionary getDictionary();
+ default Dictionary getDictionary() {
+ return null;
+ }
/**
* Transforms the data from the given value block to single-valued dictionary ids.
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/plan/FilterPlanNode.java b/pinot-core/src/main/java/org/apache/pinot/core/plan/FilterPlanNode.java
index 90701181028..9074ff6323f 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/plan/FilterPlanNode.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/plan/FilterPlanNode.java
@@ -316,8 +316,7 @@ public class FilterPlanNode implements PlanNode {
dataSource.getIFSTIndex(), dataSource.getDictionary());
} else {
predicateEvaluator =
- PredicateEvaluatorProvider.getPredicateEvaluator(predicate, dataSource.getDictionary(),
- dataSource.getDataSourceMetadata().getDataType(), _queryContext);
+ PredicateEvaluatorProvider.getPredicateEvaluator(predicate, dataSource, _queryContext);
}
} else {
if (dataSource.getFSTIndex() != null) {
@@ -325,8 +324,7 @@ public class FilterPlanNode implements PlanNode {
dataSource.getFSTIndex(), dataSource.getDictionary());
} else {
predicateEvaluator =
- PredicateEvaluatorProvider.getPredicateEvaluator(predicate, dataSource.getDictionary(),
- dataSource.getDataSourceMetadata().getDataType(), _queryContext);
+ PredicateEvaluatorProvider.getPredicateEvaluator(predicate, dataSource, _queryContext);
}
}
_predicateEvaluators.add(Pair.of(predicate, predicateEvaluator));
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/DistinctCountThetaSketchAggregationFunction.java b/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/DistinctCountThetaSketchAggregationFunction.java
index d23d4c7549a..49440638310 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/DistinctCountThetaSketchAggregationFunction.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/DistinctCountThetaSketchAggregationFunction.java
@@ -1551,7 +1551,8 @@ public class DistinctCountThetaSketchAggregationFunction
DataType valueType = valueTypes[_expressionIndex];
Object valueArray = valueArrays[_expressionIndex];
if (_predicateEvaluator == null) {
- _predicateEvaluator = PredicateEvaluatorProvider.getPredicateEvaluator(_predicate, null, valueType);
+ // Intermediate-result predicate evaluation has no DataSource — pass null to take the raw-value path.
+ _predicateEvaluator = PredicateEvaluatorProvider.getPredicateEvaluator(_predicate, null, valueType, null);
}
if (singleValue) {
switch (valueType) {
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/groupby/DefaultGroupByExecutor.java b/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/groupby/DefaultGroupByExecutor.java
index 8039ba1ce52..15cecadd3f7 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/groupby/DefaultGroupByExecutor.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/groupby/DefaultGroupByExecutor.java
@@ -87,7 +87,16 @@ public class DefaultGroupByExecutor implements GroupByExecutor {
for (ExpressionContext groupByExpression : groupByExpressions) {
ColumnContext columnContext = projectOperator.getResultColumnContext(groupByExpression);
hasMVGroupByExpression |= !columnContext.isSingleValue();
- hasNoDictionaryGroupByExpression |= columnContext.getDictionary() == null;
+ // DictionaryBasedGroupKeyGenerator does dict-id reads from the forward index — that requires the
+ // forward index to actually be dict-encoded. Columns with a shared dictionary on a RAW forward index
+ // (dict file exists but forward stores raw values) would otherwise be misrouted into the dict-id
+ // path; gate on forward-index encoding so they take the no-dict GROUP BY path instead.
+ // ColumnContext.getDataSource() is null for computed (non-identifier) transforms; in that case
+ // getDictionary() == null already covers them via the first condition.
+ hasNoDictionaryGroupByExpression |= columnContext.getDictionary() == null
+ || (columnContext.getDataSource() != null
+ && columnContext.getDataSource().getForwardIndex() != null
+ && !columnContext.getDataSource().getForwardIndex().isDictionaryEncoded());
}
_hasMVGroupByExpression = hasMVGroupByExpression;
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/query/reduce/filter/PredicateRowMatcher.java b/pinot-core/src/main/java/org/apache/pinot/core/query/reduce/filter/PredicateRowMatcher.java
index 4c108b70e72..46d6459bfbb 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/query/reduce/filter/PredicateRowMatcher.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/query/reduce/filter/PredicateRowMatcher.java
@@ -44,7 +44,7 @@ public class PredicateRowMatcher implements RowMatcher {
if (_predicateType == Predicate.Type.IS_NULL || _predicateType == Predicate.Type.IS_NOT_NULL) {
_predicateEvaluator = null;
} else {
- _predicateEvaluator = PredicateEvaluatorProvider.getPredicateEvaluator(predicate, null, _valueType);
+ _predicateEvaluator = PredicateEvaluatorProvider.getPredicateEvaluator(predicate, null, _valueType, null);
}
_nullHandlingEnabled = nullHandlingEnabled;
}
diff --git a/pinot-core/src/test/java/org/apache/pinot/core/operator/filter/predicate/PredicateEvaluatorProviderTest.java b/pinot-core/src/test/java/org/apache/pinot/core/operator/filter/predicate/PredicateEvaluatorProviderTest.java
new file mode 100644
index 00000000000..76b5fa07b7f
--- /dev/null
+++ b/pinot-core/src/test/java/org/apache/pinot/core/operator/filter/predicate/PredicateEvaluatorProviderTest.java
@@ -0,0 +1,260 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.filter.predicate;
+
+import org.apache.pinot.common.request.context.ExpressionContext;
+import org.apache.pinot.common.request.context.predicate.EqPredicate;
+import org.apache.pinot.common.request.context.predicate.RangePredicate;
+import org.apache.pinot.core.query.request.context.QueryContext;
+import org.apache.pinot.segment.spi.datasource.DataSource;
+import org.apache.pinot.segment.spi.datasource.DataSourceMetadata;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.index.reader.ForwardIndexReader;
+import org.apache.pinot.segment.spi.index.reader.InvertedIndexReader;
+import org.apache.pinot.segment.spi.index.reader.RangeIndexReader;
+import org.apache.pinot.spi.config.table.FieldConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.FieldSpec.DataType;
+import org.mockito.Mockito;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertNotNull;
+import static org.testng.Assert.assertNull;
+import static org.testng.Assert.assertTrue;
+
+
+public class PredicateEvaluatorProviderTest {
+
+ /// RAW forward index with dictionary but no inverted/range/sorted index — the planner must drop the dictionary so
+ /// that the scan-based filter operator receives a raw-value evaluator. Otherwise, the scan iterator would call
+ /// `applySV(rawValue)` on a dict-based evaluator and throw [UnsupportedOperationException].
+ @Test
+ public void rawForwardWithDictNoSecondaryIndexUsesRawValueEvaluator() {
+ DataSource dataSource = mockDataSource("col", DataType.STRING, /*sorted=*/false,
+ /*dict=*/true, /*forwardDictEncoded=*/false, /*inverted=*/false, /*range=*/false, /*rangeExact=*/false);
+ PredicateEvaluator evaluator =
+ PredicateEvaluatorProvider.getPredicateEvaluator(eqPredicate("col", "value-1"), dataSource,
+ mockQueryContext(/*allowAllIndexes=*/true));
+ assertFalse(evaluator.isDictionaryBased(),
+ "RAW forward + dict + no secondary index must produce raw-value evaluator");
+ }
+
+ /// RAW forward index with dictionary + inverted index — keep the dictionary so the inverted-index path can translate
+ /// dict IDs to docs.
+ @Test
+ public void rawForwardWithDictAndInvertedKeepsDictEvaluator() {
+ DataSource dataSource = mockDataSource("col", DataType.STRING, /*sorted=*/false,
+ /*dict=*/true, /*forwardDictEncoded=*/false, /*inverted=*/true, /*range=*/false, /*rangeExact=*/false);
+ PredicateEvaluator evaluator =
+ PredicateEvaluatorProvider.getPredicateEvaluator(eqPredicate("col", "value-1"), dataSource,
+ mockQueryContext(/*allowAllIndexes=*/true));
+ assertTrue(evaluator.isDictionaryBased(),
+ "RAW forward + dict + inverted index must keep dict-based evaluator for inverted-index lookup");
+ }
+
+ /// RAW forward index with dictionary + inverted index, but the user disabled the inverted index for this query via
+ /// `OPTION(skipIndexes=...)`. The planner must drop the dictionary because the scan path will be used.
+ @Test
+ public void rawForwardWithDictAndDisabledInvertedDropsDictEvaluator() {
+ DataSource dataSource = mockDataSource("col", DataType.STRING, /*sorted=*/false,
+ /*dict=*/true, /*forwardDictEncoded=*/false, /*inverted=*/true, /*range=*/false, /*rangeExact=*/false);
+ QueryContext queryContext = mockQueryContext(/*allowAllIndexes=*/false);
+ PredicateEvaluator evaluator =
+ PredicateEvaluatorProvider.getPredicateEvaluator(eqPredicate("col", "value-1"), dataSource, queryContext);
+ assertFalse(evaluator.isDictionaryBased(),
+ "Inverted index disallowed via skipIndexes must trigger raw-value evaluator on raw forward");
+ }
+
+ /// Standard dict-encoded forward index — keep the dictionary regardless of secondary indexes.
+ @Test
+ public void dictEncodedForwardKeepsDictEvaluator() {
+ DataSource dataSource = mockDataSource("col", DataType.STRING, /*sorted=*/false,
+ /*dict=*/true, /*forwardDictEncoded=*/true, /*inverted=*/false, /*range=*/false, /*rangeExact=*/false);
+ PredicateEvaluator evaluator =
+ PredicateEvaluatorProvider.getPredicateEvaluator(eqPredicate("col", "value-1"), dataSource,
+ mockQueryContext(/*allowAllIndexes=*/true));
+ assertTrue(evaluator.isDictionaryBased(), "Dict-encoded forward index must always keep dict-based evaluator");
+ }
+
+ /// Forward index disabled (e.g. forward-index-disabled column) — keep dictionary in place since scan is impossible.
+ @Test
+ public void forwardIndexDisabledKeepsDictEvaluator() {
+ DataSource dataSource = mockDataSource("col", DataType.STRING, /*sorted=*/false,
+ /*dict=*/true, /*forwardDictEncoded=*/false, /*inverted=*/true, /*range=*/false, /*rangeExact=*/false);
+ Mockito.when(dataSource.getForwardIndex()).thenReturn(null);
+ PredicateEvaluator evaluator =
+ PredicateEvaluatorProvider.getPredicateEvaluator(eqPredicate("col", "value-1"), dataSource,
+ mockQueryContext(/*allowAllIndexes=*/true));
+ assertTrue(evaluator.isDictionaryBased(),
+ "Disabled forward index must keep dict-based evaluator (scan is impossible)");
+ }
+
+ /// RAW forward + dictionary + inverted index but a RANGE predicate is being filtered. The inverted index does not
+ /// service RANGE filtering, so the operator selection falls through to scan. The planner must drop the dictionary
+ /// for this predicate even though an inverted index physically exists on the column.
+ @Test
+ public void rawForwardWithDictAndInvertedDropsDictForRangePredicate() {
+ DataSource dataSource = mockDataSource("col", DataType.INT, /*sorted=*/false,
+ /*dict=*/true, /*forwardDictEncoded=*/false, /*inverted=*/true, /*range=*/false, /*rangeExact=*/false);
+ Dictionary result = PredicateEvaluatorProvider.getDictionaryUsableForFiltering(dataSource,
+ mockQueryContext(/*allowAllIndexes=*/true), rangePredicate("col", "(5\t\t15)"));
+ assertNull(result,
+ "RANGE predicate on inverted-but-not-range column must drop dict so raw-value evaluator is built");
+ }
+
+ /// RAW forward + dictionary + range index + RANGE predicate. When a dictionary exists, the range index is built over
+ /// dict IDs (see `RangeIndexType#createIndexCreator`), so the dictionary must be preserved and the dict-based range
+ /// evaluator used.
+ @Test
+ public void rawForwardWithDictAndRangeIndexKeepsDictForRangePredicate() {
+ DataSource dataSource = mockDataSource("col", DataType.INT, /*sorted=*/false,
+ /*dict=*/true, /*forwardDictEncoded=*/false, /*inverted=*/false, /*range=*/true, /*rangeExact=*/true);
+ Dictionary result = PredicateEvaluatorProvider.getDictionaryUsableForFiltering(dataSource,
+ mockQueryContext(/*allowAllIndexes=*/true), rangePredicate("col", "(5\t\t15)"));
+ assertNotNull(result,
+ "RANGE predicate on dict-id-based range index must keep dict so range reader receives dict-id evaluator");
+ }
+
+ /// RAW forward + dictionary + non-exact (legacy) range index + RANGE predicate. Non-exact range readers fall back to
+ /// ScanBasedFilterOperator for partial matches; that scan applies the predicate evaluator on raw forward values, so
+ /// a dict-based evaluator would be misapplied. The dictionary must be dropped.
+ @Test
+ public void rawForwardWithDictAndNonExactRangeIndexDropsDictForRangePredicate() {
+ DataSource dataSource = mockDataSource("col", DataType.INT, /*sorted=*/false,
+ /*dict=*/true, /*forwardDictEncoded=*/false, /*inverted=*/false, /*range=*/true, /*rangeExact=*/false);
+ Dictionary result = PredicateEvaluatorProvider.getDictionaryUsableForFiltering(dataSource,
+ mockQueryContext(/*allowAllIndexes=*/true), rangePredicate("col", "(5\t\t15)"));
+ assertNull(result,
+ "Non-exact range index falls back to scan; dict must be dropped so scan uses raw-value evaluator");
+ }
+
+ /// RAW forward + dictionary + range index but the range index is disabled via skipIndexes. With no other
+ /// dict-consuming operator available, RANGE falls to scan, which needs the raw-value evaluator.
+ @Test
+ public void rawForwardWithDictAndDisabledRangeIndexDropsDictForRangePredicate() {
+ DataSource dataSource = mockDataSource("col", DataType.INT, /*sorted=*/false,
+ /*dict=*/true, /*forwardDictEncoded=*/false, /*inverted=*/false, /*range=*/true, /*rangeExact=*/true);
+ Dictionary result = PredicateEvaluatorProvider.getDictionaryUsableForFiltering(dataSource,
+ mockQueryContext(/*allowAllIndexes=*/false), rangePredicate("col", "(5\t\t15)"));
+ assertNull(result, "Range index disallowed via skipIndexes must trigger raw-value evaluator on raw forward");
+ }
+
+ /// RAW forward + dictionary + EXACT range index + EQ predicate. The exact range index can serve EQ (see
+ /// `RangeIndexBasedFilterOperator#canEvaluate`) and uses dict IDs when dict exists, so keep the dict.
+ @Test
+ public void rawForwardWithDictAndExactRangeIndexKeepsDictForEqPredicate() {
+ DataSource dataSource = mockDataSource("col", DataType.INT, /*sorted=*/false,
+ /*dict=*/true, /*forwardDictEncoded=*/false, /*inverted=*/false, /*range=*/true, /*rangeExact=*/true);
+ Dictionary result = PredicateEvaluatorProvider.getDictionaryUsableForFiltering(dataSource,
+ mockQueryContext(/*allowAllIndexes=*/true), eqPredicate("col", "7"));
+ assertNotNull(result, "Exact range index serves EQ predicate; dict must be preserved for the dict-based evaluator");
+ }
+
+ /// RAW forward + dictionary + NON-EXACT range index + EQ predicate. Non-exact range indexes don't serve EQ; EQ falls
+ /// through to scan. With no inverted/sorted, the dictionary must be dropped.
+ @Test
+ public void rawForwardWithDictAndNonExactRangeIndexDropsDictForEqPredicate() {
+ DataSource dataSource = mockDataSource("col", DataType.INT, /*sorted=*/false,
+ /*dict=*/true, /*forwardDictEncoded=*/false, /*inverted=*/false, /*range=*/true, /*rangeExact=*/false);
+ Dictionary result = PredicateEvaluatorProvider.getDictionaryUsableForFiltering(dataSource,
+ mockQueryContext(/*allowAllIndexes=*/true), eqPredicate("col", "7"));
+ assertNull(result,
+ "Non-exact range index does not serve EQ; dict must be dropped so scan uses raw-value evaluator");
+ }
+
+ /// RAW forward + dictionary + range index + NOT_EQ predicate. NOT_EQ never uses the range index, so dict must be
+ /// dropped (no inverted/sorted in this test).
+ @Test
+ public void rawForwardWithDictAndRangeIndexDropsDictForNotEqPredicate() {
+ DataSource dataSource = mockDataSource("col", DataType.INT, /*sorted=*/false,
+ /*dict=*/true, /*forwardDictEncoded=*/false, /*inverted=*/false, /*range=*/true, /*rangeExact=*/true);
+ Dictionary result = PredicateEvaluatorProvider.getDictionaryUsableForFiltering(dataSource,
+ mockQueryContext(/*allowAllIndexes=*/true),
+ new org.apache.pinot.common.request.context.predicate.NotEqPredicate(ExpressionContext.forIdentifier("col"),
+ "7"));
+ assertNull(result, "NOT_EQ never consults the range index; dict must be dropped so scan uses raw-value evaluator");
+ }
+
+ /// RAW forward + dictionary + inverted + EQ predicate. EQ uses inverted index, so dict must be preserved.
+ @Test
+ public void rawForwardWithDictAndInvertedKeepsDictForEqPredicate() {
+ DataSource dataSource = mockDataSource("col", DataType.STRING, /*sorted=*/false,
+ /*dict=*/true, /*forwardDictEncoded=*/false, /*inverted=*/true, /*range=*/false, /*rangeExact=*/false);
+ Dictionary result = PredicateEvaluatorProvider.getDictionaryUsableForFiltering(dataSource,
+ mockQueryContext(/*allowAllIndexes=*/true), eqPredicate("col", "value-1"));
+ assertNotNull(result, "EQ predicate must keep dict so InvertedIndexFilterOperator can do dict-id lookup");
+ }
+
+ private static EqPredicate eqPredicate(String column, String value) {
+ return new EqPredicate(ExpressionContext.forIdentifier(column), value);
+ }
+
+ private static RangePredicate rangePredicate(String column, String range) {
+ return new RangePredicate(ExpressionContext.forIdentifier(column), range);
+ }
+
+ @SuppressWarnings({"rawtypes", "unchecked"})
+ private static DataSource mockDataSource(String columnName, DataType dataType, boolean sorted, boolean hasDictionary,
+ boolean forwardDictEncoded, boolean hasInverted, boolean hasRange, boolean rangeExact) {
+ DataSource dataSource = Mockito.mock(DataSource.class);
+ DataSourceMetadata metadata = Mockito.mock(DataSourceMetadata.class);
+ FieldSpec fieldSpec = new org.apache.pinot.spi.data.DimensionFieldSpec(columnName, dataType, true);
+ Mockito.when(metadata.getDataType()).thenReturn(dataType);
+ Mockito.when(metadata.isSorted()).thenReturn(sorted);
+ Mockito.when(metadata.getFieldSpec()).thenReturn(fieldSpec);
+ Mockito.when(dataSource.getDataSourceMetadata()).thenReturn(metadata);
+ Mockito.when(dataSource.getColumnName()).thenReturn(columnName);
+
+ ForwardIndexReader forwardIndex = Mockito.mock(ForwardIndexReader.class);
+ Mockito.when(forwardIndex.isDictionaryEncoded()).thenReturn(forwardDictEncoded);
+ Mockito.when(dataSource.getForwardIndex()).thenReturn(forwardIndex);
+
+ if (hasDictionary) {
+ Dictionary dictionary = Mockito.mock(Dictionary.class);
+ Mockito.when(dictionary.length()).thenReturn(0);
+ Mockito.when(dataSource.getDictionary()).thenReturn(dictionary);
+ } else {
+ Mockito.when(dataSource.getDictionary()).thenReturn(null);
+ }
+ if (hasInverted) {
+ Mockito.when(dataSource.getInvertedIndex()).thenReturn(Mockito.mock(InvertedIndexReader.class));
+ } else {
+ Mockito.when(dataSource.getInvertedIndex()).thenReturn(null);
+ }
+ if (hasRange) {
+ RangeIndexReader rangeIndex = Mockito.mock(RangeIndexReader.class);
+ Mockito.when(rangeIndex.isExact()).thenReturn(rangeExact);
+ Mockito.when(dataSource.getRangeIndex()).thenReturn(rangeIndex);
+ } else {
+ Mockito.when(dataSource.getRangeIndex()).thenReturn(null);
+ }
+ return dataSource;
+ }
+
+ private static QueryContext mockQueryContext(boolean allowAllIndexes) {
+ QueryContext queryContext = Mockito.mock(QueryContext.class);
+ Mockito.when(
+ queryContext.isIndexUseAllowed(Mockito.any(DataSource.class), Mockito.any(FieldConfig.IndexType.class)))
+ .thenReturn(allowAllIndexes);
+ Mockito.when(queryContext.isIndexUseAllowed(Mockito.anyString(), Mockito.any(FieldConfig.IndexType.class)))
+ .thenReturn(allowAllIndexes);
+ return queryContext;
+ }
+}
diff --git a/pinot-core/src/test/java/org/apache/pinot/core/operator/transform/function/BaseTransformFunctionTest.java b/pinot-core/src/test/java/org/apache/pinot/core/operator/transform/function/BaseTransformFunctionTest.java
index 37c107e4ce2..727afa78ff1 100644
--- a/pinot-core/src/test/java/org/apache/pinot/core/operator/transform/function/BaseTransformFunctionTest.java
+++ b/pinot-core/src/test/java/org/apache/pinot/core/operator/transform/function/BaseTransformFunctionTest.java
@@ -108,6 +108,17 @@ public abstract class BaseTransformFunctionTest {
protected static final String DOUBLE_MV_COLUMN_2 = "doubleMV2";
protected static final String JSON_COLUMN = "json";
protected static final String DEFAULT_JSON_COLUMN = "defaultJson";
+ /// MV INT column configured as RAW forward with an explicit shared dictionary (no secondary index).
+ /// Carries the same values as {@link #INT_MV_COLUMN} so callers can compare results against the
+ /// dict-encoded baseline. Exists to exercise the predicate-evaluator code paths for shared-dict + RAW
+ /// columns: the forward index serves raw values while a dictionary file lives alongside.
+ protected static final String INT_MV_DICT_RAW_COLUMN = "intMVDictRaw";
+ /// MV INT column configured as RAW forward with a shared dictionary AND an inverted index. Same values as
+ /// {@link #INT_MV_COLUMN} and {@link #INT_MV_DICT_RAW_COLUMN}. The inverted index is irrelevant inside
+ /// filterMv (filterMv evaluates a per-value predicate on the already-fetched MV array, not at filter-plan
+ /// time), so the result must match both other variants — but exercising this shape makes sure the inverted
+ /// index sitting on the column doesn't perturb the predicate evaluator's path selection.
+ protected static final String INT_MV_DICT_RAW_INV_COLUMN = "intMVDictRawInv";
private static final String SEGMENT_NAME = "testSegment";
private static final String INDEX_DIR_PATH = FileUtils.getTempDirectoryPath() + File.separator + SEGMENT_NAME;
private static final Random RANDOM = new Random();
@@ -242,6 +253,10 @@ public abstract class BaseTransformFunctionTest {
map.put(BYTES_SV_COLUMN, _bytesSVValues[i]);
map.put(INT_MV_COLUMN, ArrayUtils.toObject(_intMVValues[i]));
+ // Same values as INT_MV_COLUMN so callers can compare results against the dict-encoded baseline and
+ // against each other (RAW + dict only vs. RAW + dict + inverted).
+ map.put(INT_MV_DICT_RAW_COLUMN, ArrayUtils.toObject(_intMVValues[i]));
+ map.put(INT_MV_DICT_RAW_INV_COLUMN, ArrayUtils.toObject(_intMVValues[i]));
if (isNullRow(i)) {
map.put(INT_MV_NULL_COLUMN, null);
} else {
@@ -292,6 +307,8 @@ public abstract class BaseTransformFunctionTest {
.addSingleValueDimension(JSON_COLUMN, FieldSpec.DataType.JSON)
.addSingleValueDimension(DEFAULT_JSON_COLUMN, FieldSpec.DataType.JSON)
.addMultiValueDimension(INT_MV_COLUMN, FieldSpec.DataType.INT)
+ .addMultiValueDimension(INT_MV_DICT_RAW_COLUMN, FieldSpec.DataType.INT)
+ .addMultiValueDimension(INT_MV_DICT_RAW_INV_COLUMN, FieldSpec.DataType.INT)
.addMultiValueDimension(INT_MV_NULL_COLUMN, FieldSpec.DataType.INT)
.addMultiValueDimension(LONG_MV_COLUMN, FieldSpec.DataType.LONG)
.addMultiValueDimension(FLOAT_MV_COLUMN, FieldSpec.DataType.FLOAT)
@@ -344,6 +361,21 @@ public abstract class BaseTransformFunctionTest {
new FieldConfig(JSON_STRING_SV_COLUMN, FieldConfig.EncodingType.DICTIONARY, null, null, null, null, indexNode,
null, null);
fieldConfigList.add(jsonFieldConfig);
+ // Configure INT_MV_DICT_RAW_COLUMN as RAW forward with an explicit shared dictionary (no secondary
+ // index). The result on disk is the bare shared-dict + RAW shape: forward stores raw values, a
+ // dictionary file lives alongside, and nothing else depends on the dictionary.
+ ObjectNode dictOnlyIndexes = JsonNodeFactory.instance.objectNode();
+ dictOnlyIndexes.set("dictionary", JsonNodeFactory.instance.objectNode());
+ fieldConfigList.add(new FieldConfig(INT_MV_DICT_RAW_COLUMN, FieldConfig.EncodingType.RAW, null, null, null, null,
+ dictOnlyIndexes, null, null));
+ // Configure INT_MV_DICT_RAW_INV_COLUMN as RAW forward with a shared dictionary AND an inverted index.
+ // Same values as the bare variant above so callers can assert filterMv produces identical results
+ // regardless of whether an inverted index is present on the column.
+ ObjectNode dictAndInvertedIndexes = JsonNodeFactory.instance.objectNode();
+ dictAndInvertedIndexes.set("dictionary", JsonNodeFactory.instance.objectNode());
+ dictAndInvertedIndexes.set("inverted", JsonNodeFactory.instance.objectNode());
+ fieldConfigList.add(new FieldConfig(INT_MV_DICT_RAW_INV_COLUMN, FieldConfig.EncodingType.RAW, null, null, null,
+ null, dictAndInvertedIndexes, null, null));
TableConfig tableConfig =
new TableConfigBuilder(TableType.OFFLINE).setTableName("test").setTimeColumnName(TIME_COLUMN)
.setFieldConfigList(fieldConfigList).setNullHandlingEnabled(true).build();
diff --git a/pinot-core/src/test/java/org/apache/pinot/core/operator/transform/function/DateTimeConversionTransformFunctionTest.java b/pinot-core/src/test/java/org/apache/pinot/core/operator/transform/function/DateTimeConversionTransformFunctionTest.java
index 86855f564db..97e1f511165 100644
--- a/pinot-core/src/test/java/org/apache/pinot/core/operator/transform/function/DateTimeConversionTransformFunctionTest.java
+++ b/pinot-core/src/test/java/org/apache/pinot/core/operator/transform/function/DateTimeConversionTransformFunctionTest.java
@@ -32,7 +32,6 @@ import org.apache.pinot.core.common.BlockValSet;
import org.apache.pinot.core.operator.ColumnContext;
import org.apache.pinot.core.operator.blocks.ValueBlock;
import org.apache.pinot.core.operator.transform.TransformResultMetadata;
-import org.apache.pinot.segment.spi.index.reader.Dictionary;
import org.apache.pinot.spi.data.DateTimeFieldSpec;
import org.apache.pinot.spi.data.DateTimeFormatPatternSpec;
import org.apache.pinot.spi.data.FieldSpec.DataType;
@@ -920,12 +919,6 @@ public class DateTimeConversionTransformFunctionTest extends BaseTransformFuncti
return new TransformResultMetadata(DataType.LONG, true, false);
}
- @Nullable
- @Override
- public Dictionary getDictionary() {
- return null;
- }
-
@Override
public int[] transformToDictIdsSV(ValueBlock valueBlock) {
return new int[0];
diff --git a/pinot-core/src/test/java/org/apache/pinot/core/operator/transform/function/FilterMvTransformFunctionTest.java b/pinot-core/src/test/java/org/apache/pinot/core/operator/transform/function/FilterMvTransformFunctionTest.java
index d5bf95e90a8..4994fd70d35 100644
--- a/pinot-core/src/test/java/org/apache/pinot/core/operator/transform/function/FilterMvTransformFunctionTest.java
+++ b/pinot-core/src/test/java/org/apache/pinot/core/operator/transform/function/FilterMvTransformFunctionTest.java
@@ -33,6 +33,7 @@ import org.apache.pinot.common.request.context.ExpressionContext;
import org.apache.pinot.common.request.context.RequestContextUtils;
import org.apache.pinot.core.function.scalar.FilterMvScalarFunction;
import org.apache.pinot.core.operator.transform.TransformResultMetadata;
+import org.apache.pinot.segment.spi.datasource.DataSource;
import org.apache.pinot.segment.spi.index.reader.Dictionary;
import org.apache.pinot.spi.data.FieldSpec.DataType;
import org.apache.pinot.spi.exception.BadQueryRequestException;
@@ -42,6 +43,8 @@ import org.testng.annotations.Test;
import static org.testng.Assert.assertEquals;
import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertNotNull;
+import static org.testng.Assert.assertNull;
import static org.testng.Assert.assertTrue;
@@ -347,6 +350,114 @@ public class FilterMvTransformFunctionTest extends BaseTransformFunctionTest {
assertEquals(filteredAll.length, 3);
}
+ /// filterMv on a column with a shared dictionary on a RAW forward index. FilterMvTransformFunction
+ /// drops the dictionary internally because RAW forward indexes throw UnsupportedOperationException
+ /// from getDictIdMV — the dict-id path is not viable. The predicate evaluator falls back to per-value
+ /// raw matching, and the transform output must still match the dict-encoded baseline produced by the
+ /// regular `INT_MV_COLUMN` (both columns hold the same values).
+ @Test(dataProvider = "filterMvIntPredicates")
+ public void testFilterMvOnSharedDictRawForwardColumn(String predicate, IntPredicate matcher) {
+ // Sanity: the data source must actually be RAW + dict for this test to be meaningful.
+ DataSource dataSource = _dataSourceMap.get(INT_MV_DICT_RAW_COLUMN);
+ assertNotNull(dataSource);
+ assertFalse(dataSource.getForwardIndex().isDictionaryEncoded(),
+ "Pre-condition: " + INT_MV_DICT_RAW_COLUMN + " must have a RAW forward index");
+ assertNotNull(dataSource.getDictionary(),
+ "Pre-condition: " + INT_MV_DICT_RAW_COLUMN + " must carry a shared dictionary alongside the RAW forward");
+
+ String escaped = predicate.replace("'", "''");
+ String expressionStr = String.format("filterMv(%s, '%s')", INT_MV_DICT_RAW_COLUMN, escaped);
+ ExpressionContext expression = RequestContextUtils.getExpression(expressionStr);
+ TransformFunction transformFunction = TransformFunctionFactory.get(expression, _dataSourceMap);
+ assertTrue(transformFunction instanceof FilterMvTransformFunction);
+ TransformResultMetadata resultMetadata = transformFunction.getResultMetadata();
+ assertEquals(resultMetadata.getDataType(), DataType.INT);
+ assertFalse(resultMetadata.isSingleValue());
+ // FilterMvTransformFunction drops the dictionary internally for RAW forward — getDictIdMV throws
+ // UnsupportedOperationException on RAW forward indexes, so the dict-id path isn't viable. The
+ // predicate evaluator runs the raw-value matching path instead.
+ assertFalse(resultMetadata.hasDictionary(),
+ "FilterMvTransformFunction over a RAW forward column must report hasDictionary=false so the predicate "
+ + "evaluator takes the raw-value matching path");
+ assertNull(transformFunction.getDictionary());
+
+ int[][] intValuesMV = transformFunction.transformToIntValuesMV(_projectionBlock);
+ for (int i = 0; i < NUM_ROWS; i++) {
+ IntList expectedList = new IntArrayList();
+ for (int value : _intMVValues[i]) {
+ if (matcher.test(value)) {
+ expectedList.add(value);
+ }
+ }
+ int[] expectedValues = expectedList.toIntArray();
+ assertEquals(intValuesMV[i].length, expectedValues.length, "Row " + i + " predicate=" + predicate);
+ for (int j = 0; j < expectedValues.length; j++) {
+ assertEquals(intValuesMV[i][j], expectedValues[j], "Row " + i + " idx " + j + " predicate=" + predicate);
+ }
+ }
+ }
+
+ /// filterMv on a column with a shared dictionary on a RAW forward index AND an inverted index. The
+ /// inverted index doesn't influence filterMv's per-value evaluation — FilterMvTransformFunction still
+ /// drops the dictionary internally because the underlying forward index is RAW (getDictIdMV would
+ /// throw), so the result matches the dict-encoded baseline via the raw-value matching path.
+ @Test(dataProvider = "filterMvIntPredicates")
+ public void testFilterMvOnSharedDictRawForwardWithInvertedColumn(String predicate, IntPredicate matcher) {
+ // Sanity: confirm the on-disk shape is dict + inverted + RAW forward.
+ DataSource dataSource = _dataSourceMap.get(INT_MV_DICT_RAW_INV_COLUMN);
+ assertNotNull(dataSource);
+ assertFalse(dataSource.getForwardIndex().isDictionaryEncoded(),
+ "Pre-condition: " + INT_MV_DICT_RAW_INV_COLUMN + " must have a RAW forward index");
+ assertNotNull(dataSource.getDictionary(),
+ "Pre-condition: " + INT_MV_DICT_RAW_INV_COLUMN + " must carry a shared dictionary");
+ assertNotNull(dataSource.getInvertedIndex(),
+ "Pre-condition: " + INT_MV_DICT_RAW_INV_COLUMN + " must carry an inverted index");
+
+ String escaped = predicate.replace("'", "''");
+ String expressionStr = String.format("filterMv(%s, '%s')", INT_MV_DICT_RAW_INV_COLUMN, escaped);
+ ExpressionContext expression = RequestContextUtils.getExpression(expressionStr);
+ TransformFunction transformFunction = TransformFunctionFactory.get(expression, _dataSourceMap);
+ assertTrue(transformFunction instanceof FilterMvTransformFunction);
+ TransformResultMetadata resultMetadata = transformFunction.getResultMetadata();
+ assertEquals(resultMetadata.getDataType(), DataType.INT);
+ assertFalse(resultMetadata.isSingleValue());
+ // FilterMvTransformFunction drops the dictionary internally for RAW forward — even with an inverted
+ // index sitting on disk, the dict-id path is not viable because getDictIdMV throws on RAW forward.
+ assertFalse(resultMetadata.hasDictionary(),
+ "FilterMvTransformFunction over a RAW forward column must report hasDictionary=false even when an "
+ + "inverted index sits on the column");
+ assertNull(transformFunction.getDictionary());
+
+ // Compute filterMv on the same predicate over INT_MV_COLUMN (dict-encoded baseline) and over
+ // INT_MV_DICT_RAW_COLUMN (RAW + dict, no inverted) and assert all three produce identical results.
+ int[][] resultWithInverted = transformFunction.transformToIntValuesMV(_projectionBlock);
+ int[][] resultDictBaseline = filterMvIntValues(INT_MV_COLUMN, predicate);
+ int[][] resultRawDictNoInverted = filterMvIntValues(INT_MV_DICT_RAW_COLUMN, predicate);
+ for (int i = 0; i < NUM_ROWS; i++) {
+ IntList expectedList = new IntArrayList();
+ for (int value : _intMVValues[i]) {
+ if (matcher.test(value)) {
+ expectedList.add(value);
+ }
+ }
+ int[] expected = expectedList.toIntArray();
+ assertEquals(resultWithInverted[i], expected, "RAW + dict + inverted, row=" + i + " predicate=" + predicate);
+ assertEquals(resultDictBaseline[i], expected, "dict-encoded baseline, row=" + i + " predicate=" + predicate);
+ assertEquals(resultRawDictNoInverted[i], expected,
+ "RAW + dict (no inverted), row=" + i + " predicate=" + predicate);
+ }
+ }
+
+ /// Helper: run filterMv on a named column with the given predicate string and return the resulting
+ /// per-row int MV arrays. Used for cross-column result-equivalence assertions.
+ private int[][] filterMvIntValues(String columnName, String predicate) {
+ String escaped = predicate.replace("'", "''");
+ String expressionStr = String.format("filterMv(%s, '%s')", columnName, escaped);
+ ExpressionContext expression = RequestContextUtils.getExpression(expressionStr);
+ TransformFunction transformFunction = TransformFunctionFactory.get(expression, _dataSourceMap);
+ return transformFunction.transformToIntValuesMV(_projectionBlock);
+ }
+
@Test(dataProvider = "illegalArguments", expectedExceptions = {BadQueryRequestException.class})
public void testIllegalArguments(String expressionStr) {
ExpressionContext expression = RequestContextUtils.getExpression(expressionStr);
diff --git a/pinot-core/src/test/java/org/apache/pinot/queries/ForwardIndexHandlerReloadQueriesTest.java b/pinot-core/src/test/java/org/apache/pinot/queries/ForwardIndexHandlerReloadQueriesTest.java
index c8411e549dd..aaf81b89fca 100644
--- a/pinot-core/src/test/java/org/apache/pinot/queries/ForwardIndexHandlerReloadQueriesTest.java
+++ b/pinot-core/src/test/java/org/apache/pinot/queries/ForwardIndexHandlerReloadQueriesTest.java
@@ -107,12 +107,10 @@ public class ForwardIndexHandlerReloadQueriesTest extends BaseQueriesTest {
List<String> noDictionaryColumns = List.of("column1", "column2", "column3", "column5", "column7", "column10");
List<String> invertedIndexColumns = List.of("column8", "column9");
- List<FieldConfig> fieldConfigs = new ArrayList<>(noDictionaryColumns.size());
- for (String column : noDictionaryColumns) {
- fieldConfigs.add(
- new FieldConfig(column, FieldConfig.EncodingType.RAW, List.of(), FieldConfig.CompressionCodec.SNAPPY, null));
- }
- TableConfig tableConfig = createTableConfig(noDictionaryColumns, invertedIndexColumns, List.of(), fieldConfigs);
+ List<String> rangeIndexColumns = List.of();
+ List<FieldConfig> fieldConfigs = buildFieldConfigsForRawColumns(noDictionaryColumns);
+ TableConfig tableConfig = createTableConfig(noDictionaryColumns, invertedIndexColumns, rangeIndexColumns,
+ fieldConfigs);
URL resource = getClass().getClassLoader().getResource(AVRO_DATA);
assertNotNull(resource);
@@ -150,6 +148,32 @@ public class ForwardIndexHandlerReloadQueriesTest extends BaseQueriesTest {
_indexSegments = List.of(segment, segment);
}
+ private List<FieldConfig> buildFieldConfigsForRawColumns(List<String> rawColumns) {
+ return buildFieldConfigsForRawColumns(rawColumns, Map.of());
+ }
+
+ private List<FieldConfig> buildFieldConfigsForRawColumns(List<String> rawColumns,
+ Map<String, FieldConfig.CompressionCodec> compressionOverrides) {
+ List<FieldConfig> fieldConfigs = new ArrayList<>(rawColumns.size());
+ for (String column : rawColumns) {
+ FieldConfig.CompressionCodec compressionCodec =
+ compressionOverrides.getOrDefault(column, FieldConfig.CompressionCodec.SNAPPY);
+ fieldConfigs.add(new FieldConfig(column, FieldConfig.EncodingType.RAW, List.of(), compressionCodec, null));
+ }
+ return fieldConfigs;
+ }
+
+ private ImmutableSegment reloadSegment(TableConfig tableConfig)
+ throws Exception {
+ IndexLoadingConfig indexLoadingConfig = new IndexLoadingConfig(tableConfig, SCHEMA);
+ File indexDir = new File(INDEX_DIR, SEGMENT_NAME);
+ ImmutableSegment segment = ImmutableSegmentLoader.load(indexDir, indexLoadingConfig);
+ _indexSegment.destroy();
+ _indexSegment = segment;
+ _indexSegments = List.of(segment, segment);
+ return segment;
+ }
+
private TableConfig createTableConfig(List<String> noDictionaryColumns, List<String> invertedIndexColumns,
List<String> rangeIndexColumns, List<FieldConfig> fieldConfigs) {
return new TableConfigBuilder(TableType.OFFLINE).setTableName(RAW_TABLE_NAME).setTimeColumnName("daysSinceEpoch")
@@ -157,6 +181,22 @@ public class ForwardIndexHandlerReloadQueriesTest extends BaseQueriesTest {
.setRangeIndexColumns(rangeIndexColumns).setFieldConfigList(fieldConfigs).build();
}
+ private String getMostFrequentColumn5Value()
+ throws Exception {
+ BrokerResponseNative response =
+ getBrokerResponse("SELECT column5, COUNT(*) FROM testTable GROUP BY column5 ORDER BY COUNT(*) DESC LIMIT 1");
+ assertTrue(response.getExceptions() == null || response.getExceptions().isEmpty());
+ ResultTable resultTable = response.getResultTable();
+ assertNotNull(resultTable);
+ List<Object[]> rows = resultTable.getRows();
+ assertFalse(rows.isEmpty());
+ return String.valueOf(rows.get(0)[0]);
+ }
+
+ private String escapeSingleQuotes(String value) {
+ return value.replace("'", "''");
+ }
+
@AfterMethod
public void tearDown() {
_indexSegment.destroy();
@@ -218,6 +258,7 @@ public class ForwardIndexHandlerReloadQueriesTest extends BaseQueriesTest {
assertEquals(brokerResponseNative.getNumSegmentsProcessed(), 4L);
assertEquals(brokerResponseNative.getNumSegmentsMatched(), 4L);
assertEquals(brokerResponseNative.getNumEntriesScannedPostFilter(), 1384L);
+ // After reload column7 gets a dictionary + inverted index; the NOT IN filter resolves to 0 forward-index scans.
assertEquals(brokerResponseNative.getNumEntriesScannedInFilter(), 250896L);
assertNotNull(brokerResponseNative.getExceptions());
@@ -530,6 +571,8 @@ public class ForwardIndexHandlerReloadQueriesTest extends BaseQueriesTest {
changePropertiesAndReloadSegment();
// TEST1 - After reload. Test where column7 is in filter.
+ // After reload column7 gets a dictionary + inverted index; the equality filter uses the inverted index
+ // and contributes 0 entries scanned in the filter phase.
brokerResponseNative = getBrokerResponse(query1);
assertTrue(brokerResponseNative.getExceptions() == null || brokerResponseNative.getExceptions().size() == 0);
resultTable = brokerResponseNative.getResultTable();
@@ -598,6 +641,8 @@ public class ForwardIndexHandlerReloadQueriesTest extends BaseQueriesTest {
changePropertiesAndReloadSegment();
+ // After reload column9 and column10 gain range indexes; the range predicates are resolved via range index
+ // bitmap lookups (0 forward-index entries scanned in the filter phase).
brokerResponseNative = getBrokerResponse(query);
assertTrue(brokerResponseNative.getExceptions() == null || brokerResponseNative.getExceptions().size() == 0);
resultTable1 = brokerResponseNative.getResultTable();
@@ -621,16 +666,64 @@ public class ForwardIndexHandlerReloadQueriesTest extends BaseQueriesTest {
validateBeforeAfterQueryResults(resultRows1, resultRows2);
}
+ @Test
+ public void testRawForwardColumnIndexAddAndRemove()
+ throws Exception {
+ String filterValue = escapeSingleQuotes(getMostFrequentColumn5Value());
+ String filterQuery = String.format("SELECT COUNT(*) FROM %s WHERE column5 = '%s'", RAW_TABLE_NAME, filterValue);
+
+ BrokerResponseNative baselineResponse = getBrokerResponse(filterQuery);
+ assertTrue(baselineResponse.getExceptions() == null || baselineResponse.getExceptions().size() == 0);
+ ResultTable baselineResultTable = baselineResponse.getResultTable();
+ assertNotNull(baselineResultTable);
+ List<Object[]> baselineRows = baselineResultTable.getRows();
+ assertEquals(baselineRows.size(), 1);
+ long baselineCount = ((Number) baselineRows.get(0)[0]).longValue();
+ long baselineEntriesScanned = baselineResponse.getNumEntriesScannedInFilter();
+ assertTrue(baselineEntriesScanned > 0);
+
+ List<String> invertedIndexColumns = List.of("column8", "column9");
+ List<String> rangeIndexColumns = List.of();
+
+ // Enable dictionary on column5.
+ List<String> noDictionaryColumns = List.of("column1", "column2", "column3", "column7", "column10");
+ TableConfig tableConfig = createTableConfig(noDictionaryColumns, invertedIndexColumns, rangeIndexColumns,
+ buildFieldConfigsForRawColumns(noDictionaryColumns));
+ ImmutableSegment segment = reloadSegment(tableConfig);
+ assertTrue(segment.getSegmentMetadata().getColumnMetadataFor("column5").hasDictionary());
+ BrokerResponseNative dictResponse = getBrokerResponse(filterQuery);
+ assertTrue(dictResponse.getExceptions() == null || dictResponse.getExceptions().size() == 0);
+ ResultTable dictResultTable = dictResponse.getResultTable();
+ assertNotNull(dictResultTable);
+ List<Object[]> dictRows = dictResultTable.getRows();
+ assertEquals(dictRows.size(), 1);
+ assertEquals(((Number) dictRows.get(0)[0]).longValue(), baselineCount);
+
+ // Disable dictionary on column5 again.
+ noDictionaryColumns = List.of("column1", "column2", "column3", "column5", "column7", "column10");
+ tableConfig = createTableConfig(noDictionaryColumns, invertedIndexColumns, rangeIndexColumns,
+ buildFieldConfigsForRawColumns(noDictionaryColumns));
+ segment = reloadSegment(tableConfig);
+ assertFalse(segment.getSegmentMetadata().getColumnMetadataFor("column5").hasDictionary());
+ BrokerResponseNative rawResponse = getBrokerResponse(filterQuery);
+ assertTrue(rawResponse.getExceptions() == null || rawResponse.getExceptions().size() == 0);
+ ResultTable rawResultTable = rawResponse.getResultTable();
+ assertNotNull(rawResultTable);
+ List<Object[]> rawRows = rawResultTable.getRows();
+ assertEquals(rawRows.size(), 1);
+ assertEquals(((Number) rawRows.get(0)[0]).longValue(), baselineCount);
+ }
+
/**
* As a part of segmentReload, the ForwardIndexHandler will perform the following operations:
*
- * column1 -> change compression.
- * column6 -> disable dictionary
- * column9 -> disable dictionary
- * column3 -> Enable dictionary.
- * column2 -> Enable dictionary. Add inverted index.
- * column7 -> Enable dictionary. Add inverted index.
- * column10 -> Enable dictionary.
+ * column1 -> change compression (SNAPPY -> ZSTANDARD).
+ * column6 -> disable dictionary.
+ * column9 -> disable dictionary. Add range index.
+ * column3 -> enable dictionary.
+ * column2 -> enable dictionary. Add inverted index.
+ * column7 -> enable dictionary. Add inverted index.
+ * column10 -> enable dictionary. Add range index.
*/
private void changePropertiesAndReloadSegment()
throws Exception {
@@ -645,16 +738,9 @@ public class ForwardIndexHandlerReloadQueriesTest extends BaseQueriesTest {
}
fieldConfigs.add(new FieldConfig(column, FieldConfig.EncodingType.RAW, List.of(), compressionCodec, null));
}
- TableConfig tableConfig =
- createTableConfig(noDictionaryColumns, invertedIndexColumns, rangeIndexColumns, fieldConfigs);
- IndexLoadingConfig indexLoadingConfig = new IndexLoadingConfig(tableConfig, SCHEMA);
-
- // Reload the segments to pick up the new configs
- File indexDir = new File(INDEX_DIR, SEGMENT_NAME);
- ImmutableSegment segment = ImmutableSegmentLoader.load(indexDir, indexLoadingConfig);
- _indexSegment.destroy();
- _indexSegment = segment;
- _indexSegments = List.of(segment, segment);
+ TableConfig tableConfig = createTableConfig(noDictionaryColumns, invertedIndexColumns, rangeIndexColumns,
+ fieldConfigs);
+ ImmutableSegment segment = reloadSegment(tableConfig);
Map<String, ColumnMetadata> columnMetadataMap = segment.getSegmentMetadata().getColumnMetadataMap();
for (Map.Entry<String, ColumnMetadata> entry : columnMetadataMap.entrySet()) {
diff --git a/pinot-core/src/test/resources/TableIndexingTest.csv b/pinot-core/src/test/resources/TableIndexingTest.csv
index 941712e8ba0..01ea7b7fee6 100644
--- a/pinot-core/src/test/resources/TableIndexingTest.csv
+++ b/pinot-core/src/test/resources/TableIndexingTest.csv
@@ -1,7 +1,7 @@
data_type;cardinality;encoding;index_type;success;error
INT;sv;raw;timestamp_index;false;Cannot create TIMESTAMP index on column: col of stored type other than LONG
INT;sv;raw;bloom_filter;true;
-INT;sv;raw;fst_index;false;Cannot create FST index on column: col without dictionary
+INT;sv;raw;fst_index;false;Cannot create FST index on column: col of stored type other than STRING
INT;sv;raw;h3_index;false;Cannot create H3 index on column: col of stored type other than BYTES
INT;sv;raw;inverted_index;false;Cannot create inverted index on column: col without dictionary
INT;sv;raw;json_index;false;Cannot create JSON index on column: col of stored type other than STRING or MAP
@@ -12,7 +12,7 @@ INT;sv;raw;vector_index;false;Cannot create vector index on single-value column:
INT;sv;raw;multi_col_text_index;false;Cannot create TEXT index on column: col of stored type other than STRING
INT;mv;raw;timestamp_index;false;Cannot create TIMESTAMP index on column: col of stored type other than LONG
INT;mv;raw;bloom_filter;true;
-INT;mv;raw;fst_index;false;Cannot create FST index on column: col without dictionary
+INT;mv;raw;fst_index;false;Cannot create FST index on multi-value column: col
INT;mv;raw;h3_index;false;Cannot create H3 index on multi-value column: col
INT;mv;raw;inverted_index;false;Cannot create inverted index on column: col without dictionary
INT;mv;raw;json_index;false;Cannot create JSON index on multi-value column: col
@@ -45,7 +45,7 @@ INT;mv;dict;vector_index;false;Cannot create vector index on column: col of stor
INT;mv;dict;multi_col_text_index;false;Cannot create TEXT index on column: col of stored type other than STRING
LONG;sv;raw;timestamp_index;true;
LONG;sv;raw;bloom_filter;true;
-LONG;sv;raw;fst_index;false;Cannot create FST index on column: col without dictionary
+LONG;sv;raw;fst_index;false;Cannot create FST index on column: col of stored type other than STRING
LONG;sv;raw;h3_index;false;Cannot create H3 index on column: col of stored type other than BYTES
LONG;sv;raw;inverted_index;false;Cannot create inverted index on column: col without dictionary
LONG;sv;raw;json_index;false;Cannot create JSON index on column: col of stored type other than STRING or MAP
@@ -56,7 +56,7 @@ LONG;sv;raw;vector_index;false;Cannot create vector index on single-value column
LONG;sv;raw;multi_col_text_index;false;Cannot create TEXT index on column: col of stored type other than STRING
LONG;mv;raw;timestamp_index;false;Cannot create TIMESTAMP index on multi-value column: col
LONG;mv;raw;bloom_filter;true;
-LONG;mv;raw;fst_index;false;Cannot create FST index on column: col without dictionary
+LONG;mv;raw;fst_index;false;Cannot create FST index on multi-value column: col
LONG;mv;raw;h3_index;false;Cannot create H3 index on multi-value column: col
LONG;mv;raw;inverted_index;false;Cannot create inverted index on column: col without dictionary
LONG;mv;raw;json_index;false;Cannot create JSON index on multi-value column: col
@@ -89,7 +89,7 @@ LONG;mv;dict;vector_index;false;Cannot create vector index on column: col of sto
LONG;mv;dict;multi_col_text_index;false;Cannot create TEXT index on column: col of stored type other than STRING
FLOAT;sv;raw;timestamp_index;false;Cannot create TIMESTAMP index on column: col of stored type other than LONG
FLOAT;sv;raw;bloom_filter;true;
-FLOAT;sv;raw;fst_index;false;Cannot create FST index on column: col without dictionary
+FLOAT;sv;raw;fst_index;false;Cannot create FST index on column: col of stored type other than STRING
FLOAT;sv;raw;h3_index;false;Cannot create H3 index on column: col of stored type other than BYTES
FLOAT;sv;raw;inverted_index;false;Cannot create inverted index on column: col without dictionary
FLOAT;sv;raw;json_index;false;Cannot create JSON index on column: col of stored type other than STRING or MAP
@@ -100,7 +100,7 @@ FLOAT;sv;raw;vector_index;false;Cannot create vector index on single-value colum
FLOAT;sv;raw;multi_col_text_index;false;Cannot create TEXT index on column: col of stored type other than STRING
FLOAT;mv;raw;timestamp_index;false;Cannot create TIMESTAMP index on column: col of stored type other than LONG
FLOAT;mv;raw;bloom_filter;true;
-FLOAT;mv;raw;fst_index;false;Cannot create FST index on column: col without dictionary
+FLOAT;mv;raw;fst_index;false;Cannot create FST index on multi-value column: col
FLOAT;mv;raw;h3_index;false;Cannot create H3 index on multi-value column: col
FLOAT;mv;raw;inverted_index;false;Cannot create inverted index on column: col without dictionary
FLOAT;mv;raw;json_index;false;Cannot create JSON index on multi-value column: col
@@ -133,7 +133,7 @@ FLOAT;mv;dict;vector_index;true;
FLOAT;mv;dict;multi_col_text_index;false;Cannot create TEXT index on column: col of stored type other than STRING
DOUBLE;sv;raw;timestamp_index;false;Cannot create TIMESTAMP index on column: col of stored type other than LONG
DOUBLE;sv;raw;bloom_filter;true;
-DOUBLE;sv;raw;fst_index;false;Cannot create FST index on column: col without dictionary
+DOUBLE;sv;raw;fst_index;false;Cannot create FST index on column: col of stored type other than STRING
DOUBLE;sv;raw;h3_index;false;Cannot create H3 index on column: col of stored type other than BYTES
DOUBLE;sv;raw;inverted_index;false;Cannot create inverted index on column: col without dictionary
DOUBLE;sv;raw;json_index;false;Cannot create JSON index on column: col of stored type other than STRING or MAP
@@ -144,7 +144,7 @@ DOUBLE;sv;raw;vector_index;false;Cannot create vector index on single-value colu
DOUBLE;sv;raw;multi_col_text_index;false;Cannot create TEXT index on column: col of stored type other than STRING
DOUBLE;mv;raw;timestamp_index;false;Cannot create TIMESTAMP index on column: col of stored type other than LONG
DOUBLE;mv;raw;bloom_filter;true;
-DOUBLE;mv;raw;fst_index;false;Cannot create FST index on column: col without dictionary
+DOUBLE;mv;raw;fst_index;false;Cannot create FST index on multi-value column: col
DOUBLE;mv;raw;h3_index;false;Cannot create H3 index on multi-value column: col
DOUBLE;mv;raw;inverted_index;false;Cannot create inverted index on column: col without dictionary
DOUBLE;mv;raw;json_index;false;Cannot create JSON index on multi-value column: col
@@ -177,7 +177,7 @@ DOUBLE;mv;dict;vector_index;false;Cannot create vector index on column: col of s
DOUBLE;mv;dict;multi_col_text_index;false;Cannot create TEXT index on column: col of stored type other than STRING
DECIMAL;sv_BIG;raw;timestamp_index;false;Cannot create TIMESTAMP index on column: col of stored type other than LONG
DECIMAL;sv_BIG;raw;bloom_filter;true;
-DECIMAL;sv_BIG;raw;fst_index;false;Cannot create FST index on column: col without dictionary
+DECIMAL;sv_BIG;raw;fst_index;false;Cannot create FST index on column: col of stored type other than STRING
DECIMAL;sv_BIG;raw;h3_index;false;Cannot create H3 index on column: col of stored type other than BYTES
DECIMAL;sv_BIG;raw;inverted_index;false;Cannot create inverted index on column: col without dictionary
DECIMAL;sv_BIG;raw;json_index;false;Cannot create JSON index on column: col of stored type other than STRING or MAP
@@ -199,7 +199,7 @@ DECIMAL;sv_BIG;dict;vector_index;false;Cannot create vector index on single-valu
DECIMAL;sv_BIG;dict;multi_col_text_index;false;Cannot create TEXT index on column: col of stored type other than STRING
BOOLEAN;sv;raw;timestamp_index;false;Cannot create TIMESTAMP index on column: col of stored type other than LONG
BOOLEAN;sv;raw;bloom_filter;false;Cannot create bloom filter on BOOLEAN column: col
-BOOLEAN;sv;raw;fst_index;false;Cannot create FST index on column: col without dictionary
+BOOLEAN;sv;raw;fst_index;false;Cannot create FST index on column: col of stored type other than STRING
BOOLEAN;sv;raw;h3_index;false;Cannot create H3 index on column: col of stored type other than BYTES
BOOLEAN;sv;raw;inverted_index;false;Cannot create inverted index on column: col without dictionary
BOOLEAN;sv;raw;json_index;false;Cannot create JSON index on column: col of stored type other than STRING or MAP
@@ -210,7 +210,7 @@ BOOLEAN;sv;raw;vector_index;false;Cannot create vector index on single-value col
BOOLEAN;sv;raw;multi_col_text_index;false;Cannot create TEXT index on column: col of stored type other than STRING
BOOLEAN;mv;raw;timestamp_index;false;Cannot create TIMESTAMP index on column: col of stored type other than LONG
BOOLEAN;mv;raw;bloom_filter;false;Cannot create bloom filter on BOOLEAN column: col
-BOOLEAN;mv;raw;fst_index;false;Cannot create FST index on column: col without dictionary
+BOOLEAN;mv;raw;fst_index;false;Cannot create FST index on multi-value column: col
BOOLEAN;mv;raw;h3_index;false;Cannot create H3 index on multi-value column: col
BOOLEAN;mv;raw;inverted_index;false;Cannot create inverted index on column: col without dictionary
BOOLEAN;mv;raw;json_index;false;Cannot create JSON index on multi-value column: col
@@ -243,7 +243,7 @@ BOOLEAN;mv;dict;vector_index;false;Cannot create vector index on column: col of
BOOLEAN;mv;dict;multi_col_text_index;false;Cannot create TEXT index on column: col of stored type other than STRING
TIMESTAMP;sv;raw;timestamp_index;true;
TIMESTAMP;sv;raw;bloom_filter;true;
-TIMESTAMP;sv;raw;fst_index;false;Cannot create FST index on column: col without dictionary
+TIMESTAMP;sv;raw;fst_index;false;Cannot create FST index on column: col of stored type other than STRING
TIMESTAMP;sv;raw;h3_index;false;Cannot create H3 index on column: col of stored type other than BYTES
TIMESTAMP;sv;raw;inverted_index;false;Cannot create inverted index on column: col without dictionary
TIMESTAMP;sv;raw;json_index;false;Cannot create JSON index on column: col of stored type other than STRING or MAP
@@ -254,7 +254,7 @@ TIMESTAMP;sv;raw;vector_index;false;Cannot create vector index on single-value c
TIMESTAMP;sv;raw;multi_col_text_index;false;Cannot create TEXT index on column: col of stored type other than STRING
TIMESTAMP;mv;raw;timestamp_index;false;Cannot create TIMESTAMP index on multi-value column: col
TIMESTAMP;mv;raw;bloom_filter;true;
-TIMESTAMP;mv;raw;fst_index;false;Cannot create FST index on column: col without dictionary
+TIMESTAMP;mv;raw;fst_index;false;Cannot create FST index on multi-value column: col
TIMESTAMP;mv;raw;h3_index;false;Cannot create H3 index on multi-value column: col
TIMESTAMP;mv;raw;inverted_index;false;Cannot create inverted index on column: col without dictionary
TIMESTAMP;mv;raw;json_index;false;Cannot create JSON index on multi-value column: col
@@ -287,7 +287,7 @@ TIMESTAMP;mv;dict;vector_index;false;Cannot create vector index on column: col o
TIMESTAMP;mv;dict;multi_col_text_index;false;Cannot create TEXT index on column: col of stored type other than STRING
STRING;sv;raw;timestamp_index;false;Cannot create TIMESTAMP index on column: col of stored type other than LONG
STRING;sv;raw;bloom_filter;true;
-STRING;sv;raw;fst_index;false;Cannot create FST index on column: col without dictionary
+STRING;sv;raw;fst_index;true;
STRING;sv;raw;h3_index;false;Cannot create H3 index on column: col of stored type other than BYTES
STRING;sv;raw;inverted_index;false;Cannot create inverted index on column: col without dictionary
STRING;sv;raw;json_index;false;Column: col Unrecognized token 'str': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false') at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 1, column: 15]
@@ -298,7 +298,7 @@ STRING;sv;raw;vector_index;false;Cannot create vector index on single-value colu
STRING;sv;raw;multi_col_text_index;true;
STRING;mv;raw;timestamp_index;false;Cannot create TIMESTAMP index on column: col of stored type other than LONG
STRING;mv;raw;bloom_filter;true;
-STRING;mv;raw;fst_index;false;Cannot create FST index on column: col without dictionary
+STRING;mv;raw;fst_index;false;Cannot create FST index on multi-value column: col
STRING;mv;raw;h3_index;false;Cannot create H3 index on multi-value column: col
STRING;mv;raw;inverted_index;false;Cannot create inverted index on column: col without dictionary
STRING;mv;raw;json_index;false;Cannot create JSON index on multi-value column: col
@@ -331,7 +331,7 @@ STRING;mv;dict;vector_index;false;Cannot create vector index on column: col of s
STRING;mv;dict;multi_col_text_index;true;
JSON;sv;raw;timestamp_index;false;Cannot create TIMESTAMP index on column: col of stored type other than LONG
JSON;sv;raw;bloom_filter;true;
-JSON;sv;raw;fst_index;false;Cannot create FST index on column: col without dictionary
+JSON;sv;raw;fst_index;true;
JSON;sv;raw;h3_index;false;Cannot create H3 index on column: col of stored type other than BYTES
JSON;sv;raw;inverted_index;false;Cannot create inverted index on column: col without dictionary
JSON;sv;raw;json_index;true;
@@ -353,7 +353,7 @@ JSON;sv;dict;vector_index;false;Cannot create vector index on single-value colum
JSON;sv;dict;multi_col_text_index;false;Cannot create TEXT index on column: col of stored type other than STRING
BYTES;sv;raw;timestamp_index;false;Cannot create TIMESTAMP index on column: col of stored type other than LONG
BYTES;sv;raw;bloom_filter;false;Caught exception while reading data
-BYTES;sv;raw;fst_index;false;Cannot create FST index on column: col without dictionary
+BYTES;sv;raw;fst_index;false;Cannot create FST index on column: col of stored type other than STRING
BYTES;sv;raw;h3_index;false;Caught exception while reading data
BYTES;sv;raw;inverted_index;false;Cannot create inverted index on column: col without dictionary
BYTES;sv;raw;json_index;false;Cannot create JSON index on column: col of stored type other than STRING or MAP
@@ -364,7 +364,7 @@ BYTES;sv;raw;vector_index;false;Cannot create vector index on single-value colum
BYTES;sv;raw;multi_col_text_index;false;Caught exception while reading data
BYTES;mv;raw;timestamp_index;false;Cannot create TIMESTAMP index on column: col of stored type other than LONG
BYTES;mv;raw;bloom_filter;false;Caught exception while reading data
-BYTES;mv;raw;fst_index;false;Cannot create FST index on column: col without dictionary
+BYTES;mv;raw;fst_index;false;Cannot create FST index on multi-value column: col
BYTES;mv;raw;h3_index;false;Cannot create H3 index on multi-value column: col
BYTES;mv;raw;inverted_index;false;Cannot create inverted index on column: col without dictionary
BYTES;mv;raw;json_index;false;Cannot create JSON index on multi-value column: col
@@ -397,7 +397,7 @@ BYTES;mv;dict;vector_index;false;Cannot create vector index on column: col of st
BYTES;mv;dict;multi_col_text_index;false;Caught exception while reading data
STRING;map;raw;timestamp_index;false;Cannot create TIMESTAMP index on column: col of stored type other than LONG
STRING;map;raw;bloom_filter;false;Cannot create bloom filter on MAP column: col
-STRING;map;raw;fst_index;false;Cannot create FST index on column: col without dictionary
+STRING;map;raw;fst_index;false;Cannot create FST index on column: col of stored type other than STRING
STRING;map;raw;h3_index;false;Cannot create H3 index on column: col of stored type other than BYTES
STRING;map;raw;inverted_index;false;Cannot create inverted index on column: col without dictionary
STRING;map;raw;json_index;true;
@@ -419,7 +419,7 @@ STRING;map;dict;vector_index;false;Cannot create vector index on single-value co
STRING;map;dict;multi_col_text_index;false;Cannot create TEXT index on column: col of stored type other than STRING
INT;map;raw;timestamp_index;false;Cannot create TIMESTAMP index on column: col of stored type other than LONG
INT;map;raw;bloom_filter;false;Cannot create bloom filter on MAP column: col
-INT;map;raw;fst_index;false;Cannot create FST index on column: col without dictionary
+INT;map;raw;fst_index;false;Cannot create FST index on column: col of stored type other than STRING
INT;map;raw;h3_index;false;Cannot create H3 index on column: col of stored type other than BYTES
INT;map;raw;inverted_index;false;Cannot create inverted index on column: col without dictionary
INT;map;raw;json_index;true;
@@ -441,7 +441,7 @@ INT;map;dict;vector_index;false;Cannot create vector index on single-value colum
INT;map;dict;multi_col_text_index;false;Cannot create TEXT index on column: col of stored type other than STRING
LONG;map;raw;timestamp_index;false;Cannot create TIMESTAMP index on column: col of stored type other than LONG
LONG;map;raw;bloom_filter;false;Cannot create bloom filter on MAP column: col
-LONG;map;raw;fst_index;false;Cannot create FST index on column: col without dictionary
+LONG;map;raw;fst_index;false;Cannot create FST index on column: col of stored type other than STRING
LONG;map;raw;h3_index;false;Cannot create H3 index on column: col of stored type other than BYTES
LONG;map;raw;inverted_index;false;Cannot create inverted index on column: col without dictionary
LONG;map;raw;json_index;true;
@@ -463,7 +463,7 @@ LONG;map;dict;vector_index;false;Cannot create vector index on single-value colu
LONG;map;dict;multi_col_text_index;false;Cannot create TEXT index on column: col of stored type other than STRING
FLOAT;map;raw;timestamp_index;false;Cannot create TIMESTAMP index on column: col of stored type other than LONG
FLOAT;map;raw;bloom_filter;false;Cannot create bloom filter on MAP column: col
-FLOAT;map;raw;fst_index;false;Cannot create FST index on column: col without dictionary
+FLOAT;map;raw;fst_index;false;Cannot create FST index on column: col of stored type other than STRING
FLOAT;map;raw;h3_index;false;Cannot create H3 index on column: col of stored type other than BYTES
FLOAT;map;raw;inverted_index;false;Cannot create inverted index on column: col without dictionary
FLOAT;map;raw;json_index;true;
@@ -485,7 +485,7 @@ FLOAT;map;dict;vector_index;false;Cannot create vector index on single-value col
FLOAT;map;dict;multi_col_text_index;false;Cannot create TEXT index on column: col of stored type other than STRING
DOUBLE;map;raw;timestamp_index;false;Cannot create TIMESTAMP index on column: col of stored type other than LONG
DOUBLE;map;raw;bloom_filter;false;Cannot create bloom filter on MAP column: col
-DOUBLE;map;raw;fst_index;false;Cannot create FST index on column: col without dictionary
+DOUBLE;map;raw;fst_index;false;Cannot create FST index on column: col of stored type other than STRING
DOUBLE;map;raw;h3_index;false;Cannot create H3 index on column: col of stored type other than BYTES
DOUBLE;map;raw;inverted_index;false;Cannot create inverted index on column: col without dictionary
DOUBLE;map;raw;json_index;true;
diff --git a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/custom/RawForwardIndexWithDictionaryTest.java b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/custom/RawForwardIndexWithDictionaryTest.java
new file mode 100644
index 00000000000..eebde67c7f8
--- /dev/null
+++ b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/custom/RawForwardIndexWithDictionaryTest.java
@@ -0,0 +1,601 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.integration.tests.custom;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import java.util.Random;
+import org.apache.avro.SchemaBuilder;
+import org.apache.avro.file.DataFileWriter;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericDatumWriter;
+import org.apache.pinot.spi.config.table.FieldConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertNotNull;
+import static org.testng.Assert.assertTrue;
+
+
+/**
+ * Integration test that verifies a column with RAW forward index encoding plus an explicitly configured dictionary
+ * returns identical query results to a baseline column that uses the default dictionary-encoded forward index.
+ * Three column shapes are exercised so the predicate-evaluator selection logic is covered for each:
+ * <ul>
+ * <li>{@code rawDictDim}: RAW forward + explicit dictionary, no secondary index — exercises the scan path that
+ * requires dropping the dictionary so the raw-value evaluator is used.</li>
+ * <li>{@code rawDictInvDim}: RAW forward + explicit dictionary + inverted index — exercises the inverted-index
+ * path that requires keeping the dictionary so dict IDs can be looked up.</li>
+ * <li>{@code dictDim}: regular dictionary-encoded baseline.</li>
+ * </ul>
+ * All three columns get identical values so query outputs must match exactly across filters, aggregations,
+ * GROUP BY, DISTINCT, IN, and REGEXP_LIKE predicates.
+ */
+@Test(suiteName = "CustomClusterIntegrationTest")
+public class RawForwardIndexWithDictionaryTest extends CustomDataQueryClusterIntegrationTest {
+ private static final String TABLE_NAME = "RawForwardIndexWithDictionaryTest";
+ private static final String DICT_DIMENSION = "dictDim";
+ private static final String RAW_DICT_DIMENSION = "rawDictDim";
+ private static final String RAW_DICT_INV_DIMENSION = "rawDictInvDim";
+ private static final String DICT_INT_DIMENSION = "dictIntDim";
+ private static final String RAW_DICT_INT_DIMENSION = "rawDictIntDim";
+ private static final String RAW_DICT_INV_INT_DIMENSION = "rawDictInvIntDim";
+ private static final String RAW_DICT_RANGE_INT_DIMENSION = "rawDictRangeIntDim";
+ private static final String METRIC_COLUMN = "metric";
+ private static final int ROW_COUNT = 1000;
+ private static final int UNIQUE_DIMENSION_VALUES = 20;
+
+ @Override
+ protected long getCountStarResult() {
+ return ROW_COUNT;
+ }
+
+ @Override
+ public String getTableName() {
+ return TABLE_NAME;
+ }
+
+ @Override
+ public TableConfig createOfflineTableConfig() {
+ return new TableConfigBuilder(TableType.OFFLINE).setTableName(getTableName())
+ .setTimeColumnName(getTimeColumnName())
+ .setNoDictionaryColumns(getNoDictionaryColumns())
+ .setInvertedIndexColumns(getInvertedIndexColumns())
+ .setRangeIndexColumns(getRangeIndexColumns())
+ .setFieldConfigList(getFieldConfigs())
+ .setNumReplicas(getNumReplicas())
+ .setSegmentVersion(getSegmentVersion())
+ .setLoadMode(getLoadMode())
+ .setBrokerTenant(getBrokerTenant())
+ .setServerTenant(getServerTenant())
+ .setIngestionConfig(getIngestionConfig())
+ .setQueryConfig(getQueryConfig())
+ .setNullHandlingEnabled(getNullHandlingEnabled())
+ .build();
+ }
+
+ @Override
+ public Schema createSchema() {
+ return new Schema.SchemaBuilder().setSchemaName(getTableName())
+ .addSingleValueDimension(DICT_DIMENSION, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(RAW_DICT_DIMENSION, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(RAW_DICT_INV_DIMENSION, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(DICT_INT_DIMENSION, FieldSpec.DataType.INT)
+ .addSingleValueDimension(RAW_DICT_INT_DIMENSION, FieldSpec.DataType.INT)
+ .addSingleValueDimension(RAW_DICT_INV_INT_DIMENSION, FieldSpec.DataType.INT)
+ .addSingleValueDimension(RAW_DICT_RANGE_INT_DIMENSION, FieldSpec.DataType.INT)
+ .addMetric(METRIC_COLUMN, FieldSpec.DataType.LONG)
+ .addDateTime(TIMESTAMP_FIELD_NAME, FieldSpec.DataType.LONG, "1:MILLISECONDS:EPOCH", "1:MILLISECONDS")
+ .build();
+ }
+
+ @Override
+ public List<File> createAvroFiles()
+ throws Exception {
+ org.apache.avro.Schema avroSchema = SchemaBuilder.record("RawForwardIndexWithDictionaryRecord").fields()
+ .requiredString(DICT_DIMENSION)
+ .requiredString(RAW_DICT_DIMENSION)
+ .requiredString(RAW_DICT_INV_DIMENSION)
+ .requiredInt(DICT_INT_DIMENSION)
+ .requiredInt(RAW_DICT_INT_DIMENSION)
+ .requiredInt(RAW_DICT_INV_INT_DIMENSION)
+ .requiredInt(RAW_DICT_RANGE_INT_DIMENSION)
+ .requiredLong(METRIC_COLUMN)
+ .requiredLong(TIMESTAMP_FIELD_NAME)
+ .endRecord();
+
+ File avroFile = new File(_tempDir, "raw-forward-with-dictionary.avro");
+ try (DataFileWriter<GenericData.Record> fileWriter = new DataFileWriter<>(new GenericDatumWriter<>(avroSchema))) {
+ fileWriter.create(avroSchema, avroFile);
+ Random random = new Random(1234);
+ long currentTimeMillis = System.currentTimeMillis();
+ for (int i = 0; i < ROW_COUNT; i++) {
+ GenericData.Record record = new GenericData.Record(avroSchema);
+ String stringValue = "value-" + (i % UNIQUE_DIMENSION_VALUES);
+ int intValue = i % UNIQUE_DIMENSION_VALUES;
+ // All three string columns and four int columns get IDENTICAL values so query results must match exactly.
+ record.put(DICT_DIMENSION, stringValue);
+ record.put(RAW_DICT_DIMENSION, stringValue);
+ record.put(RAW_DICT_INV_DIMENSION, stringValue);
+ record.put(DICT_INT_DIMENSION, intValue);
+ record.put(RAW_DICT_INT_DIMENSION, intValue);
+ record.put(RAW_DICT_INV_INT_DIMENSION, intValue);
+ record.put(RAW_DICT_RANGE_INT_DIMENSION, intValue);
+ record.put(METRIC_COLUMN, random.nextInt(10_000));
+ record.put(TIMESTAMP_FIELD_NAME, currentTimeMillis + i);
+ fileWriter.append(record);
+ }
+ }
+ return List.of(avroFile);
+ }
+
+ @Override
+ protected List<String> getNoDictionaryColumns() {
+ // Forward-index encoding for the raw-dict columns is set via FieldConfig (RAW + explicit dictionary).
+ return List.of();
+ }
+
+ @Override
+ protected List<FieldConfig> getFieldConfigs() {
+ ObjectNode dictionaryIndex = JsonUtils.newObjectNode();
+ dictionaryIndex.set("dictionary", JsonUtils.newObjectNode());
+ FieldConfig rawDictString =
+ new FieldConfig(RAW_DICT_DIMENSION, FieldConfig.EncodingType.RAW, null, null, null, null, dictionaryIndex,
+ null, null);
+ FieldConfig rawDictInvString =
+ new FieldConfig(RAW_DICT_INV_DIMENSION, FieldConfig.EncodingType.RAW, null, null, null, null,
+ dictionaryIndex.deepCopy(), null, null);
+ FieldConfig rawDictInt =
+ new FieldConfig(RAW_DICT_INT_DIMENSION, FieldConfig.EncodingType.RAW, null, null, null, null,
+ dictionaryIndex.deepCopy(), null, null);
+ FieldConfig rawDictInvInt =
+ new FieldConfig(RAW_DICT_INV_INT_DIMENSION, FieldConfig.EncodingType.RAW, null, null, null, null,
+ dictionaryIndex.deepCopy(), null, null);
+ FieldConfig rawDictRangeInt =
+ new FieldConfig(RAW_DICT_RANGE_INT_DIMENSION, FieldConfig.EncodingType.RAW, null, null, null, null,
+ dictionaryIndex.deepCopy(), null, null);
+ return List.of(rawDictString, rawDictInvString, rawDictInt, rawDictInvInt, rawDictRangeInt);
+ }
+
+ @Override
+ protected List<String> getInvertedIndexColumns() {
+ return List.of(RAW_DICT_INV_DIMENSION, RAW_DICT_INV_INT_DIMENSION);
+ }
+
+ @Override
+ protected List<String> getRangeIndexColumns() {
+ return List.of(RAW_DICT_RANGE_INT_DIMENSION);
+ }
+
+ @Test
+ public void testSegmentMetadataHasDictionaryAndRawForwardIndex()
+ throws IOException {
+ JsonNode segmentsMetadata = JsonUtils.stringToJsonNode(sendGetRequest(_controllerRequestURLBuilder
+ .forSegmentsMetadataFromServer(getTableName(),
+ List.of(RAW_DICT_DIMENSION, RAW_DICT_INV_DIMENSION, RAW_DICT_INT_DIMENSION,
+ RAW_DICT_INV_INT_DIMENSION, RAW_DICT_RANGE_INT_DIMENSION))));
+ assertFalse(segmentsMetadata.isEmpty());
+ for (String column : List.of(RAW_DICT_DIMENSION, RAW_DICT_INV_DIMENSION, RAW_DICT_INT_DIMENSION,
+ RAW_DICT_INV_INT_DIMENSION, RAW_DICT_RANGE_INT_DIMENSION)) {
+ JsonNode columnMetadata = findColumnMetadata(segmentsMetadata, column);
+ assertNotNull(columnMetadata, "Metadata for " + column + " not found");
+ assertTrue(columnMetadata.get("hasDictionary").asBoolean(),
+ "Dictionary should be built for explicitly configured dictionary on raw-encoded column: " + column);
+ }
+ }
+
+ @Test(dataProvider = "useBothQueryEngines")
+ public void testEqualityFilterReturnsSameResults(boolean useMultiStageQueryEngine)
+ throws Exception {
+ setUseMultiStageQueryEngine(useMultiStageQueryEngine);
+ for (int i = 0; i < UNIQUE_DIMENSION_VALUES; i++) {
+ String value = "value-" + i;
+ assertScalarLongMatches(
+ String.format("SELECT COUNT(*) FROM %s WHERE %s = '%s'", getTableName(), DICT_DIMENSION, value),
+ String.format("SELECT COUNT(*) FROM %s WHERE %s = '%s'", getTableName(), RAW_DICT_DIMENSION, value));
+ // RAW + dictionary + inverted index: dictionary must be preserved so the inverted-index path works.
+ assertScalarLongMatches(
+ String.format("SELECT COUNT(*) FROM %s WHERE %s = '%s'", getTableName(), DICT_DIMENSION, value),
+ String.format("SELECT COUNT(*) FROM %s WHERE %s = '%s'", getTableName(), RAW_DICT_INV_DIMENSION, value));
+ assertScalarLongMatches(
+ String.format("SELECT COUNT(*) FROM %s WHERE %s = %d", getTableName(), DICT_INT_DIMENSION, i),
+ String.format("SELECT COUNT(*) FROM %s WHERE %s = %d", getTableName(), RAW_DICT_INT_DIMENSION, i));
+ }
+ }
+
+ @Test
+ public void testEqualityFilterWithSkipInvertedIndex()
+ throws Exception {
+ // Force scan path for the inverted-indexed column. The planner must recognize that the inverted index is disabled
+ // for this query and fall back to the raw-value evaluator instead of the dict-based one.
+ setUseMultiStageQueryEngine(false);
+ String prefix = "SET skipIndexes='" + RAW_DICT_INV_DIMENSION + "=inverted'; ";
+ assertScalarLongMatches(
+ prefix + String.format("SELECT COUNT(*) FROM %s WHERE %s = 'value-3'", getTableName(), DICT_DIMENSION),
+ prefix + String.format("SELECT COUNT(*) FROM %s WHERE %s = 'value-3'", getTableName(), RAW_DICT_INV_DIMENSION));
+ }
+
+ @Test(dataProvider = "useBothQueryEngines")
+ public void testRegexpLikeReturnsSameResults(boolean useMultiStageQueryEngine)
+ throws Exception {
+ setUseMultiStageQueryEngine(useMultiStageQueryEngine);
+ // REGEXP_LIKE goes through a separate code path in FilterPlanNode and must also drop the dict for raw forward.
+ assertScalarLongMatches(
+ String.format("SELECT COUNT(*) FROM %s WHERE REGEXP_LIKE(%s, 'value-1.*')", getTableName(), DICT_DIMENSION),
+ String.format("SELECT COUNT(*) FROM %s WHERE REGEXP_LIKE(%s, 'value-1.*')", getTableName(),
+ RAW_DICT_DIMENSION));
+ assertScalarLongMatches(
+ String.format("SELECT COUNT(*) FROM %s WHERE REGEXP_LIKE(%s, 'value-1.*')", getTableName(), DICT_DIMENSION),
+ String.format("SELECT COUNT(*) FROM %s WHERE REGEXP_LIKE(%s, 'value-1.*')", getTableName(),
+ RAW_DICT_INV_DIMENSION));
+ }
+
+ @Test(dataProvider = "useBothQueryEngines")
+ public void testInequalityFilterReturnsSameResults(boolean useMultiStageQueryEngine)
+ throws Exception {
+ setUseMultiStageQueryEngine(useMultiStageQueryEngine);
+ assertScalarLongMatches(
+ String.format("SELECT COUNT(*) FROM %s WHERE %s != 'value-3'", getTableName(), DICT_DIMENSION),
+ String.format("SELECT COUNT(*) FROM %s WHERE %s != 'value-3'", getTableName(), RAW_DICT_DIMENSION));
+ assertScalarLongMatches(
+ String.format("SELECT COUNT(*) FROM %s WHERE %s > 5 AND %s < 15", getTableName(), DICT_INT_DIMENSION,
+ DICT_INT_DIMENSION),
+ String.format("SELECT COUNT(*) FROM %s WHERE %s > 5 AND %s < 15", getTableName(), RAW_DICT_INT_DIMENSION,
+ RAW_DICT_INT_DIMENSION));
+ }
+
+ /**
+ * Regression test: a column with dict + inverted + RAW forward index used to crash on RANGE predicates.
+ * The planner kept the dictionary because an inverted index existed, but RANGE predicates don't use the
+ * inverted index — they fall through to the scan operator, which then called {@code applySV(rawValue)} on
+ * the dict-based range evaluator and threw {@code UnsupportedOperationException}. The fix drops the dict
+ * per-predicate-type so RANGE on RAW forward gets a raw-value evaluator.
+ */
+ @Test(dataProvider = "useBothQueryEngines")
+ public void testRangeOnRawDictInvertedColumnReturnsSameResults(boolean useMultiStageQueryEngine)
+ throws Exception {
+ setUseMultiStageQueryEngine(useMultiStageQueryEngine);
+ // Column has: dict + inverted + RAW forward; query is RANGE -> must use scan with raw-value evaluator.
+ assertScalarLongMatches(
+ String.format("SELECT COUNT(*) FROM %s WHERE %s > 5 AND %s < 15", getTableName(), DICT_INT_DIMENSION,
+ DICT_INT_DIMENSION),
+ String.format("SELECT COUNT(*) FROM %s WHERE %s > 5 AND %s < 15", getTableName(), RAW_DICT_INV_INT_DIMENSION,
+ RAW_DICT_INV_INT_DIMENSION));
+ assertScalarLongMatches(
+ String.format("SELECT COUNT(*) FROM %s WHERE %s >= 10", getTableName(), DICT_INT_DIMENSION),
+ String.format("SELECT COUNT(*) FROM %s WHERE %s >= 10", getTableName(), RAW_DICT_INV_INT_DIMENSION));
+ }
+
+ /**
+ * Mixed-predicate query on the same column: combines an EQ that uses the inverted index (dict path) with a
+ * RANGE that scans (raw-value path). Each predicate gets its own evaluator, so dict-drop must be decided
+ * per predicate, not per column.
+ */
+ @Test(dataProvider = "useBothQueryEngines")
+ public void testMixedInvertedEqAndRangeOnSameColumnReturnsSameResults(boolean useMultiStageQueryEngine)
+ throws Exception {
+ setUseMultiStageQueryEngine(useMultiStageQueryEngine);
+ assertScalarLongMatches(
+ String.format("SELECT COUNT(*) FROM %s WHERE %s = 7 OR (%s > 12 AND %s < 18)", getTableName(),
+ DICT_INT_DIMENSION, DICT_INT_DIMENSION, DICT_INT_DIMENSION),
+ String.format("SELECT COUNT(*) FROM %s WHERE %s = 7 OR (%s > 12 AND %s < 18)", getTableName(),
+ RAW_DICT_INV_INT_DIMENSION, RAW_DICT_INV_INT_DIMENSION, RAW_DICT_INV_INT_DIMENSION));
+ }
+
+ /**
+ * Comprehensive mixed-predicate matrix on the column with RAW forward + dictionary + inverted index.
+ *
+ * <p>Exercises every predicate type that takes the dict-consuming inverted-index path together with a RANGE
+ * predicate that drops the dict and scans raw values.
+ * {@code PredicateEvaluatorProvider#getDictionaryUsableForFiltering} decides per-predicate-type whether the
+ * dictionary is preserved or dropped, so each combination below routes one
+ * predicate through {@code InvertedIndexBasedFilterOperator} (with dict IDs) and another through
+ * {@code ScanBasedFilterOperator} (with raw values) on the same physical column.
+ *
+ * <p>Predicate-to-operator mapping for the int column ({@link #RAW_DICT_INV_INT_DIMENSION}) used here:
+ * <ul>
+ * <li>{@code EQ}, {@code NOT_EQ}, {@code IN}, {@code NOT_IN} → inverted index, dict preserved</li>
+ * <li>{@code RANGE} → scan, dict dropped, raw-value evaluator</li>
+ * </ul>
+ * Results are compared against the dictionary-encoded baseline column, which always uses the dict path.
+ * The deterministic dataset has values {@code 0..19}, each appearing 50 times in 1000 rows. Each sub-case
+ * also asserts an explicit non-zero expected count to defuse vacuous-pass risk if both sides silently
+ * returned 0 (e.g. due to a planner short-circuit on a contradictory predicate).
+ */
+ @Test(dataProvider = "useBothQueryEngines")
+ public void testAllPredicateTypesMixedWithRangeOnSameColumnReturnsSameResults(boolean useMultiStageQueryEngine)
+ throws Exception {
+ setUseMultiStageQueryEngine(useMultiStageQueryEngine);
+
+ // EQ (inverted, dict) AND RANGE (scan, raw): {7} satisfies both -> 50 rows.
+ assertScalarLongMatchesExpected(
+ String.format("SELECT COUNT(*) FROM %s WHERE %s = 7 AND %s > 5", getTableName(),
+ DICT_INT_DIMENSION, DICT_INT_DIMENSION),
+ String.format("SELECT COUNT(*) FROM %s WHERE %s = 7 AND %s > 5", getTableName(),
+ RAW_DICT_INV_INT_DIMENSION, RAW_DICT_INV_INT_DIMENSION),
+ 50);
+
+ // IN (inverted, dict) OR RANGE (scan, raw): {1,3,5} union {15,16,17,18} = 7 values * 50 = 350 rows.
+ assertScalarLongMatchesExpected(
+ String.format("SELECT COUNT(*) FROM %s WHERE %s IN (1, 3, 5) OR (%s >= 15 AND %s < 19)",
+ getTableName(), DICT_INT_DIMENSION, DICT_INT_DIMENSION, DICT_INT_DIMENSION),
+ String.format("SELECT COUNT(*) FROM %s WHERE %s IN (1, 3, 5) OR (%s >= 15 AND %s < 19)",
+ getTableName(), RAW_DICT_INV_INT_DIMENSION, RAW_DICT_INV_INT_DIMENSION, RAW_DICT_INV_INT_DIMENSION),
+ 350);
+
+ // NOT_EQ (inverted, dict) AND RANGE (scan, raw): not-{9} intersect {7..12} = {7,8,10,11,12} -> 5 * 50 = 250 rows.
+ assertScalarLongMatchesExpected(
+ String.format("SELECT COUNT(*) FROM %s WHERE %s != 9 AND %s BETWEEN 7 AND 12", getTableName(),
+ DICT_INT_DIMENSION, DICT_INT_DIMENSION),
+ String.format("SELECT COUNT(*) FROM %s WHERE %s != 9 AND %s BETWEEN 7 AND 12", getTableName(),
+ RAW_DICT_INV_INT_DIMENSION, RAW_DICT_INV_INT_DIMENSION),
+ 250);
+
+ // NOT_IN (inverted, dict) AND RANGE (scan, raw):
+ // not-{2,4,6} intersect {0..10} = {0,1,3,5,7,8,9,10} -> 8 * 50 = 400 rows.
+ assertScalarLongMatchesExpected(
+ String.format("SELECT COUNT(*) FROM %s WHERE %s NOT IN (2, 4, 6) AND %s <= 10", getTableName(),
+ DICT_INT_DIMENSION, DICT_INT_DIMENSION),
+ String.format("SELECT COUNT(*) FROM %s WHERE %s NOT IN (2, 4, 6) AND %s <= 10", getTableName(),
+ RAW_DICT_INV_INT_DIMENSION, RAW_DICT_INV_INT_DIMENSION),
+ 400);
+
+ // Three-way mix on the same column: EQ via inverted, NOT_IN via inverted, RANGE via scan.
+ // ({0} union not-{1,2,3}) intersect {5..19} = {5..19} -> 15 * 50 = 750 rows.
+ assertScalarLongMatchesExpected(
+ String.format("SELECT COUNT(*) FROM %s WHERE (%s = 0 OR %s NOT IN (1, 2, 3)) AND %s > 4",
+ getTableName(), DICT_INT_DIMENSION, DICT_INT_DIMENSION, DICT_INT_DIMENSION),
+ String.format("SELECT COUNT(*) FROM %s WHERE (%s = 0 OR %s NOT IN (1, 2, 3)) AND %s > 4",
+ getTableName(), RAW_DICT_INV_INT_DIMENSION, RAW_DICT_INV_INT_DIMENSION, RAW_DICT_INV_INT_DIMENSION),
+ 750);
+ }
+
+ /**
+ * Mixed-predicate query on the STRING column with RAW forward + dictionary + inverted index. Validates that
+ * REGEXP_LIKE (inverted-index, dict-id evaluator) and string EQ on the same column both return correct results
+ * and can be combined in a single query.
+ *
+ * <p>Note: there is no string analogue of RANGE that drops the dict in this column shape (no raw-value evaluator
+ * is exercised here), so this is a sanity check for the inverted-index path on strings — including the
+ * REGEXP_LIKE-on-RAW-forward path that the dict-drop logic enables.
+ */
+ @Test(dataProvider = "useBothQueryEngines")
+ public void testMixedRegexpAndEqOnStringRawDictInvertedColumnReturnsSameResults(boolean useMultiStageQueryEngine)
+ throws Exception {
+ setUseMultiStageQueryEngine(useMultiStageQueryEngine);
+ assertScalarLongMatches(
+ String.format("SELECT COUNT(*) FROM %s WHERE %s = 'value-1' OR REGEXP_LIKE(%s, 'value-1.*')",
+ getTableName(), DICT_DIMENSION, DICT_DIMENSION),
+ String.format("SELECT COUNT(*) FROM %s WHERE %s = 'value-1' OR REGEXP_LIKE(%s, 'value-1.*')",
+ getTableName(), RAW_DICT_INV_DIMENSION, RAW_DICT_INV_DIMENSION));
+ assertScalarLongMatches(
+ String.format("SELECT COUNT(*) FROM %s WHERE %s != 'value-3' AND REGEXP_LIKE(%s, 'value-1.*')",
+ getTableName(), DICT_DIMENSION, DICT_DIMENSION),
+ String.format("SELECT COUNT(*) FROM %s WHERE %s != 'value-3' AND REGEXP_LIKE(%s, 'value-1.*')",
+ getTableName(), RAW_DICT_INV_DIMENSION, RAW_DICT_INV_DIMENSION));
+ }
+
+ /**
+ * Cross-column mixed-predicate query: the predicates target different physical columns but each routes through
+ * a different operator within the same query plan. Validates that the planner does not entangle the per-column
+ * dict-drop decisions: the EQ on the inverted+dict+raw column takes the inverted-index path (dict preserved),
+ * while the two RANGE predicates on a raw+dict column with and without an inverted index both take the scan
+ * path (dict dropped). The dict-encoded baseline column is run as a separate query and supplies the expected
+ * value of 50 rows ({@code dim = 7} matching once per appearance).
+ */
+ @Test(dataProvider = "useBothQueryEngines")
+ public void testMixedPredicatesAcrossDifferentColumnShapesReturnsSameResults(boolean useMultiStageQueryEngine)
+ throws Exception {
+ setUseMultiStageQueryEngine(useMultiStageQueryEngine);
+ assertScalarLongMatchesExpected(
+ String.format("SELECT COUNT(*) FROM %s WHERE %s = 7 AND %s > 3 AND %s < 17",
+ getTableName(), DICT_INT_DIMENSION, DICT_INT_DIMENSION, DICT_INT_DIMENSION),
+ String.format("SELECT COUNT(*) FROM %s WHERE %s = 7 AND %s > 3 AND %s < 17",
+ getTableName(), RAW_DICT_INV_INT_DIMENSION, RAW_DICT_INV_INT_DIMENSION, RAW_DICT_INT_DIMENSION),
+ 50);
+ }
+
+ /**
+ * Column with dict + RAW forward + range index. {@code RangeIndexType#createIndexCreator} chooses the
+ * dict-id-based range index when a dictionary exists at segment build time, so the planner must keep the dict
+ * for RANGE predicates and the range-index path is used.
+ *
+ * <p>Note: this test covers the healthy case where dict and range-index were built together. The pre-existing
+ * {@link org.apache.pinot.segment.local.segment.index.readers.BitSlicedRangeIndexReader} assumes dict ↔ dict-based
+ * range; a segment built with raw-value range index that later gains a dictionary (e.g. via reload without range
+ * rebuild) is not a supported state and falls outside this fix's scope.
+ */
+ @Test(dataProvider = "useBothQueryEngines")
+ public void testRangeOnRawDictRangeColumnReturnsSameResults(boolean useMultiStageQueryEngine)
+ throws Exception {
+ setUseMultiStageQueryEngine(useMultiStageQueryEngine);
+ assertScalarLongMatches(
+ String.format("SELECT COUNT(*) FROM %s WHERE %s > 5 AND %s < 15", getTableName(), DICT_INT_DIMENSION,
+ DICT_INT_DIMENSION),
+ String.format("SELECT COUNT(*) FROM %s WHERE %s > 5 AND %s < 15", getTableName(),
+ RAW_DICT_RANGE_INT_DIMENSION, RAW_DICT_RANGE_INT_DIMENSION));
+ assertScalarLongMatches(
+ String.format("SELECT COUNT(*) FROM %s WHERE %s >= 10", getTableName(), DICT_INT_DIMENSION),
+ String.format("SELECT COUNT(*) FROM %s WHERE %s >= 10", getTableName(), RAW_DICT_RANGE_INT_DIMENSION));
+ }
+
+ /**
+ * Equality on a range-indexed column. Whether range index serves EQ depends on
+ * {@link org.apache.pinot.segment.spi.index.reader.RangeIndexReader#isExact()}; the planner reasons over that
+ * directly. Either path (range-index or scan) must produce results identical to the baseline.
+ */
+ @Test(dataProvider = "useBothQueryEngines")
+ public void testEqualityOnRawDictRangeColumnReturnsSameResults(boolean useMultiStageQueryEngine)
+ throws Exception {
+ setUseMultiStageQueryEngine(useMultiStageQueryEngine);
+ for (int i = 0; i < UNIQUE_DIMENSION_VALUES; i++) {
+ assertScalarLongMatches(
+ String.format("SELECT COUNT(*) FROM %s WHERE %s = %d", getTableName(), DICT_INT_DIMENSION, i),
+ String.format("SELECT COUNT(*) FROM %s WHERE %s = %d", getTableName(), RAW_DICT_RANGE_INT_DIMENSION, i));
+ }
+ }
+
+ /**
+ * Range index disabled via {@code skipIndexes}. The planner must drop the dictionary so the scan path uses the
+ * raw-value evaluator instead of the dict-based one.
+ */
+ @Test
+ public void testRangeOnRawDictRangeColumnWithSkipRangeIndex()
+ throws Exception {
+ setUseMultiStageQueryEngine(false);
+ String prefix = "SET skipIndexes='" + RAW_DICT_RANGE_INT_DIMENSION + "=range'; ";
+ assertScalarLongMatches(
+ prefix + String.format("SELECT COUNT(*) FROM %s WHERE %s > 5 AND %s < 15", getTableName(), DICT_INT_DIMENSION,
+ DICT_INT_DIMENSION),
+ prefix + String.format("SELECT COUNT(*) FROM %s WHERE %s > 5 AND %s < 15", getTableName(),
+ RAW_DICT_RANGE_INT_DIMENSION, RAW_DICT_RANGE_INT_DIMENSION));
+ }
+
+ @Test(dataProvider = "useBothQueryEngines")
+ public void testInPredicateReturnsSameResults(boolean useMultiStageQueryEngine)
+ throws Exception {
+ setUseMultiStageQueryEngine(useMultiStageQueryEngine);
+ assertScalarLongMatches(
+ String.format("SELECT COUNT(*) FROM %s WHERE %s IN ('value-1','value-7','value-13')", getTableName(),
+ DICT_DIMENSION),
+ String.format("SELECT COUNT(*) FROM %s WHERE %s IN ('value-1','value-7','value-13')", getTableName(),
+ RAW_DICT_DIMENSION));
+ assertScalarLongMatches(
+ String.format("SELECT COUNT(*) FROM %s WHERE %s IN (2, 4, 6, 8)", getTableName(), DICT_INT_DIMENSION),
+ String.format("SELECT COUNT(*) FROM %s WHERE %s IN (2, 4, 6, 8)", getTableName(), RAW_DICT_INT_DIMENSION));
+ }
+
+ @Test(dataProvider = "useBothQueryEngines")
+ public void testGroupByReturnsSameResults(boolean useMultiStageQueryEngine)
+ throws Exception {
+ setUseMultiStageQueryEngine(useMultiStageQueryEngine);
+ JsonNode dictRows = postQuery(
+ String.format("SELECT %s, COUNT(*) FROM %s GROUP BY %s ORDER BY %s", DICT_DIMENSION, getTableName(),
+ DICT_DIMENSION, DICT_DIMENSION)).get("resultTable").get("rows");
+ JsonNode rawRows = postQuery(
+ String.format("SELECT %s, COUNT(*) FROM %s GROUP BY %s ORDER BY %s", RAW_DICT_DIMENSION, getTableName(),
+ RAW_DICT_DIMENSION, RAW_DICT_DIMENSION)).get("resultTable").get("rows");
+ assertEquals(rawRows, dictRows, "GROUP BY rows must match between dictionary-only and raw+dictionary columns");
+ }
+
+ @Test(dataProvider = "useBothQueryEngines")
+ public void testDistinctReturnsSameResults(boolean useMultiStageQueryEngine)
+ throws Exception {
+ setUseMultiStageQueryEngine(useMultiStageQueryEngine);
+ JsonNode dictRows = postQuery(
+ String.format("SELECT DISTINCT %s FROM %s ORDER BY %s", DICT_DIMENSION, getTableName(), DICT_DIMENSION))
+ .get("resultTable").get("rows");
+ JsonNode rawRows = postQuery(
+ String.format("SELECT DISTINCT %s FROM %s ORDER BY %s", RAW_DICT_DIMENSION, getTableName(),
+ RAW_DICT_DIMENSION)).get("resultTable").get("rows");
+ assertEquals(rawRows, dictRows, "DISTINCT rows must match between dictionary-only and raw+dictionary columns");
+ }
+
+ @Test(dataProvider = "useBothQueryEngines")
+ public void testAggregationWithGroupByOnRawDictColumnReturnsSameResults(boolean useMultiStageQueryEngine)
+ throws Exception {
+ setUseMultiStageQueryEngine(useMultiStageQueryEngine);
+ JsonNode dictRows = postQuery(
+ String.format("SELECT %s, SUM(%s), MIN(%s), MAX(%s) FROM %s GROUP BY %s ORDER BY %s", DICT_DIMENSION,
+ METRIC_COLUMN, METRIC_COLUMN, METRIC_COLUMN, getTableName(), DICT_DIMENSION, DICT_DIMENSION))
+ .get("resultTable").get("rows");
+ JsonNode rawRows = postQuery(
+ String.format("SELECT %s, SUM(%s), MIN(%s), MAX(%s) FROM %s GROUP BY %s ORDER BY %s", RAW_DICT_DIMENSION,
+ METRIC_COLUMN, METRIC_COLUMN, METRIC_COLUMN, getTableName(), RAW_DICT_DIMENSION, RAW_DICT_DIMENSION))
+ .get("resultTable").get("rows");
+ assertEquals(rawRows, dictRows,
+ "Aggregations grouped by raw+dictionary column must match the dictionary-only baseline");
+ }
+
+ @Test(dataProvider = "useBothQueryEngines")
+ public void testProjectionReturnsSameResults(boolean useMultiStageQueryEngine)
+ throws Exception {
+ setUseMultiStageQueryEngine(useMultiStageQueryEngine);
+ JsonNode dictRows = postQuery(
+ String.format("SELECT %s FROM %s ORDER BY %s, %s LIMIT 100", DICT_DIMENSION, getTableName(),
+ TIMESTAMP_FIELD_NAME, DICT_DIMENSION)).get("resultTable").get("rows");
+ JsonNode rawRows = postQuery(
+ String.format("SELECT %s FROM %s ORDER BY %s, %s LIMIT 100", RAW_DICT_DIMENSION, getTableName(),
+ TIMESTAMP_FIELD_NAME, RAW_DICT_DIMENSION)).get("resultTable").get("rows");
+ assertEquals(rawRows.size(), dictRows.size(), "Projection row counts must match");
+ for (int i = 0; i < dictRows.size(); i++) {
+ assertEquals(rawRows.get(i).get(0).asText(), dictRows.get(i).get(0).asText(),
+ "Projected value at row " + i + " must match between dictionary-only and raw+dictionary columns");
+ }
+ }
+
+ private void assertScalarLongMatches(String dictQuery, String rawQuery)
+ throws Exception {
+ long dictResult = scalarLong(dictQuery);
+ long rawResult = scalarLong(rawQuery);
+ assertEquals(rawResult, dictResult,
+ "Result mismatch.\n dict query: " + dictQuery + " => " + dictResult + "\n raw query: " + rawQuery
+ + " => " + rawResult);
+ }
+
+ /// Like {@link #assertScalarLongMatches} but additionally pins the expected count, so the assertion does not
+ /// pass vacuously if both sides silently return 0 (e.g. due to a planner short-circuit on a
+ /// contradictory predicate).
+ private void assertScalarLongMatchesExpected(String dictQuery, String rawQuery, long expected)
+ throws Exception {
+ long dictResult = scalarLong(dictQuery);
+ long rawResult = scalarLong(rawQuery);
+ assertEquals(dictResult, expected,
+ "Dict baseline does not match expected count.\n dict query: " + dictQuery + " => " + dictResult
+ + " (expected " + expected + ")");
+ assertEquals(rawResult, expected,
+ "Raw result does not match expected count.\n raw query: " + rawQuery + " => " + rawResult
+ + " (expected " + expected + ")");
+ }
+
+ private long scalarLong(String query)
+ throws Exception {
+ JsonNode response = postQuery(query);
+ JsonNode resultTable = response.get("resultTable");
+ if (resultTable == null) {
+ throw new AssertionError("Query failed: " + query + "\nResponse: " + response.toString());
+ }
+ return resultTable.get("rows").get(0).get(0).asLong();
+ }
+
+ private static JsonNode findColumnMetadata(JsonNode segmentsMetadata, String column) {
+ for (JsonNode segmentMetadata : segmentsMetadata) {
+ JsonNode columnsMetadata = segmentMetadata.get("columns");
+ if (columnsMetadata == null) {
+ continue;
+ }
+ for (JsonNode columnMetadata : columnsMetadata) {
+ if (column.equals(columnMetadata.get("columnName").asText())) {
+ return columnMetadata;
+ }
+ }
+ }
+ return null;
+ }
+}
diff --git a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/legacy/LegacyRawValueInvertedIndexMigrationIntegrationTest.java b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/legacy/LegacyRawValueInvertedIndexMigrationIntegrationTest.java
new file mode 100644
index 00000000000..4f73128a4df
--- /dev/null
+++ b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/legacy/LegacyRawValueInvertedIndexMigrationIntegrationTest.java
@@ -0,0 +1,311 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.integration.tests.legacy;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import java.io.File;
+import java.io.InputStream;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.integration.tests.BaseClusterIntegrationTest;
+import org.apache.pinot.segment.local.segment.index.loader.invertedindex.LegacyRawValueInvertedIndexCleanup;
+import org.apache.pinot.segment.local.segment.store.SegmentLocalFSDirectory;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.creator.SegmentVersion;
+import org.apache.pinot.segment.spi.index.StandardIndexes;
+import org.apache.pinot.segment.spi.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.apache.pinot.spi.config.table.FieldConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.apache.pinot.spi.utils.ReadMode;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.apache.pinot.util.TestUtils;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.DataProvider;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertNotNull;
+import static org.testng.Assert.assertTrue;
+
+
+/// Backward-compatibility integration test for the legacy raw-value inverted index migration path.
+///
+/// **Two fixtures** are loaded into two separate tables, exercising both on-disk segment formats:
+///
+/// - `legacy/legacyRawInverted_OFFLINE_0.tar.gz` (v1) — each index is its own file (`category.bitmap.inv`,
+/// `category.sv.raw.fwd`). Generated by an OLDER Pinot codebase that still contains the now-deleted
+/// `RawValueBitmapInvertedIndexCreator` (commit `669ef058e0`, see header for the one-line patch needed to
+/// permit STRING).
+/// - `legacy/legacyRawInverted_v3_OFFLINE_0.tar.gz` (v3) — all indexes packed into a single `columns.psf` with
+/// an `index_map` that declares offsets and sizes per index. Produced by running the v1 fixture through
+/// `SegmentV1V2ToV3FormatConverter` from Pinot 1.5.0; the legacy embedded-dictionary bytes are byte-identical
+/// to the v1 fixture (the `SingleFileIndexDirectory` magic-marker prefix is added by the writer and stripped by
+/// `getIndexFor` on read, so the bytes the format detector sees are the same as in v1).
+///
+/// **What the new code must do on segment load** to remain backward-compatible (both fixtures):
+///
+/// 1. `SegmentPreProcessor.removeLegacyRawValueInvertedIndexes` (pre-pass before all index handlers) detects
+/// the legacy 44-byte big-endian header and removes the index entry — for v1 it deletes the file, for v3 it
+/// removes the `index_map` entry (the bytes inside `columns.psf` are no longer referenced).
+/// 2. `ForwardIndexHandler.ENABLE_DICTIONARY` op materializes a standalone dictionary on the column (driven by
+/// the cluster's table config still requesting an inverted index, which now requires a dictionary).
+/// 3. `InvertedIndexHandler` builds a standard dict-id-based bitmap inverted index from the new dictionary.
+///
+/// **Verification:**
+///
+/// - **Black-box query checks** — COUNT(*), EQ, IN, NOT_EQ — would crash or report wildly wrong counts if the
+/// modern `BitmapInvertedIndexReader` mis-interpreted the legacy header.
+/// - **White-box on-disk checks** ([#assertSegmentMigratedOnDisk]) inspect the server's local segment
+/// directory after load: assert that the column metadata flips to `hasDictionary=true`, that the dictionary
+/// index is present, that the inverted index is no longer in the legacy embedded-dict format, and that the
+/// forward index is preserved (raw forward stays).
+public class LegacyRawValueInvertedIndexMigrationIntegrationTest extends BaseClusterIntegrationTest {
+ private static final String CATEGORY_COLUMN = "category";
+ /// Mirrors the deterministic values written by the fixture generator. 6-element cycle:
+ /// alpha 3× per cycle, beta/gamma/delta 1× each.
+ private static final String[] DISTINCT_VALUES = {"alpha", "beta", "gamma", "delta"};
+ private static final int NUM_DOCS = 600;
+ private static final int CYCLE_LENGTH = 6;
+ private static final String SEGMENT_NAME = "legacyRawInverted_OFFLINE_0";
+
+ private static final String V1_TABLE = "legacyRawInvertedV1";
+ private static final String V3_TABLE = "legacyRawInvertedV3";
+ private static final String V1_FIXTURE_RESOURCE = "legacy/legacyRawInverted_OFFLINE_0.tar.gz";
+ private static final String V3_FIXTURE_RESOURCE = "legacy/legacyRawInverted_v3_OFFLINE_0.tar.gz";
+
+ @Override
+ protected long getCountStarResult() {
+ return NUM_DOCS;
+ }
+
+ @BeforeClass
+ public void setUp()
+ throws Exception {
+ TestUtils.ensureDirectoriesExistAndEmpty(_tempDir, _segmentDir, _tarDir);
+
+ startZk();
+ startController();
+ startBroker();
+ startServer();
+
+ setUpFixtureTable(V1_TABLE, V1_FIXTURE_RESOURCE);
+ setUpFixtureTable(V3_TABLE, V3_FIXTURE_RESOURCE);
+ }
+
+ /// Cluster-side schema and table config request `noDictionaryColumns` overlapping `invertedIndexColumns` on
+ /// the same column — the legacy "raw forward + inverted" config that, in the old code, produced the
+ /// embedded-dict format. The new code must auto-create the missing dictionary on segment load.
+ private void setUpFixtureTable(String tableName, String fixtureResource)
+ throws Exception {
+ Schema schema = new Schema.SchemaBuilder()
+ .setSchemaName(tableName)
+ .addSingleValueDimension(CATEGORY_COLUMN, FieldSpec.DataType.STRING)
+ .build();
+ addSchema(schema);
+
+ // FieldConfig requires an explicit `dictionary: {}` block per the new
+ // `validateExplicitDictionaryForRawForwardIndex` rule: a column with `noDictionaryColumns +
+ // invertedIndexColumns` overlap must opt in to the auto-created shared dictionary.
+ ObjectNode indexes = JsonUtils.newObjectNode();
+ indexes.set("dictionary", JsonUtils.newObjectNode());
+ FieldConfig categoryFieldConfig = new FieldConfig.Builder(CATEGORY_COLUMN)
+ .withEncodingType(FieldConfig.EncodingType.RAW)
+ .withIndexes(indexes)
+ .build();
+ TableConfig tableConfig = new TableConfigBuilder(TableType.OFFLINE)
+ .setTableName(tableName)
+ .setInvertedIndexColumns(List.of(CATEGORY_COLUMN))
+ .setFieldConfigList(List.of(categoryFieldConfig))
+ .build();
+ addTableConfig(tableConfig);
+
+ File perTableTarDir = new File(_tarDir, tableName);
+ FileUtils.forceMkdir(perTableTarDir);
+ File stagedTar = new File(perTableTarDir, SEGMENT_NAME + ".tar.gz");
+ try (InputStream legacyResource =
+ getClass().getClassLoader().getResourceAsStream(fixtureResource)) {
+ assertNotNull(legacyResource,
+ "Legacy fixture resource not found on test classpath: " + fixtureResource);
+ FileUtils.copyInputStreamToFile(legacyResource, stagedTar);
+ }
+
+ uploadSegments(tableName, perTableTarDir);
+ waitForDocsLoaded(tableName, 60_000L);
+ }
+
+ private void waitForDocsLoaded(String tableName, long timeoutMs)
+ throws Exception {
+ TestUtils.waitForCondition(aVoid -> {
+ try {
+ return getCurrentCountStarResult(tableName) == NUM_DOCS;
+ } catch (Exception e) {
+ return null;
+ }
+ }, 100L, timeoutMs, "Failed to load all documents into table: " + tableName);
+ }
+
+ @AfterClass
+ public void tearDown()
+ throws Exception {
+ dropOfflineTable(V1_TABLE);
+ dropOfflineTable(V3_TABLE);
+ stopServer();
+ stopBroker();
+ stopController();
+ stopZk();
+ FileUtils.deleteDirectory(_tempDir);
+ }
+
+ @DataProvider(name = "fixtures")
+ public Object[][] fixtures() {
+ return new Object[][] {
+ {V1_TABLE, "v1"},
+ {V3_TABLE, "v3"}
+ };
+ }
+
+ /// COUNT(*) doesn't itself depend on the inverted index, but if the legacy file weren't dropped during
+ /// preprocess the segment would either fail to load or report a corrupted doc count.
+ @Test(dataProvider = "fixtures")
+ public void testCountStarMatchesIngestedDocs(String tableName, String formatLabel)
+ throws Exception {
+ JsonNode result = postQuery("SELECT COUNT(*) FROM " + tableName);
+ assertEquals(result.get("resultTable").get("rows").get(0).get(0).asLong(), NUM_DOCS,
+ formatLabel + ": doc count must equal the ingested row count after migration");
+ }
+
+ /// Each EQ predicate would, in pre-migration state, hit the legacy embedded-dict reader. After migration the
+ /// new dict-id-based `BitmapInvertedIndexReader` serves it. The two should produce the same results — verified
+ /// by checking against the inline expected counts.
+ @Test(dataProvider = "fixtures")
+ public void testEqualityCountsAfterMigration(String tableName, String formatLabel)
+ throws Exception {
+ Map<String, Long> expected = expectedCountsByValue();
+ for (Map.Entry<String, Long> e : expected.entrySet()) {
+ String value = e.getKey();
+ long expectedCount = e.getValue();
+ JsonNode result = postQuery(
+ "SELECT COUNT(*) FROM " + tableName + " WHERE " + CATEGORY_COLUMN + " = '" + value + "'");
+ long actual = result.get("resultTable").get("rows").get(0).get(0).asLong();
+ assertEquals(actual, expectedCount,
+ formatLabel + ": EQ count mismatch for '" + value + "' — migration may have failed");
+ }
+ }
+
+ @Test(dataProvider = "fixtures")
+ public void testInPredicateAfterMigration(String tableName, String formatLabel)
+ throws Exception {
+ Map<String, Long> expected = expectedCountsByValue();
+ long expectedAlphaBeta = expected.get(DISTINCT_VALUES[0]) + expected.get(DISTINCT_VALUES[1]);
+ JsonNode result = postQuery("SELECT COUNT(*) FROM " + tableName + " WHERE " + CATEGORY_COLUMN
+ + " IN ('" + DISTINCT_VALUES[0] + "', '" + DISTINCT_VALUES[1] + "')");
+ long actual = result.get("resultTable").get("rows").get(0).get(0).asLong();
+ assertEquals(actual, expectedAlphaBeta,
+ formatLabel + ": IN-predicate count mismatch — migration may have failed");
+ }
+
+ @Test(dataProvider = "fixtures")
+ public void testNotEqualPredicateAfterMigration(String tableName, String formatLabel)
+ throws Exception {
+ long expectedNotAlpha = NUM_DOCS - expectedCountsByValue().get(DISTINCT_VALUES[0]);
+ JsonNode result = postQuery("SELECT COUNT(*) FROM " + tableName + " WHERE " + CATEGORY_COLUMN
+ + " != '" + DISTINCT_VALUES[0] + "'");
+ long actual = result.get("resultTable").get("rows").get(0).get(0).asLong();
+ assertEquals(actual, expectedNotAlpha,
+ formatLabel + ": NOT_EQ count mismatch — migration may have failed");
+ }
+
+ /// White-box check: open the server's local segment directory, parse `metadata.properties` + `index_map` (or
+ /// equivalent v1 file layout), and assert that the migration mutated the on-disk state correctly:
+ ///
+ /// - `column.category.hasDictionary = true` — `ForwardIndexHandler.ENABLE_DICTIONARY` flipped this.
+ /// - The dictionary index is now registered with the segment directory.
+ /// - The inverted index is registered, and its bytes are NOT in the legacy embedded-dictionary format.
+ /// - The forward index is still raw (we kept it; the shared dictionary is standalone).
+ @Test(dataProvider = "fixtures")
+ public void testSegmentMigratedOnDisk(String tableName, String formatLabel)
+ throws Exception {
+ File segmentDir = locateServerSegmentDir(tableName);
+ assertTrue(segmentDir.isDirectory(),
+ formatLabel + ": expected server-side segment dir at " + segmentDir);
+
+ SegmentMetadataImpl segmentMetadata = new SegmentMetadataImpl(segmentDir);
+ ColumnMetadata categoryMeta = segmentMetadata.getColumnMetadataFor(CATEGORY_COLUMN);
+ assertNotNull(categoryMeta, formatLabel + ": missing column metadata for " + CATEGORY_COLUMN);
+ assertTrue(categoryMeta.hasDictionary(),
+ formatLabel + ": metadata must show hasDictionary=true after migration (was=false in fixture)");
+ assertEquals(categoryMeta.getCardinality(), DISTINCT_VALUES.length,
+ formatLabel + ": cardinality should match the fixture's distinct-value count");
+
+ // Server normalizes all loaded segments to v3 (per the test cluster's segment.format.version=v3 setting),
+ // so post-load we expect v3 regardless of fixture's source format.
+ assertEquals(segmentMetadata.getVersion(), SegmentVersion.v3,
+ formatLabel + ": server should normalize the loaded segment to v3");
+ try (SegmentDirectory directory = new SegmentLocalFSDirectory(segmentDir, ReadMode.mmap);
+ SegmentDirectory.Reader reader = directory.createReader()) {
+ assertTrue(reader.hasIndexFor(CATEGORY_COLUMN, StandardIndexes.dictionary()),
+ formatLabel + ": dictionary index must be present after migration");
+ assertTrue(reader.hasIndexFor(CATEGORY_COLUMN, StandardIndexes.inverted()),
+ formatLabel + ": inverted index must be present after migration");
+ assertTrue(reader.hasIndexFor(CATEGORY_COLUMN, StandardIndexes.forward()),
+ formatLabel + ": forward index must remain after migration (raw forward is preserved)");
+
+ try (PinotDataBuffer invBuffer =
+ reader.getIndexFor(CATEGORY_COLUMN, StandardIndexes.inverted())) {
+ assertFalse(LegacyRawValueInvertedIndexCleanup.isLegacyRawValueInvertedIndexFormat(invBuffer, categoryMeta),
+ formatLabel + ": rebuilt inverted index must NOT be in legacy embedded-dict format");
+ }
+ }
+ }
+
+ /// Resolve the on-disk segment directory under the in-process server's `INSTANCE_DATA_DIR`. Layout is
+ /// `<dataDir>/<tableNameWithType>/<segmentName>` for offline tables.
+ private File locateServerSegmentDir(String tableName) {
+ String tableNameWithType = TableNameBuilder.OFFLINE.tableNameWithType(tableName);
+ return new File(
+ TEMP_SERVER_DIR + File.separator + "dataDir-0" + File.separator + tableNameWithType,
+ SEGMENT_NAME);
+ }
+
+ /// Per-value expected multiplicity from the fixture's deterministic 6-element cycle:
+ /// alpha appears 3× per cycle, beta/gamma/delta each 1×.
+ private static Map<String, Long> expectedCountsByValue() {
+ long cycles = NUM_DOCS / CYCLE_LENGTH;
+ Map<String, Long> counts = new HashMap<>();
+ counts.put(DISTINCT_VALUES[0], cycles * 3);
+ counts.put(DISTINCT_VALUES[1], cycles);
+ counts.put(DISTINCT_VALUES[2], cycles);
+ counts.put(DISTINCT_VALUES[3], cycles);
+ long total = counts.values().stream().mapToLong(Long::longValue).sum();
+ assertEquals(total, NUM_DOCS, "Inline expected counts must sum to NUM_DOCS");
+ return counts;
+ }
+}
diff --git a/pinot-integration-tests/src/test/resources/legacy/legacyRawInverted_OFFLINE_0.tar.gz b/pinot-integration-tests/src/test/resources/legacy/legacyRawInverted_OFFLINE_0.tar.gz
new file mode 100644
index 00000000000..51d046a47b2
Binary files /dev/null and b/pinot-integration-tests/src/test/resources/legacy/legacyRawInverted_OFFLINE_0.tar.gz differ
diff --git a/pinot-integration-tests/src/test/resources/legacy/legacyRawInverted_v3_OFFLINE_0.tar.gz b/pinot-integration-tests/src/test/resources/legacy/legacyRawInverted_v3_OFFLINE_0.tar.gz
new file mode 100644
index 00000000000..b2abb40c66d
Binary files /dev/null and b/pinot-integration-tests/src/test/resources/legacy/legacyRawInverted_v3_OFFLINE_0.tar.gz differ
diff --git a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/BaseSegmentCreator.java b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/BaseSegmentCreator.java
index 0a4e712ee51..006e85612c8 100644
--- a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/BaseSegmentCreator.java
+++ b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/BaseSegmentCreator.java
@@ -370,6 +370,19 @@ public abstract class BaseSegmentCreator implements SegmentCreator {
String column = spec.getName();
FieldIndexConfigs fieldIndexConfigs = config.getIndexConfigsByColName().get(column);
+ // If any enabled index requires a dictionary for this column, create one regardless of the user's
+ // explicit dictionary setting or the optimizeDictionary heuristics — the alternative is segment
+ // creation failure (e.g., inverted index can't be built without a dict) or silent index loss
+ // (FST/IFST expect a dictionary).
+ boolean dictionaryRequired = DictionaryIndexConfig.requiresDictionary(spec, fieldIndexConfigs);
+ if (dictionaryRequired) {
+ if (fieldIndexConfigs.getConfig(StandardIndexes.dictionary()).isDisabled()) {
+ LOGGER.warn("Column: {} has dictionary disabled but required by indexes: {}; creating dictionary anyway",
+ column, DictionaryIndexConfig.getIndexTypesWithDictionaryRequired(spec, fieldIndexConfigs));
+ }
+ return true;
+ }
+
if (fieldIndexConfigs.getConfig(StandardIndexes.dictionary()).isDisabled()) {
return false;
}
diff --git a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/dictionary/DictionaryIndexType.java b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/dictionary/DictionaryIndexType.java
index 84b904fd7f6..930cfd8fc85 100644
--- a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/dictionary/DictionaryIndexType.java
+++ b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/dictionary/DictionaryIndexType.java
@@ -19,6 +19,7 @@
package org.apache.pinot.segment.local.segment.index.dictionary;
+import com.fasterxml.jackson.databind.JsonNode;
import com.google.common.base.Preconditions;
import com.google.common.collect.Lists;
import com.google.common.collect.Maps;
@@ -27,7 +28,9 @@ import java.io.IOException;
import java.io.UncheckedIOException;
import java.util.ArrayList;
import java.util.Collections;
+import java.util.HashMap;
import java.util.HashSet;
+import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.Set;
@@ -61,6 +64,7 @@ import org.apache.pinot.segment.spi.index.IndexConfigDeserializer;
import org.apache.pinot.segment.spi.index.IndexHandler;
import org.apache.pinot.segment.spi.index.IndexReaderConstraintException;
import org.apache.pinot.segment.spi.index.IndexReaderFactory;
+import org.apache.pinot.segment.spi.index.IndexService;
import org.apache.pinot.segment.spi.index.IndexType;
import org.apache.pinot.segment.spi.index.IndexUtil;
import org.apache.pinot.segment.spi.index.StandardIndexes;
@@ -127,12 +131,27 @@ public class DictionaryIndexType
ColumnConfigDeserializer<DictionaryIndexConfig> fromNoDictionaryColumns =
IndexConfigDeserializer.fromCollection(tableConfig -> tableConfig.getIndexingConfig().getNoDictionaryColumns(),
(accum, column) -> accum.put(column, DictionaryIndexConfig.DISABLED));
- ColumnConfigDeserializer<DictionaryIndexConfig> fromFieldConfigs =
- IndexConfigDeserializer.fromCollection(TableConfig::getFieldConfigList, (accum, fieldConfig) -> {
- if (fieldConfig.getEncodingType() == FieldConfig.EncodingType.RAW) {
- accum.put(fieldConfig.getName(), DictionaryIndexConfig.DISABLED);
- }
- });
+ ColumnConfigDeserializer<DictionaryIndexConfig> fromFieldConfigs = (tableConfig, schema) -> {
+ List<FieldConfig> fieldConfigList = tableConfig.getFieldConfigList();
+ if (fieldConfigList == null) {
+ return Collections.emptyMap();
+ }
+ Map<String, DictionaryIndexConfig> result = new HashMap<>();
+ for (FieldConfig fieldConfig : fieldConfigList) {
+ // encoding=RAW disables the dictionary unless an explicit dictionary config is given OR an
+ // index that requires a dictionary is enabled in the FieldConfig. In the latter case let the
+ // dictionary fall through to its default-enabled state so the runtime can build it; the
+ // auto-creation paths in BaseSegmentCreator/ForwardIndexHandler will produce a shared-dict +
+ // RAW forward index.
+ FieldSpec fieldSpec = schema == null ? null : schema.getFieldSpecFor(fieldConfig.getName());
+ if (fieldConfig.getEncodingType() == FieldConfig.EncodingType.RAW
+ && !hasExplicitDictionaryConfig(fieldConfig)
+ && (fieldSpec == null || !hasIndexRequiringDictionary(fieldConfig, fieldSpec))) {
+ result.put(fieldConfig.getName(), DictionaryIndexConfig.DISABLED);
+ }
+ }
+ return result;
+ };
ColumnConfigDeserializer<DictionaryIndexConfig> fromIndexingConfig = (tableConfig, schema) -> {
IndexingConfig indexingConfig = tableConfig.getIndexingConfig();
Set<String> onHeapDictionaryColumns = indexingConfig.getOnHeapDictionaryColumns() != null ? new HashSet<>(
@@ -154,6 +173,86 @@ public class DictionaryIndexType
.withFallbackAlternative(fromIndexingConfig);
}
+ private static boolean hasExplicitDictionaryConfig(FieldConfig fieldConfig) {
+ JsonNode indexes = fieldConfig.getIndexes();
+ return indexes != null && indexes.isObject() && indexes.has(StandardIndexes.DICTIONARY_ID);
+ }
+
+ /// True if any index enabled in the FieldConfig declares
+ /// {@link IndexType#requiresDictionary(FieldSpec, IndexConfig)} for the column. Iterates over every
+ /// registered IndexType, asks each whether it's enabled in this raw FieldConfig (legacy `indexTypes`
+ /// list or the `indexes` map without `disabled:true`), and consults `requiresDictionary` against the
+ /// IndexType's default config. (Built-in dict-requiring indexes — FST/IFST/INVERTED — return true
+ /// unconditionally, so the default config is sufficient.)
+ @SuppressWarnings({"rawtypes", "unchecked"})
+ private static boolean hasIndexRequiringDictionary(FieldConfig fieldConfig, FieldSpec fieldSpec) {
+ Set<String> enabledIds = enabledIndexIds(fieldConfig);
+ for (IndexType indexType : IndexService.getInstance().getAllIndexes()) {
+ if (StandardIndexes.DICTIONARY_ID.equals(indexType.getId()) || !enabledIds.contains(indexType.getId())) {
+ continue;
+ }
+ if (indexType.requiresDictionary(fieldSpec, indexType.getDefaultConfig())) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /// Collects the set of `StandardIndexes` IDs that are explicitly enabled on the FieldConfig — both
+ /// from the legacy `indexTypes` enum list and from the `indexes` JsonNode map (skipping entries marked
+ /// `"disabled": true`).
+ private static Set<String> enabledIndexIds(FieldConfig fieldConfig) {
+ Set<String> ids = new HashSet<>();
+ for (FieldConfig.IndexType legacy : fieldConfig.getIndexTypes()) {
+ String id = legacyIndexTypeId(legacy);
+ if (id != null) {
+ ids.add(id);
+ }
+ }
+ JsonNode indexes = fieldConfig.getIndexes();
+ if (indexes != null && indexes.isObject()) {
+ Iterator<String> names = indexes.fieldNames();
+ while (names.hasNext()) {
+ String name = names.next();
+ JsonNode entry = indexes.get(name);
+ if (entry != null && entry.isObject()) {
+ JsonNode disabled = entry.get("disabled");
+ if (disabled != null && disabled.isBoolean() && disabled.asBoolean()) {
+ continue;
+ }
+ }
+ ids.add(name);
+ }
+ }
+ return ids;
+ }
+
+ @Nullable
+ private static String legacyIndexTypeId(FieldConfig.IndexType type) {
+ switch (type) {
+ case INVERTED:
+ return StandardIndexes.INVERTED_ID;
+ case FST:
+ return StandardIndexes.FST_ID;
+ case IFST:
+ return StandardIndexes.IFST_ID;
+ case TEXT:
+ return StandardIndexes.TEXT_ID;
+ case H3:
+ return StandardIndexes.H3_ID;
+ case JSON:
+ return StandardIndexes.JSON_ID;
+ case RANGE:
+ return StandardIndexes.RANGE_ID;
+ case VECTOR:
+ return StandardIndexes.VECTOR_ID;
+ case SORTED:
+ case TIMESTAMP:
+ default:
+ return null;
+ }
+ }
+
@Override
public SegmentDictionaryCreator createIndexCreator(IndexCreationContext context, DictionaryIndexConfig indexConfig) {
boolean useVarLengthDictionary = shouldUseVarLengthDictionary(context, indexConfig);
diff --git a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/inverted/InvertedIndexType.java b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/inverted/InvertedIndexType.java
index 27291593771..94eac9c9d3f 100644
--- a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/inverted/InvertedIndexType.java
+++ b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/inverted/InvertedIndexType.java
@@ -152,10 +152,8 @@ public class InvertedIndexType
@Override
public boolean requiresDictionary(FieldSpec fieldSpec, IndexConfig indexConfig) {
- // Inverted index posting lists are keyed by dictionary IDs. Note: a separate raw-value bitmap inverted index
- // currently exists for SV no-dictionary columns (RawValueBitmapInvertedIndexCreator); under the upcoming
- // shared-dictionary contract (apache/pinot#17269) that path is removed and any enabled inverted index requires
- // a dictionary, so we declare the dependency unconditionally here.
+ // Inverted index posting lists are keyed by dictionary IDs; an enabled inverted index always requires a
+ // dictionary.
return true;
}
diff --git a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/BaseIndexHandler.java b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/BaseIndexHandler.java
index 5d4358e2a74..98b98acf5dc 100644
--- a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/BaseIndexHandler.java
+++ b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/BaseIndexHandler.java
@@ -31,6 +31,7 @@ import org.apache.pinot.segment.spi.index.IndexHandler;
import org.apache.pinot.segment.spi.index.StandardIndexes;
import org.apache.pinot.segment.spi.index.metadata.SegmentMetadataImpl;
import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.apache.pinot.spi.config.table.FieldConfig;
import org.apache.pinot.spi.config.table.TableConfig;
import org.apache.pinot.spi.data.Schema;
import org.slf4j.Logger;
@@ -112,10 +113,12 @@ public abstract class BaseIndexHandler implements IndexHandler {
FieldIndexConfigs fieldIndexConfig = _fieldIndexConfigs.get(columnName);
boolean dictionaryEnabled = fieldIndexConfig.getConfig(StandardIndexes.dictionary()).isEnabled();
ForwardIndexConfig forwardIndexConfig = fieldIndexConfig.getConfig(StandardIndexes.forward());
+ boolean dictionaryBasedForwardIndex = forwardIndexConfig.getEncodingType() == FieldConfig.EncodingType.DICTIONARY;
InvertedIndexAndDictionaryBasedForwardIndexCreator creator =
new InvertedIndexAndDictionaryBasedForwardIndexCreator(columnName, _segmentDirectory, dictionaryEnabled,
- forwardIndexConfig, segmentWriter, isTemporaryForwardIndex, _tableConfig.getTableName(),
+ dictionaryBasedForwardIndex, forwardIndexConfig, segmentWriter, isTemporaryForwardIndex,
+ _tableConfig.getTableName(),
_tableConfig.getIngestionConfig() != null && _tableConfig.getIngestionConfig().isContinueOnError());
creator.regenerateForwardIndex();
// Validate that the forward index is created.
diff --git a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java
index 4a7c7fb913c..c6cc8ef0505 100644
--- a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java
+++ b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java
@@ -23,8 +23,8 @@ import com.google.common.base.Preconditions;
import java.io.File;
import java.io.IOException;
import java.math.BigDecimal;
+import java.util.ArrayList;
import java.util.Arrays;
-import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
@@ -69,8 +69,8 @@ import org.apache.pinot.segment.spi.index.reader.ForwardIndexReaderContext;
import org.apache.pinot.segment.spi.store.SegmentDirectory;
import org.apache.pinot.segment.spi.utils.SegmentMetadataUtils;
import org.apache.pinot.spi.config.table.FieldConfig;
-import org.apache.pinot.spi.config.table.IndexingConfig;
import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.ComplexFieldSpec;
import org.apache.pinot.spi.data.FieldSpec;
import org.apache.pinot.spi.data.FieldSpec.DataType;
import org.apache.pinot.spi.data.Schema;
@@ -101,8 +101,14 @@ public class ForwardIndexHandler extends BaseIndexHandler {
private static final List<IndexType<?, ?, ?>> DICTIONARY_BASED_INDEXES_TO_REWRITE =
Arrays.asList(StandardIndexes.range(), StandardIndexes.fst(), StandardIndexes.inverted());
+ /// Re-enable operations are split by target encoding so the intent is explicit at the operation level: a
+ /// `forwardIndex.disabled` column being re-enabled may want to come back as either dict-encoded or raw,
+ /// depending on the new config. Both variants flow through the same regenerate-from-inverted-index path
+ /// (only the output writer differs), but having them as distinct operations makes the call site readable
+ /// and the test assertions specific.
protected enum Operation {
- DISABLE_FORWARD_INDEX, ENABLE_FORWARD_INDEX, DISABLE_DICTIONARY, ENABLE_DICTIONARY, CHANGE_INDEX_COMPRESSION_TYPE
+ DISABLE_FORWARD_INDEX, ENABLE_DICT_FORWARD_INDEX, ENABLE_RAW_FORWARD_INDEX, DISABLE_DICTIONARY,
+ ENABLE_DICTIONARY, CHANGE_INDEX_COMPRESSION_TYPE
}
@VisibleForTesting
@@ -131,6 +137,9 @@ public class ForwardIndexHandler extends BaseIndexHandler {
return;
}
+ Set<String> forwardIndexDisabledColumns =
+ FieldIndexConfigsUtil.columnsWithIndexDisabled(_fieldIndexConfigs.keySet(), StandardIndexes.forward(),
+ _fieldIndexConfigs);
for (Map.Entry<String, List<Operation>> entry : columnOperationsMap.entrySet()) {
String column = entry.getKey();
List<Operation> operations = entry.getValue();
@@ -142,37 +151,56 @@ public class ForwardIndexHandler extends BaseIndexHandler {
// handlers that need the forward index to construct their own indexes will have it available.
_tmpForwardIndexColumns.add(column);
break;
- case ENABLE_FORWARD_INDEX:
+ case ENABLE_DICT_FORWARD_INDEX: {
+ // Rebuilding a dict-encoded forward index requires the dictionary to be present afterwards — the
+ // forward index stores dict ids that resolve through it.
ColumnMetadata columnMetadata = createForwardIndexIfNeeded(segmentWriter, column, false);
- if (columnMetadata.hasDictionary()) {
- if (!segmentWriter.hasIndexFor(column, StandardIndexes.dictionary())) {
- throw new IllegalStateException(String.format(
- "Dictionary should still exist after rebuilding forward index for dictionary column: %s", column));
- }
+ if (!columnMetadata.hasDictionary() || !segmentWriter.hasIndexFor(column, StandardIndexes.dictionary())) {
+ throw new IllegalStateException(String.format(
+ "Dictionary should still exist after rebuilding dict-encoded forward index for column: %s", column));
+ }
+ break;
+ }
+ case ENABLE_RAW_FORWARD_INDEX: {
+ // Two on-disk shapes can land here:
+ // (a) Forward index is absent → rebuild from dict + inverted as raw forward (no dict change).
+ // (b) Forward index exists and is DICT-encoded → flip the encoding to RAW in place, keeping the
+ // dictionary because another enabled index requires it (e.g. dict + inverted + raw forward).
+ // The dictionary stays untouched in either case; if the new config wants the dictionary gone, the
+ // dict-transition step queues DISABLE_DICTIONARY separately (which on a dict-encoded forward
+ // already takes the convert-and-drop path, so the planner suppresses this op in that combo).
+ if (segmentWriter.hasIndexFor(column, StandardIndexes.forward())
+ && isForwardIndexDictionaryEncoded(column)) {
+ convertDictForwardToRawKeepingDictionary(column, segmentWriter);
} else {
- if (segmentWriter.hasIndexFor(column, StandardIndexes.dictionary())) {
- throw new IllegalStateException(
- String.format("Dictionary should not exist after rebuilding forward index for raw column: %s",
- column));
- }
+ createForwardIndexIfNeeded(segmentWriter, column, false);
+ }
+ if (!segmentWriter.hasIndexFor(column, StandardIndexes.forward())) {
+ throw new IllegalStateException(
+ String.format("Forward index was not created for column: %s", column));
}
break;
+ }
case DISABLE_DICTIONARY:
- Set<String> newForwardIndexDisabledColumns =
- FieldIndexConfigsUtil.columnsWithIndexDisabled(_fieldIndexConfigs.keySet(), StandardIndexes.forward(),
- _fieldIndexConfigs);
- if (newForwardIndexDisabledColumns.contains(column)) {
- removeDictionaryFromForwardIndexDisabledColumn(column, segmentWriter);
+ if (forwardIndexDisabledColumns.contains(column)) {
+ disableDictionary(column, segmentWriter, "Disable dictionary when no forward index exists");
if (segmentWriter.hasIndexFor(column, StandardIndexes.dictionary())) {
throw new IllegalStateException(
String.format("Dictionary should not exist after disabling dictionary for column: %s", column));
}
- } else {
+ } else if (isForwardIndexDictionaryEncoded(column)) {
disableDictionaryAndCreateRawForwardIndex(column, segmentWriter);
+ } else {
+ disableDictionary(column, segmentWriter, "Disable dictionary while keeping raw forward index");
}
break;
case ENABLE_DICTIONARY:
- createDictBasedForwardIndex(column, segmentWriter);
+ if (_fieldIndexConfigs.get(column).getConfig(StandardIndexes.forward()).getEncodingType()
+ == FieldConfig.EncodingType.DICTIONARY) {
+ createDictBasedForwardIndex(column, segmentWriter);
+ } else {
+ createDictionaryForRawForwardIndex(column, segmentWriter);
+ }
if (!segmentWriter.hasIndexFor(column, StandardIndexes.forward())) {
throw new IllegalStateException(String.format("Forward index was not created for column: %s", column));
}
@@ -197,150 +225,204 @@ public class ForwardIndexHandler extends BaseIndexHandler {
return columnOperationsMap;
}
- // Get all physical columns, excluding virtual columns like $docId, $hostName and $segmentName.
Set<String> existingAllColumns = _segmentDirectory.getSegmentMetadata().getSchema().getPhysicalColumnNames();
Set<String> existingDictColumns = _segmentDirectory.getColumnsWithIndex(StandardIndexes.dictionary());
Set<String> existingForwardIndexColumns = _segmentDirectory.getColumnsWithIndex(StandardIndexes.forward());
+ Set<String> existingInvertedIndexColumns =
+ segmentReader.toSegmentDirectory().getColumnsWithIndex(StandardIndexes.inverted());
String segmentName = _segmentDirectory.getSegmentMetadata().getName();
+
for (String column : existingAllColumns) {
if (!_schema.hasColumn(column)) {
// _schema will be null only in tests
LOGGER.info("Column: {} of segment: {} is not in schema, skipping updating forward index", column, segmentName);
continue;
}
- boolean existingHasDict = existingDictColumns.contains(column);
- boolean existingHasFwd = existingForwardIndexColumns.contains(column);
- FieldIndexConfigs newConf = _fieldIndexConfigs.get(column);
- boolean newIsFwd = newConf.getConfig(StandardIndexes.forward()).isEnabled();
- boolean newIsDict = newConf.getConfig(StandardIndexes.dictionary()).isEnabled();
- boolean newIsRange = newConf.getConfig(StandardIndexes.range()).isEnabled();
-
- if (existingHasFwd && !newIsFwd) {
- // Existing column has a forward index. New column config disables the forward index
-
- ColumnMetadata columnMetadata = _segmentDirectory.getSegmentMetadata().getColumnMetadataFor(column);
- if (columnMetadata.isSorted()) {
- // Check if the column is sorted. If sorted, disabling forward index should be a no-op. Do not return an
- // operation for this column related to disabling forward index.
- LOGGER.warn("Trying to disable the forward index for a sorted column: {} of segment: {}, ignoring", column,
- segmentName);
- continue;
- }
+ // Skip complex types (MAP, etc.) — they do not support dictionary encoding
+ FieldSpec fieldSpec = _schema.getFieldSpecFor(column);
+ if (fieldSpec instanceof ComplexFieldSpec) {
+ continue;
+ }
- if (existingHasDict) {
- if (!newIsDict) {
- // Dictionary was also disabled. Just disable the dictionary and remove it along with the forward index
- // If range index exists, don't try to regenerate it on toggling the dictionary, throw an error instead
- Preconditions.checkState(!newIsRange, String.format(
- "Must disable range index (enabled) to disable the dictionary and forward index for column: %s of "
- + "segment: %s or refresh / back-fill the forward index", column, segmentName));
- columnOperationsMap.put(column,
- Arrays.asList(Operation.DISABLE_FORWARD_INDEX, Operation.DISABLE_DICTIONARY));
- } else {
- // Dictionary is still enabled, keep it but remove the forward index
- columnOperationsMap.put(column, Collections.singletonList(Operation.DISABLE_FORWARD_INDEX));
- }
- } else {
- if (!newIsDict) {
- // Dictionary remains disabled and we should not reconstruct temporary forward index as dictionary based
- columnOperationsMap.put(column, Collections.singletonList(Operation.DISABLE_FORWARD_INDEX));
- } else {
- // Dictionary is enabled, creation of dictionary and conversion to dictionary based forward index is needed
- columnOperationsMap.put(column,
- Arrays.asList(Operation.DISABLE_FORWARD_INDEX, Operation.ENABLE_DICTIONARY));
- }
- }
- } else if (!existingHasFwd && newIsFwd) {
- // Existing column does not have a forward index. New column config enables the forward index
-
- ColumnMetadata columnMetadata = _segmentDirectory.getSegmentMetadata().getColumnMetadataFor(column);
- if (columnMetadata != null && columnMetadata.isSorted()) {
- // Check if the column is sorted. If sorted, disabling forward index should be a no-op and forward index
- // should already exist. Do not return an operation for this column related to enabling forward index.
- LOGGER.warn("Trying to enable the forward index for a sorted column: {} of segment: {}, ignoring", column,
- segmentName);
- continue;
- }
+ // Forward-index encoding is the source-of-truth for "does forward exist and how is it laid out". Three
+ // states: DICTIONARY (dict-encoded forward), RAW (raw forward), null (forward disabled / not on disk).
+ FieldConfig.EncodingType existingFwdEncoding = existingForwardIndexColumns.contains(column)
+ ? _segmentDirectory.getSegmentMetadata().getColumnMetadataFor(column).getForwardIndexEncoding() : null;
+ ForwardIndexConfig newFwdConf = _fieldIndexConfigs.get(column).getConfig(StandardIndexes.forward());
+ FieldConfig.EncodingType newFwdEncoding = newFwdConf.isEnabled() ? newFwdConf.getEncodingType() : null;
+
+ List<Operation> ops = computeColumnOperations(column, fieldSpec, segmentReader,
+ existingDictColumns.contains(column), existingFwdEncoding, newFwdEncoding,
+ existingInvertedIndexColumns.contains(column), segmentName);
+ if (!ops.isEmpty()) {
+ columnOperationsMap.put(column, ops);
+ }
+ }
+ if (!columnOperationsMap.isEmpty()) {
+ LOGGER.info("Need to apply columnOperations: {} for forward index for segment: {}", columnOperationsMap,
+ segmentName);
+ }
+ return columnOperationsMap;
+ }
- // Get list of columns with inverted index
- Set<String> existingInvertedIndexColumns =
- segmentReader.toSegmentDirectory().getColumnsWithIndex(StandardIndexes.inverted());
- if (!existingHasDict || !existingInvertedIndexColumns.contains(column)) {
- // If either dictionary or inverted index is missing on the column there is no way to re-generate the forward
- // index. Treat this as a no-op and log a warning.
+ /// Compute the operations needed to bring one column from its existing on-disk state to the desired state in
+ /// the new config. The logic decomposes into four orthogonal questions, computed in this order:
+ ///
+ /// 1. **Forward-index transition** — based on `existingFwdEncoding` vs `newFwdEncoding` (each is one of
+ /// `DICTIONARY`, `RAW`, or `null` for "forward index disabled / not on disk").
+ /// 2. **Dictionary transition** — based on `existingHasDict` vs `desiredDict`, where
+ /// `desiredDict = newIsDict || any-enabled-index-requires-dict`. The "force on if required" rule is the
+ /// only place this method consults other indexes — once `desiredDict` is computed, the rest of the logic
+ /// treats it as the source of truth.
+ /// 3. **Compression-type change** — only when no encoding change happened (forward + dict both unchanged).
+ /// 4. **Cross-cutting guards** — sorted columns can't toggle forward; range index format is incompatible
+ /// with disabling the dictionary; enabling forward needs dict + inverted on disk; enabling dict needs
+ /// forward to be on so the dict can be bootstrapped.
+ private List<Operation> computeColumnOperations(String column, FieldSpec fieldSpec,
+ SegmentDirectory.Reader segmentReader, boolean existingHasDict,
+ @Nullable FieldConfig.EncodingType existingFwdEncoding, @Nullable FieldConfig.EncodingType newFwdEncoding,
+ boolean existingHasInverted, String segmentName)
+ throws Exception {
+ FieldIndexConfigs newConf = _fieldIndexConfigs.get(column);
+ boolean newIsDict = newConf.getConfig(StandardIndexes.dictionary()).isEnabled();
+ boolean newIsRange = newConf.getConfig(StandardIndexes.range()).isEnabled();
+ ColumnMetadata columnMetadata = _segmentDirectory.getSegmentMetadata().getColumnMetadataFor(column);
+ boolean existingHasFwd = existingFwdEncoding != null;
+ boolean newHasFwd = newFwdEncoding != null;
+
+ // Force the dictionary on whenever any enabled index requires it — otherwise that index would be left
+ // orphaned after reload. This is the only place where index-level dependencies feed into the dictionary
+ // decision; the rest of this method treats `desiredDict` as authoritative.
+ boolean dictRequiredByIndex = fieldSpec != null && DictionaryIndexConfig.requiresDictionary(fieldSpec, newConf);
+ boolean desiredDict = newIsDict || dictRequiredByIndex;
+
+ List<Operation> ops = new ArrayList<>(2);
+
+ // 1. Forward-index transition. Three sub-cases:
+ // (a) existing has forward, new disables it → DISABLE_FORWARD_INDEX
+ // (b) existing disabled, new re-enables it → ENABLE_{DICT|RAW}_FORWARD_INDEX (rebuild from dict+inverted)
+ // (c) forward stays on but encoding flips DICT⇄RAW → ENABLE_{DICT|RAW}_FORWARD_INDEX (rewrite in place)
+ // When the dictionary needs to stay (because another index requires it) the encoding-flip handler keeps
+ // it. When the dictionary needs to go, the dict transition step (#2) queues DISABLE_DICTIONARY; the
+ // existing DISABLE_DICTIONARY handler covers the DICT→RAW + drop-dict path on its own, so we don't
+ // queue an ENABLE_RAW_FORWARD_INDEX in that combo to avoid double-rewrite.
+ if (existingHasFwd != newHasFwd) {
+ if (columnMetadata != null && columnMetadata.isSorted()) {
+ LOGGER.warn("Trying to {} the forward index for a sorted column: {} of segment: {}, ignoring",
+ existingHasFwd ? "disable" : "enable", column, segmentName);
+ return ops;
+ }
+ if (existingHasFwd) {
+ ops.add(Operation.DISABLE_FORWARD_INDEX);
+ } else {
+ // Re-creating the forward index requires both dict and inverted on disk. After re-enable the dict
+ // status stays whatever was needed for the rebuild — independent dict transitions don't run in this
+ // combo (the rebuild needs the dict, so dropping it in the same reload is not supported).
+ if (!existingHasDict || !existingHasInverted) {
LOGGER.warn(
"Trying to enable the forward index for a column: {} of segment: {} missing either the dictionary ({}) "
+ "and / or the inverted index ({}) is not possible. Either a refresh or back-fill is required to "
+ "get the forward index, ignoring", column, segmentName, existingHasDict ? "enabled" : "disabled",
- existingInvertedIndexColumns.contains(column) ? "enabled" : "disabled");
- continue;
+ existingHasInverted ? "enabled" : "disabled");
+ return ops;
}
+ ops.add(newFwdEncoding == FieldConfig.EncodingType.RAW
+ ? Operation.ENABLE_RAW_FORWARD_INDEX : Operation.ENABLE_DICT_FORWARD_INDEX);
+ return ops;
+ }
+ } else if (existingHasFwd && existingFwdEncoding != newFwdEncoding && existingHasDict == desiredDict) {
+ // Encoding flip with forward staying on AND the dictionary state staying the same. The other two
+ // possibilities — DICT→RAW + drop-dict and RAW→DICT + create-dict — are already handled by the
+ // dict-transition step below: DISABLE_DICTIONARY's handler converts dict-encoded forward to raw and
+ // drops the dict together, and ENABLE_DICTIONARY's handler builds the dictionary and converts raw
+ // forward to dict-encoded. Only the dict-stays cases need this explicit op:
+ // - DICT→RAW with dict kept (because another index requires it, e.g. inverted)
+ // - RAW→DICT with shared dict already present
+ if (columnMetadata != null && columnMetadata.isSorted()) {
+ LOGGER.warn("Trying to flip forward-index encoding (DICT⇄RAW) for a sorted column: {} of segment: {}, "
+ + "ignoring", column, segmentName);
+ return ops;
+ }
+ ops.add(newFwdEncoding == FieldConfig.EncodingType.RAW
+ ? Operation.ENABLE_RAW_FORWARD_INDEX : Operation.ENABLE_DICT_FORWARD_INDEX);
+ }
- columnOperationsMap.put(column, Collections.singletonList(Operation.ENABLE_FORWARD_INDEX));
- } else if (!existingHasFwd) {
- // Forward index is disabled for the existing column and should remain disabled based on the latest config
- // Need some checks to see whether the dictionary is being enabled or disabled here and take appropriate actions
-
- // If the dictionary is not enabled on the existing column it must be on the new noDictionary column list.
- // Cannot enable the dictionary for a column with forward index disabled.
- Preconditions.checkState(existingHasDict || !newIsDict, String.format(
- "Cannot regenerate the dictionary for column: %s of segment: %s with forward index disabled. Please "
- + "refresh or back-fill the data to add back the forward index", column, segmentName));
-
- if (existingHasDict && !newIsDict) {
- // Dictionary is currently enabled on this column but is supposed to be disabled. Remove the dictionary
- // and update the segment metadata If the range index exists then throw an error since we are not
- // regenerating the range index on toggling the dictionary
- Preconditions.checkState(!newIsRange, String.format(
- "Must disable range index (enabled) to disable the dictionary for a forwardIndexDisabled column: %s of "
- + "segment: %s or refresh / back-fill the forward index", column, segmentName));
- columnOperationsMap.put(column, Collections.singletonList(Operation.DISABLE_DICTIONARY));
- }
- } else if (!existingHasDict && newIsDict) {
- // Existing column is RAW. New column is dictionary enabled.
- ColumnMetadata existingColumnMetadata = _segmentDirectory.getSegmentMetadata().getColumnMetadataFor(column);
- IndexingConfig indexingConfig = _tableConfig.getIndexingConfig();
- if (existingColumnMetadata.getFieldSpec().getFieldType() != FieldSpec.FieldType.COMPLEX
- && DictionaryIndexType.ignoreDictionaryOverride(indexingConfig.isOptimizeDictionary(),
- indexingConfig.isOptimizeDictionaryForMetrics(), indexingConfig.getNoDictionarySizeRatioThreshold(),
- indexingConfig.getNoDictionaryCardinalityRatioThreshold(), existingColumnMetadata.getFieldSpec(),
- _fieldIndexConfigs.get(column), existingColumnMetadata.getCardinality(),
- existingColumnMetadata.getTotalNumberOfEntries())) {
- columnOperationsMap.put(column, Collections.singletonList(Operation.ENABLE_DICTIONARY));
- }
- } else if (existingHasDict && !newIsDict) {
- // Existing column has dictionary. New config for the column is RAW.
- if (shouldDisableDictionary(column, _segmentDirectory.getSegmentMetadata().getColumnMetadataFor(column))) {
- columnOperationsMap.put(column, Collections.singletonList(Operation.DISABLE_DICTIONARY));
- }
- } else if (!existingHasDict) {
- // Both existing and new column is RAW forward index encoded. Check if compression needs to be changed.
+ // 2. Dictionary transition.
+ if (existingHasDict && !desiredDict) {
+ // When the forward index is being kept, the range index can be rebuilt against raw values, additional
+ // guards (auto-generated cardinality-1 columns, on-disk inverted/FST) apply via shouldDisableDictionary.
+ // When the forward index is also being removed — or was already off — the range index can't be rebuilt,
+ // so dropping the dictionary while range stays in the new config leaves the segment inconsistent.
+ if (!existingHasFwd) {
+ Preconditions.checkState(!newIsRange, String.format(
+ "Must disable range index (enabled) to disable the dictionary for a forwardIndexDisabled column: %s "
+ + "of segment: %s or refresh / back-fill the forward index", column, segmentName));
+ ops.add(Operation.DISABLE_DICTIONARY);
+ } else if (ops.contains(Operation.DISABLE_FORWARD_INDEX)) {
+ Preconditions.checkState(!newIsRange, String.format(
+ "Must disable range index (enabled) to disable the dictionary and forward index for column: %s of "
+ + "segment: %s or refresh / back-fill the forward index", column, segmentName));
+ ops.add(Operation.DISABLE_DICTIONARY);
+ } else if (shouldDisableDictionary(column, columnMetadata, fieldSpec, newConf)) {
+ ops.add(Operation.DISABLE_DICTIONARY);
+ }
+ } else if (!existingHasDict && desiredDict) {
+ // Dict can only be created from forward values, so refuse if forward will be off in the final state.
+ Preconditions.checkState(existingHasFwd || newHasFwd, String.format(
+ "Cannot regenerate the dictionary for column: %s of segment: %s with forward index disabled. Please "
+ + "refresh or back-fill the data to add back the forward index", column, segmentName));
+ // If an index requires the dict, always enable it. Otherwise apply the optimize-dictionary heuristic so
+ // reload doesn't force a dictionary onto a high-cardinality column that segment creation would have kept raw.
+ if (dictRequiredByIndex || shouldEnableDictionaryHeuristically(column, fieldSpec, newConf, columnMetadata,
+ segmentName)) {
+ ops.add(Operation.ENABLE_DICTIONARY);
+ }
+ }
+
+ // 3. Compression-type change (only when no encoding change happened).
+ if (ops.isEmpty() && existingFwdEncoding != null && existingFwdEncoding == newFwdEncoding
+ && existingHasDict == desiredDict) {
+ if (existingFwdEncoding == FieldConfig.EncodingType.RAW) {
// TODO: Also check if raw index version needs to be changed
if (shouldChangeRawCompressionType(column, segmentReader)) {
- columnOperationsMap.put(column, Collections.singletonList(Operation.CHANGE_INDEX_COMPRESSION_TYPE));
- }
- } else {
- // Both existing and new column is dictionary encoded. Check if compression needs to be changed.
- if (shouldChangeDictIdCompressionType(column, segmentReader)) {
- columnOperationsMap.put(column, Collections.singletonList(Operation.CHANGE_INDEX_COMPRESSION_TYPE));
+ ops.add(Operation.CHANGE_INDEX_COMPRESSION_TYPE);
}
+ } else if (shouldChangeDictIdCompressionType(column, segmentReader)) {
+ ops.add(Operation.CHANGE_INDEX_COMPRESSION_TYPE);
}
}
- if (!columnOperationsMap.isEmpty()) {
- LOGGER.info("Need to apply columnOperations: {} for forward index for segment: {}", columnOperationsMap,
- segmentName);
+
+ return ops;
+ }
+
+ /// Optimize-dictionary heuristic for ENABLE_DICTIONARY. Used when the new config wants a dictionary but no
+ /// secondary index actually requires it; the heuristic decides whether the column's cardinality / size
+ /// characteristics make a dictionary worthwhile, mirroring what segment creation would have chosen.
+ private boolean shouldEnableDictionaryHeuristically(String column, FieldSpec fieldSpec, FieldIndexConfigs newConf,
+ @Nullable ColumnMetadata colMeta, String segmentName) {
+ if (fieldSpec == null || colMeta == null || colMeta.isSorted()) {
+ return true;
}
- return columnOperationsMap;
+ boolean keepDict = DictionaryIndexType.ignoreDictionaryOverride(
+ _tableConfig.getIndexingConfig().isOptimizeDictionary(),
+ _tableConfig.getIndexingConfig().isOptimizeDictionaryForMetrics(),
+ _tableConfig.getIndexingConfig().getNoDictionarySizeRatioThreshold(),
+ _tableConfig.getIndexingConfig().getNoDictionaryCardinalityRatioThreshold(),
+ fieldSpec, newConf, colMeta.getCardinality(), colMeta.getTotalNumberOfEntries());
+ if (!keepDict) {
+ LOGGER.info("Skipping ENABLE_DICTIONARY for column: {} of segment: {} because optimize-dictionary "
+ + "heuristics determined dictionary is not beneficial (cardinality={}, totalNumberOfEntries={})",
+ column, segmentName, colMeta.getCardinality(), colMeta.getTotalNumberOfEntries());
+ }
+ return keepDict;
}
- private void removeDictionaryFromForwardIndexDisabledColumn(String column, SegmentDirectory.Writer segmentWriter)
+ private void disableDictionary(String column, SegmentDirectory.Writer segmentWriter, String reason)
throws Exception {
// Remove the dictionary and update the metadata to indicate that the dictionary is no longer present
segmentWriter.removeIndex(column, StandardIndexes.dictionary());
String segmentName = _segmentDirectory.getSegmentMetadata().getName();
- LOGGER.info(
- "Removed dictionary for noForwardIndex column. Updating metadata properties for segment={} and column={}",
- segmentName, column);
+ LOGGER.info("{}: Updating metadata properties for segment={} and column={}", reason, segmentName, column);
Map<String, String> metadataProperties = new HashMap<>();
metadataProperties.put(getKeyFor(column, HAS_DICTIONARY), String.valueOf(false));
metadataProperties.put(getKeyFor(column, FORWARD_INDEX_ENCODING), FieldConfig.EncodingType.RAW.name());
@@ -351,17 +433,42 @@ public class ForwardIndexHandler extends BaseIndexHandler {
// Remove the inverted index, FST index and range index
removeDictRelatedIndexes(column, segmentWriter);
+
+ LOGGER.info("Removed dictionary and all corresponding indexes for segment: {}, column: {}", segmentName, column);
+ }
+
+ private boolean isForwardIndexDictionaryEncoded(String column) {
+ return _segmentDirectory.getSegmentMetadata().getColumnMetadataFor(column).getForwardIndexEncoding()
+ == FieldConfig.EncodingType.DICTIONARY;
+ }
+
+ /// Returns {@code true} when removing the dictionary on this column would leave another enabled index
+ /// orphaned. Aggregates {@link IndexType#requiresDictionary} over every enabled index in the new config
+ /// via {@link DictionaryIndexConfig#requiresDictionary(FieldSpec, FieldIndexConfigs)}, so this automatically
+ /// covers inverted, FST, IFST, and any future dict-requiring index type. Only the new config matters — any
+ /// existing on-disk index no longer present in the new config is removed by its own handler later in
+ /// preprocessing and does not need to keep the dictionary alive.
+ private boolean dictionaryRequiredByOtherIndex(String column, @Nullable FieldSpec fieldSpec,
+ FieldIndexConfigs newConf) {
+ if (fieldSpec != null && DictionaryIndexConfig.requiresDictionary(fieldSpec, newConf)) {
+ LOGGER.warn("Cannot disable dictionary for column={} because at least one enabled index in the new "
+ + "config still requires a dictionary.", column);
+ return true;
+ }
+ return false;
}
- private boolean shouldDisableDictionary(String column, ColumnMetadata existingColumnMetadata) {
+ private boolean shouldDisableDictionary(String column, ColumnMetadata existingColumnMetadata,
+ @Nullable FieldSpec fieldSpec, FieldIndexConfigs newConf) {
if (existingColumnMetadata.isAutoGenerated() && existingColumnMetadata.getCardinality() == 1) {
LOGGER.warn("Cannot disable dictionary for auto-generated column={} with cardinality=1.", column);
return false;
}
- // Allow disabling dictionary only if the new config specifies that inverted index and FST index should not
- // be present. So for existing segments where inverted index and FST index are already present, disabling
- // dictionary will only be allowed if FST and inverted index are also disabled.
+ if (dictionaryRequiredByOtherIndex(column, fieldSpec, newConf)) {
+ return false;
+ }
+
if (hasIndex(column, StandardIndexes.inverted()) || hasIndex(column, StandardIndexes.fst())) {
LOGGER.warn("Cannot disable dictionary as column={} has FST index or inverted index or both.", column);
return false;
@@ -427,17 +534,19 @@ public class ForwardIndexHandler extends BaseIndexHandler {
throws Exception {
ColumnMetadata existingColMetadata = _segmentDirectory.getSegmentMetadata().getColumnMetadataFor(column);
boolean isSingleValue = existingColMetadata.isSingleValue();
- boolean hasDictionary = existingColMetadata.hasDictionary();
+ boolean isDictionaryEncodedForwardIndex = isForwardIndexDictionaryEncoded(column);
File indexDir = _segmentDirectory.getSegmentMetadata().getIndexDir();
String segmentName = _segmentDirectory.getSegmentMetadata().getName();
File inProgress = new File(indexDir, column + ".fwd.inprogress");
String fileExtension;
if (isSingleValue) {
- fileExtension = hasDictionary ? V1Constants.Indexes.UNSORTED_SV_FORWARD_INDEX_FILE_EXTENSION
+ fileExtension = isDictionaryEncodedForwardIndex
+ ? V1Constants.Indexes.UNSORTED_SV_FORWARD_INDEX_FILE_EXTENSION
: V1Constants.Indexes.RAW_SV_FORWARD_INDEX_FILE_EXTENSION;
} else {
- fileExtension = hasDictionary ? V1Constants.Indexes.UNSORTED_MV_FORWARD_INDEX_FILE_EXTENSION
+ fileExtension = isDictionaryEncodedForwardIndex
+ ? V1Constants.Indexes.UNSORTED_MV_FORWARD_INDEX_FILE_EXTENSION
: V1Constants.Indexes.RAW_MV_FORWARD_INDEX_FILE_EXTENSION;
}
File fwdIndexFile = new File(indexDir, column + fileExtension);
@@ -481,6 +590,8 @@ public class ForwardIndexHandler extends BaseIndexHandler {
.withTableNameWithType(_tableConfig.getTableName())
.withContinueOnError(_tableConfig.getIngestionConfig() != null
&& _tableConfig.getIngestionConfig().isContinueOnError());
+ // Encoding flows through ForwardIndexConfig; for compression-change rewrite the encoding does not change so
+ // the config in _fieldIndexConfigs already carries the correct encoding.
// Set entry length info for raw index creators. No need to set this when changing dictionary id compression type.
if (!reader.isDictionaryEncoded() && !columnMetadata.getDataType().getStoredType().isFixedWidth()) {
int lengthOfLongestEntry = reader.getLengthOfLongestEntry();
@@ -505,8 +616,8 @@ public class ForwardIndexHandler extends BaseIndexHandler {
builder.withMaxRowLengthInBytes(maxRowLengthInBytes);
}
}
- IndexCreationContext context = builder.build();
ForwardIndexConfig config = _fieldIndexConfigs.get(column).getConfig(StandardIndexes.forward());
+ IndexCreationContext context = builder.build();
try (ForwardIndexCreator creator = StandardIndexes.forward().createIndexCreator(context, config)) {
if (!reader.getStoredType().equals(creator.getValueType())) {
// Creator stored type should match reader stored type for raw columns. We do not support changing datatypes.
@@ -850,7 +961,7 @@ public class ForwardIndexHandler extends BaseIndexHandler {
LOGGER.info("Creating a new dictionary for segment={} and column={}", segmentName, column);
int numDocs = existingColMetadata.getTotalDocs();
- statsCollector = getStatsCollector(column, fieldSpec.getDataType().getStoredType());
+ statsCollector = getStatsCollector(column, fieldSpec.getDataType().getStoredType(), true);
// NOTE:
// Special null handling is not necessary here. This is because, the existing default null value in the raw
// forwardIndex will be retained as such while created the dictionary and dict-based forward index. Also, null
@@ -872,6 +983,7 @@ public class ForwardIndexHandler extends BaseIndexHandler {
LOGGER.info("Built dictionary. Rewriting dictionary enabled forward index for segment={} and column={}",
segmentName, column);
+ ForwardIndexConfig config = _fieldIndexConfigs.get(column).getConfig(StandardIndexes.forward());
IndexCreationContext context = IndexCreationContext.builder()
.withIndexDir(indexDir)
.withFieldSpec(fieldSpec)
@@ -882,7 +994,6 @@ public class ForwardIndexHandler extends BaseIndexHandler {
.withContinueOnError(
_tableConfig.getIngestionConfig() != null && _tableConfig.getIngestionConfig().isContinueOnError())
.build();
- ForwardIndexConfig config = _fieldIndexConfigs.get(column).getConfig(StandardIndexes.forward());
try (ForwardIndexCreator creator = StandardIndexes.forward().createIndexCreator(context, config)) {
forwardIndexRewriteHelper(column, existingColMetadata, reader, creator, numDocs, dictionaryCreator, null);
}
@@ -922,6 +1033,77 @@ public class ForwardIndexHandler extends BaseIndexHandler {
LOGGER.info("Created dictionary based forward index for segment: {}, column: {}", segmentName, column);
}
+ private void createDictionaryForRawForwardIndex(String column, SegmentDirectory.Writer segmentWriter)
+ throws Exception {
+ SegmentMetadataImpl segmentMetadata = _segmentDirectory.getSegmentMetadata();
+ File indexDir = segmentMetadata.getIndexDir();
+ String segmentName = segmentMetadata.getName();
+ File inProgress = new File(indexDir, column + ".rawdict.inprogress");
+ File dictionaryFile = new File(indexDir, column + V1Constants.Dict.FILE_EXTENSION);
+
+ if (!inProgress.exists()) {
+ FileUtils.touch(inProgress);
+ } else {
+ FileUtils.deleteQuietly(dictionaryFile);
+ }
+
+ ColumnMetadata existingColMetadata = segmentMetadata.getColumnMetadataFor(column);
+ FieldSpec fieldSpec = existingColMetadata.getFieldSpec();
+ int dictionaryCardinality;
+ int dictionaryElementSize;
+ try (ForwardIndexReader<?> reader = StandardIndexes.forward().getReaderFactory()
+ .createIndexReader(segmentWriter, _fieldIndexConfigs.get(column), existingColMetadata)) {
+ AbstractColumnStatisticsCollector statsCollector =
+ getStatsCollector(column, fieldSpec.getDataType().getStoredType(), true);
+ try (PinotSegmentColumnReader columnReader = new PinotSegmentColumnReader(reader, null, null,
+ existingColMetadata.getMaxNumberOfMultiValues())) {
+ for (int i = 0; i < existingColMetadata.getTotalDocs(); i++) {
+ statsCollector.collect(columnReader.getValue(i));
+ }
+ statsCollector.seal();
+ }
+ DictionaryIndexConfig dictConf = _fieldIndexConfigs.get(column).getConfig(StandardIndexes.dictionary());
+ boolean optimizeDictionaryType = _tableConfig.getIndexingConfig().isOptimizeDictionaryType();
+ boolean useVarLength = dictConf.isUseVarLengthDictionary()
+ || DictionaryIndexType.shouldUseVarLengthDictionary(reader.getStoredType(), statsCollector)
+ || (optimizeDictionaryType
+ && DictionaryIndexType.optimizeTypeShouldUseVarLengthDictionary(reader.getStoredType(), statsCollector));
+ try (SegmentDictionaryCreator dictionaryCreator =
+ new SegmentDictionaryCreator(fieldSpec, indexDir, useVarLength)) {
+ // Capture cardinality from the unique-values set passed to build() rather than from the stats collector,
+ // because some collectors over-count by tracking null/duplicate occurrences separately. The dictionary on
+ // disk has exactly java.lang.reflect.Array.getLength(uniqueValues) entries.
+ Object uniqueValues = statsCollector.getUniqueValuesSet();
+ dictionaryCardinality = java.lang.reflect.Array.getLength(uniqueValues);
+ dictionaryCreator.build(uniqueValues);
+ dictionaryElementSize = dictionaryCreator.getNumBytesPerEntry();
+ }
+ }
+
+ LoaderUtils.writeIndexToV3Format(segmentWriter, column, dictionaryFile, StandardIndexes.dictionary());
+
+ Preconditions.checkState(dictionaryCardinality > 0,
+ "Dictionary cardinality for column %s must be > 0 to compute BITS_PER_ELEMENT", column);
+ Map<String, String> metadataProperties = new HashMap<>();
+ metadataProperties.put(getKeyFor(column, HAS_DICTIONARY), String.valueOf(true));
+ metadataProperties.put(getKeyFor(column, FORWARD_INDEX_ENCODING), FieldConfig.EncodingType.RAW.name());
+ metadataProperties.put(getKeyFor(column, DICTIONARY_ELEMENT_SIZE), String.valueOf(dictionaryElementSize));
+ metadataProperties.put(getKeyFor(column, CARDINALITY), String.valueOf(dictionaryCardinality));
+ metadataProperties.put(getKeyFor(column, BITS_PER_ELEMENT),
+ String.valueOf(PinotDataBitSet.getNumBitsPerValue(dictionaryCardinality - 1)));
+ SegmentMetadataUtils.updateMetadataProperties(_segmentDirectory, metadataProperties);
+
+ // Secondary indexes that require dictionary semantics should be rebuilt against the new shared dictionary.
+ removeDictRelatedIndexes(column, segmentWriter);
+
+ // Delete the inprogress marker only after the cleanup is complete. If the JVM dies between the metadata write
+ // and the secondary-index removal, the marker still exists; on next preprocess `inProgress.exists()` deletes
+ // the partial dictionary file and the rebuild starts fresh.
+ FileUtils.deleteQuietly(inProgress);
+
+ LOGGER.info("Created dictionary for raw forward index segment={} column={}", segmentName, column);
+ }
+
static void removeDictRelatedIndexes(String column, SegmentDirectory.Writer segmentWriter) {
// TODO: Move this logic as a static function in each index creator.
@@ -979,6 +1161,47 @@ public class ForwardIndexHandler extends BaseIndexHandler {
LOGGER.info("Created raw based forward index for segment: {}, column: {}", segmentName, column);
}
+ /// Flip a dict-encoded forward index to RAW while keeping the dictionary intact. Used when an enabled
+ /// secondary index (e.g. inverted, FST, IFST) requires the dictionary to remain after the encoding flip —
+ /// the secondary index continues to operate against the same dict ids, so it does not need to be removed
+ /// or rebuilt. Compare with {@link #disableDictionaryAndCreateRawForwardIndex}, which performs the same
+ /// rewrite and additionally drops the dictionary plus all dict-dependent indexes.
+ private void convertDictForwardToRawKeepingDictionary(String column, SegmentDirectory.Writer segmentWriter)
+ throws Exception {
+ ColumnMetadata existingColMetadata = _segmentDirectory.getSegmentMetadata().getColumnMetadataFor(column);
+ boolean isSingleValue = existingColMetadata.isSingleValue();
+
+ File indexDir = _segmentDirectory.getSegmentMetadata().getIndexDir();
+ String segmentName = _segmentDirectory.getSegmentMetadata().getName();
+ File inProgress = new File(indexDir, column + ".fwdraw.inprogress");
+ String fileExtension = isSingleValue ? V1Constants.Indexes.RAW_SV_FORWARD_INDEX_FILE_EXTENSION
+ : V1Constants.Indexes.RAW_MV_FORWARD_INDEX_FILE_EXTENSION;
+ File fwdIndexFile = new File(indexDir, column + fileExtension);
+
+ if (!inProgress.exists()) {
+ FileUtils.touch(inProgress);
+ } else {
+ FileUtils.deleteQuietly(fwdIndexFile);
+ }
+
+ LOGGER.info("Converting dict-encoded forward index to raw (keeping dictionary) for segment={} column={}",
+ segmentName, column);
+ rewriteDictToRawForwardIndex(existingColMetadata, segmentWriter, indexDir);
+
+ // Remove the old dict-encoded forward index file and write the new raw forward index in its place.
+ // Crucially we do NOT remove the dictionary or any dict-dependent secondary indexes — the dictionary stays
+ // and those indexes remain valid against unchanged dict ids.
+ segmentWriter.removeIndex(column, StandardIndexes.forward());
+ LoaderUtils.writeIndexToV3Format(segmentWriter, column, fwdIndexFile, StandardIndexes.forward());
+
+ Map<String, String> metadataProperties = new HashMap<>();
+ metadataProperties.put(getKeyFor(column, FORWARD_INDEX_ENCODING), FieldConfig.EncodingType.RAW.name());
+ SegmentMetadataUtils.updateMetadataProperties(_segmentDirectory, metadataProperties);
+
+ FileUtils.deleteQuietly(inProgress);
+ LOGGER.info("Converted forward index to raw (dictionary kept) for segment: {}, column: {}", segmentName, column);
+ }
+
private void rewriteDictToRawForwardIndex(ColumnMetadata columnMetadata, SegmentDirectory.Writer segmentWriter,
File indexDir)
throws Exception {
@@ -994,6 +1217,7 @@ public class ForwardIndexHandler extends BaseIndexHandler {
.withContinueOnError(_tableConfig.getIngestionConfig() != null
&& _tableConfig.getIngestionConfig().isContinueOnError());
builder.withDictionary(false);
+ // Encoding flows through ForwardIndexConfig set below.
if (!columnMetadata.getDataType().getStoredType().isFixedWidth()) {
if (columnMetadata.isSingleValue()) {
// lengthOfLongestEntry is available for dict columns from metadata.
@@ -1003,8 +1227,8 @@ public class ForwardIndexHandler extends BaseIndexHandler {
builder.withMaxRowLengthInBytes(getMaxRowLength(columnMetadata, reader, dictionary));
}
}
- IndexCreationContext context = builder.build();
ForwardIndexConfig config = _fieldIndexConfigs.get(column).getConfig(StandardIndexes.forward());
+ IndexCreationContext context = builder.build();
try (ForwardIndexCreator creator = StandardIndexes.forward().createIndexCreator(context, config)) {
forwardIndexRewriteHelper(column, columnMetadata, reader, creator, columnMetadata.getTotalDocs(), null,
dictionary);
@@ -1038,10 +1262,20 @@ public class ForwardIndexHandler extends BaseIndexHandler {
}
private AbstractColumnStatisticsCollector getStatsCollector(String column, DataType storedType) {
+ return getStatsCollector(column, storedType, false);
+ }
+
+ /// `requireUniqueValues=true` forces a per-type collector even when the no-dict optimization would apply.
+ /// Callers building a dictionary out of the raw values must opt in — `_fieldIndexConfigs` may still report
+ /// `dictionary=disabled` when the dictionary requirement was derived from a secondary-index need rather
+ /// than user config (e.g. legacy `invertedIndexColumns` triggering ENABLE_DICTIONARY in computeOperations),
+ /// in which case the optimized no-dict collector would be picked and `getUniqueValuesSet()` returns null.
+ private AbstractColumnStatisticsCollector getStatsCollector(String column, DataType storedType,
+ boolean requireUniqueValues) {
StatsCollectorConfig statsCollectorConfig = new StatsCollectorConfig(_tableConfig, _schema, null);
boolean dictionaryEnabled = hasIndex(column, StandardIndexes.dictionary());
- // MAP collector is optimised for no-dictionary collection
- if (!dictionaryEnabled && storedType != DataType.MAP) {
+ // MAP collector is optimised for no-dictionary collection.
+ if (!requireUniqueValues && !dictionaryEnabled && storedType != DataType.MAP) {
if (ClusterConfigForTable.useOptimizedNoDictCollector(_tableConfig)) {
return new NoDictColumnStatisticsCollector(column, statsCollectorConfig);
}
diff --git a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/InvertedIndexAndDictionaryBasedForwardIndexCreator.java b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/InvertedIndexAndDictionaryBasedForwardIndexCreator.java
index 0c798691069..c823b87822e 100644
--- a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/InvertedIndexAndDictionaryBasedForwardIndexCreator.java
+++ b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/InvertedIndexAndDictionaryBasedForwardIndexCreator.java
@@ -40,7 +40,6 @@ import org.apache.pinot.segment.spi.index.reader.Dictionary;
import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
import org.apache.pinot.segment.spi.store.SegmentDirectory;
import org.apache.pinot.segment.spi.utils.SegmentMetadataUtils;
-import org.apache.pinot.spi.config.table.FieldConfig;
import org.apache.pinot.spi.data.FieldSpec;
import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
import org.slf4j.Logger;
@@ -93,7 +92,18 @@ public class InvertedIndexAndDictionaryBasedForwardIndexCreator implements AutoC
private final int _maxNumberOfMultiValues;
private final FieldSpec.DataType _storedType;
private final int _totalNumberOfEntries;
- private final boolean _dictionaryEnabled;
+ /// `true` if a standalone dictionary file exists for the column. Independent of how the forward index is
+ /// encoded — both the standard "dict-encoded forward index" and the new "shared-dictionary + raw forward index"
+ /// configurations have `_dictionaryPresent == true`.
+ private final boolean _dictionaryPresent;
+ /// `true` if the forward index itself stores dict IDs (`FieldConfig.EncodingType.DICTIONARY`),
+ /// `false` if it stores raw values (`FieldConfig.EncodingType.RAW`).
+ ///
+ /// Pre-shared-dict, this field used to be a single `_dictionaryEnabled` that conflated both meanings.
+ /// After the shared-dictionary feature, the two boolean dimensions are independent: a column can have a dictionary
+ /// AND a raw forward index simultaneously. Subclasses or callers that previously expected a single combined flag
+ /// must update to the two-flag model.
+ private final boolean _dictionaryBasedForwardIndex;
private final boolean _useMMapBuffer;
// Files and temporary buffers
@@ -114,7 +124,8 @@ public class InvertedIndexAndDictionaryBasedForwardIndexCreator implements AutoC
private PinotDataBuffer _forwardIndexMaxSizeBuffer;
public InvertedIndexAndDictionaryBasedForwardIndexCreator(String columnName, SegmentDirectory segmentDirectory,
- boolean dictionaryEnabled, ForwardIndexConfig fwdConf, SegmentDirectory.Writer segmentWriter,
+ boolean dictionaryPresent, boolean dictionaryBasedForwardIndex, ForwardIndexConfig fwdConf,
+ SegmentDirectory.Writer segmentWriter,
boolean isTemporaryForwardIndex, String tableNameWithType, boolean continueOnError)
throws IOException {
_columnName = columnName;
@@ -132,7 +143,8 @@ public class InvertedIndexAndDictionaryBasedForwardIndexCreator implements AutoC
_totalNumberOfEntries = _columnMetadata.getTotalNumberOfEntries();
_maxNumberOfMultiValues = _columnMetadata.getMaxNumberOfMultiValues();
_storedType = _columnMetadata.getFieldSpec().getDataType().getStoredType();
- _dictionaryEnabled = dictionaryEnabled;
+ _dictionaryPresent = dictionaryPresent;
+ _dictionaryBasedForwardIndex = dictionaryBasedForwardIndex;
int numValues = _singleValue ? _numDocs : _totalNumberOfEntries;
_useMMapBuffer = numValues > NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER;
@@ -140,7 +152,7 @@ public class InvertedIndexAndDictionaryBasedForwardIndexCreator implements AutoC
// a no-op for sorted columns
File indexDir = _segmentMetadata.getIndexDir();
String fileExtension;
- if (_dictionaryEnabled) {
+ if (_dictionaryBasedForwardIndex) {
fileExtension = _singleValue ? V1Constants.Indexes.UNSORTED_SV_FORWARD_INDEX_FILE_EXTENSION
: V1Constants.Indexes.UNSORTED_MV_FORWARD_INDEX_FILE_EXTENSION;
} else {
@@ -230,7 +242,7 @@ public class InvertedIndexAndDictionaryBasedForwardIndexCreator implements AutoC
// Only cleanup the other indexes if the forward index to be created is permanent. If the forward index is
// temporary, it is meant to be used only for construction of other indexes and will be deleted once all the
// IndexHandlers have completed.
- if (!_dictionaryEnabled) {
+ if (!_dictionaryPresent) {
LOGGER.info("Clean up indexes no longer needed or which need to be rewritten for segment: {}, column: {}",
segmentName, _columnName);
// Delete the dictionary
@@ -271,7 +283,7 @@ public class InvertedIndexAndDictionaryBasedForwardIndexCreator implements AutoC
.withIndexDir(_segmentMetadata.getIndexDir())
.withColumnMetadata(_columnMetadata)
.withForwardIndexDisabled(false)
- .withDictionary(_dictionaryEnabled)
+ .withDictionary(_dictionaryPresent)
.withLengthOfLongestEntry(lengthOfLongestEntry)
.withTableNameWithType(_tableNameWithType)
.withContinueOnError(_continueOnError)
@@ -281,18 +293,17 @@ public class InvertedIndexAndDictionaryBasedForwardIndexCreator implements AutoC
writeToForwardIndex(dictionary, context);
// Setup and return the metadata properties to update
- if (_dictionaryEnabled) {
- return Map.of(
- getKeyFor(_columnName, FORWARD_INDEX_ENCODING), FieldConfig.EncodingType.DICTIONARY.name());
- } else {
- return Map.of(
- getKeyFor(_columnName, HAS_DICTIONARY), String.valueOf(false),
- getKeyFor(_columnName, FORWARD_INDEX_ENCODING), FieldConfig.EncodingType.RAW.name(),
- getKeyFor(_columnName, DICTIONARY_ELEMENT_SIZE), String.valueOf(0)
- // TODO: See https://github.com/apache/pinot/pull/16921 for details
- // getKeyFor(_columnName, BITS_PER_ELEMENT), String.valueOf(-1)
- );
+ if (_dictionaryPresent) {
+ return Map.of(getKeyFor(_columnName, FORWARD_INDEX_ENCODING),
+ _forwardIndexConfig.getEncodingType().name());
}
+ return Map.of(
+ getKeyFor(_columnName, HAS_DICTIONARY), String.valueOf(false),
+ getKeyFor(_columnName, FORWARD_INDEX_ENCODING), _forwardIndexConfig.getEncodingType().name(),
+ getKeyFor(_columnName, DICTIONARY_ELEMENT_SIZE), String.valueOf(0)
+ // TODO: See https://github.com/apache/pinot/pull/16921 for details
+ // getKeyFor(_columnName, BITS_PER_ELEMENT), String.valueOf(-1)
+ );
}
}
@@ -363,7 +374,7 @@ public class InvertedIndexAndDictionaryBasedForwardIndexCreator implements AutoC
.withIndexDir(_segmentMetadata.getIndexDir())
.withColumnMetadata(_columnMetadata)
.withForwardIndexDisabled(false)
- .withDictionary(_dictionaryEnabled)
+ .withDictionary(_dictionaryPresent)
.withTotalNumberOfEntries(_nextValueId)
.withMaxNumberOfMultiValueElements(maxNumberOfMultiValues[0])
.withMaxRowLengthInBytes(maxRowLengthInBytes[0])
@@ -380,8 +391,8 @@ public class InvertedIndexAndDictionaryBasedForwardIndexCreator implements AutoC
String.valueOf(maxNumberOfMultiValues[0]));
metadataProperties.put(getKeyFor(_columnName, TOTAL_NUMBER_OF_ENTRIES), String.valueOf(_nextValueId));
metadataProperties.put(getKeyFor(_columnName, FORWARD_INDEX_ENCODING),
- (_dictionaryEnabled ? FieldConfig.EncodingType.DICTIONARY : FieldConfig.EncodingType.RAW).name());
- if (!_dictionaryEnabled) {
+ _forwardIndexConfig.getEncodingType().name());
+ if (!_dictionaryPresent) {
metadataProperties.put(getKeyFor(_columnName, HAS_DICTIONARY), String.valueOf(false));
metadataProperties.put(getKeyFor(_columnName, DICTIONARY_ELEMENT_SIZE), String.valueOf(0));
// TODO: See https://github.com/apache/pinot/pull/16921 for details
@@ -401,7 +412,7 @@ public class InvertedIndexAndDictionaryBasedForwardIndexCreator implements AutoC
private void writeToForwardIndex(Dictionary dictionary, IndexCreationContext context)
throws IOException {
try (ForwardIndexCreator creator = StandardIndexes.forward().createIndexCreator(context, _forwardIndexConfig)) {
- if (_dictionaryEnabled) {
+ if (creator.isDictionaryEncoded()) {
if (_singleValue) {
for (int docId = 0; docId < _numDocs; docId++) {
creator.putDictId(getInt(_forwardIndexValueBuffer, docId));
diff --git a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/SegmentPreProcessor.java b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/SegmentPreProcessor.java
index 825175b6230..cef94ee26ba 100644
--- a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/SegmentPreProcessor.java
+++ b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/SegmentPreProcessor.java
@@ -34,7 +34,7 @@ import org.apache.pinot.segment.local.segment.index.loader.columnminmaxvalue.Col
import org.apache.pinot.segment.local.segment.index.loader.columnminmaxvalue.ColumnMinMaxValueGeneratorMode;
import org.apache.pinot.segment.local.segment.index.loader.defaultcolumn.DefaultColumnHandler;
import org.apache.pinot.segment.local.segment.index.loader.defaultcolumn.DefaultColumnHandlerFactory;
-import org.apache.pinot.segment.local.segment.index.loader.invertedindex.InvertedIndexHandler;
+import org.apache.pinot.segment.local.segment.index.loader.invertedindex.LegacyRawValueInvertedIndexCleanup;
import org.apache.pinot.segment.local.segment.index.loader.invertedindex.MultiColumnTextIndexHandler;
import org.apache.pinot.segment.local.startree.StarTreeBuilderUtils;
import org.apache.pinot.segment.local.startree.v2.builder.MultipleTreesBuilder;
@@ -113,6 +113,12 @@ public class SegmentPreProcessor implements AutoCloseable {
removeInvertedIndexTempFiles(indexDir);
try (SegmentDirectory.Writer segmentWriter = _segmentDirectory.createWriter()) {
+ // Backward-compat shim: invalidate any legacy raw-value embedded-dictionary inverted indexes left over from
+ // PR #17060 (reverted by PR #18410) so the standard handlers can rebuild them in the dict-id format. Must
+ // run before any handler that may try to read the inverted-index buffer. Safe to delete after Pinot 1.7;
+ // see [LegacyRawValueInvertedIndexCleanup] javadoc for the full sunset checklist.
+ LegacyRawValueInvertedIndexCleanup.removeLegacyRawValueInvertedIndexes(segmentWriter);
+
// Update default columns according to the schema.
DefaultColumnHandler defaultColumnHandler =
DefaultColumnHandlerFactory.getDefaultColumnHandler(indexDir, segmentMetadata, _indexLoadingConfig,
@@ -124,9 +130,14 @@ public class SegmentPreProcessor implements AutoCloseable {
List<IndexHandler> indexHandlers = new ArrayList<>();
// We cannot just create all the index handlers in a random order.
- // Specifically, ForwardIndexHandler needs to be executed first. This is because it modifies the segment metadata
- // while rewriting forward index to create a dictionary. Some other handlers (like the range one) assume that
- // metadata was already been modified by ForwardIndexHandler.
+ // Specifically, ForwardIndexHandler MUST run first. It is the only handler that:
+ // (a) creates the shared dictionary for a RAW forward index column when a secondary index requires one
+ // (ENABLE_DICTIONARY operation in ForwardIndexHandler.createDictionaryForRawForwardIndex);
+ // (b) updates the segment metadata's HAS_DICTIONARY / FORWARD_INDEX_ENCODING properties accordingly.
+ // The InvertedIndexHandler / RangeIndexHandler / FSTIndexHandler then read the freshly-reloaded metadata and
+ // build dict-id-based indexes on top of the new shared dictionary. If this order is violated, downstream
+ // handlers fail with an IllegalStateException because the dictionary they require does not yet exist.
+ // Any future change to handler scheduling MUST preserve: ForwardIndexHandler → reloadMetadata → other handlers.
IndexHandler forwardHandler = createHandler(StandardIndexes.forward());
indexHandlers.add(forwardHandler);
forwardHandler.updateIndices(segmentWriter);
diff --git a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/invertedindex/DictionaryBasedIndexBuilder.java b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/invertedindex/DictionaryBasedIndexBuilder.java
new file mode 100644
index 00000000000..cc323735f1f
--- /dev/null
+++ b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/invertedindex/DictionaryBasedIndexBuilder.java
@@ -0,0 +1,198 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader.invertedindex;
+
+import java.math.BigDecimal;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.index.creator.DictionaryBasedInvertedIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.index.reader.ForwardIndexReader;
+import org.apache.pinot.segment.spi.index.reader.ForwardIndexReaderContext;
+import org.apache.pinot.spi.utils.ByteArray;
+
+
+/// Shared helper for index handlers that need to populate a [DictionaryBasedInvertedIndexCreator]-shaped index
+/// (inverted index, range index) by reading raw values from a forward index and translating them to dict IDs via a
+/// [Dictionary].
+///
+/// [InvertedIndexHandler] and [RangeIndexHandler] both build dict-id-based indexes from raw forward
+/// data when a shared dictionary exists. The data-type dispatch and per-row read/lookup/add loops are identical
+/// between the two — only the creator argument differs. Both creator types implement
+/// [DictionaryBasedInvertedIndexCreator], so this single method handles both.
+public final class DictionaryBasedIndexBuilder {
+ private DictionaryBasedIndexBuilder() {
+ }
+
+ /// Reads raw values from `forwardIndexReader`, looks each value up in `dictionary`, and feeds the
+ /// (value, dictId) pair into `creator` for every doc in the segment.
+ ///
+ /// **MV allocation note:** the multi-value [DictionaryBasedInvertedIndexCreator#addIntMV] family takes
+ /// `(values, dictIds)` where `dictIds.length` must equal `values.length`, and the reader's no-buffer overload
+ /// returns an exact-length values array. As a result, both arrays are necessarily fresh per-row allocations under
+ /// the current SPI; eliminating either one would require a length-aware `addXxxMV(values, dictIds, length)` SPI
+ /// method.
+ ///
+ /// @throws IllegalStateException if the column data type is unsupported, or if the column is multi-value with a
+ /// data type that does not support MV (e.g. BIG_DECIMAL).
+ @SuppressWarnings("rawtypes")
+ public static void addRawValuesViaDictionary(DictionaryBasedInvertedIndexCreator creator,
+ ForwardIndexReader forwardIndexReader, ForwardIndexReaderContext readerContext, Dictionary dictionary,
+ ColumnMetadata columnMetadata, int numDocs) {
+ String columnName = columnMetadata.getColumnName();
+ boolean singleValue = columnMetadata.isSingleValue();
+ switch (columnMetadata.getDataType().getStoredType()) {
+ case INT:
+ if (singleValue) {
+ for (int i = 0; i < numDocs; i++) {
+ int value = forwardIndexReader.getInt(i, readerContext);
+ creator.addInt(value, dictionary.indexOf(value));
+ }
+ } else {
+ for (int i = 0; i < numDocs; i++) {
+ int[] values = forwardIndexReader.getIntMV(i, readerContext);
+ creator.addIntMV(values, lookupDictIds(dictionary, values));
+ }
+ }
+ return;
+ case LONG:
+ if (singleValue) {
+ for (int i = 0; i < numDocs; i++) {
+ long value = forwardIndexReader.getLong(i, readerContext);
+ creator.addLong(value, dictionary.indexOf(value));
+ }
+ } else {
+ for (int i = 0; i < numDocs; i++) {
+ long[] values = forwardIndexReader.getLongMV(i, readerContext);
+ creator.addLongMV(values, lookupDictIds(dictionary, values));
+ }
+ }
+ return;
+ case FLOAT:
+ if (singleValue) {
+ for (int i = 0; i < numDocs; i++) {
+ float value = forwardIndexReader.getFloat(i, readerContext);
+ creator.addFloat(value, dictionary.indexOf(value));
+ }
+ } else {
+ for (int i = 0; i < numDocs; i++) {
+ float[] values = forwardIndexReader.getFloatMV(i, readerContext);
+ creator.addFloatMV(values, lookupDictIds(dictionary, values));
+ }
+ }
+ return;
+ case DOUBLE:
+ if (singleValue) {
+ for (int i = 0; i < numDocs; i++) {
+ double value = forwardIndexReader.getDouble(i, readerContext);
+ creator.addDouble(value, dictionary.indexOf(value));
+ }
+ } else {
+ for (int i = 0; i < numDocs; i++) {
+ double[] values = forwardIndexReader.getDoubleMV(i, readerContext);
+ creator.addDoubleMV(values, lookupDictIds(dictionary, values));
+ }
+ }
+ return;
+ case BIG_DECIMAL:
+ if (!singleValue) {
+ throw new IllegalStateException(
+ "Dictionary-based index over raw values not supported for multi-value BIG_DECIMAL column: " + columnName);
+ }
+ for (int i = 0; i < numDocs; i++) {
+ BigDecimal value = forwardIndexReader.getBigDecimal(i, readerContext);
+ creator.add(value, dictionary.indexOf(value));
+ }
+ return;
+ case STRING:
+ if (singleValue) {
+ for (int i = 0; i < numDocs; i++) {
+ String value = forwardIndexReader.getString(i, readerContext);
+ creator.addString(value, dictionary.indexOf(value));
+ }
+ } else {
+ for (int i = 0; i < numDocs; i++) {
+ String[] values = forwardIndexReader.getStringMV(i, readerContext);
+ creator.addStringMV(values, lookupDictIds(dictionary, values));
+ }
+ }
+ return;
+ case BYTES:
+ if (singleValue) {
+ for (int i = 0; i < numDocs; i++) {
+ byte[] value = forwardIndexReader.getBytes(i, readerContext);
+ creator.addBytes(value, dictionary.indexOf(new ByteArray(value)));
+ }
+ } else {
+ for (int i = 0; i < numDocs; i++) {
+ byte[][] values = forwardIndexReader.getBytesMV(i, readerContext);
+ int[] dictIds = new int[values.length];
+ for (int j = 0; j < values.length; j++) {
+ dictIds[j] = dictionary.indexOf(new ByteArray(values[j]));
+ }
+ creator.addBytesMV(values, dictIds);
+ }
+ }
+ return;
+ default:
+ throw new IllegalStateException(
+ "Unsupported data type for dictionary-based index over raw values: " + columnMetadata.getDataType()
+ + " (column: " + columnName + ")");
+ }
+ }
+
+ private static int[] lookupDictIds(Dictionary dictionary, int[] values) {
+ int[] dictIds = new int[values.length];
+ for (int j = 0; j < values.length; j++) {
+ dictIds[j] = dictionary.indexOf(values[j]);
+ }
+ return dictIds;
+ }
+
+ private static int[] lookupDictIds(Dictionary dictionary, long[] values) {
+ int[] dictIds = new int[values.length];
+ for (int j = 0; j < values.length; j++) {
+ dictIds[j] = dictionary.indexOf(values[j]);
+ }
+ return dictIds;
+ }
+
+ private static int[] lookupDictIds(Dictionary dictionary, float[] values) {
+ int[] dictIds = new int[values.length];
+ for (int j = 0; j < values.length; j++) {
+ dictIds[j] = dictionary.indexOf(values[j]);
+ }
+ return dictIds;
+ }
+
+ private static int[] lookupDictIds(Dictionary dictionary, double[] values) {
+ int[] dictIds = new int[values.length];
+ for (int j = 0; j < values.length; j++) {
+ dictIds[j] = dictionary.indexOf(values[j]);
+ }
+ return dictIds;
+ }
+
+ private static int[] lookupDictIds(Dictionary dictionary, String[] values) {
+ int[] dictIds = new int[values.length];
+ for (int j = 0; j < values.length; j++) {
+ dictIds[j] = dictionary.indexOf(values[j]);
+ }
+ return dictIds;
+ }
+}
diff --git a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/invertedindex/InvertedIndexHandler.java b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/invertedindex/InvertedIndexHandler.java
index ff4029b52d1..967a23ba076 100644
--- a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/invertedindex/InvertedIndexHandler.java
+++ b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/invertedindex/InvertedIndexHandler.java
@@ -23,6 +23,7 @@ import java.util.HashSet;
import java.util.Map;
import java.util.Set;
import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.segment.index.dictionary.DictionaryIndexType;
import org.apache.pinot.segment.local.segment.index.loader.BaseIndexHandler;
import org.apache.pinot.segment.local.segment.index.loader.LoaderUtils;
import org.apache.pinot.segment.spi.ColumnMetadata;
@@ -31,9 +32,9 @@ import org.apache.pinot.segment.spi.creator.IndexCreationContext;
import org.apache.pinot.segment.spi.creator.SegmentVersion;
import org.apache.pinot.segment.spi.index.FieldIndexConfigs;
import org.apache.pinot.segment.spi.index.FieldIndexConfigsUtil;
-import org.apache.pinot.segment.spi.index.IndexReaderFactory;
import org.apache.pinot.segment.spi.index.StandardIndexes;
import org.apache.pinot.segment.spi.index.creator.DictionaryBasedInvertedIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
import org.apache.pinot.segment.spi.index.reader.ForwardIndexReader;
import org.apache.pinot.segment.spi.index.reader.ForwardIndexReaderContext;
import org.apache.pinot.segment.spi.store.SegmentDirectory;
@@ -44,6 +45,17 @@ import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+/// Handler that creates inverted indexes during segment preprocessing.
+///
+/// **Handler-ordering contract:** this handler requires that the column has a dictionary on disk by the time
+/// [#updateIndices] runs. [org.apache.pinot.segment.local.segment.index.loader.SegmentPreProcessor]
+/// enforces this by always running [org.apache.pinot.segment.local.segment.index.loader.ForwardIndexHandler]
+/// first and reloading segment metadata before this handler is invoked, so the new
+/// `ENABLE_DICTIONARY` → "create shared dictionary on a RAW forward column" path completes before this
+/// handler asserts `columnMetadata.hasDictionary() == true` in [#createInvertedIndexForColumn].
+///
+/// Plugins implementing custom IndexHandler scheduling must preserve this ordering or
+/// [#createInvertedIndexForColumn] will fail with an [IllegalStateException] at reload time.
@SuppressWarnings({"rawtypes", "unchecked"})
public class InvertedIndexHandler extends BaseIndexHandler {
private static final Logger LOGGER = LoggerFactory.getLogger(InvertedIndexHandler.class);
@@ -57,7 +69,8 @@ public class InvertedIndexHandler extends BaseIndexHandler {
}
@Override
- public boolean needUpdateIndices(SegmentDirectory.Reader segmentReader) {
+ public boolean needUpdateIndices(SegmentDirectory.Reader segmentReader)
+ throws Exception {
String segmentName = _segmentDirectory.getSegmentMetadata().getName();
Set<String> columnsToAddIdx = new HashSet<>(_columnsToAddIdx);
Set<String> existingColumns =
@@ -68,6 +81,12 @@ public class InvertedIndexHandler extends BaseIndexHandler {
LOGGER.info("Need to remove existing inverted index from segment: {}, column: {}", segmentName, column);
return true;
}
+ ColumnMetadata columnMetadata = _segmentDirectory.getSegmentMetadata().getColumnMetadataFor(column);
+ if (LegacyRawValueInvertedIndexCleanup.isLegacyRawValueInvertedIndexFormat(
+ segmentReader.getIndexFor(column, StandardIndexes.inverted()), columnMetadata)) {
+ LOGGER.info("Need to rebuild inverted index for segment: {}, column: {}", segmentName, column);
+ return true;
+ }
}
// Check if any new index need to be added.
for (String column : columnsToAddIdx) {
@@ -93,6 +112,14 @@ public class InvertedIndexHandler extends BaseIndexHandler {
LOGGER.info("Removing existing inverted index from segment: {}, column: {}", segmentName, column);
segmentWriter.removeIndex(column, StandardIndexes.inverted());
LOGGER.info("Removed existing inverted index from segment: {}, column: {}", segmentName, column);
+ } else {
+ ColumnMetadata columnMetadata = _segmentDirectory.getSegmentMetadata().getColumnMetadataFor(column);
+ if (LegacyRawValueInvertedIndexCleanup.isLegacyRawValueInvertedIndexFormat(
+ segmentWriter.getIndexFor(column, StandardIndexes.inverted()), columnMetadata)) {
+ LOGGER.info("Rebuilding existing inverted index for segment: {}, column: {}", segmentName, column);
+ segmentWriter.removeIndex(column, StandardIndexes.inverted());
+ columnsToAddIdx.add(column);
+ }
}
}
for (String column : columnsToAddIdx) {
@@ -143,24 +170,42 @@ public class InvertedIndexHandler extends BaseIndexHandler {
.withContinueOnError(_tableConfig.getIngestionConfig() != null
&& _tableConfig.getIngestionConfig().isContinueOnError())
.build();
+ // Raw-forward columns that require an inverted index must have a shared standalone dictionary created first by
+ // ForwardIndexHandler (which runs before this handler). If we reach this point without a dictionary, it indicates
+ // a bug in the handler pipeline — fail fast rather than leave the segment in an inconsistent state.
+ if (!columnMetadata.hasDictionary()) {
+ FileUtils.deleteQuietly(inProgress);
+ throw new IllegalStateException(
+ "Cannot create inverted index for segment: " + segmentName + ", column: " + columnName
+ + " — no dictionary present. ForwardIndexHandler must create a shared standalone dictionary before "
+ + "InvertedIndexHandler runs for columns that require a dictionary-backed inverted index.");
+ }
try (DictionaryBasedInvertedIndexCreator creator = StandardIndexes.inverted()
.createIndexCreator(context, IndexConfig.ENABLED)) {
- IndexReaderFactory<ForwardIndexReader> readerFactory = StandardIndexes.forward().getReaderFactory();
- try (ForwardIndexReader forwardIndexReader = readerFactory.createIndexReader(segmentWriter,
- _fieldIndexConfigs.get(columnMetadata.getColumnName()), columnMetadata);
+ try (
+ ForwardIndexReader forwardIndexReader = StandardIndexes.forward()
+ .getReaderFactory()
+ .createIndexReader(segmentWriter, _fieldIndexConfigs.get(columnName), columnMetadata);
ForwardIndexReaderContext readerContext = forwardIndexReader.createContext()) {
- if (columnMetadata.isSingleValue()) {
- // Single-value column.
- for (int i = 0; i < numDocs; i++) {
- creator.add(forwardIndexReader.getDictId(i, readerContext));
+ if (forwardIndexReader.isDictionaryEncoded()) {
+ if (columnMetadata.isSingleValue()) {
+ // Single-value column.
+ for (int i = 0; i < numDocs; i++) {
+ creator.add(forwardIndexReader.getDictId(i, readerContext));
+ }
+ } else {
+ // Multi-value column.
+ int[] dictIds = new int[columnMetadata.getMaxNumberOfMultiValues()];
+ for (int i = 0; i < numDocs; i++) {
+ int length = forwardIndexReader.getDictIdMV(i, dictIds, readerContext);
+ creator.add(dictIds, length);
+ }
}
} else {
- // Multi-value column.
- int[] dictIds = new int[columnMetadata.getMaxNumberOfMultiValues()];
- for (int i = 0; i < numDocs; i++) {
- int length = forwardIndexReader.getDictIdMV(i, dictIds, readerContext);
- creator.add(dictIds, length);
+ try (Dictionary dictionary = DictionaryIndexType.read(segmentWriter, columnMetadata)) {
+ DictionaryBasedIndexBuilder.addRawValuesViaDictionary(creator, forwardIndexReader, readerContext,
+ dictionary, columnMetadata, numDocs);
}
}
creator.seal();
diff --git a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/invertedindex/LegacyRawValueInvertedIndexCleanup.java b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/invertedindex/LegacyRawValueInvertedIndexCleanup.java
new file mode 100644
index 00000000000..9cdcaee2ce5
--- /dev/null
+++ b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/invertedindex/LegacyRawValueInvertedIndexCleanup.java
@@ -0,0 +1,137 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader.invertedindex;
+
+import java.io.IOException;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.index.StandardIndexes;
+import org.apache.pinot.segment.spi.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/// Backward-compatibility shim for inverted-index files written by the now-deleted
+/// `RawValueBitmapInvertedIndexCreator` (introduced in PR #17060, reverted in PR #18410).
+///
+/// <h2>What this exists for</h2>
+///
+/// PR #17060 added a creator that wrote a non-standard "embedded-dictionary" inverted-index file directly into
+/// the column's `bitmap.inv` slot. PR #18410 reverted the creator. Any segment built between those two PRs that
+/// has a RAW forward column with an inverted index still carries the legacy file on disk and must be migrated to
+/// the modern shared-dictionary + standard `BitmapInvertedIndexReader` format on the next segment load.
+///
+/// This class isolates everything needed for that migration:
+///
+/// 1. [#isLegacyRawValueInvertedIndexFormat] — detects the legacy 44-byte big-endian header by byte pattern.
+/// 2. [#removeLegacyRawValueInvertedIndexes] — runs once at the start of `SegmentPreProcessor` to drop every
+/// legacy file from the segment directory, so the standard [InvertedIndexHandler] can rebuild it (with a
+/// shared dictionary auto-created by `ForwardIndexHandler` if the column is still RAW).
+///
+/// <h2>Sunset</h2>
+///
+/// **Safe to delete after Pinot 1.7 is released.** By then, every segment in the wild that was built with the
+/// reverted creator has had multiple opportunities to be reloaded and migrated. Removing this class — along with
+/// the call site in `SegmentPreProcessor.process` — is a one-shot delete:
+///
+/// - Delete this file.
+/// - Remove the `LegacyRawValueInvertedIndexCleanup.removeLegacyRawValueInvertedIndexes(segmentWriter)` call near
+/// the top of `SegmentPreProcessor.process`.
+/// - Delete the unit tests in `InvertedIndexHandlerTest` that exercise [#isLegacyRawValueInvertedIndexFormat].
+/// - Delete `LegacyRawValueInvertedIndexMigrationIntegrationTest` and its two `legacy/legacyRawInverted_*.tar.gz`
+/// fixtures under `pinot-integration-tests/src/test/resources/legacy/`.
+public final class LegacyRawValueInvertedIndexCleanup {
+ private static final Logger LOGGER = LoggerFactory.getLogger(LegacyRawValueInvertedIndexCleanup.class);
+
+ private LegacyRawValueInvertedIndexCleanup() {
+ }
+
+ /// Drops any inverted-index file that uses the legacy raw-value embedded-dictionary format written by the
+ /// now-deleted `RawValueBitmapInvertedIndexCreator`. Must run before any other index handler so they don't try
+ /// to interpret the legacy buffer as the modern bitmap-inverted-index format.
+ ///
+ /// Two outcomes after this method runs:
+ ///
+ /// - If the user's table config still requests an inverted index on the column, [InvertedIndexHandler] (or
+ /// `ForwardIndexHandler` first, if a shared dictionary must be created) rebuilds it in the dict-id format on
+ /// the same reload pass.
+ /// - If the user has removed the inverted index from the config, the index simply stays absent — the column
+ /// still has its raw forward index and queries scan it.
+ public static void removeLegacyRawValueInvertedIndexes(SegmentDirectory.Writer segmentWriter)
+ throws IOException {
+ SegmentMetadataImpl segmentMetadata = (SegmentMetadataImpl) segmentWriter.toSegmentDirectory().getSegmentMetadata();
+ String segmentName = segmentMetadata.getName();
+ for (ColumnMetadata columnMetadata : segmentMetadata.getColumnMetadataMap().values()) {
+ String column = columnMetadata.getColumnName();
+ if (!segmentWriter.hasIndexFor(column, StandardIndexes.inverted())) {
+ continue;
+ }
+ // NOTE: getIndexFor returns a cached buffer reference; do NOT close it here, otherwise the underlying
+ // segment-writer cache entry becomes invalid for subsequent handler reads.
+ PinotDataBuffer buffer = segmentWriter.getIndexFor(column, StandardIndexes.inverted());
+ if (!isLegacyRawValueInvertedIndexFormat(buffer, columnMetadata)) {
+ continue;
+ }
+ LOGGER.info("Removing legacy raw-value inverted index for segment: {}, column: {}; standard handlers will "
+ + "rebuild a dict-id-based inverted index if the column config still requires one", segmentName, column);
+ segmentWriter.removeIndex(column, StandardIndexes.inverted());
+ }
+ }
+
+ /// Returns `true` if the inverted-index buffer uses the legacy raw-value format written by the now-deleted
+ /// `RawValueBitmapInvertedIndexCreator` (before this format was replaced by a shared standalone dictionary +
+ /// standard bitmap inverted index).
+ ///
+ /// Both the legacy file and modern bitmap inverted-index files are mapped via [PinotDataBuffer] in big-endian
+ /// order (see `FilePerIndexDirectory`), so all reads here are big-endian. The legacy format starts with a
+ /// 44-byte header:
+ ///
+ /// - offset 0: version int (always 1)
+ /// - offset 4: cardinality int
+ /// - offset 8: max-length int (0 for fixed-width data types)
+ /// - offsets 12, 20, 28, 36: 8-byte longs (dict offset, dict length, inverted-index offset, inverted-index length)
+ ///
+ /// The modern format written by
+ /// [org.apache.pinot.segment.local.segment.creator.impl.inv.BitmapInvertedIndexWriter] starts with an offset
+ /// table whose first int is the start offset of bitmap 0, equal to `(cardinality + 1) * 4` — the size of the
+ /// offset table itself. False positives against a legacy-format check are therefore effectively impossible
+ /// because:
+ ///
+ /// - For a modern file to pass `getInt(0) == 1`, the offset table size would have to be 1 byte, which is
+ /// impossible (each entry is 4 bytes and the smallest valid value with one bitmap is 8).
+ /// - The follow-up `getInt(4) == cardinality` further narrows: in the modern format that position holds the
+ /// end-offset of bitmap 0 (typically a large number unrelated to cardinality).
+ /// - The trailing offset/length range checks against `dataBuffer.size()` reject random byte patterns.
+ public static boolean isLegacyRawValueInvertedIndexFormat(PinotDataBuffer dataBuffer, ColumnMetadata columnMetadata) {
+ if (dataBuffer.size() < 44) {
+ return false;
+ }
+ if (dataBuffer.getInt(0) != 1 || dataBuffer.getInt(4) != columnMetadata.getCardinality()) {
+ return false;
+ }
+ long dictionaryOffset = dataBuffer.getLong(12);
+ long dictionaryLength = dataBuffer.getLong(20);
+ long invertedIndexOffset = dataBuffer.getLong(28);
+ long invertedIndexLength = dataBuffer.getLong(36);
+ long dataBufferSize = dataBuffer.size();
+ return dictionaryOffset >= 44 && dictionaryLength >= 0 && invertedIndexOffset >= dictionaryOffset + dictionaryLength
+ && invertedIndexLength >= 0 && invertedIndexOffset + invertedIndexLength <= dataBufferSize;
+ }
+}
diff --git a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/invertedindex/RangeIndexHandler.java b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/invertedindex/RangeIndexHandler.java
index 1af29b87067..30069a601d2 100644
--- a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/invertedindex/RangeIndexHandler.java
+++ b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/invertedindex/RangeIndexHandler.java
@@ -24,6 +24,7 @@ import java.util.HashSet;
import java.util.Map;
import java.util.Set;
import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.segment.index.dictionary.DictionaryIndexType;
import org.apache.pinot.segment.local.segment.index.loader.BaseIndexHandler;
import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
import org.apache.pinot.segment.local.segment.index.loader.LoaderUtils;
@@ -37,8 +38,10 @@ import org.apache.pinot.segment.spi.index.IndexReaderFactory;
import org.apache.pinot.segment.spi.index.RangeIndexConfig;
import org.apache.pinot.segment.spi.index.StandardIndexes;
import org.apache.pinot.segment.spi.index.creator.CombinedInvertedIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
import org.apache.pinot.segment.spi.index.reader.ForwardIndexReader;
import org.apache.pinot.segment.spi.index.reader.ForwardIndexReaderContext;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
import org.apache.pinot.segment.spi.store.SegmentDirectory;
import org.apache.pinot.spi.config.table.TableConfig;
import org.apache.pinot.spi.data.Schema;
@@ -65,7 +68,8 @@ public class RangeIndexHandler extends BaseIndexHandler {
}
@Override
- public boolean needUpdateIndices(SegmentDirectory.Reader segmentReader) {
+ public boolean needUpdateIndices(SegmentDirectory.Reader segmentReader)
+ throws Exception {
String segmentName = _segmentDirectory.getSegmentMetadata().getName();
Set<String> columnsToAddIdx = new HashSet<>(_columnsToAddIdx);
Set<String> existingColumns = segmentReader.toSegmentDirectory().getColumnsWithIndex(StandardIndexes.range());
@@ -73,11 +77,15 @@ public class RangeIndexHandler extends BaseIndexHandler {
// Check if any index updates are required.
boolean rangeIndexUpdated = false;
- // Check if any existing index need to be removed.
+ // Check if any existing index need to be removed or rebuilt due to a version change.
for (String column : existingColumns) {
if (!columnsToAddIdx.remove(column)) {
LOGGER.info("Need to remove existing range index from segment: {}, column: {}", segmentName, column);
rangeIndexUpdated = true;
+ } else if (existingRangeIndexVersionDiffers(segmentReader, column)) {
+ LOGGER.info("Need to rebuild range index for segment: {}, column: {} due to version change", segmentName,
+ column);
+ rangeIndexUpdated = true;
}
}
for (String column : columnsToAddIdx) {
@@ -96,10 +104,23 @@ public class RangeIndexHandler extends BaseIndexHandler {
return rangeIndexUpdated;
}
+ /// Returns `true` if the on-disk range index version doesn't match the configured version. Range index v1
+ /// (RangeIndexCreator) and v2 (BitSlicedRangeIndexCreator) have incompatible on-disk layouts and serve
+ /// different query semantics (v1 is non-exact, v2 is exact), so a version change requires rebuild.
+ private boolean existingRangeIndexVersionDiffers(SegmentDirectory.Reader segmentReader, String column)
+ throws Exception {
+ int configuredVersion = _fieldIndexConfigs.get(column).getConfig(StandardIndexes.range()).getVersion();
+ // The buffer is owned by SegmentDirectory; don't close it here (mmap regions are shared).
+ PinotDataBuffer rangeIndexBuffer = segmentReader.getIndexFor(column, StandardIndexes.range());
+ int onDiskVersion = rangeIndexBuffer.getInt(0);
+ return onDiskVersion != configuredVersion;
+ }
+
@Override
public void updateIndices(SegmentDirectory.Writer segmentWriter)
throws Exception {
- // Remove indices not set in table config any more
+ // Remove indices not set in table config any more, or those whose on-disk version differs from the
+ // configured version (v1↔v2 require rebuild).
String segmentName = _segmentDirectory.getSegmentMetadata().getName();
Set<String> columnsToAddIdx = new HashSet<>(_columnsToAddIdx);
Set<String> existingColumns = segmentWriter.toSegmentDirectory().getColumnsWithIndex(StandardIndexes.range());
@@ -108,6 +129,13 @@ public class RangeIndexHandler extends BaseIndexHandler {
LOGGER.info("Removing existing range index from segment: {}, column: {}", segmentName, column);
segmentWriter.removeIndex(column, StandardIndexes.range());
LOGGER.info("Removed existing range index from segment: {}, column: {}", segmentName, column);
+ } else if (existingRangeIndexVersionDiffers(segmentWriter, column)) {
+ LOGGER.info("Rebuilding range index for segment: {}, column: {} due to version change", segmentName, column);
+ segmentWriter.removeIndex(column, StandardIndexes.range());
+ ColumnMetadata columnMetadata = _segmentDirectory.getSegmentMetadata().getColumnMetadataFor(column);
+ if (columnMetadata != null && !columnMetadata.isSorted()) {
+ createRangeIndexForColumn(segmentWriter, columnMetadata);
+ }
}
}
for (String column : columnsToAddIdx) {
@@ -167,17 +195,24 @@ public class RangeIndexHandler extends BaseIndexHandler {
_fieldIndexConfigs.get(columnMetadata.getColumnName()), columnMetadata);
ForwardIndexReaderContext readerContext = forwardIndexReader.createContext();
CombinedInvertedIndexCreator rangeIndexCreator = newRangeIndexCreator(columnMetadata)) {
- if (columnMetadata.isSingleValue()) {
- // Single-value column
- for (int i = 0; i < numDocs; i++) {
- rangeIndexCreator.add(forwardIndexReader.getDictId(i, readerContext));
+ if (forwardIndexReader.isDictionaryEncoded()) {
+ if (columnMetadata.isSingleValue()) {
+ for (int i = 0; i < numDocs; i++) {
+ rangeIndexCreator.add(forwardIndexReader.getDictId(i, readerContext));
+ }
+ } else {
+ int[] dictIds = new int[columnMetadata.getMaxNumberOfMultiValues()];
+ for (int i = 0; i < numDocs; i++) {
+ int length = forwardIndexReader.getDictIdMV(i, dictIds, readerContext);
+ rangeIndexCreator.add(dictIds, length);
+ }
}
} else {
- // Multi-value column
- int[] dictIds = new int[columnMetadata.getMaxNumberOfMultiValues()];
- for (int i = 0; i < numDocs; i++) {
- int length = forwardIndexReader.getDictIdMV(i, dictIds, readerContext);
- rangeIndexCreator.add(dictIds, length);
+ // RAW forward + shared standalone dictionary: read raw values and look each up in the dictionary to feed
+ // dict IDs into the range index.
+ try (Dictionary dictionary = DictionaryIndexType.read(segmentWriter, columnMetadata)) {
+ DictionaryBasedIndexBuilder.addRawValuesViaDictionary(rangeIndexCreator, forwardIndexReader, readerContext,
+ dictionary, columnMetadata, numDocs);
}
}
rangeIndexCreator.seal();
@@ -219,7 +254,7 @@ public class RangeIndexHandler extends BaseIndexHandler {
throw new IllegalStateException("Unsupported data type: " + columnMetadata.getDataType());
}
} else {
- // Multi-value column
+ // Multi-value column.
int maxNumValuesPerMVEntry = columnMetadata.getMaxNumberOfMultiValues();
switch (columnMetadata.getDataType().getStoredType()) {
case INT:
diff --git a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/range/RangeIndexType.java b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/range/RangeIndexType.java
index fad0bbdcc72..dd072a94f1d 100644
--- a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/range/RangeIndexType.java
+++ b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/range/RangeIndexType.java
@@ -85,6 +85,18 @@ public class RangeIndexType
storedType.isNumeric() || indexConfigs.getConfig(StandardIndexes.dictionary()).isEnabled(),
"Cannot create range index on non-numeric column: %s without dictionary", column);
Preconditions.checkState(storedType != DataType.MAP, "Cannot create range index on MAP column: %s", column);
+ // Legacy range index (version 1) over dict IDs is non-exact: query-time partial matches fall back to
+ // ScanBasedFilterOperator, which on a RAW forward column would have to apply a dict-based evaluator
+ // against raw values — silently producing wrong results. Reject this combination upfront and require
+ // the BitSliced range index (version 2) for shared-dict + RAW forward columns.
+ boolean rawForward = indexConfigs.getConfig(StandardIndexes.forward()).getEncodingType()
+ == FieldConfig.EncodingType.RAW;
+ boolean dictionaryEnabled = indexConfigs.getConfig(StandardIndexes.dictionary()).isEnabled();
+ Preconditions.checkState(
+ !(rawForward && dictionaryEnabled && rangeIndexConfig.getVersion() == RangeIndexCreator.VERSION),
+ "Cannot create range index version %s on column: %s with RAW forward index and dictionary; "
+ + "use BitSliced range index version %s instead",
+ RangeIndexCreator.VERSION, column, BitSlicedRangeIndexCreator.VERSION);
}
}
diff --git a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/readers/PinotSegmentColumnReader.java b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/readers/PinotSegmentColumnReader.java
index c0f4e165185..e5ac0e94f75 100644
--- a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/readers/PinotSegmentColumnReader.java
+++ b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/readers/PinotSegmentColumnReader.java
@@ -96,8 +96,8 @@ public class PinotSegmentColumnReader implements Closeable {
}
public Object getValue(int docId) {
- if (_dictionary != null) {
- // Dictionary based
+ if (_forwardIndexReader.isDictionaryEncoded()) {
+ // Dictionary-encoded forward index
if (_forwardIndexReader.isSingleValue()) {
return _dictionary.get(_forwardIndexReader.getDictId(docId, _forwardIndexReaderContext));
} else {
@@ -195,7 +195,7 @@ public class PinotSegmentColumnReader implements Closeable {
// Single-value accessors
public int getInt(int docId) {
- if (_dictionary != null) {
+ if (_forwardIndexReader.isDictionaryEncoded()) {
return _dictionary.getIntValue(_forwardIndexReader.getDictId(docId, _forwardIndexReaderContext));
} else {
return _forwardIndexReader.getInt(docId, _forwardIndexReaderContext);
@@ -203,7 +203,7 @@ public class PinotSegmentColumnReader implements Closeable {
}
public long getLong(int docId) {
- if (_dictionary != null) {
+ if (_forwardIndexReader.isDictionaryEncoded()) {
return _dictionary.getLongValue(_forwardIndexReader.getDictId(docId, _forwardIndexReaderContext));
} else {
return _forwardIndexReader.getLong(docId, _forwardIndexReaderContext);
@@ -211,7 +211,7 @@ public class PinotSegmentColumnReader implements Closeable {
}
public float getFloat(int docId) {
- if (_dictionary != null) {
+ if (_forwardIndexReader.isDictionaryEncoded()) {
return _dictionary.getFloatValue(_forwardIndexReader.getDictId(docId, _forwardIndexReaderContext));
} else {
return _forwardIndexReader.getFloat(docId, _forwardIndexReaderContext);
@@ -219,7 +219,7 @@ public class PinotSegmentColumnReader implements Closeable {
}
public double getDouble(int docId) {
- if (_dictionary != null) {
+ if (_forwardIndexReader.isDictionaryEncoded()) {
return _dictionary.getDoubleValue(_forwardIndexReader.getDictId(docId, _forwardIndexReaderContext));
} else {
return _forwardIndexReader.getDouble(docId, _forwardIndexReaderContext);
@@ -235,7 +235,7 @@ public class PinotSegmentColumnReader implements Closeable {
}
public String getString(int docId) {
- if (_dictionary != null) {
+ if (_forwardIndexReader.isDictionaryEncoded()) {
return _dictionary.getStringValue(_forwardIndexReader.getDictId(docId, _forwardIndexReaderContext));
} else {
return _forwardIndexReader.getString(docId, _forwardIndexReaderContext);
@@ -243,7 +243,7 @@ public class PinotSegmentColumnReader implements Closeable {
}
public byte[] getBytes(int docId) {
- if (_dictionary != null) {
+ if (_forwardIndexReader.isDictionaryEncoded()) {
return _dictionary.getBytesValue(_forwardIndexReader.getDictId(docId, _forwardIndexReaderContext));
} else {
return _forwardIndexReader.getBytes(docId, _forwardIndexReaderContext);
@@ -253,7 +253,7 @@ public class PinotSegmentColumnReader implements Closeable {
// Multi-value accessors
public int[] getIntMV(int docId) {
- if (_dictionary != null) {
+ if (_forwardIndexReader.isDictionaryEncoded()) {
int numValues = _forwardIndexReader.getDictIdMV(docId, _dictIdBuffer, _forwardIndexReaderContext);
int[] values = new int[numValues];
_dictionary.readIntValues(_dictIdBuffer, numValues, values);
@@ -264,7 +264,7 @@ public class PinotSegmentColumnReader implements Closeable {
}
public long[] getLongMV(int docId) {
- if (_dictionary != null) {
+ if (_forwardIndexReader.isDictionaryEncoded()) {
int numValues = _forwardIndexReader.getDictIdMV(docId, _dictIdBuffer, _forwardIndexReaderContext);
long[] values = new long[numValues];
_dictionary.readLongValues(_dictIdBuffer, numValues, values);
@@ -275,7 +275,7 @@ public class PinotSegmentColumnReader implements Closeable {
}
public float[] getFloatMV(int docId) {
- if (_dictionary != null) {
+ if (_forwardIndexReader.isDictionaryEncoded()) {
int numValues = _forwardIndexReader.getDictIdMV(docId, _dictIdBuffer, _forwardIndexReaderContext);
float[] values = new float[numValues];
_dictionary.readFloatValues(_dictIdBuffer, numValues, values);
@@ -286,7 +286,7 @@ public class PinotSegmentColumnReader implements Closeable {
}
public double[] getDoubleMV(int docId) {
- if (_dictionary != null) {
+ if (_forwardIndexReader.isDictionaryEncoded()) {
int numValues = _forwardIndexReader.getDictIdMV(docId, _dictIdBuffer, _forwardIndexReaderContext);
double[] values = new double[numValues];
_dictionary.readDoubleValues(_dictIdBuffer, numValues, values);
@@ -308,7 +308,7 @@ public class PinotSegmentColumnReader implements Closeable {
}
public String[] getStringMV(int docId) {
- if (_dictionary != null) {
+ if (_forwardIndexReader.isDictionaryEncoded()) {
int numValues = _forwardIndexReader.getDictIdMV(docId, _dictIdBuffer, _forwardIndexReaderContext);
String[] values = new String[numValues];
_dictionary.readStringValues(_dictIdBuffer, numValues, values);
@@ -319,7 +319,7 @@ public class PinotSegmentColumnReader implements Closeable {
}
public byte[][] getBytesMV(int docId) {
- if (_dictionary != null) {
+ if (_forwardIndexReader.isDictionaryEncoded()) {
int numValues = _forwardIndexReader.getDictIdMV(docId, _dictIdBuffer, _forwardIndexReaderContext);
byte[][] values = new byte[numValues][];
_dictionary.readBytesValues(_dictIdBuffer, numValues, values);
diff --git a/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/dictionary/DictionaryIndexTypeTest.java b/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/dictionary/DictionaryIndexTypeTest.java
index b8c4c8b0c68..bdc7b813752 100644
--- a/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/dictionary/DictionaryIndexTypeTest.java
+++ b/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/dictionary/DictionaryIndexTypeTest.java
@@ -535,4 +535,33 @@ public class DictionaryIndexTypeTest {
fieldIndexConfigs = new FieldIndexConfigs.Builder().add(StandardIndexes.inverted(), indexConfig).build();
assertTrue(DictionaryIndexType.ignoreDictionaryOverride(true, true, 5, null, metric, fieldIndexConfigs, 5, 20));
}
+
+ @Test
+ public void testIsDictionaryRequiredDetectsInvertedIndex() {
+ FieldSpec fieldSpec = new DimensionFieldSpec("rawCol", FieldSpec.DataType.STRING, true);
+ FieldIndexConfigs withInverted = new FieldIndexConfigs.Builder()
+ .add(StandardIndexes.inverted(), IndexConfig.ENABLED)
+ .build();
+ assertTrue(DictionaryIndexConfig.requiresDictionary(fieldSpec, withInverted),
+ "Inverted index should require dictionary");
+
+ FieldIndexConfigs withoutInverted = new FieldIndexConfigs.Builder().build();
+ assertFalse(DictionaryIndexConfig.requiresDictionary(fieldSpec, withoutInverted),
+ "No dictionary-requiring index configured");
+ }
+
+ @Test
+ public void testGetIndexTypesWithDictionaryRequiredReturnsFstAndInverted() {
+ FieldSpec fieldSpec = new DimensionFieldSpec("rawCol", FieldSpec.DataType.STRING, true);
+ FieldIndexConfigs withBoth = new FieldIndexConfigs.Builder()
+ .add(StandardIndexes.inverted(), IndexConfig.ENABLED)
+ .add(StandardIndexes.fst(), new org.apache.pinot.segment.spi.index.FstIndexConfig())
+ .build();
+ java.util.List<IndexType<?, ?, ?>> required =
+ DictionaryIndexConfig.getIndexTypesWithDictionaryRequired(fieldSpec, withBoth);
+ assertTrue(required.stream().anyMatch(t -> t.getId().equals(StandardIndexes.INVERTED_ID)),
+ "Inverted index should be in the required list");
+ assertTrue(required.stream().anyMatch(t -> t.getId().equals(StandardIndexes.FST_ID)),
+ "FST index should be in the required list");
+ }
}
diff --git a/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandlerTest.java b/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandlerTest.java
index 73a84049fd0..a5dc39d6f99 100644
--- a/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandlerTest.java
+++ b/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandlerTest.java
@@ -40,7 +40,9 @@ import org.apache.commons.lang3.StringUtils;
import org.apache.pinot.segment.local.io.util.PinotDataBitSet;
import org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl;
import org.apache.pinot.segment.local.segment.index.dictionary.DictionaryIndexType;
+import org.apache.pinot.segment.local.segment.index.loader.invertedindex.InvertedIndexHandler;
import org.apache.pinot.segment.local.segment.index.loader.invertedindex.RangeIndexHandler;
+import org.apache.pinot.segment.local.segment.index.readers.BitmapInvertedIndexReader;
import org.apache.pinot.segment.local.segment.readers.GenericRowRecordReader;
import org.apache.pinot.segment.local.segment.readers.PinotSegmentColumnReader;
import org.apache.pinot.segment.local.segment.store.SegmentLocalFSDirectory;
@@ -59,6 +61,7 @@ import org.apache.pinot.segment.spi.index.StandardIndexes;
import org.apache.pinot.segment.spi.index.metadata.SegmentMetadataImpl;
import org.apache.pinot.segment.spi.index.reader.Dictionary;
import org.apache.pinot.segment.spi.index.reader.ForwardIndexReader;
+import org.apache.pinot.segment.spi.index.reader.InvertedIndexReader;
import org.apache.pinot.segment.spi.store.SegmentDirectory;
import org.apache.pinot.spi.config.table.FieldConfig;
import org.apache.pinot.spi.config.table.FieldConfig.CompressionCodec;
@@ -649,7 +652,8 @@ public class ForwardIndexHandlerTest {
assertEquals(computeOperations(),
Map.of(METRIC_LZ4_INTEGER, List.of(ForwardIndexHandler.Operation.ENABLE_DICTIONARY)));
- // TEST5: Enable Dictionary for sorted column.
+ // TEST5: Sorted column currently configured as RAW. Removing RAW config means the new config expects a
+ // dictionary, so ForwardIndexHandler should detect ENABLE_DICTIONARY.
resetIndexConfigs();
_noDictionaryColumns.remove(DIM_RAW_SORTED_INTEGER);
_fieldConfigMap.remove(DIM_RAW_SORTED_INTEGER);
@@ -677,11 +681,15 @@ public class ForwardIndexHandlerTest {
assertEquals(computeOperations(),
Map.of(DIM_DICT_MV_BYTES, List.of(ForwardIndexHandler.Operation.DISABLE_DICTIONARY)));
- // TEST3: Disable dictionary and enable inverted index. Should be a no-op.
+ // TEST3: Disable dictionary and enable inverted index on a previously dict-encoded column. The inverted
+ // index requires a dictionary, so the auto-keep-dictionary-when-required-by-index logic keeps the dict.
+ // The forward index encoding still has to flip from DICT to RAW per the user's noDictionaryColumns
+ // request, so ENABLE_RAW_FORWARD_INDEX is queued (the encoding-flip handler keeps the dictionary).
resetIndexConfigs();
_noDictionaryColumns.add(DIM_DICT_STRING);
_invertedIndexColumns.add(DIM_DICT_STRING);
- assertTrue(computeOperations().isEmpty());
+ assertEquals(computeOperations(),
+ Map.of(DIM_DICT_STRING, List.of(ForwardIndexHandler.Operation.ENABLE_RAW_FORWARD_INDEX)));
}
}
@@ -874,8 +882,10 @@ public class ForwardIndexHandlerTest {
+ "testSegment or refresh / back-fill the forward index");
}
- // TEST13: Disable dictionary on a column that already has forward index disabled and inverted index enabled with
- // a range index
+ // TEST13: Disable dictionary on a column that already has forward index disabled and inverted index
+ // enabled with a range index. The new config still has inverted index enabled, which requires a
+ // dictionary — so the auto-keep-dictionary-when-required-by-index logic prevents removing the dict at
+ // all. The range Preconditions never fires because no DISABLE_DICTIONARY operation is ever queued.
resetIndexConfigs();
_noDictionaryColumns.add(DIM_SV_FORWARD_INDEX_DISABLED_INTEGER);
_rangeIndexColumns.add(DIM_SV_FORWARD_INDEX_DISABLED_INTEGER);
@@ -883,15 +893,10 @@ public class ForwardIndexHandlerTest {
new FieldConfig(DIM_SV_FORWARD_INDEX_DISABLED_INTEGER, FieldConfig.EncodingType.RAW,
List.of(FieldConfig.IndexType.INVERTED, FieldConfig.IndexType.RANGE), CompressionCodec.LZ4,
Map.of(FieldConfig.FORWARD_INDEX_DISABLED, "true")));
- try {
- computeOperations();
- fail("Disabling dictionary on forward index disabled column with inverted index and a range index "
- + "is not possible");
- } catch (IllegalStateException e) {
- assertEquals(e.getMessage(), "Must disable range index (enabled) to disable the dictionary for a "
- + "forwardIndexDisabled column: DIM_SV_FORWARD_INDEX_DISABLED_INTEGER of segment: testSegment or refresh "
- + "/ back-fill the forward index");
- }
+ Map<String, List<ForwardIndexHandler.Operation>> ops = computeOperations();
+ assertFalse(ops.containsKey(DIM_SV_FORWARD_INDEX_DISABLED_INTEGER),
+ "Dictionary must stay because the new config still requires it for the inverted index; "
+ + "no operation should be queued.");
}
}
@@ -912,13 +917,15 @@ public class ForwardIndexHandlerTest {
_invertedIndexColumns.remove(column);
_fieldConfigMap.put(column,
new FieldConfig(column, FieldConfig.EncodingType.RAW, List.of(), CompressionCodec.LZ4, null));
- assertEquals(computeOperations(), Map.of(column, List.of(ForwardIndexHandler.Operation.ENABLE_FORWARD_INDEX)));
+ assertEquals(computeOperations(),
+ Map.of(column, List.of(ForwardIndexHandler.Operation.ENABLE_RAW_FORWARD_INDEX)));
// TEST2: Enable forward index in dictionary format for a column with forward index disabled
resetIndexConfigs();
_fieldConfigMap.remove(DIM_SV_FORWARD_INDEX_DISABLED_BYTES);
assertEquals(computeOperations(),
- Map.of(DIM_SV_FORWARD_INDEX_DISABLED_BYTES, List.of(ForwardIndexHandler.Operation.ENABLE_FORWARD_INDEX)));
+ Map.of(DIM_SV_FORWARD_INDEX_DISABLED_BYTES,
+ List.of(ForwardIndexHandler.Operation.ENABLE_DICT_FORWARD_INDEX)));
// TEST3: Enable forward index in raw format for a column with forward index disabled. Remove column from inverted
// index as well (inverted index needs dictionary)
@@ -929,7 +936,8 @@ public class ForwardIndexHandlerTest {
new FieldConfig(DIM_MV_FORWARD_INDEX_DISABLED_INTEGER, FieldConfig.EncodingType.RAW, List.of(),
CompressionCodec.LZ4, null));
assertEquals(computeOperations(),
- Map.of(DIM_MV_FORWARD_INDEX_DISABLED_INTEGER, List.of(ForwardIndexHandler.Operation.ENABLE_FORWARD_INDEX)));
+ Map.of(DIM_MV_FORWARD_INDEX_DISABLED_INTEGER,
+ List.of(ForwardIndexHandler.Operation.ENABLE_RAW_FORWARD_INDEX)));
// TEST4: Enable forward index in dictionary format for two columns with forward index disabled. Disable inverted
// index for one of them
@@ -938,8 +946,8 @@ public class ForwardIndexHandlerTest {
_fieldConfigMap.remove(DIM_SV_FORWARD_INDEX_DISABLED_LONG);
_fieldConfigMap.remove(DIM_MV_FORWARD_INDEX_DISABLED_STRING);
assertEquals(computeOperations(),
- Map.of(DIM_SV_FORWARD_INDEX_DISABLED_LONG, List.of(ForwardIndexHandler.Operation.ENABLE_FORWARD_INDEX),
- DIM_MV_FORWARD_INDEX_DISABLED_STRING, List.of(ForwardIndexHandler.Operation.ENABLE_FORWARD_INDEX)));
+ Map.of(DIM_SV_FORWARD_INDEX_DISABLED_LONG, List.of(ForwardIndexHandler.Operation.ENABLE_DICT_FORWARD_INDEX),
+ DIM_MV_FORWARD_INDEX_DISABLED_STRING, List.of(ForwardIndexHandler.Operation.ENABLE_DICT_FORWARD_INDEX)));
// TEST5: Enable forward index in raw format for two columns with forward index disabled. Remove column from
// inverted index as well (inverted index needs dictionary)
@@ -955,8 +963,8 @@ public class ForwardIndexHandlerTest {
new FieldConfig(DIM_MV_FORWARD_INDEX_DISABLED_LONG, FieldConfig.EncodingType.RAW, List.of(),
CompressionCodec.LZ4, null));
assertEquals(computeOperations(),
- Map.of(DIM_SV_FORWARD_INDEX_DISABLED_STRING, List.of(ForwardIndexHandler.Operation.ENABLE_FORWARD_INDEX),
- DIM_MV_FORWARD_INDEX_DISABLED_LONG, List.of(ForwardIndexHandler.Operation.ENABLE_FORWARD_INDEX)));
+ Map.of(DIM_SV_FORWARD_INDEX_DISABLED_STRING, List.of(ForwardIndexHandler.Operation.ENABLE_RAW_FORWARD_INDEX),
+ DIM_MV_FORWARD_INDEX_DISABLED_LONG, List.of(ForwardIndexHandler.Operation.ENABLE_RAW_FORWARD_INDEX)));
// TEST6: Enable forward index in dictionary format and one in raw format for columns with forward index disabled
resetIndexConfigs();
@@ -967,8 +975,8 @@ public class ForwardIndexHandlerTest {
CompressionCodec.LZ4, null));
_fieldConfigMap.remove(DIM_SV_FORWARD_INDEX_DISABLED_BYTES);
assertEquals(computeOperations(),
- Map.of(DIM_MV_FORWARD_INDEX_DISABLED_LONG, List.of(ForwardIndexHandler.Operation.ENABLE_FORWARD_INDEX),
- DIM_SV_FORWARD_INDEX_DISABLED_BYTES, List.of(ForwardIndexHandler.Operation.ENABLE_FORWARD_INDEX)));
+ Map.of(DIM_MV_FORWARD_INDEX_DISABLED_LONG, List.of(ForwardIndexHandler.Operation.ENABLE_RAW_FORWARD_INDEX),
+ DIM_SV_FORWARD_INDEX_DISABLED_BYTES, List.of(ForwardIndexHandler.Operation.ENABLE_DICT_FORWARD_INDEX)));
// TEST7: Enable forward index for a raw column with forward index disabled and keep it as raw
resetIndexConfigs();
@@ -988,6 +996,9 @@ public class ForwardIndexHandlerTest {
public void testChangeCompressionForSingleColumn()
throws Exception {
for (String column : RAW_COLUMNS_WITH_FORWARD_INDEX) {
+ if (RAW_SORTED_COLUMNS.contains(column)) {
+ continue;
+ }
// For every noDictionaryColumn, change the compressionType to all available types, one by one.
for (CompressionCodec compressionType : RAW_COMPRESSION_TYPES) {
SegmentMetadataImpl existingSegmentMetadata;
@@ -1027,9 +1038,8 @@ public class ForwardIndexHandlerTest {
@Test
public void testChangeCompressionAndIndexVersion()
throws Exception {
- List<String> columns = new ArrayList<>(RAW_SNAPPY_COLUMNS.size() + RAW_SORTED_COLUMNS.size());
+ List<String> columns = new ArrayList<>(RAW_SNAPPY_COLUMNS.size());
columns.addAll(RAW_SNAPPY_COLUMNS);
- columns.addAll(RAW_SORTED_COLUMNS);
for (String column : columns) {
// Convert from SNAPPY v2 to LZ4 v4
SegmentMetadataImpl existingSegmentMetadata;
@@ -1195,6 +1205,9 @@ public class ForwardIndexHandlerTest {
public void testEnableDictionaryForSingleColumn()
throws Exception {
for (String column : RAW_COLUMNS_WITH_FORWARD_INDEX) {
+ if (RAW_SORTED_COLUMNS.contains(column)) {
+ continue;
+ }
SegmentMetadataImpl existingSegmentMetadata;
try (SegmentDirectory segmentDirectory = new SegmentLocalFSDirectory(INDEX_DIR, ReadMode.mmap);
SegmentDirectory.Writer writer = segmentDirectory.createWriter()) {
@@ -1203,7 +1216,14 @@ public class ForwardIndexHandlerTest {
existingSegmentMetadata = segmentDirectory.getSegmentMetadata();
_noDictionaryColumns.remove(column);
- _fieldConfigMap.remove(column);
+ FieldConfig existingFieldConfig = _fieldConfigMap.get(column);
+ // Explicitly add dictionary config since RAW encoding alone disables dictionary
+ ObjectNode dictIndexes = JsonUtils.newObjectNode();
+ dictIndexes.set("dictionary", JsonUtils.newObjectNode());
+ _fieldConfigMap.put(column,
+ new FieldConfig(column, FieldConfig.EncodingType.RAW, null,
+ List.of(FieldConfig.IndexType.INVERTED),
+ existingFieldConfig.getCompressionCodec(), null, dictIndexes, null, null));
updateIndices();
// Tear down before validation. Because columns.psf and index map cleanup happens at segmentDirectory.close()
}
@@ -1212,18 +1232,349 @@ public class ForwardIndexHandlerTest {
testIndexExists(column, StandardIndexes.forward());
testIndexExists(column, StandardIndexes.dictionary());
validateIndexMap(column, true, false);
- validateForwardIndex(column, null, metadata.isSorted());
+ if (metadata.isSorted()) {
+ validateForwardIndex(column, null, true);
+ } else {
+ validateForwardIndex(column, getExpectedRawCompressionCodec(column), false, true);
+ }
// In column metadata, nothing other than hasDictionary and bitsPerElement should change.
validateMetadataProperties(new SegmentMetadataImpl(INDEX_DIR).getColumnMetadataFor(column), metadata, true);
}
}
+ @Test
+ public void testEnableDictionaryAndInvertedIndexKeepsRawForward()
+ throws Exception {
+ String column = DIM_ZSTANDARD_STRING;
+
+ try (SegmentDirectory segmentDirectory = new SegmentLocalFSDirectory(INDEX_DIR, ReadMode.mmap);
+ SegmentDirectory.Writer writer = segmentDirectory.createWriter()) {
+ _segmentDirectory = segmentDirectory;
+ _writer = writer;
+
+ _invertedIndexColumns.add(column);
+ // Explicitly enable dictionary for this RAW column while keeping RAW forward encoding.
+ // Must remove from noDictionaryColumns to avoid ConfigDeclaredTwiceException.
+ _noDictionaryColumns.remove(column);
+ ObjectNode rawDictIndexes = JsonUtils.newObjectNode();
+ rawDictIndexes.set("dictionary", JsonUtils.newObjectNode());
+ FieldConfig existingConfig = _fieldConfigMap.get(column);
+ _fieldConfigMap.put(column,
+ new FieldConfig(column, FieldConfig.EncodingType.RAW, null,
+ List.of(FieldConfig.IndexType.INVERTED),
+ existingConfig != null ? existingConfig.getCompressionCodec() : null,
+ null, rawDictIndexes, null, null));
+
+ ForwardIndexHandler forwardIndexHandler = createForwardIndexHandler();
+ assertEquals(forwardIndexHandler.computeOperations(writer),
+ Map.of(column, List.of(ForwardIndexHandler.Operation.ENABLE_DICTIONARY)));
+ assertTrue(forwardIndexHandler.needUpdateIndices(writer));
+ forwardIndexHandler.updateIndices(writer);
+ forwardIndexHandler.postUpdateIndicesCleanup(writer);
+
+ InvertedIndexHandler invertedIndexHandler = new InvertedIndexHandler(segmentDirectory,
+ createIndexLoadingConfig().getFieldIndexConfigByColName(), createTableConfig(), SCHEMA);
+ assertTrue(invertedIndexHandler.needUpdateIndices(writer));
+ invertedIndexHandler.updateIndices(writer);
+ invertedIndexHandler.postUpdateIndicesCleanup(writer);
+ }
+
+ try (SegmentDirectory segmentDirectory = new SegmentLocalFSDirectory(INDEX_DIR, ReadMode.mmap);
+ SegmentDirectory.Reader reader = segmentDirectory.createReader()) {
+ ColumnMetadata columnMetadata = segmentDirectory.getSegmentMetadata().getColumnMetadataFor(column);
+ assertTrue(columnMetadata.hasDictionary());
+ assertEquals(columnMetadata.getForwardIndexEncoding(), FieldConfig.EncodingType.RAW);
+ assertTrue(reader.hasIndexFor(column, StandardIndexes.dictionary()));
+ assertTrue(reader.hasIndexFor(column, StandardIndexes.inverted()));
+
+ FieldIndexConfigs fieldIndexConfigs = createFieldIndexConfigsFromMetadata(columnMetadata);
+ IndexReaderFactory<ForwardIndexReader> forwardReaderFactory = StandardIndexes.forward().getReaderFactory();
+ try (ForwardIndexReader<?> forwardIndexReader =
+ forwardReaderFactory.createIndexReader(reader, fieldIndexConfigs, columnMetadata)) {
+ assertFalse(forwardIndexReader.isDictionaryEncoded());
+ assertEquals(forwardIndexReader.getCompressionType(), ChunkCompressionType.ZSTANDARD);
+ }
+
+ try (Dictionary dictionary = DictionaryIndexType.read(reader, columnMetadata)) {
+ assertEquals(dictionary.length(), columnMetadata.getCardinality());
+ }
+
+ IndexReaderFactory<InvertedIndexReader> invertedReaderFactory = StandardIndexes.inverted().getReaderFactory();
+ try (InvertedIndexReader<?> invertedIndexReader =
+ invertedReaderFactory.createIndexReader(reader, fieldIndexConfigs, columnMetadata)) {
+ assertTrue(invertedIndexReader instanceof BitmapInvertedIndexReader);
+ }
+ }
+ }
+
+ /**
+ * End-to-end "enable dictionary on a RAW column with a range index" scenario.
+ *
+ * <p>{@link org.apache.pinot.segment.local.segment.index.range.RangeIndexType#requiresDictionary} returns
+ * {@code true}: a range index is always built over dictionary IDs. A user who wants a range index on a RAW
+ * forward column must opt in to a shared standalone dictionary by adding {@code indexes.dictionary: {}} to the
+ * column's {@code FieldConfig}; on segment reload, {@code ForwardIndexHandler.ENABLE_DICTIONARY} materializes
+ * the dictionary on disk and {@link RangeIndexHandler} then builds the range index over those dict IDs.
+ *
+ * <p>This test pins down the full handler chain: starting from a RAW column with no dictionary and no range
+ * index, after the config change and one reload pass the column ends up with {@code hasDictionary=true} (forward
+ * index still RAW-encoded), a dictionary file on disk, and a freshly built dict-id-based range index. If a
+ * future change re-orders handlers, drops range from {@code DICTIONARY_BASED_INDEXES_TO_REWRITE}, or breaks the
+ * RAW-with-shared-dict path in either handler, this test catches the regression.
+ */
+ @Test
+ public void testEnableDictionaryAndRangeOnRawForwardColumn()
+ throws Exception {
+ String column = METRIC_LZ4_INTEGER;
+ SegmentMetadataImpl existingSegmentMetadata;
+
+ // Pre-condition: column starts as RAW (no dictionary, no range index). A range index on a no-dictionary column
+ // is no longer a valid configuration; a user reaching this state migrates by adding both the explicit
+ // dictionary and the range index in the same reload.
+ try (SegmentDirectory segmentDirectory = new SegmentLocalFSDirectory(INDEX_DIR, ReadMode.mmap);
+ SegmentDirectory.Reader reader = segmentDirectory.createReader()) {
+ existingSegmentMetadata = segmentDirectory.getSegmentMetadata();
+ ColumnMetadata columnMetadata = existingSegmentMetadata.getColumnMetadataFor(column);
+ assertFalse(columnMetadata.hasDictionary(),
+ "Pre-condition: column should start without a dictionary");
+ assertFalse(reader.hasIndexFor(column, StandardIndexes.range()),
+ "Pre-condition: column should start without a range index");
+ }
+
+ // Update config: opt-in to a shared dictionary (RAW + indexes.dictionary: {}) and add a range index.
+ _noDictionaryColumns.remove(column);
+ _rangeIndexColumns.add(column);
+ FieldConfig existingFieldConfig = _fieldConfigMap.get(column);
+ ObjectNode dictIndexes = JsonUtils.newObjectNode();
+ dictIndexes.set("dictionary", JsonUtils.newObjectNode());
+ _fieldConfigMap.put(column,
+ new FieldConfig(column, FieldConfig.EncodingType.RAW, null, null,
+ existingFieldConfig != null ? existingFieldConfig.getCompressionCodec() : null, null, dictIndexes,
+ null, null));
+
+ // Step 1: ForwardIndexHandler emits ENABLE_DICTIONARY, materializing a standalone dictionary on the RAW column.
+ try (SegmentDirectory segmentDirectory = new SegmentLocalFSDirectory(INDEX_DIR, ReadMode.mmap);
+ SegmentDirectory.Writer writer = segmentDirectory.createWriter()) {
+ _segmentDirectory = segmentDirectory;
+ _writer = writer;
+
+ ForwardIndexHandler forwardIndexHandler = createForwardIndexHandler();
+ assertEquals(forwardIndexHandler.computeOperations(writer),
+ Map.of(column, List.of(ForwardIndexHandler.Operation.ENABLE_DICTIONARY)));
+ forwardIndexHandler.updateIndices(writer);
+ forwardIndexHandler.postUpdateIndicesCleanup(writer);
+ }
+
+ // Verify: dictionary on disk, forward index still RAW-encoded, no range index yet.
+ try (SegmentDirectory segmentDirectory = new SegmentLocalFSDirectory(INDEX_DIR, ReadMode.mmap);
+ SegmentDirectory.Reader reader = segmentDirectory.createReader()) {
+ ColumnMetadata columnMetadata = segmentDirectory.getSegmentMetadata().getColumnMetadataFor(column);
+ assertTrue(columnMetadata.hasDictionary(), "Dictionary should be enabled after ENABLE_DICTIONARY");
+ assertEquals(columnMetadata.getForwardIndexEncoding(), FieldConfig.EncodingType.RAW,
+ "Forward index should remain RAW-encoded");
+ assertTrue(reader.hasIndexFor(column, StandardIndexes.dictionary()));
+ assertFalse(reader.hasIndexFor(column, StandardIndexes.range()),
+ "RangeIndexHandler has not run yet, so the range index should not exist");
+ }
+
+ // Step 2: RangeIndexHandler now sees columnMetadata.hasDictionary() == true and builds the range index over
+ // dict IDs (reading raw forward values and looking each up in the shared dictionary).
+ try (SegmentDirectory segmentDirectory = new SegmentLocalFSDirectory(INDEX_DIR, ReadMode.mmap);
+ SegmentDirectory.Writer writer = segmentDirectory.createWriter()) {
+ _segmentDirectory = segmentDirectory;
+ _writer = writer;
+
+ RangeIndexHandler rangeIndexHandler = new RangeIndexHandler(segmentDirectory, createIndexLoadingConfig());
+ assertTrue(rangeIndexHandler.needUpdateIndices(writer),
+ "RangeIndexHandler should detect the new range index in the config");
+ rangeIndexHandler.updateIndices(writer);
+ rangeIndexHandler.postUpdateIndicesCleanup(writer);
+ }
+
+ // Final state: dict + range present, forward index still RAW-encoded.
+ try (SegmentDirectory segmentDirectory = new SegmentLocalFSDirectory(INDEX_DIR, ReadMode.mmap);
+ SegmentDirectory.Reader reader = segmentDirectory.createReader()) {
+ ColumnMetadata columnMetadata = segmentDirectory.getSegmentMetadata().getColumnMetadataFor(column);
+ assertTrue(columnMetadata.hasDictionary());
+ assertEquals(columnMetadata.getForwardIndexEncoding(), FieldConfig.EncodingType.RAW);
+ assertTrue(reader.hasIndexFor(column, StandardIndexes.range()),
+ "Range index should be built by RangeIndexHandler over the new dictionary");
+ validateMetadataProperties(new SegmentMetadataImpl(INDEX_DIR).getColumnMetadataFor(column),
+ existingSegmentMetadata.getColumnMetadataFor(column), true);
+ }
+ }
+
+ /**
+ * Symmetric "disable dictionary on a column with a range index" scenario, under the new contract that a range
+ * index always requires a dictionary. The valid migration path is to drop the range index in the same reload as
+ * the dictionary; otherwise the table-config validation rejects the change.
+ *
+ * <p>This test asserts the handler chain for the supported flow: starting from a dict-encoded column with a
+ * range index, removing the column from both the dictionary set and the {@code rangeIndexColumns} set causes
+ * {@code ForwardIndexHandler.DISABLE_DICTIONARY} to drop the dictionary and (via
+ * {@code removeDictRelatedIndexes}) the now-stale range index. {@code RangeIndexHandler} then sees the column
+ * is no longer requested and stays a no-op.
+ */
+ @Test
+ public void testDisableDictionaryAndRangeIndexTogether()
+ throws Exception {
+ String column = DIM_DICT_INTEGER;
+ SegmentMetadataImpl existingSegmentMetadata;
+
+ // Step 1: build a dict-id-based range index on the dict-encoded column.
+ try (SegmentDirectory segmentDirectory = new SegmentLocalFSDirectory(INDEX_DIR, ReadMode.mmap);
+ SegmentDirectory.Writer writer = segmentDirectory.createWriter()) {
+ _segmentDirectory = segmentDirectory;
+ _writer = writer;
+ existingSegmentMetadata = segmentDirectory.getSegmentMetadata();
+
+ _rangeIndexColumns.add(column);
+ RangeIndexHandler rangeIndexHandler = new RangeIndexHandler(segmentDirectory, createIndexLoadingConfig());
+ assertTrue(rangeIndexHandler.needUpdateIndices(writer));
+ rangeIndexHandler.updateIndices(writer);
+ rangeIndexHandler.postUpdateIndicesCleanup(writer);
+ }
+
+ // Pre-condition: column has dict + range; range was built in the dict-id-based variant.
+ try (SegmentDirectory segmentDirectory = new SegmentLocalFSDirectory(INDEX_DIR, ReadMode.mmap);
+ SegmentDirectory.Reader reader = segmentDirectory.createReader()) {
+ ColumnMetadata columnMetadata = segmentDirectory.getSegmentMetadata().getColumnMetadataFor(column);
+ assertTrue(columnMetadata.hasDictionary(), "Pre-condition: column should have a dictionary");
+ assertTrue(reader.hasIndexFor(column, StandardIndexes.range()),
+ "Pre-condition: dict-id-based range index should exist");
+ }
+
+ // Step 2: disable dictionary AND remove range index from the config (the only valid migration under the new
+ // contract). ForwardIndexHandler emits DISABLE_DICTIONARY and via removeDictRelatedIndexes also drops the
+ // now-stale range index.
+ try (SegmentDirectory segmentDirectory = new SegmentLocalFSDirectory(INDEX_DIR, ReadMode.mmap);
+ SegmentDirectory.Writer writer = segmentDirectory.createWriter()) {
+ _segmentDirectory = segmentDirectory;
+ _writer = writer;
+
+ _noDictionaryColumns.add(column);
+ _rangeIndexColumns.remove(column);
+
+ ForwardIndexHandler forwardIndexHandler = createForwardIndexHandler();
+ assertEquals(forwardIndexHandler.computeOperations(writer),
+ Map.of(column, List.of(ForwardIndexHandler.Operation.DISABLE_DICTIONARY)));
+ forwardIndexHandler.updateIndices(writer);
+ forwardIndexHandler.postUpdateIndicesCleanup(writer);
+ }
+
+ // Verify: dict and range are both gone after ForwardIndexHandler's DISABLE_DICTIONARY pass.
+ try (SegmentDirectory segmentDirectory = new SegmentLocalFSDirectory(INDEX_DIR, ReadMode.mmap);
+ SegmentDirectory.Reader reader = segmentDirectory.createReader()) {
+ ColumnMetadata columnMetadata = segmentDirectory.getSegmentMetadata().getColumnMetadataFor(column);
+ assertFalse(columnMetadata.hasDictionary(), "Dictionary should be disabled after DISABLE_DICTIONARY");
+ assertFalse(reader.hasIndexFor(column, StandardIndexes.dictionary()));
+ assertFalse(reader.hasIndexFor(column, StandardIndexes.range()),
+ "Stale dict-id-based range index must be removed when dictionary is disabled");
+ }
+
+ // Step 3: RangeIndexHandler is a no-op because the column is no longer in the range-index list.
+ try (SegmentDirectory segmentDirectory = new SegmentLocalFSDirectory(INDEX_DIR, ReadMode.mmap);
+ SegmentDirectory.Writer writer = segmentDirectory.createWriter()) {
+ _segmentDirectory = segmentDirectory;
+ _writer = writer;
+
+ RangeIndexHandler rangeIndexHandler = new RangeIndexHandler(segmentDirectory, createIndexLoadingConfig());
+ assertFalse(rangeIndexHandler.needUpdateIndices(writer),
+ "RangeIndexHandler should not need updates after the range index was removed from the config");
+ }
+
+ // Final state: dict gone, range gone, forward index regenerated as raw.
+ try (SegmentDirectory segmentDirectory = new SegmentLocalFSDirectory(INDEX_DIR, ReadMode.mmap);
+ SegmentDirectory.Reader reader = segmentDirectory.createReader()) {
+ ColumnMetadata columnMetadata = segmentDirectory.getSegmentMetadata().getColumnMetadataFor(column);
+ assertFalse(columnMetadata.hasDictionary());
+ assertFalse(reader.hasIndexFor(column, StandardIndexes.range()));
+ validateMetadataProperties(new SegmentMetadataImpl(INDEX_DIR).getColumnMetadataFor(column),
+ existingSegmentMetadata.getColumnMetadataFor(column), false);
+ }
+ }
+
+ /**
+ * Pins down the handler-ordering contract documented on InvertedIndexHandler: the dictionary file (created by
+ * ForwardIndexHandler under the ENABLE_DICTIONARY operation for a RAW forward column with an inverted index)
+ * must already exist on disk before InvertedIndexHandler attempts to build the dict-id-based inverted index.
+ *
+ * <p>If a future change reorders handlers so InvertedIndexHandler runs before ForwardIndexHandler, the
+ * Preconditions.checkState(columnMetadata.hasDictionary(), ...) inside InvertedIndexHandler.createInvertedIndex
+ * ForColumn fires and this test catches the regression.
+ */
+ @Test
+ public void testHandlerOrderingForwardBeforeInvertedOnRawWithDictRequired()
+ throws Exception {
+ String column = DIM_ZSTANDARD_STRING;
+
+ // Configure: RAW forward index + explicit dictionary + inverted index. ForwardIndexHandler must emit
+ // ENABLE_DICTIONARY (creating a shared dictionary on the RAW column); InvertedIndexHandler must then build
+ // the inverted index on top of that dictionary.
+ try (SegmentDirectory segmentDirectory = new SegmentLocalFSDirectory(INDEX_DIR, ReadMode.mmap);
+ SegmentDirectory.Writer writer = segmentDirectory.createWriter()) {
+ _segmentDirectory = segmentDirectory;
+ _writer = writer;
+
+ _invertedIndexColumns.add(column);
+ _noDictionaryColumns.remove(column);
+ ObjectNode rawDictIndexes = JsonUtils.newObjectNode();
+ rawDictIndexes.set("dictionary", JsonUtils.newObjectNode());
+ FieldConfig existingConfig = _fieldConfigMap.get(column);
+ _fieldConfigMap.put(column,
+ new FieldConfig(column, FieldConfig.EncodingType.RAW, null, List.of(FieldConfig.IndexType.INVERTED),
+ existingConfig != null ? existingConfig.getCompressionCodec() : null, null, rawDictIndexes, null,
+ null));
+
+ // Step 1: ForwardIndexHandler must run first (matching SegmentPreProcessor's ordering).
+ ForwardIndexHandler forwardIndexHandler = createForwardIndexHandler();
+ assertEquals(forwardIndexHandler.computeOperations(writer),
+ Map.of(column, List.of(ForwardIndexHandler.Operation.ENABLE_DICTIONARY)));
+ forwardIndexHandler.updateIndices(writer);
+ forwardIndexHandler.postUpdateIndicesCleanup(writer);
+ }
+
+ // Verify ForwardIndexHandler created the dictionary on disk and updated metadata.
+ try (SegmentDirectory segmentDirectory = new SegmentLocalFSDirectory(INDEX_DIR, ReadMode.mmap);
+ SegmentDirectory.Reader reader = segmentDirectory.createReader()) {
+ ColumnMetadata columnMetadata = segmentDirectory.getSegmentMetadata().getColumnMetadataFor(column);
+ assertTrue(columnMetadata.hasDictionary(),
+ "Pre-condition for InvertedIndexHandler: dictionary must exist after ForwardIndexHandler runs");
+ assertTrue(reader.hasIndexFor(column, StandardIndexes.dictionary()));
+ }
+
+ // Step 2: InvertedIndexHandler runs next, sees the freshly-created dictionary, and builds the inverted index.
+ try (SegmentDirectory segmentDirectory = new SegmentLocalFSDirectory(INDEX_DIR, ReadMode.mmap);
+ SegmentDirectory.Writer writer = segmentDirectory.createWriter()) {
+ _segmentDirectory = segmentDirectory;
+ _writer = writer;
+
+ InvertedIndexHandler invertedIndexHandler = new InvertedIndexHandler(segmentDirectory,
+ createIndexLoadingConfig().getFieldIndexConfigByColName(), createTableConfig(), SCHEMA);
+ assertTrue(invertedIndexHandler.needUpdateIndices(writer));
+ invertedIndexHandler.updateIndices(writer);
+ invertedIndexHandler.postUpdateIndicesCleanup(writer);
+ }
+
+ // Final state: dict + raw forward + inverted index, all consistent.
+ try (SegmentDirectory segmentDirectory = new SegmentLocalFSDirectory(INDEX_DIR, ReadMode.mmap);
+ SegmentDirectory.Reader reader = segmentDirectory.createReader()) {
+ ColumnMetadata columnMetadata = segmentDirectory.getSegmentMetadata().getColumnMetadataFor(column);
+ assertTrue(columnMetadata.hasDictionary());
+ assertEquals(columnMetadata.getForwardIndexEncoding(), FieldConfig.EncodingType.RAW);
+ assertTrue(reader.hasIndexFor(column, StandardIndexes.inverted()));
+ }
+ }
+
@Test
public void testEnableDictionaryForMultipleColumns()
throws Exception {
String column1 = RAW_COLUMNS_WITH_FORWARD_INDEX.get(RANDOM.nextInt(RAW_COLUMNS_WITH_FORWARD_INDEX.size()));
String column2 = RAW_COLUMNS_WITH_FORWARD_INDEX.get(RANDOM.nextInt(RAW_COLUMNS_WITH_FORWARD_INDEX.size()));
+ if (RAW_SORTED_COLUMNS.contains(column1) || RAW_SORTED_COLUMNS.contains(column2)) {
+ return;
+ }
SegmentMetadataImpl existingSegmentMetadata;
try (SegmentDirectory segmentDirectory = new SegmentLocalFSDirectory(INDEX_DIR, ReadMode.mmap);
SegmentDirectory.Writer writer = segmentDirectory.createWriter()) {
@@ -1233,22 +1584,49 @@ public class ForwardIndexHandlerTest {
_noDictionaryColumns.remove(column1);
_noDictionaryColumns.remove(column2);
- _fieldConfigMap.remove(column1);
- _fieldConfigMap.remove(column2);
+ FieldConfig existingFieldConfig1 = _fieldConfigMap.get(column1);
+ ObjectNode dictIndexes1 = JsonUtils.newObjectNode();
+ dictIndexes1.set("dictionary", JsonUtils.newObjectNode());
+ _fieldConfigMap.put(column1,
+ new FieldConfig(column1, FieldConfig.EncodingType.RAW, null,
+ List.of(FieldConfig.IndexType.INVERTED),
+ existingFieldConfig1.getCompressionCodec(), null, dictIndexes1, null, null));
+ FieldConfig existingFieldConfig2 = _fieldConfigMap.get(column2);
+ ObjectNode dictIndexes2 = JsonUtils.newObjectNode();
+ dictIndexes2.set("dictionary", JsonUtils.newObjectNode());
+ _fieldConfigMap.put(column2,
+ new FieldConfig(column2, FieldConfig.EncodingType.RAW, null,
+ List.of(FieldConfig.IndexType.INVERTED),
+ existingFieldConfig2.getCompressionCodec(), null, dictIndexes2, null, null));
updateIndices();
// Tear down before validation. Because columns.psf and index map cleanup happens at segmentDirectory.close()
}
- SegmentMetadataImpl newSegmentMetadata = new SegmentMetadataImpl(INDEX_DIR);
- for (String column : List.of(column1, column2)) {
- ColumnMetadata metadata = existingSegmentMetadata.getColumnMetadataFor(column);
- testIndexExists(column, StandardIndexes.forward());
- testIndexExists(column, StandardIndexes.dictionary());
- validateIndexMap(column, true, false);
- validateForwardIndex(column, null, metadata.isSorted());
- // In column metadata, nothing other than hasDictionary and bitsPerElement should change.
- validateMetadataProperties(newSegmentMetadata.getColumnMetadataFor(column), metadata, true);
+ // Col1 validation.
+ ColumnMetadata metadata = existingSegmentMetadata.getColumnMetadataFor(column1);
+ testIndexExists(column1, StandardIndexes.forward());
+ testIndexExists(column1, StandardIndexes.dictionary());
+ validateIndexMap(column1, true, false);
+ if (metadata.isSorted()) {
+ validateForwardIndex(column1, null, true);
+ } else {
+ validateForwardIndex(column1, getExpectedRawCompressionCodec(column1), false, true);
+ }
+ // In column metadata, nothing other than hasDictionary and bitsPerElement should change.
+ validateMetadataProperties(new SegmentMetadataImpl(INDEX_DIR).getColumnMetadataFor(column1), metadata, true);
+
+ // Col2 validation.
+ metadata = existingSegmentMetadata.getColumnMetadataFor(column2);
+ testIndexExists(column2, StandardIndexes.forward());
+ testIndexExists(column2, StandardIndexes.dictionary());
+ validateIndexMap(column2, true, false);
+ if (metadata.isSorted()) {
+ validateForwardIndex(column2, null, true);
+ } else {
+ validateForwardIndex(column2, getExpectedRawCompressionCodec(column2), false, true);
}
+ // In column metadata, nothing other than hasDictionary and bitsPerElement should change.
+ validateMetadataProperties(new SegmentMetadataImpl(INDEX_DIR).getColumnMetadataFor(column2), metadata, true);
}
@Test
@@ -1869,19 +2247,32 @@ public class ForwardIndexHandlerTest {
throws IOException, ConfigurationException {
try (SegmentDirectory segmentDirectory = new SegmentLocalFSDirectory(INDEX_DIR, ReadMode.mmap);
SegmentDirectory.Reader reader = segmentDirectory.createReader()) {
- validateForwardIndex(segmentDirectory, reader, columnName, expectedCompressionType, isSorted);
+ validateForwardIndex(segmentDirectory, reader, columnName, expectedCompressionType, isSorted,
+ expectedCompressionType == null);
+ } catch (IndexReaderConstraintException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ private void validateForwardIndex(String columnName, @Nullable CompressionCodec expectedCompressionType,
+ boolean isSorted, boolean expectDictionary)
+ throws IOException, ConfigurationException {
+ try (SegmentDirectory segmentDirectory = new SegmentLocalFSDirectory(INDEX_DIR, ReadMode.mmap);
+ SegmentDirectory.Reader reader = segmentDirectory.createReader()) {
+ validateForwardIndex(segmentDirectory, reader, columnName, expectedCompressionType, isSorted, expectDictionary);
} catch (IndexReaderConstraintException e) {
throw new RuntimeException(e);
}
}
private void validateForwardIndex(SegmentDirectory segmentDirectory, SegmentDirectory.Reader reader,
- String columnName, @Nullable CompressionCodec expectedCompressionType, boolean isSorted)
+ String columnName, @Nullable CompressionCodec expectedCompressionType, boolean isSorted,
+ boolean expectDictionary)
throws IOException, IndexReaderConstraintException {
ColumnMetadata columnMetadata = segmentDirectory.getSegmentMetadata().getColumnMetadataFor(columnName);
boolean isSingleValue = columnMetadata.isSingleValue();
- if (expectedCompressionType == null) {
+ if (expectDictionary) {
assertTrue(reader.hasIndexFor(columnName, StandardIndexes.dictionary()));
} else {
assertFalse(reader.hasIndexFor(columnName, StandardIndexes.dictionary()));
@@ -1903,12 +2294,18 @@ public class ForwardIndexHandlerTest {
try (ForwardIndexReader<?> forwardIndexReader = readerFactory.createIndexReader(reader, fieldIndexConfigs,
columnMetadata)) {
Dictionary dictionary = null;
- if (columnMetadata.hasDictionary()) {
+ if (expectDictionary && forwardIndexReader.isDictionaryEncoded()) {
dictionary = DictionaryIndexType.read(reader, columnMetadata);
}
PinotSegmentColumnReader columnReader = new PinotSegmentColumnReader(forwardIndexReader, dictionary, null,
columnMetadata.getMaxNumberOfMultiValues());
+ if (expectDictionary && !forwardIndexReader.isDictionaryEncoded()) {
+ try (Dictionary loadedDictionary = DictionaryIndexType.read(reader, columnMetadata)) {
+ assertEquals(loadedDictionary.length(), columnMetadata.getCardinality());
+ }
+ }
+
for (int rowIdx = 0; rowIdx < columnMetadata.getTotalDocs(); rowIdx++) {
// For MV forward index disabled columns cannot do this validation as we had to create a unique set of elements
if (!MV_FORWARD_INDEX_DISABLED_COLUMNS.contains(columnName) && (rowIdx % 10 == 0)) {
@@ -2043,6 +2440,25 @@ public class ForwardIndexHandlerTest {
}
}
+ private CompressionCodec getExpectedRawCompressionCodec(String columnName) {
+ if (RAW_SNAPPY_COLUMNS.contains(columnName)) {
+ return CompressionCodec.SNAPPY;
+ }
+ if (RAW_ZSTANDARD_COLUMNS.contains(columnName)) {
+ return CompressionCodec.ZSTANDARD;
+ }
+ if (RAW_PASS_THROUGH_COLUMNS.contains(columnName)) {
+ return CompressionCodec.PASS_THROUGH;
+ }
+ if (RAW_LZ4_COLUMNS.contains(columnName)) {
+ return CompressionCodec.LZ4;
+ }
+ if (RAW_GZIP_COLUMNS.contains(columnName)) {
+ return CompressionCodec.GZIP;
+ }
+ throw new IllegalArgumentException("Unexpected raw column: " + columnName);
+ }
+
private void testIndexExists(String columnName, IndexType<?, ?, ?> indexType)
throws Exception {
try (SegmentDirectory segmentDirectory = new SegmentLocalFSDirectory(INDEX_DIR, ReadMode.mmap);
@@ -2136,8 +2552,7 @@ public class ForwardIndexHandlerTest {
FieldIndexConfigs.Builder builder = new FieldIndexConfigs.Builder();
// Add forward index config
- ForwardIndexConfig forwardIndexConfig =
- ForwardIndexConfig.getDefault(columnMetadata.getForwardIndexEncoding());
+ ForwardIndexConfig forwardIndexConfig = ForwardIndexConfig.getDefault(columnMetadata.getForwardIndexEncoding());
builder.add(StandardIndexes.forward(), forwardIndexConfig);
// Add dictionary config if the column has dictionary
diff --git a/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/loader/SegmentPreProcessorTest.java b/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/loader/SegmentPreProcessorTest.java
index 15dfa5df39b..1bc242a3a9c 100644
--- a/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/loader/SegmentPreProcessorTest.java
+++ b/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/loader/SegmentPreProcessorTest.java
@@ -18,6 +18,7 @@
*/
package org.apache.pinot.segment.local.segment.index.loader;
+import com.fasterxml.jackson.databind.node.ObjectNode;
import java.io.File;
import java.io.IOException;
import java.io.RandomAccessFile;
@@ -237,6 +238,7 @@ public class SegmentPreProcessorTest implements PinotBuffersAfterClassCheckRule
_ingestionConfig = new IngestionConfig();
_ingestionConfig.setRowTimeValueCheck(false);
_ingestionConfig.setSegmentTimeValueCheck(false);
+ _ingestionConfig.setContinueOnError(true);
_columnMinMaxValueGeneratorMode = null;
_bloomFilterConfigs = null;
@@ -354,15 +356,27 @@ public class SegmentPreProcessorTest implements PinotBuffersAfterClassCheckRule
checkTextIndexCreation(_newColumnsSchemaWithText, NEWLY_ADDED_STRING_MV_COL_RAW, false, 1, true, false, 4, 1, true);
}
- @Test(dataProvider = "bothV1AndV3", expectedExceptions = UnsupportedOperationException.class,
- expectedExceptionsMessageRegExp = "FST index is currently only supported on dictionary encoded columns: column4")
+ @Test(dataProvider = "bothV1AndV3")
public void testEnableFSTIndexOnExistingColumnRaw(SegmentVersion segmentVersion)
throws Exception {
buildSegment(segmentVersion);
+ // FST on RAW column requires explicit dictionary config.
+ // Must remove from noDictionaryColumns to avoid ConfigDeclaredTwiceException.
+ _noDictionaryColumns.remove(EXISTING_STRING_COL_RAW);
+ ObjectNode fstIndexes = org.apache.pinot.spi.utils.JsonUtils.newObjectNode();
+ fstIndexes.set("dictionary", org.apache.pinot.spi.utils.JsonUtils.newObjectNode());
_fieldConfigMap.put(EXISTING_STRING_COL_RAW,
- new FieldConfig(EXISTING_STRING_COL_RAW, FieldConfig.EncodingType.RAW, List.of(FieldConfig.IndexType.FST), null,
- null));
+ new FieldConfig(EXISTING_STRING_COL_RAW, FieldConfig.EncodingType.RAW, FieldConfig.IndexType.FST,
+ null, null, null, fstIndexes, null, null));
+ if (segmentVersion == SegmentVersion.v1) {
+ assertThrows(UnsupportedOperationException.class, this::runPreProcessor);
+ return;
+ }
runPreProcessor();
+ SegmentMetadataImpl segmentMetadata = new SegmentMetadataImpl(INDEX_DIR);
+ ColumnMetadata columnMetadata = segmentMetadata.getColumnMetadataFor(EXISTING_STRING_COL_RAW);
+ assertTrue(columnMetadata.hasDictionary());
+ assertTrue(columnMetadata.getIndexSizeFor(StandardIndexes.fst()) > 0);
}
@Test(dataProvider = "bothV1AndV3")
@@ -630,6 +644,108 @@ public class SegmentPreProcessorTest implements PinotBuffersAfterClassCheckRule
null);
}
+ /// Verifies that the range index is rebuilt every time the dictionary state of a column changes — both via
+ /// the explicit toggle path and via the auto-toggle path triggered by an index that requires a dictionary
+ /// (e.g. inverted). The on-disk range index format differs between dict-id-based (when a dictionary exists)
+ /// and raw-value-based (when not), so the index size must change in lockstep with the dictionary state.
+ @Test
+ public void testRangeIndexRebuiltOnDictionaryToggle()
+ throws Exception {
+ // Setup: V3 segment with a raw INT column carrying a range index. The range index is built against raw
+ // values at this point.
+ buildV3Segment();
+ _rangeIndexColumns.add(EXISTING_INT_COL_RAW);
+ runPreProcessor(_schema);
+ long rawRangeSize1 = new SegmentMetadataImpl(INDEX_DIR).getColumnMetadataFor(EXISTING_INT_COL_RAW)
+ .getIndexSizeFor(StandardIndexes.range());
+ assertTrue(rawRangeSize1 > 0, "Range index must exist before toggling dictionary state");
+ assertFalse(new SegmentMetadataImpl(INDEX_DIR).getColumnMetadataFor(EXISTING_INT_COL_RAW).hasDictionary(),
+ "Column must start without a dictionary");
+
+ // (1) Auto-enable dictionary by adding an inverted index. ForwardIndexHandler now queues
+ // ENABLE_DICTIONARY because the inverted index requires a dictionary. The range index must be rebuilt
+ // against dict ids — its on-disk size therefore changes.
+ _invertedIndexColumns.add(EXISTING_INT_COL_RAW);
+ runPreProcessor(_schema);
+ SegmentMetadataImpl afterAutoEnable = new SegmentMetadataImpl(INDEX_DIR);
+ long dictRangeSize = afterAutoEnable.getColumnMetadataFor(EXISTING_INT_COL_RAW)
+ .getIndexSizeFor(StandardIndexes.range());
+ assertTrue(afterAutoEnable.getColumnMetadataFor(EXISTING_INT_COL_RAW).hasDictionary(),
+ "Dictionary must be auto-created when inverted index requires it");
+ assertNotEquals(dictRangeSize, rawRangeSize1,
+ "Range index must be rebuilt with dict-id format when dictionary is auto-enabled");
+
+ // (2) Auto-disable dictionary by removing the inverted index. The dictionary is no longer required and
+ // gets dropped; the range index must be rebuilt against raw values again.
+ _invertedIndexColumns.remove(EXISTING_INT_COL_RAW);
+ runPreProcessor(_schema);
+ SegmentMetadataImpl afterAutoDisable = new SegmentMetadataImpl(INDEX_DIR);
+ long rawRangeSize2 = afterAutoDisable.getColumnMetadataFor(EXISTING_INT_COL_RAW)
+ .getIndexSizeFor(StandardIndexes.range());
+ assertFalse(afterAutoDisable.getColumnMetadataFor(EXISTING_INT_COL_RAW).hasDictionary(),
+ "Dictionary must be auto-removed when no enabled index requires it");
+ assertNotEquals(rawRangeSize2, dictRangeSize,
+ "Range index must be rebuilt with raw-value format when dictionary is auto-disabled");
+
+ // (3) Explicit toggle path: enable dictionary by removing the column from noDictionaryColumns. Range
+ // index must be rebuilt yet again.
+ _noDictionaryColumns.remove(EXISTING_INT_COL_RAW);
+ runPreProcessor(_schema);
+ SegmentMetadataImpl afterExplicitEnable = new SegmentMetadataImpl(INDEX_DIR);
+ long dictRangeSize2 = afterExplicitEnable.getColumnMetadataFor(EXISTING_INT_COL_RAW)
+ .getIndexSizeFor(StandardIndexes.range());
+ assertTrue(afterExplicitEnable.getColumnMetadataFor(EXISTING_INT_COL_RAW).hasDictionary(),
+ "Dictionary must exist after explicit ENABLE_DICTIONARY");
+ assertNotEquals(dictRangeSize2, rawRangeSize2,
+ "Range index must be rebuilt with dict-id format when dictionary is explicitly enabled");
+
+ // (4) Explicit disable: re-add to noDictionaryColumns. Range index rebuilt back to raw-value format.
+ _noDictionaryColumns.add(EXISTING_INT_COL_RAW);
+ runPreProcessor(_schema);
+ SegmentMetadataImpl afterExplicitDisable = new SegmentMetadataImpl(INDEX_DIR);
+ long rawRangeSize3 = afterExplicitDisable.getColumnMetadataFor(EXISTING_INT_COL_RAW)
+ .getIndexSizeFor(StandardIndexes.range());
+ assertFalse(afterExplicitDisable.getColumnMetadataFor(EXISTING_INT_COL_RAW).hasDictionary(),
+ "Dictionary must be removed after explicit DISABLE_DICTIONARY");
+ assertNotEquals(rawRangeSize3, dictRangeSize2,
+ "Range index must be rebuilt with raw-value format when dictionary is explicitly disabled");
+ }
+
+ /// Verifies the encoding-flip transition where the forward index goes from dict-encoded to raw, while the
+ /// dictionary stays because an inverted index requires it. End state: dict + inverted + raw forward index.
+ @Test
+ public void testFlipDictForwardToRawForwardKeepingDictionaryForInvertedIndex()
+ throws Exception {
+ // Set up: COLUMN1 starts as dict-encoded INT with an inverted index.
+ _invertedIndexColumns.add(COLUMN1_NAME);
+ buildV3Segment();
+ SegmentMetadataImpl initial = new SegmentMetadataImpl(INDEX_DIR);
+ assertEquals(initial.getColumnMetadataFor(COLUMN1_NAME).getForwardIndexEncoding(),
+ FieldConfig.EncodingType.DICTIONARY,
+ "Pre-condition: COLUMN1 must start dict-encoded so the encoding flip is meaningful");
+ assertTrue(initial.getColumnMetadataFor(COLUMN1_NAME).hasDictionary());
+
+ // Flip encoding to RAW while keeping the inverted index. The inverted index requires a dictionary, so
+ // the auto-keep-dictionary-when-required-by-index logic must keep the dictionary across the flip.
+ _noDictionaryColumns.add(COLUMN1_NAME);
+ runPreProcessor(_schema);
+
+ SegmentMetadataImpl afterFlip = new SegmentMetadataImpl(INDEX_DIR);
+ assertEquals(afterFlip.getColumnMetadataFor(COLUMN1_NAME).getForwardIndexEncoding(),
+ FieldConfig.EncodingType.RAW, "Forward index must be raw after the flip");
+ assertTrue(afterFlip.getColumnMetadataFor(COLUMN1_NAME).hasDictionary(),
+ "Dictionary must be kept because the inverted index still requires it");
+ try (SegmentDirectory dir = new SegmentLocalFSDirectory(INDEX_DIR, ReadMode.mmap);
+ SegmentDirectory.Reader reader = dir.createReader()) {
+ assertTrue(reader.hasIndexFor(COLUMN1_NAME, StandardIndexes.dictionary()),
+ "Dictionary file must remain on disk");
+ assertTrue(reader.hasIndexFor(COLUMN1_NAME, StandardIndexes.inverted()),
+ "Inverted index must remain on disk and continue to point at the same dict ids");
+ assertTrue(reader.hasIndexFor(COLUMN1_NAME, StandardIndexes.forward()),
+ "Raw forward index must exist after the flip");
+ }
+ }
+
@Test
public void testForwardIndexHandlerChangeCompression()
throws Exception {
@@ -1525,10 +1641,22 @@ public class SegmentPreProcessorTest implements PinotBuffersAfterClassCheckRule
_rangeIndexColumns.add("daysSinceEpoch");
verifyProcessNotNeeded();
_rangeIndexColumns.remove("daysSinceEpoch");
- // Add inverted index to raw column
+ // Add inverted index to raw column. On v3 segments the inverted index requires a dictionary, so the
+ // auto-create-dictionary path now triggers ENABLE_DICTIONARY (and InvertedIndexHandler then builds the
+ // inverted index against it). v1 segments still skip ForwardIndexHandler entirely, so the result is a
+ // no-op there. Pre-PR behavior on v3 was the same silent no-op as v1, leaving the inverted-index request
+ // orphaned. Removing inverted on v3 then triggers DISABLE_DICTIONARY to put the column back to its
+ // original raw-no-dict state so the rest of this test's "no processing needed" assertions hold.
_invertedIndexColumns.add(EXISTING_STRING_COL_RAW);
- verifyProcessNotNeeded();
+ if (segmentVersion == SegmentVersion.v3) {
+ verifyProcessNeeded();
+ } else {
+ verifyProcessNotNeeded();
+ }
_invertedIndexColumns.remove(EXISTING_STRING_COL_RAW);
+ if (segmentVersion == SegmentVersion.v3) {
+ verifyProcessNeeded();
+ }
// Add inverted index to non-existing column
_invertedIndexColumns.add("newColumnX");
verifyProcessNotNeeded();
@@ -1811,7 +1939,7 @@ public class SegmentPreProcessorTest implements PinotBuffersAfterClassCheckRule
new IndexLoadingConfig(tableConfig, schema))) {
// Should need processing due to new star-tree config
assertTrue(processor.needProcess());
- // Process should complete without throwing exception, but star-tree should not be created
+ // Process logs error and skips invalid star-tree config; no exception is thrown
processor.process(SEGMENT_OPERATIONS_THROTTLER);
}
@@ -1865,7 +1993,7 @@ public class SegmentPreProcessorTest implements PinotBuffersAfterClassCheckRule
new IndexLoadingConfig(tableConfig, schema))) {
// Should need processing due to changed star-tree config
assertTrue(processor.needProcess());
- // Process should complete without throwing exception, but star-tree should not be updated
+ // Process logs error and skips invalid star-tree config; no exception is thrown
processor.process(SEGMENT_OPERATIONS_THROTTLER);
}
diff --git a/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/loader/invertedindex/InvertedIndexHandlerTest.java b/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/loader/invertedindex/InvertedIndexHandlerTest.java
new file mode 100644
index 00000000000..9756982f996
--- /dev/null
+++ b/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/loader/invertedindex/InvertedIndexHandlerTest.java
@@ -0,0 +1,80 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader.invertedindex;
+
+import java.io.File;
+import java.nio.ByteOrder;
+import java.nio.file.Files;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
+import org.mockito.Mockito;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertTrue;
+
+
+/// Unit tests for the [LegacyRawValueInvertedIndexCleanup#isLegacyRawValueInvertedIndexFormat] format detector.
+/// The end-to-end rebuild path (preprocessor detects legacy bytes, drops the file, handler rebuilds in modern
+/// format) is covered by `LegacyRawValueInvertedIndexMigrationIntegrationTest` against real v1 and v3 fixtures,
+/// so this file only pins down the byte-pattern detector itself.
+public class InvertedIndexHandlerTest {
+
+ @Test
+ public void testDetectsLegacyRawValueInvertedIndexFormat()
+ throws Exception {
+ File indexFile = Files.createTempFile("legacy-raw-inverted", ".inv").toFile();
+ ColumnMetadata columnMetadata = Mockito.mock(ColumnMetadata.class);
+ Mockito.when(columnMetadata.getCardinality()).thenReturn(3);
+ try (PinotDataBuffer dataBuffer =
+ PinotDataBuffer.mapFile(indexFile, false, 0, 128, ByteOrder.BIG_ENDIAN, "legacy-raw-inverted")) {
+ dataBuffer.putInt(0, 1);
+ dataBuffer.putInt(4, 3);
+ dataBuffer.putInt(8, 16);
+ dataBuffer.putLong(12, 44);
+ dataBuffer.putLong(20, 16);
+ dataBuffer.putLong(28, 60);
+ dataBuffer.putLong(36, 32);
+
+ assertTrue(LegacyRawValueInvertedIndexCleanup.isLegacyRawValueInvertedIndexFormat(dataBuffer, columnMetadata));
+ } finally {
+ FileUtils.deleteQuietly(indexFile);
+ }
+ }
+
+ @Test
+ public void testStandardBitmapInvertedIndexIsNotDetectedAsLegacyRawValueFormat()
+ throws Exception {
+ File indexFile = Files.createTempFile("standard-inverted", ".inv").toFile();
+ ColumnMetadata columnMetadata = Mockito.mock(ColumnMetadata.class);
+ Mockito.when(columnMetadata.getCardinality()).thenReturn(3);
+ try (PinotDataBuffer dataBuffer =
+ PinotDataBuffer.mapFile(indexFile, false, 0, 128, ByteOrder.BIG_ENDIAN, "standard-inverted")) {
+ dataBuffer.putInt(0, 16);
+ dataBuffer.putInt(4, 24);
+ dataBuffer.putInt(8, 40);
+ dataBuffer.putInt(12, 64);
+
+ assertFalse(LegacyRawValueInvertedIndexCleanup.isLegacyRawValueInvertedIndexFormat(dataBuffer, columnMetadata));
+ } finally {
+ FileUtils.deleteQuietly(indexFile);
+ }
+ }
+}
diff --git a/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/readers/LazyRowTest.java b/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/readers/LazyRowTest.java
index 54ac86c25c3..d941988f30c 100644
--- a/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/readers/LazyRowTest.java
+++ b/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/readers/LazyRowTest.java
@@ -102,8 +102,10 @@ public class LazyRowTest {
ForwardIndexReader col1ForwardIndexReader = mock(ForwardIndexReader.class);
when(col1ForwardIndexReader.isSingleValue()).thenReturn(true);
+ when(col1ForwardIndexReader.isDictionaryEncoded()).thenReturn(true);
ForwardIndexReader col2ForwardIndexReader = mock(ForwardIndexReader.class);
when(col2ForwardIndexReader.isSingleValue()).thenReturn(true);
+ when(col2ForwardIndexReader.isDictionaryEncoded()).thenReturn(true);
when(_col1Datasource.getForwardIndex()).thenReturn(col1ForwardIndexReader);
when(col2Datasource.getForwardIndex()).thenReturn(col2ForwardIndexReader);
when(col2ForwardIndexReader.getDictId(eq(1), any())).thenReturn(1);
diff --git a/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/utils/IndexCombinationValidationTest.java b/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/utils/IndexCombinationValidationTest.java
new file mode 100644
index 00000000000..a61c2d81316
--- /dev/null
+++ b/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/utils/IndexCombinationValidationTest.java
@@ -0,0 +1,665 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.utils;
+
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.pinot.spi.config.table.FieldConfig;
+import org.apache.pinot.spi.config.table.FieldConfig.CompressionCodec;
+import org.apache.pinot.spi.config.table.FieldConfig.EncodingType;
+import org.apache.pinot.spi.config.table.FieldConfig.IndexType;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec.DataType;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.assertTrue;
+import static org.testng.Assert.fail;
+
+
+/**
+ * Exhaustive tests for all valid and invalid combinations of:
+ * - Forward index encoding: DICTIONARY vs RAW
+ * - Dictionary: enabled (default), disabled (noDictionaryColumns), or explicit (indexes.dictionary)
+ * - Secondary indexes: inverted, FST, IFST, range, text, JSON, bloom filter
+ * - Column data types: STRING (SV), INT (SV), INT (MV), FLOAT (SV)
+ *
+ * The core invariant under test: any column that uses an index requiring a dictionary
+ * (inverted, FST, IFST) must have an explicit dictionary config when the forward index is raw.
+ */
+public class IndexCombinationValidationTest {
+
+ private static final String TABLE_NAME = "testTable";
+ private static final String STR_COL = "strCol"; // STRING SV
+ private static final String INT_COL = "intCol"; // INT SV
+ private static final String INT_MV_COL = "intMvCol"; // INT MV
+ private static final String FLOAT_COL = "floatCol"; // FLOAT SV
+
+ private static final Schema SCHEMA = new Schema.SchemaBuilder()
+ .setSchemaName(TABLE_NAME)
+ .addSingleValueDimension(STR_COL, DataType.STRING)
+ .addSingleValueDimension(INT_COL, DataType.INT)
+ .addMultiValueDimension(INT_MV_COL, DataType.INT)
+ .addSingleValueDimension(FLOAT_COL, DataType.FLOAT)
+ .build();
+
+ // ----- helpers -----
+
+ /** Build a FieldConfig with an explicit dictionary node in the indexes section. */
+ private static FieldConfig rawWithExplicitDict(String col, List<IndexType> indexes) {
+ ObjectNode indexesNode = JsonUtils.newObjectNode();
+ indexesNode.set("dictionary", JsonUtils.newObjectNode());
+ return new FieldConfig(col, EncodingType.RAW, null, indexes, null, null, indexesNode, null, null);
+ }
+
+ /** Validate and expect success. */
+ private static void assertValid(TableConfig tableConfig) {
+ try {
+ TableConfigUtils.validate(tableConfig, SCHEMA);
+ } catch (Exception e) {
+ fail("Expected validation to pass but got: " + e.getMessage(), e);
+ }
+ }
+
+ /** Validate and expect failure containing the given message fragment. */
+ private static void assertInvalid(TableConfig tableConfig, String errorFragment) {
+ try {
+ TableConfigUtils.validate(tableConfig, SCHEMA);
+ fail("Expected validation to fail with: " + errorFragment);
+ } catch (Exception e) {
+ assertTrue(e.getMessage() != null && e.getMessage().contains(errorFragment),
+ "Expected '" + errorFragment + "' in error but got: " + e.getMessage());
+ }
+ }
+
+ /// Validate and expect failure containing any one of the given message fragments. Used when the same misconfig is
+ /// caught by multiple validation checks and the order of checks (and hence which error surfaces) is not
+ /// guaranteed across releases. For example, "raw forward + inverted with no explicit dictionary" can be flagged
+ /// either by `validateExplicitDictionaryForRawForwardIndex` ("require a dictionary") or by the older legacy
+ /// check that ran from `InvertedIndexType.validate` ("without dictionary"); both reject the same scenario.
+ private static void assertInvalidContains(TableConfig tableConfig, String... errorFragments) {
+ try {
+ TableConfigUtils.validate(tableConfig, SCHEMA);
+ fail("Expected validation to fail with one of: " + String.join(" | ", errorFragments));
+ } catch (Exception e) {
+ String message = e.getMessage();
+ boolean matched = false;
+ if (message != null) {
+ for (String fragment : errorFragments) {
+ if (message.contains(fragment)) {
+ matched = true;
+ break;
+ }
+ }
+ }
+ assertTrue(matched,
+ "Expected any of [" + String.join(" | ", errorFragments) + "] in error but got: " + message);
+ }
+ }
+
+ // ============================================================
+ // 1. Dictionary-requiring indexes (inverted / FST / IFST)
+ // with dictionary-encoded forward index
+ // ============================================================
+
+ @Test
+ public void testDictEncodedWithInvertedIndexPasses() {
+ // dict-encoded forward + inverted: standard, always valid
+ TableConfig tc = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setInvertedIndexColumns(List.of(STR_COL)).build();
+ assertValid(tc);
+ }
+
+ @Test
+ public void testDictEncodedWithFstIndexPasses() {
+ FieldConfig fc = new FieldConfig(STR_COL, EncodingType.DICTIONARY, IndexType.FST, null, null);
+ TableConfig tc = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setFieldConfigList(List.of(fc)).build();
+ assertValid(tc);
+ }
+
+ @Test
+ public void testDictEncodedWithInvertedAndTextAndBloomPasses() {
+ // multiple indexes on a dict-encoded column
+ TableConfig tc = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setInvertedIndexColumns(List.of(STR_COL))
+ .setBloomFilterColumns(List.of(STR_COL))
+ .build();
+ assertValid(tc);
+ }
+
+ // ============================================================
+ // 2. RAW forward index without explicit dictionary
+ // + inverted/FST/IFST index → FAILS (all require a dictionary)
+ // ============================================================
+
+ @Test
+ public void testRawWithInvertedIndexNoExplicitDictFails() {
+ TableConfig tc = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setNoDictionaryColumns(List.of(STR_COL))
+ .setInvertedIndexColumns(List.of(STR_COL))
+ .build();
+ assertInvalidContains(tc, "require a dictionary", "without dictionary");
+ }
+
+ @Test
+ public void testRawWithInvertedIndexIntColumnNoExplicitDictFails() {
+ TableConfig tc = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setNoDictionaryColumns(List.of(INT_COL))
+ .setInvertedIndexColumns(List.of(INT_COL))
+ .build();
+ assertInvalidContains(tc, "require a dictionary", "without dictionary");
+ }
+
+ @Test
+ public void testRawWithFstIndexNoExplicitDictFails() {
+ FieldConfig fc = new FieldConfig(STR_COL, EncodingType.RAW, IndexType.FST, null, null);
+ TableConfig tc = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setNoDictionaryColumns(List.of(STR_COL))
+ .setFieldConfigList(List.of(fc))
+ .build();
+ assertInvalid(tc, "dictionary");
+ }
+
+ @Test
+ public void testRawWithIfstIndexNoExplicitDictFails() {
+ // IFSTIndexType.validate() fires: "Cannot create IFST index on column: <col> without dictionary"
+ FieldConfig fc = new FieldConfig(STR_COL, EncodingType.RAW, IndexType.IFST, null, null);
+ TableConfig tc = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setNoDictionaryColumns(List.of(STR_COL))
+ .setFieldConfigList(List.of(fc))
+ .build();
+ assertInvalid(tc, "dictionary");
+ }
+
+ @Test
+ public void testRawWithMultipleDictRequiringIndexesNoExplicitDictFails() {
+ // Both inverted and FST require dictionary; at least one validator fires mentioning "dictionary"
+ FieldConfig fc = new FieldConfig(STR_COL, EncodingType.RAW, null, List.of(IndexType.FST),
+ null, null, null, null, null);
+ TableConfig tc = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setNoDictionaryColumns(List.of(STR_COL))
+ .setInvertedIndexColumns(List.of(STR_COL))
+ .setFieldConfigList(List.of(fc))
+ .build();
+ assertInvalid(tc, "dictionary");
+ }
+
+ // ============================================================
+ // 3. RAW forward index WITH explicit dictionary
+ // + dictionary-requiring indexes → must PASS
+ // ============================================================
+
+ @Test
+ public void testRawWithExplicitDictAndInvertedIndexPasses() {
+ ObjectNode indexes = JsonUtils.newObjectNode();
+ indexes.set("dictionary", JsonUtils.newObjectNode());
+ FieldConfig fc = new FieldConfig(STR_COL, EncodingType.RAW, null, null, null, null, indexes, null, null);
+ TableConfig tc = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setInvertedIndexColumns(List.of(STR_COL))
+ .setFieldConfigList(List.of(fc))
+ .build();
+ assertValid(tc);
+ }
+
+ @Test
+ public void testRawWithExplicitDictAndFstIndexPasses() {
+ assertValid(new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setFieldConfigList(List.of(rawWithExplicitDict(STR_COL, List.of(IndexType.FST))))
+ .build());
+ }
+
+ @Test
+ public void testRawWithExplicitDictAndIfstIndexPasses() {
+ assertValid(new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setFieldConfigList(List.of(rawWithExplicitDict(STR_COL, List.of(IndexType.IFST))))
+ .build());
+ }
+
+ @Test
+ public void testRawWithExplicitDictAndInvertedPlusFstPasses() {
+ ObjectNode indexes = JsonUtils.newObjectNode();
+ indexes.set("dictionary", JsonUtils.newObjectNode());
+ FieldConfig fc = new FieldConfig(STR_COL, EncodingType.RAW, null, List.of(IndexType.FST),
+ null, null, indexes, null, null);
+ TableConfig tc = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setInvertedIndexColumns(List.of(STR_COL))
+ .setFieldConfigList(List.of(fc))
+ .build();
+ assertValid(tc);
+ }
+
+ @Test
+ public void testRawWithExplicitDictAndInvertedIndexIntColumnPasses() {
+ ObjectNode indexes = JsonUtils.newObjectNode();
+ indexes.set("dictionary", JsonUtils.newObjectNode());
+ FieldConfig fc = new FieldConfig(INT_COL, EncodingType.RAW, null, null, null, null, indexes, null, null);
+ TableConfig tc = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setInvertedIndexColumns(List.of(INT_COL))
+ .setFieldConfigList(List.of(fc))
+ .build();
+ assertValid(tc);
+ }
+
+ // ============================================================
+ // 4. RAW forward index with indexes that do NOT require dict
+ // (text, bloom, JSON, range-on-numeric) → must PASS
+ // ============================================================
+
+ @Test
+ public void testRawWithTextIndexNoDictPasses() {
+ FieldConfig fc = new FieldConfig(STR_COL, EncodingType.RAW, IndexType.TEXT, null, null);
+ TableConfig tc = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setNoDictionaryColumns(List.of(STR_COL))
+ .setFieldConfigList(List.of(fc))
+ .build();
+ assertValid(tc);
+ }
+
+ @Test
+ public void testRawWithBloomFilterNoDictPasses() {
+ TableConfig tc = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setNoDictionaryColumns(List.of(STR_COL))
+ .setBloomFilterColumns(List.of(STR_COL))
+ .build();
+ assertValid(tc);
+ }
+
+ @Test
+ public void testRawWithJsonIndexStringColumnNoDictPasses() {
+ FieldConfig fc = new FieldConfig(STR_COL, EncodingType.RAW, IndexType.JSON, null, null);
+ TableConfig tc = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setNoDictionaryColumns(List.of(STR_COL))
+ .setFieldConfigList(List.of(fc))
+ .build();
+ assertValid(tc);
+ }
+
+ @Test
+ public void testRawWithRangeIndexNumericColumnNoDictPasses() {
+ // Range index on numeric (INT) column does NOT require a dictionary
+ TableConfig tc = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setNoDictionaryColumns(List.of(INT_COL))
+ .setRangeIndexColumns(List.of(INT_COL))
+ .build();
+ assertValid(tc);
+ }
+
+ @Test
+ public void testRawWithRangeIndexFloatColumnNoDictPasses() {
+ TableConfig tc = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setNoDictionaryColumns(List.of(FLOAT_COL))
+ .setRangeIndexColumns(List.of(FLOAT_COL))
+ .build();
+ assertValid(tc);
+ }
+
+ // ============================================================
+ // 5. Range index on non-numeric column: requires dictionary
+ // ============================================================
+
+ @Test
+ public void testDictWithRangeIndexStringColumnPasses() {
+ // dict-encoded STRING + range: valid
+ TableConfig tc = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setRangeIndexColumns(List.of(STR_COL))
+ .build();
+ assertValid(tc);
+ }
+
+ @Test
+ public void testRawWithRangeIndexStringColumnNoDictFails() {
+ // RAW STRING + range without dict: invalid (range on non-numeric needs dict)
+ TableConfig tc = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setNoDictionaryColumns(List.of(STR_COL))
+ .setRangeIndexColumns(List.of(STR_COL))
+ .build();
+ assertInvalid(tc, "Cannot create range index on non-numeric column");
+ }
+
+ @Test
+ public void testRawWithRangeIndexStringColumnWithExplicitDictPasses() {
+ ObjectNode indexes = JsonUtils.newObjectNode();
+ indexes.set("dictionary", JsonUtils.newObjectNode());
+ FieldConfig fc = new FieldConfig(STR_COL, EncodingType.RAW, null, null, null, null, indexes, null, null);
+ TableConfig tc = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setRangeIndexColumns(List.of(STR_COL))
+ .setFieldConfigList(List.of(fc))
+ .build();
+ assertValid(tc);
+ }
+
+ // ============================================================
+ // 6. Multi-value column combinations
+ // ============================================================
+
+ @Test
+ public void testDictMvWithInvertedIndexPasses() {
+ // MV INT + dict + inverted: valid
+ TableConfig tc = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setInvertedIndexColumns(List.of(INT_MV_COL))
+ .build();
+ assertValid(tc);
+ }
+
+ @Test
+ public void testRawMvWithInvertedIndexNoDictFails() {
+ TableConfig tc = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setNoDictionaryColumns(List.of(INT_MV_COL))
+ .setInvertedIndexColumns(List.of(INT_MV_COL))
+ .build();
+ assertInvalidContains(tc, "require a dictionary", "without dictionary");
+ }
+
+ @Test
+ public void testRawMvWithInvertedIndexWithExplicitDictPasses() {
+ ObjectNode indexes = JsonUtils.newObjectNode();
+ indexes.set("dictionary", JsonUtils.newObjectNode());
+ FieldConfig fc = new FieldConfig(INT_MV_COL, EncodingType.RAW, null, null, null, null, indexes, null, null);
+ TableConfig tc = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setInvertedIndexColumns(List.of(INT_MV_COL))
+ .setFieldConfigList(List.of(fc))
+ .build();
+ assertValid(tc);
+ }
+
+ @Test
+ public void testFstIndexMvColumnFails() {
+ // FST on MV column is never valid regardless of encoding
+ FieldConfig fc = new FieldConfig(INT_MV_COL, EncodingType.DICTIONARY, IndexType.FST, null, null);
+ TableConfig tc = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setFieldConfigList(List.of(fc))
+ .build();
+ assertInvalid(tc, "Cannot create FST index on multi-value column");
+ }
+
+ // ============================================================
+ // 7. Multiple columns: one valid, one invalid
+ // ============================================================
+
+ @Test
+ public void testMixedColumnsOneRawInvertedNoDictFails() {
+ // strCol is raw + inverted without dict → fails; intCol is dict + inverted → valid on its own
+ TableConfig tc = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setNoDictionaryColumns(List.of(STR_COL))
+ .setInvertedIndexColumns(Arrays.asList(STR_COL, INT_COL))
+ .build();
+ assertInvalidContains(tc, "require a dictionary", "without dictionary");
+ }
+
+ @Test
+ public void testMixedColumnsAllValidPasses() {
+ // strCol: raw + text (no dict required); intCol: dict + inverted; floatCol: dict + range
+ FieldConfig fc = new FieldConfig(STR_COL, EncodingType.RAW, IndexType.TEXT, null, null);
+ TableConfig tc = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setNoDictionaryColumns(List.of(STR_COL))
+ .setInvertedIndexColumns(List.of(INT_COL))
+ .setRangeIndexColumns(List.of(FLOAT_COL))
+ .setFieldConfigList(List.of(fc))
+ .build();
+ assertValid(tc);
+ }
+
+ @Test
+ public void testMixedColumnsRawDictAndRawNoDictInvertedOnlyFails() {
+ // strCol: raw + explicit dict + inverted (valid); intCol: raw + no dict + inverted → fails
+ ObjectNode indexes = JsonUtils.newObjectNode();
+ indexes.set("dictionary", JsonUtils.newObjectNode());
+ FieldConfig fc = new FieldConfig(STR_COL, EncodingType.RAW, null, null, null, null, indexes, null, null);
+ TableConfig tc = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setNoDictionaryColumns(List.of(INT_COL))
+ .setInvertedIndexColumns(Arrays.asList(STR_COL, INT_COL))
+ .setFieldConfigList(List.of(fc))
+ .build();
+ assertInvalidContains(tc, "require a dictionary", "without dictionary");
+ }
+
+ // ============================================================
+ // 8. Forward index disabled combinations
+ // ============================================================
+
+ @Test
+ public void testForwardIndexDisabledWithDictAndInvertedPasses() {
+ // Forward index disabled but dictionary + inverted still enabled: valid
+ ObjectNode indexes = JsonUtils.newObjectNode();
+ ObjectNode fwdDisabled = JsonUtils.newObjectNode();
+ fwdDisabled.put("disabled", true);
+ indexes.set("forward", fwdDisabled);
+ FieldConfig fc = new FieldConfig(INT_COL, EncodingType.DICTIONARY, null,
+ null, null, null, indexes, null, null);
+ TableConfig tc = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setInvertedIndexColumns(List.of(INT_COL))
+ .setFieldConfigList(List.of(fc))
+ .build();
+ assertValid(tc);
+ }
+
+ // ============================================================
+ // 9. Compression codec validation
+ // ============================================================
+
+ @Test
+ public void testRawWithLz4CodecPasses() {
+ FieldConfig fc = new FieldConfig(STR_COL, EncodingType.RAW, (IndexType) null, CompressionCodec.LZ4, null);
+ TableConfig tc = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setNoDictionaryColumns(List.of(STR_COL))
+ .setFieldConfigList(List.of(fc))
+ .build();
+ assertValid(tc);
+ }
+
+ @Test
+ public void testRawWithSnappyCodecPasses() {
+ FieldConfig fc = new FieldConfig(STR_COL, EncodingType.RAW, (IndexType) null, CompressionCodec.SNAPPY, null);
+ TableConfig tc = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setNoDictionaryColumns(List.of(STR_COL))
+ .setFieldConfigList(List.of(fc))
+ .build();
+ assertValid(tc);
+ }
+
+ @Test
+ public void testRawWithZstdCodecPasses() {
+ FieldConfig fc = new FieldConfig(STR_COL, EncodingType.RAW, (IndexType) null, CompressionCodec.ZSTANDARD, null);
+ TableConfig tc = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setNoDictionaryColumns(List.of(STR_COL))
+ .setFieldConfigList(List.of(fc))
+ .build();
+ assertValid(tc);
+ }
+
+ @Test
+ public void testRawWithClpCodecStringColumnPasses() {
+ // CLP codecs are valid for raw STRING columns
+ FieldConfig fc = new FieldConfig(STR_COL, EncodingType.RAW, (IndexType) null, CompressionCodec.CLP, null);
+ TableConfig tc = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setNoDictionaryColumns(List.of(STR_COL))
+ .setFieldConfigList(List.of(fc))
+ .build();
+ assertValid(tc);
+ }
+
+ @Test
+ public void testRawWithClpCodecNonStringColumnFails() {
+ // CLP is only valid on STRING stored type
+ FieldConfig fc = new FieldConfig(INT_COL, EncodingType.RAW, (IndexType) null, CompressionCodec.CLP, null);
+ TableConfig tc = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setNoDictionaryColumns(List.of(INT_COL))
+ .setFieldConfigList(List.of(fc))
+ .build();
+ assertInvalid(tc, "CLP");
+ }
+
+ @Test
+ public void testDeltaDeltaCodecNonNumericColumnFails() {
+ // DELTADELTA only valid on INT/LONG columns
+ FieldConfig fc = new FieldConfig(STR_COL, EncodingType.RAW, (IndexType) null, CompressionCodec.DELTADELTA, null);
+ TableConfig tc = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setNoDictionaryColumns(List.of(STR_COL))
+ .setFieldConfigList(List.of(fc))
+ .build();
+ assertInvalid(tc, "DELTADELTA can only be used on INT/LONG");
+ }
+
+ @Test
+ public void testDeltaDeltaCodecRawWithExplicitDictFails() {
+ // EncodingType.RAW + explicit dictionary + DELTADELTA codec is rejected by ForwardIndexType.validate:
+ // when the dictionary is enabled (regardless of forward-index encoding), the codec must be applicable to a
+ // dict-encoded index. DELTADELTA is applicable to neither raw nor dict-encoded forward indexes (it is a
+ // dict-id stream codec), so the validation surfaces the dict-encoded message even though the column will end
+ // up with a RAW forward index alongside the standalone dictionary at segment-creation time.
+ ObjectNode indexes = JsonUtils.newObjectNode();
+ indexes.set("dictionary", JsonUtils.newObjectNode());
+ FieldConfig fc = new FieldConfig(INT_COL, EncodingType.RAW, null, null, CompressionCodec.DELTADELTA, null,
+ indexes, null, null);
+ TableConfig tc = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setFieldConfigList(List.of(fc))
+ .build();
+ assertInvalid(tc, "DELTADELTA");
+ }
+
+ // ============================================================
+ // 10. Index type restrictions by data type / cardinality
+ // ============================================================
+
+ @Test
+ public void testFstIndexNonStringColumnFails() {
+ FieldConfig fc = new FieldConfig(INT_COL, EncodingType.DICTIONARY, IndexType.FST, null, null);
+ TableConfig tc = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setFieldConfigList(List.of(fc))
+ .build();
+ assertInvalid(tc, "Cannot create FST index on column: " + INT_COL + " of stored type other than STRING");
+ }
+
+ @Test
+ public void testTextIndexNonStringColumnFails() {
+ FieldConfig fc = new FieldConfig(INT_COL, EncodingType.DICTIONARY, IndexType.TEXT, null, null);
+ TableConfig tc = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setFieldConfigList(List.of(fc))
+ .build();
+ assertInvalid(tc, "Cannot create TEXT index on column: " + INT_COL + " of stored type other than STRING");
+ }
+
+ @Test
+ public void testTextIndexStringColumnRawPasses() {
+ // Text index does not require dictionary
+ FieldConfig fc = new FieldConfig(STR_COL, EncodingType.RAW, IndexType.TEXT, null, null);
+ TableConfig tc = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setNoDictionaryColumns(List.of(STR_COL))
+ .setFieldConfigList(List.of(fc))
+ .build();
+ assertValid(tc);
+ }
+
+ @Test
+ public void testTextIndexStringColumnDictPasses() {
+ FieldConfig fc = new FieldConfig(STR_COL, EncodingType.DICTIONARY, IndexType.TEXT, null, null);
+ TableConfig tc = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setFieldConfigList(List.of(fc))
+ .build();
+ assertValid(tc);
+ }
+
+ @Test
+ public void testJsonIndexStringColumnRawPasses() {
+ FieldConfig fc = new FieldConfig(STR_COL, EncodingType.RAW, IndexType.JSON, null, null);
+ TableConfig tc = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setNoDictionaryColumns(List.of(STR_COL))
+ .setFieldConfigList(List.of(fc))
+ .build();
+ assertValid(tc);
+ }
+
+ @Test
+ public void testJsonIndexStringColumnDictPasses() {
+ FieldConfig fc = new FieldConfig(STR_COL, EncodingType.DICTIONARY, IndexType.JSON, null, null);
+ TableConfig tc = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setFieldConfigList(List.of(fc))
+ .build();
+ assertValid(tc);
+ }
+
+ @Test
+ public void testJsonIndexIntColumnFails() {
+ // JSON index not valid on non-STRING/non-MAP column
+ FieldConfig fc = new FieldConfig(INT_COL, EncodingType.DICTIONARY, IndexType.JSON, null, null);
+ TableConfig tc = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setFieldConfigList(List.of(fc))
+ .build();
+ assertInvalid(tc, "Cannot create JSON index on column: " + INT_COL);
+ }
+
+ // ============================================================
+ // 11. Bloom filter combinations
+ // ============================================================
+
+ @Test
+ public void testBloomFilterDictEncodedPasses() {
+ TableConfig tc = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setBloomFilterColumns(List.of(INT_COL))
+ .build();
+ assertValid(tc);
+ }
+
+ @Test
+ public void testBloomFilterRawEncodedPasses() {
+ // Bloom filter does not require dictionary
+ TableConfig tc = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setNoDictionaryColumns(List.of(INT_COL))
+ .setBloomFilterColumns(List.of(INT_COL))
+ .build();
+ assertValid(tc);
+ }
+
+ @Test
+ public void testBloomFilterAllTypesRawPasses() {
+ // Bloom filter is valid on all types (except BOOLEAN) without dictionary
+ TableConfig tc = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setNoDictionaryColumns(Arrays.asList(STR_COL, INT_COL, FLOAT_COL))
+ .setBloomFilterColumns(Arrays.asList(STR_COL, INT_COL, FLOAT_COL))
+ .build();
+ assertValid(tc);
+ }
+
+ // ============================================================
+ // 12. Error message quality: FST index without dict must name the index type
+ // ============================================================
+
+ @Test
+ public void testErrorMessageNamesFstIndex() {
+ FieldConfig fc = new FieldConfig(STR_COL, EncodingType.RAW, IndexType.FST, null, null);
+ TableConfig tc = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setNoDictionaryColumns(List.of(STR_COL))
+ .setFieldConfigList(List.of(fc))
+ .build();
+ try {
+ TableConfigUtils.validate(tc, SCHEMA);
+ fail("Expected validation to fail");
+ } catch (Exception e) {
+ // FstIndexType.validate() fires: "Cannot create FST index on column: <col> without dictionary"
+ String msg = e.getMessage();
+ assertTrue(msg.contains(STR_COL), "Error should name the column");
+ assertTrue(msg.contains("without dictionary"), "Error should explain the problem");
+ }
+ }
+}
diff --git a/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/utils/TableConfigUtilsTest.java b/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/utils/TableConfigUtilsTest.java
index 4ddde893b71..b6eac19916d 100644
--- a/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/utils/TableConfigUtilsTest.java
+++ b/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/utils/TableConfigUtilsTest.java
@@ -1243,18 +1243,15 @@ public class TableConfigUtilsTest {
assertEquals(e.getMessage(), "FieldConfig encoding type is different from indexingConfig for column: myCol1");
}
- tableConfig = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
- .setNoDictionaryColumns(Arrays.asList("myCol1"))
- .build();
- try {
- FieldConfig fieldConfig =
- new FieldConfig("myCol1", FieldConfig.EncodingType.RAW, FieldConfig.IndexType.FST, null, null);
- tableConfig.setFieldConfigList(Arrays.asList(fieldConfig));
- TableConfigUtils.validate(tableConfig, schema);
- fail("Should fail since FST index is enabled on RAW encoding type");
- } catch (Exception e) {
- assertEquals(e.getMessage(), "Cannot create FST index on column: myCol1 without dictionary");
- }
+ // FST on RAW column is valid when explicit dictionary config is provided in indexes
+ tableConfig = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME).build();
+ ObjectNode rawFstIndexes = JsonUtils.newObjectNode();
+ rawFstIndexes.set("dictionary", JsonUtils.newObjectNode());
+ FieldConfig rawFstFieldConfig =
+ new FieldConfig("myCol1", FieldConfig.EncodingType.RAW, null, Arrays.asList(FieldConfig.IndexType.FST),
+ null, null, rawFstIndexes, null, null);
+ tableConfig.setFieldConfigList(Arrays.asList(rawFstFieldConfig));
+ TableConfigUtils.validate(tableConfig, schema);
tableConfig = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME).build();
try {
@@ -1464,19 +1461,16 @@ public class TableConfigUtilsTest {
assertEquals(e.getMessage(), "Cannot create inverted index on column: myCol2 without dictionary");
}
- // Tests the case when the field-config list marks a column as raw (non-dictionary) and enables
- // inverted index on it
+ // FieldConfig with encoding=RAW + INVERTED: dictionary is auto-enabled (INVERTED requires it),
+ // producing a shared-dict + RAW forward index. No validation error expected.
tableConfig = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME).build();
- try {
+ {
FieldConfig fieldConfig =
new FieldConfig.Builder("myCol2").withIndexTypes(Arrays.asList(FieldConfig.IndexType.INVERTED))
.withEncodingType(FieldConfig.EncodingType.RAW)
.build();
tableConfig.setFieldConfigList(Arrays.asList(fieldConfig));
TableConfigUtils.validate(tableConfig, schema);
- fail("Should not be able to disable dictionary but keep inverted index");
- } catch (Exception e) {
- assertEquals(e.getMessage(), "Cannot create inverted index on column: myCol2 without dictionary");
}
tableConfig = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
@@ -1486,14 +1480,17 @@ public class TableConfigUtilsTest {
// Enable forward index disabled flag for a column with FST index and disable dictionary
Map<String, String> fieldConfigProperties = new HashMap<>();
fieldConfigProperties.put(FieldConfig.FORWARD_INDEX_DISABLED, Boolean.TRUE.toString());
- FieldConfig fieldConfig =
+ FieldConfig fieldConfigWithFst =
new FieldConfig("myCol2", FieldConfig.EncodingType.RAW, FieldConfig.IndexType.FST, null, null, null,
fieldConfigProperties);
- tableConfig.setFieldConfigList(Arrays.asList(fieldConfig));
+ tableConfig.setFieldConfigList(Arrays.asList(fieldConfigWithFst));
TableConfigUtils.validate(tableConfig, schema);
- fail("Should not be able to disable dictionary but keep inverted index");
+ fail("Should not be able to disable dictionary but keep FST index");
} catch (Exception e) {
- assertEquals(e.getMessage(), "Cannot create FST index on column: myCol2 without dictionary");
+ // FST requires dictionary; validation may catch dictionary-required or FST-specific error
+ assertTrue(e.getMessage().contains("require a dictionary")
+ || e.getMessage().contains("without dictionary"),
+ "Expected dictionary-related error but got: " + e.getMessage());
}
tableConfig = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
@@ -1504,10 +1501,10 @@ public class TableConfigUtilsTest {
// Enable forward index disabled flag for a column with FST index and disable dictionary
Map<String, String> fieldConfigProperties = new HashMap<>();
fieldConfigProperties.put(FieldConfig.FORWARD_INDEX_DISABLED, Boolean.TRUE.toString());
- FieldConfig fieldConfig =
+ FieldConfig fieldConfigWithRange =
new FieldConfig("intCol", FieldConfig.EncodingType.RAW, FieldConfig.IndexType.RANGE, null, null, null,
fieldConfigProperties);
- tableConfig.setFieldConfigList(Arrays.asList(fieldConfig));
+ tableConfig.setFieldConfigList(Arrays.asList(fieldConfigWithRange));
TableConfigUtils.validate(tableConfig, schema);
} catch (Exception e) {
fail("Range index with forward index disabled no dictionary column is allowed");
@@ -1517,21 +1514,47 @@ public class TableConfigUtilsTest {
Map<String, String> streamConfigs = getStreamConfigs();
tableConfig = new TableConfigBuilder(TableType.REALTIME).setTableName(TABLE_NAME)
.setTimeColumnName(TIME_COLUMN)
- .setNoDictionaryColumns(Arrays.asList("intCol"))
.setStreamConfigs(streamConfigs)
.build();
try {
// Enable forward index disabled flag for a column with inverted index index and disable dictionary
Map<String, String> fieldConfigProperties = new HashMap<>();
fieldConfigProperties.put(FieldConfig.FORWARD_INDEX_DISABLED, Boolean.TRUE.toString());
- FieldConfig fieldConfig =
- new FieldConfig("intCol", FieldConfig.EncodingType.RAW, FieldConfig.IndexType.INVERTED, null, null, null,
- fieldConfigProperties);
- tableConfig.setFieldConfigList(Arrays.asList(fieldConfig));
+ ObjectNode indexes = JsonUtils.newObjectNode();
+ indexes.set("dictionary", JsonUtils.newObjectNode());
+ FieldConfig realtimeFieldConfig =
+ new FieldConfig("intCol", FieldConfig.EncodingType.RAW, null, Arrays.asList(FieldConfig.IndexType.INVERTED),
+ null, null, indexes, fieldConfigProperties, null);
+ tableConfig.setFieldConfigList(Arrays.asList(realtimeFieldConfig));
TableConfigUtils.validate(tableConfig, schema);
} catch (Exception e) {
assertEquals(e.getMessage(), "Cannot disable forward index for column: intCol, as the table type is REALTIME");
}
+
+ // Inverted index on a RAW column auto-enables the dictionary (see DictionaryIndexType.fromFieldConfigs)
+ // and validation passes — the runtime builds a shared-dict + RAW forward index.
+ tableConfig = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME).build();
+ {
+ FieldConfig rawInvertedConfig =
+ new FieldConfig("intCol", FieldConfig.EncodingType.RAW,
+ Arrays.asList(FieldConfig.IndexType.INVERTED), null, null);
+ tableConfig.setFieldConfigList(Arrays.asList(rawInvertedConfig));
+ TableConfigUtils.validate(tableConfig, schema);
+ }
+
+ // Validate that inverted index on RAW column WITH explicit dictionary passes validation
+ tableConfig = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME).build();
+ try {
+ ObjectNode dictIndexes = JsonUtils.newObjectNode();
+ dictIndexes.set("dictionary", JsonUtils.newObjectNode());
+ FieldConfig rawInvertedWithDictConfig =
+ new FieldConfig("intCol", FieldConfig.EncodingType.RAW, null,
+ Arrays.asList(FieldConfig.IndexType.INVERTED), null, null, dictIndexes, null, null);
+ tableConfig.setFieldConfigList(Arrays.asList(rawInvertedWithDictConfig));
+ TableConfigUtils.validate(tableConfig, schema);
+ } catch (Exception e) {
+ fail("Should pass since inverted index has explicit dictionary config, but got: " + e.getMessage());
+ }
}
@Test
diff --git a/pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/DictionaryIndexConfig.java b/pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/DictionaryIndexConfig.java
index 7983c86b66c..f92fc3c9793 100644
--- a/pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/DictionaryIndexConfig.java
+++ b/pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/DictionaryIndexConfig.java
@@ -148,7 +148,7 @@ public class DictionaryIndexConfig extends IndexConfig {
private static <C extends IndexConfig> boolean requiresDictionaryBy(IndexType<C, ?, ?> indexType,
FieldSpec fieldSpec, FieldIndexConfigs fieldIndexConfigs) {
C config = fieldIndexConfigs.getConfig(indexType);
- if (config == null || config.isDisabled()) {
+ if (config.isDisabled()) {
return false;
}
return indexType.requiresDictionary(fieldSpec, config);
diff --git a/pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/creator/ForwardIndexCreator.java b/pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/creator/ForwardIndexCreator.java
index 8de96458e0b..6c79a6a5e98 100644
--- a/pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/creator/ForwardIndexCreator.java
+++ b/pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/creator/ForwardIndexCreator.java
@@ -39,7 +39,7 @@ public interface ForwardIndexCreator extends IndexCreator {
@Override
default void add(Object cellValue, int dictId) {
- if (dictId >= 0) {
+ if (isDictionaryEncoded()) {
putDictId(dictId);
} else {
switch (getValueType()) {
@@ -86,7 +86,7 @@ public interface ForwardIndexCreator extends IndexCreator {
@Override
default void add(Object[] cellValues, @Nullable int[] dictIds)
throws IOException {
- if (dictIds != null) {
+ if (isDictionaryEncoded()) {
putDictIdMV(dictIds);
} else {
int length = cellValues.length;
@@ -170,7 +170,7 @@ public interface ForwardIndexCreator extends IndexCreator {
@Override
default void addInt(int value, int dictId)
throws IOException {
- if (dictId >= 0) {
+ if (isDictionaryEncoded()) {
putDictId(dictId);
} else {
putInt(value);
@@ -180,7 +180,7 @@ public interface ForwardIndexCreator extends IndexCreator {
@Override
default void addLong(long value, int dictId)
throws IOException {
- if (dictId >= 0) {
+ if (isDictionaryEncoded()) {
putDictId(dictId);
} else {
putLong(value);
@@ -190,7 +190,7 @@ public interface ForwardIndexCreator extends IndexCreator {
@Override
default void addFloat(float value, int dictId)
throws IOException {
- if (dictId >= 0) {
+ if (isDictionaryEncoded()) {
putDictId(dictId);
} else {
putFloat(value);
@@ -200,7 +200,7 @@ public interface ForwardIndexCreator extends IndexCreator {
@Override
default void addDouble(double value, int dictId)
throws IOException {
- if (dictId >= 0) {
+ if (isDictionaryEncoded()) {
putDictId(dictId);
} else {
putDouble(value);
@@ -210,7 +210,7 @@ public interface ForwardIndexCreator extends IndexCreator {
@Override
default void addString(String value, int dictId)
throws IOException {
- if (dictId >= 0) {
+ if (isDictionaryEncoded()) {
putDictId(dictId);
} else {
putString(value);
@@ -220,7 +220,7 @@ public interface ForwardIndexCreator extends IndexCreator {
@Override
default void addBytes(byte[] value, int dictId)
throws IOException {
- if (dictId >= 0) {
+ if (isDictionaryEncoded()) {
putDictId(dictId);
} else {
putBytes(value);
@@ -230,7 +230,7 @@ public interface ForwardIndexCreator extends IndexCreator {
@Override
default void addIntMV(int[] values, @Nullable int[] dictIds)
throws IOException {
- if (dictIds != null) {
+ if (isDictionaryEncoded()) {
putDictIdMV(dictIds);
} else {
putIntMV(values);
@@ -240,7 +240,7 @@ public interface ForwardIndexCreator extends IndexCreator {
@Override
default void addLongMV(long[] values, @Nullable int[] dictIds)
throws IOException {
- if (dictIds != null) {
+ if (isDictionaryEncoded()) {
putDictIdMV(dictIds);
} else {
putLongMV(values);
@@ -250,7 +250,7 @@ public interface ForwardIndexCreator extends IndexCreator {
@Override
default void addFloatMV(float[] values, @Nullable int[] dictIds)
throws IOException {
- if (dictIds != null) {
+ if (isDictionaryEncoded()) {
putDictIdMV(dictIds);
} else {
putFloatMV(values);
@@ -260,7 +260,7 @@ public interface ForwardIndexCreator extends IndexCreator {
@Override
default void addDoubleMV(double[] values, @Nullable int[] dictIds)
throws IOException {
- if (dictIds != null) {
+ if (isDictionaryEncoded()) {
putDictIdMV(dictIds);
} else {
putDoubleMV(values);
@@ -270,7 +270,7 @@ public interface ForwardIndexCreator extends IndexCreator {
@Override
default void addBigDecimalMV(BigDecimal[] values, @Nullable int[] dictIds)
throws IOException {
- if (dictIds != null) {
+ if (isDictionaryEncoded()) {
putDictIdMV(dictIds);
} else {
putBigDecimalMV(values);
@@ -280,7 +280,7 @@ public interface ForwardIndexCreator extends IndexCreator {
@Override
default void addStringMV(String[] values, @Nullable int[] dictIds)
throws IOException {
- if (dictIds != null) {
+ if (isDictionaryEncoded()) {
putDictIdMV(dictIds);
} else {
putStringMV(values);
@@ -290,7 +290,7 @@ public interface ForwardIndexCreator extends IndexCreator {
@Override
default void addBytesMV(byte[][] values, @Nullable int[] dictIds)
throws IOException {
- if (dictIds != null) {
+ if (isDictionaryEncoded()) {
putDictIdMV(dictIds);
} else {
putBytesMV(values);
diff --git a/pinot-segment-spi/src/test/java/org/apache/pinot/segment/spi/index/metadata/ColumnMetadataImplTest.java b/pinot-segment-spi/src/test/java/org/apache/pinot/segment/spi/index/metadata/ColumnMetadataImplTest.java
new file mode 100644
index 00000000000..0b994799933
--- /dev/null
+++ b/pinot-segment-spi/src/test/java/org/apache/pinot/segment/spi/index/metadata/ColumnMetadataImplTest.java
@@ -0,0 +1,118 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.index.metadata;
+
+import org.apache.commons.configuration2.PropertiesConfiguration;
+import org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column;
+import org.apache.pinot.spi.config.table.FieldConfig.EncodingType;
+import org.apache.pinot.spi.data.FieldSpec.DataType;
+import org.apache.pinot.spi.data.FieldSpec.FieldType;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertTrue;
+
+
+/**
+ * Unit tests for {@link ColumnMetadataImpl#fromPropertiesConfiguration} focused on the
+ * {@code FORWARD_INDEX_ENCODING} property's rolling-upgrade behavior.
+ *
+ * <p>{@code FORWARD_INDEX_ENCODING} was added in this release. Old segments built before this release won't have
+ * the key in {@code metadata.properties}, so {@link ColumnMetadataImpl#fromPropertiesConfiguration} falls back to
+ * deriving the encoding from {@code HAS_DICTIONARY}: dict means {@code DICTIONARY}-encoded forward, no dict means
+ * {@code RAW}. The new "shared dictionary on RAW forward" segment shape is only representable when the key is
+ * explicitly written by the new segment creator.
+ */
+public class ColumnMetadataImplTest {
+
+ /**
+ * Old-segment fallback path: no FORWARD_INDEX_ENCODING in metadata, dict present → encoding inferred as DICTIONARY.
+ */
+ @Test
+ public void fallsBackToDictionaryEncodingWhenKeyAbsentAndHasDictionary() {
+ PropertiesConfiguration config = baseConfig("col");
+ config.setProperty(Column.getKeyFor("col", Column.HAS_DICTIONARY), true);
+ // FORWARD_INDEX_ENCODING intentionally NOT set, simulating an old segment.
+
+ ColumnMetadataImpl metadata = ColumnMetadataImpl.fromPropertiesConfiguration(config, 1, "col");
+
+ assertTrue(metadata.hasDictionary());
+ assertEquals(metadata.getForwardIndexEncoding(), EncodingType.DICTIONARY,
+ "Old segments without FORWARD_INDEX_ENCODING and HAS_DICTIONARY=true must infer DICTIONARY encoding");
+ }
+
+ /**
+ * Old-segment fallback path: no FORWARD_INDEX_ENCODING in metadata, no dict → encoding inferred as RAW.
+ */
+ @Test
+ public void fallsBackToRawEncodingWhenKeyAbsentAndNoDictionary() {
+ PropertiesConfiguration config = baseConfig("col");
+ config.setProperty(Column.getKeyFor("col", Column.HAS_DICTIONARY), false);
+ // FORWARD_INDEX_ENCODING intentionally NOT set, simulating an old raw-forward segment.
+
+ ColumnMetadataImpl metadata = ColumnMetadataImpl.fromPropertiesConfiguration(config, 1, "col");
+
+ assertFalse(metadata.hasDictionary());
+ assertEquals(metadata.getForwardIndexEncoding(), EncodingType.RAW,
+ "Old segments without FORWARD_INDEX_ENCODING and HAS_DICTIONARY=false must infer RAW encoding");
+ }
+
+ /**
+ * New shared-dict shape: FORWARD_INDEX_ENCODING=RAW + HAS_DICTIONARY=true. The new segment creator writes both
+ * keys; the metadata loader must honor the explicit FORWARD_INDEX_ENCODING and not fall back to inference.
+ */
+ @Test
+ public void honorsExplicitRawEncodingEvenWhenHasDictionary() {
+ PropertiesConfiguration config = baseConfig("col");
+ config.setProperty(Column.getKeyFor("col", Column.HAS_DICTIONARY), true);
+ config.setProperty(Column.getKeyFor("col", Column.FORWARD_INDEX_ENCODING), EncodingType.RAW.name());
+
+ ColumnMetadataImpl metadata = ColumnMetadataImpl.fromPropertiesConfiguration(config, 1, "col");
+
+ assertTrue(metadata.hasDictionary());
+ assertEquals(metadata.getForwardIndexEncoding(), EncodingType.RAW,
+ "Explicit FORWARD_INDEX_ENCODING=RAW must override inference even when HAS_DICTIONARY=true (shared-dict)");
+ }
+
+ /**
+ * New segment with explicit FORWARD_INDEX_ENCODING=DICTIONARY; verify it round-trips.
+ */
+ @Test
+ public void honorsExplicitDictionaryEncoding() {
+ PropertiesConfiguration config = baseConfig("col");
+ config.setProperty(Column.getKeyFor("col", Column.HAS_DICTIONARY), true);
+ config.setProperty(Column.getKeyFor("col", Column.FORWARD_INDEX_ENCODING), EncodingType.DICTIONARY.name());
+
+ ColumnMetadataImpl metadata = ColumnMetadataImpl.fromPropertiesConfiguration(config, 1, "col");
+
+ assertTrue(metadata.hasDictionary());
+ assertEquals(metadata.getForwardIndexEncoding(), EncodingType.DICTIONARY);
+ }
+
+ private static PropertiesConfiguration baseConfig(String column) {
+ PropertiesConfiguration config = new PropertiesConfiguration();
+ config.setProperty(Column.getKeyFor(column, Column.COLUMN_NAME), column);
+ config.setProperty(Column.getKeyFor(column, Column.COLUMN_TYPE), FieldType.DIMENSION.name());
+ config.setProperty(Column.getKeyFor(column, Column.DATA_TYPE), DataType.STRING.name());
+ config.setProperty(Column.getKeyFor(column, Column.IS_SINGLE_VALUED), true);
+ config.setProperty(Column.getKeyFor(column, Column.CARDINALITY), 1);
+ return config;
+ }
+}
diff --git a/pinot-tools/src/main/resources/examples/batch/githubComplexTypeEvents/githubComplexTypeEvents_offline_table_config.json b/pinot-tools/src/main/resources/examples/batch/githubComplexTypeEvents/githubComplexTypeEvents_offline_table_config.json
index 031be26219f..77c4f10c53e 100644
--- a/pinot-tools/src/main/resources/examples/batch/githubComplexTypeEvents/githubComplexTypeEvents_offline_table_config.json
+++ b/pinot-tools/src/main/resources/examples/batch/githubComplexTypeEvents/githubComplexTypeEvents_offline_table_config.json
@@ -11,6 +11,18 @@
"tableIndexConfig": {
"loadMode": "MMAP"
},
+ "fieldConfigList": [
+ {
+ "name": "commits_url",
+ "encodingType": "RAW",
+ "indexes": {
+ "dictionary": {},
+ "inverted": {
+ "enabled": "true"
+ }
+ }
+ }
+ ],
"ingestionConfig": {
"transformConfigs": [
{
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org