You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "xiangfu0 (via GitHub)" <gi...@apache.org> on 2026/04/28 23:43:16 UTC
[PR] Add forward-index encoding to ForwardIndexConfig SPI (Phase 1) [pinot]
xiangfu0 opened a new pull request, #18364:
URL: https://github.com/apache/pinot/pull/18364
## Summary
Adds the SPI surface needed to track forward-index encoding (`RAW` vs `DICTIONARY`) explicitly on `ForwardIndexConfig`, decoupled from dictionary presence. **Pure additive — no existing callers updated, no behavior change.**
This is **Phase 1** of a multi-phase plan to enable shared-dictionary segments (a column may keep a dictionary purely for secondary indexes — inverted, FST, range — while still using a raw forward index). Subsequent phases will wire callers to read the encoding from the new accessor and add the corresponding column-metadata key. Splitting the SPI surface from the consumer/feature work keeps later PRs reviewable.
## What changes
- **Reuse existing enum** — `FieldConfig.EncodingType` (`RAW`, `DICTIONARY`) is the canonical encoding enum on the public SPI; this PR does not introduce a parallel enum.
- **`ForwardIndexConfig`:**
- New `DEFAULT_FORWARD_INDEX_ENCODING` constant (= `EncodingType.DICTIONARY`).
- New `getForwardIndexEncoding()` getter, included in `equals`/`hashCode` and JSON round-trip.
- New `Builder.withForwardIndexEncoding(@Nullable EncodingType)` setter; copy-constructor preserves the field.
- New `@JsonCreator` (10-arg) extends the prior 9-arg form; the prior 9-arg public constructor is preserved as `@Deprecated` so plugin authors that constructed `ForwardIndexConfig` directly continue to compile and link.
- `getDisabled()` factory: documented that `getForwardIndexEncoding()` is reported but has no observable effect when the forward index is disabled.
- **`IndexCreationContext`:**
- New `getForwardIndexConfig()` interface method as a `default` returning `null` (existing implementers do not need to change).
- New `Builder.withForwardIndexConfig(...)` setter.
- The `@Nullable` contract is documented; the Javadoc also notes there is no in-tree consumer in this PR.
## Backward compatibility
- All existing public constructors of `ForwardIndexConfig` are preserved (the 9-arg form is now `@Deprecated` but kept for binary/source compat).
- The new `IndexCreationContext.getForwardIndexConfig()` is a `default` method returning `null`, so plugin implementers do not break.
- JSON deserialization of pre-existing payloads is preserved (the new field is optional; `null` is coalesced to `DEFAULT_FORWARD_INDEX_ENCODING`).
- `equals`/`hashCode` semantics: two configs differing only in encoding are not equal — intentional, since `RAW` vs `DICTIONARY` is a real semantic distinction.
## Test plan
- [x] `pinot-segment-spi` builds clean (main + test).
- [x] Downstream `pinot-segment-local`, `pinot-core`, `pinot-server`, `pinot-broker` compile clean against the new SPI.
- [x] `IndexCreationContextTest`: encoding flows through `ForwardIndexConfig`; `getForwardIndexConfig()` returns null when not set.
- [x] `ForwardIndexConfigTest`: default encoding, builder set/copy, equals/hashCode distinction, JSON round-trip (both explicit field and absent field).
- [x] `spotless:apply` / `checkstyle:check` / `license:check` all pass on `pinot-segment-spi`.
## Notes for plugin maintainers
This change touches `pinot-segment-spi`. Plugin authors do **not** need to update anything (the new interface method is `default`, the new constructor is additive, the deprecated 9-arg constructor is preserved). Subsequent phases will adopt the new accessor in `ForwardIndexCreatorFactory` and elsewhere.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add forward-index encoding to ForwardIndexConfig SPI (Phase 1) [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18364:
URL: https://github.com/apache/pinot/pull/18364#issuecomment-4340064671
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/18364?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `63.63636%` with `8 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 55.43%. Comparing base ([`9b2c3dc`](https://app.codecov.io/gh/apache/pinot/commit/9b2c3dc49e61d0c87c23785079d2b7395970bdf2?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`5994d2a`](https://app.codecov.io/gh/apache/pinot/commit/5994d2a4f0358230ccf10d415ecf5f0a09519853?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 1 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/pinot/pull/18364?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...he/pinot/segment/spi/index/ForwardIndexConfig.java](https://app.codecov.io/gh/apache/pinot/pull/18364?src=pr&el=tree&filepath=pinot-segment-spi%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Fspi%2Findex%2FForwardIndexConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2luZGV4L0ZvcndhcmRJbmRleENvbmZpZy5qYXZh) | 56.25% | [6 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/pinot/pull/18364?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...inot/segment/spi/creator/IndexCreationContext.java](https://app.codecov.io/gh/apache/pinot/pull/18364?src=pr&el=tree&filepath=pinot-segment-spi%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Fspi%2Fcreator%2FIndexCreationContext.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2NyZWF0b3IvSW5kZXhDcmVhdGlvbkNvbnRleHQuamF2YQ==) | 83.33% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/18364?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (9b2c3dc) and HEAD (5994d2a). Click for more details.
>
> <details><summary>HEAD has 16 uploads less than BASE</summary>
>
>| Flag | BASE (9b2c3dc) | HEAD (5994d2a) |
>|------|------|------|
>|java-21|5|1|
>|unittests|2|1|
>|temurin|5|1|
>|unittests2|1|0|
>|integration|3|0|
>|integration1|1|0|
>|integration2|1|0|
>|custom-integration1|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18364 +/- ##
============================================
- Coverage 63.42% 55.43% -7.99%
+ Complexity 1683 846 -837
============================================
Files 3253 2543 -710
Lines 198833 147390 -51443
Branches 30794 23736 -7058
============================================
- Hits 126106 81704 -44402
+ Misses 62643 58679 -3964
+ Partials 10084 7007 -3077
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/18364/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/18364/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/18364/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/18364/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/18364/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-21](https://app.codecov.io/gh/apache/pinot/pull/18364/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.43% <63.63%> (-7.99%)` | :arrow_down: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/18364/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.43% <63.63%> (-7.99%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/18364/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.43% <63.63%> (-7.99%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/18364/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.43% <63.63%> (+0.08%)` | :arrow_up: |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/18364/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/18364?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Forward-index encoding: introduce Encoding SPI surface and use it for raw vs dict checks [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #18364:
URL: https://github.com/apache/pinot/pull/18364#discussion_r3164331729
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/forward/ForwardIndexCreatorFactory.java:
##########
@@ -52,8 +52,11 @@ public static ForwardIndexCreator createIndexCreator(IndexCreationContext contex
String columnName = fieldSpec.getName();
int numTotalDocs = context.getTotalDocs();
- if (context.hasDictionary()) {
- // Dictionary enabled columns
+ // The forward-index encoding is carried on ForwardIndexConfig. A column may have a dictionary purely for use by
+ // secondary indexes (inverted, FST, range) while still using a raw forward index — that is, hasDictionary() does
+ // not by itself imply a dictionary-encoded forward index. Both signals must agree before we pick the dict path.
+ if (context.hasDictionary() && indexConfig.getForwardIndexEncoding() == ForwardIndexConfig.Encoding.DICTIONARY) {
Review Comment:
Do we need the first check? I'd put that as an assert to make sure the wiring is correct.
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/ForwardIndexConfig.java:
##########
@@ -35,6 +35,22 @@
public class ForwardIndexConfig extends IndexConfig {
+ /**
+ * Forward-index encoding for a column.
+ *
+ * <p>Tracked separately from dictionary presence because a column may keep a dictionary purely for secondary
+ * indexes (inverted, FST, range) while still using a raw forward index.
+ */
+ public enum Encoding {
Review Comment:
This enum already exists in `FieldConfig.EncodingType`. It should already be available, just wire it in.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/BaseSegmentCreator.java:
##########
@@ -546,15 +554,36 @@ protected void writeMetadata()
/**
* Adds column metadata information to the properties configuration.
+ *
Review Comment:
When modifying a javadoc, change it to markdown style, which is much easier to read. Use `` for code, [] for link.
Same for other javadocs
##########
pinot-core/src/test/java/org/apache/pinot/core/util/CrcUtilsTest.java:
##########
@@ -105,10 +105,11 @@ public void testCrc()
driver.build();
File indexDir = driver.getOutputDirectory();
- assertEquals(CrcUtils.computeCrc(indexDir), 1243018203L);
+ // CRC values updated after introducing the FORWARD_INDEX_ENCODING column metadata key.
Review Comment:
(nit) No need to add this comment. Any update to metadata will change the crc
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/IndexCreationContext.java:
##########
@@ -331,14 +349,15 @@ final class Common implements IndexCreationContext {
private final int[] _mutableToImmutableDocIdMap;
private final String _tableNameWithType;
private final boolean _continueOnError;
+ private final ForwardIndexConfig _forwardIndexConfig;
Review Comment:
Do we need the entire `ForwardIndexConfig` here or only the encoding type?
If we need the entire `ForwardIndexConfig`, remove the overlapping configs such as `_forwardIndexDisabled`
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/ForwardIndexConfig.java:
##########
@@ -91,23 +114,61 @@ public static ForwardIndexConfig getDisabled() {
private final DictIdCompressionType _dictIdCompressionType;
@Nullable
private final Map<String, Object> _configs;
+ // Always non-null — defaulted to DEFAULT_FORWARD_INDEX_ENCODING when no explicit value is provided.
+ // Mirrors the established pattern of FieldConfig._encodingType (which also coalesces null to DICTIONARY).
+ private final Encoding _forwardIndexEncoding;
Review Comment:
I'd put this as the first member variable. This is the top level branch
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/forward/ForwardIndexCreatorFactory.java:
##########
@@ -70,7 +73,7 @@ public static ForwardIndexCreator createIndexCreator(IndexCreationContext contex
}
}
} else {
- // Dictionary disabled columns
+ // Raw forward index — either no dictionary at all, or shared dictionary (kept for secondary indexes only)
Review Comment:
(nit) Shared dictionary is confusing. I'd only call out it is raw encoding
```suggestion
// Raw forward index
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Forward-index encoding: introduce Encoding SPI surface and use it for raw vs dict checks [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on PR #18364:
URL: https://github.com/apache/pinot/pull/18364#issuecomment-4346865848
Pushed c0ad3ebdec to address CI failures:
**Pinot Binary Compatibility Check** — `japicmp` flagged the removal of the previous public 7-arg and 9-arg `ForwardIndexConfig` constructors. Restored both as `@Deprecated` delegates that forward to the canonical 10-arg `@JsonCreator`. Plugin authors that linked against either old constructor continue to compile and link unchanged; new code should use the `Builder` or the canonical `@JsonCreator`.
**Pinot Unit Test Set 2** — `RealtimeSegmentConverterTest.testOptimizeDictionaryTypeConversion` failed with the expected segment CRC mismatched against the actual: same root cause as the `CrcUtilsTest` fixes already in this PR — the new `FORWARD_INDEX_ENCODING` metadata key changes segment CRC. Updated the two expected CRCs in the test's data provider to the new values.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Forward-index encoding: introduce Encoding SPI surface and use it for raw vs dict checks [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #18364:
URL: https://github.com/apache/pinot/pull/18364#discussion_r3159665360
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/BaseSegmentCreator.java:
##########
@@ -555,6 +555,13 @@ public static void addColumnMetadataInfo(PropertiesConfiguration properties, Str
int cardinality = columnStatistics.getCardinality();
properties.setProperty(getKeyFor(column, CARDINALITY), String.valueOf(cardinality));
properties.setProperty(getKeyFor(column, HAS_DICTIONARY), String.valueOf(hasDictionary));
+ // Persist the forward-index encoding alongside HAS_DICTIONARY so future readers can distinguish columns that
+ // keep a dictionary purely for secondary indexes (encoding=RAW, hasDictionary=true) from the standard
+ // dictionary-encoded forward index (encoding=DICTIONARY, hasDictionary=true). Until shared-dictionary segments
+ // are produced, encoding tracks hasDictionary 1:1 — readers fall back to deriving from HAS_DICTIONARY when
+ // FORWARD_INDEX_ENCODING is absent (old segments).
+ properties.setProperty(getKeyFor(column, FORWARD_INDEX_ENCODING),
+ (hasDictionary ? ForwardIndexConfig.Encoding.DICTIONARY : ForwardIndexConfig.Encoding.RAW).name());
Review Comment:
Fixed in ac70b644a5. The caller () now computes the encoding from the configured `ForwardIndexConfig` (`hasDictionary ? fwdConfig.getForwardIndexEncoding() : RAW`) and passes it to a new `addColumnMetadataInfo` overload that takes the encoding explicitly. The 7-arg overload is preserved and delegates with encoding derived from `hasDictionary` for backward compatibility with external callers.
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/V1Constants.java:
##########
@@ -138,6 +138,10 @@ public static class Column {
public static final String CARDINALITY = "cardinality";
// Mandatory, treated as `true` when missing for backward compatibility
public static final String HAS_DICTIONARY = "hasDictionary";
+ // Optional. Records the forward-index encoding (RAW or DICTIONARY) chosen at segment creation time. Old
+ // segments written before this key was introduced do not have it; readers fall back to deriving the encoding
+ // from HAS_DICTIONARY for backward compatibility.
+ public static final String FORWARD_INDEX_ENCODING = "forwardIndexEncoding";
Review Comment:
Done in ac70b644a5 — comment updated to "Mandatory for new segments. ... Old segments written before this key was introduced do not have it; readers fall back to inferring the encoding from HAS_DICTIONARY in that case."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Forward-index encoding: introduce Encoding SPI surface and use it for raw vs dict checks [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #18364:
URL: https://github.com/apache/pinot/pull/18364#discussion_r3167794070
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/ColumnMetadata.java:
##########
@@ -62,10 +63,29 @@ default boolean isSingleValue() {
/// [Constants#UNKNOWN_CARDINALITY].
int getCardinality();
- /// Returns `true` when the column is dictionary-encoded, `false` otherwise.
+ /// Returns `true` when the column has a dictionary, `false` otherwise.
+ ///
+ /// NOTE: dictionary presence is no longer the same as "forward index is dictionary-encoded". A column may keep a
Review Comment:
This contract change is not safe on its own because the common read path still treats a non-null dictionary as "forward index is dictionary-encoded". `PinotSegmentColumnReader` grabs `dataSource.getDictionary()` and then calls `getDictId*()` whenever that dictionary is present, so a shared-dictionary RAW segment will start interpreting raw values as dictIds. That affects record readers, lazy rows, upsert comparison/delete-column scans, star-tree builders, etc. before any of them consult `isForwardIndexDictionaryEncoded()`. The PR needs to update those readers (or hide the auxiliary dictionary from raw-forward datasources) before `hasDictionary()` can mean "dictionary exists for secondary indexes".
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexConfigUtils.java:
##########
@@ -0,0 +1,45 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import org.apache.pinot.segment.spi.index.ForwardIndexConfig;
+import org.apache.pinot.spi.config.table.FieldConfig;
+
+
+/**
+ * Utilities for normalizing forward-index configs in segment reload/rebuild paths.
+ */
+public final class ForwardIndexConfigUtils {
+ private ForwardIndexConfigUtils() {
+ }
+
+ public static ForwardIndexConfig withEncoding(ForwardIndexConfig config, FieldConfig.EncodingType encoding) {
+ if (config.isEnabled() && config.getForwardIndexEncoding() == encoding) {
+ return config;
+ }
+ return new ForwardIndexConfig.Builder(config)
+ .withForwardIndexEncoding(encoding)
+ .build();
+ }
+
+ public static ForwardIndexConfig forDictionaryState(ForwardIndexConfig config, boolean dictionaryEnabled) {
Review Comment:
This helper collapses two states that the PR just separated. For shared-dictionary columns the table config can legitimately be `dictionaryEnabled == true` and `forwardIndexEncoding == RAW`; every call site that goes through `forDictionaryState(...)` rewrites that explicit RAW choice back to DICTIONARY, so reload / temporary-forward-index regeneration silently changes the on-disk encoding. The normalization here should preserve `config.getForwardIndexEncoding()` for enabled forward indexes and only derive from dictionary state when the encoding is actually absent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Forward-index encoding: introduce Encoding SPI surface and use it for raw vs dict checks [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #18364:
URL: https://github.com/apache/pinot/pull/18364#discussion_r3159606715
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/V1Constants.java:
##########
@@ -138,6 +138,10 @@ public static class Column {
public static final String CARDINALITY = "cardinality";
// Mandatory, treated as `true` when missing for backward compatibility
public static final String HAS_DICTIONARY = "hasDictionary";
+ // Optional. Records the forward-index encoding (RAW or DICTIONARY) chosen at segment creation time. Old
+ // segments written before this key was introduced do not have it; readers fall back to deriving the encoding
+ // from HAS_DICTIONARY for backward compatibility.
+ public static final String FORWARD_INDEX_ENCODING = "forwardIndexEncoding";
Review Comment:
we should make this Mandatory. For reading old segments, we can infer from the `hasDictionary`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Forward-index encoding: introduce Encoding SPI surface and use it for raw vs dict checks [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #18364:
URL: https://github.com/apache/pinot/pull/18364#discussion_r3159574939
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/BaseSegmentCreator.java:
##########
@@ -555,6 +555,13 @@ public static void addColumnMetadataInfo(PropertiesConfiguration properties, Str
int cardinality = columnStatistics.getCardinality();
properties.setProperty(getKeyFor(column, CARDINALITY), String.valueOf(cardinality));
properties.setProperty(getKeyFor(column, HAS_DICTIONARY), String.valueOf(hasDictionary));
+ // Persist the forward-index encoding alongside HAS_DICTIONARY so future readers can distinguish columns that
+ // keep a dictionary purely for secondary indexes (encoding=RAW, hasDictionary=true) from the standard
+ // dictionary-encoded forward index (encoding=DICTIONARY, hasDictionary=true). Until shared-dictionary segments
+ // are produced, encoding tracks hasDictionary 1:1 — readers fall back to deriving from HAS_DICTIONARY when
+ // FORWARD_INDEX_ENCODING is absent (old segments).
+ properties.setProperty(getKeyFor(column, FORWARD_INDEX_ENCODING),
+ (hasDictionary ? ForwardIndexConfig.Encoding.DICTIONARY : ForwardIndexConfig.Encoding.RAW).name());
Review Comment:
this is wrong, we should read from the forwardIndexConfig, fallback to use `hasDictionary` to check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Forward-index encoding: introduce Encoding SPI surface and use it for raw vs dict checks [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #18364:
URL: https://github.com/apache/pinot/pull/18364#discussion_r3164965640
##########
pinot-core/src/test/java/org/apache/pinot/core/util/CrcUtilsTest.java:
##########
@@ -105,10 +105,11 @@ public void testCrc()
driver.build();
File indexDir = driver.getOutputDirectory();
- assertEquals(CrcUtils.computeCrc(indexDir), 1243018203L);
+ // CRC values updated after introducing the FORWARD_INDEX_ENCODING column metadata key.
Review Comment:
Done in 81f8dcdc97 — comment removed.
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/ForwardIndexConfig.java:
##########
@@ -35,6 +35,22 @@
public class ForwardIndexConfig extends IndexConfig {
+ /**
+ * Forward-index encoding for a column.
+ *
+ * <p>Tracked separately from dictionary presence because a column may keep a dictionary purely for secondary
+ * indexes (inverted, FST, range) while still using a raw forward index.
+ */
+ public enum Encoding {
Review Comment:
Done in 81f8dcdc97 — switched to `FieldConfig.EncodingType` everywhere; removed the duplicate `ForwardIndexConfig.Encoding` enum. Builder, getter, member field, default constant, and JSON deserialization all use `FieldConfig.EncodingType` now.
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/ForwardIndexConfig.java:
##########
@@ -91,23 +114,61 @@ public static ForwardIndexConfig getDisabled() {
private final DictIdCompressionType _dictIdCompressionType;
@Nullable
private final Map<String, Object> _configs;
+ // Always non-null — defaulted to DEFAULT_FORWARD_INDEX_ENCODING when no explicit value is provided.
+ // Mirrors the established pattern of FieldConfig._encodingType (which also coalesces null to DICTIONARY).
+ private final Encoding _forwardIndexEncoding;
Review Comment:
Done in 81f8dcdc97 — `_forwardIndexEncoding` is now the first member variable in both the class body and the `Builder`, with a comment explaining it's the top-level branch. The constructor body initializes it before any compression-related fields.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/BaseSegmentCreator.java:
##########
@@ -546,15 +554,36 @@ protected void writeMetadata()
/**
* Adds column metadata information to the properties configuration.
+ *
Review Comment:
Done in 81f8dcdc97 — converted javadocs on touched APIs (`getDisabled`, `getActualCompressionCodec`, the deprecated constructors, both `addColumnMetadataInfo` overloads, plus the previously-existing javadocs on `ColumnMetadata`/`IndexCreationContext` accessors) to the markdown `///` style with backticks for code and `[]` for links.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/forward/ForwardIndexCreatorFactory.java:
##########
@@ -70,7 +73,7 @@ public static ForwardIndexCreator createIndexCreator(IndexCreationContext contex
}
}
} else {
- // Dictionary disabled columns
+ // Raw forward index — either no dictionary at all, or shared dictionary (kept for secondary indexes only)
Review Comment:
Done in 81f8dcdc97 — comment is now just `// Raw forward index`.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/forward/ForwardIndexCreatorFactory.java:
##########
@@ -52,8 +52,11 @@ public static ForwardIndexCreator createIndexCreator(IndexCreationContext contex
String columnName = fieldSpec.getName();
int numTotalDocs = context.getTotalDocs();
- if (context.hasDictionary()) {
- // Dictionary enabled columns
+ // The forward-index encoding is carried on ForwardIndexConfig. A column may have a dictionary purely for use by
+ // secondary indexes (inverted, FST, range) while still using a raw forward index — that is, hasDictionary() does
+ // not by itself imply a dictionary-encoded forward index. Both signals must agree before we pick the dict path.
+ if (context.hasDictionary() && indexConfig.getForwardIndexEncoding() == ForwardIndexConfig.Encoding.DICTIONARY) {
Review Comment:
I tried this initially (kept just `encoding == DICTIONARY`, with an `assert context.hasDictionary()`), but `ForwardIndexHandlerTest.setUp` immediately tripped the assert: existing legacy callers build a default `ForwardIndexConfig` (encoding=DICTIONARY) for columns that end up with no dictionary at segment build time (e.g. `DIM_MV_PASS_THROUGH_INTEGER` in that test). The `hasDictionary()` check is load-bearing for those callers; both signals must agree before we pick the dict path. Restored both checks with a comment in 81f8dcdc97 explaining why the AND is needed.
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/IndexCreationContext.java:
##########
@@ -331,14 +349,15 @@ final class Common implements IndexCreationContext {
private final int[] _mutableToImmutableDocIdMap;
private final String _tableNameWithType;
private final boolean _continueOnError;
+ private final ForwardIndexConfig _forwardIndexConfig;
Review Comment:
Done in 81f8dcdc97 — kept the entire `ForwardIndexConfig` (consumers like `ForwardIndexCreatorFactory` need the full config: encoding, compressionCodec, targetMaxChunkSize, etc.) and removed the overlapping `_forwardIndexDisabled` field/getter/setter from `IndexCreationContext`. `forwardIndexDisabled()` is now a default method that delegates to `getForwardIndexConfig().isDisabled()`. Updated `BaseSegmentCreator` and `InvertedIndexAndDictionaryBasedForwardIndexCreator` to call `withForwardIndexConfig(...)` instead of `withForwardIndexDisabled(...)`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Forward-index encoding: introduce Encoding SPI surface and use it for raw vs dict checks [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 merged PR #18364:
URL: https://github.com/apache/pinot/pull/18364
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Forward-index encoding: introduce Encoding SPI surface and use it for raw vs dict checks [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #18364:
URL: https://github.com/apache/pinot/pull/18364#discussion_r3177724574
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/BaseSegmentCreator.java:
##########
@@ -531,9 +545,16 @@ protected void writeMetadata()
String column = entry.getKey();
ColumnStatistics columnStatistics = entry.getValue();
SegmentDictionaryCreator dictionaryCreator = _colIndexes.get(column).getDictionaryCreator();
- int dictionaryElementSize = (dictionaryCreator != null) ? dictionaryCreator.getNumBytesPerEntry() : 0;
+ boolean hasDictionary = dictionaryCreator != null;
+ int dictionaryElementSize = hasDictionary ? dictionaryCreator.getNumBytesPerEntry() : 0;
+ ForwardIndexConfig fwdConfig =
Review Comment:
(MAJOR) This config is from the original config, not the adapted one (line 177). We need to pass the adapted one. You may pass the adapted `FieldIndexConfigs` through `ColumnIndexCreators`
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/forward/ForwardIndexCreatorFactory.java:
##########
@@ -52,25 +52,8 @@ public static ForwardIndexCreator createIndexCreator(IndexCreationContext contex
String columnName = fieldSpec.getName();
int numTotalDocs = context.getTotalDocs();
- if (context.hasDictionary()) {
- // Dictionary enabled columns
- int cardinality = context.getCardinality();
- if (fieldSpec.isSingleValueField()) {
- if (context.isSorted()) {
- return new SingleValueSortedForwardIndexCreator(indexDir, columnName, cardinality);
- } else {
- return new SingleValueUnsortedForwardIndexCreator(indexDir, columnName, cardinality, numTotalDocs);
- }
- } else {
- if (indexConfig.getDictIdCompressionType() == DictIdCompressionType.MV_ENTRY_DICT) {
- return new MultiValueEntryDictForwardIndexCreator(indexDir, columnName, cardinality, numTotalDocs);
- } else {
- return new MultiValueUnsortedForwardIndexCreator(indexDir, columnName, cardinality, numTotalDocs,
- context.getTotalNumberOfEntries());
- }
- }
- } else {
- // Dictionary disabled columns
+ if (indexConfig.getEncodingType() == FieldConfig.EncodingType.RAW) {
Review Comment:
(minor) Keep the order as the original code by checking `FieldConfig.EncodingType.DICTIONARY` here
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/forward/ForwardIndexType.java:
##########
@@ -176,21 +220,40 @@ protected ColumnConfigDeserializer<ForwardIndexConfig> createDeserializerForLega
Maps.newHashMapWithExpectedSize(Math.max(dictConfigs.size(), schema.size()));
Collection<FieldConfig> fieldConfigs = tableConfig.getFieldConfigList();
+ Set<String> columnsWithExplicitForwardIndex = new HashSet<>();
if (fieldConfigs != null) {
for (FieldConfig fieldConfig : fieldConfigs) {
+ if (fieldConfig.getIndexes().has(INDEX_DISPLAY_NAME)) {
Review Comment:
This is incorrect. We want to detect conflicting config
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/forward/ForwardIndexType.java:
##########
@@ -166,6 +179,37 @@ public String getPrettyName() {
return INDEX_DISPLAY_NAME;
}
+ @Override
+ protected ColumnConfigDeserializer<ForwardIndexConfig> createDeserializer() {
+ // Override the default `fromIndexes` deserializer so that legacy `indexes.forward` JSON blocks that lack
Review Comment:
(MAJOR) This part of the code has become very messy, with a lot of cross reference between `createDeserializer()` and `createDeserializerForLegacyConfigs()`, and I cannot really tell if the logic is correct.
We don't need to follow the way of `fromIndexes.withExclusiveAlternative(createDeserializerForLegacyConfigs())`. For forward index, it is much more clear if we directly implement `createDeserializer`. We need to check the following properties:
- IndexingConfig.getNoDictionaryColumns()
- IndexingConfig.getNoDictionaryConfig()
- FieldConfig.getEncodingType()
- FieldConfig.getCompressionCodec()
- FieldConfig.getIndexes()
Within the deserializer, we want to detect conflicts for:
- No dictionary column, but DICT encoding
- Different encoding in FieldConfig vs indexes JSON
- Different compression in FieldConfig vs indexes
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/forward/ForwardIndexType.java:
##########
@@ -166,6 +179,37 @@ public String getPrettyName() {
return INDEX_DISPLAY_NAME;
}
+ @Override
+ protected ColumnConfigDeserializer<ForwardIndexConfig> createDeserializer() {
+ // Override the default `fromIndexes` deserializer so that legacy `indexes.forward` JSON blocks that lack
Review Comment:
Please add a test for these different conflicting scenarios
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/forward/ForwardIndexType.java:
##########
@@ -166,6 +179,37 @@ public String getPrettyName() {
return INDEX_DISPLAY_NAME;
}
+ @Override
+ protected ColumnConfigDeserializer<ForwardIndexConfig> createDeserializer() {
+ // Override the default `fromIndexes` deserializer so that legacy `indexes.forward` JSON blocks that lack
+ // `encodingType` inherit it from `FieldConfig.encodingType` instead of silently defaulting to DICTIONARY.
+ ColumnConfigDeserializer<ForwardIndexConfig> fromIndexes = (tableConfig, schema) -> {
+ Map<String, ForwardIndexConfig> result = new HashMap<>();
+ Collection<FieldConfig> fieldConfigs = tableConfig.getFieldConfigList();
+ if (fieldConfigs != null) {
+ for (FieldConfig fieldConfig : fieldConfigs) {
+ JsonNode jsonNode = fieldConfig.getIndexes().get(INDEX_DISPLAY_NAME);
+ if (jsonNode == null) {
+ continue;
+ }
+ Preconditions.checkState(jsonNode.isObject(), "Invalid forward index config for column: %s",
+ fieldConfig.getName());
+ ObjectNode configNode = ((ObjectNode) jsonNode).deepCopy();
Review Comment:
(nit) redundant cast
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/BaseSegmentCreator.java:
##########
@@ -307,14 +309,26 @@ private boolean isNullable(FieldSpec fieldSpec) {
* Adapts field index configs based on column properties.
*/
private FieldIndexConfigs adaptConfig(String columnName, FieldIndexConfigs config,
- ColumnStatistics columnStatistics, SegmentGeneratorConfig segmentCreationSpec) {
+ ColumnStatistics columnStatistics, SegmentGeneratorConfig segmentCreationSpec, boolean dictEnabledColumn) {
FieldIndexConfigs.Builder builder = new FieldIndexConfigs.Builder(config);
// Sorted columns treat the 'forwardIndexDisabled' flag as a no-op
ForwardIndexConfig fwdConfig = config.getConfig(StandardIndexes.forward());
if (!fwdConfig.isEnabled() && columnStatistics.isSorted()) {
builder.add(StandardIndexes.forward(),
new ForwardIndexConfig.Builder(fwdConfig).withLegacyProperties(segmentCreationSpec.getColumnProperties(),
columnName).build());
+ } else if (fwdConfig.isEnabled() && !dictEnabledColumn
+ && fwdConfig.getEncodingType() == FieldConfig.EncodingType.DICTIONARY) {
+ // Segment-time dictionary optimizers (`optimizeDictionary` / `optimizeDictionaryForMetrics`) can flip
+ // dictEnabledColumn=false for a column whose configured encoding is DICTIONARY. Without a dictionary the
+ // forward index must be RAW, so reconcile here before the creator factory branches on encoding.
+ builder.add(StandardIndexes.forward(), new ForwardIndexConfig.Builder(FieldConfig.EncodingType.RAW)
Review Comment:
Builder can directly take the config, which simplifies the code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Forward-index encoding: introduce Encoding SPI surface and use it for raw vs dict checks [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #18364:
URL: https://github.com/apache/pinot/pull/18364#discussion_r3177818993
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/forward/ForwardIndexType.java:
##########
@@ -166,32 +178,96 @@ public String getPrettyName() {
return INDEX_DISPLAY_NAME;
}
+ /// Resolves per-column [ForwardIndexConfig] in a single pass over [TableConfig], reconciling the legacy signals
+ /// (`indexingConfig.noDictionaryColumns`, `indexingConfig.noDictionaryConfig`, `fieldConfig.encodingType`,
+ /// `fieldConfig.compressionCodec`) with the modern `fieldConfig.indexes.forward` JSON block. The deserializer
+ /// fails fast on conflicting signals.
@Override
- protected ColumnConfigDeserializer<ForwardIndexConfig> createDeserializerForLegacyConfigs() {
- // reads tableConfig.fieldConfigList and decides what to create using the FieldConfig properties and encoding
+ protected ColumnConfigDeserializer<ForwardIndexConfig> createDeserializer() {
return (tableConfig, schema) -> {
- Map<String, DictionaryIndexConfig> dictConfigs = StandardIndexes.dictionary().getConfig(tableConfig, schema);
+ Map<String, ForwardIndexConfig> result = new HashMap<>();
- Map<String, ForwardIndexConfig> fwdConfig =
- Maps.newHashMapWithExpectedSize(Math.max(dictConfigs.size(), schema.size()));
+ // Legacy noDictionary signals — both indicate RAW forward index for the listed columns.
+ IndexingConfig indexingConfig = tableConfig.getIndexingConfig();
+ Set<String> noDictionaryColumns = new HashSet<>();
+ if (indexingConfig.getNoDictionaryColumns() != null) {
+ noDictionaryColumns.addAll(indexingConfig.getNoDictionaryColumns());
+ }
+ Map<String, String> noDictionaryConfig = indexingConfig.getNoDictionaryConfig();
+ if (noDictionaryConfig != null) {
+ noDictionaryColumns.addAll(noDictionaryConfig.keySet());
+ }
+ Set<String> processed = new HashSet<>();
Collection<FieldConfig> fieldConfigs = tableConfig.getFieldConfigList();
if (fieldConfigs != null) {
for (FieldConfig fieldConfig : fieldConfigs) {
+ String column = fieldConfig.getName();
+ processed.add(column);
+
+ // `forwardIndexDisabled` short-circuits everything else.
Map<String, String> properties = fieldConfig.getProperties();
if (properties != null && isDisabled(properties)) {
- fwdConfig.put(fieldConfig.getName(), ForwardIndexConfig.getDisabled());
- } else {
- ForwardIndexConfig config = createConfigFromFieldConfig(fieldConfig);
- if (!config.equals(ForwardIndexConfig.getDefault())) {
- fwdConfig.put(fieldConfig.getName(), config);
+ result.put(column, ForwardIndexConfig.getDisabled());
+ continue;
+ }
+
+ FieldConfig.EncodingType encodingType = fieldConfig.getEncodingType();
+
+ // Conflict: noDictionaryColumns/noDictionaryConfig says RAW but FieldConfig.encodingType says DICTIONARY.
+ Preconditions.checkState(!noDictionaryColumns.contains(column)
+ || encodingType == FieldConfig.EncodingType.RAW,
+ "Conflicting forward-index config for column: %s — listed in noDictionaryColumns/noDictionaryConfig "
+ + "but FieldConfig.encodingType is DICTIONARY", column);
+
+ JsonNode forwardIndexNode = fieldConfig.getIndexes().get(INDEX_DISPLAY_NAME);
+ if (forwardIndexNode != null) {
+ Preconditions.checkState(forwardIndexNode.isObject(), "Invalid forward index config for column: %s",
+ column);
+ ObjectNode configNode = (ObjectNode) forwardIndexNode.deepCopy();
+
+ // Conflict: encodingType mismatch between FieldConfig and indexes.forward; otherwise inject from
+ // FieldConfig so the resolved ForwardIndexConfig.encodingType always matches the column-level encoding.
+ if (configNode.hasNonNull("encodingType")) {
Review Comment:
Check before deep copy.
To save map lookup, you may directly call get, then check `node != null && !node.isNull()`
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/ColumnIndexCreators.java:
##########
@@ -41,19 +42,22 @@ public class ColumnIndexCreators implements Closeable {
private final List<IndexCreator> _indexCreators;
private final NullValueVectorCreator _nullValueVectorCreator; // null if column is not nullable
private final boolean _isDictionaryEncoded;
+ private final FieldIndexConfigs _adaptedConfig;
Review Comment:
(minor) Just call it `_indexConfigs`. Suggest putting it after `_fieldSpec`
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/forward/ForwardIndexType.java:
##########
@@ -166,32 +178,96 @@ public String getPrettyName() {
return INDEX_DISPLAY_NAME;
}
+ /// Resolves per-column [ForwardIndexConfig] in a single pass over [TableConfig], reconciling the legacy signals
+ /// (`indexingConfig.noDictionaryColumns`, `indexingConfig.noDictionaryConfig`, `fieldConfig.encodingType`,
+ /// `fieldConfig.compressionCodec`) with the modern `fieldConfig.indexes.forward` JSON block. The deserializer
+ /// fails fast on conflicting signals.
@Override
- protected ColumnConfigDeserializer<ForwardIndexConfig> createDeserializerForLegacyConfigs() {
- // reads tableConfig.fieldConfigList and decides what to create using the FieldConfig properties and encoding
+ protected ColumnConfigDeserializer<ForwardIndexConfig> createDeserializer() {
return (tableConfig, schema) -> {
- Map<String, DictionaryIndexConfig> dictConfigs = StandardIndexes.dictionary().getConfig(tableConfig, schema);
+ Map<String, ForwardIndexConfig> result = new HashMap<>();
- Map<String, ForwardIndexConfig> fwdConfig =
- Maps.newHashMapWithExpectedSize(Math.max(dictConfigs.size(), schema.size()));
+ // Legacy noDictionary signals — both indicate RAW forward index for the listed columns.
+ IndexingConfig indexingConfig = tableConfig.getIndexingConfig();
+ Set<String> noDictionaryColumns = new HashSet<>();
+ if (indexingConfig.getNoDictionaryColumns() != null) {
+ noDictionaryColumns.addAll(indexingConfig.getNoDictionaryColumns());
+ }
+ Map<String, String> noDictionaryConfig = indexingConfig.getNoDictionaryConfig();
+ if (noDictionaryConfig != null) {
+ noDictionaryColumns.addAll(noDictionaryConfig.keySet());
+ }
+ Set<String> processed = new HashSet<>();
Collection<FieldConfig> fieldConfigs = tableConfig.getFieldConfigList();
if (fieldConfigs != null) {
for (FieldConfig fieldConfig : fieldConfigs) {
+ String column = fieldConfig.getName();
+ processed.add(column);
Review Comment:
You can do `noDictionaryColumns.remove(column)` instead
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Forward-index encoding: introduce Encoding SPI surface and use it for raw vs dict checks [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #18364:
URL: https://github.com/apache/pinot/pull/18364#discussion_r3171757891
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/ColumnMetadata.java:
##########
@@ -62,10 +63,29 @@ default boolean isSingleValue() {
/// [Constants#UNKNOWN_CARDINALITY].
int getCardinality();
- /// Returns `true` when the column is dictionary-encoded, `false` otherwise.
+ /// Returns `true` when the column has a dictionary, `false` otherwise.
+ ///
+ /// NOTE: dictionary presence is no longer the same as "forward index is dictionary-encoded". A column may keep a
Review Comment:
Resolved in ff0f1c7802. `PinotSegmentColumnReader` now keys dictionary decoding off `ForwardIndexReader.isDictionaryEncoded()` / explicit forward-index encoding instead of dictionary presence, and the targeted reader tests cover RAW forward indexes that still have a dictionary.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexConfigUtils.java:
##########
@@ -0,0 +1,45 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import org.apache.pinot.segment.spi.index.ForwardIndexConfig;
+import org.apache.pinot.spi.config.table.FieldConfig;
+
+
+/**
+ * Utilities for normalizing forward-index configs in segment reload/rebuild paths.
+ */
+public final class ForwardIndexConfigUtils {
+ private ForwardIndexConfigUtils() {
+ }
+
+ public static ForwardIndexConfig withEncoding(ForwardIndexConfig config, FieldConfig.EncodingType encoding) {
+ if (config.isEnabled() && config.getForwardIndexEncoding() == encoding) {
+ return config;
+ }
+ return new ForwardIndexConfig.Builder(config)
+ .withForwardIndexEncoding(encoding)
+ .build();
+ }
+
+ public static ForwardIndexConfig forDictionaryState(ForwardIndexConfig config, boolean dictionaryEnabled) {
Review Comment:
Resolved in ff0f1c7802. `forDictionaryState(...)` now returns configs with explicit `encodingType` unchanged and only derives RAW/DICTIONARY when `getEncodingType()` is null; `ForwardIndexConfigUtilsTest` covers the shared-dictionary RAW case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Forward-index encoding: introduce Encoding SPI surface and use it for raw vs dict checks [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #18364:
URL: https://github.com/apache/pinot/pull/18364#discussion_r3170323195
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/ColumnMetadata.java:
##########
@@ -62,10 +63,29 @@ default boolean isSingleValue() {
/// [Constants#UNKNOWN_CARDINALITY].
int getCardinality();
- /// Returns `true` when the column is dictionary-encoded, `false` otherwise.
+ /// Returns `true` when the column has a dictionary, `false` otherwise.
+ ///
+ /// NOTE: dictionary presence is no longer the same as "forward index is dictionary-encoded". A column may keep a
+ /// dictionary purely for secondary indexes (inverted, FST, range) while the forward index is RAW. Use
+ /// [#getForwardIndexEncoding()] / [#isForwardIndexDictionaryEncoded()] to ask about forward-index encoding.
@JsonProperty("hasDictionary")
boolean hasDictionary();
+ /// Returns the forward-index encoding for this column.
+ ///
+ /// Segment-backed implementations override this to return the value persisted in the
+ /// [V1Constants.MetadataKeys.Column#FORWARD_INDEX_ENCODING] property — falling back to deriving it from
+ /// [#hasDictionary()] for old segments that pre-date that key. The default below is for in-memory stubs that do
+ /// not represent a persisted segment.
+ default FieldConfig.EncodingType getForwardIndexEncoding() {
+ return hasDictionary() ? FieldConfig.EncodingType.DICTIONARY : FieldConfig.EncodingType.RAW;
+ }
+
+ /// Returns `true` when the column's forward index is dictionary-encoded.
+ default boolean isForwardIndexDictionaryEncoded() {
Review Comment:
This should be json ignored. Add `@JsonIgnore`
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/metadata/ColumnMetadataImpl.java:
##########
@@ -124,6 +128,16 @@ public boolean hasDictionary() {
return _hasDictionary;
}
+ @Override
+ public FieldConfig.EncodingType getForwardIndexEncoding() {
+ return _forwardIndexEncoding;
+ }
+
+ @Override
+ public boolean isForwardIndexDictionaryEncoded() {
Review Comment:
No need to override
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/metadata/ColumnMetadataImpl.java:
##########
@@ -279,11 +293,18 @@ public String toString() {
public static ColumnMetadataImpl fromPropertiesConfiguration(PropertiesConfiguration config, int totalDocs,
String column) {
FieldSpec fieldSpec = generateFieldSpec(column, config);
+ boolean hasDictionary = config.getBoolean(Column.getKeyFor(column, Column.HAS_DICTIONARY), true);
+ // Read FORWARD_INDEX_ENCODING from the segment metadata if present; otherwise fall back to the derived value
+ // from HAS_DICTIONARY for backward compatibility with old segments.
+ String encodingProp = config.getString(Column.getKeyFor(column, Column.FORWARD_INDEX_ENCODING),
Review Comment:
Use `getEnum()`
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/ColumnMetadata.java:
##########
@@ -62,10 +63,29 @@ default boolean isSingleValue() {
/// [Constants#UNKNOWN_CARDINALITY].
int getCardinality();
- /// Returns `true` when the column is dictionary-encoded, `false` otherwise.
+ /// Returns `true` when the column has a dictionary, `false` otherwise.
+ ///
+ /// NOTE: dictionary presence is no longer the same as "forward index is dictionary-encoded". A column may keep a
+ /// dictionary purely for secondary indexes (inverted, FST, range) while the forward index is RAW. Use
+ /// [#getForwardIndexEncoding()] / [#isForwardIndexDictionaryEncoded()] to ask about forward-index encoding.
@JsonProperty("hasDictionary")
boolean hasDictionary();
+ /// Returns the forward-index encoding for this column.
+ ///
+ /// Segment-backed implementations override this to return the value persisted in the
+ /// [V1Constants.MetadataKeys.Column#FORWARD_INDEX_ENCODING] property — falling back to deriving it from
+ /// [#hasDictionary()] for old segments that pre-date that key. The default below is for in-memory stubs that do
+ /// not represent a persisted segment.
+ default FieldConfig.EncodingType getForwardIndexEncoding() {
Review Comment:
Don't add default impl for this. We want to avoid bad wiring
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/metadata/ColumnMetadataImpl.java:
##########
@@ -566,9 +593,16 @@ public ColumnMetadataImpl build() {
_bitsPerElement = UNAVAILABLE;
}
- return new ColumnMetadataImpl(_fieldSpec, _totalDocs, _cardinality, _hasDictionary, _sorted, _minValue, _maxValue,
- _minMaxValueInvalid, _lengthOfShortestElement, _lengthOfLongestElement, _isAscii, _totalNumberOfEntries,
- _maxNumberOfMultiValues, _bitsPerElement, _partitionFunction, _partitions, _autoGenerated);
+ // Default encoding: derive from hasDictionary if the caller did not explicitly set one. This preserves the
+ // pre-FORWARD_INDEX_ENCODING invariant for in-memory builds and segment-loading code paths that have not yet
+ // been updated to set the encoding directly.
+ FieldConfig.EncodingType encoding = _forwardIndexEncoding != null ? _forwardIndexEncoding
+ : (_hasDictionary ? FieldConfig.EncodingType.DICTIONARY : FieldConfig.EncodingType.RAW);
Review Comment:
```suggestion
if (_forwardIndexEncoding == null) {
_forwardIndexEncoding = _hasDictionary ? FieldConfig.EncodingType.DICTIONARY : FieldConfig.EncodingType.RAW;
}
```
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/ForwardIndexConfig.java:
##########
@@ -92,22 +107,57 @@ public static ForwardIndexConfig getDisabled() {
@Nullable
private final Map<String, Object> _configs;
+ /// Backward-compatible 7-arg constructor preserved for binary compatibility with master. New code should use
+ /// the [Builder] or the canonical 10-arg `@JsonCreator` which threads forward-index encoding through.
+ @Deprecated
public ForwardIndexConfig(@Nullable Boolean disabled, @Nullable CompressionCodec compressionCodec,
@Nullable Boolean deriveNumDocsPerChunk, @Nullable Integer rawIndexWriterVersion,
@Nullable String targetMaxChunkSize, @Nullable Integer targetDocsPerChunk,
@Nullable Map<String, Object> configs) {
+ this(disabled, compressionCodec, null, null, deriveNumDocsPerChunk, rawIndexWriterVersion, targetMaxChunkSize,
+ targetDocsPerChunk, configs, null);
+ }
+
+ /// Backward-compatible 9-arg constructor preserved for binary compatibility with master (was the previous
+ /// `@JsonCreator`). New code should use the [Builder] or the canonical 10-arg `@JsonCreator`.
+ @Deprecated
+ public ForwardIndexConfig(@Nullable Boolean disabled, @Nullable CompressionCodec compressionCodec,
+ @Nullable ChunkCompressionType chunkCompressionType, @Nullable DictIdCompressionType dictIdCompressionType,
+ @Nullable Boolean deriveNumDocsPerChunk, @Nullable Integer rawIndexWriterVersion,
+ @Nullable String targetMaxChunkSize, @Nullable Integer targetDocsPerChunk,
+ @Nullable Map<String, Object> configs) {
+ this(disabled, compressionCodec, chunkCompressionType, dictIdCompressionType, deriveNumDocsPerChunk,
+ rawIndexWriterVersion, targetMaxChunkSize, targetDocsPerChunk, configs, null);
+ }
+
+ @JsonCreator
+ public ForwardIndexConfig(@JsonProperty("disabled") @Nullable Boolean disabled,
+ @JsonProperty("compressionCodec") @Nullable CompressionCodec compressionCodec,
+ @Deprecated @JsonProperty("chunkCompressionType") @Nullable ChunkCompressionType chunkCompressionType,
+ @Deprecated @JsonProperty("dictIdCompressionType") @Nullable DictIdCompressionType dictIdCompressionType,
+ @JsonProperty("deriveNumDocsPerChunk") @Nullable Boolean deriveNumDocsPerChunk,
+ @JsonProperty("rawIndexWriterVersion") @Nullable Integer rawIndexWriterVersion,
+ @JsonProperty("targetMaxChunkSize") @Nullable String targetMaxChunkSize,
+ @JsonProperty("targetDocsPerChunk") @Nullable Integer targetDocsPerChunk,
+ @JsonProperty("configs") @Nullable Map<String, Object> configs,
+ @JsonProperty("forwardIndexEncoding") @Nullable FieldConfig.EncodingType forwardIndexEncoding) {
Review Comment:
Name it `encodingType` (match `FieldConfig`). Put it as the second argument
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/ForwardIndexConfig.java:
##########
@@ -73,10 +77,21 @@ public static ForwardIndexConfig getDefault() {
return new Builder().build();
}
+ /// Returns a `ForwardIndexConfig` that represents a disabled forward index.
+ ///
+ /// The returned instance reports [#getForwardIndexEncoding()] as [#DEFAULT_FORWARD_INDEX_ENCODING] for non-null
+ /// safety, but the encoding has no observable effect when the forward index is disabled — consumers should not
+ /// branch on it for disabled configs.
public static ForwardIndexConfig getDisabled() {
- return new ForwardIndexConfig(true, null, null, null, null, null, null, null, null);
+ return new ForwardIndexConfig(true, null, null, null, null, null, null, null, null, null);
}
+ // Forward-index encoding is the top-level branch for this config; it must be declared first so dependent fields
+ // (compression, chunk size, etc.) can be read in the context of the chosen encoding.
+ // Always non-null — coalesces to DEFAULT_FORWARD_INDEX_ENCODING when no explicit value is provided. Mirrors
+ // FieldConfig._encodingType which also coalesces null to DICTIONARY.
+ private final FieldConfig.EncodingType _forwardIndexEncoding;
Review Comment:
This is under the scope of ForwardIndexConfig, name it `_encodingType` (also match `FieldConfig`)
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/ForwardIndexConfig.java:
##########
@@ -248,6 +285,23 @@ public Map<String, Object> getConfigs() {
return _configs;
}
+ @JsonIgnore
+ public FieldConfig.EncodingType getForwardIndexEncoding() {
+ return _forwardIndexEncoding;
+ }
+
+ @JsonIgnore
+ public boolean isForwardIndexEncodingExplicitlySet() {
+ return _forwardIndexEncodingExplicitlySet;
+ }
+
+ @JsonProperty("forwardIndexEncoding")
+ @JsonInclude(JsonInclude.Include.NON_NULL)
+ @Nullable
+ private FieldConfig.EncodingType getForwardIndexEncodingForJson() {
Review Comment:
Don't change value during ser/de. This will make future default behavior change hard
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/metadata/ColumnMetadataImpl.java:
##########
@@ -80,7 +82,8 @@ public class ColumnMetadataImpl implements ColumnMetadata {
/// Use non-default size to save space for most columns
private final LongArrayList _indexTypeSizeList = new LongArrayList(2);
- private ColumnMetadataImpl(FieldSpec fieldSpec, int totalDocs, int cardinality, boolean hasDictionary, boolean sorted,
+ private ColumnMetadataImpl(FieldSpec fieldSpec, int totalDocs, int cardinality, boolean hasDictionary,
+ FieldConfig.EncodingType forwardIndexEncoding, boolean sorted,
Review Comment:
(nit) Reformat the constructor arguments
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/ForwardIndexConfig.java:
##########
@@ -35,6 +36,9 @@
public class ForwardIndexConfig extends IndexConfig {
+ /// Default forward-index encoding when the field is absent from JSON and no explicit value is set on the
+ /// [Builder].
+ public static final FieldConfig.EncodingType DEFAULT_FORWARD_INDEX_ENCODING = FieldConfig.EncodingType.DICTIONARY;
Review Comment:
(minor) Directly import `EncodingType`
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/metadata/ColumnMetadataImpl.java:
##########
@@ -80,7 +82,8 @@ public class ColumnMetadataImpl implements ColumnMetadata {
/// Use non-default size to save space for most columns
private final LongArrayList _indexTypeSizeList = new LongArrayList(2);
- private ColumnMetadataImpl(FieldSpec fieldSpec, int totalDocs, int cardinality, boolean hasDictionary, boolean sorted,
+ private ColumnMetadataImpl(FieldSpec fieldSpec, int totalDocs, int cardinality, boolean hasDictionary,
+ FieldConfig.EncodingType forwardIndexEncoding, boolean sorted,
Review Comment:
Directly import `EncodingType`
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/ForwardIndexConfig.java:
##########
@@ -73,10 +77,21 @@ public static ForwardIndexConfig getDefault() {
return new Builder().build();
}
+ /// Returns a `ForwardIndexConfig` that represents a disabled forward index.
+ ///
+ /// The returned instance reports [#getForwardIndexEncoding()] as [#DEFAULT_FORWARD_INDEX_ENCODING] for non-null
+ /// safety, but the encoding has no observable effect when the forward index is disabled — consumers should not
+ /// branch on it for disabled configs.
public static ForwardIndexConfig getDisabled() {
- return new ForwardIndexConfig(true, null, null, null, null, null, null, null, null);
+ return new ForwardIndexConfig(true, null, null, null, null, null, null, null, null, null);
}
+ // Forward-index encoding is the top-level branch for this config; it must be declared first so dependent fields
+ // (compression, chunk size, etc.) can be read in the context of the chosen encoding.
+ // Always non-null — coalesces to DEFAULT_FORWARD_INDEX_ENCODING when no explicit value is provided. Mirrors
+ // FieldConfig._encodingType which also coalesces null to DICTIONARY.
+ private final FieldConfig.EncodingType _forwardIndexEncoding;
+ private final boolean _forwardIndexEncodingExplicitlySet;
Review Comment:
Why is this needed? We should keep the JSON as is during ser/de
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/ForwardIndexConfig.java:
##########
@@ -73,10 +77,21 @@ public static ForwardIndexConfig getDefault() {
return new Builder().build();
}
+ /// Returns a `ForwardIndexConfig` that represents a disabled forward index.
+ ///
+ /// The returned instance reports [#getForwardIndexEncoding()] as [#DEFAULT_FORWARD_INDEX_ENCODING] for non-null
+ /// safety, but the encoding has no observable effect when the forward index is disabled — consumers should not
+ /// branch on it for disabled configs.
public static ForwardIndexConfig getDisabled() {
- return new ForwardIndexConfig(true, null, null, null, null, null, null, null, null);
+ return new ForwardIndexConfig(true, null, null, null, null, null, null, null, null, null);
}
+ // Forward-index encoding is the top-level branch for this config; it must be declared first so dependent fields
+ // (compression, chunk size, etc.) can be read in the context of the chosen encoding.
+ // Always non-null — coalesces to DEFAULT_FORWARD_INDEX_ENCODING when no explicit value is provided. Mirrors
+ // FieldConfig._encodingType which also coalesces null to DICTIONARY.
+ private final FieldConfig.EncodingType _forwardIndexEncoding;
Review Comment:
This should be `@Nullable`
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/ForwardIndexConfig.java:
##########
@@ -192,16 +231,14 @@ public static CompressionCodec getActualCompressionCodec(@Nullable CompressionCo
default:
throw new IllegalStateException("Unsupported chunk compression type: " + chunkCompressionType);
}
- } else if (dictIdCompressionType != null) {
- switch (dictIdCompressionType) {
- case MV_ENTRY_DICT:
- return CompressionCodec.MV_ENTRY_DICT;
- default:
- throw new IllegalStateException("Unsupported dictionary compression type: " + dictIdCompressionType);
+ }
Review Comment:
Why is this change needed?
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/IndexCreationContext.java:
##########
@@ -143,6 +152,7 @@ final class Builder {
private int[] _mutableToImmutableDocIdMap;
private String _tableNameWithType;
private boolean _continueOnError;
+ private ForwardIndexConfig _forwardIndexConfig;
Review Comment:
What is the usage of this? The old `_forwardIndexDisabled` is actually not used, and should be cleaned up. `IndexConfig` shouldn't be accessed from here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Forward-index encoding: introduce Encoding SPI surface and use it for raw vs dict checks [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #18364:
URL: https://github.com/apache/pinot/pull/18364#discussion_r3172427369
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -999,8 +1010,11 @@ private void rewriteDictToRawForwardIndex(ColumnMetadata columnMetadata, Segment
builder.withMaxRowLengthInBytes(getMaxRowLength(columnMetadata, reader, dictionary));
}
}
+ ForwardIndexConfig config =
+ ForwardIndexConfigUtils.forDictionaryState(
Review Comment:
(nit) reformat
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/readers/PinotSegmentColumnReader.java:
##########
@@ -92,11 +105,43 @@ public Dictionary getDictionary() {
}
public int getDictId(int docId) {
- return _forwardIndexReader.getDictId(docId, _forwardIndexReaderContext);
+ if (_forwardIndexDictionaryEncoded) {
Review Comment:
Consider not changing this one. We need to avoid hitting this code path when forward index is not dictionary encoded. You may add an `assert` to avoid hitting this code path in test
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/forward/ForwardIndexType.java:
##########
@@ -230,35 +277,22 @@ public IndexHandler createIndexHandler(SegmentDirectory segmentDirectory, Map<St
return new ForwardIndexHandler(segmentDirectory, configsByCol, tableConfig, schema);
}
- @Override
- public boolean requiresDictionary(FieldSpec fieldSpec, ForwardIndexConfig indexConfig) {
- // Forward index supports both DICT and RAW encodings; it does not require a dictionary to function.
- return false;
- }
-
- @Override
- public boolean shouldInvalidateOnDictionaryChange(FieldSpec fieldSpec, ForwardIndexConfig indexConfig) {
- // The forward index encoding is reconciled with FieldConfig.encodingType independently of dictionary state
- // (apache/pinot#18364), so adding or removing a dictionary alone does not change the on-disk forward index
- // layout — the existing forward index can be reused.
- return false;
- }
-
@Override
protected IndexReaderFactory<ForwardIndexReader> createReaderFactory() {
return ForwardIndexReaderFactory.getInstance();
}
public String getFileExtension(ColumnMetadata columnMetadata) {
+ boolean isRawEncoded = columnMetadata.getForwardIndexEncoding() == FieldConfig.EncodingType.RAW;
Review Comment:
(minor)
```suggestion
boolean isRaw = columnMetadata.getForwardIndexEncoding() == FieldConfig.EncodingType.RAW;
```
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/ForwardIndexConfig.java:
##########
@@ -73,10 +76,16 @@ public static ForwardIndexConfig getDefault() {
return new Builder().build();
}
+ /// Returns a `ForwardIndexConfig` that represents a disabled forward index.
+ ///
+ /// The returned instance stores a null [#getEncodingType()] like a default config; the encoding has no observable
+ /// effect when the forward index is disabled.
public static ForwardIndexConfig getDisabled() {
- return new ForwardIndexConfig(true, null, null, null, null, null, null, null, null);
+ return new ForwardIndexConfig(true, null, null, null, null, null, null, null, null, null);
}
+ @Nullable
+ private final EncodingType _encodingType;
Review Comment:
Is `EncodingType` always available? It can always be derived from the table config
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexConfigUtils.java:
##########
@@ -0,0 +1,51 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import org.apache.pinot.segment.spi.index.ForwardIndexConfig;
+import org.apache.pinot.spi.config.table.FieldConfig;
+
+
+/**
+ * Utilities for normalizing forward-index configs in segment reload/rebuild paths.
+ */
+public final class ForwardIndexConfigUtils {
+ private ForwardIndexConfigUtils() {
+ }
+
+ public static ForwardIndexConfig withEncoding(ForwardIndexConfig config, FieldConfig.EncodingType encoding) {
+ if (!config.isEnabled()) {
+ return config;
+ }
+ if (config.isEnabled() && config.getForwardIndexEncoding() == encoding) {
+ return config;
+ }
+ return new ForwardIndexConfig.Builder(config)
+ .withEncodingType(encoding)
+ .build();
+ }
+
+ public static ForwardIndexConfig forDictionaryState(ForwardIndexConfig config, boolean dictionaryEnabled) {
Review Comment:
We should be able to simplify the handling if `EncodingType` is always available.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Forward-index encoding: introduce Encoding SPI surface and use it for raw vs dict checks [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #18364:
URL: https://github.com/apache/pinot/pull/18364#discussion_r3175863578
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/IndexCreationContext.java:
##########
@@ -143,7 +140,6 @@ final class Builder {
private int[] _mutableToImmutableDocIdMap;
private String _tableNameWithType;
private boolean _continueOnError;
-
Review Comment:
(nit) Keep the empty line
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/readers/PinotSegmentColumnReader.java:
##########
@@ -92,11 +105,13 @@ public Dictionary getDictionary() {
}
public int getDictId(int docId) {
+ Preconditions.checkState(_forwardIndexDictionaryEncoded,
+ "Cannot read dictIds from RAW forward index");
Review Comment:
This is hot path, consider using `assert !_forwardIndexDictionaryEncoded;` which doesn't have overhead
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/dictionary/DictionaryIndexType.java:
##########
@@ -129,7 +130,7 @@ protected ColumnConfigDeserializer<DictionaryIndexConfig> createDeserializerForL
(accum, column) -> accum.put(column, DictionaryIndexConfig.DISABLED));
ColumnConfigDeserializer<DictionaryIndexConfig> fromFieldConfigs =
IndexConfigDeserializer.fromCollection(TableConfig::getFieldConfigList, (accum, fieldConfig) -> {
- if (fieldConfig.getEncodingType() == FieldConfig.EncodingType.RAW) {
+ if (fieldConfig.getEncodingType() == FieldConfig.EncodingType.RAW && !hasForwardIndexConfig(fieldConfig)) {
Review Comment:
Why do we need this?
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/RealtimeSegmentConfig.java:
##########
@@ -441,5 +445,26 @@ public RealtimeSegmentConfig build() {
_ingestionAggregationConfigs, _defaultNullHandlingEnabled, _partitionUpsertMetadataManager,
_partitionDedupMetadataManager, _consumerDir, _textIndexConfig, _dropRecordOnPartitionMismatch);
}
+
+ private void normalizeForwardIndexEncoding() {
Review Comment:
What is this method trying to do? Can you add some javadoc?
I don't think this is doing the right thing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org