You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "Vamsi-klu (via GitHub)" <gi...@apache.org> on 2026/07/25 06:19:00 UTC
[PR] Fail-soft MutableSegmentImpl.index to avoid partial-row corruption [pinot]
Vamsi-klu opened a new pull request, #19088:
URL: https://github.com/apache/pinot/pull/19088
## Why
`MutableSegmentImpl.index()` is not transactional. On exception mid-row (dictionary/forward/agg), some columns were updated and others were not. `RealtimeSegmentDataManager` catches the error and **continues consuming** on the same mutable segment, so corruption becomes durable until seal fails (`IndexOutOfBoundsException`) or queries lie.
Production effect: a single bad row can **poison a consuming partition** (seal failure, stuck consumption, data loss risk).
## Impact
- **Mutable segments stay sealable** after transform/index errors.
- **Per-row consistency** across dictionaries/forward indexes (complete-the-row with defaults/nulls instead of half-written docs).
- **Observable quality**: incomplete/errored rows are metered rather than silent.
- Also clears the stale multi-value + `aggregateMetrics` FIXME (aggregation is already disabled/rejected for MV dims).
## How
- Fail-soft path in `index()` / `updateDictionary` / `addNewRow`: on failure, fill remaining columns with field default/null values for that docId and advance doc count consistently.
- Treat forward-index failure as row-level incomplete (do not silently skip only the forward index).
- Meter incomplete rows; avoid full dictionary transactional rollback (shared dict + concurrent readers).
## Test plan
- [x] Extended `IndexingFailureTest` for mid-row failure → segment remains consistent / sealable.
- [ ] `./mvnw -pl pinot-segment-local -am -Dtest=IndexingFailureTest -Dsurefire.failIfNoSpecifiedTests=false test`
- [ ] Spot-check with ingestion aggregation bad input after #16317 lands (linked).
## Related
fixes: #16316
related: #16317
## Reviewers
Suggested: Jackie-Jiang (issue guidance on fail-soft defaults)
---
##### Was generative AI tooling used to co-author this PR?
- [x] Yes — Grok Build (xAI)
Generated-by: Grok Build (xAI)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fail-soft MutableSegmentImpl.index to avoid partial-row corruption [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19088:
URL: https://github.com/apache/pinot/pull/19088#issuecomment-5077641402
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/19088?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `41.28788%` with `155 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 65.43%. Comparing base ([`2bcbdfe`](https://app.codecov.io/gh/apache/pinot/commit/2bcbdfed0af40edde0d4c12ebdff2a2463338d67?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`c854a0f`](https://app.codecov.io/gh/apache/pinot/commit/c854a0fabad0edd62a9f1dc6a754462061f13b63?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/pinot/pull/19088?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...local/indexsegment/mutable/MutableSegmentImpl.java](https://app.codecov.io/gh/apache/pinot/pull/19088?src=pr&el=tree&filepath=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Findexsegment%2Fmutable%2FMutableSegmentImpl.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9pbmRleHNlZ21lbnQvbXV0YWJsZS9NdXRhYmxlU2VnbWVudEltcGwuamF2YQ==) | 41.28% | [136 Missing and 19 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/19088?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19088 +/- ##
============================================
- Coverage 65.47% 65.43% -0.04%
+ Complexity 1421 1415 -6
============================================
Files 3426 3426
Lines 217315 217498 +183
Branches 34509 34543 +34
============================================
+ Hits 142283 142319 +36
- Misses 63513 63644 +131
- Partials 11519 11535 +16
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/19088/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/19088/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/19088/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/19088/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/19088/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-25](https://app.codecov.io/gh/apache/pinot/pull/19088/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `65.43% <41.28%> (-0.04%)` | :arrow_down: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/19088/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `65.43% <41.28%> (-0.04%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/19088/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `65.43% <41.28%> (-0.04%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/19088/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.76% <9.84%> (-0.07%)` | :arrow_down: |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/19088/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.87% <41.28%> (-0.01%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/pinot/pull/19088?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fail-soft MutableSegmentImpl.index to avoid partial-row corruption [pinot]
Posted by "Vamsi-klu (via GitHub)" <gi...@apache.org>.
Vamsi-klu commented on PR #19088:
URL: https://github.com/apache/pinot/pull/19088#issuecomment-5642791034
@noob-se7en _continueOnError is wired on the current head for index(), updateDictionary(), and addNewRow().
When the flag is false, we still finish the physical row so column lengths stay aligned, then rethrow. When it is true, we record the incomplete row and keep consuming.
Apache CI is still running on this head. Unit, integration, and compat jobs are pending, so I am not calling it green yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fail-soft MutableSegmentImpl.index to avoid partial-row corruption [pinot]
Posted by "Vamsi-klu (via GitHub)" <gi...@apache.org>.
Vamsi-klu commented on code in PR #19088:
URL: https://github.com/apache/pinot/pull/19088#discussion_r3725701821
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImpl.java:
##########
@@ -820,182 +831,419 @@ private void validateLengthOfMVColumns(GenericRow row)
}
}
- private void updateDictionary(GenericRow row) {
+ /// Runs dictionary + forward/secondary indexing for a new docId and meters an incomplete row when either step had
+ /// to fall back to defaults (issue #16316).
+ private void indexPhysicalRow(int docId, GenericRow row) {
+ boolean dictHadError = updateDictionary(row);
+ boolean rowHadError = addNewRow(docId, row);
+ if (dictHadError || rowHadError) {
+ recordIncompleteRow();
+ }
+ }
+
+ /// @return {@code true} if any column required a default/fallback while updating dictionaries
+ private boolean updateDictionary(GenericRow row) {
+ boolean hadError = false;
for (Map.Entry<String, IndexContainer> entry : _indexContainerMap.entrySet()) {
IndexContainer indexContainer = entry.getValue();
MutableDictionary dictionary = indexContainer._dictionary;
if (dictionary == null) {
continue;
}
-
- Object value = row.getValue(entry.getKey());
- if (value == null) {
- recordIndexingError("DICTIONARY");
- } else {
+ String column = entry.getKey();
+ Object value = row.getValue(column);
+ try {
+ if (value == null) {
+ // Prefer default-null dict entry so addNewRow can still complete the forward index for this docId
+ // (fail-soft; issue #16316). Meter and fall back to field-spec default.
+ recordIndexingError("DICTIONARY");
+ hadError = true;
+ value = getDefaultNullValueForIndexing(indexContainer._fieldSpec);
+ row.putDefaultNullValue(column, value);
+ }
if (indexContainer._fieldSpec.isSingleValueField()) {
indexContainer._dictId = dictionary.index(value);
} else {
indexContainer._dictIds = dictionary.index((Object[]) value);
}
-
// Update min/max value from dictionary
indexContainer._minValue = dictionary.getMinVal();
indexContainer._maxValue = dictionary.getMaxVal();
+ } catch (Exception e) {
+ // Do not abort the row mid-dictionary: remaining columns still get a chance, and addNewRow will fill
+ // defaults for this column if dict ids are unset (Integer.MIN_VALUE / null).
+ hadError = true;
+ recordIndexingError("DICTIONARY", e);
+ indexContainer._dictId = Integer.MIN_VALUE;
+ indexContainer._dictIds = null;
}
Review Comment:
Good catch, this is a real gap on the aggregation path. index() calls updateDictionary() and then getOrCreateDocId(), and the key is built straight from indexContainer._dictId for every dimension and time column, so the Integer.MIN_VALUE sentinel does flow into the rollup key. Two failed rows with different real values would collapse into one key, and the default recovery in addPhysicalColumn happens only after keying. I will move the fallback into the updateDictionary() catch block: index getDefaultNullValueForIndexing(fieldSpec) into the dictionary and set _dictId/_dictIds from that, keeping Integer.MIN_VALUE only if indexing the default also throws. The MIN_VALUE recovery in addPhysicalColumn stays as a last resort, and the row is still metered incomplete via dictHadError.
##########
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/indexsegment/mutable/IndexingFailureTest.java:
##########
@@ -133,8 +133,46 @@ public void testIndexingFailures()
assertEquals(_mutableSegment.getDataSource(JSON_COL).getJsonIndex().getMatchingDocIds("valid = 'json'"),
ImmutableRoaringBitmap.bitmapOf(0, 2, 3));
assertTrue(_mutableSegment.getDataSource(STRING_COL).getNullValueVector().isNull(3));
- // null string value skipped
+ // Fail-soft (#16316): null string is completed with the field default so forward lengths stay aligned.
+ GenericRow nullResult = _mutableSegment.getRecord(3, new GenericRow());
+ assertEquals(nullResult.getValue(STRING_COL), FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_STRING);
verify(_serverMetrics, times(1)).addMeteredTableValue(matches("DICTIONARY-indexingError$"),
eq(ServerMeter.INDEXING_FAILURES), eq(1L));
Review Comment:
I checked this and the matcher does match the emitted key. Mockito's ArgumentMatchers.matches() delegates to org.mockito.internal.matchers.Matches, which calls pattern.matcher(actual).find(), a substring search rather than a whole-string match (verified against mockito-core 5.23.0, the version pinned in the root pom). So matches("DICTIONARY-indexingError$") finds the anchored suffix inside "testTable_REALTIME-DICTIONARY-indexingError". This assertion is also unchanged from master (it shows up as context in the diff, and the pre-existing checks at lines 90 and 104 use the same style), and IndexingFailureTest passes in CI on this PR. I will keep it as is.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImpl.java:
##########
@@ -820,182 +831,419 @@ private void validateLengthOfMVColumns(GenericRow row)
}
}
- private void updateDictionary(GenericRow row) {
+ /// Runs dictionary + forward/secondary indexing for a new docId and meters an incomplete row when either step had
+ /// to fall back to defaults (issue #16316).
+ private void indexPhysicalRow(int docId, GenericRow row) {
+ boolean dictHadError = updateDictionary(row);
+ boolean rowHadError = addNewRow(docId, row);
+ if (dictHadError || rowHadError) {
+ recordIncompleteRow();
+ }
+ }
+
+ /// @return {@code true} if any column required a default/fallback while updating dictionaries
+ private boolean updateDictionary(GenericRow row) {
+ boolean hadError = false;
for (Map.Entry<String, IndexContainer> entry : _indexContainerMap.entrySet()) {
IndexContainer indexContainer = entry.getValue();
MutableDictionary dictionary = indexContainer._dictionary;
if (dictionary == null) {
continue;
}
-
- Object value = row.getValue(entry.getKey());
- if (value == null) {
- recordIndexingError("DICTIONARY");
- } else {
+ String column = entry.getKey();
+ Object value = row.getValue(column);
+ try {
+ if (value == null) {
+ // Prefer default-null dict entry so addNewRow can still complete the forward index for this docId
+ // (fail-soft; issue #16316). Meter and fall back to field-spec default.
+ recordIndexingError("DICTIONARY");
+ hadError = true;
+ value = getDefaultNullValueForIndexing(indexContainer._fieldSpec);
+ row.putDefaultNullValue(column, value);
+ }
if (indexContainer._fieldSpec.isSingleValueField()) {
indexContainer._dictId = dictionary.index(value);
} else {
indexContainer._dictIds = dictionary.index((Object[]) value);
}
-
// Update min/max value from dictionary
indexContainer._minValue = dictionary.getMinVal();
indexContainer._maxValue = dictionary.getMaxVal();
+ } catch (Exception e) {
+ // Do not abort the row mid-dictionary: remaining columns still get a chance, and addNewRow will fill
+ // defaults for this column if dict ids are unset (Integer.MIN_VALUE / null).
+ hadError = true;
+ recordIndexingError("DICTIONARY", e);
+ indexContainer._dictId = Integer.MIN_VALUE;
+ indexContainer._dictIds = null;
}
- updateIndexCapacityThresholdBreached(dictionary, entry.getKey());
+ updateIndexCapacityThresholdBreached(dictionary, column);
}
+ return hadError;
}
- private void addNewRow(int docId, GenericRow row) {
+ /// Indexes a new physical row. Fail-soft (issue #16316): every physical column must end up with a forward-index
+ /// (or OPEN_STRUCT) entry for [docId] so seal/query lengths stay aligned with [_numDocsIndexed]. On forward-index
+ /// failure the column is completed with the field default/null instead of being left blank. Secondary index
+ /// failures are still metered and swallowed. Aggregation path failures also fall back to a default initial value.
+ ///
+ /// @return {@code true} if any column required a default/fallback while indexing
+ private boolean addNewRow(int docId, GenericRow row) {
+ boolean rowHadError = false;
for (Map.Entry<String, IndexContainer> entry : _indexContainerMap.entrySet()) {
String column = entry.getKey();
IndexContainer indexContainer = entry.getValue();
-
- // Handle ingestion aggregation
- ValueAggregator valueAggregator = indexContainer._valueAggregator;
- if (valueAggregator != null) {
- String sourceColumn = indexContainer._sourceColumn;
- // NOTE: value can be null if the column is not specified in the schema.
- Object value = row.getValue(sourceColumn);
- // Handle COUNT(*)
- if (value == null && sourceColumn.equals(AggregationFunctionColumnPair.STAR)) {
- assert valueAggregator.getAggregationType() == AggregationFunctionType.COUNT;
- value = 1;
+ try {
+ if (indexContainer._valueAggregator != null) {
+ if (!addAggregatedColumn(docId, row, column, indexContainer)) {
+ rowHadError = true;
+ }
+ } else if (!addPhysicalColumn(docId, row, column, indexContainer)) {
+ rowHadError = true;
}
-
- // Update numValues info
- indexContainer._valuesInfo.updateSVNumValues();
-
- MutableIndex forwardIndex = indexContainer._mutableIndexes.get(StandardIndexes.forward());
- FieldSpec fieldSpec = indexContainer._fieldSpec;
-
- DataType dataType = fieldSpec.getDataType();
- value = valueAggregator.getInitialAggregatedValue(value);
- // BIG_DECIMAL is actually stored as byte[] and hence can be supported here.
- switch (dataType.getStoredType()) {
- case INT:
- forwardIndex.add(((Number) value).intValue(), -1, docId);
- break;
- case LONG:
- forwardIndex.add(((Number) value).longValue(), -1, docId);
- break;
- case FLOAT:
- forwardIndex.add(((Number) value).floatValue(), -1, docId);
- break;
- case DOUBLE:
- forwardIndex.add(((Number) value).doubleValue(), -1, docId);
- break;
- case BIG_DECIMAL:
- case BYTES:
- forwardIndex.add(valueAggregator.serializeAggregatedValue(value), -1, docId);
- break;
- default:
- throw new UnsupportedOperationException(
- "Unsupported data type: " + dataType + " for aggregation: " + column);
+ } catch (Exception e) {
+ // Last-resort complete-the-row so a single bad column cannot leave a half-written docId.
+ rowHadError = true;
+ recordIndexingError("ROW", e);
+ try {
+ indexDefaultNullColumn(docId, indexContainer);
+ } catch (Exception fallbackError) {
+ _logger.error("Failed to index default null for column: {} at docId: {}", column, docId, fallbackError);
}
- continue;
}
+ }
- // Update the null value vector even if a null value is somehow produced
- if (indexContainer._nullValueVector != null && row.isNullValue(column)) {
- indexContainer._nullValueVector.setNull(docId);
+ if (_multiColumnValues != null) {
+ try {
+ _multiColumnTextIndex.add(_multiColumnValues);
+ } catch (Exception e) {
+ rowHadError = true;
+ recordIndexingError("MULTI_COLUMN_TEXT", e);
+ } finally {
+ Collections.fill(_multiColumnValues, null);
}
+ }
+ return rowHadError;
+ }
- Object value = row.getValue(column);
- if (value == null) {
- // the value should not be null unless something is broken upstream but this will lead to inappropriate reuse
- // of the dictionary id if this somehow happens. An NPE here can corrupt indexes leading to incorrect query
- // results, hence the extra care. A metric will already have been emitted when trying to update the dictionary.
- continue;
- }
+ /// Returns {@code true} when the aggregated column was written without error.
+ private boolean addAggregatedColumn(int docId, GenericRow row, String column, IndexContainer indexContainer) {
+ ValueAggregator valueAggregator = indexContainer._valueAggregator;
+ String sourceColumn = indexContainer._sourceColumn;
+ // NOTE: value can be null if the column is not specified in the schema.
+ Object value = row.getValue(sourceColumn);
+ // Handle COUNT(*)
+ if (value == null && sourceColumn.equals(AggregationFunctionColumnPair.STAR)) {
+ assert valueAggregator.getAggregationType() == AggregationFunctionType.COUNT;
+ value = 1;
+ }
- FieldSpec fieldSpec = indexContainer._fieldSpec;
- DataType dataType = fieldSpec.getDataType();
+ MutableIndex forwardIndex = indexContainer._mutableIndexes.get(StandardIndexes.forward());
+ FieldSpec fieldSpec = indexContainer._fieldSpec;
+ DataType dataType = fieldSpec.getDataType();
+ try {
+ value = valueAggregator.getInitialAggregatedValue(value);
+ // BIG_DECIMAL is actually stored as byte[] and hence can be supported here.
+ switch (dataType.getStoredType()) {
+ case INT:
+ forwardIndex.add(((Number) value).intValue(), -1, docId);
+ break;
+ case LONG:
+ forwardIndex.add(((Number) value).longValue(), -1, docId);
+ break;
+ case FLOAT:
+ forwardIndex.add(((Number) value).floatValue(), -1, docId);
+ break;
+ case DOUBLE:
+ forwardIndex.add(((Number) value).doubleValue(), -1, docId);
+ break;
+ case BIG_DECIMAL:
+ case BYTES:
+ forwardIndex.add(valueAggregator.serializeAggregatedValue(value), -1, docId);
+ break;
+ default:
+ throw new UnsupportedOperationException(
+ "Unsupported data type: " + dataType + " for aggregation: " + column);
+ }
+ indexContainer._valuesInfo.updateSVNumValues();
+ return true;
+ } catch (Exception e) {
+ recordIndexingError(StandardIndexes.forward(), e);
+ indexDefaultAggregatedValue(docId, indexContainer);
+ return false;
+ }
+ }
- if (fieldSpec.isSingleValueField()) {
- // Update numValues info
+ /// Returns {@code true} when the physical column was written without error.
+ private boolean addPhysicalColumn(int docId, GenericRow row, String column, IndexContainer indexContainer) {
+ FieldSpec fieldSpec = indexContainer._fieldSpec;
+ DataType dataType = fieldSpec.getDataType();
+ boolean isNull = row.isNullValue(column);
+ Object value = row.getValue(column);
+ if (value == null) {
+ // Should not happen after NullValueTransformer, but complete the row with defaults rather than leaving a hole.
+ recordIndexingError("NULL_VALUE");
+ value = getDefaultNullValueForIndexing(fieldSpec);
+ isNull = true;
+ }
Review Comment:
Agreed. The null-value fallback at the top of addPhysicalColumn() substitutes the field default and meters NULL_VALUE, but the hadError flags are declared after that block, so the method can still return true and addNewRow() never meters the row as incomplete. The OPEN_STRUCT branch's early return true has the same gap. I will track the fallback in a flag initialized before the null check and fold it into all three return paths (single-value, multi-value, and OPEN_STRUCT) so any default substitution propagates to rowHadError and triggers recordIncompleteRow() once for the row. This only changes behavior for raw/no-dictionary columns, since updateDictionary() already flags the null case for dictionary columns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fail-soft MutableSegmentImpl.index to avoid partial-row corruption [pinot]
Posted by "Vamsi-klu (via GitHub)" <gi...@apache.org>.
Vamsi-klu commented on PR #19088:
URL: https://github.com/apache/pinot/pull/19088#issuecomment-5249499643
Pushed `4a0dfe3` with both actionable threads addressed, details on each.
I checked the new tests by reverting only the `MutableSegmentImpl` hunks and re-running: 3 of the 5 tests in `IndexingFailureTest` fail without the fix and all 5 pass with it, so they genuinely pin the behavior rather than just passing. The other mutable-segment suites are green locally as well, 29 tests.
One deliberate deferral worth stating rather than leaving silent: `addPhysicalColumn` still iterates `indexContainer._mutableIndexes` in HashMap order instead of writing the forward index first. It is a separate concern from the two threads here and none of the new tests depend on ordering, so I would rather do it as a follow-up than widen this PR. Happy to fold it in if a reviewer prefers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fail-soft MutableSegmentImpl.index to avoid partial-row corruption [pinot]
Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps commented on PR #19088:
URL: https://github.com/apache/pinot/pull/19088#issuecomment-6335924523
This comes from a model of upsert thread orders that StarTree runs on upsert PRs.
**On a `dropOutOfOrderRecord` table, this adds one new bad order, where a key owns a doc that `index()` never writes. It clears none of the model's known ones.**
I replayed the 45 histories behind the model's known bad orders and none changed. Then I searched 12 `dropOutOfOrderRecord` table shapes for 61 minutes. The search found this order 14 times, never on today's code.
**The new order.** `isOutOfOrderRecord()` reads the key without its lock, and the drop branch still calls `addRecord()` with docId `numDocsIndexed`. If a Helix thread takes the key's segment OFFLINE in between, `addRecord()` sees a new key and marks that unwritten doc valid, so the next record lands in it. It needs a segment reset at that exact moment, so it is rare.
```mermaid
sequenceDiagram
participant C as Consumer N1
participant M as Upsert metadata
participant H as Helix thread
Note over M: A points at N0, comparison 5
C->>M: index(A, comparison 3), the peek says out of order
H->>M: N0 ONLINE to OFFLINE, removeSegment drops A
C->>M: drop branch addRecord(A, docId 0) finds no A, marks doc 0 valid
Note over C: doc 0 is never written, numDocsIndexed stays 0
C->>M: index(B, value 7) writes doc 0, so A and B share it
C->>M: index(A, comparison 6) merges with doc 0, which is B's row
C->>M: addRecord(A) moves A to doc 1 and clears doc 0
Note over M: A's value is 8 instead of 2, and B is gone from queries
```
<details><summary>Model history</summary>
```
('append', 'A', (5,), 0, {'v': 1}, False)
('append', 'A', (3,), 1, {'v': 1}, False)
('append', 'B', (1,), 2, {'v': 7}, False)
('append', 'A', (6,), 3, {'v': 1}, False)
('assign_consumer', "N0'")
('acquire', "N0'")
('consume', "N0'", 0)
('finish_index', "N0'", 0)
('commit_local', "N0'", 0)
('step', 'replace', "N0'", 'start_replace')
('step', 'replace', "N0'", 'add_committed_row:A@0')
('step', 'replace', "N0'", 'begin_leftover')
('step', 'replace', "N0'", 'untrack_and_offload')
('assign_consumer', "N1'")
('acquire', "N1'")
('consume', "N1'", 1)
('offline_immutable', 'N0')
('step', 'offline', 'N0', 'begin_remove')
('step', 'offline', 'N0', 'remove:A')
('step', 'offline', 'N0', 'untrack')
('finish_index', "N1'", 1)
('consume', "N1'", 2)
('finish_index', "N1'", 2)
('consume', "N1'", 3)
('finish_index', "N1'", 3)
('add_immutable', 'N0')
('step', 'add_immutable', 'N0', 'start_add')
('step', 'add_immutable', 'N0', 'add_row:A@0')
('step', 'add_immutable', 'N0', 'register')
```
</details>
Metering the drop without calling `addRecord()` would close it.
The model has no indexing failures or dedup, so the rest of this PR was not checked.
<!-- upsert-model-check head=b46dd0c4490fa4469bda91649e6545075f88534f -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fail-soft MutableSegmentImpl.index to avoid partial-row corruption [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on PR #19088:
URL: https://github.com/apache/pinot/pull/19088#issuecomment-5109975638
cc @swaminathanmanish @KKcorps @noob-se7en
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fail-soft MutableSegmentImpl.index to avoid partial-row corruption [pinot]
Posted by "Vamsi-klu (via GitHub)" <gi...@apache.org>.
Vamsi-klu commented on code in PR #19088:
URL: https://github.com/apache/pinot/pull/19088#discussion_r3755547427
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImpl.java:
##########
@@ -820,182 +831,419 @@ private void validateLengthOfMVColumns(GenericRow row)
}
}
- private void updateDictionary(GenericRow row) {
+ /// Runs dictionary + forward/secondary indexing for a new docId and meters an incomplete row when either step had
+ /// to fall back to defaults (issue #16316).
+ private void indexPhysicalRow(int docId, GenericRow row) {
+ boolean dictHadError = updateDictionary(row);
+ boolean rowHadError = addNewRow(docId, row);
+ if (dictHadError || rowHadError) {
+ recordIncompleteRow();
+ }
+ }
+
+ /// @return {@code true} if any column required a default/fallback while updating dictionaries
+ private boolean updateDictionary(GenericRow row) {
+ boolean hadError = false;
for (Map.Entry<String, IndexContainer> entry : _indexContainerMap.entrySet()) {
IndexContainer indexContainer = entry.getValue();
MutableDictionary dictionary = indexContainer._dictionary;
if (dictionary == null) {
continue;
}
-
- Object value = row.getValue(entry.getKey());
- if (value == null) {
- recordIndexingError("DICTIONARY");
- } else {
+ String column = entry.getKey();
+ Object value = row.getValue(column);
+ try {
+ if (value == null) {
+ // Prefer default-null dict entry so addNewRow can still complete the forward index for this docId
+ // (fail-soft; issue #16316). Meter and fall back to field-spec default.
+ recordIndexingError("DICTIONARY");
+ hadError = true;
+ value = getDefaultNullValueForIndexing(indexContainer._fieldSpec);
+ row.putDefaultNullValue(column, value);
+ }
if (indexContainer._fieldSpec.isSingleValueField()) {
indexContainer._dictId = dictionary.index(value);
} else {
indexContainer._dictIds = dictionary.index((Object[]) value);
}
-
// Update min/max value from dictionary
indexContainer._minValue = dictionary.getMinVal();
indexContainer._maxValue = dictionary.getMaxVal();
+ } catch (Exception e) {
+ // Do not abort the row mid-dictionary: remaining columns still get a chance, and addNewRow will fill
+ // defaults for this column if dict ids are unset (Integer.MIN_VALUE / null).
+ hadError = true;
+ recordIndexingError("DICTIONARY", e);
+ indexContainer._dictId = Integer.MIN_VALUE;
+ indexContainer._dictIds = null;
}
Review Comment:
Fixed in `4a0dfe3`. The `updateDictionary` catch now indexes the column's default null value into the dictionary before falling back to the sentinel, and it also puts the default back on the row. Without that second half the forward index would hold the default while the secondary indexes still held the raw value, and `setNull` would never fire. The sentinel is kept only if indexing the default throws as well, and the row is metered incomplete either way. Min and max are refreshed from the dictionary in the same block, otherwise seal can still throw the class of error this PR exists to prevent.
One correction on the framing: two failed rows still share the default-value key after this, by design. The real defect was that `Integer.MIN_VALUE` is not a real dict id, so a failed row was keyed on something that could never match the default the row actually stored, and a later row legitimately carrying that default then landed on a second docId with identical dimension content. The new test pins exactly that: two failed rows roll into docId 0, then a row carrying the default joins the same docId. Before the fix it produces two docIds.
##########
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/indexsegment/mutable/IndexingFailureTest.java:
##########
@@ -133,8 +133,46 @@ public void testIndexingFailures()
assertEquals(_mutableSegment.getDataSource(JSON_COL).getJsonIndex().getMatchingDocIds("valid = 'json'"),
ImmutableRoaringBitmap.bitmapOf(0, 2, 3));
assertTrue(_mutableSegment.getDataSource(STRING_COL).getNullValueVector().isNull(3));
- // null string value skipped
+ // Fail-soft (#16316): null string is completed with the field default so forward lengths stay aligned.
+ GenericRow nullResult = _mutableSegment.getRecord(3, new GenericRow());
+ assertEquals(nullResult.getValue(STRING_COL), FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_STRING);
verify(_serverMetrics, times(1)).addMeteredTableValue(matches("DICTIONARY-indexingError$"),
eq(ServerMeter.INDEXING_FAILURES), eq(1L));
Review Comment:
I do not think this one holds. Mockito's `Matches` delegates to `Matcher.find()`, which is substring semantics rather than full-match, so the anchored suffix does match inside `testTable_REALTIME-DICTIONARY-indexingError`. The assertion is also unchanged from master and green in CI, so I have left it as is.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImpl.java:
##########
@@ -820,182 +831,419 @@ private void validateLengthOfMVColumns(GenericRow row)
}
}
- private void updateDictionary(GenericRow row) {
+ /// Runs dictionary + forward/secondary indexing for a new docId and meters an incomplete row when either step had
+ /// to fall back to defaults (issue #16316).
+ private void indexPhysicalRow(int docId, GenericRow row) {
+ boolean dictHadError = updateDictionary(row);
+ boolean rowHadError = addNewRow(docId, row);
+ if (dictHadError || rowHadError) {
+ recordIncompleteRow();
+ }
+ }
+
+ /// @return {@code true} if any column required a default/fallback while updating dictionaries
+ private boolean updateDictionary(GenericRow row) {
+ boolean hadError = false;
for (Map.Entry<String, IndexContainer> entry : _indexContainerMap.entrySet()) {
IndexContainer indexContainer = entry.getValue();
MutableDictionary dictionary = indexContainer._dictionary;
if (dictionary == null) {
continue;
}
-
- Object value = row.getValue(entry.getKey());
- if (value == null) {
- recordIndexingError("DICTIONARY");
- } else {
+ String column = entry.getKey();
+ Object value = row.getValue(column);
+ try {
+ if (value == null) {
+ // Prefer default-null dict entry so addNewRow can still complete the forward index for this docId
+ // (fail-soft; issue #16316). Meter and fall back to field-spec default.
+ recordIndexingError("DICTIONARY");
+ hadError = true;
+ value = getDefaultNullValueForIndexing(indexContainer._fieldSpec);
+ row.putDefaultNullValue(column, value);
+ }
if (indexContainer._fieldSpec.isSingleValueField()) {
indexContainer._dictId = dictionary.index(value);
} else {
indexContainer._dictIds = dictionary.index((Object[]) value);
}
-
// Update min/max value from dictionary
indexContainer._minValue = dictionary.getMinVal();
indexContainer._maxValue = dictionary.getMaxVal();
+ } catch (Exception e) {
+ // Do not abort the row mid-dictionary: remaining columns still get a chance, and addNewRow will fill
+ // defaults for this column if dict ids are unset (Integer.MIN_VALUE / null).
+ hadError = true;
+ recordIndexingError("DICTIONARY", e);
+ indexContainer._dictId = Integer.MIN_VALUE;
+ indexContainer._dictIds = null;
}
- updateIndexCapacityThresholdBreached(dictionary, entry.getKey());
+ updateIndexCapacityThresholdBreached(dictionary, column);
}
+ return hadError;
}
- private void addNewRow(int docId, GenericRow row) {
+ /// Indexes a new physical row. Fail-soft (issue #16316): every physical column must end up with a forward-index
+ /// (or OPEN_STRUCT) entry for [docId] so seal/query lengths stay aligned with [_numDocsIndexed]. On forward-index
+ /// failure the column is completed with the field default/null instead of being left blank. Secondary index
+ /// failures are still metered and swallowed. Aggregation path failures also fall back to a default initial value.
+ ///
+ /// @return {@code true} if any column required a default/fallback while indexing
+ private boolean addNewRow(int docId, GenericRow row) {
+ boolean rowHadError = false;
for (Map.Entry<String, IndexContainer> entry : _indexContainerMap.entrySet()) {
String column = entry.getKey();
IndexContainer indexContainer = entry.getValue();
-
- // Handle ingestion aggregation
- ValueAggregator valueAggregator = indexContainer._valueAggregator;
- if (valueAggregator != null) {
- String sourceColumn = indexContainer._sourceColumn;
- // NOTE: value can be null if the column is not specified in the schema.
- Object value = row.getValue(sourceColumn);
- // Handle COUNT(*)
- if (value == null && sourceColumn.equals(AggregationFunctionColumnPair.STAR)) {
- assert valueAggregator.getAggregationType() == AggregationFunctionType.COUNT;
- value = 1;
+ try {
+ if (indexContainer._valueAggregator != null) {
+ if (!addAggregatedColumn(docId, row, column, indexContainer)) {
+ rowHadError = true;
+ }
+ } else if (!addPhysicalColumn(docId, row, column, indexContainer)) {
+ rowHadError = true;
}
-
- // Update numValues info
- indexContainer._valuesInfo.updateSVNumValues();
-
- MutableIndex forwardIndex = indexContainer._mutableIndexes.get(StandardIndexes.forward());
- FieldSpec fieldSpec = indexContainer._fieldSpec;
-
- DataType dataType = fieldSpec.getDataType();
- value = valueAggregator.getInitialAggregatedValue(value);
- // BIG_DECIMAL is actually stored as byte[] and hence can be supported here.
- switch (dataType.getStoredType()) {
- case INT:
- forwardIndex.add(((Number) value).intValue(), -1, docId);
- break;
- case LONG:
- forwardIndex.add(((Number) value).longValue(), -1, docId);
- break;
- case FLOAT:
- forwardIndex.add(((Number) value).floatValue(), -1, docId);
- break;
- case DOUBLE:
- forwardIndex.add(((Number) value).doubleValue(), -1, docId);
- break;
- case BIG_DECIMAL:
- case BYTES:
- forwardIndex.add(valueAggregator.serializeAggregatedValue(value), -1, docId);
- break;
- default:
- throw new UnsupportedOperationException(
- "Unsupported data type: " + dataType + " for aggregation: " + column);
+ } catch (Exception e) {
+ // Last-resort complete-the-row so a single bad column cannot leave a half-written docId.
+ rowHadError = true;
+ recordIndexingError("ROW", e);
+ try {
+ indexDefaultNullColumn(docId, indexContainer);
+ } catch (Exception fallbackError) {
+ _logger.error("Failed to index default null for column: {} at docId: {}", column, docId, fallbackError);
}
- continue;
}
+ }
- // Update the null value vector even if a null value is somehow produced
- if (indexContainer._nullValueVector != null && row.isNullValue(column)) {
- indexContainer._nullValueVector.setNull(docId);
+ if (_multiColumnValues != null) {
+ try {
+ _multiColumnTextIndex.add(_multiColumnValues);
+ } catch (Exception e) {
+ rowHadError = true;
+ recordIndexingError("MULTI_COLUMN_TEXT", e);
+ } finally {
+ Collections.fill(_multiColumnValues, null);
}
+ }
+ return rowHadError;
+ }
- Object value = row.getValue(column);
- if (value == null) {
- // the value should not be null unless something is broken upstream but this will lead to inappropriate reuse
- // of the dictionary id if this somehow happens. An NPE here can corrupt indexes leading to incorrect query
- // results, hence the extra care. A metric will already have been emitted when trying to update the dictionary.
- continue;
- }
+ /// Returns {@code true} when the aggregated column was written without error.
+ private boolean addAggregatedColumn(int docId, GenericRow row, String column, IndexContainer indexContainer) {
+ ValueAggregator valueAggregator = indexContainer._valueAggregator;
+ String sourceColumn = indexContainer._sourceColumn;
+ // NOTE: value can be null if the column is not specified in the schema.
+ Object value = row.getValue(sourceColumn);
+ // Handle COUNT(*)
+ if (value == null && sourceColumn.equals(AggregationFunctionColumnPair.STAR)) {
+ assert valueAggregator.getAggregationType() == AggregationFunctionType.COUNT;
+ value = 1;
+ }
- FieldSpec fieldSpec = indexContainer._fieldSpec;
- DataType dataType = fieldSpec.getDataType();
+ MutableIndex forwardIndex = indexContainer._mutableIndexes.get(StandardIndexes.forward());
+ FieldSpec fieldSpec = indexContainer._fieldSpec;
+ DataType dataType = fieldSpec.getDataType();
+ try {
+ value = valueAggregator.getInitialAggregatedValue(value);
+ // BIG_DECIMAL is actually stored as byte[] and hence can be supported here.
+ switch (dataType.getStoredType()) {
+ case INT:
+ forwardIndex.add(((Number) value).intValue(), -1, docId);
+ break;
+ case LONG:
+ forwardIndex.add(((Number) value).longValue(), -1, docId);
+ break;
+ case FLOAT:
+ forwardIndex.add(((Number) value).floatValue(), -1, docId);
+ break;
+ case DOUBLE:
+ forwardIndex.add(((Number) value).doubleValue(), -1, docId);
+ break;
+ case BIG_DECIMAL:
+ case BYTES:
+ forwardIndex.add(valueAggregator.serializeAggregatedValue(value), -1, docId);
+ break;
+ default:
+ throw new UnsupportedOperationException(
+ "Unsupported data type: " + dataType + " for aggregation: " + column);
+ }
+ indexContainer._valuesInfo.updateSVNumValues();
+ return true;
+ } catch (Exception e) {
+ recordIndexingError(StandardIndexes.forward(), e);
+ indexDefaultAggregatedValue(docId, indexContainer);
+ return false;
+ }
+ }
- if (fieldSpec.isSingleValueField()) {
- // Update numValues info
+ /// Returns {@code true} when the physical column was written without error.
+ private boolean addPhysicalColumn(int docId, GenericRow row, String column, IndexContainer indexContainer) {
+ FieldSpec fieldSpec = indexContainer._fieldSpec;
+ DataType dataType = fieldSpec.getDataType();
+ boolean isNull = row.isNullValue(column);
+ Object value = row.getValue(column);
+ if (value == null) {
+ // Should not happen after NullValueTransformer, but complete the row with defaults rather than leaving a hole.
+ recordIndexingError("NULL_VALUE");
+ value = getDefaultNullValueForIndexing(fieldSpec);
+ isNull = true;
+ }
Review Comment:
Fixed in `4a0dfe3`. `addPhysicalColumn` tracks a `defaultSubstituted` flag now, folded into the SV, MV and OPEN_STRUCT return paths, so `addNewRow` meters the row incomplete exactly once per row no matter how many columns fell back. There is a test with one row that only substitutes and one that substitutes and also fails its JSON index, asserting exactly two meters for two rows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fail-soft MutableSegmentImpl.index to avoid partial-row corruption [pinot]
Posted by "Vamsi-klu (via GitHub)" <gi...@apache.org>.
Vamsi-klu commented on PR #19088:
URL: https://github.com/apache/pinot/pull/19088#issuecomment-5077683610
**Ready for review** — all required CI checks are green on this PR.
Issue: #16316
Could the following folks take a look when convenient?
@Jackie-Jiang
Formal GitHub "Request review" is unavailable from a fork contributor account on `apache/pinot` (API returns 404/403), so this is the review ping.
Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fail-soft MutableSegmentImpl.index to avoid partial-row corruption [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #19088:
URL: https://github.com/apache/pinot/pull/19088#discussion_r3762697935
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImpl.java:
##########
@@ -695,21 +696,31 @@ public boolean index(GenericRow row, @Nullable StreamMessageMetadata metadata)
_numDocsIndexed = numDocsIndexed;
}
} else {
- // Update dictionary first
- updateDictionary(row);
-
// If metrics aggregation is enabled and if the dimension values were already seen, this will return existing
- // docId, else this will return a new docId.
+ // docId, else this will return a new docId. Dictionary must be updated before the rollup key is computed.
+ boolean dictHadError = updateDictionary(row);
int docId = getOrCreateDocId();
if (docId == numDocsIndexed) {
- // New row
- addNewRow(numDocsIndexed, row);
+ // New row — fail-soft complete-the-row so _numDocsIndexed and per-column lengths stay aligned (#16316).
+ boolean rowHadError = addNewRow(numDocsIndexed, row);
+ if (dictHadError || rowHadError) {
+ recordIncompleteRow();
+ }
// Update number of documents indexed at last to make the latest row queryable
canTakeMore = numDocsIndexed++ < _capacity;
} else {
assert isAggregateMetricsEnabled();
- aggregateMetrics(row, docId);
+ try {
+ aggregateMetrics(row, docId);
+ if (dictHadError) {
+ recordIncompleteRow();
+ }
+ } catch (Exception e) {
+ // In-place rollup already has a complete prior row; meter and keep previous metric values.
+ recordIndexingError("AGGREGATE_METRICS", e);
+ recordIncompleteRow();
Review Comment:
`aggregateMetrics()` mutates each metric as it iterates, so this catch cannot “keep previous metric values.” If one metric succeeds and a later aggregator rejects its input (the multi-metric form of #16317), the first metric remains updated while the later metric remains unchanged, silently applying only part of the row. Compute/validate all metric updates before committing them, or restore every previously written metric when any aggregation fails.
This issue also appears in the following locations of the same file:
- line 924
- line 1071
- line 1162
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fail-soft MutableSegmentImpl.index to avoid partial-row corruption [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #19088:
URL: https://github.com/apache/pinot/pull/19088#discussion_r3669414589
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImpl.java:
##########
@@ -820,182 +831,419 @@ private void validateLengthOfMVColumns(GenericRow row)
}
}
- private void updateDictionary(GenericRow row) {
+ /// Runs dictionary + forward/secondary indexing for a new docId and meters an incomplete row when either step had
+ /// to fall back to defaults (issue #16316).
+ private void indexPhysicalRow(int docId, GenericRow row) {
+ boolean dictHadError = updateDictionary(row);
+ boolean rowHadError = addNewRow(docId, row);
+ if (dictHadError || rowHadError) {
+ recordIncompleteRow();
+ }
+ }
+
+ /// @return {@code true} if any column required a default/fallback while updating dictionaries
+ private boolean updateDictionary(GenericRow row) {
+ boolean hadError = false;
for (Map.Entry<String, IndexContainer> entry : _indexContainerMap.entrySet()) {
IndexContainer indexContainer = entry.getValue();
MutableDictionary dictionary = indexContainer._dictionary;
if (dictionary == null) {
continue;
}
-
- Object value = row.getValue(entry.getKey());
- if (value == null) {
- recordIndexingError("DICTIONARY");
- } else {
+ String column = entry.getKey();
+ Object value = row.getValue(column);
+ try {
+ if (value == null) {
+ // Prefer default-null dict entry so addNewRow can still complete the forward index for this docId
+ // (fail-soft; issue #16316). Meter and fall back to field-spec default.
+ recordIndexingError("DICTIONARY");
+ hadError = true;
+ value = getDefaultNullValueForIndexing(indexContainer._fieldSpec);
+ row.putDefaultNullValue(column, value);
+ }
if (indexContainer._fieldSpec.isSingleValueField()) {
indexContainer._dictId = dictionary.index(value);
} else {
indexContainer._dictIds = dictionary.index((Object[]) value);
}
-
// Update min/max value from dictionary
indexContainer._minValue = dictionary.getMinVal();
indexContainer._maxValue = dictionary.getMaxVal();
+ } catch (Exception e) {
+ // Do not abort the row mid-dictionary: remaining columns still get a chance, and addNewRow will fill
+ // defaults for this column if dict ids are unset (Integer.MIN_VALUE / null).
+ hadError = true;
+ recordIndexingError("DICTIONARY", e);
+ indexContainer._dictId = Integer.MIN_VALUE;
+ indexContainer._dictIds = null;
}
Review Comment:
In updateDictionary(), the exception path sets _dictId=Integer.MIN_VALUE. When metrics aggregation is enabled, getOrCreateDocId() uses these dictIds as the rollup key, so a dict failure can silently roll up unrelated rows together (or otherwise corrupt aggregation keying).
Instead of leaving a sentinel, try to fall back to the field default-null value and index that into the dictionary so key columns remain valid; only fall back to Integer.MIN_VALUE if even the default cannot be indexed.
This issue also appears in the following locations of the same file:
- line 1028
- line 1119
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImpl.java:
##########
@@ -820,182 +831,419 @@ private void validateLengthOfMVColumns(GenericRow row)
}
}
- private void updateDictionary(GenericRow row) {
+ /// Runs dictionary + forward/secondary indexing for a new docId and meters an incomplete row when either step had
+ /// to fall back to defaults (issue #16316).
+ private void indexPhysicalRow(int docId, GenericRow row) {
+ boolean dictHadError = updateDictionary(row);
+ boolean rowHadError = addNewRow(docId, row);
+ if (dictHadError || rowHadError) {
+ recordIncompleteRow();
+ }
+ }
+
+ /// @return {@code true} if any column required a default/fallback while updating dictionaries
+ private boolean updateDictionary(GenericRow row) {
+ boolean hadError = false;
for (Map.Entry<String, IndexContainer> entry : _indexContainerMap.entrySet()) {
IndexContainer indexContainer = entry.getValue();
MutableDictionary dictionary = indexContainer._dictionary;
if (dictionary == null) {
continue;
}
-
- Object value = row.getValue(entry.getKey());
- if (value == null) {
- recordIndexingError("DICTIONARY");
- } else {
+ String column = entry.getKey();
+ Object value = row.getValue(column);
+ try {
+ if (value == null) {
+ // Prefer default-null dict entry so addNewRow can still complete the forward index for this docId
+ // (fail-soft; issue #16316). Meter and fall back to field-spec default.
+ recordIndexingError("DICTIONARY");
+ hadError = true;
+ value = getDefaultNullValueForIndexing(indexContainer._fieldSpec);
+ row.putDefaultNullValue(column, value);
+ }
if (indexContainer._fieldSpec.isSingleValueField()) {
indexContainer._dictId = dictionary.index(value);
} else {
indexContainer._dictIds = dictionary.index((Object[]) value);
}
-
// Update min/max value from dictionary
indexContainer._minValue = dictionary.getMinVal();
indexContainer._maxValue = dictionary.getMaxVal();
+ } catch (Exception e) {
+ // Do not abort the row mid-dictionary: remaining columns still get a chance, and addNewRow will fill
+ // defaults for this column if dict ids are unset (Integer.MIN_VALUE / null).
+ hadError = true;
+ recordIndexingError("DICTIONARY", e);
+ indexContainer._dictId = Integer.MIN_VALUE;
+ indexContainer._dictIds = null;
}
- updateIndexCapacityThresholdBreached(dictionary, entry.getKey());
+ updateIndexCapacityThresholdBreached(dictionary, column);
}
+ return hadError;
}
- private void addNewRow(int docId, GenericRow row) {
+ /// Indexes a new physical row. Fail-soft (issue #16316): every physical column must end up with a forward-index
+ /// (or OPEN_STRUCT) entry for [docId] so seal/query lengths stay aligned with [_numDocsIndexed]. On forward-index
+ /// failure the column is completed with the field default/null instead of being left blank. Secondary index
+ /// failures are still metered and swallowed. Aggregation path failures also fall back to a default initial value.
+ ///
+ /// @return {@code true} if any column required a default/fallback while indexing
+ private boolean addNewRow(int docId, GenericRow row) {
+ boolean rowHadError = false;
for (Map.Entry<String, IndexContainer> entry : _indexContainerMap.entrySet()) {
String column = entry.getKey();
IndexContainer indexContainer = entry.getValue();
-
- // Handle ingestion aggregation
- ValueAggregator valueAggregator = indexContainer._valueAggregator;
- if (valueAggregator != null) {
- String sourceColumn = indexContainer._sourceColumn;
- // NOTE: value can be null if the column is not specified in the schema.
- Object value = row.getValue(sourceColumn);
- // Handle COUNT(*)
- if (value == null && sourceColumn.equals(AggregationFunctionColumnPair.STAR)) {
- assert valueAggregator.getAggregationType() == AggregationFunctionType.COUNT;
- value = 1;
+ try {
+ if (indexContainer._valueAggregator != null) {
+ if (!addAggregatedColumn(docId, row, column, indexContainer)) {
+ rowHadError = true;
+ }
+ } else if (!addPhysicalColumn(docId, row, column, indexContainer)) {
+ rowHadError = true;
}
-
- // Update numValues info
- indexContainer._valuesInfo.updateSVNumValues();
-
- MutableIndex forwardIndex = indexContainer._mutableIndexes.get(StandardIndexes.forward());
- FieldSpec fieldSpec = indexContainer._fieldSpec;
-
- DataType dataType = fieldSpec.getDataType();
- value = valueAggregator.getInitialAggregatedValue(value);
- // BIG_DECIMAL is actually stored as byte[] and hence can be supported here.
- switch (dataType.getStoredType()) {
- case INT:
- forwardIndex.add(((Number) value).intValue(), -1, docId);
- break;
- case LONG:
- forwardIndex.add(((Number) value).longValue(), -1, docId);
- break;
- case FLOAT:
- forwardIndex.add(((Number) value).floatValue(), -1, docId);
- break;
- case DOUBLE:
- forwardIndex.add(((Number) value).doubleValue(), -1, docId);
- break;
- case BIG_DECIMAL:
- case BYTES:
- forwardIndex.add(valueAggregator.serializeAggregatedValue(value), -1, docId);
- break;
- default:
- throw new UnsupportedOperationException(
- "Unsupported data type: " + dataType + " for aggregation: " + column);
+ } catch (Exception e) {
+ // Last-resort complete-the-row so a single bad column cannot leave a half-written docId.
+ rowHadError = true;
+ recordIndexingError("ROW", e);
+ try {
+ indexDefaultNullColumn(docId, indexContainer);
+ } catch (Exception fallbackError) {
+ _logger.error("Failed to index default null for column: {} at docId: {}", column, docId, fallbackError);
}
- continue;
}
+ }
- // Update the null value vector even if a null value is somehow produced
- if (indexContainer._nullValueVector != null && row.isNullValue(column)) {
- indexContainer._nullValueVector.setNull(docId);
+ if (_multiColumnValues != null) {
+ try {
+ _multiColumnTextIndex.add(_multiColumnValues);
+ } catch (Exception e) {
+ rowHadError = true;
+ recordIndexingError("MULTI_COLUMN_TEXT", e);
+ } finally {
+ Collections.fill(_multiColumnValues, null);
}
+ }
+ return rowHadError;
+ }
- Object value = row.getValue(column);
- if (value == null) {
- // the value should not be null unless something is broken upstream but this will lead to inappropriate reuse
- // of the dictionary id if this somehow happens. An NPE here can corrupt indexes leading to incorrect query
- // results, hence the extra care. A metric will already have been emitted when trying to update the dictionary.
- continue;
- }
+ /// Returns {@code true} when the aggregated column was written without error.
+ private boolean addAggregatedColumn(int docId, GenericRow row, String column, IndexContainer indexContainer) {
+ ValueAggregator valueAggregator = indexContainer._valueAggregator;
+ String sourceColumn = indexContainer._sourceColumn;
+ // NOTE: value can be null if the column is not specified in the schema.
+ Object value = row.getValue(sourceColumn);
+ // Handle COUNT(*)
+ if (value == null && sourceColumn.equals(AggregationFunctionColumnPair.STAR)) {
+ assert valueAggregator.getAggregationType() == AggregationFunctionType.COUNT;
+ value = 1;
+ }
- FieldSpec fieldSpec = indexContainer._fieldSpec;
- DataType dataType = fieldSpec.getDataType();
+ MutableIndex forwardIndex = indexContainer._mutableIndexes.get(StandardIndexes.forward());
+ FieldSpec fieldSpec = indexContainer._fieldSpec;
+ DataType dataType = fieldSpec.getDataType();
+ try {
+ value = valueAggregator.getInitialAggregatedValue(value);
+ // BIG_DECIMAL is actually stored as byte[] and hence can be supported here.
+ switch (dataType.getStoredType()) {
+ case INT:
+ forwardIndex.add(((Number) value).intValue(), -1, docId);
+ break;
+ case LONG:
+ forwardIndex.add(((Number) value).longValue(), -1, docId);
+ break;
+ case FLOAT:
+ forwardIndex.add(((Number) value).floatValue(), -1, docId);
+ break;
+ case DOUBLE:
+ forwardIndex.add(((Number) value).doubleValue(), -1, docId);
+ break;
+ case BIG_DECIMAL:
+ case BYTES:
+ forwardIndex.add(valueAggregator.serializeAggregatedValue(value), -1, docId);
+ break;
+ default:
+ throw new UnsupportedOperationException(
+ "Unsupported data type: " + dataType + " for aggregation: " + column);
+ }
+ indexContainer._valuesInfo.updateSVNumValues();
+ return true;
+ } catch (Exception e) {
+ recordIndexingError(StandardIndexes.forward(), e);
+ indexDefaultAggregatedValue(docId, indexContainer);
+ return false;
+ }
+ }
- if (fieldSpec.isSingleValueField()) {
- // Update numValues info
+ /// Returns {@code true} when the physical column was written without error.
+ private boolean addPhysicalColumn(int docId, GenericRow row, String column, IndexContainer indexContainer) {
+ FieldSpec fieldSpec = indexContainer._fieldSpec;
+ DataType dataType = fieldSpec.getDataType();
+ boolean isNull = row.isNullValue(column);
+ Object value = row.getValue(column);
+ if (value == null) {
+ // Should not happen after NullValueTransformer, but complete the row with defaults rather than leaving a hole.
+ recordIndexingError("NULL_VALUE");
+ value = getDefaultNullValueForIndexing(fieldSpec);
+ isNull = true;
+ }
Review Comment:
When row.getValue(column) is null, addPhysicalColumn() falls back to the field default and logs an indexing error, but it can still return true (because this fallback isn't included in the hadError/return value). That means callers won't meter the row as incomplete even though a default was substituted.
If the intent is that any default substitution counts as an incomplete row (per PR description and method contract), the method should propagate this fallback as a failure signal (or otherwise ensure recordIncompleteRow() is triggered once per row).
##########
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/indexsegment/mutable/IndexingFailureTest.java:
##########
@@ -133,8 +133,46 @@ public void testIndexingFailures()
assertEquals(_mutableSegment.getDataSource(JSON_COL).getJsonIndex().getMatchingDocIds("valid = 'json'"),
ImmutableRoaringBitmap.bitmapOf(0, 2, 3));
assertTrue(_mutableSegment.getDataSource(STRING_COL).getNullValueVector().isNull(3));
- // null string value skipped
+ // Fail-soft (#16316): null string is completed with the field default so forward lengths stay aligned.
+ GenericRow nullResult = _mutableSegment.getRecord(3, new GenericRow());
+ assertEquals(nullResult.getValue(STRING_COL), FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_STRING);
verify(_serverMetrics, times(1)).addMeteredTableValue(matches("DICTIONARY-indexingError$"),
eq(ServerMeter.INDEXING_FAILURES), eq(1L));
Review Comment:
This matcher won't match the actual metric key being emitted. MutableSegmentImpl.recordIndexingError("DICTIONARY") prefixes the key with the realtime table name (e.g. "testTable_REALTIME-DICTIONARY-indexingError"), so matching only "DICTIONARY-indexingError$" will fail.
Update the expectation to include the table prefix (or use a regex with ".*-" prefix) so the test asserts the same key that production code emits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fail-soft MutableSegmentImpl.index to avoid partial-row corruption [pinot]
Posted by "Vamsi-klu (via GitHub)" <gi...@apache.org>.
Vamsi-klu commented on PR #19088:
URL: https://github.com/apache/pinot/pull/19088#issuecomment-5642196438
cc @Jackie-Jiang @noob-se7en fail-soft row indexing for #16316
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fail-soft MutableSegmentImpl.index to avoid partial-row corruption [pinot]
Posted by "Jackie-Jiang (via GitHub) via commits" <co...@pinot.apache.org>.
Jackie-Jiang commented on code in PR #19088:
URL: https://github.com/apache/pinot/pull/19088#discussion_r4449000588
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/dedup/PartitionDedupMetadataManager.java:
##########
@@ -55,6 +55,14 @@ default void replaceSegment(IndexSegment oldSegment, IndexSegment newSegment) {
/// Remove the expired primary keys from the metadata when TTL is enabled.
void removeExpiredPrimaryKeys();
+ /// Returns true if the primary key is already present and within TTL. Does not insert the key.
+ ///
+ /// Call this before writing a new consuming row so a failed write cannot leave a ghost key that drops a later retry.
+ /// Default is absent so existing test doubles keep the previous always-insert behavior until they override it.
+ default boolean isRecordPresent(DedupRecordInfo dedupRecordInfo) {
+ return false;
Review Comment:
[P1] Preserve deduplication for existing metadata-manager plugins
A custom metadata manager implementing the existing interface inherits false here. MutableSegmentImpl now publishes the row before calling checkRecordPresentOrUpdate(), and ignores that method's duplicate result. Thus a previously working plugin can return true for an existing key and still retain every duplicate after this change. These are supported production extensions via DedupConfig.metadataManagerClass, not just test doubles. Please preserve the old contract for implementations without the new capability, or provide a compatible migration. Add a regression using a legacy implementation that only overrides checkRecordPresentOrUpdate(), and ingest the same key twice.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImpl.java:
##########
@@ -674,82 +697,105 @@ public boolean index(GenericRow row, @Nullable StreamMessageMetadata metadata)
boolean canTakeMore;
// NOTE: out-of-order records can not be dropped or marked when consistent upsert view is enabled.
- // Since Indexing the record and updation of _numDocsIndexed counter happens before updating the upsert
- // metadata, we wouldn't be able to actually drop or mark those records as dropped. This order is important for
- // consistent upsert view, otherwise the latest doc can be missed by query due to 'docId < _numDocs' check
- // in query filter operators. Here the record becomes queryable before validDocIds bitmaps are updated.
- if (_upsertConsistencyMode != UpsertConfig.ConsistencyMode.NONE) {
- updateDictionary(updatedRow);
- addNewRow(numDocsIndexed, updatedRow);
- numDocsIndexed++;
- canTakeMore = numDocsIndexed < _capacity;
- _numDocsIndexed = numDocsIndexed;
- // Index the record and update _numDocsIndexed counter before updating the upsert metadata so that the record
- // becomes queryable before validDocIds bitmaps are updated. This order is important for consistent upsert view,
- // otherwise the latest doc can be missed by query due to 'docId < _numDocs' check in query filter operators.
- // NOTE: out-of-order records can not be dropped or marked when consistent upsert view is enabled.
- _partitionUpsertMetadataManager.addRecord(this, recordInfo);
- } else {
- // if record doesn't need to be dropped, then persist in segment and update metadata hashmap
- // we are doing metadata update first followed by segment data update here, there can be a scenario where
- // segment indexing or addNewRow call errors out in those scenario, there can be metadata inconsistency where
- // a key is pointing to some other key's docID
- // TODO fix this metadata mismatch scenario
- boolean isOutOfOrderRecord = !_partitionUpsertMetadataManager.addRecord(this, recordInfo);
+ // Peek first so drop/mark can skip the write without mutating validDocIds. Then finish the physical row
+ // before addRecord so metadata never points at a hole (issue #16316).
+ if (!_partitionUpsertMetadataManager.isAcceptingRecords()) {
+ updateIndexedAndIngestionTime(metadata);
+ return canAddMore();
+ }
+ if (_upsertConsistencyMode == UpsertConfig.ConsistencyMode.NONE) {
+ boolean isOutOfOrderRecord = _partitionUpsertMetadataManager.isOutOfOrderRecord(recordInfo);
if (_upsertOutOfOrderRecordColumn != null) {
updatedRow.putValue(_upsertOutOfOrderRecordColumn, BooleanUtils.toInt(isOutOfOrderRecord));
}
- if (!isOutOfOrderRecord || !_upsertDropOutOfOrderRecord) {
- updateDictionary(updatedRow);
- addNewRow(numDocsIndexed, updatedRow);
- // Update number of documents indexed before handling the upsert metadata so that the record becomes queryable
- // once validated
- numDocsIndexed++;
+ if (isOutOfOrderRecord && _upsertDropOutOfOrderRecord) {
+ // Not addRecord(): the check above ran without the key's lock, and if the key's segment was removed since
+ // then, addRecord() would treat it as new and mark the unwritten doc `numDocsIndexed` valid.
+ _partitionUpsertMetadataManager.dropOutOfOrderRecord(this, recordInfo);
+ updateIndexedAndIngestionTime(metadata);
+ return canAddMore();
}
- canTakeMore = numDocsIndexed < _capacity;
- _numDocsIndexed = numDocsIndexed;
}
+ indexPhysicalRow(numDocsIndexed, updatedRow);
+ numDocsIndexed++;
+ canTakeMore = numDocsIndexed < _capacity;
+ _numDocsIndexed = numDocsIndexed;
+ // Index the record and update _numDocsIndexed counter before updating the upsert metadata so that the record
+ // becomes queryable before validDocIds bitmaps are updated. This order is important for consistent upsert view,
+ // otherwise the latest doc can be missed by query due to 'docId < _numDocs' check in query filter operators.
+ _partitionUpsertMetadataManager.addRecord(this, recordInfo);
Review Comment:
[P1] Keep repaired upsert fields consistent with the metadata being claimed
recordInfo was captured before indexPhysicalRow(), but that method can replace primary-key, comparison, or delete fields with defaults. This call then claims metadata for the original values while publishing a different physical row. For example, a recoverable dictionary failure on comparison value 20 can store its default while metadata still claims version 20, causing a later valid version 10 to be rejected; rebuilding metadata from the segment yields different ordering. A repaired primary key similarly disagrees with the map key, and a repaired delete flag can disagree with the queryable bitmap. Please handle metadata-defining field failures explicitly: avoid validating a substituted row under the original metadata, or finalize the physical values before constructing metadata with a defined recovery policy. Cover PK, comparison, and delete-field fallback.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImpl.java:
##########
@@ -852,192 +898,527 @@ private void validateNumMultiValues(GenericRow row) {
}
}
- private void updateDictionary(GenericRow row) {
+ /// Runs dictionary + forward/secondary indexing for a new docId and meters an incomplete row when either step had
+ /// to fall back to defaults (issue #16316).
+ private void indexPhysicalRow(int docId, GenericRow row) {
+ RowWriteState state = _rowWriteState;
+ boolean dictHadError = updateDictionary(row, state);
+ boolean rowHadError = addNewRow(docId, row, state);
+ if (dictHadError || rowHadError) {
+ recordIncompleteRow();
+ }
+ if (state._repairFailed) {
+ markUnrecoverable(state);
+ throwPendingRowIndexingExceptionIfStrict();
+ throw new IllegalStateException(
+ "Mutable segment " + _segmentName + " is terminal after an unrecoverable indexing failure");
+ }
+ }
+
+ /// @return {@code true} if any column required a default/fallback while updating dictionaries.
+ /// When `continueOnError` is false, dictionary failures are stashed and rethrown after the row is published.
+ private boolean updateDictionary(GenericRow row, RowWriteState state) {
+ boolean hadError = false;
for (Map.Entry<String, IndexContainer> entry : _indexContainerMap.entrySet()) {
IndexContainer indexContainer = entry.getValue();
MutableDictionary dictionary = indexContainer._dictionary;
if (dictionary == null) {
continue;
}
-
- Object value = row.getValue(entry.getKey());
- if (value == null) {
- recordIndexingError("DICTIONARY");
- } else {
+ String column = entry.getKey();
+ Object value = row.getValue(column);
+ try {
+ if (value == null) {
+ recordIndexingError("DICTIONARY");
+ hadError = true;
+ state.markIncomplete();
+ value = getDefaultNullValueForIndexing(indexContainer._fieldSpec);
+ row.putDefaultNullValue(column, value);
+ }
if (indexContainer._fieldSpec.isSingleValueField()) {
indexContainer._dictId = dictionary.index(value);
} else {
indexContainer._dictIds = dictionary.index((Object[]) value);
}
-
- // Update min/max value from dictionary
indexContainer._minValue = dictionary.getMinVal();
indexContainer._maxValue = dictionary.getMaxVal();
+ } catch (Exception e) {
+ recordOrDeferIndexingError("DICTIONARY", e);
+ hadError = true;
+ state.noteError(e);
+ indexContainer._dictId = Integer.MIN_VALUE;
+ indexContainer._dictIds = null;
+ try {
+ // Index the field default instead of leaving the sentinel: with metrics aggregation the rollup key is built
+ // from the dict ids (see getOrCreateDocId), and Integer.MIN_VALUE is not a real dict id.
+ Object defaultValue = getDefaultNullValueForIndexing(indexContainer._fieldSpec);
+ if (indexContainer._fieldSpec.isSingleValueField()) {
+ indexContainer._dictId = dictionary.index(defaultValue);
+ } else {
+ indexContainer._dictIds = dictionary.index((Object[]) defaultValue);
+ }
+ row.putDefaultNullValue(column, defaultValue);
+ indexContainer._minValue = dictionary.getMinVal();
+ indexContainer._maxValue = dictionary.getMaxVal();
+ } catch (Exception fallbackError) {
+ _logger.error("Failed to index default null value into dictionary for column: {}", column, fallbackError);
+ state.noteRepairFailure(fallbackError);
+ }
}
- updateIndexCapacityThresholdBreached(dictionary, entry.getKey());
+ updateIndexCapacityThresholdBreached(dictionary, column);
}
+ return hadError;
}
- private void addNewRow(int docId, GenericRow row) {
+ /// Indexes a new physical row. Always completes the row so seal/query lengths stay aligned with [_numDocsIndexed]
+ /// (issue #16316). Forward indexes are written first from a finalized value; secondary indexes then consume that
+ /// same value. On unrecoverable repair the segment is marked terminal.
+ ///
+ /// @return {@code true} if any column required a default/fallback while indexing
+ private boolean addNewRow(int docId, GenericRow row, RowWriteState state) {
+ boolean rowHadError = false;
for (Map.Entry<String, IndexContainer> entry : _indexContainerMap.entrySet()) {
String column = entry.getKey();
IndexContainer indexContainer = entry.getValue();
-
- // Handle ingestion aggregation
- ValueAggregator valueAggregator = indexContainer._valueAggregator;
- if (valueAggregator != null) {
- String sourceColumn = indexContainer._sourceColumn;
- // NOTE: value can be null if the column is not specified in the schema.
- Object value = row.getValue(sourceColumn);
- // Handle COUNT(*)
- if (value == null && sourceColumn.equals(AggregationFunctionColumnPair.STAR)) {
- assert valueAggregator.getAggregationType() == AggregationFunctionType.COUNT;
- value = 1;
+ try {
+ if (indexContainer._valueAggregator != null) {
+ if (!addAggregatedColumn(docId, row, column, indexContainer, state)) {
+ rowHadError = true;
+ }
+ } else if (!addPhysicalColumn(docId, row, column, indexContainer, state)) {
+ rowHadError = true;
}
-
- // Update numValues info
- indexContainer._valuesInfo.updateSVNumValues();
-
- MutableIndex forwardIndex = indexContainer._mutableIndexes.get(StandardIndexes.forward());
- FieldSpec fieldSpec = indexContainer._fieldSpec;
-
- DataType dataType = fieldSpec.getDataType();
- value = valueAggregator.getInitialAggregatedValue(value);
- // BIG_DECIMAL is actually stored as byte[] and hence can be supported here.
- switch (dataType.getStoredType()) {
- case INT:
- forwardIndex.add(((Number) value).intValue(), -1, docId);
- break;
- case LONG:
- forwardIndex.add(((Number) value).longValue(), -1, docId);
- break;
- case FLOAT:
- forwardIndex.add(((Number) value).floatValue(), -1, docId);
- break;
- case DOUBLE:
- forwardIndex.add(((Number) value).doubleValue(), -1, docId);
- break;
- case BIG_DECIMAL:
- case BYTES:
- forwardIndex.add(valueAggregator.serializeAggregatedValue(value), -1, docId);
- break;
- default:
- throw new UnsupportedOperationException(
- "Unsupported data type: " + dataType + " for aggregation: " + column);
+ } catch (Exception e) {
+ recordOrDeferIndexingError("ROW", e);
+ rowHadError = true;
+ state.noteError(e);
+ try {
+ indexDefaultNullColumn(docId, column, indexContainer, state);
+ } catch (Exception fallbackError) {
+ _logger.error("Failed to index default null for column: {} at docId: {}", column, docId, fallbackError);
+ state.noteRepairFailure(fallbackError);
}
- continue;
}
+ }
- // Update the null value vector even if a null value is somehow produced
- if (indexContainer._nullValueVector != null && row.isNullValue(column)) {
- indexContainer._nullValueVector.setNull(docId);
+ if (_multiColumnValues != null) {
+ try {
+ // Multi-column text is a row-level secondary index. It is written only after every column has a finalized
+ // value in `_multiColumnValues`. There is no per-doc rollback; a failure leaves this doc absent from the
+ // text index while forward indexes remain complete. Counted once as a secondary-index error.
+ _multiColumnTextIndex.add(_multiColumnValues);
+ } catch (Exception e) {
+ recordOrDeferIndexingError("MULTI_COLUMN_TEXT", e);
+ rowHadError = true;
+ state.noteError(e);
+ } finally {
+ Collections.fill(_multiColumnValues, null);
}
+ }
+ if (state._repairFailed) {
+ markUnrecoverable(state);
+ }
+ return rowHadError;
+ }
- Object value = row.getValue(column);
- if (value == null) {
- // the value should not be null unless something is broken upstream but this will lead to inappropriate reuse
- // of the dictionary id if this somehow happens. An NPE here can corrupt indexes leading to incorrect query
- // results, hence the extra care. A metric will already have been emitted when trying to update the dictionary.
- continue;
+ /// Returns {@code true} when the aggregated column was written without error.
+ private boolean addAggregatedColumn(int docId, GenericRow row, String column, IndexContainer indexContainer,
+ RowWriteState state) {
+ ValueAggregator valueAggregator = indexContainer._valueAggregator;
+ String sourceColumn = indexContainer._sourceColumn;
+ Object value = row.getValue(sourceColumn);
+ if (value == null && sourceColumn.equals(AggregationFunctionColumnPair.STAR)) {
+ assert valueAggregator.getAggregationType() == AggregationFunctionType.COUNT;
+ value = 1;
+ }
+
+ MutableIndex forwardIndex = indexContainer._mutableIndexes.get(StandardIndexes.forward());
+ FieldSpec fieldSpec = indexContainer._fieldSpec;
+ DataType dataType = fieldSpec.getDataType();
+ try {
+ value = valueAggregator.getInitialAggregatedValue(value);
+ addAggregatedForwardValue(forwardIndex, valueAggregator, dataType, value, docId, column);
+ indexContainer._valuesInfo.updateSVNumValues();
+ state._columnsWithCanonicalForward.add(column);
+ return true;
+ } catch (Exception e) {
+ recordOrDeferIndexingError(StandardIndexes.forward(), e);
+ state.noteError(e);
+ try {
+ indexDefaultAggregatedValue(docId, column, indexContainer, state);
+ } catch (Exception fallbackError) {
+ state.noteRepairFailure(fallbackError);
}
+ return false;
+ }
+ }
- FieldSpec fieldSpec = indexContainer._fieldSpec;
- DataType dataType = fieldSpec.getDataType();
+ private void addAggregatedForwardValue(MutableIndex forwardIndex, ValueAggregator valueAggregator, DataType dataType,
+ Object value, int docId, String column) {
+ maybeIntercept(column, StandardIndexes.forward(), value, docId);
+ switch (dataType.getStoredType()) {
+ case INT:
+ forwardIndex.add(((Number) value).intValue(), -1, docId);
+ break;
+ case LONG:
+ forwardIndex.add(((Number) value).longValue(), -1, docId);
+ break;
+ case FLOAT:
+ forwardIndex.add(((Number) value).floatValue(), -1, docId);
+ break;
+ case DOUBLE:
+ forwardIndex.add(((Number) value).doubleValue(), -1, docId);
+ break;
+ case BIG_DECIMAL:
+ case BYTES:
+ forwardIndex.add(valueAggregator.serializeAggregatedValue(value), -1, docId);
+ break;
+ default:
+ throw new UnsupportedOperationException("Unsupported data type: " + dataType + " for aggregation: " + column);
+ }
+ }
- if (fieldSpec.isSingleValueField()) {
- // Update numValues info
- indexContainer._valuesInfo.updateSVNumValues();
-
- // Route OPEN_STRUCT values to the dedicated mutable index. OPEN_STRUCT has no forward
- // index / dictionary / min-max, so the standard per-IndexType loop and the comparable
- // tracking below would be no-ops at best and crash at worst (Map is not Comparable).
- if (dataType == DataType.OPEN_STRUCT) {
- MutableIndex openStructIndex = indexContainer._mutableIndexes.get(StandardIndexes.openStruct());
- if (openStructIndex != null) {
- openStructIndex.add(value, -1, docId);
- }
- continue;
+ /// Returns {@code true} when the physical column was written from the row value without error and without falling
+ /// back to the field default.
+ private boolean addPhysicalColumn(int docId, GenericRow row, String column, IndexContainer indexContainer,
+ RowWriteState state) {
+ FieldSpec fieldSpec = indexContainer._fieldSpec;
+ DataType dataType = fieldSpec.getDataType();
+ boolean isNull = row.isNullValue(column);
+ Object value = row.getValue(column);
+ boolean defaultSubstituted = false;
+ if (value == null) {
+ recordIndexingError("NULL_VALUE");
+ value = getDefaultNullValueForIndexing(fieldSpec);
+ isNull = true;
+ defaultSubstituted = true;
+ state.markIncomplete();
+ }
+
+ if (fieldSpec.isSingleValueField()) {
+ if (dataType == DataType.OPEN_STRUCT) {
+ return addOpenStructColumn(docId, column, indexContainer, value, defaultSubstituted, state);
+ }
+ int dictId = indexContainer._dictId;
+ if (indexContainer._dictionary != null && dictId == Integer.MIN_VALUE) {
+ try {
+ Object defaultValue = getDefaultNullValueForIndexing(fieldSpec);
+ dictId = indexContainer._dictionary.index(defaultValue);
+ indexContainer._dictId = dictId;
+ value = defaultValue;
+ isNull = true;
+ defaultSubstituted = true;
+ state.markIncomplete();
+ } catch (Exception e) {
+ recordOrDeferIndexingError("DICTIONARY", e);
+ state.noteError(e);
+ state.noteRepairFailure(e);
+ return false;
}
+ }
+ Object parsedJsonValue = defaultSubstituted ? null : row.getParsedJsonValue(column);
+ boolean hadError = writeSingleValueIndexes(docId, column, indexContainer, fieldSpec, dataType, value, dictId,
+ isNull, parsedJsonValue, state);
+ return !hadError && !defaultSubstituted;
+ }
- // Update indexes
- int dictId = indexContainer._dictId;
- for (Map.Entry<IndexType, MutableIndex> indexEntry : indexContainer._mutableIndexes.entrySet()) {
- try {
- MutableIndex mutableIndex = indexEntry.getValue();
- if (mutableIndex instanceof MutableJsonIndexImpl) {
- // Feed the JSON index the parsed value cached by the DataTypeTransformer (if any), so it flattens the
- // document directly instead of re-parsing the string this column was serialized into for the forward
- // index; the index falls back to that string for documents it cannot flatten directly. This is
- // intentionally scoped to Pinot's built-in JSON index implementation.
- Object parsedValue = row.getParsedJsonValue(column);
- if (parsedValue != null && value instanceof String) {
- ((MutableJsonIndexImpl) mutableIndex).addParsed(parsedValue, (String) value);
- updateIndexCapacityThresholdBreached(mutableIndex, indexEntry.getKey(), column);
- continue;
- }
- }
- mutableIndex.add(value, dictId, docId);
- updateIndexCapacityThresholdBreached(mutableIndex, indexEntry.getKey(), column);
- } catch (Exception e) {
- recordIndexingError(indexEntry.getKey(), e);
- }
+ Object[] values = value instanceof Object[] ? (Object[]) value : new Object[]{value};
+ int[] dictIds = indexContainer._dictIds;
+ if (indexContainer._dictionary != null && dictIds == null) {
+ try {
+ Object[] defaultValues = (Object[]) getDefaultNullValueForIndexing(fieldSpec);
+ dictIds = indexContainer._dictionary.index(defaultValues);
+ indexContainer._dictIds = dictIds;
+ values = defaultValues;
+ isNull = true;
+ defaultSubstituted = true;
+ state.markIncomplete();
+ } catch (Exception e) {
+ recordOrDeferIndexingError("DICTIONARY", e);
+ state.noteError(e);
+ state.noteRepairFailure(e);
+ return false;
+ }
+ }
+ boolean hadError = writeMultiValueIndexes(docId, column, indexContainer, dataType, values, dictIds, isNull, state);
+ return !hadError && !defaultSubstituted;
+ }
+
+ private boolean addOpenStructColumn(int docId, String column, IndexContainer indexContainer, Object value,
+ boolean defaultSubstituted, RowWriteState state) {
+ MutableIndex openStructIndex = indexContainer._mutableIndexes.get(StandardIndexes.openStruct());
+ if (openStructIndex != null) {
+ try {
+ maybeIntercept(column, StandardIndexes.openStruct(), value, docId);
+ openStructIndex.add(value, -1, docId);
+ state._columnsWithCanonicalForward.add(column);
+ } catch (Exception e) {
+ recordOrDeferIndexingError(StandardIndexes.openStruct(), e);
+ state.noteError(e);
+ try {
+ Object defaultValue = getDefaultNullValueForIndexing(indexContainer._fieldSpec);
+ maybeIntercept(column, StandardIndexes.openStruct(), defaultValue, docId);
+ openStructIndex.add(defaultValue, -1, docId);
+ state._columnsWithCanonicalForward.add(column);
+ } catch (Exception fallbackError) {
+ state.noteRepairFailure(fallbackError);
+ return false;
}
+ defaultSubstituted = true;
+ }
+ } else {
+ state._columnsWithCanonicalForward.add(column);
+ }
+ indexContainer._valuesInfo.updateSVNumValues();
+ return !defaultSubstituted;
+ }
- if (dictId < 0) {
- // Update min/max value from raw value
- // NOTE: Skip updating min/max value for aggregated metrics because the value will change over time.
- if (!isAggregateMetricsEnabled() || fieldSpec.getFieldType() != FieldSpec.FieldType.METRIC) {
- Comparable comparable = toComparableValue(value, dataType, column);
- if (indexContainer._minValue == null) {
- indexContainer._minValue = comparable;
- indexContainer._maxValue = comparable;
- } else {
- if (comparable.compareTo(indexContainer._minValue) < 0) {
- indexContainer._minValue = comparable;
- }
- if (comparable.compareTo(indexContainer._maxValue) > 0) {
- indexContainer._maxValue = comparable;
- }
- }
+ /// Writes the canonical forward index first, then every secondary index from that same finalized value.
+ private boolean writeSingleValueIndexes(int docId, String column, IndexContainer indexContainer, FieldSpec fieldSpec,
+ DataType dataType, Object value, int dictId, boolean isNull, @Nullable Object parsedJsonValue,
+ RowWriteState state) {
+ boolean hadError = false;
+ MutableIndex forwardIndex = indexContainer._mutableIndexes.get(StandardIndexes.forward());
+ if (forwardIndex != null && !state._columnsWithCanonicalForward.contains(column)) {
+ try {
+ maybeIntercept(column, StandardIndexes.forward(), value, docId);
+ forwardIndex.add(value, dictId, docId);
+ updateIndexCapacityThresholdBreached(forwardIndex, StandardIndexes.forward(), column);
+ state._columnsWithCanonicalForward.add(column);
+ } catch (Exception e) {
+ recordOrDeferIndexingError(StandardIndexes.forward(), e);
+ state.noteError(e);
+ hadError = true;
+ try {
+ Object defaultValue = getDefaultNullValueForIndexing(fieldSpec);
+ int defaultDictId = dictId;
+ if (indexContainer._dictionary != null) {
+ defaultDictId = indexContainer._dictionary.index(defaultValue);
+ indexContainer._dictId = defaultDictId;
}
+ value = defaultValue;
+ dictId = defaultDictId;
Review Comment:
[P1] Keep the registered aggregation key consistent with repaired dimensions
getOrCreateDocId() has already registered the original dimension dictionary IDs before this forward-index fallback changes the dimension and _dictId to the default. With aggregation enabled, a forward failure for dim=42, metric=1 can publish dim=DEFAULT while _recordIdMap still maps the key for 42 to that document. A later successful dim=42, metric=2 then aggregates into the default-valued row, permanently assigning valid input to the wrong group. Please ensure a repaired dimension cannot disagree with the registered key, or treat key-column repair as unrecoverable. The existing dictionary-failure aggregation test fails before key creation and does not cover this forward-failure case.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImpl.java:
##########
@@ -1129,77 +1582,159 @@ private void aggregateMetrics(GenericRow row, int docId) {
}
MutableForwardIndex forwardIndex =
(MutableForwardIndex) indexContainer._mutableIndexes.get(StandardIndexes.forward());
- DataType dataType = metricFieldSpec.getDataType();
-
- switch (valueAggregator.getAggregatedValueType()) {
- case DOUBLE:
- double oldDoubleValue;
- double newDoubleValue;
- switch (dataType) {
- case INT:
- oldDoubleValue = forwardIndex.getInt(docId);
- newDoubleValue = (double) valueAggregator.applyRawValue(oldDoubleValue, value);
- forwardIndex.setInt(docId, (int) newDoubleValue);
- break;
- case LONG:
- oldDoubleValue = forwardIndex.getLong(docId);
- newDoubleValue = (double) valueAggregator.applyRawValue(oldDoubleValue, value);
- forwardIndex.setLong(docId, (long) newDoubleValue);
- break;
- case FLOAT:
- oldDoubleValue = forwardIndex.getFloat(docId);
- newDoubleValue = (double) valueAggregator.applyRawValue(oldDoubleValue, value);
- forwardIndex.setFloat(docId, (float) newDoubleValue);
- break;
- case DOUBLE:
- oldDoubleValue = forwardIndex.getDouble(docId);
- newDoubleValue = (double) valueAggregator.applyRawValue(oldDoubleValue, value);
- forwardIndex.setDouble(docId, newDoubleValue);
- break;
- default:
- throw new UnsupportedOperationException(String.format("Aggregation type %s of %s not supported for %s",
- valueAggregator.getAggregatedValueType(), valueAggregator.getAggregationType(), dataType));
- }
- break;
- case LONG:
- long oldLongValue;
- long newLongValue;
- switch (dataType) {
- case INT:
- oldLongValue = forwardIndex.getInt(docId);
- newLongValue = (long) valueAggregator.applyRawValue(oldLongValue, value);
- forwardIndex.setInt(docId, (int) newLongValue);
- break;
- case LONG:
- oldLongValue = forwardIndex.getLong(docId);
- newLongValue = (long) valueAggregator.applyRawValue(oldLongValue, value);
- forwardIndex.setLong(docId, newLongValue);
- break;
- case FLOAT:
- oldLongValue = (long) forwardIndex.getFloat(docId);
- newLongValue = (long) valueAggregator.applyRawValue(oldLongValue, value);
- forwardIndex.setFloat(docId, (float) newLongValue);
- break;
- case DOUBLE:
- oldLongValue = (long) forwardIndex.getDouble(docId);
- newLongValue = (long) valueAggregator.applyRawValue(oldLongValue, value);
- forwardIndex.setDouble(docId, (double) newLongValue);
- break;
- default:
- throw new UnsupportedOperationException(String.format("Aggregation type %s of %s not supported for %s",
- valueAggregator.getAggregatedValueType(), valueAggregator.getAggregationType(), dataType));
+ prepared.add(
+ prepareMetricWrite(column, valueAggregator, forwardIndex, metricFieldSpec.getDataType(), value, docId));
+ }
+
+ List<PreparedMetricWrite> committed = new ArrayList<>();
+ try {
+ for (PreparedMetricWrite write : prepared) {
+ maybeIntercept(write._column, StandardIndexes.forward(), write.interceptValue(), docId);
+ commitMetricWrite(write, docId);
+ committed.add(write);
+ }
+ } catch (Exception e) {
+ Exception restoreError = null;
+ for (int i = committed.size() - 1; i >= 0; i--) {
+ try {
+ maybeIntercept(committed.get(i)._column, StandardIndexes.forward(), committed.get(i).restoreInterceptValue(),
+ docId);
+ restoreMetricWrite(committed.get(i), docId);
+ } catch (Exception restoreException) {
+ if (restoreError == null) {
+ restoreError = restoreException;
+ } else {
+ restoreError.addSuppressed(restoreException);
}
- break;
- case BYTES:
- Object oldValue = valueAggregator.deserializeAggregatedValue(forwardIndex.getBytes(docId));
- Object newValue = valueAggregator.applyRawValue(oldValue, value);
- forwardIndex.setBytes(docId, valueAggregator.serializeAggregatedValue(newValue));
- break;
- default:
- throw new UnsupportedOperationException(
- String.format("Aggregation type %s of %s not supported for %s", valueAggregator.getAggregatedValueType(),
- valueAggregator.getAggregationType(), dataType));
+ }
+ }
+ if (restoreError != null) {
+ e.addSuppressed(restoreError);
+ _unrecoverableIndexingFailure = true;
+ }
+ throw wrapIndexingException("AGGREGATE_METRICS", e);
+ }
+ }
+
+ private PreparedMetricWrite prepareMetricWrite(String column, ValueAggregator valueAggregator,
+ MutableForwardIndex forwardIndex, DataType dataType, Object rawValue, int docId) {
+ switch (valueAggregator.getAggregatedValueType()) {
+ case DOUBLE: {
+ double oldDoubleValue;
+ double newDoubleValue;
+ MetricWriteKind kind;
+ switch (dataType) {
+ case INT:
+ oldDoubleValue = forwardIndex.getInt(docId);
+ newDoubleValue = (double) valueAggregator.applyRawValue(oldDoubleValue, rawValue);
+ kind = MetricWriteKind.INT;
+ break;
+ case LONG:
+ oldDoubleValue = forwardIndex.getLong(docId);
+ newDoubleValue = (double) valueAggregator.applyRawValue(oldDoubleValue, rawValue);
+ kind = MetricWriteKind.LONG;
+ break;
+ case FLOAT:
+ oldDoubleValue = forwardIndex.getFloat(docId);
+ newDoubleValue = (double) valueAggregator.applyRawValue(oldDoubleValue, rawValue);
+ kind = MetricWriteKind.FLOAT;
+ break;
+ case DOUBLE:
+ oldDoubleValue = forwardIndex.getDouble(docId);
+ newDoubleValue = (double) valueAggregator.applyRawValue(oldDoubleValue, rawValue);
+ kind = MetricWriteKind.DOUBLE;
+ break;
+ default:
+ throw new UnsupportedOperationException(String.format("Aggregation type %s of %s not supported for %s",
+ valueAggregator.getAggregatedValueType(), valueAggregator.getAggregationType(), dataType));
+ }
+ return PreparedMetricWrite.numeric(column, forwardIndex, kind, oldDoubleValue, newDoubleValue);
}
+ case LONG: {
+ long oldLongValue;
+ long newLongValue;
+ MetricWriteKind kind;
+ switch (dataType) {
+ case INT:
+ oldLongValue = forwardIndex.getInt(docId);
+ newLongValue = (long) valueAggregator.applyRawValue(oldLongValue, rawValue);
+ kind = MetricWriteKind.INT;
+ break;
+ case LONG:
+ oldLongValue = forwardIndex.getLong(docId);
+ newLongValue = (long) valueAggregator.applyRawValue(oldLongValue, rawValue);
+ kind = MetricWriteKind.LONG;
+ break;
+ case FLOAT:
+ oldLongValue = (long) forwardIndex.getFloat(docId);
+ newLongValue = (long) valueAggregator.applyRawValue(oldLongValue, rawValue);
+ kind = MetricWriteKind.FLOAT;
+ break;
+ case DOUBLE:
+ oldLongValue = (long) forwardIndex.getDouble(docId);
+ newLongValue = (long) valueAggregator.applyRawValue(oldLongValue, rawValue);
+ kind = MetricWriteKind.DOUBLE;
+ break;
+ default:
+ throw new UnsupportedOperationException(String.format("Aggregation type %s of %s not supported for %s",
+ valueAggregator.getAggregatedValueType(), valueAggregator.getAggregationType(), dataType));
+ }
+ return PreparedMetricWrite.numeric(column, forwardIndex, kind, oldLongValue, newLongValue);
+ }
+ case BYTES: {
+ Object oldValue = valueAggregator.deserializeAggregatedValue(forwardIndex.getBytes(docId));
+ Object newValue = valueAggregator.applyRawValue(oldValue, rawValue);
+ return PreparedMetricWrite.bytes(column, forwardIndex, valueAggregator.serializeAggregatedValue(oldValue),
+ valueAggregator.serializeAggregatedValue(newValue));
Review Comment:
[P1] Capture the original aggregate bytes before applying the update
applyRawValue() can mutate oldValue and return that same object: AvgValueAggregator calls value.apply(), and HLL aggregators mutate their sketch. Serializing oldValue afterward therefore saves the updated aggregate as the rollback value. If this metric commits and a later metric write fails, restoreMetricWrite() restores the new bytes, silently retaining only part of the failed row. Please capture the original bytes before applyRawValue(), or clone the old aggregate first. Extend rollback coverage with a mutable BYTES aggregate such as AVG followed by a failing metric; the current numeric LONG tests cannot expose this aliasing.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImpl.java:
##########
@@ -852,192 +898,527 @@ private void validateNumMultiValues(GenericRow row) {
}
}
- private void updateDictionary(GenericRow row) {
+ /// Runs dictionary + forward/secondary indexing for a new docId and meters an incomplete row when either step had
+ /// to fall back to defaults (issue #16316).
+ private void indexPhysicalRow(int docId, GenericRow row) {
+ RowWriteState state = _rowWriteState;
+ boolean dictHadError = updateDictionary(row, state);
+ boolean rowHadError = addNewRow(docId, row, state);
+ if (dictHadError || rowHadError) {
+ recordIncompleteRow();
+ }
+ if (state._repairFailed) {
+ markUnrecoverable(state);
+ throwPendingRowIndexingExceptionIfStrict();
+ throw new IllegalStateException(
+ "Mutable segment " + _segmentName + " is terminal after an unrecoverable indexing failure");
+ }
+ }
+
+ /// @return {@code true} if any column required a default/fallback while updating dictionaries.
+ /// When `continueOnError` is false, dictionary failures are stashed and rethrown after the row is published.
+ private boolean updateDictionary(GenericRow row, RowWriteState state) {
+ boolean hadError = false;
for (Map.Entry<String, IndexContainer> entry : _indexContainerMap.entrySet()) {
IndexContainer indexContainer = entry.getValue();
MutableDictionary dictionary = indexContainer._dictionary;
if (dictionary == null) {
continue;
}
-
- Object value = row.getValue(entry.getKey());
- if (value == null) {
- recordIndexingError("DICTIONARY");
- } else {
+ String column = entry.getKey();
+ Object value = row.getValue(column);
+ try {
+ if (value == null) {
+ recordIndexingError("DICTIONARY");
+ hadError = true;
+ state.markIncomplete();
+ value = getDefaultNullValueForIndexing(indexContainer._fieldSpec);
+ row.putDefaultNullValue(column, value);
+ }
if (indexContainer._fieldSpec.isSingleValueField()) {
indexContainer._dictId = dictionary.index(value);
} else {
indexContainer._dictIds = dictionary.index((Object[]) value);
}
-
- // Update min/max value from dictionary
indexContainer._minValue = dictionary.getMinVal();
indexContainer._maxValue = dictionary.getMaxVal();
+ } catch (Exception e) {
+ recordOrDeferIndexingError("DICTIONARY", e);
+ hadError = true;
+ state.noteError(e);
+ indexContainer._dictId = Integer.MIN_VALUE;
+ indexContainer._dictIds = null;
+ try {
+ // Index the field default instead of leaving the sentinel: with metrics aggregation the rollup key is built
+ // from the dict ids (see getOrCreateDocId), and Integer.MIN_VALUE is not a real dict id.
+ Object defaultValue = getDefaultNullValueForIndexing(indexContainer._fieldSpec);
+ if (indexContainer._fieldSpec.isSingleValueField()) {
+ indexContainer._dictId = dictionary.index(defaultValue);
+ } else {
+ indexContainer._dictIds = dictionary.index((Object[]) defaultValue);
+ }
+ row.putDefaultNullValue(column, defaultValue);
+ indexContainer._minValue = dictionary.getMinVal();
+ indexContainer._maxValue = dictionary.getMaxVal();
+ } catch (Exception fallbackError) {
+ _logger.error("Failed to index default null value into dictionary for column: {}", column, fallbackError);
+ state.noteRepairFailure(fallbackError);
+ }
}
- updateIndexCapacityThresholdBreached(dictionary, entry.getKey());
+ updateIndexCapacityThresholdBreached(dictionary, column);
}
+ return hadError;
}
- private void addNewRow(int docId, GenericRow row) {
+ /// Indexes a new physical row. Always completes the row so seal/query lengths stay aligned with [_numDocsIndexed]
+ /// (issue #16316). Forward indexes are written first from a finalized value; secondary indexes then consume that
+ /// same value. On unrecoverable repair the segment is marked terminal.
+ ///
+ /// @return {@code true} if any column required a default/fallback while indexing
+ private boolean addNewRow(int docId, GenericRow row, RowWriteState state) {
+ boolean rowHadError = false;
for (Map.Entry<String, IndexContainer> entry : _indexContainerMap.entrySet()) {
String column = entry.getKey();
IndexContainer indexContainer = entry.getValue();
-
- // Handle ingestion aggregation
- ValueAggregator valueAggregator = indexContainer._valueAggregator;
- if (valueAggregator != null) {
- String sourceColumn = indexContainer._sourceColumn;
- // NOTE: value can be null if the column is not specified in the schema.
- Object value = row.getValue(sourceColumn);
- // Handle COUNT(*)
- if (value == null && sourceColumn.equals(AggregationFunctionColumnPair.STAR)) {
- assert valueAggregator.getAggregationType() == AggregationFunctionType.COUNT;
- value = 1;
+ try {
+ if (indexContainer._valueAggregator != null) {
+ if (!addAggregatedColumn(docId, row, column, indexContainer, state)) {
+ rowHadError = true;
+ }
+ } else if (!addPhysicalColumn(docId, row, column, indexContainer, state)) {
+ rowHadError = true;
}
-
- // Update numValues info
- indexContainer._valuesInfo.updateSVNumValues();
-
- MutableIndex forwardIndex = indexContainer._mutableIndexes.get(StandardIndexes.forward());
- FieldSpec fieldSpec = indexContainer._fieldSpec;
-
- DataType dataType = fieldSpec.getDataType();
- value = valueAggregator.getInitialAggregatedValue(value);
- // BIG_DECIMAL is actually stored as byte[] and hence can be supported here.
- switch (dataType.getStoredType()) {
- case INT:
- forwardIndex.add(((Number) value).intValue(), -1, docId);
- break;
- case LONG:
- forwardIndex.add(((Number) value).longValue(), -1, docId);
- break;
- case FLOAT:
- forwardIndex.add(((Number) value).floatValue(), -1, docId);
- break;
- case DOUBLE:
- forwardIndex.add(((Number) value).doubleValue(), -1, docId);
- break;
- case BIG_DECIMAL:
- case BYTES:
- forwardIndex.add(valueAggregator.serializeAggregatedValue(value), -1, docId);
- break;
- default:
- throw new UnsupportedOperationException(
- "Unsupported data type: " + dataType + " for aggregation: " + column);
+ } catch (Exception e) {
+ recordOrDeferIndexingError("ROW", e);
+ rowHadError = true;
+ state.noteError(e);
+ try {
+ indexDefaultNullColumn(docId, column, indexContainer, state);
+ } catch (Exception fallbackError) {
+ _logger.error("Failed to index default null for column: {} at docId: {}", column, docId, fallbackError);
+ state.noteRepairFailure(fallbackError);
}
- continue;
}
+ }
- // Update the null value vector even if a null value is somehow produced
- if (indexContainer._nullValueVector != null && row.isNullValue(column)) {
- indexContainer._nullValueVector.setNull(docId);
+ if (_multiColumnValues != null) {
+ try {
+ // Multi-column text is a row-level secondary index. It is written only after every column has a finalized
+ // value in `_multiColumnValues`. There is no per-doc rollback; a failure leaves this doc absent from the
+ // text index while forward indexes remain complete. Counted once as a secondary-index error.
+ _multiColumnTextIndex.add(_multiColumnValues);
+ } catch (Exception e) {
+ recordOrDeferIndexingError("MULTI_COLUMN_TEXT", e);
+ rowHadError = true;
+ state.noteError(e);
+ } finally {
+ Collections.fill(_multiColumnValues, null);
}
+ }
+ if (state._repairFailed) {
+ markUnrecoverable(state);
+ }
+ return rowHadError;
+ }
- Object value = row.getValue(column);
- if (value == null) {
- // the value should not be null unless something is broken upstream but this will lead to inappropriate reuse
- // of the dictionary id if this somehow happens. An NPE here can corrupt indexes leading to incorrect query
- // results, hence the extra care. A metric will already have been emitted when trying to update the dictionary.
- continue;
+ /// Returns {@code true} when the aggregated column was written without error.
+ private boolean addAggregatedColumn(int docId, GenericRow row, String column, IndexContainer indexContainer,
+ RowWriteState state) {
+ ValueAggregator valueAggregator = indexContainer._valueAggregator;
+ String sourceColumn = indexContainer._sourceColumn;
+ Object value = row.getValue(sourceColumn);
+ if (value == null && sourceColumn.equals(AggregationFunctionColumnPair.STAR)) {
+ assert valueAggregator.getAggregationType() == AggregationFunctionType.COUNT;
+ value = 1;
+ }
+
+ MutableIndex forwardIndex = indexContainer._mutableIndexes.get(StandardIndexes.forward());
+ FieldSpec fieldSpec = indexContainer._fieldSpec;
+ DataType dataType = fieldSpec.getDataType();
+ try {
+ value = valueAggregator.getInitialAggregatedValue(value);
+ addAggregatedForwardValue(forwardIndex, valueAggregator, dataType, value, docId, column);
+ indexContainer._valuesInfo.updateSVNumValues();
+ state._columnsWithCanonicalForward.add(column);
+ return true;
+ } catch (Exception e) {
+ recordOrDeferIndexingError(StandardIndexes.forward(), e);
+ state.noteError(e);
+ try {
+ indexDefaultAggregatedValue(docId, column, indexContainer, state);
+ } catch (Exception fallbackError) {
+ state.noteRepairFailure(fallbackError);
}
+ return false;
+ }
+ }
- FieldSpec fieldSpec = indexContainer._fieldSpec;
- DataType dataType = fieldSpec.getDataType();
+ private void addAggregatedForwardValue(MutableIndex forwardIndex, ValueAggregator valueAggregator, DataType dataType,
+ Object value, int docId, String column) {
+ maybeIntercept(column, StandardIndexes.forward(), value, docId);
+ switch (dataType.getStoredType()) {
+ case INT:
+ forwardIndex.add(((Number) value).intValue(), -1, docId);
+ break;
+ case LONG:
+ forwardIndex.add(((Number) value).longValue(), -1, docId);
+ break;
+ case FLOAT:
+ forwardIndex.add(((Number) value).floatValue(), -1, docId);
+ break;
+ case DOUBLE:
+ forwardIndex.add(((Number) value).doubleValue(), -1, docId);
+ break;
+ case BIG_DECIMAL:
+ case BYTES:
+ forwardIndex.add(valueAggregator.serializeAggregatedValue(value), -1, docId);
+ break;
+ default:
+ throw new UnsupportedOperationException("Unsupported data type: " + dataType + " for aggregation: " + column);
+ }
+ }
- if (fieldSpec.isSingleValueField()) {
- // Update numValues info
- indexContainer._valuesInfo.updateSVNumValues();
-
- // Route OPEN_STRUCT values to the dedicated mutable index. OPEN_STRUCT has no forward
- // index / dictionary / min-max, so the standard per-IndexType loop and the comparable
- // tracking below would be no-ops at best and crash at worst (Map is not Comparable).
- if (dataType == DataType.OPEN_STRUCT) {
- MutableIndex openStructIndex = indexContainer._mutableIndexes.get(StandardIndexes.openStruct());
- if (openStructIndex != null) {
- openStructIndex.add(value, -1, docId);
- }
- continue;
+ /// Returns {@code true} when the physical column was written from the row value without error and without falling
+ /// back to the field default.
+ private boolean addPhysicalColumn(int docId, GenericRow row, String column, IndexContainer indexContainer,
+ RowWriteState state) {
+ FieldSpec fieldSpec = indexContainer._fieldSpec;
+ DataType dataType = fieldSpec.getDataType();
+ boolean isNull = row.isNullValue(column);
+ Object value = row.getValue(column);
+ boolean defaultSubstituted = false;
+ if (value == null) {
+ recordIndexingError("NULL_VALUE");
+ value = getDefaultNullValueForIndexing(fieldSpec);
+ isNull = true;
+ defaultSubstituted = true;
+ state.markIncomplete();
+ }
+
+ if (fieldSpec.isSingleValueField()) {
+ if (dataType == DataType.OPEN_STRUCT) {
+ return addOpenStructColumn(docId, column, indexContainer, value, defaultSubstituted, state);
+ }
+ int dictId = indexContainer._dictId;
+ if (indexContainer._dictionary != null && dictId == Integer.MIN_VALUE) {
+ try {
+ Object defaultValue = getDefaultNullValueForIndexing(fieldSpec);
+ dictId = indexContainer._dictionary.index(defaultValue);
+ indexContainer._dictId = dictId;
+ value = defaultValue;
+ isNull = true;
+ defaultSubstituted = true;
+ state.markIncomplete();
+ } catch (Exception e) {
+ recordOrDeferIndexingError("DICTIONARY", e);
+ state.noteError(e);
+ state.noteRepairFailure(e);
+ return false;
}
+ }
+ Object parsedJsonValue = defaultSubstituted ? null : row.getParsedJsonValue(column);
+ boolean hadError = writeSingleValueIndexes(docId, column, indexContainer, fieldSpec, dataType, value, dictId,
+ isNull, parsedJsonValue, state);
+ return !hadError && !defaultSubstituted;
+ }
- // Update indexes
- int dictId = indexContainer._dictId;
- for (Map.Entry<IndexType, MutableIndex> indexEntry : indexContainer._mutableIndexes.entrySet()) {
- try {
- MutableIndex mutableIndex = indexEntry.getValue();
- if (mutableIndex instanceof MutableJsonIndexImpl) {
- // Feed the JSON index the parsed value cached by the DataTypeTransformer (if any), so it flattens the
- // document directly instead of re-parsing the string this column was serialized into for the forward
- // index; the index falls back to that string for documents it cannot flatten directly. This is
- // intentionally scoped to Pinot's built-in JSON index implementation.
- Object parsedValue = row.getParsedJsonValue(column);
- if (parsedValue != null && value instanceof String) {
- ((MutableJsonIndexImpl) mutableIndex).addParsed(parsedValue, (String) value);
- updateIndexCapacityThresholdBreached(mutableIndex, indexEntry.getKey(), column);
- continue;
- }
- }
- mutableIndex.add(value, dictId, docId);
- updateIndexCapacityThresholdBreached(mutableIndex, indexEntry.getKey(), column);
- } catch (Exception e) {
- recordIndexingError(indexEntry.getKey(), e);
- }
+ Object[] values = value instanceof Object[] ? (Object[]) value : new Object[]{value};
+ int[] dictIds = indexContainer._dictIds;
+ if (indexContainer._dictionary != null && dictIds == null) {
+ try {
+ Object[] defaultValues = (Object[]) getDefaultNullValueForIndexing(fieldSpec);
+ dictIds = indexContainer._dictionary.index(defaultValues);
+ indexContainer._dictIds = dictIds;
+ values = defaultValues;
+ isNull = true;
+ defaultSubstituted = true;
+ state.markIncomplete();
+ } catch (Exception e) {
+ recordOrDeferIndexingError("DICTIONARY", e);
+ state.noteError(e);
+ state.noteRepairFailure(e);
+ return false;
+ }
+ }
+ boolean hadError = writeMultiValueIndexes(docId, column, indexContainer, dataType, values, dictIds, isNull, state);
+ return !hadError && !defaultSubstituted;
+ }
+
+ private boolean addOpenStructColumn(int docId, String column, IndexContainer indexContainer, Object value,
+ boolean defaultSubstituted, RowWriteState state) {
+ MutableIndex openStructIndex = indexContainer._mutableIndexes.get(StandardIndexes.openStruct());
+ if (openStructIndex != null) {
+ try {
+ maybeIntercept(column, StandardIndexes.openStruct(), value, docId);
+ openStructIndex.add(value, -1, docId);
+ state._columnsWithCanonicalForward.add(column);
+ } catch (Exception e) {
+ recordOrDeferIndexingError(StandardIndexes.openStruct(), e);
+ state.noteError(e);
+ try {
+ Object defaultValue = getDefaultNullValueForIndexing(indexContainer._fieldSpec);
+ maybeIntercept(column, StandardIndexes.openStruct(), defaultValue, docId);
+ openStructIndex.add(defaultValue, -1, docId);
+ state._columnsWithCanonicalForward.add(column);
+ } catch (Exception fallbackError) {
+ state.noteRepairFailure(fallbackError);
+ return false;
}
+ defaultSubstituted = true;
+ }
+ } else {
+ state._columnsWithCanonicalForward.add(column);
+ }
+ indexContainer._valuesInfo.updateSVNumValues();
+ return !defaultSubstituted;
+ }
- if (dictId < 0) {
- // Update min/max value from raw value
- // NOTE: Skip updating min/max value for aggregated metrics because the value will change over time.
- if (!isAggregateMetricsEnabled() || fieldSpec.getFieldType() != FieldSpec.FieldType.METRIC) {
- Comparable comparable = toComparableValue(value, dataType, column);
- if (indexContainer._minValue == null) {
- indexContainer._minValue = comparable;
- indexContainer._maxValue = comparable;
- } else {
- if (comparable.compareTo(indexContainer._minValue) < 0) {
- indexContainer._minValue = comparable;
- }
- if (comparable.compareTo(indexContainer._maxValue) > 0) {
- indexContainer._maxValue = comparable;
- }
- }
+ /// Writes the canonical forward index first, then every secondary index from that same finalized value.
+ private boolean writeSingleValueIndexes(int docId, String column, IndexContainer indexContainer, FieldSpec fieldSpec,
+ DataType dataType, Object value, int dictId, boolean isNull, @Nullable Object parsedJsonValue,
+ RowWriteState state) {
+ boolean hadError = false;
+ MutableIndex forwardIndex = indexContainer._mutableIndexes.get(StandardIndexes.forward());
+ if (forwardIndex != null && !state._columnsWithCanonicalForward.contains(column)) {
+ try {
+ maybeIntercept(column, StandardIndexes.forward(), value, docId);
+ forwardIndex.add(value, dictId, docId);
+ updateIndexCapacityThresholdBreached(forwardIndex, StandardIndexes.forward(), column);
+ state._columnsWithCanonicalForward.add(column);
+ } catch (Exception e) {
+ recordOrDeferIndexingError(StandardIndexes.forward(), e);
+ state.noteError(e);
+ hadError = true;
+ try {
+ Object defaultValue = getDefaultNullValueForIndexing(fieldSpec);
+ int defaultDictId = dictId;
+ if (indexContainer._dictionary != null) {
+ defaultDictId = indexContainer._dictionary.index(defaultValue);
+ indexContainer._dictId = defaultDictId;
Review Comment:
[P1] Refresh cached dictionary bounds after inserting the fallback value
_minValue and _maxValue were copied from the dictionary in updateDictionary(), before this insertion. If the default is a new extreme, those cached bounds remain stale. For example, with null handling disabled, repairing the only dictionary INT row from 42 to Integer.MIN_VALUE leaves datasource bounds [42,42]. ColumnValueSegmentPruner then incorrectly prunes the consuming segment for WHERE col < 0 even though its repaired row matches. Please refresh the cached bounds in fallback dictionary-insertion paths, and cover the datasource bounds/range-pruning behavior with null handling disabled.
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java:
##########
@@ -735,12 +736,28 @@ private boolean processStreamEvents(MessageBatch messageBatch, long idlePipeSlee
_serverMetrics.addMeteredGlobalValue(ServerMeter.REALTIME_BYTES_CONSUMED, recordSerializedValueLength);
}
} catch (Exception e) {
- _numRowsErrored++;
- _numBytesDropped += rowSizeInBytes;
String errorMessage =
"Caught exception while indexing the record at offset: " + offset + " , row: " + transformedRow;
_segmentLogger.error(errorMessage, e);
_realtimeTableDataManager.addSegmentError(_segmentNameStr, new SegmentErrorInfo(now(), errorMessage, e));
+ // continueOnError=false finishes a started row, then rethrows. A published repair is consumed;
+ // only an unpublished failure is a drop. An unrecoverable repair is terminal.
+ if (_realtimeSegment.getNumDocsIndexed() > docsBeforeIndex) {
+ indexedMessageCount++;
+ _lastRowMetadata = metadata;
+ _lastConsumedTimestampMs = System.currentTimeMillis();
+ realtimeRowsConsumedMeter =
+ _serverMetrics.addMeteredTableValue(_clientId, ServerMeter.REALTIME_ROWS_CONSUMED, 1,
+ realtimeRowsConsumedMeter);
+ _serverMetrics.addMeteredGlobalValue(ServerMeter.REALTIME_ROWS_CONSUMED, 1L);
+ } else {
+ _numRowsErrored++;
+ _numBytesDropped += rowSizeInBytes;
+ }
+ canTakeMore = _realtimeSegment.canTakeMoreRows();
+ if (!canTakeMore) {
+ throw new RuntimeException(errorMessage, e);
Review Comment:
[P1] Distinguish normal capacity completion from an unrecoverable failure
With the default continueOnError=false, a recoverable indexing error on the row that fills the segment publishes the repaired row and then rethrows. canTakeMoreRows() is false simply because _numDocsIndexed == _capacity, so this branch throws before _currentOffset and _numRowsIndexed advance. PartitionConsumer then transitions to State.ERROR instead of completing the segment normally. A malformed JSON secondary-index value on the capacity-filling row is one reachable example. Please check unrecoverable state separately and let successfully published rows advance their offsets and complete normally. StatelessRealtimeSegmentWriter.indexTransformedRow has the same conflation. A real-segment regression with capacity one and a recoverable JSON error would cover the distinction missing from the mocked tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org