[I] MutableJsonIndexImpl: unpaired UTF-16 surrogate in JSON value corrupts doc ID mapping [pinot]

["spapin (via GitHub)" <gi...@apache.org>]

spapin opened a new issue, #18737:
URL: https://github.com/apache/pinot/issues/18737

   ## Bug description
   
   A JSON value containing an unpaired UTF-16 surrogate corrupts the JSON index of a consuming (mutable) segment.
   
   > **What's a surrogate?** Java represents characters above U+FFFF (emoji, some CJK, etc.) as a *pair* of `char` values called surrogates. An *unpaired* surrogate is one half of that pair with its partner missing — it's not valid text and not encodable as UTF-8. It commonly appears when a string is truncated in the middle of such a character.
   
   ## Why it happens
   
   For each document, `MutableJsonIndexImpl.addFlattenedRecords` does two steps:
   
   1. **Step 1** — appends one entry per flattened record to `_docIdMapping`.
   2. **Step 2** — loops over those records to build the posting lists, incrementing `_nextFlattenedDocId` as it goes.
   
   Step 2 calls `Utf8.encodedLength(...)` only to track an approximate memory size — and that call **throws** on an unpaired surrogate. The throw aborts step 2 partway, *after* step 1 has already grown `_docIdMapping`. Result: `_docIdMapping` is left permanently one entry longer than `_nextFlattenedDocId`.
   
   ## Effect
   
   With the two counters out of sync, every later document's posting-list entries map back to the *wrong* real doc id, so `json_match()` on that segment returns rows shifted by one. `MutableSegmentImpl` swallows the exception, so it's **completely silent** — no query error, no log warning — and only heals once the segment is committed and the immutable index is rebuilt from scratch.
   
   ## Reproduction
   
   Insert these three records in order into a JSON-indexed column on a consuming (realtime) segment:
   
   ```
   {"name": "first"}
   {"name": "\uD800"}   // unpaired high surrogate
   {"name": "third"}
   ```
   
   Expected results:
   - `json_match(col, '"$.name = ''first''")` → doc 0
   - `json_match(col, '"$.name = ''\uD800''")` → doc 1
   - `json_match(col, '"$.name = ''third''")` → doc 2
   
   Actual results after the corrupted document:
   - `json_match(col, '"$.name = ''third''")` → doc 3 (off by one)
   
   ## Suggested fix
   
   `_bytesSize` is only a memory-accounting heuristic, so a byte-length calculation should never be able to break indexing. The fix wraps the two `Utf8.encodedLength` calls in a helper that catches `IllegalArgumentException` and falls back to `String.getBytes(UTF_8)` — which never throws — when the string is malformed. This keeps the allocation-free fast path for well-formed strings and only pays for the allocation on the rare malformed input where the exact count doesn't matter anyway.
   
   ## Environment
   
   - Component: `pinot-segment-local` / `MutableJsonIndexImpl`
   - Index type: JSON index (mutable/realtime segment)
   - Trigger: any JSON column value containing an unpaired UTF-16 surrogate character


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

Re: [I] MutableJsonIndexImpl: unpaired UTF-16 surrogate in JSON value corrupts doc ID mapping [pinot]

["xiangfu0 (via GitHub)" <gi...@apache.org>]

xiangfu0 closed issue #18737: MutableJsonIndexImpl: unpaired UTF-16 surrogate in JSON value corrupts doc ID mapping
URL: https://github.com/apache/pinot/issues/18737


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org