[GitHub] [pinot] siddharthteotia edited a comment on pull request #7638: Allow MV Field Support For Raw Columns in Text Indices

[GitBox <gi...@apache.org>]

siddharthteotia edited a comment on pull request #7638:
URL: https://github.com/apache/pinot/pull/7638#issuecomment-954477445


   - For offline, in `LuceneTextIndexReader`, we build a mapping file (luceneDocId -> pinotDocId) during segment load to avoid expensive retrieval of entire Lucene document. That file is built by iterating over numDocs in the Pinot segment which is equal to numDocs in lucene index fo SV column. For MV column, this is not true since for each Pinot doc, we are adding docs equal to length of array. So, during query processing when `LuceneDocIdCollector` looks up the mapping file in `DocIdTranslator`, it can seg fault as Lucene will return a docId > max pinot docId
   
   - For realtime, can you also add this new interface support in `RealtimeLuceneTextIndexReader` that acts as both reader and writer and uses Lucene NRT search ? Separate PR is also fine


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org