[GitHub] [pinot] somandal commented on pull request #9333: Add an option to disable the creation of the forward index for a column

[GitBox <gi...@apache.org>]

somandal commented on PR #9333:
URL: https://github.com/apache/pinot/pull/9333#issuecomment-1241174482

   > We should allow disabling forward index for sorted column, which is a no-op because the inverted index for sorted column can be used as forward index. I believe it is already allowed in the implementation because the default column is always sorted. No need to add this in the validation.
   
   Thanks for the suggestion @Jackie-Jiang. From going through the code it looks like for sorted columns the forward index and inverted index are essentially the same. Code snippet from the `PhysicalColumnIndexCreator`:
   
   ```
           // Single-value
           if (metadata.isSorted()) {
             // Sorted
             // forwardIndexDisabled columns do not need to be handled here as forward index cannot be disabled on a
             // sorted column
             SortedIndexReader<?> sortedIndexReader = indexReaderProvider.newSortedIndexReader(fwdIndexBuffer, metadata);
             _forwardIndex = sortedIndexReader;
             _invertedIndex = sortedIndexReader;
             _fstIndex = null;
             return;
           }
   ```
   
   I also see code on the segment creation path where we skip creating the inverted index if the column is sorted even if the inverted index is enabled for the column. So keeping these in mind, we didn't think it made much sense to disable forward index for such columns.
   
   Let me know your thoughts based on the above. cc @siddharthteotia 
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org