[GitHub] [pinot] Jackie-Jiang commented on issue #8962: Segments go into error state after adding sortedColumn on already existing realtime table

[GitBox <gi...@apache.org>]

Jackie-Jiang commented on issue #8962:
URL: https://github.com/apache/pinot/issues/8962#issuecomment-1164927712

   In order to sort records within a consuming segment, inverted index is required. Currently there is no easy way to add an inverted index into a consuming segment, so there are 2 options:
   1. Re-design segment reload for consuming segment. Currently segment reload can only add new column with default value, which is not robust and can cause unexpected behavior when there are other changes in the table config. In order to apply the complete table config into the current consuming segment without waiting for it to commit, we need to re-consume the segment. To avoid inconsistent query result, we can consider pausing the current consuming segment and start a new one, wait for it to catch up and then replace the old one.
   2. Make sorting possible without inverted index.
   
   IMO option 1 is more desired because option 2 can cause extra overhead when committing the segment, and won't fix the other issues caused by reloading a consuming segment.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org