[GitHub] [pinot] mcvsubbu commented on pull request #10869: Bug Fix: Segment Purger cannot purge old segments after schema evolution

["mcvsubbu (via GitHub)" <gi...@apache.org>]

mcvsubbu commented on PR #10869:
URL: https://github.com/apache/pinot/pull/10869#issuecomment-1581532821

   > This doesn't seem to be the correct fix.
   > 
   > The real problem is that we are using the table config from the ZK and schema from the segment file (we picked this way before because schema might not be stored in ZK when this task was implemented). Instead, we should also read the schema from the ZK so that schema is in-sync with the table config. You may read the schema in `PurgeTaskExecutor` and pass it into the `SegmentPurger`, then use it to create the `SegmentGeneratorConfig`.
   
   Minion purge task can purge data from old segments, which may have been built with an older schema (minus some columns). I thought the idea was to keep the old segments without the new column (and let the server auto-load default value into new column instead of baking it into the segment).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org