[GitHub] [pinot] Jackie-Jiang commented on issue #10951: After segment purge, the segment start and end time in segment zk metadata may not reflect the correct values

["Jackie-Jiang (via GitHub)" <gi...@apache.org>]

Jackie-Jiang commented on issue #10951:
URL: https://github.com/apache/pinot/issues/10951#issuecomment-1601955203

   Checking the code in `SegmentPurger` and find the following note:
   
   ```
         // The time column type info is not stored in the segment metadata.
         // Keep segment start/end time to properly handle time column type other than EPOCH (e.g.SIMPLE_FORMAT).
         if (segmentMetadata.getTimeInterval() != null) {
           config.setTimeColumnName(_tableConfig.getValidationConfig().getTimeColumnName());
           config.setStartTime(Long.toString(segmentMetadata.getStartTime()));
           config.setEndTime(Long.toString(segmentMetadata.getEndTime()));
           config.setSegmentTimeUnit(segmentMetadata.getTimeUnit());
         }
   ```
   
   I think at the time when we added the purge task (in 2018), schema might not be available in the cluster, and we don't have the time type info (more context in #2846). Now with #10869 we always use the schema from ZK to generate the new segment, so we can safely remove these special handling to reflect the actual time range


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org