[GitHub] [pinot] itschrispeck commented on issue #11004: Follower server redownloads segments upon server restart due to CRC change

["itschrispeck (via GitHub)" <gi...@apache.org>]

itschrispeck commented on issue #11004:
URL: https://github.com/apache/pinot/issues/11004#issuecomment-1638567703

   You're right, I think it's coincidence that the clusters we saw issue with are running a newer version of Pinot. We've jumped ahead to a commit that includes both fixes and we're still seeing the behavior. 
   
   It looks like the lucene index may not be created deterministically also:
   ```
   Binary files replica_1/json_data.lucene.index/_0.cfe and replica_2/json_data.lucene.index/_0.cfe differ
   72 | Binary files replica_1/json_data.lucene.index/_0.cfs and replica_2/json_data.lucene.index/_0.cfs differ
   73 | Binary files replica_1/json_data.lucene.index/_0.si and replica_2/json_data.lucene.index/_0.si differ
   74 | Binary files replica_1/json_data.lucene.index/segments_1 and replica_2/json_data.lucene.index/segments_1 differ
   75 | Binary files replica_1/message_dictionaryVars.lucene.index/_0.cfe and replica_2/message_dictionaryVars.lucene.index/_0.cfe differ
   76 | Binary files replica_1/message_dictionaryVars.lucene.index/_0.cfs and replica_2/message_dictionaryVars.lucene.index/_0.cfs differ
   77 | Binary files replica_1/message_dictionaryVars.lucene.index/_0.si and replica_2/message_dictionaryVars.lucene.index/_0.si differ
   78 | Binary files replica_1/message_dictionaryVars.lucene.index/segments_1 and replica_2/message_dictionaryVars.lucene.index/segments_1 differ
   79 | Binary files replica_1/message_logtype.lucene.index/_0.cfe and replica_2/message_logtype.lucene.index/_0.cfe differ
   80 | Binary files replica_1/message_logtype.lucene.index/_0.cfs and replica_2/message_logtype.lucene.index/_0.cfs differ
   81 | Binary files replica_1/message_logtype.lucene.index/_0.si and replica_2/message_logtype.lucene.index/_0.si differ
   82 | Binary files replica_1/message_logtype.lucene.index/segments_1 and replica_2/message_logtype.lucene.index/segments_1 differ
   83
   ```
   
   And I think this issue would also present if we use certain transform functions (e.g. storing ingestion time via `now()`). 
   
   What necessitates computing the CRC for all segments each restart? Assuming no data corruption happened, it seems that any data difference was already being 'served' as valid data. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org