Re: [PR] Set skipCrcCheckOnLoad in table config [pinot]

["cypherean (via GitHub)" <gi...@apache.org>]

cypherean commented on PR #17249:
URL: https://github.com/apache/pinot/pull/17249#issuecomment-3740253274

   > Since the segment management is done via a state machine, the only way to get consistent behavior is to always ignore the crc check when loading segment, which also means segment refresh (in-place replacement) won't be supported. Do you see this as an okay trade-off? If so, let's clearly document the behavior
   
   @Jackie-Jiang this gap is already exposed as an instance level config. Going forward 
   1. Do we want to ensure this is only exposed on add, reload paths and we're able to detect replace even when server is down and treat it as such? 
   2. Or do we allow to skip crc check for all 3 as is the current behavior and document the gaps for the user? 
   3. Or try add/replacement on a best effort basis and serve from local copy on failure (same as reload), but actively alert and repair? 
   
   My take - We need a combination of 1 and 3, not just as a short term fix but as a fallback for reliability.  Suggested fixes:
   1. Detect replace even when server is down and treat it as replace not add
   2. Try add/replacement on a best effort basis and serve from local copy on failure - this will be behind the flag that let's us skip crc check right now.
   
   Pushing the changes for (2), I'll look into 1 in a follow up PR 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org