[I] Support API for checking if segments need to be reloaded for a table [pinot]

["dang-stripe (via GitHub)" <gi...@apache.org>]

dang-stripe opened a new issue, #12117:
URL: https://github.com/apache/pinot/issues/12117

   We're interested in having an API to know whether or not segments need to be reloaded for a table. The use case is to build robust automation around segment reloads after table/schema changes without having to depend directly on the table/schema change triggering the reload.
   
   Currently, reload segments fans out ZK messages to all servers hosting segments for the table and the servers process them 1 at a time w/ the default config. While there's an API to check the column/index information for a segment, it's expensive to do a scatter-gather for all segments and compare that data against the current table/schema config.
   
   One idea is to have servers track the znode version of both the table and schema configs that a segment was last reloaded with in memory and expose an API to the controller to fetch this version and compare it with the latest znode version to return to the caller whether there are segments that need to be reloaded.
   
   cc @Jackie-Jiang @jadami10 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

Re: [I] Support API for checking if segments need to be reloaded for a table [pinot]

["nitin-startree (via GitHub)" <gi...@apache.org>]

nitin-startree commented on issue #12117:
URL: https://github.com/apache/pinot/issues/12117#issuecomment-2058802380

   @Jackie-Jiang - Could you provide an estimated timeline or any updates regarding this Feature request? I'm currently reviewing all tickets labelled as "Waiting on Feature" to ascertain their current status.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org