You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "dang-stripe (via GitHub)" <gi...@apache.org> on 2023/12/07 23:23:08 UTC
[I] Support API for checking if segments need to be reloaded for a table [pinot]
dang-stripe opened a new issue, #12117:
URL: https://github.com/apache/pinot/issues/12117
We're interested in having an API to know whether or not segments need to be reloaded for a table. The use case is to build robust automation around segment reloads after table/schema changes without having to depend directly on the table/schema change triggering the reload.
Currently, reload segments fans out ZK messages to all servers hosting segments for the table and the servers process them 1 at a time w/ the default config. While there's an API to check the column/index information for a segment, it's expensive to do a scatter-gather for all segments and compare that data against the current table/schema config.
One idea is to have servers track the znode version of both the table and schema configs that a segment was last reloaded with in memory and expose an API to the controller to fetch this version and compare it with the latest znode version to return to the caller whether there are segments that need to be reloaded.
cc @Jackie-Jiang @jadami10
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Support API for checking if segments need to be reloaded for a table [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang closed issue #12117: Support API for checking if segments need to be reloaded for a table
URL: https://github.com/apache/pinot/issues/12117
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Support API for checking if segments need to be reloaded for a table [pinot]
Posted by "nitin-startree (via GitHub)" <gi...@apache.org>.
nitin-startree commented on issue #12117:
URL: https://github.com/apache/pinot/issues/12117#issuecomment-2058802380
@Jackie-Jiang - Could you provide an estimated timeline or any updates regarding this Feature request? I'm currently reviewing all tickets labelled as "Waiting on Feature" to ascertain their current status.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Support API for checking if segments need to be reloaded for a table [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #12117:
URL: https://github.com/apache/pinot/issues/12117#issuecomment-2237710986
A more desired solution would be:
- Controller asks all servers serving a table if the local loaded segment aligns with the current table config/schema
- Each server pulls the latest table config and schema and compare them with the local loaded segments, then return info about whether a reload is required
Pros:
- Some table config/schema update doesn't require reloading the segments (e.g. ingestion related settings)
Cons:
- Each server needs to pull table config/schema during the check
- The check itself is more expensive than caching the table config/schema ZNode version
We can also consider combining the 2 approaches: when ZNode version doesn't match, we perform the check. When server detect that there is no need to reload the index, it updates the ZNode version within the segment without reloading it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org