You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "KKcorps (via GitHub)" <gi...@apache.org> on 2023/04/13 15:32:24 UTC
[GitHub] [pinot] KKcorps opened a new issue, #10603: Make Table Deletion Idempotent
KKcorps opened a new issue, #10603:
URL: https://github.com/apache/pinot/issues/10603
The table deletion flow was changed in #8422 where we send a message to servers and let them take care of waiting for EV convergence and then remove the segments
On the controller side, we only takes care of deleting the configs and ideal states.
However, this leads to a lot of exceptions.
For eg. when servers try to close consuming segments it starts getting `TableNotFound` exception
The new deletion flow already has the following TODO which will help solve this issue.
```java
public void deleteOfflineTable(String tableName, @Nullable String retentionPeriod) {
String offlineTableName = TableNameBuilder.OFFLINE.tableNameWithType(tableName);
LOGGER.info("Deleting table {}: Start", offlineTableName);
// Remove the table from brokerResource
HelixHelper.removeResourceFromBrokerIdealState(_helixZkManager, offlineTableName);
LOGGER.info("Deleting table {}: Removed from broker resource", offlineTableName);
// Drop the table on servers
// TODO: Make this api idempotent and blocking by waiting for externalview to converge on controllers
// instead of servers. This is because if externalview gets updated with significant delay,
// we may have the race condition for table recreation that the new table will use the old states
// (old table data manager) on the servers.
// Steps needed:
// 1. Drop the helix resource first (set idealstate as null)
// 2. Wait for the externalview to converge
// 3. Get servers for the tenant, and send delete table message to these servers
deleteTableOnServer(offlineTableName);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] KKcorps commented on issue #10603: Make Table Deletion Idempotent
Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps commented on issue #10603:
URL: https://github.com/apache/pinot/issues/10603#issuecomment-1507359268
@jtao15 I am planning to go with background job option because Ev convergence can take long especially when upserts are enabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] jtao15 commented on issue #10603: Make Table Deletion Idempotent
Posted by "jtao15 (via GitHub)" <gi...@apache.org>.
jtao15 commented on issue #10603:
URL: https://github.com/apache/pinot/issues/10603#issuecomment-1507300674
I think we can either:
1. Make it blocking, but client side needs to wait if externalview is slow to converge.
2. As @KKcorps mentioned, create a background job and return the job id to the client. Maybe we need to add a new table deletion api since it's not backward compatible and deprecate the existing one? And add another api to check the job status.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on issue #10603: Make Table Deletion Idempotent
Posted by "mcvsubbu (via GitHub)" <gi...@apache.org>.
mcvsubbu commented on issue #10603:
URL: https://github.com/apache/pinot/issues/10603#issuecomment-1507223775
cc:@jtao15
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] KKcorps commented on issue #10603: Make Table Deletion Idempotent
Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps commented on issue #10603:
URL: https://github.com/apache/pinot/issues/10603#issuecomment-1507196334
@Jackie-Jiang Should we just create a background thread to take care of waiting in the controller because we do need to return some response to the user that deletion is being taken care of and not keep on waiting for X seconds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #10603: Make Table Deletion Idempotent
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #10603:
URL: https://github.com/apache/pinot/issues/10603#issuecomment-1507517125
Background job sounds good to me. We can build it similar to reload job.
There is another issue we want to address: we delete segments from deep store and ZK metadata separately. If controller crashes after ZK metadata is deleted, but before segments are deleted, the segments will left orphan, and rerun the table deletion won't be able to find the segments. The fix should be always first delete the segment from deep store, then delete the ZK metadata for each segment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org