You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "wirybeaver (via GitHub)" <gi...@apache.org> on 2026/03/22 03:27:04 UTC
[I] [TableCopy] Controller crash recovery for in-progress table replication jobs [pinot]
wirybeaver opened a new issue, #17933:
URL: https://github.com/apache/pinot/issues/17933
## Background
During a table copy operation (PR #17521), the replication job runs on the controller that received the API request. If that controller crashes or restarts mid-job, the in-progress job is lost and does not automatically resume.
## Current behavior
Users need to delete the target table and restart the copy from the beginning.
## Proposed improvements
**Short-term (manual resume API):**
- Provide a `/resume` API that cancels the job on the old controller, computes the remaining segments to copy (source segments − segments already above the watermark − segments already in target), and re-triggers the copy.
**Long-term (automatic recovery by lead controller):**
- The cluster's lead controller runs a background scan thread:
1. Scans all table replication controller jobs.
2. If the controller that owns a job UUID is dead, select a new controller and invoke the resume endpoint.
3. If the controller is alive but no longer has the job in its local executor, send a Helix message to the assigned controller; if the controller notices it owns the job UUID but has no running task, recreate the copy tasks.
## Related
- PR: https://github.com/apache/pinot/pull/17521
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] [TableCopy] Controller crash recovery for in-progress table replication jobs [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #17933:
URL: https://github.com/apache/pinot/issues/17933#issuecomment-5403182962
This issue has been automatically marked as stale because it has not had any activity in the last 150 days. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] [TableCopy] Controller crash recovery for in-progress table replication jobs [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #17933: [TableCopy] Controller crash recovery for in-progress table replication jobs
URL: https://github.com/apache/pinot/issues/17933
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] [TableCopy] Controller crash recovery for in-progress table replication jobs [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #17933:
URL: https://github.com/apache/pinot/issues/17933#issuecomment-5805182063
This issue has been closed because it has been inactive for 180 days. If you believe this issue is still relevant, please feel free to reopen it or create a new issue. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org