You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "suddendust (via GitHub)" <gi...@apache.org> on 2023/02/20 19:58:20 UTC
[GitHub] [pinot] suddendust opened a new issue, #10313: API for Rebalance Status
suddendust opened a new issue, #10313:
URL: https://github.com/apache/pinot/issues/10313
Currently, there's no way to check if the rebalance is complete without checking server logs. This is arduous, especially for tables that have a large no of segments. It'd be helpful to add an API that shows the rebalance status, just like the reload API. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #10313: API for Rebalance Status
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #10313:
URL: https://github.com/apache/pinot/issues/10313#issuecomment-1444623797
> @Jackie-Jiang We can run a rebalance with `dryRun = true` and check the status when the API is invoked?
That is possible, but there is also a corner case as following:
1. Rebalance is triggered and finished
2. Some new segment is pushed/committed
3. Dry run gives a different result from the current assignment, but the rebalance is already done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #10313: API for Rebalance Status
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #10313:
URL: https://github.com/apache/pinot/issues/10313#issuecomment-1442331999
Rebalance is slightly different from reload because the target assignment (IS) can change multiple times with no-downtime settings, also if the IS changes while rebalancing, pinot will re-compute the target assignment.
To solve this, we can pass in a hook to the rebalancer, and let the rebalancer pass back the current rebalancing state. Only rebalancer knows the progress of the rebalance since IS can change dynamically.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang closed issue #10313: API for Rebalance Status
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang closed issue #10313: API for Rebalance Status
URL: https://github.com/apache/pinot/issues/10313
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #10313: API for Rebalance Status
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #10313:
URL: https://github.com/apache/pinot/issues/10313#issuecomment-1442336767
cc @navina
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #10313: API for Rebalance Status
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #10313:
URL: https://github.com/apache/pinot/issues/10313#issuecomment-1442358418
This is duplicate with #9558
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] suddendust commented on issue #10313: API for Rebalance Status
Posted by "suddendust (via GitHub)" <gi...@apache.org>.
suddendust commented on issue #10313:
URL: https://github.com/apache/pinot/issues/10313#issuecomment-1442856890
@Jackie-Jiang But we can calculate the target and the current assigned when the API is invoked? We can just run a rebalance with `dryRun = true` and check the status?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] swaminathanmanish commented on issue #10313: API for Rebalance Status
Posted by "swaminathanmanish (via GitHub)" <gi...@apache.org>.
swaminathanmanish commented on issue #10313:
URL: https://github.com/apache/pinot/issues/10313#issuecomment-1442330492
I'll be looking into this and will take it up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] swaminathanmanish commented on issue #10313: API for Rebalance Status
Posted by "swaminathanmanish (via GitHub)" <gi...@apache.org>.
swaminathanmanish commented on issue #10313:
URL: https://github.com/apache/pinot/issues/10313#issuecomment-1535171764
Completed -
Look for `checking the status` section in this doc
https://docs.pinot.apache.org/operators/operating-pinot/rebalance/rebalance-servers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] suddendust commented on issue #10313: API for Rebalance Status
Posted by "suddendust (via GitHub)" <gi...@apache.org>.
suddendust commented on issue #10313:
URL: https://github.com/apache/pinot/issues/10313#issuecomment-1446113945
@Jackie-Jiang Got it, controller should report back on the status. I did this using an Observer and registering it with the rebalancer: https://github.com/apache/pinot/pull/10345. I wasn't aware @swaminathanmanish has already taken this up. @swaminathanmanish Please see if you can use this PR to get a head-start, thanks :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] suddendust commented on issue #10313: API for Rebalance Status
Posted by "suddendust (via GitHub)" <gi...@apache.org>.
suddendust commented on issue #10313:
URL: https://github.com/apache/pinot/issues/10313#issuecomment-1446987553
@swaminathanmanish I think providing the users with the current status (IN_PROGRESS, DONE), the start time, the finish time (if complete) and the time elapsed is def needed. Additional info about which segments are yet to be rebalanced can be returned too, but in my exp, especially for tables with a large no. of segments, this info isn't really helpful - As users can hardly do anything but wait. Still, we can give a count of the no. of segments remaining, which can act as a progress indicator for the user. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] swaminathanmanish commented on issue #10313: API for Rebalance Status
Posted by "swaminathanmanish (via GitHub)" <gi...@apache.org>.
swaminathanmanish commented on issue #10313:
URL: https://github.com/apache/pinot/issues/10313#issuecomment-1446811536
Thanks @suddendust for sharing your ideas. I'll take a look. Im also trying to figure out what kind of response will be useful to users (more like requirements gathering).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] swaminathanmanish commented on issue #10313: API for Rebalance Status
Posted by "swaminathanmanish (via GitHub)" <gi...@apache.org>.
swaminathanmanish commented on issue #10313:
URL: https://github.com/apache/pinot/issues/10313#issuecomment-1447002029
Thanks @suddendust . This is useful.Yes the number of pending segments to be rebalanced can help detect the state when rebalance is stuck (or no progress when it continues to be in_progress indefinitely) for some reason.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] suddendust commented on issue #10313: API for Rebalance Status
Posted by "suddendust (via GitHub)" <gi...@apache.org>.
suddendust commented on issue #10313:
URL: https://github.com/apache/pinot/issues/10313#issuecomment-1441430986
I think this can be pretty similar to the reload API. During rebalance, we persist the rebalance config and the segments to monitor in ZK. When the status API is called, we can fetch the IS, the EV and check if they have converged (don't need to calculate the segments to move as it's already persisted in ZK). We can either respond with a yes or no, or can return the segments that have actually not converged yet. @Jackie-Jiang @saurabhd336 Does this sound like a correct approach? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org