You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "jadami10 (via GitHub)" <gi...@apache.org> on 2025/08/09 20:48:34 UTC
[I] Server returns `/health = OK` before server is ready to server queries [pinot]
jadami10 opened a new issue, #16565:
URL: https://github.com/apache/pinot/issues/16565
We've been hit by a bug related to #14529 and external orchestration of Pinot restarts.
Let's assume you have a table with 2 replica groups
- External system sends SIGTERM to pinot-server-1
- pinot-server-1 sets `IS_SHUTDOWN_IN_PROGRESS
- Pinot broker stops routing to pinot-server-1
- pinot-server-1 starts back up with `/health` not returning `OK`
- pinot-server-1 [startupServiceStatusCheck](https://github.com/apache/pinot/blob/642bf00501ef0cc0ddb79ade00b2eff695590ea0/pinot-server/src/main/java/org/apache/pinot/server/starter/helix/BaseServerStarter.java#L150) completes.
- *start of problem*: External system seems `/health` return OK
- *problem*: External system restarts pinot-server-2
- *problem*: Queries fail because `pinot-server-1` and `pinot-server-2` are both not serving queries
- `pinot-server-1` sets `IS_SHUTDOWN_IN_PROGRESS` false
- Broker adds `pinot-server-1` back to the routing table, and queries succeed again
In our case, this caused ~17 seconds of down time.
It's not clear how to orchestrate this correctly in Pinot. It seems you have to check the broker routing table for every table to ensure your server is found in there. But there's no clear API for "Is X server available for all necessary segments" or "is Y server going to cause downtime if I take it down". So if you're performing a rolling restart, you're kind of crossing your fingers you wait long enough between replica group restarts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Server returns `/health = OK` before server is ready to server queries [pinot]
Posted by "jadami10 (via GitHub)" <gi...@apache.org>.
jadami10 commented on issue #16565:
URL: https://github.com/apache/pinot/issues/16565#issuecomment-3189966117
we discussed this live, there's 2 items we think are right to do here:
1. server `/health` for readiness should be updated to check [IS_SHUTDOWN_IN_PROGRESS](https://github.com/apache/pinot/blob/642bf00501ef0cc0ddb79ade00b2eff695590ea0/pinot-server/src/main/java/org/apache/pinot/server/starter/helix/BaseServerStarter.java#L799C38-L800C40) is set to false. This doesn't completely cover the issue, but it ensures `/health` returns OK _after_ the query server has started and gives the broker more time to process the server status change
2. we need some form of coordination between the external view the server knows about and the external view the broker routing map is based on. We need some form of watermark to be able to say "broker X has processed external view changes up to version V" in order to ensure either 1) servers are not marked healthy until brokers have rebuilt the routing map with the latest external view changes or 2) an external orchestrating system is able to correctly await broker routing map changes in order to inform when the next batch of servers can be restarted
the first item is much more straightforward, so we can start there.
I likely won't have time to get to this for a few months, and we've papered over this by adding `sleep(30s)` between batches of servers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Server returns `/health = OK` before server is ready to serve queries [pinot]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on issue #16565:
URL: https://github.com/apache/pinot/issues/16565#issuecomment-5212314495
@Jackie-Jiang I opened #19178 for item 1. It keeps `/health` and `/health/readiness` at `503` until the query-serving path is ready, while leaving liveness unchanged. The change reuses the existing local readiness signal and includes focused endpoint coverage. The broker watermark discussed in item 2 is intentionally out of scope for this PR. Let me know if any suggestions. thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Server returns `/health = OK` before server is ready to serve queries [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #16565:
URL: https://github.com/apache/pinot/issues/16565#issuecomment-4113109726
This issue has been automatically marked as stale because it has not had any activity in the last 150 days. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Server returns `/health = OK` before server is ready to serve queries [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #16565:
URL: https://github.com/apache/pinot/issues/16565#issuecomment-4320903047
This issue has been closed because it has been inactive for 180 days. If you believe this issue is still relevant, please feel free to reopen it or create a new issue. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Server returns `/health = OK` before server is ready to serve queries [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #16565:
URL: https://github.com/apache/pinot/issues/16565#issuecomment-5208675728
cc @J-HowHuang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Server returns `/health = OK` before server is ready to server queries [pinot]
Posted by "jadami10 (via GitHub)" <gi...@apache.org>.
jadami10 commented on issue #16565:
URL: https://github.com/apache/pinot/issues/16565#issuecomment-3184079164
well 14529 is semi-related. The majority of the issue is for sure the race condition between server 1 reporting healthy and broker routing table updates. But the issue is exacerbated is server 1 comes up healthy without all segments being ONLINE (either they're in ERROR state or they're still OFFLINE and the server timed out polling)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Server returns `/health = OK` before server is ready to server queries [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #16565:
URL: https://github.com/apache/pinot/issues/16565#issuecomment-3182247528
I don't think #14529 is for the same issue. It is mainly about handling `ERROR` in external view.
Seems you are hitting the race condition on broker routing table updates, where at a given point both server 1 and 2 are marked as `IS_SHUTDOWN_IN_PROGRESS`. Can you confirm if this is the issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Server returns `/health = OK` before server is ready to serve queries [pinot]
Posted by "pri1712 (via GitHub)" <gi...@apache.org>.
pri1712 commented on issue #16565:
URL: https://github.com/apache/pinot/issues/16565#issuecomment-5194684336
@Jackie-Jiang seeing multiple issues that may be related, I would like to investigate into this, pls assign any one of them to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Server returns `/health = OK` before server is ready to server queries [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #16565:
URL: https://github.com/apache/pinot/issues/16565#issuecomment-3185041884
I see. On this [line](https://github.com/apache/pinot/blob/master/pinot-common/src/main/java/org/apache/pinot/common/utils/ServiceStatus.java#L480) we log an error when segment goes into error state, but allow it to proceed because segment won't auto recover. We can consider adding a config to do status check in **strict** mode, so that server fails to start when encountering segment load error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Server returns `/health = OK` before server is ready to server queries [pinot]
Posted by "jadami10 (via GitHub)" <gi...@apache.org>.
jadami10 commented on issue #16565:
URL: https://github.com/apache/pinot/issues/16565#issuecomment-3185956882
sorry, let's forget the error segment case and strictly look at the example in the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Server returns `/health = OK` before server is ready to serve queries [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #16565: Server returns `/health = OK` before server is ready to serve queries
URL: https://github.com/apache/pinot/issues/16565
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org