You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/07/14 00:52:35 UTC

[GitHub] [pinot] dang-stripe opened a new issue, #9056: Queries being routed to broker even though not all server segments are online yet

dang-stripe opened a new issue, #9056:
URL: https://github.com/apache/pinot/issues/9056

Our offline segment servers take a very long time to load all segments. We've set a very high value for `pinot.server.startup.timeoutMs` to compensate for this and to avoid having the server receive queries until all segments have moved to online state.

We noticed that there are still queries being routed to the server while it's not yet healthy though. Is there a configuration we're missing here that would prevent this?

The remaining pinot server conf is mostly using defaults. `pinot.server.startup.minResourcePercent` is not set, which I believe defaults to 100%.

```
[2022-07-13 19:22:48.586429] INFO [PinotServiceManager] [Start a Pinot [SERVER]] Trying to start Pinot Server...

// seeing this server in broker logs at with failure to respond (responseTimeMs == -1):
2022-07-13 22:20:44.229
2022-07-13 22:32:30.344
2022-07-13 22:47:29.760
...

[2022-07-13 23:11:04.748064] INFO [BaseServerStarter] [Start a Pinot [SERVER]] Service status is GOOD after 13683066ms
```

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] Jackie-Jiang commented on issue #9056: Queries being routed to broker even though not all server segments are online yet

Posted by GitBox <gi...@apache.org>.

Jackie-Jiang commented on issue #9056:
URL: https://github.com/apache/pinot/issues/9056#issuecomment-1188405356

   Sync'ed offline, and the root cause of the issue is that the server is new added, so broker will start routing queries to it once the segments become online in external view. #8525 should be able to fix this


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] mcvsubbu commented on issue #9056: Queries being routed to broker even though not all server segments are online yet

Posted by GitBox <gi...@apache.org>.

mcvsubbu commented on issue #9056:
URL: https://github.com/apache/pinot/issues/9056#issuecomment-1184732425

   There is a heatlhcheck call on the server that returns good only if all segments for the server are loaded and available in external view.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] Jackie-Jiang commented on issue #9056: Queries being routed to server even though not all server segments are online yet

Posted by GitBox <gi...@apache.org>.

Jackie-Jiang commented on issue #9056:
URL: https://github.com/apache/pinot/issues/9056#issuecomment-1208753739

   @suddendust Just to clarify, do you main queries get routed to unavailable servers? The fix in #8525 will be available in the next release (0.11.0)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] Jackie-Jiang closed issue #9056: Queries being routed to server even though not all server segments are online yet

Posted by GitBox <gi...@apache.org>.

Jackie-Jiang closed issue #9056: Queries being routed to server even though not all server segments are online yet
URL: https://github.com/apache/pinot/issues/9056


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] suddendust commented on issue #9056: Queries being routed to broker even though not all server segments are online yet

Posted by GitBox <gi...@apache.org>.

suddendust commented on issue #9056:
URL: https://github.com/apache/pinot/issues/9056#issuecomment-1207243384

   We've seen this too in 0.9.x. Queries get routed to unavailable brokers.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] Jackie-Jiang commented on issue #9056: Queries being routed to broker even though not all server segments are online yet

Posted by GitBox <gi...@apache.org>.

Jackie-Jiang commented on issue #9056:
URL: https://github.com/apache/pinot/issues/9056#issuecomment-1184704593

   Wow, it takes over 3 hours to get all segments loaded?
   
   Is this a new added server? If not, did it shut down regularly? #8525 fixes the issue of broker hitting a server when it is new added or not shut down regularly.
   #8785 fixes the issue of server opening up query port prematurely.
   
   Based on the broker log, seems the server didn't even process the request (query port might not open yet), so it could be due to helix callback delay. Can you please verify if this is the case, and check if the broker gets the instance config change callback and removes the server from the routing table?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] dang-stripe commented on issue #9056: Queries being routed to broker even though not all server segments are online yet

Posted by GitBox <gi...@apache.org>.

dang-stripe commented on issue #9056:
URL: https://github.com/apache/pinot/issues/9056#issuecomment-1184731288

   This is a newly added server. We don't have those fixes in our version so I'll try to pull them in. Which helix callback does this happen in just so I know what to search for or is there a metric for this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org