You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "piby180 (via GitHub)" <gi...@apache.org> on 2026/07/01 17:06:55 UTC
[I] Brief window of query failures immediately after a server pod finishes restarting and becomes healthy [pinot]
piby180 opened a new issue, #18902:
URL: https://github.com/apache/pinot/issues/18902
We regularly experience a brief window of query failures immediately after a Pinot server pod transitions to a healthy state following a restart. Notably, there is no observed disruption during the initial termination phase or while the pod is in a 0/1 Runningstate; the instability occurs specifically as the service becomes ready to handle traffic again.
### Error Details
The failures manifest as QueryExecutionError exceptions (likely caused by gRPC connection timeouts). The logs indicate that the query dispatcher is unable to establish a connection to the newly restarted pod.
Full Java Stack Trace:
```
QueryExecutionError: Error dispatching query: 667717545000000055 to server: pinot-server-8.pinot-server-headless.pinot.svc.cluster.local@{8421,8442}
at org.apache.pinot.query.service.dispatch.QueryDispatcher.processResults(QueryDispatcher.java:689)
at org.apache.pinot.query.service.dispatch.QueryDispatcher.execute(QueryDispatcher.java:643)
at org.apache.pinot.query.service.dispatch.QueryDispatcher.submit(QueryDispatcher.java:584)
at org.apache.pinot.query.service.dispatch.QueryDispatcher.submitAndReduce(QueryDispatcher.java:219)
Caused by: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
at io.grpc.Status.asRuntimeException(Status.java:532)
at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:581)
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:566)
at io.grpc.internal.ClientCallImpl.access$100(ClientCallImpl.java:72)
at io.grpc.internal.ClientCallImpl$ClientCallListenerImpl.onClose(ClientCallImpl.java:547)
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:566)
at io.grpc.internal.ClientCallImpl.access$100(ClientCallImpl.java:72)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:739)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:720)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.io.IOException: connection timed out after 30000 ms: pinot-server-8.pinot-server-headless.pinot.svc.cluster.local/100.64.62.48:8421
at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$2.run(AbstractEpollChannel.java:615)
at io.grpc.netty.shaded.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
at io.grpc.netty.shaded.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:160)
at io.grpc.netty.shaded.io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
at io.grpc.netty.shaded.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.base/java.lang.Thread.run(Thread.java:840)
```
### Environment Context
Pinot Version: 1.5.0
EKS 1.34
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Brief window of query failures immediately after a server pod finishes restarting and becomes healthy [pinot]
Posted by "piby180 (via GitHub)" <gi...@apache.org>.
piby180 commented on issue #18902:
URL: https://github.com/apache/pinot/issues/18902#issuecomment-5033632679
@pri1712 Thanks for the tip. I will test this on our dev cluster within this week and will report back here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Brief window of query failures immediately after a server pod finishes restarting and becomes healthy [pinot]
Posted by "pri1712 (via GitHub)" <gi...@apache.org>.
pri1712 commented on issue #18902:
URL: https://github.com/apache/pinot/issues/18902#issuecomment-5011664330
can you define 'brief' is it few seconds/minutes/hours? would help scope out this issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Brief window of query failures immediately after a server pod finishes restarting and becomes healthy [pinot]
Posted by "piby180 (via GitHub)" <gi...@apache.org>.
piby180 commented on issue #18902:
URL: https://github.com/apache/pinot/issues/18902#issuecomment-5011813152
We see it during graceful termination by kubernetes. We use kubernetes RollungUpdate strategy and servers pods are restarted one by one.
We also set terminationGracePeriod to 10 mins so it has enough time to gracefully shut down the server.
These errors come when a new server starts to become healthy and not during shutdown process
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Brief window of query failures immediately after a server pod finishes restarting and becomes healthy [pinot]
Posted by "pri1712 (via GitHub)" <gi...@apache.org>.
pri1712 commented on issue #18902:
URL: https://github.com/apache/pinot/issues/18902#issuecomment-5011817597
thats weird, let me see if i can come up with some reasoning
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Brief window of query failures immediately after a server pod finishes restarting and becomes healthy [pinot]
Posted by "pri1712 (via GitHub)" <gi...@apache.org>.
pri1712 commented on issue #18902:
URL: https://github.com/apache/pinot/issues/18902#issuecomment-5156173796
let me set this up in a dev env and experiment. thanks for pointing out this issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Brief window of query failures immediately after a server pod finishes restarting and becomes healthy [pinot]
Posted by "piby180 (via GitHub)" <gi...@apache.org>.
piby180 commented on issue #18902:
URL: https://github.com/apache/pinot/issues/18902#issuecomment-5011710445
> can you define 'brief' is it a few seconds/minutes/hours? would help scope out this issue @piby180
It is around 5-10 second window for each server.
It seems to be related to
https://github.com/apache/pinot/issues/17465
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Brief window of query failures immediately after a server pod finishes restarting and becomes healthy [pinot]
Posted by "pri1712 (via GitHub)" <gi...@apache.org>.
pri1712 commented on issue #18902:
URL: https://github.com/apache/pinot/issues/18902#issuecomment-5033237745
@piby180 please confirm once you have had a chance to try this out
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Brief window of query failures immediately after a server pod finishes restarting and becomes healthy [pinot]
Posted by "timothy-e (via GitHub)" <gi...@apache.org>.
timothy-e commented on issue #18902:
URL: https://github.com/apache/pinot/issues/18902#issuecomment-5180161987
I think this may also be related to https://github.com/apache/pinot/issues/16565
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Brief window of query failures immediately after a server pod finishes restarting and becomes healthy [pinot]
Posted by "pri1712 (via GitHub)" <gi...@apache.org>.
pri1712 commented on issue #18902:
URL: https://github.com/apache/pinot/issues/18902#issuecomment-5014620524
this is probably an issue with how fast the Helix is able to propagate changes about a servers state to Brokers. When a broker detects that a given server is up, it tries routing queries to it but due to various caches it may be using the older IP address leading to connection timeouts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Brief window of query failures immediately after a server pod finishes restarting and becomes healthy [pinot]
Posted by "pri1712 (via GitHub)" <gi...@apache.org>.
pri1712 commented on issue #18902:
URL: https://github.com/apache/pinot/issues/18902#issuecomment-5011800370
that involved killing the pinot servers, are you seeing this for graceful termination of server pods as well? In the other case it was attributed to grpc connections doing an exponential backoff which lead to them hitting servers even post they were dead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Brief window of query failures immediately after a server pod finishes restarting and becomes healthy [pinot]
Posted by "piby180 (via GitHub)" <gi...@apache.org>.
piby180 commented on issue #18902:
URL: https://github.com/apache/pinot/issues/18902#issuecomment-5118220012
@pri1712 I tested `pinot.broker.failure.detector.type = CONNECTION` but still same error.
I also tested adding tag `queriesDisabled=true` before restarting a server and removing it after the server is restarted.
As soon as I remove the tag `queriesDisabled=true` from the server and fresh queries are routed to it, we get the error for first 5 seconds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org