You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "ankitsultana (via GitHub)" <gi...@apache.org> on 2025/05/08 04:18:49 UTC
[I] [multistage] Mailbox Errs Without Logs on Inbound Message Limit Breach [pinot]
ankitsultana opened a new issue, #15741:
URL: https://github.com/apache/pinot/issues/15741
We were seeing an issue where queries would fail with "Client cancelled" on the receiver side, and "RST_STREAM" error on the sender side. Outside of this, there were no logs to indicate what the issue was.
Enabling gRPC Debug logs showed that the issue was `errorCode=8`, which refers to resource exhaustion: https://grpc.github.io/grpc/core/md_doc_statuscodes.html
After increasing the buffer limit the queries work fine.
```
{"@timestamp":"2025-05-08T02:47:38.502+00:00","@version":1,"message":"[id: 0x047e448d, L:/10.148.247.202:44029 - R:/10.148.33.163:32918] OUTBOUND RST_STREAM: streamId=15 errorCode=8","logger_name":"io.grpc.netty.shaded.io.grpc.netty.NettyServerHandler","thread_name":"grpc-default-worker-ELG-3-1","level":"DEBUG","level_value":10000,"HOSTNAME":"some-hostname"}
```
As part of this ticket we should improve logging for this scenario.
This is kinda unexpected because in the past I have seen "RESOURCE_EXHAUSTED" errors when gRPC message size limits are breached, but maybe BiDir calls are not handled gracefully by gRPC?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] [multistage] Mailbox Errs Without Logs on Inbound Message Limit Breach [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #15741:
URL: https://github.com/apache/pinot/issues/15741#issuecomment-4340080888
This issue has been closed because it has been inactive for 180 days. If you believe this issue is still relevant, please feel free to reopen it or create a new issue. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] [multistage] Mailbox Errs Without Logs on Inbound Message Limit Breach [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #15741:
URL: https://github.com/apache/pinot/issues/15741#issuecomment-4151441659
This issue has been automatically marked as stale because it has not had any activity in the last 150 days. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] [multistage] Mailbox Errs Without Logs on Inbound Message Limit Breach [pinot]
Posted by "ankitsultana (via GitHub)" <gi...@apache.org>.
ankitsultana commented on issue #15741:
URL: https://github.com/apache/pinot/issues/15741#issuecomment-2967476638
@shauryachats : can you confirm if your patch has fixed this issue?
As in, before your patch, queries with this error would fail silently and block till timeout.
After your patch, queries should fail fast and return a proper error message to users.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] [multistage] Mailbox Errs Without Logs on Inbound Message Limit Breach [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #15741: [multistage] Mailbox Errs Without Logs on Inbound Message Limit Breach
URL: https://github.com/apache/pinot/issues/15741
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org