[I] [multistage][bug] error propagation during GRPC failure is not handled when setup is not finished. [pinot]

["walterddr (via GitHub)" <gi...@apache.org>]

walterddr opened a new issue, #11919:
URL: https://github.com/apache/pinot/issues/11919

   when GRPC internal failure occurs. the ReceivingMailbox doesn't propagate that error back to the next block fetcher if the message was the FIRST block
   error from broker:
   ```
   2023/11/01 08:45:52.204 ERROR [MultiStageBrokerRequestHandler] [jersey-server-managed-async-executor-7] Caught exception executing request 1879852788000000005: select * from tbl limit 6000, 
   Received error query execution result block: {250=ExecutionTimeoutError
   ProcessingException(errorCode:250, message:ExecutionTimeoutError)
   	at org.apache.pinot.common.exception.QueryException.<clinit>(QueryException.java:113)
   	at org.apache.pinot.common.response.broker.BrokerResponseNative.<clinit>(BrokerResponseNative.java:53)
   	at org.apache.pinot.broker.requesthandler.MultiStageBrokerRequestHandler.handleRequest(MultiStageBrokerRequestHandler.java:156)
   	at org.apache.pinot.broker.requesthandler.BaseBrokerRequestHandler.handleRequest(BaseBrokerRequestHandler.java:279)}
   org.apache.pinot.query.service.dispatch.QueryDispatcher.getResultTable(QueryDispatcher.java:266)
   ```
   
   but the server log indicates that:
   ```
   Nov 01, 2023 8:45:42 AM io.grpc.netty.shaded.io.grpc.netty.NettyServerStream$TransportState deframeFailed
   WARNING: Exception processing message
   io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: gRPC message exceeds maximum size 16777216: 17562988
   	at io.grpc.Status.asRuntimeException(Status.java:530)
   	at io.grpc.internal.MessageDeframer.processHeader(MessageDeframer.java:392)
   	at io.grpc.internal.MessageDeframer.deliver(MessageDeframer.java:272)
   	at io.grpc.internal.MessageDeframer.deframe(MessageDeframer.java:178)
   ...
   2023/11/01 08:45:42.788 ERROR [MailboxContentObserver] [grpc-default-executor-12] Got error before mailbox is set up
   io.grpc.StatusRuntimeException: CANCELLED: client cancelled
   	at io.grpc.Status.asRuntimeException(Status.java:530) ~[grpc-api-1.53.0.jar:1.53.0]
   	at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:291) [grpc-stub-1.53.0.jar:1.53.0]
   	at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:378) [grpc-core-1.53.0.jar:1.53.0]
   	at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:365) [grpc-core-1.53.0.jar:1.53.0]
   	at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:923) [grpc-core-1.53.0.jar:1.53.0]
   	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) [grpc-core-1.53.0.jar:1.53.0]
   	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) [grpc-core-1.53.0.jar:1.53.0]
   	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
   	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
   	at java.lang.Thread.run(Thread.java:829) [?:?]
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org