[GitHub] [incubator-pinot] sajjad-moradi commented on pull request #5705: Add error message to broker response in case of broker send error

[GitBox <gi...@apache.org>]

sajjad-moradi commented on pull request #5705:
URL: https://github.com/apache/incubator-pinot/pull/5705#issuecomment-658860655


   > An ECONNREFUSED situation (indicating that the remote server refused connection) means that the pinot-server process on the server is not running.
   > It will happen very quickly (the kernel will return the error), definitely much quicker than any response from other requests for other servers. So, when the broker gets notified of a connection request failure, it is likely the first response seen in the map of responses.
   > 
   > If the pinot-server process has gone down, then it is likely the case that we have not gotten notification from Helix as yet (otherwise, we would have taken down the server from the routing table and would not be routing to that server).
   > 
   > So, we are trying to address a window between the time the server went down, and the time we get notified by Helix.
   > 
   > True, Helix can sometimes take longer time to notify (depending on zookeeper speed).
   > 
   > It is good to note in the PR that this is the case we are addressing.
   
   That's true. That's the issue that we faced and we want to prevent that, but please note that we're catching not only connection refused exception but all types of exceptions in `submitQuery()` and adding the exception stack trace to the returned broker response will help debugging for other issues as well.
   ```java
       for (Map.Entry<ServerRoutingInstance, InstanceRequest> entry : requestMap.entrySet()) {
         ServerRoutingInstance serverRoutingInstance = entry.getKey();
         try {
           _serverChannels.sendRequest(serverRoutingInstance, entry.getValue());
           asyncQueryResponse.markRequestSubmitted(serverRoutingInstance);
         } catch (Exception e) {
           LOGGER.error("Caught exception while sending request {} to server: {}, marking query failed", requestId,
               serverRoutingInstance, e);
           _brokerMetrics.addMeteredTableValue(rawTableName, BrokerMeter.REQUEST_SEND_EXCEPTIONS, 1);
           asyncQueryResponse.setBrokerRequestSendException(e);
           asyncQueryResponse.markQueryFailed();
           break;
         }
       }
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org