You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@calcite.apache.org by "Josh Elser (JIRA)" <ji...@apache.org> on 2016/07/14 21:57:21 UTC
[jira] [Resolved] (CALCITE-1315) Automatically retry request when a
server fails to deliver an HTTP response
[ https://issues.apache.org/jira/browse/CALCITE-1315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Elser resolved CALCITE-1315.
---------------------------------
Resolution: Fixed
Fixed in https://git1-us-west.apache.org/repos/asf?p=calcite.git;a=commit;h=799e1b083b57d2295681d3e58c4d88563f7ccc91
> Automatically retry request when a server fails to deliver an HTTP response
> ---------------------------------------------------------------------------
>
> Key: CALCITE-1315
> URL: https://issues.apache.org/jira/browse/CALCITE-1315
> Project: Calcite
> Issue Type: Bug
> Components: avatica
> Reporter: Josh Elser
> Assignee: Josh Elser
> Fix For: avatica-1.9.0
>
>
> I've been running my load-balancer test using HAProxy between an Avatica client and multiple avatica servers. With the help of [~speleato], we've noticed that we're still seeing some intermittent failures in the client when we take down a backend server.
> The client reports a NoHttpResponseException:
> {noformat}
> java.lang.RuntimeException: org.apache.http.NoHttpResponseException: localhost:8888 failed to respond
> at org.apache.calcite.avatica.remote.AvaticaCommonsHttpClientImpl.send(AvaticaCommonsHttpClientImpl.java:149)
> at org.apache.calcite.avatica.remote.RemoteProtobufService._apply(RemoteProtobufService.java:44)
> at org.apache.calcite.avatica.remote.ProtobufService.apply(ProtobufService.java:69)
> at org.apache.calcite.avatica.remote.RemoteMeta$14.call(RemoteMeta.java:307)
> at org.apache.calcite.avatica.remote.RemoteMeta$14.call(RemoteMeta.java:305)
> at org.apache.calcite.avatica.AvaticaConnection.invokeWithRetries(AvaticaConnection.java:715)
> at org.apache.calcite.avatica.remote.RemoteMeta.fetch(RemoteMeta.java:304)
> at org.apache.calcite.avatica.MetaImpl$FetchIterator.moveNext(MetaImpl.java:908)
> at org.apache.calcite.avatica.MetaImpl$FetchIterator.next(MetaImpl.java:893)
> at org.apache.calcite.avatica.util.IteratorCursor.next(IteratorCursor.java:46)
> at org.apache.calcite.avatica.AvaticaResultSet.next(AvaticaResultSet.java:218)
> at sqlline.BufferedRows.<init>(BufferedRows.java:37)
> at sqlline.SqlLine.print(SqlLine.java:1649)
> at sqlline.Commands.execute(Commands.java:833)
> at sqlline.Commands.sql(Commands.java:732)
> at sqlline.SqlLine.dispatch(SqlLine.java:807)
> at sqlline.SqlLine.runCommands(SqlLine.java:1710)
> at sqlline.Commands.run(Commands.java:1285)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)
> at sqlline.SqlLine.dispatch(SqlLine.java:803)
> at sqlline.SqlLine.initArgs(SqlLine.java:613)
> at sqlline.SqlLine.begin(SqlLine.java:656)
> at sqlline.SqlLine.start(SqlLine.java:398)
> at sqlline.SqlLine.main(SqlLine.java:292)
> at org.apache.phoenix.queryserver.client.SqllineWrapper.main(SqllineWrapper.java:83)
> Caused by: org.apache.http.NoHttpResponseException: localhost:8888 failed to respond
> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
> at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
> at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
> at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
> at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
> at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
> at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
> at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
> at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
> at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
> at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
> at org.apache.calcite.avatica.remote.AvaticaCommonsHttpClientImpl.execute(AvaticaCommonsHttpClientImpl.java:157)
> at org.apache.calcite.avatica.remote.AvaticaCommonsHttpClientImpl.send(AvaticaCommonsHttpClientImpl.java:134)
> ... 28 more
> {noformat}
> I'm pretty sure what's happening (and what I can't see as something which we can change in the loadbalancer itself), is that a client submits a request to the LB which is then sent to the backend server, the backend server starts processing that request, but fails to ever respond. Instead of the LB sending back an HTTP/503 (as we've seen it do before), the client just gets a terminated connection while waiting for the response. I'm not surprised that HAProxy does not automatically retry this for us.
> So, the Avatica client can just retry this instead like we do on HTTP/503's already.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)