You are viewing a plain text version of this content. The canonical link for it is here.
Posted to httpclient-users@hc.apache.org by Pete Keyes <PK...@starbucks.com> on 2016/05/04 18:13:11 UTC

NoHttpResponseException received at startup

We have an odd issue wherein at Tomcat(TomEE) startup apache-hc execution requests throw a NoHttpResponseException within less than 1 second.  After a few minutes this behavior completely stops and isn’t seen again until the next time we stop and restart Tomcat(TomEE).

Basic apache-hc configuration information:
  - single static CloseableHttpClient instance initialized at container startup;
  - all requests managed via PoolingHttpClientConnectionManager
  - connection timeout 6000ms
  - read timeout 6000ms
  - we use 4 retries
  - the typical timing for each HttpClient.execute(HttpUriRequest,HttpContext) during this startup period is that they fail very fast:
      try-1: ~  5ms
      try-2: ~250ms
      try-3: ~500ms
      try-5: ~700ms
We measure the elapsed milli-seconds with respect to “execute()”.  It seems very odd that execute() fails so fast given the apache-hc settings.  It isn’t every execution request that fails.  But, a very high percentage of them do exhibit this behavior during the initial ~5 minutes after Tomcat(TomEE) begins dispatching inbound requests.

The inbound HTTP request volume inbound is ~70 RPS(requests per second).

After ~5 minutes this type of error never repeats.  We have logging in place that verifies all the apache-hc components are fully instantiated prior to any request being processed.

Version information:
  apache-hc: 4.5.2
  java     : 1.8r60
  tomcat   : 7.0.68
  tomee    : 1.7.4


Any ideas at all given the minimal information that I have to provide are greatly appreciated.

—
Pete


Re: NoHttpResponseException received at startup

Posted by Pete Keyes <PK...@starbucks.com>.
Thanks Oleg.  

The keyword “reset” is enough to help us point networking in a plausible
direction.
  - since this is a symptom associated with a broker connection this
confirms why each retry always fails
    within the same 5ms elapsed time;
  - we currently allow a retry using the same execution context;
      - obviously a fruitless waste of resource time;

— 
Pete






On 5/6/16, 5:24 AM, "Oleg Kalnichevski" <ol...@apache.org> wrote:

>On Thu, 2016-05-05 at 14:46 +0000, Pete Keyes wrote:
>> Update for the elapsed time for these calls that fail strictly at
>>startup.
>>  The stair-step 5/25/500... was strictly due to a pause we insert
>>between
>> retries.  All 4 tries fail within 5ms.
>> 
>> Given the configuration settings for CloseableHttpClient a failure
>>within
>> 5ms seems unlikely.  The trip to the F5 VIP and back would barely
>>complete
>> within 5ms.
>> 
>> 
>> Any thoughts on how to stabilize this a bit at startup?  Or glean more
>> information about why apache-hc would fail with NoHttpResponseException
>>so
>> quickly.  
>
>The best course of action would be to capture IP packets with a traffic
>analyzer such as Wireshark and to try to figure out why TCP connections
>get reset.
>
>Oleg 
>
>
>
>> Stack trace for the error:
>>     org.apache.http.NoHttpResponseException:
>> crm-asp-prod.starbucks.net:443 failed to respond
>>     
>> 
>>org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttp
>>Re
>> sponseParser.java:143)
>>     
>> 
>>org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttp
>>Re
>> sponseParser.java:57)
>>     
>> 
>>org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser
>>.j
>> ava:261)
>>     
>> 
>>org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(D
>>ef
>> aultBHttpClientConnection.java:165)
>>     
>> 
>>org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.jav
>>a:
>> 167)
>>     
>> 
>>org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpReques
>>tE
>> xecutor.java:272)
>>     
>> 
>>org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.
>>ja
>> va:124)
>>     
>> 
>>org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java
>>:2
>> 71)
>>     
>> 
>>org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184
>>)
>>     org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
>>     
>> 
>>org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110
>>)
>>     
>> 
>>org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClie
>>nt
>> .java:184)
>>     
>> 
>>org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClie
>>nt
>> .java:82)
>> 
>> 
>> 
>> ‹ 
>> Pete Keyes
>> 
>> 
>> 
>> 
>> 
>> 
>> On 5/4/16, 11:13 AM, "Pete Keyes" <PK...@starbucks.com> wrote:
>> 
>> >We have an odd issue wherein at Tomcat(TomEE) startup apache-hc
>>execution
>> >requests throw a NoHttpResponseException within less than 1 second.
>> >After a few minutes this behavior completely stops and isn¹t seen again
>> >until the next time we stop and restart Tomcat(TomEE).
>> >
>> >Basic apache-hc configuration information:
>> >  - single static CloseableHttpClient instance initialized at container
>> >startup;
>> >  - all requests managed via PoolingHttpClientConnectionManager
>> >  - connection timeout 6000ms
>> >  - read timeout 6000ms
>> >  - we use 4 retries
>> >  - the typical timing for each
>> >HttpClient.execute(HttpUriRequest,HttpContext) during this startup
>>period
>> >is that they fail very fast:
>> >      try-1: ~  5ms
>> >      try-2: ~250ms
>> >      try-3: ~500ms
>> >      try-5: ~700ms
>> >We measure the elapsed milli-seconds with respect to ³execute()².  It
>> >seems very odd that execute() fails so fast given the apache-hc
>>settings.
>> > It isn¹t every execution request that fails.  But, a very high
>> >percentage of them do exhibit this behavior during the initial ~5
>>minutes
>> >after Tomcat(TomEE) begins dispatching inbound requests.
>> >
>> >The inbound HTTP request volume inbound is ~70 RPS(requests per
>>second).
>> >
>> >After ~5 minutes this type of error never repeats.  We have logging in
>> >place that verifies all the apache-hc components are fully instantiated
>> >prior to any request being processed.
>> >
>> >Version information:
>> >  apache-hc: 4.5.2
>> >  java     : 1.8r60
>> >  tomcat   : 7.0.68
>> >  tomee    : 1.7.4
>> >
>> >
>> >Any ideas at all given the minimal information that I have to provide
>>are
>> >greatly appreciated.
>> >
>> >‹
>> >Pete
>> >
>> 
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>> 
>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>For additional commands, e-mail: httpclient-users-help@hc.apache.org
>


Re: NoHttpResponseException received at startup

Posted by Oleg Kalnichevski <ol...@apache.org>.
On Thu, 2016-05-05 at 14:46 +0000, Pete Keyes wrote:
> Update for the elapsed time for these calls that fail strictly at startup.
>  The stair-step 5/25/500... was strictly due to a pause we insert between
> retries.  All 4 tries fail within 5ms.
> 
> Given the configuration settings for CloseableHttpClient a failure within
> 5ms seems unlikely.  The trip to the F5 VIP and back would barely complete
> within 5ms.
> 
> 
> Any thoughts on how to stabilize this a bit at startup?  Or glean more
> information about why apache-hc would fail with NoHttpResponseException so
> quickly.  

The best course of action would be to capture IP packets with a traffic
analyzer such as Wireshark and to try to figure out why TCP connections
get reset.

Oleg 



> Stack trace for the error:
>     org.apache.http.NoHttpResponseException:
> crm-asp-prod.starbucks.net:443 failed to respond
>     
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpRe
> sponseParser.java:143)
>     
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpRe
> sponseParser.java:57)
>     
> org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.j
> ava:261)
>     
> org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(Def
> aultBHttpClientConnection.java:165)
>     
> org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:
> 167)
>     
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestE
> xecutor.java:272)
>     
> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.ja
> va:124)
>     
> org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:2
> 71)
>     
> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
>     org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
>     
> org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
>     
> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient
> .java:184)
>     
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient
> .java:82)
> 
> 
> 
> \u2039 
> Pete Keyes
> 
> 
> 
> 
> 
> 
> On 5/4/16, 11:13 AM, "Pete Keyes" <PK...@starbucks.com> wrote:
> 
> >We have an odd issue wherein at Tomcat(TomEE) startup apache-hc execution
> >requests throw a NoHttpResponseException within less than 1 second.
> >After a few minutes this behavior completely stops and isn�t seen again
> >until the next time we stop and restart Tomcat(TomEE).
> >
> >Basic apache-hc configuration information:
> >  - single static CloseableHttpClient instance initialized at container
> >startup;
> >  - all requests managed via PoolingHttpClientConnectionManager
> >  - connection timeout 6000ms
> >  - read timeout 6000ms
> >  - we use 4 retries
> >  - the typical timing for each
> >HttpClient.execute(HttpUriRequest,HttpContext) during this startup period
> >is that they fail very fast:
> >      try-1: ~  5ms
> >      try-2: ~250ms
> >      try-3: ~500ms
> >      try-5: ~700ms
> >We measure the elapsed milli-seconds with respect to �execute()�.  It
> >seems very odd that execute() fails so fast given the apache-hc settings.
> > It isn�t every execution request that fails.  But, a very high
> >percentage of them do exhibit this behavior during the initial ~5 minutes
> >after Tomcat(TomEE) begins dispatching inbound requests.
> >
> >The inbound HTTP request volume inbound is ~70 RPS(requests per second).
> >
> >After ~5 minutes this type of error never repeats.  We have logging in
> >place that verifies all the apache-hc components are fully instantiated
> >prior to any request being processed.
> >
> >Version information:
> >  apache-hc: 4.5.2
> >  java     : 1.8r60
> >  tomcat   : 7.0.68
> >  tomee    : 1.7.4
> >
> >
> >Any ideas at all given the minimal information that I have to provide are
> >greatly appreciated.
> >
> >\u2039
> >Pete
> >
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
> 



---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org


Re: NoHttpResponseException received at startup

Posted by Pete Keyes <PK...@starbucks.com>.
Update for the elapsed time for these calls that fail strictly at startup.
 The stair-step 5/25/500... was strictly due to a pause we insert between
retries.  All 4 tries fail within 5ms.

Given the configuration settings for CloseableHttpClient a failure within
5ms seems unlikely.  The trip to the F5 VIP and back would barely complete
within 5ms.


Any thoughts on how to stabilize this a bit at startup?  Or glean more
information about why apache-hc would fail with NoHttpResponseException so
quickly.  Stack trace for the error:
    org.apache.http.NoHttpResponseException:
crm-asp-prod.starbucks.net:443 failed to respond
    
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpRe
sponseParser.java:143)
    
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpRe
sponseParser.java:57)
    
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.j
ava:261)
    
org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(Def
aultBHttpClientConnection.java:165)
    
org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:
167)
    
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestE
xecutor.java:272)
    
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.ja
va:124)
    
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:2
71)
    
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
    org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
    
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
    
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient
.java:184)
    
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient
.java:82)



‹ 
Pete Keyes






On 5/4/16, 11:13 AM, "Pete Keyes" <PK...@starbucks.com> wrote:

>We have an odd issue wherein at Tomcat(TomEE) startup apache-hc execution
>requests throw a NoHttpResponseException within less than 1 second.
>After a few minutes this behavior completely stops and isn¹t seen again
>until the next time we stop and restart Tomcat(TomEE).
>
>Basic apache-hc configuration information:
>  - single static CloseableHttpClient instance initialized at container
>startup;
>  - all requests managed via PoolingHttpClientConnectionManager
>  - connection timeout 6000ms
>  - read timeout 6000ms
>  - we use 4 retries
>  - the typical timing for each
>HttpClient.execute(HttpUriRequest,HttpContext) during this startup period
>is that they fail very fast:
>      try-1: ~  5ms
>      try-2: ~250ms
>      try-3: ~500ms
>      try-5: ~700ms
>We measure the elapsed milli-seconds with respect to ³execute()².  It
>seems very odd that execute() fails so fast given the apache-hc settings.
> It isn¹t every execution request that fails.  But, a very high
>percentage of them do exhibit this behavior during the initial ~5 minutes
>after Tomcat(TomEE) begins dispatching inbound requests.
>
>The inbound HTTP request volume inbound is ~70 RPS(requests per second).
>
>After ~5 minutes this type of error never repeats.  We have logging in
>place that verifies all the apache-hc components are fully instantiated
>prior to any request being processed.
>
>Version information:
>  apache-hc: 4.5.2
>  java     : 1.8r60
>  tomcat   : 7.0.68
>  tomee    : 1.7.4
>
>
>Any ideas at all given the minimal information that I have to provide are
>greatly appreciated.
>
>‹
>Pete
>


---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org