You are viewing a plain text version of this content. The canonical link for it is here.
Posted to httpclient-users@hc.apache.org by Mark Rekveld <mr...@sonatype.com> on 2016/06/22 11:00:06 UTC

Help/Advise debugging SocketException in BHttpConnectionBase.fillInputBuffer()

Hi,

I would like to ask for your assistance to resolve an issue that we are
facing that has the nature of being a race condition.

The problem is that a SocketException is thrown in the finally clause of
BHttpConnectionBase.fillInputBuffer() [
https://github.com/apache/httpcore/blob/4.3.x/httpcore/src/main/java/org/apache/http/impl/BHttpConnectionBase.java#L326
]

We use 4.3.6 of the HttpClient with 4.3.3 of HttpCore, and have a custom
extension of org.apache.http.protocol.HttpRequestExecutor that sends the
request entity asynchronously and immediately try to read the response from
the sever.

The exception occurs within this custom HttpRequestExecutor when the
conn.isResponseAvailable(1000 * 5) call is made in this code snippet. See
stack trace below

private void sendRequestEntity(final HttpEntityEnclosingRequest req) throws
IOException {
  Thread entitySender = new Thread(getClass().getSimpleName() + "-" +
System.currentTimeMillis())
  {
    @Override
    public void run() {
      try {
        conn.sendRequestEntity(req);
        conn.flush();
        context.setAttribute(HttpCoreContext.HTTP_REQ_SENT, Boolean.TRUE);
      }
      catch (Throwable t) {
        error = t;
      }
    }
  };
  entitySender.start();

  while (entitySender.isAlive() && !conn.isResponseAvailable(1000 * 5)) {
    /*
     * Wait here until either we have fully uploaded the request entity or
the server sends a response. If
     * we were to return immediately and called doReceiveResponse() right
away we would provoke socket
     * timeouts.
     */
  }
}

The full stack trace:

2016-06-03 02:36:04,149+1000 ERROR [qtp761509273-12469 - PUT
/rest/ci/scan/treasury-web-deployable] jenkins-server
com.sonatype.insight.brain.hds.HdsClient - Socket is closed
java.net.SocketException: Socket is closed
 at java.net.Socket.setSoTimeout(Socket.java:1137) ~[na:1.8.0_91]
 at
sun.security.ssl.BaseSSLSocketImpl.setSoTimeout(BaseSSLSocketImpl.java:633)
~[na:1.8.0_91]
 at sun.security.ssl.SSLSocketImpl.setSoTimeout(SSLSocketImpl.java:2526)
~[na:1.8.0_91]
 at
org.apache.http.impl.BHttpConnectionBase.fillInputBuffer(SourceFile:326)
~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at org.apache.http.impl.BHttpConnectionBase.awaitInput(SourceFile:334)
~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
org.apache.http.impl.DefaultBHttpClientConnection.isResponseAvailable(SourceFile:131)
~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
org.apache.http.impl.conn.CPoolProxy.isResponseAvailable(SourceFile:141)
~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
com.sonatype.insight.client.utils.HttpRequestExecutor$HttpExchange.sendRequestEntity(SourceFile:201)
~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
com.sonatype.insight.client.utils.HttpRequestExecutor$HttpExchange.doSendRequest(SourceFile:177)
~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
com.sonatype.insight.client.utils.HttpRequestExecutor$HttpExchange.perform(SourceFile:108)
~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
com.sonatype.insight.client.utils.HttpRequestExecutor.execute(SourceFile:53)
~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at org.apache.http.impl.execchain.MainClientExec.execute(SourceFile:254)
~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at org.apache.http.impl.execchain.ProtocolExec.execute(SourceFile:195)
~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at org.apache.http.impl.execchain.RetryExec.execute(SourceFile:86)
~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at org.apache.http.impl.execchain.RedirectExec.execute(SourceFile:108)
~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
org.apache.http.impl.client.InternalHttpClient.doExecute(SourceFile:184)
~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at org.apache.http.impl.client.CloseableHttpClient.execute(SourceFile:82)
~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at org.apache.http.impl.client.CloseableHttpClient.execute(SourceFile:106)
~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at org.apache.http.impl.client.CloseableHttpClient.execute(SourceFile:57)
~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at com.sonatype.insight.brain.hds.HdsClient.execute(SourceFile:380)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at com.sonatype.insight.brain.hds.HdsClient.execute(SourceFile:329)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at com.sonatype.insight.brain.hds.HdsClient.getResponse(SourceFile:132)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at com.sonatype.insight.brain.hds.HdsClient.get(SourceFile:178)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at com.sonatype.insight.brain.hds.ScanUploader.upload(SourceFile:69)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at com.sonatype.insight.brain.hds.CIResource.putScan(SourceFile:50)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
com.sonatype.insight.brain.hds.CIResource$$EnhancerByGuice$$88667ab8.CGLIB$putScan$0(<generated>)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
com.sonatype.insight.brain.hds.CIResource$$EnhancerByGuice$$88667ab8$$FastClassByGuice$$99131cf1.invoke(<generated>)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(SourceFile:228)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(SourceFile:75)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
org.apache.shiro.guice.aop.AopAllianceMethodInvocationAdapter.proceed(SourceFile:49)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
com.sonatype.insight.brain.security.AuthorizeMethodInterceptor.invoke(SourceFile:62)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
org.apache.shiro.guice.aop.AopAllianceMethodInterceptorAdapter.invoke(SourceFile:36)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(SourceFile:75)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
com.google.inject.internal.InterceptorStackCallback.intercept(SourceFile:55)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
com.sonatype.insight.brain.hds.CIResource$$EnhancerByGuice$$88667ab8.putScan(<generated>)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at sun.reflect.GeneratedMethodAccessor196.invoke(Unknown Source) ~[na:na]
 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[na:1.8.0_91]
 at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_91]
 at
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(SourceFile:60)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(SourceFile:185)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(SourceFile:75)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
com.yammer.dropwizard.jersey.OptionalResourceMethodDispatchAdapter$OptionalRequestDispatcher.dispatch(SourceFile:37)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(SourceFile:302)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(SourceFile:147)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(SourceFile:108)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(SourceFile:147)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(SourceFile:84)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(SourceFile:1511)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(SourceFile:1442)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(SourceFile:1391)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(SourceFile:1381)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
com.sun.jersey.spi.container.servlet.WebComponent.service(SourceFile:416)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
com.sun.jersey.spi.container.servlet.ServletContainer.service(SourceFile:538)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
com.sun.jersey.spi.container.servlet.ServletContainer.service(SourceFile:716)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at javax.servlet.http.HttpServlet.service(SourceFile:848)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at org.eclipse.jetty.servlet.ServletHolder.handle(SourceFile:669)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(SourceFile:1448)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
com.sonatype.insight.brain.security.AuthenticationLoggingFilter.doFilter(SourceFile:53)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(SourceFile:1419)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(SourceFile:61)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at org.apache.shiro.web.servlet.AdviceFilter.executeChain(SourceFile:108)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(SourceFile:137)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(SourceFile:125)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(SourceFile:66)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(SourceFile:118)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(SourceFile:66)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at org.apache.shiro.web.servlet.AdviceFilter.executeChain(SourceFile:108)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(SourceFile:137)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(SourceFile:125)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(SourceFile:66)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at org.apache.shiro.web.servlet.AdviceFilter.executeChain(SourceFile:108)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(SourceFile:137)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(SourceFile:125)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(SourceFile:66)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(SourceFile:449)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(SourceFile:365)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at org.apache.shiro.subject.support.SubjectCallable.doCall(SourceFile:90)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at org.apache.shiro.subject.support.SubjectCallable.call(SourceFile:83)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
org.apache.shiro.subject.support.DelegatingSubject.execute(SourceFile:383)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(SourceFile:362)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(SourceFile:125)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(SourceFile:1419)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
com.sonatype.insight.brain.security.TraceMethodBlockFilter.doFilter(SourceFile:48)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(SourceFile:1419)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at com.yammer.dropwizard.servlets.ThreadNameFilter.doFilter(SourceFile:29)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(SourceFile:1419)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at org.eclipse.jetty.servlet.ServletHandler.doHandle(SourceFile:455)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(SourceFile:1075)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at org.eclipse.jetty.servlet.ServletHandler.doScope(SourceFile:384)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
org.eclipse.jetty.server.handler.ContextHandler.doScope(SourceFile:1009)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at org.eclipse.jetty.server.handler.ScopedHandler.handle(SourceFile:135)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at org.eclipse.jetty.server.handler.HandlerWrapper.handle(SourceFile:116)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at com.yammer.metrics.jetty.InstrumentedHandler.handle(SourceFile:200)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at org.eclipse.jetty.server.handler.GzipHandler.handle(SourceFile:264)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at com.yammer.dropwizard.jetty.BiDiGzipHandler.handle(SourceFile:123)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
org.eclipse.jetty.server.handler.HandlerCollection.handle(SourceFile:154)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at org.eclipse.jetty.server.handler.HandlerWrapper.handle(SourceFile:116)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at org.eclipse.jetty.server.Server.handle(SourceFile:368)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(SourceFile:489)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at org.eclipse.jetty.server.AbstractHttpConnection.content(SourceFile:953)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(SourceFile:1014)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at org.eclipse.jetty.http.HttpParser.parseNext(SourceFile:861)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at org.eclipse.jetty.http.HttpParser.parseAvailable(SourceFile:240)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at org.eclipse.jetty.server.AsyncHttpConnection.handle(SourceFile:82)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SourceFile:628)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SourceFile:52)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(SourceFile:608)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(SourceFile:543)
[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
2016-06-03 02:36:04,149+1000 DEBUG [qtp761509273-12469 - PUT
/rest/ci/scan/treasury-web-deployable] jenkins-server
com.sonatype.insight.brain.hds.HdsClient - Completed HDS request in 9381 ms.
2016-06-03 02:36:04,151+1000 ERROR [qtp761509273-12469 - PUT
/rest/ci/scan/treasury-web-deployable] jenkins-server
com.sonatype.insight.jaxrs.error.ErrorResponseGenerator - The request to
Sonatype HDS failed, please retry in a bit.
com.sonatype.insight.error.exception.BadGatewayException: The request to
Sonatype HDS failed, please retry in a bit.
 at com.sonatype.insight.brain.hds.HdsClient.execute(SourceFile:391)
~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at com.sonatype.insight.brain.hds.HdsClient.execute(SourceFile:329)
~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at com.sonatype.insight.brain.hds.HdsClient.getResponse(SourceFile:132)
~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at com.sonatype.insight.brain.hds.HdsClient.get(SourceFile:178)
~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at com.sonatype.insight.brain.hds.ScanUploader.upload(SourceFile:69)
~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at com.sonatype.insight.brain.hds.CIResource.putScan(SourceFile:50)
~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
org.apache.shiro.guice.aop.AopAllianceMethodInvocationAdapter.proceed(SourceFile:49)
~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
com.sonatype.insight.brain.security.AuthorizeMethodInterceptor.invoke(SourceFile:62)
~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
 at
org.apache.shiro.guice.aop.AopAllianceMethodInterceptorAdapter.invoke(SourceFile:36)
~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]

-- 
Cheers,
Mark Rekveld

Re: Help/Advise debugging SocketException in BHttpConnectionBase.fillInputBuffer()

Posted by Mark Rekveld <mr...@sonatype.com>.
Hi Oleg,

Thanks for your quick reply.
I'll try upgrading to see if that resolves this issue,

On Wed, Jun 22, 2016 at 5:23 PM Oleg Kalnichevski <ol...@apache.org> wrote:

> On Wed, 2016-06-22 at 13:00 +0200, Mark Rekveld wrote:
> > Hi,
> >
> > I would like to ask for your assistance to resolve an issue that we are
> > facing that has the nature of being a race condition.
> >
> > The problem is that a SocketException is thrown in the finally clause of
> > BHttpConnectionBase.fillInputBuffer() [
> >
> https://github.com/apache/httpcore/blob/4.3.x/httpcore/src/main/java/org/apache/http/impl/BHttpConnectionBase.java#L326
> > ]
> >
> > We use 4.3.6 of the HttpClient with 4.3.3 of HttpCore, and have a custom
> > extension of org.apache.http.protocol.HttpRequestExecutor that sends the
> > request entity asynchronously and immediately try to read the response
> from
> > the sever.
> >
>
> We can only adequately support two branches of HttpClient: stable
> (currently 4.5.x) and dev (currently 5.0-alpha)
>
> Please consider upgrading to HttpClient 4.5. There have been numerous
> changes since 4.3.6.
>
> Oleg
>
> > The exception occurs within this custom HttpRequestExecutor when the
> > conn.isResponseAvailable(1000 * 5) call is made in this code snippet. See
> > stack trace below
> >
> > private void sendRequestEntity(final HttpEntityEnclosingRequest req)
> throws
> > IOException {
> >   Thread entitySender = new Thread(getClass().getSimpleName() + "-" +
> > System.currentTimeMillis())
> >   {
> >     @Override
> >     public void run() {
> >       try {
> >         conn.sendRequestEntity(req);
> >         conn.flush();
> >         context.setAttribute(HttpCoreContext.HTTP_REQ_SENT,
> Boolean.TRUE);
> >       }
> >       catch (Throwable t) {
> >         error = t;
> >       }
> >     }
> >   };
> >   entitySender.start();
> >
> >   while (entitySender.isAlive() && !conn.isResponseAvailable(1000 * 5)) {
> >     /*
> >      * Wait here until either we have fully uploaded the request entity
> or
> > the server sends a response. If
> >      * we were to return immediately and called doReceiveResponse() right
> > away we would provoke socket
> >      * timeouts.
> >      */
> >   }
> > }
> >
> > The full stack trace:
> >
> > 2016-06-03 02:36:04,149+1000 ERROR [qtp761509273-12469 - PUT
> > /rest/ci/scan/treasury-web-deployable] jenkins-server
> > com.sonatype.insight.brain.hds.HdsClient - Socket is closed
> > java.net.SocketException: Socket is closed
> >  at java.net.Socket.setSoTimeout(Socket.java:1137) ~[na:1.8.0_91]
> >  at
> >
> sun.security.ssl.BaseSSLSocketImpl.setSoTimeout(BaseSSLSocketImpl.java:633)
> > ~[na:1.8.0_91]
> >  at sun.security.ssl.SSLSocketImpl.setSoTimeout(SSLSocketImpl.java:2526)
> > ~[na:1.8.0_91]
> >  at
> > org.apache.http.impl.BHttpConnectionBase.fillInputBuffer(SourceFile:326)
> > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at org.apache.http.impl.BHttpConnectionBase.awaitInput(SourceFile:334)
> > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> org.apache.http.impl.DefaultBHttpClientConnection.isResponseAvailable(SourceFile:131)
> > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> > org.apache.http.impl.conn.CPoolProxy.isResponseAvailable(SourceFile:141)
> > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> com.sonatype.insight.client.utils.HttpRequestExecutor$HttpExchange.sendRequestEntity(SourceFile:201)
> > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> com.sonatype.insight.client.utils.HttpRequestExecutor$HttpExchange.doSendRequest(SourceFile:177)
> > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> com.sonatype.insight.client.utils.HttpRequestExecutor$HttpExchange.perform(SourceFile:108)
> > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> com.sonatype.insight.client.utils.HttpRequestExecutor.execute(SourceFile:53)
> > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at org.apache.http.impl.execchain.MainClientExec.execute(SourceFile:254)
> > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at org.apache.http.impl.execchain.ProtocolExec.execute(SourceFile:195)
> > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at org.apache.http.impl.execchain.RetryExec.execute(SourceFile:86)
> > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at org.apache.http.impl.execchain.RedirectExec.execute(SourceFile:108)
> > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> > org.apache.http.impl.client.InternalHttpClient.doExecute(SourceFile:184)
> > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> org.apache.http.impl.client.CloseableHttpClient.execute(SourceFile:82)
> > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> org.apache.http.impl.client.CloseableHttpClient.execute(SourceFile:106)
> > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> org.apache.http.impl.client.CloseableHttpClient.execute(SourceFile:57)
> > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at com.sonatype.insight.brain.hds.HdsClient.execute(SourceFile:380)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at com.sonatype.insight.brain.hds.HdsClient.execute(SourceFile:329)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at com.sonatype.insight.brain.hds.HdsClient.getResponse(SourceFile:132)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at com.sonatype.insight.brain.hds.HdsClient.get(SourceFile:178)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at com.sonatype.insight.brain.hds.ScanUploader.upload(SourceFile:69)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at com.sonatype.insight.brain.hds.CIResource.putScan(SourceFile:50)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> com.sonatype.insight.brain.hds.CIResource$$EnhancerByGuice$$88667ab8.CGLIB$putScan$0(<generated>)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> com.sonatype.insight.brain.hds.CIResource$$EnhancerByGuice$$88667ab8$$FastClassByGuice$$99131cf1.invoke(<generated>)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(SourceFile:228)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(SourceFile:75)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> org.apache.shiro.guice.aop.AopAllianceMethodInvocationAdapter.proceed(SourceFile:49)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> com.sonatype.insight.brain.security.AuthorizeMethodInterceptor.invoke(SourceFile:62)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> org.apache.shiro.guice.aop.AopAllianceMethodInterceptorAdapter.invoke(SourceFile:36)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(SourceFile:75)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> com.google.inject.internal.InterceptorStackCallback.intercept(SourceFile:55)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> com.sonatype.insight.brain.hds.CIResource$$EnhancerByGuice$$88667ab8.putScan(<generated>)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at sun.reflect.GeneratedMethodAccessor196.invoke(Unknown Source)
> ~[na:na]
> >  at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > ~[na:1.8.0_91]
> >  at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_91]
> >  at
> >
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(SourceFile:60)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(SourceFile:185)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(SourceFile:75)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> com.yammer.dropwizard.jersey.OptionalResourceMethodDispatchAdapter$OptionalRequestDispatcher.dispatch(SourceFile:37)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(SourceFile:302)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(SourceFile:147)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(SourceFile:108)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(SourceFile:147)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(SourceFile:84)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(SourceFile:1511)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(SourceFile:1442)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(SourceFile:1391)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(SourceFile:1381)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> > com.sun.jersey.spi.container.servlet.WebComponent.service(SourceFile:416)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> com.sun.jersey.spi.container.servlet.ServletContainer.service(SourceFile:538)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> com.sun.jersey.spi.container.servlet.ServletContainer.service(SourceFile:716)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at javax.servlet.http.HttpServlet.service(SourceFile:848)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at org.eclipse.jetty.servlet.ServletHolder.handle(SourceFile:669)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(SourceFile:1448)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> com.sonatype.insight.brain.security.AuthenticationLoggingFilter.doFilter(SourceFile:53)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(SourceFile:1419)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(SourceFile:61)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> org.apache.shiro.web.servlet.AdviceFilter.executeChain(SourceFile:108)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(SourceFile:137)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(SourceFile:125)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(SourceFile:66)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(SourceFile:118)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(SourceFile:66)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> org.apache.shiro.web.servlet.AdviceFilter.executeChain(SourceFile:108)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(SourceFile:137)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(SourceFile:125)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(SourceFile:66)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> org.apache.shiro.web.servlet.AdviceFilter.executeChain(SourceFile:108)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(SourceFile:137)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(SourceFile:125)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(SourceFile:66)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(SourceFile:449)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(SourceFile:365)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> org.apache.shiro.subject.support.SubjectCallable.doCall(SourceFile:90)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at org.apache.shiro.subject.support.SubjectCallable.call(SourceFile:83)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> org.apache.shiro.subject.support.DelegatingSubject.execute(SourceFile:383)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(SourceFile:362)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(SourceFile:125)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(SourceFile:1419)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> com.sonatype.insight.brain.security.TraceMethodBlockFilter.doFilter(SourceFile:48)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(SourceFile:1419)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> com.yammer.dropwizard.servlets.ThreadNameFilter.doFilter(SourceFile:29)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(SourceFile:1419)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at org.eclipse.jetty.servlet.ServletHandler.doHandle(SourceFile:455)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> > org.eclipse.jetty.server.handler.ContextHandler.doHandle(SourceFile:1075)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at org.eclipse.jetty.servlet.ServletHandler.doScope(SourceFile:384)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> > org.eclipse.jetty.server.handler.ContextHandler.doScope(SourceFile:1009)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at org.eclipse.jetty.server.handler.ScopedHandler.handle(SourceFile:135)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(SourceFile:116)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at com.yammer.metrics.jetty.InstrumentedHandler.handle(SourceFile:200)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at org.eclipse.jetty.server.handler.GzipHandler.handle(SourceFile:264)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at com.yammer.dropwizard.jetty.BiDiGzipHandler.handle(SourceFile:123)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> > org.eclipse.jetty.server.handler.HandlerCollection.handle(SourceFile:154)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(SourceFile:116)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at org.eclipse.jetty.server.Server.handle(SourceFile:368)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(SourceFile:489)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> org.eclipse.jetty.server.AbstractHttpConnection.content(SourceFile:953)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(SourceFile:1014)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at org.eclipse.jetty.http.HttpParser.parseNext(SourceFile:861)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at org.eclipse.jetty.http.HttpParser.parseAvailable(SourceFile:240)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at org.eclipse.jetty.server.AsyncHttpConnection.handle(SourceFile:82)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SourceFile:628)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SourceFile:52)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(SourceFile:608)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(SourceFile:543)
> > [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> > 2016-06-03 02:36:04,149+1000 DEBUG [qtp761509273-12469 - PUT
> > /rest/ci/scan/treasury-web-deployable] jenkins-server
> > com.sonatype.insight.brain.hds.HdsClient - Completed HDS request in 9381
> ms.
> > 2016-06-03 02:36:04,151+1000 ERROR [qtp761509273-12469 - PUT
> > /rest/ci/scan/treasury-web-deployable] jenkins-server
> > com.sonatype.insight.jaxrs.error.ErrorResponseGenerator - The request to
> > Sonatype HDS failed, please retry in a bit.
> > com.sonatype.insight.error.exception.BadGatewayException: The request to
> > Sonatype HDS failed, please retry in a bit.
> >  at com.sonatype.insight.brain.hds.HdsClient.execute(SourceFile:391)
> > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at com.sonatype.insight.brain.hds.HdsClient.execute(SourceFile:329)
> > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at com.sonatype.insight.brain.hds.HdsClient.getResponse(SourceFile:132)
> > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at com.sonatype.insight.brain.hds.HdsClient.get(SourceFile:178)
> > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at com.sonatype.insight.brain.hds.ScanUploader.upload(SourceFile:69)
> > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at com.sonatype.insight.brain.hds.CIResource.putScan(SourceFile:50)
> > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> org.apache.shiro.guice.aop.AopAllianceMethodInvocationAdapter.proceed(SourceFile:49)
> > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> com.sonatype.insight.brain.security.AuthorizeMethodInterceptor.invoke(SourceFile:62)
> > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >  at
> >
> org.apache.shiro.guice.aop.AopAllianceMethodInterceptorAdapter.invoke(SourceFile:36)
> > ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
> --

Cheers,
Mark Rekveld

Re: Help/Advise debugging SocketException in BHttpConnectionBase.fillInputBuffer()

Posted by Oleg Kalnichevski <ol...@apache.org>.
On Wed, 2016-06-22 at 13:00 +0200, Mark Rekveld wrote:
> Hi,
> 
> I would like to ask for your assistance to resolve an issue that we are
> facing that has the nature of being a race condition.
> 
> The problem is that a SocketException is thrown in the finally clause of
> BHttpConnectionBase.fillInputBuffer() [
> https://github.com/apache/httpcore/blob/4.3.x/httpcore/src/main/java/org/apache/http/impl/BHttpConnectionBase.java#L326
> ]
> 
> We use 4.3.6 of the HttpClient with 4.3.3 of HttpCore, and have a custom
> extension of org.apache.http.protocol.HttpRequestExecutor that sends the
> request entity asynchronously and immediately try to read the response from
> the sever.
> 

We can only adequately support two branches of HttpClient: stable
(currently 4.5.x) and dev (currently 5.0-alpha)

Please consider upgrading to HttpClient 4.5. There have been numerous
changes since 4.3.6.  

Oleg

> The exception occurs within this custom HttpRequestExecutor when the
> conn.isResponseAvailable(1000 * 5) call is made in this code snippet. See
> stack trace below
> 
> private void sendRequestEntity(final HttpEntityEnclosingRequest req) throws
> IOException {
>   Thread entitySender = new Thread(getClass().getSimpleName() + "-" +
> System.currentTimeMillis())
>   {
>     @Override
>     public void run() {
>       try {
>         conn.sendRequestEntity(req);
>         conn.flush();
>         context.setAttribute(HttpCoreContext.HTTP_REQ_SENT, Boolean.TRUE);
>       }
>       catch (Throwable t) {
>         error = t;
>       }
>     }
>   };
>   entitySender.start();
> 
>   while (entitySender.isAlive() && !conn.isResponseAvailable(1000 * 5)) {
>     /*
>      * Wait here until either we have fully uploaded the request entity or
> the server sends a response. If
>      * we were to return immediately and called doReceiveResponse() right
> away we would provoke socket
>      * timeouts.
>      */
>   }
> }
> 
> The full stack trace:
> 
> 2016-06-03 02:36:04,149+1000 ERROR [qtp761509273-12469 - PUT
> /rest/ci/scan/treasury-web-deployable] jenkins-server
> com.sonatype.insight.brain.hds.HdsClient - Socket is closed
> java.net.SocketException: Socket is closed
>  at java.net.Socket.setSoTimeout(Socket.java:1137) ~[na:1.8.0_91]
>  at
> sun.security.ssl.BaseSSLSocketImpl.setSoTimeout(BaseSSLSocketImpl.java:633)
> ~[na:1.8.0_91]
>  at sun.security.ssl.SSLSocketImpl.setSoTimeout(SSLSocketImpl.java:2526)
> ~[na:1.8.0_91]
>  at
> org.apache.http.impl.BHttpConnectionBase.fillInputBuffer(SourceFile:326)
> ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at org.apache.http.impl.BHttpConnectionBase.awaitInput(SourceFile:334)
> ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> org.apache.http.impl.DefaultBHttpClientConnection.isResponseAvailable(SourceFile:131)
> ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> org.apache.http.impl.conn.CPoolProxy.isResponseAvailable(SourceFile:141)
> ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> com.sonatype.insight.client.utils.HttpRequestExecutor$HttpExchange.sendRequestEntity(SourceFile:201)
> ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> com.sonatype.insight.client.utils.HttpRequestExecutor$HttpExchange.doSendRequest(SourceFile:177)
> ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> com.sonatype.insight.client.utils.HttpRequestExecutor$HttpExchange.perform(SourceFile:108)
> ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> com.sonatype.insight.client.utils.HttpRequestExecutor.execute(SourceFile:53)
> ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at org.apache.http.impl.execchain.MainClientExec.execute(SourceFile:254)
> ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at org.apache.http.impl.execchain.ProtocolExec.execute(SourceFile:195)
> ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at org.apache.http.impl.execchain.RetryExec.execute(SourceFile:86)
> ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at org.apache.http.impl.execchain.RedirectExec.execute(SourceFile:108)
> ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> org.apache.http.impl.client.InternalHttpClient.doExecute(SourceFile:184)
> ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at org.apache.http.impl.client.CloseableHttpClient.execute(SourceFile:82)
> ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at org.apache.http.impl.client.CloseableHttpClient.execute(SourceFile:106)
> ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at org.apache.http.impl.client.CloseableHttpClient.execute(SourceFile:57)
> ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at com.sonatype.insight.brain.hds.HdsClient.execute(SourceFile:380)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at com.sonatype.insight.brain.hds.HdsClient.execute(SourceFile:329)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at com.sonatype.insight.brain.hds.HdsClient.getResponse(SourceFile:132)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at com.sonatype.insight.brain.hds.HdsClient.get(SourceFile:178)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at com.sonatype.insight.brain.hds.ScanUploader.upload(SourceFile:69)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at com.sonatype.insight.brain.hds.CIResource.putScan(SourceFile:50)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> com.sonatype.insight.brain.hds.CIResource$$EnhancerByGuice$$88667ab8.CGLIB$putScan$0(<generated>)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> com.sonatype.insight.brain.hds.CIResource$$EnhancerByGuice$$88667ab8$$FastClassByGuice$$99131cf1.invoke(<generated>)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(SourceFile:228)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(SourceFile:75)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> org.apache.shiro.guice.aop.AopAllianceMethodInvocationAdapter.proceed(SourceFile:49)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> com.sonatype.insight.brain.security.AuthorizeMethodInterceptor.invoke(SourceFile:62)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> org.apache.shiro.guice.aop.AopAllianceMethodInterceptorAdapter.invoke(SourceFile:36)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(SourceFile:75)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> com.google.inject.internal.InterceptorStackCallback.intercept(SourceFile:55)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> com.sonatype.insight.brain.hds.CIResource$$EnhancerByGuice$$88667ab8.putScan(<generated>)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at sun.reflect.GeneratedMethodAccessor196.invoke(Unknown Source) ~[na:na]
>  at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.8.0_91]
>  at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_91]
>  at
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(SourceFile:60)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(SourceFile:185)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(SourceFile:75)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> com.yammer.dropwizard.jersey.OptionalResourceMethodDispatchAdapter$OptionalRequestDispatcher.dispatch(SourceFile:37)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(SourceFile:302)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(SourceFile:147)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(SourceFile:108)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(SourceFile:147)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(SourceFile:84)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(SourceFile:1511)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(SourceFile:1442)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(SourceFile:1391)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(SourceFile:1381)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> com.sun.jersey.spi.container.servlet.WebComponent.service(SourceFile:416)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(SourceFile:538)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(SourceFile:716)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at javax.servlet.http.HttpServlet.service(SourceFile:848)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at org.eclipse.jetty.servlet.ServletHolder.handle(SourceFile:669)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(SourceFile:1448)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> com.sonatype.insight.brain.security.AuthenticationLoggingFilter.doFilter(SourceFile:53)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(SourceFile:1419)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(SourceFile:61)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at org.apache.shiro.web.servlet.AdviceFilter.executeChain(SourceFile:108)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(SourceFile:137)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(SourceFile:125)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(SourceFile:66)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(SourceFile:118)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(SourceFile:66)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at org.apache.shiro.web.servlet.AdviceFilter.executeChain(SourceFile:108)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(SourceFile:137)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(SourceFile:125)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(SourceFile:66)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at org.apache.shiro.web.servlet.AdviceFilter.executeChain(SourceFile:108)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(SourceFile:137)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(SourceFile:125)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(SourceFile:66)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(SourceFile:449)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(SourceFile:365)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at org.apache.shiro.subject.support.SubjectCallable.doCall(SourceFile:90)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at org.apache.shiro.subject.support.SubjectCallable.call(SourceFile:83)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> org.apache.shiro.subject.support.DelegatingSubject.execute(SourceFile:383)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(SourceFile:362)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(SourceFile:125)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(SourceFile:1419)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> com.sonatype.insight.brain.security.TraceMethodBlockFilter.doFilter(SourceFile:48)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(SourceFile:1419)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at com.yammer.dropwizard.servlets.ThreadNameFilter.doFilter(SourceFile:29)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(SourceFile:1419)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at org.eclipse.jetty.servlet.ServletHandler.doHandle(SourceFile:455)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(SourceFile:1075)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at org.eclipse.jetty.servlet.ServletHandler.doScope(SourceFile:384)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(SourceFile:1009)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at org.eclipse.jetty.server.handler.ScopedHandler.handle(SourceFile:135)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(SourceFile:116)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at com.yammer.metrics.jetty.InstrumentedHandler.handle(SourceFile:200)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at org.eclipse.jetty.server.handler.GzipHandler.handle(SourceFile:264)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at com.yammer.dropwizard.jetty.BiDiGzipHandler.handle(SourceFile:123)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(SourceFile:154)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(SourceFile:116)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at org.eclipse.jetty.server.Server.handle(SourceFile:368)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(SourceFile:489)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at org.eclipse.jetty.server.AbstractHttpConnection.content(SourceFile:953)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(SourceFile:1014)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at org.eclipse.jetty.http.HttpParser.parseNext(SourceFile:861)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at org.eclipse.jetty.http.HttpParser.parseAvailable(SourceFile:240)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at org.eclipse.jetty.server.AsyncHttpConnection.handle(SourceFile:82)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SourceFile:628)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SourceFile:52)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(SourceFile:608)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(SourceFile:543)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> 2016-06-03 02:36:04,149+1000 DEBUG [qtp761509273-12469 - PUT
> /rest/ci/scan/treasury-web-deployable] jenkins-server
> com.sonatype.insight.brain.hds.HdsClient - Completed HDS request in 9381 ms.
> 2016-06-03 02:36:04,151+1000 ERROR [qtp761509273-12469 - PUT
> /rest/ci/scan/treasury-web-deployable] jenkins-server
> com.sonatype.insight.jaxrs.error.ErrorResponseGenerator - The request to
> Sonatype HDS failed, please retry in a bit.
> com.sonatype.insight.error.exception.BadGatewayException: The request to
> Sonatype HDS failed, please retry in a bit.
>  at com.sonatype.insight.brain.hds.HdsClient.execute(SourceFile:391)
> ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at com.sonatype.insight.brain.hds.HdsClient.execute(SourceFile:329)
> ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at com.sonatype.insight.brain.hds.HdsClient.getResponse(SourceFile:132)
> ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at com.sonatype.insight.brain.hds.HdsClient.get(SourceFile:178)
> ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at com.sonatype.insight.brain.hds.ScanUploader.upload(SourceFile:69)
> ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at com.sonatype.insight.brain.hds.CIResource.putScan(SourceFile:50)
> ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> org.apache.shiro.guice.aop.AopAllianceMethodInvocationAdapter.proceed(SourceFile:49)
> ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> com.sonatype.insight.brain.security.AuthorizeMethodInterceptor.invoke(SourceFile:62)
> ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> org.apache.shiro.guice.aop.AopAllianceMethodInterceptorAdapter.invoke(SourceFile:36)
> ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
> 



---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org