[jira] [Comment Edited] (CXF-7122) Infinite loop due to AsyncHTTPConduit read timeout with exhausted connection pool

["William Montaz (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CXF-7122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15644021#comment-15644021 ] 

William Montaz edited comment on CXF-7122 at 11/7/16 12:18 PM:
---------------------------------------------------------------

Hi Freeman,

use your test class, use testAsyncTimeout, debug and set a breakpoint on the getHttpresponse first line (while(httpresponse == null). You will see that even if you consider response timeout in the timer task, the request is still being sent and response is given back to the client. Ok your exception in the timer task hides it but from a client/server perspective everything happened in a nice fashion. If your client timeout, you should close the connection.

Another thing that does not work, put testTimeoutAsync in this fashion

    @Test
    public void testTimeoutAsync() throws Exception {
        updateAddressPort(g, PORT);
        HTTPConduit c = (HTTPConduit)ClientProxy.getClient(g).getConduit();
        c.getClient().setReceiveTimeout(3000);
        try {
            long start = System.currentTimeMillis();
            Response<GreetMeLaterResponse> future = g.greetMeLaterAsync(-5000L);
            try{
                future.get();
            } catch (Exception e){
                System.out.println("First get timeout");
            }
            long firstGet = System.currentTimeMillis();
            System.out.println("First get took " + (firstGet - start) + " ms");
            Thread.sleep(1000);
            long startSecond = System.currentTimeMillis();
            future = g.greetMeLaterAsync(1000);
            try{
                future.get();
            } catch (Exception e){
                System.out.println("Second get timeout");
            }
            long secondGet = System.currentTimeMillis();
            System.out.println("Second get took " + (secondGet - startSecond) + " ms");
            Thread.sleep(30000);
            future.get();
            fail();
        } catch (Exception ex) {
            //expected!!!
            ex.printStackTrace();
        }
    }

We agree that second request should not timeout. The first one took 3 sec and the second one should only take 1 second, but since the connection is not really closed, it is still not available and thus request two will timeout because it is waiting for the connection still held by first request.


was (Author: willymontaz):
Hi Freeman,

use your test class, use testAsyncTimeout, debug and set a breakpoint on the getHttpresponse first line (while(httpresponse == null). You will see that even if you consider response timeout in the timer task, the request is still being sent and response is given back to the client. Ok your exception in the timer task hides it but from a client/server perspective everything happened in a nice fashion. If your client timeout, you should close the connection.

Another thing that does not work, put testTimeoutAsync in this fashion


> Infinite loop due to AsyncHTTPConduit read timeout with exhausted connection pool
> ---------------------------------------------------------------------------------
>
>                 Key: CXF-7122
>                 URL: https://issues.apache.org/jira/browse/CXF-7122
>             Project: CXF
>          Issue Type: Bug
>          Components: Transports
>            Reporter: William Montaz
>            Assignee: Freeman Fang
>            Priority: Critical
>             Fix For: 3.2.0, 3.1.9
>
>         Attachments: AsyncHTTPConduitTest.java
>
>
> Using AsyncHTTPConduit, when the underlying connection pool gets exhausted, requests waiting for a connection will lead to an infinite loop if they reach receive timeout.
> The problem occured on all versions of CXF above 3.0.5 (we did not tested other ones). 
> Let's imagine a backend that's broken and leads to timeout for all requests.
> When handling requests, the cxf worker thread will eventually go in wait state (AsyncHTTPConduit:618), with a timeout that matches the HTTPClientPolicy.setReceiveTimeout() value, waiting for the NIO stack to complete and call notifyAll via responseCallback (AsyncHTTPConduit:455). 
> The timeout on the wait is the big problem :
> With our broken backend, the connection pool is exhausted waiting for other requests to timeout. When a new request is made by cxf against this backend, after timeout time this will happen :
>  - on the one side the reactor threads will get a connection from the pool and try to write to the output stream. Waiting in the pool is not considered as receive timeout.
>  - on the other side the cxf worker thread will wake up (because of the timedout wait), and shutdown SharedOutputBuffer and SharedInputBuffer (AsyncHTTPClient:624)
>  - reactor threads will go to infinite loop because they will try to produceContent from a shutdown buffer (SharedOutputBuffer:120)
>  
>  From there, application recovery is compromised.
>   
>  To fix that, timeout should be handled only via the client callback (AsyncHTTPConduit:463).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)