You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cxf.apache.org by Mandy Warren <ma...@gmail.com> on 2014/12/12 15:23:58 UTC
Async client Http transport and keep-alive timeouts
Hi,
We are using cxf version 2.7.12 and are using Asynchronous Client HTTP Transport to send a request to an external service. This external service runs on an Apache server and is configured as follows:-
Keep-Alive: timeout=2, max=100
Connection: Keep-Alive
i.e. the external server will close the connection after 2 seconds of idle time.
Under heavy load, every so often we see an attempt to send a request on a connection fail because at the exact same time, the external server has closed the connection. I would then expect CXF to retry the sending of the request again but it doesn't, the call just fails. Here is some relevant debug to indicate the issue:
2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> POST /Service/ws/ Controls/1.0/GetStuff HTTP/1.1]
2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> txId: 1234]
2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> User-Agent: Apache CXF 2.7.12]
2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> SOAPAction: executeService]
2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> Accept: */*]
2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> Content-Length: 1547]
2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> Content-Type: application/xml]
2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> Host: a-b.x.com:80]
2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.reactor.IOSessionImpl [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 xxx.x.xx.xx:xxxxx<->yyy.y.yyy.yy:yy[ACTIVE][rw:w]: Event set [w]]
2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.reactor.IOSessionImpl [txId=|client=|entryService=|callingService=] [] [http-outgoing-268xxx.x.xx.xx:xxxxx<->yyy.y.yyy.yy:yy[ACTIVE][rw:rw]: -1 bytes read]
2014-12-09 16:10:29,091 [I/O dispatcher 2] DEBUG org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduitFactory$4 [txId=|client=|entryService=|callingService=] [] [[exchange: 9268] aborting connection [id:268][route:{}->http://a-b.x.com:80][state:null]]
2014-12-09 16:10:29,091 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.reactor.IOSessionImpl [txId=|client=|entryService=|callingService=] [] [http-outgoing-268xxx.x.xx.xx:xxxxx<->yyy.y.yyy.yy:yy[ACTIVE][rw:rw]: Shutdown]
2014-12-09 16:10:29,091 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.reactor.IOSessionImpl [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 0.0.0.0:11111<->yyy.y.yyy.yy:yy[CLOSED][]: Shutdown]
2014-12-09 16:10:29,091 [I/O dispatcher 2] DEBUG org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduitFactory$3 [txId=|client=|entryService=|callingService=] [] [Connection released: [id: 268][route: {}->http://a-b.x.com:80][total kept alive: 0; route allocated: 1 of 1000; total allocated: 1 of 5000]
The only thing I've noticed which could be relevant is that when we send a request to the external service, CXF doesn't explicitly send the Keep-alive headers (as it is implicit through the use of HTTP 1.1) which means the external service does not send back the explicit Keep-Alive: timeout=2, max=100 header.
Please could someone confirm whether or not you would expect the async transport to retry sending the request when a connection is closed due to a keep-alive idle timeout? Am I missing some configuration to allow this?
Many thanks
Mandy
Sent from a mobile device
Re: Async client Http transport and keep-alive timeouts
Posted by Sergey Beryozkin <sb...@gmail.com>.
Hi Mandy
On 21/12/14 22:40, Mandy Warren wrote:
> Hi Sergey, I tried using HTTPUrlConnection for my synchronous calls and it worked very nicely, I did a tcpdump and could see it respected the keep alive setting and retried with a new connection if a connection was unexpectedly closed by the server.
>
> I would however like the option to also use async transport as we are looking at using the async/Future methods on WebClient. I do feel there is a bug in it but before I raise a jira it would be good to get Dan/Aki or Oleg's opinions.
I'm not as polite as you so I've created a JIRA issue :-)
https://issues.apache.org/jira/browse/CXF-6169
It can be a tricky issue to fix right and it might be either CXF ir Http
Client issue, but you've spent a lot to time to analyze it so it is
worth starting tracking it
>
> I am also happy to send the tcpdump output for the async transport test showing the FIN from the server and subsequent processing if this helps. Note I have also tried explicitly setting the Connection: keep-alive header but still no luck..
>
Please attach the info you have to CXF-6169
Thanks, Sergey
> Many thanks
> Mandy
>
> Sent from my iPad
>
>> On 16 Dec 2014, at 12:36, Sergey Beryozkin <sb...@gmail.com> wrote:
>>
>> Hi Mandy
>>
>> I'd like to ask Dan, Aki or Oleg to comment - hopefully one of HTTP Client experts would read it :-)
>>
>> by the way, what about HTTPUrlConnection - does it work less efficiently ?
>>
>> Thanks, Sergey
>>> On 12/12/14 14:23, Mandy Warren wrote:
>>> Hi,
>>>
>>> We are using cxf version 2.7.12 and are using Asynchronous Client HTTP Transport to send a request to an external service. This external service runs on an Apache server and is configured as follows:-
>>>
>>> Keep-Alive: timeout=2, max=100
>>> Connection: Keep-Alive
>>>
>>> i.e. the external server will close the connection after 2 seconds of idle time.
>>>
>>> Under heavy load, every so often we see an attempt to send a request on a connection fail because at the exact same time, the external server has closed the connection. I would then expect CXF to retry the sending of the request again but it doesn't, the call just fails. Here is some relevant debug to indicate the issue:
>>>
>>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> POST /Service/ws/ Controls/1.0/GetStuff HTTP/1.1]
>>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> txId: 1234]
>>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> User-Agent: Apache CXF 2.7.12]
>>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> SOAPAction: executeService]
>>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> Accept: */*]
>>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> Content-Length: 1547]
>>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> Content-Type: application/xml]
>>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> Host: a-b.x.com:80]
>>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.reactor.IOSessionImpl [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 xxx.x.xx.xx:xxxxx<->yyy.y.yyy.yy:yy[ACTIVE][rw:w]: Event set [w]]
>>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.reactor.IOSessionImpl [txId=|client=|entryService=|callingService=] [] [http-outgoing-268xxx.x.xx.xx:xxxxx<->yyy.y.yyy.yy:yy[ACTIVE][rw:rw]: -1 bytes read]
>>> 2014-12-09 16:10:29,091 [I/O dispatcher 2] DEBUG org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduitFactory$4 [txId=|client=|entryService=|callingService=] [] [[exchange: 9268] aborting connection [id:268][route:{}->http://a-b.x.com:80][state:null]]
>>> 2014-12-09 16:10:29,091 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.reactor.IOSessionImpl [txId=|client=|entryService=|callingService=] [] [http-outgoing-268xxx.x.xx.xx:xxxxx<->yyy.y.yyy.yy:yy[ACTIVE][rw:rw]: Shutdown]
>>> 2014-12-09 16:10:29,091 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.reactor.IOSessionImpl [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 0.0.0.0:11111<->yyy.y.yyy.yy:yy[CLOSED][]: Shutdown]
>>> 2014-12-09 16:10:29,091 [I/O dispatcher 2] DEBUG org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduitFactory$3 [txId=|client=|entryService=|callingService=] [] [Connection released: [id: 268][route: {}->http://a-b.x.com:80][total kept alive: 0; route allocated: 1 of 1000; total allocated: 1 of 5000]
>>>
>>> The only thing I've noticed which could be relevant is that when we send a request to the external service, CXF doesn't explicitly send the Keep-alive headers (as it is implicit through the use of HTTP 1.1) which means the external service does not send back the explicit Keep-Alive: timeout=2, max=100 header.
>>>
>>> Please could someone confirm whether or not you would expect the async transport to retry sending the request when a connection is closed due to a keep-alive idle timeout? Am I missing some configuration to allow this?
>>>
>>> Many thanks
>>> Mandy
>>>
>>>
>>>
>>> Sent from a mobile device
>>>
>>
Re: Async client Http transport and keep-alive timeouts
Posted by Daniel Kulp <dk...@apache.org>.
> On Dec 22, 2014, at 4:11 PM, Mandy Warren <ma...@gmail.com> wrote:
>
> Hi Dan
>
> At the moment our code is based on the 2.7.x branch, we hadn't planned to upgrade to 3 til early next year. However maybe this will give me the incentive to upgrade sooner!
>
> Can I just confirm that if we want to use a later version of httpcore we need to upgrade to 3.0.x? I seem to remember an earlier post where you said you couldn't upgrade 2.7.x due to api incompatibilities?
Yea. There were a BUNCH of changes in the httpcore that we had to take into account for the 3.0.x upgrade.
Dan
>
> Thanks
> Mandy
>
> Sent from a mobile device
>
>> On 22 Dec 2014, at 18:39, Daniel Kulp <dk...@apache.org> wrote:
>>
>>
>> This is likely down in HTTP Client someplace, but I am questioning one thing: if I use wireshark to grab the raw messages, I’m seeing the Connection: Keep-Alive header:
>>
>> POST /SoapContext/SoapPort HTTP/1.1
>> Accept: */*
>> User-Agent: Apache CXF ${project.version}
>> SOAPAction: ""
>> Content-Length: 615
>> Content-Type: text/xml; charset=UTF-8
>> Host: localhost:9001
>> Connection: Keep-Alive
>>
>> At least with the latest code on trunk, but that shouldn’t be any different than what’s on 3.0.x. 2.7.x does use an older version of the HTTP components libraries so there might be some issues there. Could you try updating to 3.0.3 and seeing if the problems still remain?
>>
>>
>> Dan
>>
>>
>>
>>> On Dec 21, 2014, at 5:40 PM, Mandy Warren <ma...@gmail.com> wrote:
>>>
>>> Hi Sergey, I tried using HTTPUrlConnection for my synchronous calls and it worked very nicely, I did a tcpdump and could see it respected the keep alive setting and retried with a new connection if a connection was unexpectedly closed by the server.
>>>
>>> I would however like the option to also use async transport as we are looking at using the async/Future methods on WebClient. I do feel there is a bug in it but before I raise a jira it would be good to get Dan/Aki or Oleg's opinions.
>>>
>>> I am also happy to send the tcpdump output for the async transport test showing the FIN from the server and subsequent processing if this helps. Note I have also tried explicitly setting the Connection: keep-alive header but still no luck..
>>>
>>> Many thanks
>>> Mandy
>>>
>>> Sent from my iPad
>>>
>>>> On 16 Dec 2014, at 12:36, Sergey Beryozkin <sb...@gmail.com> wrote:
>>>>
>>>> Hi Mandy
>>>>
>>>> I'd like to ask Dan, Aki or Oleg to comment - hopefully one of HTTP Client experts would read it :-)
>>>>
>>>> by the way, what about HTTPUrlConnection - does it work less efficiently ?
>>>>
>>>> Thanks, Sergey
>>>>> On 12/12/14 14:23, Mandy Warren wrote:
>>>>> Hi,
>>>>>
>>>>> We are using cxf version 2.7.12 and are using Asynchronous Client HTTP Transport to send a request to an external service. This external service runs on an Apache server and is configured as follows:-
>>>>>
>>>>> Keep-Alive: timeout=2, max=100
>>>>> Connection: Keep-Alive
>>>>>
>>>>> i.e. the external server will close the connection after 2 seconds of idle time.
>>>>>
>>>>> Under heavy load, every so often we see an attempt to send a request on a connection fail because at the exact same time, the external server has closed the connection. I would then expect CXF to retry the sending of the request again but it doesn't, the call just fails. Here is some relevant debug to indicate the issue:
>>>>>
>>>>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> POST /Service/ws/ Controls/1.0/GetStuff HTTP/1.1]
>>>>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> txId: 1234]
>>>>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> User-Agent: Apache CXF 2.7.12]
>>>>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> SOAPAction: executeService]
>>>>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> Accept: */*]
>>>>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> Content-Length: 1547]
>>>>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> Content-Type: application/xml]
>>>>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> Host: a-b.x.com:80]
>>>>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.reactor.IOSessionImpl [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 xxx.x.xx.xx:xxxxx<->yyy.y.yyy.yy:yy[ACTIVE][rw:w]: Event set [w]]
>>>>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.reactor.IOSessionImpl [txId=|client=|entryService=|callingService=] [] [http-outgoing-268xxx.x.xx.xx:xxxxx<->yyy.y.yyy.yy:yy[ACTIVE][rw:rw]: -1 bytes read]
>>>>> 2014-12-09 16:10:29,091 [I/O dispatcher 2] DEBUG org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduitFactory$4 [txId=|client=|entryService=|callingService=] [] [[exchange: 9268] aborting connection [id:268][route:{}->http://a-b.x.com:80][state:null]]
>>>>> 2014-12-09 16:10:29,091 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.reactor.IOSessionImpl [txId=|client=|entryService=|callingService=] [] [http-outgoing-268xxx.x.xx.xx:xxxxx<->yyy.y.yyy.yy:yy[ACTIVE][rw:rw]: Shutdown]
>>>>> 2014-12-09 16:10:29,091 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.reactor.IOSessionImpl [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 0.0.0.0:11111<->yyy.y.yyy.yy:yy[CLOSED][]: Shutdown]
>>>>> 2014-12-09 16:10:29,091 [I/O dispatcher 2] DEBUG org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduitFactory$3 [txId=|client=|entryService=|callingService=] [] [Connection released: [id: 268][route: {}->http://a-b.x.com:80][total kept alive: 0; route allocated: 1 of 1000; total allocated: 1 of 5000]
>>>>>
>>>>> The only thing I've noticed which could be relevant is that when we send a request to the external service, CXF doesn't explicitly send the Keep-alive headers (as it is implicit through the use of HTTP 1.1) which means the external service does not send back the explicit Keep-Alive: timeout=2, max=100 header.
>>>>>
>>>>> Please could someone confirm whether or not you would expect the async transport to retry sending the request when a connection is closed due to a keep-alive idle timeout? Am I missing some configuration to allow this?
>>>>>
>>>>> Many thanks
>>>>> Mandy
>>>>>
>>>>>
>>>>>
>>>>> Sent from a mobile device
>>
>> --
>> Daniel Kulp
>> dkulp@apache.org - http://dankulp.com/blog
>> Talend Community Coder - http://coders.talend.com
>>
--
Daniel Kulp
dkulp@apache.org - http://dankulp.com/blog
Talend Community Coder - http://coders.talend.com
Re: Async client Http transport and keep-alive timeouts
Posted by Mandy Warren <ma...@gmail.com>.
Hi Dan
At the moment our code is based on the 2.7.x branch, we hadn't planned to upgrade to 3 til early next year. However maybe this will give me the incentive to upgrade sooner!
Can I just confirm that if we want to use a later version of httpcore we need to upgrade to 3.0.x? I seem to remember an earlier post where you said you couldn't upgrade 2.7.x due to api incompatibilities?
Thanks
Mandy
Sent from a mobile device
> On 22 Dec 2014, at 18:39, Daniel Kulp <dk...@apache.org> wrote:
>
>
> This is likely down in HTTP Client someplace, but I am questioning one thing: if I use wireshark to grab the raw messages, I’m seeing the Connection: Keep-Alive header:
>
> POST /SoapContext/SoapPort HTTP/1.1
> Accept: */*
> User-Agent: Apache CXF ${project.version}
> SOAPAction: ""
> Content-Length: 615
> Content-Type: text/xml; charset=UTF-8
> Host: localhost:9001
> Connection: Keep-Alive
>
> At least with the latest code on trunk, but that shouldn’t be any different than what’s on 3.0.x. 2.7.x does use an older version of the HTTP components libraries so there might be some issues there. Could you try updating to 3.0.3 and seeing if the problems still remain?
>
>
> Dan
>
>
>
>> On Dec 21, 2014, at 5:40 PM, Mandy Warren <ma...@gmail.com> wrote:
>>
>> Hi Sergey, I tried using HTTPUrlConnection for my synchronous calls and it worked very nicely, I did a tcpdump and could see it respected the keep alive setting and retried with a new connection if a connection was unexpectedly closed by the server.
>>
>> I would however like the option to also use async transport as we are looking at using the async/Future methods on WebClient. I do feel there is a bug in it but before I raise a jira it would be good to get Dan/Aki or Oleg's opinions.
>>
>> I am also happy to send the tcpdump output for the async transport test showing the FIN from the server and subsequent processing if this helps. Note I have also tried explicitly setting the Connection: keep-alive header but still no luck..
>>
>> Many thanks
>> Mandy
>>
>> Sent from my iPad
>>
>>> On 16 Dec 2014, at 12:36, Sergey Beryozkin <sb...@gmail.com> wrote:
>>>
>>> Hi Mandy
>>>
>>> I'd like to ask Dan, Aki or Oleg to comment - hopefully one of HTTP Client experts would read it :-)
>>>
>>> by the way, what about HTTPUrlConnection - does it work less efficiently ?
>>>
>>> Thanks, Sergey
>>>> On 12/12/14 14:23, Mandy Warren wrote:
>>>> Hi,
>>>>
>>>> We are using cxf version 2.7.12 and are using Asynchronous Client HTTP Transport to send a request to an external service. This external service runs on an Apache server and is configured as follows:-
>>>>
>>>> Keep-Alive: timeout=2, max=100
>>>> Connection: Keep-Alive
>>>>
>>>> i.e. the external server will close the connection after 2 seconds of idle time.
>>>>
>>>> Under heavy load, every so often we see an attempt to send a request on a connection fail because at the exact same time, the external server has closed the connection. I would then expect CXF to retry the sending of the request again but it doesn't, the call just fails. Here is some relevant debug to indicate the issue:
>>>>
>>>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> POST /Service/ws/ Controls/1.0/GetStuff HTTP/1.1]
>>>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> txId: 1234]
>>>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> User-Agent: Apache CXF 2.7.12]
>>>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> SOAPAction: executeService]
>>>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> Accept: */*]
>>>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> Content-Length: 1547]
>>>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> Content-Type: application/xml]
>>>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> Host: a-b.x.com:80]
>>>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.reactor.IOSessionImpl [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 xxx.x.xx.xx:xxxxx<->yyy.y.yyy.yy:yy[ACTIVE][rw:w]: Event set [w]]
>>>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.reactor.IOSessionImpl [txId=|client=|entryService=|callingService=] [] [http-outgoing-268xxx.x.xx.xx:xxxxx<->yyy.y.yyy.yy:yy[ACTIVE][rw:rw]: -1 bytes read]
>>>> 2014-12-09 16:10:29,091 [I/O dispatcher 2] DEBUG org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduitFactory$4 [txId=|client=|entryService=|callingService=] [] [[exchange: 9268] aborting connection [id:268][route:{}->http://a-b.x.com:80][state:null]]
>>>> 2014-12-09 16:10:29,091 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.reactor.IOSessionImpl [txId=|client=|entryService=|callingService=] [] [http-outgoing-268xxx.x.xx.xx:xxxxx<->yyy.y.yyy.yy:yy[ACTIVE][rw:rw]: Shutdown]
>>>> 2014-12-09 16:10:29,091 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.reactor.IOSessionImpl [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 0.0.0.0:11111<->yyy.y.yyy.yy:yy[CLOSED][]: Shutdown]
>>>> 2014-12-09 16:10:29,091 [I/O dispatcher 2] DEBUG org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduitFactory$3 [txId=|client=|entryService=|callingService=] [] [Connection released: [id: 268][route: {}->http://a-b.x.com:80][total kept alive: 0; route allocated: 1 of 1000; total allocated: 1 of 5000]
>>>>
>>>> The only thing I've noticed which could be relevant is that when we send a request to the external service, CXF doesn't explicitly send the Keep-alive headers (as it is implicit through the use of HTTP 1.1) which means the external service does not send back the explicit Keep-Alive: timeout=2, max=100 header.
>>>>
>>>> Please could someone confirm whether or not you would expect the async transport to retry sending the request when a connection is closed due to a keep-alive idle timeout? Am I missing some configuration to allow this?
>>>>
>>>> Many thanks
>>>> Mandy
>>>>
>>>>
>>>>
>>>> Sent from a mobile device
>
> --
> Daniel Kulp
> dkulp@apache.org - http://dankulp.com/blog
> Talend Community Coder - http://coders.talend.com
>
Re: Async client Http transport and keep-alive timeouts
Posted by Daniel Kulp <dk...@apache.org>.
This is likely down in HTTP Client someplace, but I am questioning one thing: if I use wireshark to grab the raw messages, I’m seeing the Connection: Keep-Alive header:
POST /SoapContext/SoapPort HTTP/1.1
Accept: */*
User-Agent: Apache CXF ${project.version}
SOAPAction: ""
Content-Length: 615
Content-Type: text/xml; charset=UTF-8
Host: localhost:9001
Connection: Keep-Alive
At least with the latest code on trunk, but that shouldn’t be any different than what’s on 3.0.x. 2.7.x does use an older version of the HTTP components libraries so there might be some issues there. Could you try updating to 3.0.3 and seeing if the problems still remain?
Dan
> On Dec 21, 2014, at 5:40 PM, Mandy Warren <ma...@gmail.com> wrote:
>
> Hi Sergey, I tried using HTTPUrlConnection for my synchronous calls and it worked very nicely, I did a tcpdump and could see it respected the keep alive setting and retried with a new connection if a connection was unexpectedly closed by the server.
>
> I would however like the option to also use async transport as we are looking at using the async/Future methods on WebClient. I do feel there is a bug in it but before I raise a jira it would be good to get Dan/Aki or Oleg's opinions.
>
> I am also happy to send the tcpdump output for the async transport test showing the FIN from the server and subsequent processing if this helps. Note I have also tried explicitly setting the Connection: keep-alive header but still no luck..
>
> Many thanks
> Mandy
>
> Sent from my iPad
>
>> On 16 Dec 2014, at 12:36, Sergey Beryozkin <sb...@gmail.com> wrote:
>>
>> Hi Mandy
>>
>> I'd like to ask Dan, Aki or Oleg to comment - hopefully one of HTTP Client experts would read it :-)
>>
>> by the way, what about HTTPUrlConnection - does it work less efficiently ?
>>
>> Thanks, Sergey
>>> On 12/12/14 14:23, Mandy Warren wrote:
>>> Hi,
>>>
>>> We are using cxf version 2.7.12 and are using Asynchronous Client HTTP Transport to send a request to an external service. This external service runs on an Apache server and is configured as follows:-
>>>
>>> Keep-Alive: timeout=2, max=100
>>> Connection: Keep-Alive
>>>
>>> i.e. the external server will close the connection after 2 seconds of idle time.
>>>
>>> Under heavy load, every so often we see an attempt to send a request on a connection fail because at the exact same time, the external server has closed the connection. I would then expect CXF to retry the sending of the request again but it doesn't, the call just fails. Here is some relevant debug to indicate the issue:
>>>
>>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> POST /Service/ws/ Controls/1.0/GetStuff HTTP/1.1]
>>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> txId: 1234]
>>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> User-Agent: Apache CXF 2.7.12]
>>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> SOAPAction: executeService]
>>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> Accept: */*]
>>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> Content-Length: 1547]
>>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> Content-Type: application/xml]
>>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> Host: a-b.x.com:80]
>>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.reactor.IOSessionImpl [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 xxx.x.xx.xx:xxxxx<->yyy.y.yyy.yy:yy[ACTIVE][rw:w]: Event set [w]]
>>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.reactor.IOSessionImpl [txId=|client=|entryService=|callingService=] [] [http-outgoing-268xxx.x.xx.xx:xxxxx<->yyy.y.yyy.yy:yy[ACTIVE][rw:rw]: -1 bytes read]
>>> 2014-12-09 16:10:29,091 [I/O dispatcher 2] DEBUG org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduitFactory$4 [txId=|client=|entryService=|callingService=] [] [[exchange: 9268] aborting connection [id:268][route:{}->http://a-b.x.com:80][state:null]]
>>> 2014-12-09 16:10:29,091 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.reactor.IOSessionImpl [txId=|client=|entryService=|callingService=] [] [http-outgoing-268xxx.x.xx.xx:xxxxx<->yyy.y.yyy.yy:yy[ACTIVE][rw:rw]: Shutdown]
>>> 2014-12-09 16:10:29,091 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.reactor.IOSessionImpl [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 0.0.0.0:11111<->yyy.y.yyy.yy:yy[CLOSED][]: Shutdown]
>>> 2014-12-09 16:10:29,091 [I/O dispatcher 2] DEBUG org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduitFactory$3 [txId=|client=|entryService=|callingService=] [] [Connection released: [id: 268][route: {}->http://a-b.x.com:80][total kept alive: 0; route allocated: 1 of 1000; total allocated: 1 of 5000]
>>>
>>> The only thing I've noticed which could be relevant is that when we send a request to the external service, CXF doesn't explicitly send the Keep-alive headers (as it is implicit through the use of HTTP 1.1) which means the external service does not send back the explicit Keep-Alive: timeout=2, max=100 header.
>>>
>>> Please could someone confirm whether or not you would expect the async transport to retry sending the request when a connection is closed due to a keep-alive idle timeout? Am I missing some configuration to allow this?
>>>
>>> Many thanks
>>> Mandy
>>>
>>>
>>>
>>> Sent from a mobile device
>>>
>>
--
Daniel Kulp
dkulp@apache.org - http://dankulp.com/blog
Talend Community Coder - http://coders.talend.com
Re: Async client Http transport and keep-alive timeouts
Posted by Mandy Warren <ma...@gmail.com>.
Hi Sergey, I tried using HTTPUrlConnection for my synchronous calls and it worked very nicely, I did a tcpdump and could see it respected the keep alive setting and retried with a new connection if a connection was unexpectedly closed by the server.
I would however like the option to also use async transport as we are looking at using the async/Future methods on WebClient. I do feel there is a bug in it but before I raise a jira it would be good to get Dan/Aki or Oleg's opinions.
I am also happy to send the tcpdump output for the async transport test showing the FIN from the server and subsequent processing if this helps. Note I have also tried explicitly setting the Connection: keep-alive header but still no luck..
Many thanks
Mandy
Sent from my iPad
> On 16 Dec 2014, at 12:36, Sergey Beryozkin <sb...@gmail.com> wrote:
>
> Hi Mandy
>
> I'd like to ask Dan, Aki or Oleg to comment - hopefully one of HTTP Client experts would read it :-)
>
> by the way, what about HTTPUrlConnection - does it work less efficiently ?
>
> Thanks, Sergey
>> On 12/12/14 14:23, Mandy Warren wrote:
>> Hi,
>>
>> We are using cxf version 2.7.12 and are using Asynchronous Client HTTP Transport to send a request to an external service. This external service runs on an Apache server and is configured as follows:-
>>
>> Keep-Alive: timeout=2, max=100
>> Connection: Keep-Alive
>>
>> i.e. the external server will close the connection after 2 seconds of idle time.
>>
>> Under heavy load, every so often we see an attempt to send a request on a connection fail because at the exact same time, the external server has closed the connection. I would then expect CXF to retry the sending of the request again but it doesn't, the call just fails. Here is some relevant debug to indicate the issue:
>>
>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> POST /Service/ws/ Controls/1.0/GetStuff HTTP/1.1]
>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> txId: 1234]
>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> User-Agent: Apache CXF 2.7.12]
>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> SOAPAction: executeService]
>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> Accept: */*]
>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> Content-Length: 1547]
>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> Content-Type: application/xml]
>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> Host: a-b.x.com:80]
>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.reactor.IOSessionImpl [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 xxx.x.xx.xx:xxxxx<->yyy.y.yyy.yy:yy[ACTIVE][rw:w]: Event set [w]]
>> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.reactor.IOSessionImpl [txId=|client=|entryService=|callingService=] [] [http-outgoing-268xxx.x.xx.xx:xxxxx<->yyy.y.yyy.yy:yy[ACTIVE][rw:rw]: -1 bytes read]
>> 2014-12-09 16:10:29,091 [I/O dispatcher 2] DEBUG org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduitFactory$4 [txId=|client=|entryService=|callingService=] [] [[exchange: 9268] aborting connection [id:268][route:{}->http://a-b.x.com:80][state:null]]
>> 2014-12-09 16:10:29,091 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.reactor.IOSessionImpl [txId=|client=|entryService=|callingService=] [] [http-outgoing-268xxx.x.xx.xx:xxxxx<->yyy.y.yyy.yy:yy[ACTIVE][rw:rw]: Shutdown]
>> 2014-12-09 16:10:29,091 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.reactor.IOSessionImpl [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 0.0.0.0:11111<->yyy.y.yyy.yy:yy[CLOSED][]: Shutdown]
>> 2014-12-09 16:10:29,091 [I/O dispatcher 2] DEBUG org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduitFactory$3 [txId=|client=|entryService=|callingService=] [] [Connection released: [id: 268][route: {}->http://a-b.x.com:80][total kept alive: 0; route allocated: 1 of 1000; total allocated: 1 of 5000]
>>
>> The only thing I've noticed which could be relevant is that when we send a request to the external service, CXF doesn't explicitly send the Keep-alive headers (as it is implicit through the use of HTTP 1.1) which means the external service does not send back the explicit Keep-Alive: timeout=2, max=100 header.
>>
>> Please could someone confirm whether or not you would expect the async transport to retry sending the request when a connection is closed due to a keep-alive idle timeout? Am I missing some configuration to allow this?
>>
>> Many thanks
>> Mandy
>>
>>
>>
>> Sent from a mobile device
>>
>
Re: Async client Http transport and keep-alive timeouts
Posted by Sergey Beryozkin <sb...@gmail.com>.
Hi Mandy
I'd like to ask Dan, Aki or Oleg to comment - hopefully one of HTTP
Client experts would read it :-)
by the way, what about HTTPUrlConnection - does it work less efficiently ?
Thanks, Sergey
On 12/12/14 14:23, Mandy Warren wrote:
> Hi,
>
> We are using cxf version 2.7.12 and are using Asynchronous Client HTTP Transport to send a request to an external service. This external service runs on an Apache server and is configured as follows:-
>
> Keep-Alive: timeout=2, max=100
> Connection: Keep-Alive
>
> i.e. the external server will close the connection after 2 seconds of idle time.
>
> Under heavy load, every so often we see an attempt to send a request on a connection fail because at the exact same time, the external server has closed the connection. I would then expect CXF to retry the sending of the request again but it doesn't, the call just fails. Here is some relevant debug to indicate the issue:
>
> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> POST /Service/ws/ Controls/1.0/GetStuff HTTP/1.1]
> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> txId: 1234]
> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> User-Agent: Apache CXF 2.7.12]
> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> SOAPAction: executeService]
> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> Accept: */*]
> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> Content-Length: 1547]
> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> Content-Type: application/xml]
> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.headers [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 >> Host: a-b.x.com:80]
> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.reactor.IOSessionImpl [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 xxx.x.xx.xx:xxxxx<->yyy.y.yyy.yy:yy[ACTIVE][rw:w]: Event set [w]]
> 2014-12-09 16:10:29,090 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.reactor.IOSessionImpl [txId=|client=|entryService=|callingService=] [] [http-outgoing-268xxx.x.xx.xx:xxxxx<->yyy.y.yyy.yy:yy[ACTIVE][rw:rw]: -1 bytes read]
> 2014-12-09 16:10:29,091 [I/O dispatcher 2] DEBUG org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduitFactory$4 [txId=|client=|entryService=|callingService=] [] [[exchange: 9268] aborting connection [id:268][route:{}->http://a-b.x.com:80][state:null]]
> 2014-12-09 16:10:29,091 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.reactor.IOSessionImpl [txId=|client=|entryService=|callingService=] [] [http-outgoing-268xxx.x.xx.xx:xxxxx<->yyy.y.yyy.yy:yy[ACTIVE][rw:rw]: Shutdown]
> 2014-12-09 16:10:29,091 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.reactor.IOSessionImpl [txId=|client=|entryService=|callingService=] [] [http-outgoing-268 0.0.0.0:11111<->yyy.y.yyy.yy:yy[CLOSED][]: Shutdown]
> 2014-12-09 16:10:29,091 [I/O dispatcher 2] DEBUG org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduitFactory$3 [txId=|client=|entryService=|callingService=] [] [Connection released: [id: 268][route: {}->http://a-b.x.com:80][total kept alive: 0; route allocated: 1 of 1000; total allocated: 1 of 5000]
>
> The only thing I've noticed which could be relevant is that when we send a request to the external service, CXF doesn't explicitly send the Keep-alive headers (as it is implicit through the use of HTTP 1.1) which means the external service does not send back the explicit Keep-Alive: timeout=2, max=100 header.
>
> Please could someone confirm whether or not you would expect the async transport to retry sending the request when a connection is closed due to a keep-alive idle timeout? Am I missing some configuration to allow this?
>
> Many thanks
> Mandy
>
>
>
> Sent from a mobile device
>