axis 1.5.1, httpclient 3.1 and missing FINACK

["Brooks, Kenneth S" <ke...@chase.com>]

Oleg,

Another team in my company is using Axis2 1.5.1 (with HttpClient 3.1) as clients of a webservice.

They are seeing the following:

*       When communicating from source -> destination via http, everything works fine.
o       Responses come in less than 2 seconds and the FIN, FINACK happens within 5 seconds. Almost no RST happening.
*       When communicating from source -> destination via https (no other changes) they are seeing these symptoms:
o       Over 90% of all calls never see the FINACK sent back by client. This causes a very large number of RST to happen. The majority of the RST get RSTACK back, but not all.
o       About 1.4% of all calls get "Connect Timeout" errors because they think that the socket is still in use.

That is the very high level that I can gather from them so far.

Basically what I'm asking you is
1) Have you seen any issues with httpclient (or axis use of httpclient) when using https
2) Is httpclient even in the mix when it comes to low level protocol communication like the FIN and FINACK? Or are we barking up the wrong tree and need to look at some OS/Hardware level area instead?

Thanks a bunch,
ken

Just for more reference, here is their writeup on what they've done and what their thinking is.

Enabled additional captures (tcpdump) on the Business Partner firewall to understand the SSL transmission behavior. The theory is there are no FIN_ACK being received by the Server (ALG) from the Client (Chase). Since the Server doesn't receive a FIN_ACK on time (typically 8 to 10 secs) it sends a RESET to the Client. In 98.6% of the cases the client is able to send a RESET_ACK back to the server, thereby freeing up the source port for future communication on that port. However 1.4% of the cases it trying to reuse the source port without having sent RESET_ACK back to the server and we get the "Connect Timeout" errors.

Some interesting observations: when we communicate over HTTP, there are very few resets happening, reason being most transactions finish in 3-5 secs there by avoiding the need for a RESET from the server. But for HTTPS very few transactions end with FIN_ACK from the client, and the once that end gracefully with a FIN_ACK end under 10secs. Since most transactions here are running longer the server typically ends up sending a RESET and a RESET_ACK from the client finishes the transactions within 15-25 secs. (I will call this as SSL overhead). Another observation is lot of close_waits on the server during HTTPS, which explains the 15-25 secs to close the  connection. The pay load is delivered from the server to the client in 2-3 secs, which matches with the response times we see with most calls.

Here is my take on it and we should discuss, it appears the AIX2 framework may not be very efficient when it comes to SSL and we might be in need of a SSL Accelerator (H/w or S/w). ALG has a SSL Accelerator on their load balancer. However on a good note we are not seeing any response time degradation, make me believe we are not passing this overhead to the client, the packet gets delivered to the client the handshake overhead within the framework happens behind the scenes.




This transmission may contain information that is privileged,
confidential, legally privileged, and/or exempt from disclosure
under applicable law.  If you are not the intended recipient, you
are hereby notified that any disclosure, copying, distribution, or
use of the information contained herein (including any reliance
thereon) is STRICTLY PROHIBITED.  Although this transmission and
any attachments are believed to be free of any virus or other
defect that might affect any computer system into which it is
received and opened, it is the responsibility of the recipient to
ensure that it is virus free and no responsibility is accepted by
JPMorgan Chase & Co., its subsidiaries and affiliates, as
applicable, for any loss or damage arising in any way from its use.
 If you received this transmission in error, please immediately
contact the sender and destroy the material in its entirety,
whether in electronic or hard copy format. Thank you.

Re: axis 1.5.1, httpclient 3.1 and missing FINACK

[Oleg Kalnichevski <ol...@apache.org>]

On Thu, 2010-07-22 at 13:20 -0400, Brooks, Kenneth S wrote:
> Oleg,
> 
> Another team in my company is using Axis2 1.5.1 (with HttpClient 3.1) as clients of a webservice.
> 
> They are seeing the following:
> 
> *       When communicating from source -> destination via http, everything works fine.
> o       Responses come in less than 2 seconds and the FIN, FINACK happens within 5 seconds. Almost no RST happening.

This probably means that the application is not re-using connections. Is
this intentional? 


> *       When communicating from source -> destination via https (no other changes) they are seeing these symptoms:
> o       Over 90% of all calls never see the FINACK sent back by client. This causes a very large number of RST to happen. The majority of the RST get RSTACK back, but not all.
> o       About 1.4% of all calls get "Connect Timeout" errors because they think that the socket is still in use.
> 
> That is the very high level that I can gather from them so far.
> 
> Basically what I'm asking you is
> 1) Have you seen any issues with httpclient (or axis use of httpclient) when using https

Personally I have never experienced any issues with HTTPS using both HC
3.1 and HC 4.0. I have never really used Axis2, though. Never managed to
come to terms with it no matter how hard I tried. 

> 2) Is httpclient even in the mix when it comes to low level protocol communication like the FIN and FINACK? Or are we barking up the wrong tree and need to look at some OS/Hardware level area instead?
> 

Java applications in general have no means to controlling TCP/IP
networking on such a low level. What might well be the case, though,
that Java application does not explicitly close the socket on its end
thus leaving the connection is a half open state until the socket gets
garbage collected. I _believe_ Axis2 used to have issues with connection
management and used to be prone to leaking client-side HTTP connections.

Is there a reason connections are not being re-used? Maybe the problem
could be mitigated by reducing the number of connections that need to be
established in the first place?

I am not a TCP/IP specialist by any stretch of imagination so take it
for what it is worth to you.

Cheers

Oleg  



---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org