You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@servicemix.apache.org by Eric Anderson <er...@ultracode.com> on 2006/06/08 22:49:42 UTC
HTTP sockets not being closed
Hello,
I'm having a nagging problem with a lingering (not in the TCP
terminology way) HTTP socket connection after using a slightly
customized HttpInvoker component to send a message to a slightly
modified HttpConnector component, running on port 4900. The symptom is,
that after about a minute when the HTTP exchange is complete, I get the
following debug output on the console:
DEBUG - JCLLoggerAdapter.debug(155) | EXCEPTION
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:99)
at
org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:141)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:255)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:190)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:298)
at
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:153)
at
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:412)
DEBUG - JCLLoggerAdapter.debug(121) | EOF
After this exchange, but prior to the above exception, I see:
netstat -a -p |grep 4900
tcp6 0 0 localhost.localdom:4900 *:*
LISTEN 15130/java
tcp6 0 0 localhost.localdo:32956 localhost.localdom:4900
ESTABLISHED15130/java
tcp6 0 0 localhost.localdom:4900 localhost.localdo:32956
ESTABLISHED15130/java
After the exception I see:
netstat -a -p |grep 4900
tcp6 0 0 localhost.localdom:4900 *:*
LISTEN 15130/java
tcp6 1 0 localhost.localdo:32956 localhost.localdom:4900
CLOSE_WAIT 15130/java
tcp6 0 0 localhost.localdom:4900 localhost.localdo:32956
FIN_WAIT2 -
And eventually the connection goes away entirely.
My HttpConnector consists of a copy of the ServiceMix HttpConnector and
is essentially modified to "eat" the message - it's an HTTP /dev/null .
On each request, it's executing this:
public void processInOut(HttpServletRequest request,
HttpServletResponse response) throws ServletException, IOException,
JBIException
{
log.debug("Processing HTTP request from " +
request.getRemoteAddr());
BufferedReader reader = request.getReader();
String s;
while (null != (s = reader.readLine()))
{
log.debug(s);
}
reader.close();
response.setStatus(HttpServletResponse.SC_OK);
response.setHeader("X-CJIS-HTTP-Server-Debug", "done");
response.setContentLength(0);
}
Everything *appears* to work correctly - I can see the on-the-wire trace
output with the correct data flowing in each direction. The sender sees
an SC_OK response code and is happy. The Sender makes sure to close the
connection by calling
org.apache.commons.httpclient.methods.PostMethod.releaseConnection()
just like the stock HttpInvoker does.
Adding a bit of confusion to this is that I have another, unmodified
HttpConnector in defaultInOut=false mode running on another port, and
being sent HTTP messages from an external test client, not another
Component. This seems to work fine, but in that case the test client
terminates after sending a single message, so perhaps that test is
irrelevant.
This *seems* like a Jetty issue of some sort. I upgraded to 6.0.0
beta12 but it didn't change anything. From looking at the Jetty source,
it simply appears that Jetty thinks there's more to read on the HTTP
connection, but why this would be I don't understand, especially given
the fact that the response has already been sent.
Thoughts, suggestions anyone?
Thanks,
Eric
Re: HTTP sockets not being closed
Posted by jpoley <ja...@yahoo.com>.
This looks like it may be related to the tcp/ip settings on our OS.
what OS name / version are you running?
you might have to change a tcp_time_wait interval...
you can read similar documentatoin here
http://e-docs.bea.com/wls/docs81/perform/HWTuning.html#1105552
let me know if this helps
-jp
--
View this message in context: http://www.nabble.com/HTTP-sockets-not-being-closed-t1758148.html#a4785996
Sent from the ServiceMix - User forum at Nabble.com.
Re: HTTP sockets not being closed
Posted by jpoley <ja...@yahoo.com>.
This looks like it may be related to the tcp/ip settings on your OS.
what OS name / version are you running?
you might have to change a tcp_time_wait interval...
you can read similar documentatoin here
http://e-docs.bea.com/wls/docs81/perform/HWTuning.html#1105552
let me know if this helps
-jp
--
View this message in context: http://www.nabble.com/HTTP-sockets-not-being-closed-t1758148.html#a4785996
Sent from the ServiceMix - User forum at Nabble.com.
Re: HTTP sockets not being closed
Posted by Eric Anderson <er...@ultracode.com>.
I tried beta17, no luck. Then I traced through Jetty and found that it
was leaving the HTTP connection open and trying to read another request
from it, as long as the protocol version was 1.1 and the client had not
closed the connection. As it turns out, this is exactly what's supposed
to be done with HTTP/1.1 connections. After a minute or so Jetty
times-out the connection and closes it, which resulted in debug output
that I was seeing.
Apache HTTPClient, via the MultiThreadedHttpConnectionManager, seems to
be doing HTTP connection pooling and is keeping previously used
connections alive for later re-use. I was able to get around this two
ways, 1) by setting the protocol version to 1.0, and 2) by sending a
"Connection: close" HTTP parameter to the server. The latter would be
the preferred way to accomplish this I would think.
Under load, I can imagine that HTTP connection pooling in the
HttpInvoker would be a Good Thing. But under light loads, it results in
the socket timeout exceptions at the server, and of course in some
amount of server resources being used as well. So I added a config
parameter to *my* HttpInvoker which can be set via Spring. The code now
has in it
su/servicemix.xml:
<property name="closeConnections" value="true"/>
HttpInvoker.java :
protected boolean transform(MessageExchange exchange,
NormalizedMessage in, NormalizedMessage out) throws MessagingException
{
PostMethod method = new PostMethod(url);
try {
...
if (true == getCloseConnections())
{
method.setRequestHeader("Connection", "close");
}
....
Seems to work for me. This might be a useful addition to the stock
HttpInvoker class.
Eric
Guillaume Nodet wrote:
> I think I have read something like that on jetty mailing list.
> Try to upgrade to latest beta17 maybe.
> Then ask on jetty mailing list.
>
> Cheers,
> Guillaume Nodet
>
> On 6/8/06, Eric Anderson <er...@ultracode.com> wrote:
>>
>> Hello,
>>
>> I'm having a nagging problem with a lingering (not in the TCP
>> terminology way) HTTP socket connection after using a slightly
>> customized HttpInvoker component to send a message to a slightly
>> modified HttpConnector component, running on port 4900. The symptom is,
>> that after about a minute when the HTTP exchange is complete, I get the
>> following debug output on the console:
>>
>> DEBUG - JCLLoggerAdapter.debug(155) | EXCEPTION
>> java.net.SocketTimeoutException: Read timed out
>> at java.net.SocketInputStream.socketRead0(Native Method)
>> at java.net.SocketInputStream.read(SocketInputStream.java:129)
>> at
>> org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:99)
>> at
>> org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java
>>
>> :141)
>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:255)
>> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java
>> :190)
>> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java
>> :298)
>> at
>> org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java
>>
>> :153)
>> at
>> org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java
>>
>> :412)
>> DEBUG - JCLLoggerAdapter.debug(121) | EOF
>>
>> After this exchange, but prior to the above exception, I see:
>>
>> netstat -a -p |grep 4900
>> tcp6 0 0 localhost.localdom:4900 *:*
>> LISTEN 15130/java
>> tcp6 0 0 localhost.localdo:32956 localhost.localdom:4900
>> ESTABLISHED15130/java
>> tcp6 0 0 localhost.localdom:4900 localhost.localdo:32956
>> ESTABLISHED15130/java
>>
>> After the exception I see:
>>
>> netstat -a -p |grep 4900
>> tcp6 0 0 localhost.localdom:4900 *:*
>> LISTEN 15130/java
>> tcp6 1 0 localhost.localdo:32956 localhost.localdom:4900
>> CLOSE_WAIT 15130/java
>> tcp6 0 0 localhost.localdom:4900 localhost.localdo:32956
>> FIN_WAIT2 -
>>
>> And eventually the connection goes away entirely.
>>
>> My HttpConnector consists of a copy of the ServiceMix HttpConnector and
>> is essentially modified to "eat" the message - it's an HTTP /dev/null .
>>
>> On each request, it's executing this:
>>
>> public void processInOut(HttpServletRequest request,
>> HttpServletResponse response) throws ServletException, IOException,
>> JBIException
>> {
>> log.debug("Processing HTTP request from " +
>> request.getRemoteAddr());
>> BufferedReader reader = request.getReader();
>> String s;
>> while (null != (s = reader.readLine()))
>> {
>> log.debug(s);
>> }
>> reader.close();
>> response.setStatus(HttpServletResponse.SC_OK);
>> response.setHeader("X-CJIS-HTTP-Server-Debug", "done");
>> response.setContentLength(0);
>> }
>>
>> Everything *appears* to work correctly - I can see the on-the-wire trace
>> output with the correct data flowing in each direction. The sender sees
>> an SC_OK response code and is happy. The Sender makes sure to close the
>> connection by calling
>> org.apache.commons.httpclient.methods.PostMethod.releaseConnection()
>> just like the stock HttpInvoker does.
>>
>> Adding a bit of confusion to this is that I have another, unmodified
>> HttpConnector in defaultInOut=false mode running on another port, and
>> being sent HTTP messages from an external test client, not another
>> Component. This seems to work fine, but in that case the test client
>> terminates after sending a single message, so perhaps that test is
>> irrelevant.
>>
>> This *seems* like a Jetty issue of some sort. I upgraded to 6.0.0
>> beta12 but it didn't change anything. From looking at the Jetty source,
>> it simply appears that Jetty thinks there's more to read on the HTTP
>> connection, but why this would be I don't understand, especially given
>> the fact that the response has already been sent.
>>
>> Thoughts, suggestions anyone?
>>
>> Thanks,
>> Eric
>>
>>
>
Re: HTTP sockets not being closed
Posted by Guillaume Nodet <gn...@gmail.com>.
I think I have read something like that on jetty mailing list.
Try to upgrade to latest beta17 maybe.
Then ask on jetty mailing list.
Cheers,
Guillaume Nodet
On 6/8/06, Eric Anderson <er...@ultracode.com> wrote:
>
> Hello,
>
> I'm having a nagging problem with a lingering (not in the TCP
> terminology way) HTTP socket connection after using a slightly
> customized HttpInvoker component to send a message to a slightly
> modified HttpConnector component, running on port 4900. The symptom is,
> that after about a minute when the HTTP exchange is complete, I get the
> following debug output on the console:
>
> DEBUG - JCLLoggerAdapter.debug(155) | EXCEPTION
> java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:129)
> at org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:99)
> at
> org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java
> :141)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:255)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java
> :190)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java
> :298)
> at
> org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java
> :153)
> at
> org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java
> :412)
> DEBUG - JCLLoggerAdapter.debug(121) | EOF
>
> After this exchange, but prior to the above exception, I see:
>
> netstat -a -p |grep 4900
> tcp6 0 0 localhost.localdom:4900 *:*
> LISTEN 15130/java
> tcp6 0 0 localhost.localdo:32956 localhost.localdom:4900
> ESTABLISHED15130/java
> tcp6 0 0 localhost.localdom:4900 localhost.localdo:32956
> ESTABLISHED15130/java
>
> After the exception I see:
>
> netstat -a -p |grep 4900
> tcp6 0 0 localhost.localdom:4900 *:*
> LISTEN 15130/java
> tcp6 1 0 localhost.localdo:32956 localhost.localdom:4900
> CLOSE_WAIT 15130/java
> tcp6 0 0 localhost.localdom:4900 localhost.localdo:32956
> FIN_WAIT2 -
>
> And eventually the connection goes away entirely.
>
> My HttpConnector consists of a copy of the ServiceMix HttpConnector and
> is essentially modified to "eat" the message - it's an HTTP /dev/null .
>
> On each request, it's executing this:
>
> public void processInOut(HttpServletRequest request,
> HttpServletResponse response) throws ServletException, IOException,
> JBIException
> {
> log.debug("Processing HTTP request from " +
> request.getRemoteAddr());
> BufferedReader reader = request.getReader();
> String s;
> while (null != (s = reader.readLine()))
> {
> log.debug(s);
> }
> reader.close();
> response.setStatus(HttpServletResponse.SC_OK);
> response.setHeader("X-CJIS-HTTP-Server-Debug", "done");
> response.setContentLength(0);
> }
>
> Everything *appears* to work correctly - I can see the on-the-wire trace
> output with the correct data flowing in each direction. The sender sees
> an SC_OK response code and is happy. The Sender makes sure to close the
> connection by calling
> org.apache.commons.httpclient.methods.PostMethod.releaseConnection()
> just like the stock HttpInvoker does.
>
> Adding a bit of confusion to this is that I have another, unmodified
> HttpConnector in defaultInOut=false mode running on another port, and
> being sent HTTP messages from an external test client, not another
> Component. This seems to work fine, but in that case the test client
> terminates after sending a single message, so perhaps that test is
> irrelevant.
>
> This *seems* like a Jetty issue of some sort. I upgraded to 6.0.0
> beta12 but it didn't change anything. From looking at the Jetty source,
> it simply appears that Jetty thinks there's more to read on the HTTP
> connection, but why this would be I don't understand, especially given
> the fact that the response has already been sent.
>
> Thoughts, suggestions anyone?
>
> Thanks,
> Eric
>
>