You are viewing a plain text version of this content. The canonical link for it is here.
Posted to xmlrpc-dev@ws.apache.org by Alan Burlison <Al...@sun.com> on 2009/04/06 23:47:23 UTC
BUG: clients can hang the ServletWebServer class
We are using the ServletWebServer class, and we are getting occasional
hangs, where the listener thread gets stuck as follows:
"XML-RPC Weblistener" prio=3 tid=0x089bd000 nid=0x15 runnable
[0xe22ce000..0xe22ceb60]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
at
com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
- locked <0xeb722510> (a java.lang.Object)
at
com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:746)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
- locked <0xeb722838> (a com.sun.net.ssl.internal.ssl.AppInputStream)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
- locked <0xeb72ae70> (a
org.apache.xmlrpc.webserver.HttpServletRequestImpl$1)
at
org.apache.xmlrpc.webserver.HttpServletRequestImpl$2.read(HttpServletRequestImpl.java:94)
at javax.servlet.ServletInputStream.readLine(ServletInputStream.java:94)
at
org.apache.xmlrpc.webserver.HttpServletRequestImpl.readLine(HttpServletRequestImpl.java:170)
at org.apache.xmlrpc.webserver.HttpServletRequestImpl.
(HttpServletRequestImpl.java:106)
at org.apache.xmlrpc.webserver.ServletConnection.
(ServletConnection.java:50)
at
org.apache.xmlrpc.webserver.ServletWebServer.newTask(ServletWebServer.java:145)
at org.apache.xmlrpc.webserver.WebServer.run(WebServer.java:335)
at java.lang.Thread.run(Thread.java:619)
The problem is this: when an incoming connection comes in, a new
HttpServletRequestImpl object is created. The constructor of that class
reads in the HTTP headers. The problem with *that* is that the
HttpServletRequestImpl constructor runs in the same thread that accepts
incoming connections, so if it blocks for any reason (e.g. the client
hangs) it blocks the accept thread, which means that any new connections
also hang.
I'm therefore proposing that the reading of the HTTP headers be factored
out of the HttpServletRequestImpl constructor and into a new method that
is called from the 'run' method of the thread that handles the request.
That way, a client that hangs will only hang the thread that is
servicing it and not the entire XML-RPC server.
Can anyone think of any reasons why this might cause problems? If not
I'll do the changes and submit a patch.
I also note that Connection class does this:
// set read timeout to 30 seconds
socket.setSoTimeout (30000);
That *should* mean that the maximum time a client can hang the server
for is 30 seconds, but by observation that's clearly not happening and
needs further investigation.
Thanks,
--
Alan Burlison
--
Re: BUG: clients can hang the ServletWebServer class
Posted by Alan Burlison <Al...@sun.com>.
Daniel Noll wrote:
>> I'll never stop wondering why people are using this stuff for serious
>> jobs.
>
> (a) Because it is there.
> (b) Because they have no need for a full web server as their XML-RPC
> service is the only thing on the entire server.
In our case it's a little more complex: We are running the Apache
XML-RPC server embedded inside Tomcat. Tomcat is providing a HTTPS web
service on port 443, and the XML-RPC service is also running over SSL on
port 1443.
The HTTPS service uses a normal server cert, the XML-RPC service uses a
different set of certs issued by a private CA, and does both client and
server cert validation so we can secure the XML-RPC interface and only
permit specific client applications to connect to it.
Although you can configure Tomcat to do client cert validation, you
can't configure it to use different sets of certs for different ports -
or at least I haven't been able to find how to do it.
We have custom socket factories that we use for the XML-RPC service that
use the XML-RPC service certs, and we use the web server class that
comes with XML-RPC web so that we can control how the sockets are
created, and thereby make sure they use the cert store containing the
private certs.
We could probably write our own plugin for Tomcat that did the same, but
that would be more complicated, and then we would be tied to a
particular servlet container. Using the Apache XML-RPC webserver class
was the simplest option.
--
Alan Burlison
--
Re: BUG: clients can hang the ServletWebServer class
Posted by Daniel Noll <da...@nuix.com>.
Jochen Wiedmann wrote:
> On Mon, Apr 6, 2009 at 11:47 PM, Alan Burlison <Al...@sun.com> wrote:
>
>> We are using the ServletWebServer class, and we are getting occasional
>> hangs, where the listener thread gets stuck as follows:
>
> I'll never stop wondering why people are using this stuff for serious jobs.
(a) Because it is there.
(b) Because they have no need for a full web server as their XML-RPC
service is the only thing on the entire server.
Daniel
--
Daniel Noll Forensic and eDiscovery Software
Senior Developer The world's most advanced
Nuix email data analysis
http://nuix.com/ and eDiscovery software
Re: BUG: clients can hang the ServletWebServer class
Posted by Alan Burlison <Al...@sun.com>.
Logged as https://issues.apache.org/jira/browse/XMLRPC-166
--
Alan Burlison
--
Re: BUG: clients can hang the ServletWebServer class
Posted by Alan Burlison <Al...@sun.com>.
Jochen Wiedmann wrote:
>> Can anyone think of any reasons why this might cause problems? If not I'll
>> do the changes and submit a patch.
>
> Please do. Not the first valuable patch from you.
Attached.
Here's what I've done: In the existing code the HttpServletRequestImpl
constructor reads the HTTP headers, and if the client doesn't send any
headers, the constructor hangs, and that hangs the thread that accepts
incoming connections. I've factored out the header reading code into a
new method (readHttpHeaders) which is called from the
ServletConnection.run method of the thread that processes the request:
public void run() throws Throwable {
try {
request.readHttpHeaders();
servlet.service(request, response);
} catch (Throwable t) {
if (!shuttingDown) {
throw t;
}
}
}
>> I also note that Connection class does this:
>>
>> // set read timeout to 30 seconds
>> socket.setSoTimeout (30000);
>>
>> That *should* mean that the maximum time a client can hang the server for is
>> 30 seconds, but by observation that's clearly not happening and needs
>> further investigation.
>
> Patches in that area should be welcome as well.
Also fixed. The problem was that although the Connection class set a 30
second timeout, the ServletConnection didn't - the constructor was
missing the call to setSoTimeout. The Connection class is used by the
WebServer class, ServletConnection is used by ServletWebServer.
In combination, these changes mean that a client that doesn't send any
headers can't hang the accept() thread, and that after 30 seconds, any
such misbehaving client will have its connection terminated.
--
Alan Burlison
--
Re: BUG: clients can hang the ServletWebServer class
Posted by Jochen Wiedmann <jo...@gmail.com>.
On Mon, Apr 6, 2009 at 11:47 PM, Alan Burlison <Al...@sun.com> wrote:
> We are using the ServletWebServer class, and we are getting occasional
> hangs, where the listener thread gets stuck as follows:
I'll never stop wondering why people are using this stuff for serious jobs.
> The problem is this: when an incoming connection comes in, a new
> HttpServletRequestImpl object is created. The constructor of that class
> reads in the HTTP headers. The problem with *that* is that the
> HttpServletRequestImpl constructor runs in the same thread that accepts
> incoming connections, so if it blocks for any reason (e.g. the client hangs)
> it blocks the accept thread, which means that any new connections also hang.
Interesting. News to me, that the accept thread must not block, but one never
stops to learn.
> Can anyone think of any reasons why this might cause problems? If not I'll
> do the changes and submit a patch.
Please do. Not the first valuable patch from you.
> I also note that Connection class does this:
>
> // set read timeout to 30 seconds
> socket.setSoTimeout (30000);
>
> That *should* mean that the maximum time a client can hang the server for is
> 30 seconds, but by observation that's clearly not happening and needs
> further investigation.
Patches in that area should be welcome as well.
Jochen
--
I have always wished for my computer to be as easy to use as my
telephone; my wish has come true because I can no longer figure out
how to use my telephone.
-- (Bjarne Stroustrup,
http://www.research.att.com/~bs/bs_faq.html#really-say-that
My guess: Nokia E50)