Re: Connection drops with tomcat-based game server (very long version)

[Justin Ruthenbeck <ju...@nextengine.com>]

Hi Wouter,

I can't give you concrete responses to your questions, but I can offer some 
anectodal observations from experience.

We also have a connection-sensitive, high message frequency, small message 
size portion of our application.  During bursts, the system must send and 
receive data as quickly as possible (every ~80ms or so) between the client 
(a browser) and the app server (Tomcat).

The application in general is a standard browser/server web based 
application and uses standard J2EE Servlet components.  We originally tried 
to do the same thing for the messaging component, but ran into excessive 
overhead and connection maintenance issues as well.  Specifically, we saw 
clients who occasionally had to wait long periods for their responses 
(~1000ms) or simply never received them.  Performance was better if we 
connected directly to the appserver instead of through IIS or Apache, but 
we still saw occasional connection loss.

These losses really killed the user experience and sometimes caused clients 
to ultimately mangle their internal state.  I would add that we saw similar 
problems with Weblogic.

After quite a bit of debugging, we were unable to determine a consistent 
cause of these problems, though it looked as though the appserver always 
responded if a valid Http request got to them.  This left us with the 
browser and network as potential offenders.  Fortunately, we had the option 
of applets.

We finally decided that relying on the browser and it's http stack to 
maintain connection was simply too risky and difficult to recover from.  We 
wrote our own lightweight mini Http client (as part of a client-side 
applet) and accompanying server (as an HttpServlet) running within our 
webapp on Tomcat.  With the finer grained control, error detection and 
recovery is much easier and we haven't been losing connections like before 
since the mini-http client has full control of when and why connections are 
closed.  Our request/response turnaround times are in the 20ms range for 
local networks.

This isn't to say such a solution would be best for you.  Implementing your 
own http client is a ripe field for frustration.  It also seems pointless 
to "reinvent" the http wheel when so many stable implementations exist.  In 
our case, however, it's given us the predictability we need while still 
allowing flexibility such as support for firewalled users and http proxy 
(or not) configurations.

Don't know how much that helps, but it's always good to hear of others' 
experiences.

justin

At 10:56 AM 3/21/2003, you wrote:
>Hi,
>
>I'm currently co-developing a game (draughts, see www.damserver.nl) server 
>that utilises Tomcat for message passing and relaying, which is in a 
>testing phase right now. During development of the server (on a fast 
>network) we encountered no big problems with Tomcat, but since testing has 
>intensified somewhat (especially by testers on slow modem connections) it 
>sometimes seems like Tomcat drops connections to clients randomly every 
>now and then.
>
>In short, the server works a bit like a chat server, with the exception 
>that not only chat messages are relayed by it, but also moves, connection 
>state information, user actions etc. The server keeps no internal state 
>about board setups etc. (this is all handled by the clients), but only 
>which users are logged in, where these users are at any moment (portal, 
>room, table screen), and what games are being played by which users. 
>Clients communicate with the servlet engine through only 1 servlet (this 
>is because the servlet specs advise against sharing state information 
>between servlets), which acts like a 'proxy' between the network and the 
>server logic. We don't use the HTTP protocol (ie: we dont use HttpServlet 
>but GenericServlet), but we do use the HTTP port.
>
>Messages are encoded and sent (by 'requesting' the game server servlet) as 
>raw data to the servlet engine (both server and client use 
>DataInputStream's and DataOutputStreams to read/write data). The server 
>then decodes the data, relays the message to clients if necessary, passes 
>new messages to clients if the received message demands it and returns a 
>response to the client that sent the message. This response also contains 
>only raw data, which is usually (but not always) no more than an 
>acknowledgement. Clients get their messages by polling the game server 2 
>times/second (they do this by sending a special message to the game server 
>servlet, which in turn writes back all of the clients' messages as a 
>response to the 'request' containing the message). The clients need to 
>poll because we can only use the request-response paradigm, since the 
>server should be accessible from behind a firewall. The server expects 
>connected clients to fetch their messages at least once every 2 minut
>  es, and checks this by dropping a 'ping' message in all its client 
> message queues, and verifying that it gets consumed within 2 minutes.
>
>The problems we encounter seem to surface only when a client connects 
>using a 'slow' connection, especially with modem connections but maybe 
>also with cable connections (which have little upstream bandwith). The 
>symptoms are that after some time playing a game a client does not receive 
>messages anymore. The game server keeps working fine, and the client also 
>goes on happily. It just does not receive any more messages, although the 
>server definitely sends them. Ofcourse this probably looks like bug in our 
>code, but we do not think this is the case, since we cannot reproduce the 
>bug on a fast network. It seems as if the Tomcat server sometimes randomly 
>kills a request (especially under heavy load) without sending a response. 
>The client thread that polls the server still expects this response but it 
>does not get it, so it blocks, and no new requests for messages are made. 
>After some time the game server detects that the client did not consume 
>its messages, and logs out the user.
>   As a side note I would like to mention the fact that we had to increase 
> the maximum number of simultaneous threads per servlet to get acceptable 
> performance if more than 8 people were logged in at the same time,
>
>Now for the question this post is all about: is it possible that tomcat 
>sometimes randomly kills a request, does not process a request or queues a 
>request indefinitely, thereby blocking all further communication with a 
>client? Is this possible because the server communicates with the clients 
>using only raw data and InputStreams/OutputStreams? And: is it a good idea 
>to develop a game server like the one described here using Tomcat, 
>considering the fact that tomcats main purpose is to act as web 
>application server using a strict request-response paradigm. Our project 
>development team suspects that tomcat was never meant for this type of 
>applications (that produce very small, but very frequent requests, and are 
>really connection-sensitive), and that it is not safe to assume that every 
>request made by a client always yields a proper response.
>
>Don't get me wrong: I do *not* think there's something wrong with tomcat! 
>As a matter of fact I have really good experiences with it using it for 
>web applications!!
>
>Kind Regards,
>
>         Wouter Bijlsma,
>         j.w.bijlsma@stud.tue.nl


____________________________________
Justin Ruthenbeck
Software Engineer, NextEngine Inc.
justinr - AT - nextengine DOT com
Confidential
    See http://www.nextengine.com/confidentiality.php
____________________________________


---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org

Re: Connection drops with tomcat-based game server (very long version)

[Wouter Bijlsma <j....@stud.tue.nl>]

Hi Justin,

First of all, thanks for your reply. Some of the problems you describe look an awful lot like the ones we're experiencing:

On Fri, 21 Mar 2003 12:10:49 -0800
Justin Ruthenbeck <ju...@nextengine.com> wrote:

> The application in general is a standard browser/server web based 
> application and uses standard J2EE Servlet components.  We originally
> tried to do the same thing for the messaging component, but ran into
> excessive overhead and connection maintenance issues as well. 
> Specifically, we saw clients who occasionally had to wait long periods
> for their responses (~1000ms) or simply never received them. 
> Performance was better if we connected directly to the appserver instead
> of through IIS or Apache, but we still saw occasional connection loss.

This is exactly what happens in our application. At first we'd get response times up to ~19000ms (yes, 19 seconds) when more than 10 users were logged in, but this was because the number of simultaneous threads per servlet was still at the default value (75). After bumping this number up to 1000 (which might very well be the source of a lot of other performance problems, but it's the only thing we could do to ensure reasonable performance for up to ~100 users) the response times stayed under ~1000ms. We already set up Tomcat as a standalone server because response times were *a lot* better without Apache.


> These losses really killed the user experience and sometimes caused
> clients to ultimately mangle their internal state.  I would add that we
> saw similar problems with Weblogic.

This is probably what happens in the client, resulting in the message polling thread to crash/deadlock/block/stop or something like that.

> After quite a bit of debugging, we were unable to determine a consistent
> cause of these problems, though it looked as though the appserver always
> responded if a valid Http request got to them.  This left us with the 
> browser and network as potential offenders.  Fortunately, we had the
> option of applets.

We debugged both client and server extensively, but we too could not locate a consistent cause of problems. I do have to note that the game client already IS an applet, and that we weren't using the HTTP protocol in the first place. I don't know if tomcat makes an HTTP request anyways, but we don't tell it to, we are only reading/writing data to the InputStream's/OutputStream's we got from the URLConnection (client side) and the ServletRequest (server side).

> With the finer grained control, error detection and 
> recovery is much easier and we haven't been losing connections like
> before since the mini-http client has full control of when and why
> connections are closed. 

Does this mean you don't have lost connection at all anymore?

> Don't know how much that helps, but it's always good to hear of others' 
> experiences.

Exactly :-)

	Wouter



---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org