You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Bob DeRemer <bo...@thingworx.com> on 2013/11/03 03:29:22 UTC
weird behavior using Tomcat 7 JSR-356 client implementation in
standalone multi-threaded test client
BACKGROUND:
We've created a test client that spins up multiple websocket clients - each in their own thread. While using their own thread isn't very resource efficient, that's ok. We want each client's send/receive to be isolated from each other. Our client logic is using a JSR356 client that extends Endpoint and implements Whole<byte[]>. In addition, we have message synchronization, so each client will not send a new request until it's received the corresponding respone (i.e. we have our own req/resp messages with unique IDs).
Our client is running the JVM with the following:
java -Dserver -Dd64 -Xms24G -Xmx48g -XX:+UseNUMA -XX:+UseG1GC
PROBLEM
We are seeing a problem when running multiple clients. It appears that we're sending messages on 1 websocket client, but sometimes receiving the message on a different websocket. It can be as little as 1000 or as many as 25000. It seems to depend on how fast responses are received whether we see the problem. We have checked the server logging and have verified that all messages have been received and sent back on the proper websocket connection.
QUESTION
When using Tomcat's JSR-356 client implementation in a standalone java app, what kind of threading does it use under the hood when messages are received? Is there any possibility that there can be any cross-talk?
NOTE: we're using the following jar(s) in our standalone client:
tomcat7-websocket.jar
tomcat-api.jar
tomcat-coyote.jar
tomcat-i18n-es.jar
tomcat-util.jar
websocket-api.jar
tomcat-juli.jar
Thanks
Bob
http://www.thingworx.com<http://www.thingworx.com/>
Skype: bob.deremer.thingworx
O: 610.594.6200 x812
M: 717.881.3986
RE: weird behavior using Tomcat 7 JSR-356 client implementation in
standalone multi-threaded test client
Posted by Bob DeRemer <bo...@thingworx.com>.
> -----Original Message-----
> From: Mark Thomas [mailto:markt@apache.org]
> Sent: Sunday, November 03, 2013 11:24 AM
> To: Tomcat Users List
> Subject: Re: weird behavior using Tomcat 7 JSR-356 client implementation in
> standalone multi-threaded test client
>
> On 03/11/2013 02:29, Bob DeRemer wrote:
> > BACKGROUND: We've created a test client that spins up multiple
> > websocket clients - each in their own thread. While using their own
> > thread isn't very resource efficient, that's ok. We want each
> > client's send/receive to be isolated from each other. Our client
> > logic is using a JSR356 client that extends Endpoint and implements
> > Whole<byte[]>. In addition, we have message synchronization, so each
> > client will not send a new request until it's received the
> > corresponding respone (i.e. we have our own req/resp messages with
> > unique IDs).
> >
> > Our client is running the JVM with the following: java -Dserver -Dd64
> > -Xms24G -Xmx48g -XX:+UseNUMA -XX:+UseG1GC
> >
> > PROBLEM We are seeing a problem when running multiple clients. It
> > appears that we're sending messages on 1 websocket client, but
> > sometimes receiving the message on a different websocket. It can be
> > as little as 1000 or as many as 25000. It seems to depend on how fast
> > responses are received whether we see the problem. We have checked
> > the server logging and have verified that all messages have been
> > received and sent back on the proper websocket connection.
> >
> > QUESTION When using Tomcat's JSR-356 client implementation in a
> > standalone java app, what kind of threading does it use under the hood
> > when messages are received? Is there any possibility that there can
> > be any cross-talk?
>
> The client uses AsynchronousSocketChannel under the covers. It maintains its
> own thread pool although Tomcat provides a thread factory so the threads get
> sensible names. Each incoming block of data should be processed with a
> separate thread.
>
> Much of the code is shared between the client and server implementations.
>
> I don't see any obvious chances for cross-talk. Is the error (when it
> occurs) consistent? ie. client 4 always gets client 5's responses or is it random?
> Consistent would point to an issue during connection setup, random to an issue
> during connection processing.
>
> It is certainly possible that there is a Tomcat bug here. If it is hard to reproduce
> then code inspection might find it but we need to narrow down the area to look
> in. Keep in mind it could be a client bug too.
>
> When it occurs, it would be worth checking the clients concerned. E.g.
> have the managed to end up with the same message handler for instance.
>
> Mark
>
Thanks Mark - yeah, we haven't ruled out our client code yet, so we're going to make sure that's as simple as possible. If we can come up with a somewhere reproducible case, we'll post back.
-bob
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: weird behavior using Tomcat 7 JSR-356 client implementation in
standalone multi-threaded test client
Posted by Mark Thomas <ma...@apache.org>.
On 03/11/2013 02:29, Bob DeRemer wrote:
> BACKGROUND: We've created a test client that spins up multiple
> websocket clients - each in their own thread. While using their own
> thread isn't very resource efficient, that's ok. We want each
> client's send/receive to be isolated from each other. Our client
> logic is using a JSR356 client that extends Endpoint and implements
> Whole<byte[]>. In addition, we have message synchronization, so each
> client will not send a new request until it's received the
> corresponding respone (i.e. we have our own req/resp messages with
> unique IDs).
>
> Our client is running the JVM with the following: java -Dserver -Dd64
> -Xms24G -Xmx48g -XX:+UseNUMA -XX:+UseG1GC
>
> PROBLEM We are seeing a problem when running multiple clients. It
> appears that we're sending messages on 1 websocket client, but
> sometimes receiving the message on a different websocket. It can be
> as little as 1000 or as many as 25000. It seems to depend on how
> fast responses are received whether we see the problem. We have
> checked the server logging and have verified that all messages have
> been received and sent back on the proper websocket connection.
>
> QUESTION When using Tomcat's JSR-356 client implementation in a
> standalone java app, what kind of threading does it use under the
> hood when messages are received? Is there any possibility that there
> can be any cross-talk?
The client uses AsynchronousSocketChannel under the covers. It maintains
its own thread pool although Tomcat provides a thread factory so the
threads get sensible names. Each incoming block of data should be
processed with a separate thread.
Much of the code is shared between the client and server implementations.
I don't see any obvious chances for cross-talk. Is the error (when it
occurs) consistent? ie. client 4 always gets client 5's responses or is
it random? Consistent would point to an issue during connection setup,
random to an issue during connection processing.
It is certainly possible that there is a Tomcat bug here. If it is hard
to reproduce then code inspection might find it but we need to narrow
down the area to look in. Keep in mind it could be a client bug too.
When it occurs, it would be worth checking the clients concerned. E.g.
have the managed to end up with the same message handler for instance.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org