You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Peter Cameron <p....@monitormedia.co.uk> on 2011/07/08 10:25:57 UTC
Tomcat appears to be sending duplicate requests
I'm having an interesting problem which seems to be Tomcat-related and
which occurs randomly under heavy load. I'd appreciate some advice/help.
I'm developing a Java web application and deploying to Tomcat 6.0.32. (I
get the same behaviour with the latest Tomcat 7).
The web app uses Spring 3.05 and has a RESTful interface implemented
with Jersey 1.8. It only runs on my machine and nothing else talks to it
apart from a Java integration test which throws data at it.
The app is a "black box" which is interested in counting combinations of
inputs from forms. Randomly, the counts in the database would be wrong
-- never ever less than expected, but always one or two more than
expected. In other words, it's not as if there is a race condition and
updates are being overwritten since the counts are never less than they
should be. The back-end is thoroughly thread-safe.
My test harness generates 10 batches of loads. Each batch is submitted
concurrently, and each request within a batch is submitted serially with
a 1/10 second delay. (The batch sizes are: 5372, 801, 3005, 637, 9311,
313, 1024, 777, 412, 1201.)
Anyway, I incorporated the following diagnostics in my application:
o various statistics using JMX beans and atomic integer counts
o atomic integer request count in the test harness
o http header in each request containing the unique request count
These diagnostics show that the test harness sends out the correct
number of requests but that the web app receives more requests than the
test harness is sending. I put a BitSet into the Jersey resource
endpoint (which receives each request), large enough for the total
number of requests, and set the appropriate bit corresponding to the
unique count I pass as an http header. Debugging shows that I sometimes
receive more than one request with the same count.
Why is Tomcat sending duplicate requests? Is there some timeout that is
silently happening under load and causing Tomcat to resend the request?
Are there any JMX diagnostics I can look at/turn on for Tomcat?
Something is sending those duplicate requests and it isn't me!
cheers,
Peter
Re: Tomcat appears to be sending duplicate requests
Posted by peterjca <p....@monitormedia.co.uk>.
>Now you just need a test harness for your test harness. :)
Ad infinitum!
It's not strictly a bug in the harness. The test harness implements one
correct scenario for a concurrent "thrashing". It uses the Jersey client API
to post requests, by constructing a new client each time. The only state
maintained by the harness are atomic counts for the debugging I've been
doing. The only states passed within the harness are either Strings or
immutable collections of name/value pairs. There is no session state
maintained on the server as each request is context complete.
With both end-point Jersey resources in operation (each with completely
distinct URLs), one resource gets fed duplicates of requests it has already
received, the other resource (which I haven't audited) certainly gets some
(if not all) of its requests (as it it solely responsible for persisting a
specific type of data to the database). It's all rather bizarre and points
to a thread-safety issue with the client. Either a bug or I'm using it
incorrectly (though not according to the docs I've read).
The Jersey client can have Apache's HttpClient 3 plugged in which may be
worth a shot. There's an experimental extension for HttpClient 4. At any
rate, I need to gather more info to pass on to the Jersey folks. Best
download the Jersey source. Sorry if I've disturbed you here!
--
View this message in context: http://old.nabble.com/Tomcat-appears-to-be-sending-duplicate-requests-tp32019702p32024536.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat appears to be sending duplicate requests
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Peter,
On 7/8/2011 12:29 PM, peterjca wrote:
> Taking the web service request for the computation step out of my
> harness results in my app behaving correctly. Some of the computation
> web service requests are randomly being routed to the wrong
> end-point! The bug lies somewhere in Jersey...
Now you just need a test harness for your test harness. :)
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk4XYeQACgkQ9CaO5/Lv0PD//ACguRj6ManZzQYduYwiqPwFfj96
aNYAnieGDS1dQlpOBQcMhHW3CYB7fdU+
=YSW1
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat appears to be sending duplicate requests
Posted by peterjca <p....@monitormedia.co.uk>.
Thanks for your response. Yes, I was a bit sloppy with my "sending requests"
language. Also, I'm not saying that third party libraries used by my code
aren't to blame. Certainly something in between obviously is... ...and I've
just found out what.
There are two web service endpoints being used by the test harness. One
deals with the data counting, the other computes a result once my app has
received all that it needs.
Taking the web service request for the computation step out of my harness
results in my app behaving correctly. Some of the computation web service
requests are randomly being routed to the wrong end-point! The bug lies
somewhere in Jersey...
--
View this message in context: http://old.nabble.com/Tomcat-appears-to-be-sending-duplicate-requests-tp32019702p32022820.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat appears to be sending duplicate requests
Posted by André Warnier <aw...@ice-sa.com>.
peterjca wrote:
> It's running now on Tomcat 7. I've had duplicate requests so checked the
> access logs. All requests have 200 response code.
>
> :confused:
Hi.
I am not going to provide the solution, just intervening on the "form" aspect of the dialog.
You have obviously put a lot of effort in checking out what happens, and in phrasing your
original question. Bonus points for that.
This being said,
- Tomcat is not "sending duplicate requests to the webapp". Tomcat receives requests, and
for each request it "runs" the corresponding webapp. The "sending requests" bit is a bit
confusing, and since the rest of your communication is quite precise, it would be better
to continue in that spirit.
- Seen in the absolute, it is not impossible that there is a bug in Tomcat. But it is
relatively unlikely that Tomcat would receive one HTTP request, and then run the
corresponding webapp twice with the same request.
It is even more unlikely that two different versions of Tomcat would be doing that
(because with the thousands of Tomcats in use, someone would have noticed this earlier).
This is why the people answering so far obviously tend to look for some other explanation
to what you think you are seeing. They may be wrong, but "extraordinary claims need to be
backed by extraordinary evidence".
- What Konstantin was saying about authentication, and which maybe you overlooked, was
that when authentication is involved, the normal cycle of events is :
- the client sends a first HTTP request, without authentication
- the server sends back a response saying "this needs authentication"
- the client then sends *the same request* again, this time with some form of
authentication
- this time, the server accepts the request
So, depending on how your client (your test application) exactly works, and the type of
authentication used, and on where exactly you are keeping track of the requests and
executed webapps, you may (or may not) be miscounting the actual number of HTTP
requests/responses.
Since we do not have access to your code and cannot inspect it in the same detail as
yourself, we are asking you for some precise details, and hoping that you are not
providing answers without actually checking them carefully.
You may for instance want to disable the authentication part (if possible), and redo your
tests, to make sure that the discrepancy does not originate there.
- In your test client application, to send the requests, you are using several layers of
existing Java or other HTTP-related and TCP-related code. One of these layers, under
particular circumstances, may be repeating some requests, without your application even
knowing about it. Granted, this is unlikely, but it is no more unlikely than Tomcat
executing the same request twice.
That is why Pid was recommending that you use some form of protocol analyser (which is
independent from both your test app and from Tomcat), to make sure that, really, the
requests are only being sent one time over the connection.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat appears to be sending duplicate requests
Posted by peterjca <p....@monitormedia.co.uk>.
It's running now on Tomcat 7. I've had duplicate requests so checked the
access logs. All requests have 200 response code.
:confused:
--
View this message in context: http://old.nabble.com/Tomcat-appears-to-be-sending-duplicate-requests-tp32019702p32020404.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat appears to be sending duplicate requests
Posted by Pid <pi...@pidster.com>.
On 08/07/2011 11:07, peterjca wrote:
>
> Yes, I do mean duplicate requests. I know they are duplicates because the
> header contains the unique (thread-safe) counter that my test harness
> increments. Each request should contain its unique "request count", but some
> requests contain counts already seen before -- hence they are being
> duplicated/resent for some reason by something which is not my web app nor
> my test harness.
>
> I maintain a unique thread-safe count in the test harness. That shows the
> test is sending the correct number of requests.
I'm not sure that's a safe assumption. It most likely only shows that
the test is sending the right number of requests, rather than that the
IO component it relies on is performing correctly.
If you're saying that Tomcat is seeing the same headers in multiple
requests it points to the client resending the request.
Tomcat doesn't arbitrarily resend a request to itself if a timeout occurs.
Can you use a network sniffer to inspect the network traffic to Tomcat?
p
> On the server-side in the Jersey resource end-point, purely for debug
> purposes, I am using a bitset to maintain the counts already seen from the
> headers of the incoming requests. That demonstrates that I am receiving
> duplicate requests on the server.
>
> Everything is standalone on my machine. Nothing else is using the web app as
> it's a greenfield project under my sole development.
>
> It's the standard standalone Tomcat configuration.
>
> My web app uses 2-legged OAuth for authentication using a Jersey
> ResourceFilter. That has always worked and is not the cause of any failures.
>
> In fact, all of my Jersey client requests from the test harness return 200
> response codes.
>
> I'll run Tomcat 7 and check the access logs.
>
Re: Tomcat appears to be sending duplicate requests
Posted by peterjca <p....@monitormedia.co.uk>.
Yes, I do mean duplicate requests. I know they are duplicates because the
header contains the unique (thread-safe) counter that my test harness
increments. Each request should contain its unique "request count", but some
requests contain counts already seen before -- hence they are being
duplicated/resent for some reason by something which is not my web app nor
my test harness.
I maintain a unique thread-safe count in the test harness. That shows the
test is sending the correct number of requests.
On the server-side in the Jersey resource end-point, purely for debug
purposes, I am using a bitset to maintain the counts already seen from the
headers of the incoming requests. That demonstrates that I am receiving
duplicate requests on the server.
Everything is standalone on my machine. Nothing else is using the web app as
it's a greenfield project under my sole development.
It's the standard standalone Tomcat configuration.
My web app uses 2-legged OAuth for authentication using a Jersey
ResourceFilter. That has always worked and is not the cause of any failures.
In fact, all of my Jersey client requests from the test harness return 200
response codes.
I'll run Tomcat 7 and check the access logs.
--
View this message in context: http://old.nabble.com/Tomcat-appears-to-be-sending-duplicate-requests-tp32019702p32020307.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat appears to be sending duplicate requests
Posted by Konstantin Kolinko <kn...@gmail.com>.
2011/7/8 Peter Cameron <p....@monitormedia.co.uk>:
> I'm having an interesting problem which seems to be Tomcat-related and which
> occurs randomly under heavy load. I'd appreciate some advice/help.
>
> I'm developing a Java web application and deploying to Tomcat 6.0.32. (I get
> the same behaviour with the latest Tomcat 7).
>
>(...)
> Why is Tomcat sending duplicate requests? Is there some timeout that is
> silently happening under load and causing Tomcat to resend the request? Are
> there any JMX diagnostics I can look at/turn on for Tomcat? Something is
> sending those duplicate requests and it isn't me!
You mean, receiving duplicate requests?
Is there anything in the access logs? (In TC7 they are turned on by
default, so maybe you already have them). Look for response codes
that are not equal to "200".
How do you count - in a servlet or in a filter? Do you use
authentication? (If server requests authentication credentials, a web
client will repeat its request, now with credentials).
Is Tomcat running standalone, or behind something? What connectors are
you using (Bio/Nio/Apr -- see logs during startup).
Best regards,
Konstantin Kolinko
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org