You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by "Roldan, Miguel Angel [Global Data & Content]" <Mi...@solera.com> on 2016/11/03 16:52:16 UTC
Apache tomcat concurrency issue
Hi,
We are facing some random concurrency issues in our tomcat environments.
1. We have a web application deployed on Tomcat that handles requests from browsers (AJAX and non-AJAX) and also from an applet.
2. The requests send by the browser and always gets and posts (with application/x-www-form-urlencoded format)
3. The requests made by the applet are compressed and send in binary, they are handled by a different Servlet that reads the content by executing request.getInputStream().
4. Randomly, mainly after opening the applet from the current session or in another session executed on the same server (which send some requests in parallel) requests send from the browser arrives to the server with an empty request.getParameterMap()
5. We have reviewed that the request was well formed when it leaves the browser and also we are able to monitor it with TCP Dump and see that arrives correctly to tomcat.
6. We have introduced a filter on the first position of the filter chain: org.apache.catalina.filters.RequestDumperFilter and we have validated that at this point we have already lost all of the parameters
7. If we try to execute the request.getInputStream() for this problematic requests what we see is that the byte array that returns belongs to another request made before to the server.
8. If we re-submit the same request after, it works fine. They don't have any kind of special character or exceed a certain length.
9. Environment configuration:
Httpd apache server on the front-end v. 2.2 connected to apache tomcat using AJP connector 1.3
Apache tomcat 7.0.56
O.S. - Oracle linux
10. Sometimes we are able to reproduce the issue on a eclipse-tomcat environment using http connector and we have validated that the flag used by the class org.apache.catalina.connector.Request parametersParsed is set to true.
Do you have any kind of suggestions on how to further investigate the issue?
Many thanks and regards,
Miguel A. Roldán.
________________________________
"This e-mail, any associated files and the information contained in them are confidential and is intended for the addressee(s) only. If you have received this message in error please notify the originator and delete the email immediately. The unauthorised use, disclosure, copying or alteration of this message is strictly forbidden. E-mails to and from the company are monitored for operational reasons and in accordance with lawful business practices. Any opinions expressed are those of the individual and do not necessarily represent the views of the company. The company does not conclude contracts by email and all negotiations are subject to contract. We make every effort to maintain our network free from computer viruses but accept no responsibility for any viruses which might be transferred by this e-mail."
Re: Apache tomcat concurrency issue
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Migual Angel,
On 11/10/16 5:51 AM, Roldan, Miguel Angel [Global Data & Content] wrote:
> Thanks for your help. After a lot of tests we finally realized it
> was related with having tomcat configured to recycle requests and
> using apache velocity.
I use Apache Velocity and have no such concurrency issues. Note that
"recycling requests" doesn't mean what you might think it does: in
Tomcat, "recycling request facades" means that the facades are discarded
.
> We modify the configuration RECYCLE_FACADES setting it to true and
> the problem disappear.
Then the problem is with your application retaining references to
request and/or response objects after the request should have completed.
99% of the time this is the reason for reports of the kind you gave.
This is not a bug in Tomcat, nor is it a bug in Velocity. It's a bug
in your application.
- -chris
> -----Original Message----- From: Christopher Schultz
> [mailto:chris@christopherschultz.net] Sent: viernes, 04 de
> noviembre de 2016 23:26 To: Tomcat Users List
> <us...@tomcat.apache.org> Subject: Re: Apache tomcat concurrency
> issue
>
> Miguel Angel,
>
> On 11/3/16 12:52 PM, Roldan, Miguel Angel [Global Data & Content]
> wrote:
>
> (Moving this up to the top:)
>
>> Httpd apache server on the front-end v. 2.2 connected to apache
>> tomcat using AJP connector 1.3
>
>> Apache tomcat 7.0.56
>
> Thanks.
>
>> We are facing some random concurrency issues in our tomcat
>> environments.
>
>
>> 1. We have a web application deployed on Tomcat that handles
>> requests from browsers (AJAX and non-AJAX) and also from an
>> applet.
>
>> 2. The requests send by the browser and always gets and
>> posts (with application/x-www-form-urlencoded format)
>
>> 3. The requests made by the applet are compressed and send
>> in binary, they are handled by a different Servlet that reads
>> the content by executing request.getInputStream().
>
> Are you compressing and sending a big blob called "data" or
> something similar? Or are you sending multiple parameters
> separately compressed?
>
>> 4. Randomly, mainly after opening the applet from the
>> current session or in another session executed on the same server
>> (which send some requests in parallel) requests send from the
>> browser arrives to the server with an empty
>> request.getParameterMap()
>
> If you are calling request.getInputStream, then you will never get
> anything in the parameter map. If your code consumes the request
> body in a POST, then Tomcat cannot parse it again.
>
>> 5. We have reviewed that the request was well formed when it
>> leaves the browser and also we are able to monitor it with TCP
>> Dump and see that arrives correctly to tomcat.
>
> Good.
>
>> 6. We have introduced a filter on the first position of the
>> filter chain: org.apache.catalina.filters.RequestDumperFilter
>> and we have validated that at this point we have already lost all
>> of the parameters
>
> Good thinking. Are you SURE it's the first filter in the chain? We
> aware that the RequestDumperValve will rob the data from your
> applicatio n.
>
>> 7. If we try to execute the request.getInputStream() for
>> this problematic requests what we see is that the byte array
>> that returns belongs to another request made before to the
>> server.
>
> Obviously, that shouldn't happen.
>
>> 8. If we re-submit the same request after, it works fine.
>> They don't have any kind of special character or exceed a
>> certain length.
>
>> 9. Environment configuration:
>
>
>> O.S. - Oracle linux
>
>
>
>> 10. Sometimes we are able to reproduce the issue on a
>> eclipse-tomcat environment using http connector and we have
>> validated that the flag used by the class
>> org.apache.catalina.connector.Request parametersParsed is set to
>> true.
>
>
>
>> Do you have any kind of suggestions on how to further
>> investigate the issue?
>
> Are you able to re-test with an updated version of Tomcat 7?
> Current version is 7.0.72. Your version is ~2 years old.
>
> Can you post your <Connector> configuration?
>
> Thoughts about what might be happening:
>
> 1. Application concurrency/request-object retention bug 2.
> maxPostSize value too low -- Tomcat is rejecting the request in a
> way unexpected by the application 3. Edge case/bug in Tomcat that
> has been fixed in a more-recent version?
>
> -chris
>
> ---------------------------------------------------------------------
>
>
>
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ________________________________
>
> "This e-mail, any associated files and the information contained
> in them are confidential and is intended for the addressee(s) only.
> If you have received this message in error please notify the
> originator and delete the email immediately. The unauthorised use,
> disclosure, copying or alteration of this message is strictly
> forbidden. E-mails to and from the company are monitored for
> operational reasons and in accordance with lawful business
> practices. Any opinions expressed are those of the individual and
> do not necessarily represent the views of the company. The company
> does not conclude contracts by email and all negotiations are
> subject to contract. We make every effort to maintain our network
> free from computer viruses but accept no responsibility for any
> viruses which might be transferred by this e-mail."
>
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJYJIepAAoJEBzwKT+lPKRY0ncQAMXX2vsxP92jZ1YK+wP7uEFw
s/TA3ZOBUu8QR1wg75H6o8/JnjYYTo7C/jrfHQTmgbFVkiJdMq8mA3F6HNgFc7bA
upm7M13vK42JR/UdxrjaH5ae3mrkbX2z7Xj5SlOWt2PqwGyO0GYmt5TJe4wooKMD
nHkfgiS/TIvzCaH2KwbfC8Z/dRjj7Jc4woXA3wOFYUj40n99dzwsGwS8i2pA9Q5c
SywNLGVU7uIaiiHzznj+HITyfAEJ3ZQZllXt/o4+WFBIm+Im4nUY2TbXQdbWwEE9
pmTGFPKTzpUJblHNzw+1jCtIDf5SaGvRzlrt9AHkdT4TBZIoMeSUVziepWnvYWm/
LvXk3JbKxoF7hMcghJ6OQAlI29QXdKu4qmENZG4hkgtOH/HPtL/+YhFi0i66xvxo
EQ49/5lJu+Jmi91OeECxwwxISKNf/ZqcC1gA+cVEyqvLSPfIOhMa1f83BD+tXRzM
t6PNWn6itGjwULXYCxC4dgpKYl2FJh69DJYuSY8UUzXHXDcUXUDbU/UapRQ5i88X
K5vwJnnY/jrJZjdBYzNmFEqgfDLySNewusrL3jm6FpTEPQrjYykuQw2X6tahte/o
frph/Cg+E8bSbAxAMRtJubL4z5ux9R83zGEGLI9d9XwZ2wZgOKhb+VWOwRRxu39n
8hFLu7YdgW5dAWNwpcG5
=B8sv
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Apache tomcat concurrency issue
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Miguel Angel,
On 11/3/16 12:52 PM, Roldan, Miguel Angel [Global Data & Content] wrote:
(Moving this up to the top:)
> Httpd apache server on the front-end v. 2.2 connected to apache
> tomcat using AJP connector 1.3
>
> Apache tomcat 7.0.56
Thanks.
> We are facing some random concurrency issues in our tomcat
> environments.
>
>
> 1. We have a web application deployed on Tomcat that handles
> requests from browsers (AJAX and non-AJAX) and also from an
> applet.
>
> 2. The requests send by the browser and always gets and posts
> (with application/x-www-form-urlencoded format)
>
> 3. The requests made by the applet are compressed and send in
> binary, they are handled by a different Servlet that reads the
> content by executing request.getInputStream().
Are you compressing and sending a big blob called "data" or something
similar? Or are you sending multiple parameters separately compressed?
> 4. Randomly, mainly after opening the applet from the current
> session or in another session executed on the same server (which
> send some requests in parallel) requests send from the browser
> arrives to the server with an empty request.getParameterMap()
If you are calling request.getInputStream, then you will never get
anything in the parameter map. If your code consumes the request body
in a POST, then Tomcat cannot parse it again.
> 5. We have reviewed that the request was well formed when it
> leaves the browser and also we are able to monitor it with TCP Dump
> and see that arrives correctly to tomcat.
Good.
> 6. We have introduced a filter on the first position of the
> filter chain: org.apache.catalina.filters.RequestDumperFilter and
> we have validated that at this point we have already lost all of
> the parameters
Good thinking. Are you SURE it's the first filter in the chain? We
aware that the RequestDumperValve will rob the data from your applicatio
n.
> 7. If we try to execute the request.getInputStream() for this
> problematic requests what we see is that the byte array that
> returns belongs to another request made before to the server.
Obviously, that shouldn't happen.
> 8. If we re-submit the same request after, it works fine. They
> don't have any kind of special character or exceed a certain
> length.
>
> 9. Environment configuration:
>
>
> O.S. - Oracle linux
>
>
>
> 10. Sometimes we are able to reproduce the issue on a
> eclipse-tomcat environment using http connector and we have
> validated that the flag used by the class
> org.apache.catalina.connector.Request parametersParsed is set to
> true.
>
>
>
> Do you have any kind of suggestions on how to further investigate
> the issue?
Are you able to re-test with an updated version of Tomcat 7? Current
version is 7.0.72. Your version is ~2 years old.
Can you post your <Connector> configuration?
Thoughts about what might be happening:
1. Application concurrency/request-object retention bug
2. maxPostSize value too low -- Tomcat is rejecting the request in a
way unexpected by the application
3. Edge case/bug in Tomcat that has been fixed in a more-recent version?
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJYHQsTAAoJEBzwKT+lPKRYUMgP/3vpg67S/g8jGkdNbWz6PhjP
/MJjzV8RnThF+NladYyQrJLjw+29eQ9MiQozf4YaXVC5NOfiYKy0kKz1QPw+Cz8q
Mqc0ExXYkvjX1/0Op2mlkA2GY9OU976VrSFxE0yFqUomWswtar/MCmbB++fqGWdS
8xE8y62ZuW1gi0EoXU/9UiTqz3AI+ni6vxvXOFuTbzES/qudwLe5ojShHVA8DiIB
zityp3gXYvTU0QDPYcVbJBP1PNGGlgqhW0cT4jlujztb8pOieLSpMYb1o1jY00SX
Z2E781GCl63H1avQFr4tNNsRkDn7J2Xlsg0X7ixDyeJ3RDrNG+2WjYs8XoQ8WSEI
pQ1JSCKa8EVjY87jQFocsI/DH3y9LhiMqo+80EWG3tE932JiTcqxfi0ZamPWF0qn
w/fotttCaH638QWQBc4eQErDEFsvBDi0uDP0+lergEptjvduovXmjYTmD0SRtoPM
BdmZNGXdIpQtldhvnP0LwhrnKlryVw+Zo5MB3a2IelwxLAAH8BmDYAyRmKc3RTDH
lPa9gKWLPdq5NrTKpXqSSbrq0UriEiRpZ66XyBe5IKFc1tX+AvJtChj0ss2wQdKY
6WMPoJsKAiUXii6eYJ9uVSEDfvF1LJr31VyQENHOvN7elRxkvo963XQ6esOTSAbd
jEqvpcdkgyz9dvo240kT
=Ycbm
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org