You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Marvin Lillehaug <Ma...@kantega.no> on 2013/08/07 12:23:02 UTC
[users@httpd] Responses of two different requests concatinated
Hi!
We recently got an error report from a user of one of the systems we have developed, showing that the response from a different request had been appended to the original response.
The original response was the front page of a site, generated with jsp, and the appended response was a excel file generated a few seconds earlier.
Our current hypothesis is that some buffer in either httpd(2.2.22) or Tomcat (7.0.35) has been recycled.
Httpd is connected to Tomcat using http proxypass.
I have started trying to reproduce the problem, but thought I should try the mailing lists of both httpd and tomcat before continuing.
What I have done thus far is: concatenating html and excel to verify that it is possible to open and looks the way the user experienced; building a custom version of Tomcat that uses the same Processor for each request and configured to use only one thread.
This seems a bit similar to the issue described in http://tomcat.apache.org/security-7.html#Fixed_in_Apache_Tomcat_7.0.12 (CVE-2011-1475)
Some results when googleling suggests that this could happen when jsp tags are not coded properly, but I have not found any such code in our applications.
Does anyone have any ideas or suggestions?
Thanks and regards,
Marvin B. Lillehaug
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: Responses of two different requests concatinated
Posted by Mark Thomas <ma...@apache.org>.
On 07/08/2013 12:23, Marvin Lillehaug wrote:
> Hi!
> We recently got an error report from a user of one of the systems we have developed, showing that the response from a different request had been appended to the original response.
> The original response was the front page of a site, generated with jsp, and the appended response was a excel file generated a few seconds earlier.
> Our current hypothesis is that some buffer in either httpd(2.2.22) or Tomcat (7.0.35) has been recycled.
> Httpd is connected to Tomcat using http proxypass.
>
> I have started trying to reproduce the problem, but thought I should try the mailing lists of both httpd and tomcat before continuing.
> What I have done thus far is: concatenating html and excel to verify that it is possible to open and looks the way the user experienced; building a custom version of Tomcat that uses the same Processor for each request and configured to use only one thread.
>
> This seems a bit similar to the issue described in http://tomcat.apache.org/security-7.html#Fixed_in_Apache_Tomcat_7.0.12 (CVE-2011-1475)
> Some results when googleling suggests that this could happen when jsp tags are not coded properly, but I have not found any such code in our applications.
>
> Does anyone have any ideas or suggestions?
In order of likelihood:
- app bug
- Tomcat bug
- httpd bug
I'd look for code that retains a reference to the request and/or
response object or maybe an OutputStream. The usual cause of this type
of issue is retaining a reference across requests and re-using the
object from the old request rather than the current one.
Did the appended response include HTTP headers? If yes, this could just
be the result of pipe-lining.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Responses of two different requests concatinated
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Marvin,
On 8/7/13 6:23 AM, Marvin Lillehaug wrote:
> We recently got an error report from a user of one of the systems
> we have developed, showing that the response from a different
> request had been appended to the original response. The original
> response was the front page of a site, generated with jsp, and the
> appended response was a excel file generated a few seconds
> earlier.
> Our current hypothesis is that some buffer in either httpd(2.2.22)
> or Tomcat (7.0.35) has been recycled.
Note that the Tomcat documentation uses the word "recycled" with
respect to connections and façades to mean "not re-used". You appear
to be using the term "recycled" to mean "re-used". I just wanted to be
clear because there is a related system property that uses the term
"recycle" (see below).
> Httpd is connected to Tomcat using http proxypass.
First, you should probably upgrade Tomcat to the latest 7.0 version
which is 7.0.42. Is that a possibility?
Second, you should post your whole <Connector> configuration from
Tomcat (minus any sensitive information of course).
If you want a possible quick-fix, you should look at the
RECYCLE_FACADES system property documented here:
http://tomcat.apache.org/tomcat-7.0-doc/config/systemprops.html#Security
If you set that system property to "on", then potential
response-leakages should be significantly reduced at a slight cost in
terms of heap and GC activity (but fairly negligible, as request
façades shouldn't really be surviving any minor collections... they
will just cause minor collections to occur more often).
> I have started trying to reproduce the problem, but thought I
> should try the mailing lists of both httpd and tomcat before
> continuing.
While that's okay, we generally prefer not to cross-post. Here, I have
replied only to the Tomcat users' mailing list as my reply is
Tomcat-specific.
> What I have done thus far is: concatenating html and excel to
> verify that it is possible to open and looks the way the user
> experienced; building a custom version of Tomcat that uses the same
> Processor for each request and configured to use only one thread.
>
> This seems a bit similar to the issue described in
> http://tomcat.apache.org/security-7.html#Fixed_in_Apache_Tomcat_7.0.12
>
>
(CVE-2011-1475)
This is not likely to be your problem, but we'll see when you post
your configuration.
> Some results when googleling suggests that this could happen when
> jsp tags are not coded properly, but I have not found any such code
> in our applications.
Any references? I'm not familiar with any suggestions that JSP tags
are leaky in any way. Are you using any JSPs in these transactions
that appear to have been mixed-up? I'd be surprised to find a JSP that
produces an XLS document.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.14 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJSAsXsAAoJEBzwKT+lPKRYoBoQALBjqpchPLbwCKpbh2WFcMVY
k++Xc4CopqNwe4uF0PDkU3JHzfWJUIsqZzF1mijOGIwt2EgwQKEZigCyJUv86NTZ
WG2SAc6DupNYwky0i4NnJE2F+VBcdUvTxeQ4LS0ATzr1sNqrmtQxdCbU/VeyAmyb
u5UCmFZiKvVAaepfAEzchTGTUPNqE7Xp57atPfS+EAGwLguE3ARksI9ipEBkOznm
ukRVJBTnrDuQ2/uMsAGW+miWyIeFx64WHGFWXBv/21K3GaSAN2Ut83OWCxCIO7Xr
vZPBtqzFlsfvcjWX3949Zoz0kvNZMCdMkjZ9G7cQhCQT6hWFToqdnXYkQ6U2aqkD
LIryyNiIP5zQFrJOKFKtexyugVNWBcsil4549Wwgi65rmgUeZ8qlEgr1wK4hgvKV
GG+O83kzrxMA4cyyMpgItFIset3+MccCoGHVLQg51afNoDlSfn9pAL8uLZJQW2FT
Prq9r5vD9qu5XyIFUJsHGXGTwWvkn4OPJhcITZsAeBKJxZyYhDNhqjj19fae+o+a
wgmi8XzoDqzCemefOEVXXVidaK0ceuhqUnZF729NavLZV8OUHaPItvS/5jQ4bSBU
BD/UnoDXPL8ZFzPw1WTj1ZotxjDZEAkjnZDUsADfHt2zBtWCxNyizdyIao0B0NAA
j6W/wT1bhYHVZ1eBJR2a
=oSt1
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Responses of two different requests concatinated
Posted by Marvin Lillehaug <Ma...@kantega.no>.
Thank you both for your replies :)
I only have a screenshot of how it looked to the user, so I don't know whether the headers was included.
I have given up trying to investigate further, so I guess the only thing to do is to activate RECYCLE_FACADES and hope for the best.
-----Original Message-----
From: Konstantin Kolinko [mailto:knst.kolinko@gmail.com]
Sent: Wednesday, August 07, 2013 12:35 PM
To: Tomcat Users List
Cc: users@httpd.apache.org
Subject: Re: Responses of two different requests concatinated
2013/8/7 Marvin Lillehaug <Ma...@kantega.no>:
> Hi!
> We recently got an error report from a user of one of the systems we have developed, showing that the response from a different request had been appended to the original response.
> The original response was the front page of a site, generated with jsp, and the appended response was a excel file generated a few seconds earlier.
> Our current hypothesis is that some buffer in either httpd(2.2.22) or Tomcat (7.0.35) has been recycled.
> Httpd is connected to Tomcat using http proxypass.
>
> I have started trying to reproduce the problem, but thought I should try the mailing lists of both httpd and tomcat before continuing.
> What I have done thus far is: concatenating html and excel to verify that it is possible to open and looks the way the user experienced; building a custom version of Tomcat that uses the same Processor for each request and configured to use only one thread.
>
> This seems a bit similar to the issue described in
> http://tomcat.apache.org/security-7.html#Fixed_in_Apache_Tomcat_7.0.12 (CVE-2011-1475) Some results when googleling suggests that this could happen when jsp tags are not coded properly, but I have not found any such code in our applications.
>
> Does anyone have any ideas or suggestions?
>
Usual culprit is a bug in web application that uses request/response objects outside of their life cycle.
The first step that I'd recommend is to set org.apache.catalina.connector.RECYCLE_FACADES=true
for better security and to ease detection of such misuse.
See
http://tomcat.apache.org/tomcat-7.0-doc/security-howto.html#System_Properties
http://tomcat.apache.org/tomcat-7.0-doc/config/systemprops.html
There is also exists a known issue in Java ImageIO API, https://wiki.apache.org/tomcat/FAQ/KnownIssues#ImageIOIssues
There also exists CVE-2013-2071 (fixed in 7.0.40).
Best regards,
Konstantin Kolinko
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Responses of two different requests concatinated
Posted by Konstantin Kolinko <kn...@gmail.com>.
2013/8/7 Marvin Lillehaug <Ma...@kantega.no>:
> Hi!
> We recently got an error report from a user of one of the systems we have developed, showing that the response from a different request had been appended to the original response.
> The original response was the front page of a site, generated with jsp, and the appended response was a excel file generated a few seconds earlier.
> Our current hypothesis is that some buffer in either httpd(2.2.22) or Tomcat (7.0.35) has been recycled.
> Httpd is connected to Tomcat using http proxypass.
>
> I have started trying to reproduce the problem, but thought I should try the mailing lists of both httpd and tomcat before continuing.
> What I have done thus far is: concatenating html and excel to verify that it is possible to open and looks the way the user experienced; building a custom version of Tomcat that uses the same Processor for each request and configured to use only one thread.
>
> This seems a bit similar to the issue described in http://tomcat.apache.org/security-7.html#Fixed_in_Apache_Tomcat_7.0.12 (CVE-2011-1475)
> Some results when googleling suggests that this could happen when jsp tags are not coded properly, but I have not found any such code in our applications.
>
> Does anyone have any ideas or suggestions?
>
Usual culprit is a bug in web application that uses request/response
objects outside of their life cycle.
The first step that I'd recommend is to set
org.apache.catalina.connector.RECYCLE_FACADES=true
for better security and to ease detection of such misuse.
See
http://tomcat.apache.org/tomcat-7.0-doc/security-howto.html#System_Properties
http://tomcat.apache.org/tomcat-7.0-doc/config/systemprops.html
There is also exists a known issue in Java ImageIO API,
https://wiki.apache.org/tomcat/FAQ/KnownIssues#ImageIOIssues
There also exists CVE-2013-2071 (fixed in 7.0.40).
Best regards,
Konstantin Kolinko
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org