You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Ya...@finnova.ch on 2014/07/09 09:43:23 UTC
Tomcat (v 6.0.32) write response to wrong client socket
Hello,
We got a serious problem on our online banking applications: a user U1 of bank A got to see the data of another user U2 of another bank B.
It happened only once, bevor and after that went everything well.
The two same online banking applications are running on one instance of tomcat (V.6.0.32) and have different backend path.
The logs show that the problem muss be not on the side of web application (frondend), backend and firewall. The senario looks as following:
1. U1 log on the online banking of bank A and U2 log on the online banking of
bank B, and the two sessions have been running well a while till
2. later at the exactly same time U1 wanted to see the start page and U2
booking
detail
3. U1 got to see the false data of the booking detail of U2.
4. after that the two sessions went further well without any problem
In the access log of the tomcat seems everything went well:
Access log Bank A (U1):
10.25.4.8 - - F862B9AD5DA9AC4A0D38B46D4E5B0D6C [15/Jun/2014:16:03:41 +0200]
HTTP/1.1 GET /finprdcbo/defAccountStartPage.account
?DIRTY=Y&DEFAULT=1&node=STARTSEITE 200 69000 /ebanking/defAccountStatementDetail.account Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:29.0) Gecko/20100101 Firefox/29.0
Access log Bank B (U2):
10.25.4.8 - - 12A6E21F0D6321A95C553B160DBCC9A0 [15/Jun/2014:16:03:40 +0200]
HTTP/1.1 POST /finprdzrb/defAccountStatementOfAccount.account 200 45765 /ebanking/defAccountAssetOverview.account Mozilla/5.0 (X11; Linux i686;
rv:24.0) Gecko/20100101 Firefox/24.0
But the reponse of the request for booking detail from U 2 was mistakenly sent to U1. We cann confirm this by looking into the logs of firewall as
following:
Firewall log of Session U2:
Jun 15 16:04:16 Web-Requests Access m:WR-SG-SUMMARY
vhost:wwwsec.zrb.clientis.ch:443 (https) POST /ebanking/defAccountStatementOfAccount.account => https://10.25.2.43:5215/finprdzrb/defAccountStatementOfAccount.account , status:<n/a> , redirection URL:<n/a> , referer:/ebanking/defAccountAssetOverview.account , mapping:blappl-zrb , request size: 1089 , backend response size: <n/a> , audit
token:308792478954626740 rid:U52nvH8AAAEAAAoy1CIAAAno
sid:384490af19eb229b7f0874b6ef0323c8 ip:84.74.211.190 12
Fact : "backend response size: <n/a>" means there is no reponse from tomcat for the request of POST /ebanking/defAccountStatementOfAccount.account and a timeout is trigged
Firewall log of Session U1:
Jun 15 16:03:40 Web-Requests Access SG_child[14145]: m:WR-SG-SUMMARY
vhost:wwwsec.oberuzwil.clientis.ch:443 (https) GET /ebanking/defAccountStartPage.account?DIRTY=Y&DEFAULT=1&node=STARTSEITE => https://10.25.2.43:5215/finprdcbo/defAccountStartPage.account , status:200 , redirection URL:<n/a> , referer:/ebanking/defAccountStatementDetail.account , mapping:blappl-cbo , request size: 614 , backend response size: 45765 , audit
token:147310158715306970 request total 287469 , allow/deny filters 3780 , backend responsiveness 208455 , response processing 74282 , ICAP reqmod <n/a> , ICAP respmod <n/a> rid:U52nvH8AAAEAAATYu6cAAATi sid:1dad4372b4a2f67d980e6e195aa954fe ip:84.73.20.65 12
Fact: "backend response size: 45765" means the reponse of Request of U2 is mistakenly passed to U1. See the same size of the response in the access log Bank B
(U2):
My questions:
1. Is such a problem (bug) already known?
2. When will the tomcat access log be written? after sent of response or
bevor?
3. how could the problem happen on the side of tomcat?
4. wo could the problem be hidden otherwise?
Thanks for your help !
Yanchun Yang
Re: Tomcat (v 6.0.32) write response to wrong client socket
Posted by "Igal @ getRailo.org" <ig...@getrailo.org>.
if Tomcat is fronted by a web server then I would also check the connector.
I've seen similar issues in the past with IIS and a faulty AJP connector.
switching to nginx with an APR connector resolved it in my case.
Igal
On 7/9/2014 2:49 AM, Mark Thomas wrote:
> On 09/07/2014 08:43, Yanchun.Yang@finnova.ch wrote:
>> Hello,
>>
>> We got a serious problem on our online banking applications: a user U1 of bank A got to see the data of another user U2 of another bank B.
>> It happened only once, before and after that went everything well.
>>
>> The two same online banking applications are running on one instance of tomcat (V.6.0.32) and have different backend path.
> That is quite an old version with a number of public security
> vulnerabilities to be using for a banking application.
>
>> The logs show that the problem must be not on the side of web application (frontend), backend and firewall. The scenario looks as following:
>> 1. U1 log on the online banking of bank A and U2 log on the online banking of
>> bank B, and the two sessions have been running well a while till
>> 2. later at the exactly same time U1 wanted to see the start page and U2
>> booking
>> detail
>> 3. U1 got to see the false data of the booking detail of U2.
>> 4. after that the two sessions went further well without any problem
>>
>> In the access log of the tomcat seems everything went well:
>>
>> Access log Bank A (U1):
>> 10.25.4.8 - - F862B9AD5DA9AC4A0D38B46D4E5B0D6C [15/Jun/2014:16:03:41 +0200]
>> HTTP/1.1 GET /finprdcbo/defAccountStartPage.account
>> ?DIRTY=Y&DEFAULT=1&node=STARTSEITE 200 69000 /ebanking/defAccountStatementDetail.account Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:29.0) Gecko/20100101 Firefox/29.0
>>
>> Access log Bank B (U2):
>> 10.25.4.8 - - 12A6E21F0D6321A95C553B160DBCC9A0 [15/Jun/2014:16:03:40 +0200]
>> HTTP/1.1 POST /finprdzrb/defAccountStatementOfAccount.account 200 45765 /ebanking/defAccountAssetOverview.account Mozilla/5.0 (X11; Linux i686;
>> rv:24.0) Gecko/20100101 Firefox/24.0
> These logs show the requests coming from the same IP address. Since the
> requests are from different users then there must be something acting as
> a reverse proxy. Is it the firewall? Is it some other component?
>
>> But the response of the request for booking detail from U2 was mistakenly sent to U1. We can confirm this by looking into the logs of firewall as
>> following:
>>
>> Firewall log of Session U2:
>> Jun 15 16:04:16 Web-Requests Access m:WR-SG-SUMMARY
>> vhost:wwwsec.zrb.clientis.ch:443 (https) POST /ebanking/defAccountStatementOfAccount.account => https://10.25.2.43:5215/finprdzrb/defAccountStatementOfAccount.account , status:<n/a> , redirection URL:<n/a> , referer:/ebanking/defAccountAssetOverview.account , mapping:blappl-zrb , request size: 1089 , backend response size: <n/a> , audit
>> token:308792478954626740 rid:U52nvH8AAAEAAAoy1CIAAAno
>> sid:384490af19eb229b7f0874b6ef0323c8 ip:84.74.211.190 12
>>
>> Fact : "backend response size: <n/a>" means there is no reponse from tomcat for the request of POST /ebanking/defAccountStatementOfAccount.account and a timeout is trigged
>>
>> Firewall log of Session U1:
>> Jun 15 16:03:40 Web-Requests Access SG_child[14145]: m:WR-SG-SUMMARY
>> vhost:wwwsec.oberuzwil.clientis.ch:443 (https) GET /ebanking/defAccountStartPage.account?DIRTY=Y&DEFAULT=1&node=STARTSEITE => https://10.25.2.43:5215/finprdcbo/defAccountStartPage.account , status:200 , redirection URL:<n/a> , referer:/ebanking/defAccountStatementDetail.account , mapping:blappl-cbo , request size: 614 , backend response size: 45765 , audit
>> token:147310158715306970 request total 287469 , allow/deny filters 3780 , backend responsiveness 208455 , response processing 74282 , ICAP reqmod <n/a> , ICAP respmod <n/a> rid:U52nvH8AAAEAAATYu6cAAATi sid:1dad4372b4a2f67d980e6e195aa954fe ip:84.73.20.65 12
>>
>> Fact: "backend response size: 45765" means the reponse of Request of U2 is mistakenly passed to U1. See the same size of the response in the access log Bank B
>> (U2):s
> The firewall appears to be transforming the URL. That suggests it is the
> firewall that is acting as the reverse proxy. (Aside: Transforming the
> URL like this is doable but does require very careful configuration to
> ensure everything still works properly)
>
>> My questions:
>>
>> 1. Is such a problem (bug) already known?
> Have you looked in the changelog?
> Mixing up responses would be a security issue so have you looked in
> http://tomcat.apache.org/security-6.html
>
>> 2. When will the tomcat access log be written? after sent of response or
>> before?
> It isn't quite that simple. In Tomcat 6 (it changes in 7.0.x) the access
> log is written after the application has finished writing the response
> to the buffer but before the content of that buffer is written to the
> client.
>
>> 3. how could the problem happen on the side of tomcat?
> Depending on exactly how the application works, CVE-2011-3375 might be
> able to trigger this but given that requires an error to occur it looks
> unlikely.
>
>
>> 4. how could the problem be hidden otherwise?
> Most errors of this type are eventually traced to application errors.
> The most frequent error is the application retaining a reference to the
> response and/or request objects beyond a single request.
>
> Mark
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
--
Igal Sapir
Railo Core Developer
http://getRailo.org/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
AW: Tomcat (v 6.0.32) write response to wrong client socket
Posted by Ya...@finnova.ch.
Hello Mark,
thank you for your answer. According to your informations I did some research as following:
1. I have looked for the key words of mix and response in http://tomcat.apache.org/security-6.html and in http://tomcat.apache.org/security-7.html,
And I got only one bug fix report of CVE-2011-1475 in this context. but it is not the same case like ours, because:
1. the bug is concerning BIO, but in our case it is NIO connector (Http11NioProtocol)
2. it 's Tomcat 7.X but we got the problem on Tomcat 6.0.32
2. CVE-2011-3375 is about information leakage when certain error occurs and therefor is not applicable to this case
3. I checked all of the filters and servlets in our application again and cannot find the case of retaining a reference to the response and/or request objects beyond a single request or any clues of thread unsafe.
The problem muss happed when the two requests income (income at the same time but response time various ). Is it somehow possible that tomcat mix them up ?
Any other suggestions?
Thank a lot for your effort!
Yanchun Yang
-----Ursprüngliche Nachricht-----
Von: Mark Thomas [mailto:]
Gesendet: Mittwoch, 9. Juli 2014 11:49
An: Tomcat Users List
Betreff: Re: Tomcat (v 6.0.32) write response to wrong client socket
On 09/07/2014 08:43, Yanchun.Yang@finnova.ch wrote:
> Hello,
>
> We got a serious problem on our online banking applications: a user U1 of bank A got to see the data of another user U2 of another bank B.
> It happened only once, before and after that went everything well.
>
> The two same online banking applications are running on one instance of tomcat (V.6.0.32) and have different backend path.
That is quite an old version with a number of public security vulnerabilities to be using for a banking application.
> The logs show that the problem must be not on the side of web application (frontend), backend and firewall. The scenario looks as following:
> 1. U1 log on the online banking of bank A and U2 log on the online banking of
> bank B, and the two sessions have been running well a while till
> 2. later at the exactly same time U1 wanted to see the start page and U2
> booking
> detail
> 3. U1 got to see the false data of the booking detail of U2.
> 4. after that the two sessions went further well without any problem
>
> In the access log of the tomcat seems everything went well:
>
> Access log Bank A (U1):
> 10.25.4.8 - - F862B9AD5DA9AC4A0D38B46D4E5B0D6C [15/Jun/2014:16:03:41
> +0200]
> HTTP/1.1 GET /finprdcbo/defAccountStartPage.account
> ?DIRTY=Y&DEFAULT=1&node=STARTSEITE 200 69000
> /ebanking/defAccountStatementDetail.account Mozilla/5.0 (Macintosh;
> Intel Mac OS X 10.9; rv:29.0) Gecko/20100101 Firefox/29.0
>
> Access log Bank B (U2):
> 10.25.4.8 - - 12A6E21F0D6321A95C553B160DBCC9A0 [15/Jun/2014:16:03:40
> +0200]
> HTTP/1.1 POST /finprdzrb/defAccountStatementOfAccount.account 200
> 45765 /ebanking/defAccountAssetOverview.account Mozilla/5.0 (X11;
> Linux i686;
> rv:24.0) Gecko/20100101 Firefox/24.0
These logs show the requests coming from the same IP address. Since the requests are from different users then there must be something acting as a reverse proxy. Is it the firewall? Is it some other component?
> But the response of the request for booking detail from U2 was
> mistakenly sent to U1. We can confirm this by looking into the logs
> of firewall as
> following:
>
> Firewall log of Session U2:
> Jun 15 16:04:16 Web-Requests Access m:WR-SG-SUMMARY
> vhost:wwwsec.zrb.clientis.ch:443 (https) POST
> /ebanking/defAccountStatementOfAccount.account =>
> https://10.25.2.43:5215/finprdzrb/defAccountStatementOfAccount.account
> , status:<n/a> , redirection URL:<n/a> ,
> referer:/ebanking/defAccountAssetOverview.account , mapping:blappl-zrb
> , request size: 1089 , backend response size: <n/a> , audit
> token:308792478954626740 rid:U52nvH8AAAEAAAoy1CIAAAno
> sid:384490af19eb229b7f0874b6ef0323c8 ip:84.74.211.190 12
>
> Fact : "backend response size: <n/a>" means there is no reponse from
> tomcat for the request of POST
> /ebanking/defAccountStatementOfAccount.account and a timeout is
> trigged
>
> Firewall log of Session U1:
> Jun 15 16:03:40 Web-Requests Access SG_child[14145]: m:WR-SG-SUMMARY
> vhost:wwwsec.oberuzwil.clientis.ch:443 (https) GET
> /ebanking/defAccountStartPage.account?DIRTY=Y&DEFAULT=1&node=STARTSEIT
> E => https://10.25.2.43:5215/finprdcbo/defAccountStartPage.account ,
> status:200 , redirection URL:<n/a> ,
> referer:/ebanking/defAccountStatementDetail.account ,
> mapping:blappl-cbo , request size: 614 , backend response size: 45765
> , audit
> token:147310158715306970 request total 287469 , allow/deny filters
> 3780 , backend responsiveness 208455 , response processing 74282 ,
> ICAP reqmod <n/a> , ICAP respmod <n/a> rid:U52nvH8AAAEAAATYu6cAAATi
> sid:1dad4372b4a2f67d980e6e195aa954fe ip:84.73.20.65 12
>
> Fact: "backend response size: 45765" means the reponse of Request of
> U2 is mistakenly passed to U1. See the same size of the response in
> the access log Bank B (U2):s
The firewall appears to be transforming the URL. That suggests it is the firewall that is acting as the reverse proxy. (Aside: Transforming the URL like this is doable but does require very careful configuration to ensure everything still works properly)
> My questions:
>
> 1. Is such a problem (bug) already known?
Have you looked in the changelog?
Mixing up responses would be a security issue so have you looked in http://tomcat.apache.org/security-6.html
> 2. When will the tomcat access log be written? after sent of response or
> before?
It isn't quite that simple. In Tomcat 6 (it changes in 7.0.x) the access log is written after the application has finished writing the response to the buffer but before the content of that buffer is written to the client.
> 3. how could the problem happen on the side of tomcat?
Depending on exactly how the application works, CVE-2011-3375 might be able to trigger this but given that requires an error to occur it looks unlikely.
> 4. how could the problem be hidden otherwise?
Most errors of this type are eventually traced to application errors.
The most frequent error is the application retaining a reference to the response and/or request objects beyond a single request.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat (v 6.0.32) write response to wrong client socket
Posted by Mark Thomas <ma...@apache.org>.
On 09/07/2014 08:43, Yanchun.Yang@finnova.ch wrote:
> Hello,
>
> We got a serious problem on our online banking applications: a user U1 of bank A got to see the data of another user U2 of another bank B.
> It happened only once, before and after that went everything well.
>
> The two same online banking applications are running on one instance of tomcat (V.6.0.32) and have different backend path.
That is quite an old version with a number of public security
vulnerabilities to be using for a banking application.
> The logs show that the problem must be not on the side of web application (frontend), backend and firewall. The scenario looks as following:
> 1. U1 log on the online banking of bank A and U2 log on the online banking of
> bank B, and the two sessions have been running well a while till
> 2. later at the exactly same time U1 wanted to see the start page and U2
> booking
> detail
> 3. U1 got to see the false data of the booking detail of U2.
> 4. after that the two sessions went further well without any problem
>
> In the access log of the tomcat seems everything went well:
>
> Access log Bank A (U1):
> 10.25.4.8 - - F862B9AD5DA9AC4A0D38B46D4E5B0D6C [15/Jun/2014:16:03:41 +0200]
> HTTP/1.1 GET /finprdcbo/defAccountStartPage.account
> ?DIRTY=Y&DEFAULT=1&node=STARTSEITE 200 69000 /ebanking/defAccountStatementDetail.account Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:29.0) Gecko/20100101 Firefox/29.0
>
> Access log Bank B (U2):
> 10.25.4.8 - - 12A6E21F0D6321A95C553B160DBCC9A0 [15/Jun/2014:16:03:40 +0200]
> HTTP/1.1 POST /finprdzrb/defAccountStatementOfAccount.account 200 45765 /ebanking/defAccountAssetOverview.account Mozilla/5.0 (X11; Linux i686;
> rv:24.0) Gecko/20100101 Firefox/24.0
These logs show the requests coming from the same IP address. Since the
requests are from different users then there must be something acting as
a reverse proxy. Is it the firewall? Is it some other component?
> But the response of the request for booking detail from U2 was mistakenly sent to U1. We can confirm this by looking into the logs of firewall as
> following:
>
> Firewall log of Session U2:
> Jun 15 16:04:16 Web-Requests Access m:WR-SG-SUMMARY
> vhost:wwwsec.zrb.clientis.ch:443 (https) POST /ebanking/defAccountStatementOfAccount.account => https://10.25.2.43:5215/finprdzrb/defAccountStatementOfAccount.account , status:<n/a> , redirection URL:<n/a> , referer:/ebanking/defAccountAssetOverview.account , mapping:blappl-zrb , request size: 1089 , backend response size: <n/a> , audit
> token:308792478954626740 rid:U52nvH8AAAEAAAoy1CIAAAno
> sid:384490af19eb229b7f0874b6ef0323c8 ip:84.74.211.190 12
>
> Fact : "backend response size: <n/a>" means there is no reponse from tomcat for the request of POST /ebanking/defAccountStatementOfAccount.account and a timeout is trigged
>
> Firewall log of Session U1:
> Jun 15 16:03:40 Web-Requests Access SG_child[14145]: m:WR-SG-SUMMARY
> vhost:wwwsec.oberuzwil.clientis.ch:443 (https) GET /ebanking/defAccountStartPage.account?DIRTY=Y&DEFAULT=1&node=STARTSEITE => https://10.25.2.43:5215/finprdcbo/defAccountStartPage.account , status:200 , redirection URL:<n/a> , referer:/ebanking/defAccountStatementDetail.account , mapping:blappl-cbo , request size: 614 , backend response size: 45765 , audit
> token:147310158715306970 request total 287469 , allow/deny filters 3780 , backend responsiveness 208455 , response processing 74282 , ICAP reqmod <n/a> , ICAP respmod <n/a> rid:U52nvH8AAAEAAATYu6cAAATi sid:1dad4372b4a2f67d980e6e195aa954fe ip:84.73.20.65 12
>
> Fact: "backend response size: 45765" means the reponse of Request of U2 is mistakenly passed to U1. See the same size of the response in the access log Bank B
> (U2):s
The firewall appears to be transforming the URL. That suggests it is the
firewall that is acting as the reverse proxy. (Aside: Transforming the
URL like this is doable but does require very careful configuration to
ensure everything still works properly)
> My questions:
>
> 1. Is such a problem (bug) already known?
Have you looked in the changelog?
Mixing up responses would be a security issue so have you looked in
http://tomcat.apache.org/security-6.html
> 2. When will the tomcat access log be written? after sent of response or
> before?
It isn't quite that simple. In Tomcat 6 (it changes in 7.0.x) the access
log is written after the application has finished writing the response
to the buffer but before the content of that buffer is written to the
client.
> 3. how could the problem happen on the side of tomcat?
Depending on exactly how the application works, CVE-2011-3375 might be
able to trigger this but given that requires an error to occur it looks
unlikely.
> 4. how could the problem be hidden otherwise?
Most errors of this type are eventually traced to application errors.
The most frequent error is the application retaining a reference to the
response and/or request objects beyond a single request.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org