You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Chris Dumoulin <ch...@blaze.io> on 2011/04/13 13:15:27 UTC
Problem with merged responses - possibly a reused Response or OutputBuffer
I'm seeing an intermittent problem with my webapp where a request is sent and the response contains 8184 bytes from some other response followed by the correct response.
The setup being used is Nginx 0.8.54 reverse proxying to Tomcat 7.0.11. AJP is the protocol between Nginx and Tomcat.
The webapp in Tomcat is doing Servlet 3.0 async requests.
This issue is extremely difficult to reproduce and at this point I'm not sure if the problem is in the webapp, Tomcat, or Nginx.
I know that 8184 bytes is the size of an AJP packet, and in Tomcat's org.apache.catalina.connector.Response, I see the following code:
if("AJP/1.3".equals(connector.getProtocol())) {
// default size to size of one ajp-packet
outputBuffer = new OutputBuffer(8184);
}
So, right now I'm following the theory that something is being reused in Tomcat without having been properly completed or recycled. Obviously it's most likely that this is an application bug.
Does anyone have any ideas about what kind of problem in the application could cause this behaviour, or other ideas about what the cause might be?
Thanks,
Chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Problem with merged responses - possibly a reused Response or OutputBuffer
Posted by Chris Dumoulin <ch...@blaze.io>.
It turns out the problem was with the AsyncContext timing out and then writing to the HttpServletResponse afterward.
Be sure to use an AsyncListener to be notified of these events.
Hopefully this might help someone with a similar problem in the future.
- Chris
On April 13, 2011 11:12:30 am Chris Dumoulin wrote:
> In a discussion on Bug 50957 (https://issues.apache.org/bugzilla/show_bug.cgi?id=50957) Mark Thomas said the following:
>
> "Experience has shown that most instances of this type of error are triggered by
> application bugs rather than Tomcat bugs - usually in the form of retaining and
> re-using a reference to the request or response object. One way to test this is
> to set the system property org.apache.catalina.connector.RECYCLE_FACADES to
> true. If you see NPEs then that is indicative of an application bug."
>
> I think I'm in just such a situation. I tried setting RECYCLE_FACADES to true and I did indeed see an NPE:
>
> java.lang.NullPointerException
> at org.apache.catalina.connector.ResponseFacade.getOutputStream(ResponseFacade.java:189)
> at com.blaze.proxy.ProxyHttpExchange.getOutputStream(ProxyHttpExchange.java:133)
> at com.blaze.proxy.ProxyHttpExchange.onResponseComplete(ProxyHttpExchange.java:276)
> at org.eclipse.jetty.client.HttpExchange$Listener.onResponseComplete(HttpExchange.java:900)
> at org.eclipse.jetty.client.HttpExchange.setStatus(HttpExchange.java:261)
> at org.eclipse.jetty.client.HttpConnection$Handler.messageComplete(HttpConnection.java:604)
> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:810)
> at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:212)
> at org.eclipse.jetty.client.HttpConnection.handle(HttpConnection.java:262)
> at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:510)
> at org.eclipse.jetty.io.nio.SelectChannelEndPoint.access$000(SelectChannelEndPoint.java:34)
> at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:40)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:450)
> at java.lang.Thread.run(Thread.java:662)
>
> Would anyone have any suggestions about what steps I might be doing or not doing to get into this situation?
> Is it:
> - Not calling complete() on the AsyncContext, or calling complete() and then writing to the outputStream?
> - Writing to the output stream after it has been closed?
> - Something else?
>
> Thanks,
> Chris
>
>
> On April 13, 2011 07:31:45 am Chris Dumoulin wrote:
> > Actually, I saw that notice and tried Tomcat 7.0.12, but saw the same behaviour. I should have mentioned that before.
> > So, I think this is a different issue.
> >
> > - Chris
> >
> > On April 13, 2011 07:27:51 am André Warnier wrote:
> > > Hi.
> > >
> > > An earlier message to this list [[SECURITY] CVE-2011-1475 Apache Tomcat information
> > > disclosure] /may/ have something to do with this.
> > > (It talks only about the HTTP connector, but also about content mixup with async requests,
> > > so maybe there is a link)
> > >
> > > Chris Dumoulin wrote:
> > > > I'm seeing an intermittent problem with my webapp where a request is sent and the response contains 8184 bytes from some other response followed by the correct response.
> > > >
> > > > The setup being used is Nginx 0.8.54 reverse proxying to Tomcat 7.0.11. AJP is the protocol between Nginx and Tomcat.
> > > > The webapp in Tomcat is doing Servlet 3.0 async requests.
> > > >
> > > > This issue is extremely difficult to reproduce and at this point I'm not sure if the problem is in the webapp, Tomcat, or Nginx.
> > > > I know that 8184 bytes is the size of an AJP packet, and in Tomcat's org.apache.catalina.connector.Response, I see the following code:
> > > >
> > > > if("AJP/1.3".equals(connector.getProtocol())) {
> > > > // default size to size of one ajp-packet
> > > > outputBuffer = new OutputBuffer(8184);
> > > > }
> > > >
> > > > So, right now I'm following the theory that something is being reused in Tomcat without having been properly completed or recycled. Obviously it's most likely that this is an application bug.
> > > >
> > > > Does anyone have any ideas about what kind of problem in the application could cause this behaviour, or other ideas about what the cause might be?
> > > >
> > > > Thanks,
> > > > Chris
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > > > For additional commands, e-mail: users-help@tomcat.apache.org
> > > >
> > > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > > For additional commands, e-mail: users-help@tomcat.apache.org
> > >
> > >
> >
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Problem with merged responses - possibly a reused Response or OutputBuffer
Posted by Chris Dumoulin <ch...@blaze.io>.
In a discussion on Bug 50957 (https://issues.apache.org/bugzilla/show_bug.cgi?id=50957) Mark Thomas said the following:
"Experience has shown that most instances of this type of error are triggered by
application bugs rather than Tomcat bugs - usually in the form of retaining and
re-using a reference to the request or response object. One way to test this is
to set the system property org.apache.catalina.connector.RECYCLE_FACADES to
true. If you see NPEs then that is indicative of an application bug."
I think I'm in just such a situation. I tried setting RECYCLE_FACADES to true and I did indeed see an NPE:
java.lang.NullPointerException
at org.apache.catalina.connector.ResponseFacade.getOutputStream(ResponseFacade.java:189)
at com.blaze.proxy.ProxyHttpExchange.getOutputStream(ProxyHttpExchange.java:133)
at com.blaze.proxy.ProxyHttpExchange.onResponseComplete(ProxyHttpExchange.java:276)
at org.eclipse.jetty.client.HttpExchange$Listener.onResponseComplete(HttpExchange.java:900)
at org.eclipse.jetty.client.HttpExchange.setStatus(HttpExchange.java:261)
at org.eclipse.jetty.client.HttpConnection$Handler.messageComplete(HttpConnection.java:604)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:810)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:212)
at org.eclipse.jetty.client.HttpConnection.handle(HttpConnection.java:262)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:510)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.access$000(SelectChannelEndPoint.java:34)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:40)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:450)
at java.lang.Thread.run(Thread.java:662)
Would anyone have any suggestions about what steps I might be doing or not doing to get into this situation?
Is it:
- Not calling complete() on the AsyncContext, or calling complete() and then writing to the outputStream?
- Writing to the output stream after it has been closed?
- Something else?
Thanks,
Chris
On April 13, 2011 07:31:45 am Chris Dumoulin wrote:
> Actually, I saw that notice and tried Tomcat 7.0.12, but saw the same behaviour. I should have mentioned that before.
> So, I think this is a different issue.
>
> - Chris
>
> On April 13, 2011 07:27:51 am André Warnier wrote:
> > Hi.
> >
> > An earlier message to this list [[SECURITY] CVE-2011-1475 Apache Tomcat information
> > disclosure] /may/ have something to do with this.
> > (It talks only about the HTTP connector, but also about content mixup with async requests,
> > so maybe there is a link)
> >
> > Chris Dumoulin wrote:
> > > I'm seeing an intermittent problem with my webapp where a request is sent and the response contains 8184 bytes from some other response followed by the correct response.
> > >
> > > The setup being used is Nginx 0.8.54 reverse proxying to Tomcat 7.0.11. AJP is the protocol between Nginx and Tomcat.
> > > The webapp in Tomcat is doing Servlet 3.0 async requests.
> > >
> > > This issue is extremely difficult to reproduce and at this point I'm not sure if the problem is in the webapp, Tomcat, or Nginx.
> > > I know that 8184 bytes is the size of an AJP packet, and in Tomcat's org.apache.catalina.connector.Response, I see the following code:
> > >
> > > if("AJP/1.3".equals(connector.getProtocol())) {
> > > // default size to size of one ajp-packet
> > > outputBuffer = new OutputBuffer(8184);
> > > }
> > >
> > > So, right now I'm following the theory that something is being reused in Tomcat without having been properly completed or recycled. Obviously it's most likely that this is an application bug.
> > >
> > > Does anyone have any ideas about what kind of problem in the application could cause this behaviour, or other ideas about what the cause might be?
> > >
> > > Thanks,
> > > Chris
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > > For additional commands, e-mail: users-help@tomcat.apache.org
> > >
> > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: users-help@tomcat.apache.org
> >
> >
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Problem with merged responses - possibly a reused Response or OutputBuffer
Posted by Chris Dumoulin <ch...@blaze.io>.
Actually, I saw that notice and tried Tomcat 7.0.12, but saw the same behaviour. I should have mentioned that before.
So, I think this is a different issue.
- Chris
On April 13, 2011 07:27:51 am André Warnier wrote:
> Hi.
>
> An earlier message to this list [[SECURITY] CVE-2011-1475 Apache Tomcat information
> disclosure] /may/ have something to do with this.
> (It talks only about the HTTP connector, but also about content mixup with async requests,
> so maybe there is a link)
>
> Chris Dumoulin wrote:
> > I'm seeing an intermittent problem with my webapp where a request is sent and the response contains 8184 bytes from some other response followed by the correct response.
> >
> > The setup being used is Nginx 0.8.54 reverse proxying to Tomcat 7.0.11. AJP is the protocol between Nginx and Tomcat.
> > The webapp in Tomcat is doing Servlet 3.0 async requests.
> >
> > This issue is extremely difficult to reproduce and at this point I'm not sure if the problem is in the webapp, Tomcat, or Nginx.
> > I know that 8184 bytes is the size of an AJP packet, and in Tomcat's org.apache.catalina.connector.Response, I see the following code:
> >
> > if("AJP/1.3".equals(connector.getProtocol())) {
> > // default size to size of one ajp-packet
> > outputBuffer = new OutputBuffer(8184);
> > }
> >
> > So, right now I'm following the theory that something is being reused in Tomcat without having been properly completed or recycled. Obviously it's most likely that this is an application bug.
> >
> > Does anyone have any ideas about what kind of problem in the application could cause this behaviour, or other ideas about what the cause might be?
> >
> > Thanks,
> > Chris
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: users-help@tomcat.apache.org
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Problem with merged responses - possibly a reused Response or
OutputBuffer
Posted by André Warnier <aw...@ice-sa.com>.
Hi.
An earlier message to this list [[SECURITY] CVE-2011-1475 Apache Tomcat information
disclosure] /may/ have something to do with this.
(It talks only about the HTTP connector, but also about content mixup with async requests,
so maybe there is a link)
Chris Dumoulin wrote:
> I'm seeing an intermittent problem with my webapp where a request is sent and the response contains 8184 bytes from some other response followed by the correct response.
>
> The setup being used is Nginx 0.8.54 reverse proxying to Tomcat 7.0.11. AJP is the protocol between Nginx and Tomcat.
> The webapp in Tomcat is doing Servlet 3.0 async requests.
>
> This issue is extremely difficult to reproduce and at this point I'm not sure if the problem is in the webapp, Tomcat, or Nginx.
> I know that 8184 bytes is the size of an AJP packet, and in Tomcat's org.apache.catalina.connector.Response, I see the following code:
>
> if("AJP/1.3".equals(connector.getProtocol())) {
> // default size to size of one ajp-packet
> outputBuffer = new OutputBuffer(8184);
> }
>
> So, right now I'm following the theory that something is being reused in Tomcat without having been properly completed or recycled. Obviously it's most likely that this is an application bug.
>
> Does anyone have any ideas about what kind of problem in the application could cause this behaviour, or other ideas about what the cause might be?
>
> Thanks,
> Chris
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org