You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Steve Mactaggart <st...@whitesquaresoft.com> on 2013/01/17 12:07:21 UTC
BufferOverflowException in AjpNioProcessor
Hi All,
I have been hunting a BufferOverflowException on our production system for
the last while, and initially thought this was resolved by issue 53119, we
have subsequently upgraded to 7.0.32 and have still seen the issues.
Reading through the mailing list archive it seems I am not the only one
having issues with the NIO AJP Processor.
http://www.mailinglistarchive.com/html/users@tomcat.apache.org/2012-10/msg00477.html
Below is what we are seeing pretty regularly in our production environment.
This is probably load based, as we don't seem to see this in any of our
development environments.
Caused by: java.nio.BufferOverflowException
at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:183)
at
org.apache.coyote.ajp.AjpNioProcessor.output(AjpNioProcessor.java:281)
at
org.apache.coyote.ajp.AbstractAjpProcessor$SocketOutputBuffer.doWrite(AbstractAjpProcessor.java:1122)
at org.apache.coyote.Response.doWrite(Response.java:504)
at
org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:383)
at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:342)
at
org.apache.tomcat.util.buf.IntermediateOutputStream.write(C2BConverter.java:278)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207)
at
org.apache.tomcat.util.buf.WriteConvertor.write(C2BConverter.java:242)
at java.io.BufferedWriter.write(BufferedWriter.java:188)
at
org.apache.tomcat.util.buf.C2BConverter.convert(C2BConverter.java:125)
at
org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:476)
at
org.apache.catalina.connector.CoyoteWriter.write(CoyoteWriter.java:152)
at
org.apache.jasper.runtime.JspWriterImpl.flushBuffer(JspWriterImpl.java:119)
at
org.apache.jasper.runtime.JspWriterImpl.write(JspWriterImpl.java:336)
at
org.apache.jasper.runtime.JspWriterImpl.write(JspWriterImpl.java:353)
at ... 72 more
As mentioned in the mailing list, could this be an issue that just needs to
be re-applied to the nio version of the Processor from the original issue I
thought this was?
I don't really know why this is happening, so can't provide a test case but
do know this is happening pretty regularly.
Steve
Re: BufferOverflowException in AjpNioProcessor
Posted by Mark Thomas <ma...@apache.org>.
On 18/01/2013 14:11, Konstantin Preißer wrote:
> Hi Mark,
>
>>> If you set the following system property:
>>> org.apache.catalina.connector.RECYCLE_FACADES
>>>
>>> to true (e.g. in setenv.[bat|sh]) and your application is retaining
>>> references you should start to see NPEs and the stack trace should
>> point
>>> to where the problem is.
>>>
>>> If you still see BufferOverflowException with that setting in place
>> then
>>> that would make a Tomcat bug more likely.
>
> Since I also got BufferOverflowExceptions with AJP-NIO, I tried your
> suggestion setting the system property
> "org.apache.catalina.connector.RECYCLE_FACADES" to "true" (by setting it in
> the Java options using tomcat7w.exe) and switched from AJP-BIO to the
> AJP-NIO connector.
>
> Today, I got a BufferOverflowException again, this time using the following
> environment:
> Windows Server 2008 R2 SP1 x64,
> Java 1.7.0_10 (64 bit),
> Tomcat 7.0.35,
> Frontend: IIS 7.5 with ISAPI Redirector 1.2.37
>
>
> In the logfile, the following stacktrace was written:
>
> Jan 18, 2013 2:43:21 PM org.apache.coyote.ajp.AjpNioProcessor process
> SEVERE: Error processing request
> java.nio.BufferOverflowException
> at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:183)
> at
> org.apache.coyote.ajp.AjpNioProcessor.output(AjpNioProcessor.java:281)
> at
> org.apache.coyote.ajp.AbstractAjpProcessor$SocketOutputBuffer.doWrite(Abstra
> ctAjpProcessor.java:1124)
> at org.apache.coyote.Response.doWrite(Response.java:505)
> at
> org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:
> 383)
> at
> org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:462)
> at
> org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:334)
> at
> org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:283)
> at
> org.apache.catalina.connector.Response.finishResponse(Response.java:513)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:434)
> at
> org.apache.coyote.ajp.AjpNioProcessor.process(AjpNioProcessor.java:184)
> at
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(Abstrac
> tProtocol.java:589)
> at
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:
> 1680)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
> 10)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
> 03)
> at java.lang.Thread.run(Thread.java:722)
>
> The log of the isapi-redirector contains for the given time (but not only)
> an error:
> [Fri Jan 18 14:43:21.904 2013] [5040:1780] [error]
> isapi_write_client::jk_isapi_plugin.c (1286): WriteClient failed with 995
> (0x000003e3)
>
> The isapi log contains about 43 such errors ("WriteClient failed with 995")
> per hour, which probably means that in such a case, the ISAPI redirector
> closes the AJP connection before the full response has been sent.
>
>
> I'm not sure how many requests my Tomcat does process, but I would guess
> about 50-100 requests per second. The BufferOverflowExceptions seems to
> occur rarely, it took ~ 12 hours after switching to the NIO connector for
> the exception to occur.
>
>
> Do you have any ideas?
Only that it makes it more likely that there is a problem somewhere in
the NIO AJP connector.
With that number of requests adding additional logging needs to be very
carefully targeted. That usually means having a pretty good idea where
the problem is in which case just fixing the problem is as easy to do.
I think it is time you raised a Bugzilla report that brings together the
information in this thread. With an idea of what causes the problem it
should be possible to track down the bug by code inspection but from
past experience that is not as easy as it sounds.
Mark
>
>
> Thanks,
>
> Konstantin Preißer
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: BufferOverflowException in AjpNioProcessor
Posted by Konstantin Preißer <ve...@t-online.de>.
Hi Mark,
> > If you set the following system property:
> > org.apache.catalina.connector.RECYCLE_FACADES
> >
> > to true (e.g. in setenv.[bat|sh]) and your application is retaining
> > references you should start to see NPEs and the stack trace should
> point
> > to where the problem is.
> >
> > If you still see BufferOverflowException with that setting in place
> then
> > that would make a Tomcat bug more likely.
Since I also got BufferOverflowExceptions with AJP-NIO, I tried your
suggestion setting the system property
"org.apache.catalina.connector.RECYCLE_FACADES" to "true" (by setting it in
the Java options using tomcat7w.exe) and switched from AJP-BIO to the
AJP-NIO connector.
Today, I got a BufferOverflowException again, this time using the following
environment:
Windows Server 2008 R2 SP1 x64,
Java 1.7.0_10 (64 bit),
Tomcat 7.0.35,
Frontend: IIS 7.5 with ISAPI Redirector 1.2.37
In the logfile, the following stacktrace was written:
Jan 18, 2013 2:43:21 PM org.apache.coyote.ajp.AjpNioProcessor process
SEVERE: Error processing request
java.nio.BufferOverflowException
at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:183)
at
org.apache.coyote.ajp.AjpNioProcessor.output(AjpNioProcessor.java:281)
at
org.apache.coyote.ajp.AbstractAjpProcessor$SocketOutputBuffer.doWrite(Abstra
ctAjpProcessor.java:1124)
at org.apache.coyote.Response.doWrite(Response.java:505)
at
org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:
383)
at
org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:462)
at
org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:334)
at
org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:283)
at
org.apache.catalina.connector.Response.finishResponse(Response.java:513)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:434)
at
org.apache.coyote.ajp.AjpNioProcessor.process(AjpNioProcessor.java:184)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(Abstrac
tProtocol.java:589)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:
1680)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
10)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
03)
at java.lang.Thread.run(Thread.java:722)
The log of the isapi-redirector contains for the given time (but not only)
an error:
[Fri Jan 18 14:43:21.904 2013] [5040:1780] [error]
isapi_write_client::jk_isapi_plugin.c (1286): WriteClient failed with 995
(0x000003e3)
The isapi log contains about 43 such errors ("WriteClient failed with 995")
per hour, which probably means that in such a case, the ISAPI redirector
closes the AJP connection before the full response has been sent.
I'm not sure how many requests my Tomcat does process, but I would guess
about 50-100 requests per second. The BufferOverflowExceptions seems to
occur rarely, it took ~ 12 hours after switching to the NIO connector for
the exception to occur.
Do you have any ideas?
Thanks,
Konstantin Preißer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: BufferOverflowException in AjpNioProcessor
Posted by Konstantin Preißer <ve...@t-online.de>.
Hi,
> -----Original Message-----
> From: Mark Thomas [mailto:markt@apache.org]
> Sent: Thursday, January 17, 2013 1:11 PM
>
> On 17/01/2013 11:07, Steve Mactaggart wrote:
> > Hi All,
> >
> > I have been hunting a BufferOverflowException on our production system
for
> > the last while, and initially thought this was resolved by issue 53119,
we
> > have subsequently upgraded to 7.0.32 and have still seen the issues.
> >
> > Reading through the mailing list archive it seems I am not the only one
> > having issues with the NIO AJP Processor.
Yes, I got these BufferOverflowExceptions when switching from AJP-APR to
AJP-NIO.
I added a comment to the bug for the AJP-APR connector:
https://issues.apache.org/bugzilla/show_bug.cgi?id=53119
Mark,
> The two possibilities at this point are:
> a) Tomcat bug
> b) Application bug
>
> b would be caused by something (e.g. servlet, filter) retaining a
> reference to a request and using it after the request had finished
> processing.
>
> If you set the following system property:
> org.apache.catalina.connector.RECYCLE_FACADES
>
> to true (e.g. in setenv.[bat|sh]) and your application is retaining
> references you should start to see NPEs and the stack trace should point
> to where the problem is.
>
> If you still see BufferOverflowException with that setting in place then
> that would make a Tomcat bug more likely.
I'm certain that my applications don't retain references to request objects,
but I will also try your suggestion with setting the RECYCLE_FACADES
property and switching back to the AJP-NIO connector, to see if these
BufferOverflowExceptions still occur.
Thanks!
Regards,
Konstantin Preißer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: BufferOverflowException in AjpNioProcessor
Posted by Mark Thomas <ma...@apache.org>.
On 17/01/2013 11:07, Steve Mactaggart wrote:
> Hi All,
>
> I have been hunting a BufferOverflowException on our production system for
> the last while, and initially thought this was resolved by issue 53119, we
> have subsequently upgraded to 7.0.32 and have still seen the issues.
>
> Reading through the mailing list archive it seems I am not the only one
> having issues with the NIO AJP Processor.
>
> http://www.mailinglistarchive.com/html/users@tomcat.apache.org/2012-10/msg00477.html
>
> Below is what we are seeing pretty regularly in our production environment.
> This is probably load based, as we don't seem to see this in any of our
> development environments.
>
> Caused by: java.nio.BufferOverflowException
> at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:183)
> at
> org.apache.coyote.ajp.AjpNioProcessor.output(AjpNioProcessor.java:281)
> at
> org.apache.coyote.ajp.AbstractAjpProcessor$SocketOutputBuffer.doWrite(AbstractAjpProcessor.java:1122)
> at org.apache.coyote.Response.doWrite(Response.java:504)
> at
> org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:383)
> at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:342)
> at
> org.apache.tomcat.util.buf.IntermediateOutputStream.write(C2BConverter.java:278)
> at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
> at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
> at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
> at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207)
> at
> org.apache.tomcat.util.buf.WriteConvertor.write(C2BConverter.java:242)
> at java.io.BufferedWriter.write(BufferedWriter.java:188)
> at
> org.apache.tomcat.util.buf.C2BConverter.convert(C2BConverter.java:125)
> at
> org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:476)
> at
> org.apache.catalina.connector.CoyoteWriter.write(CoyoteWriter.java:152)
> at
> org.apache.jasper.runtime.JspWriterImpl.flushBuffer(JspWriterImpl.java:119)
> at
> org.apache.jasper.runtime.JspWriterImpl.write(JspWriterImpl.java:336)
> at
> org.apache.jasper.runtime.JspWriterImpl.write(JspWriterImpl.java:353)
> at ... 72 more
>
> As mentioned in the mailing list, could this be an issue that just needs to
> be re-applied to the nio version of the Processor from the original issue I
> thought this was?
Unlikely. I have taken a look at the AjpNioProcessor code and I don't
see any exception paths that could lead to the writeBuffer not being
cleared.
> I don't really know why this is happening, so can't provide a test case but
> do know this is happening pretty regularly.
How regularly? How many requests an hour and how many errors like the
above an hour? (I'm trying to figure out what additional logging could
be added that provides useful information without filling your disks
with logs).
The two possibilities at this point are:
a) Tomcat bug
b) Application bug
b would be caused by something (e.g. servlet, filter) retaining a
reference to a request and using it after the request had finished
processing.
If you set the following system property:
org.apache.catalina.connector.RECYCLE_FACADES
to true (e.g. in setenv.[bat|sh]) and your application is retaining
references you should start to see NPEs and the stack trace should point
to where the problem is.
If you still see BufferOverflowException with that setting in place then
that would make a Tomcat bug more likely.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org