You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Vince Stewart <st...@gmail.com> on 2013/08/26 02:07:25 UTC
buffer expand warning in Tomcat (apache-tomcat-8.0.0-RC1-embed)
greetings all,
since commencing with version 8 (embedded) I have encountered no problems
except when the HttpServletRequest object receives a message bigger than
16384 k. In that circumstance a warning(below) is repeated X times for
messages bigger than 8192+(X*8192).
I notice in the InternalNioInputBuffer.expand method, the comment "//should
not happen" precedes the logging of the warning message.
I am using a modified BufferedReader class with a buffer size of 8192 and
the warning seems to coincide with each read. This same class has operated
without a problem for years but I am sure my reader is causing this new
problem.
So the question is "how best to read a large message from the
ServletInputStream in tomcat-8"
=============================
Aug 26, 2013 10:59:33 AM org.apache.coyote.http11.InternalNioInputBuffer
expand
WARNING: Expanding buffer size. Old size: 16384, new size: 24576
java.lang.Exception
at
org.apache.coyote.http11.InternalNioInputBuffer.expand(InternalNioInputBuffer.java:420)
at
org.apache.coyote.http11.InternalNioInputBuffer.readSocket(InternalNioInputBuffer.java:467)
at
org.apache.coyote.http11.InternalNioInputBuffer.nbRead(InternalNioInputBuffer.java:187)
at
org.apache.coyote.http11.InternalNioInputBuffer.available(InternalNioInputBuffer.java:171)
at
org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Processor.java:818)
at org.apache.coyote.Request.action(Request.java:373)
at
org.apache.catalina.connector.InputBuffer.available(InputBuffer.java:243)
at
org.apache.catalina.connector.CoyoteInputStream.available(CoyoteInputStream.java:137)
at sun.nio.cs.StreamDecoder.inReady(StreamDecoder.java:362)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:323)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at oodbpds.BigBufferedReader.read1(BigBufferedReader.java:136)
at oodbpds.BigBufferedReader.read(BigBufferedReader.java:156)
at oodbpds.ASIFTI.doPost(ASIFTI.java:90)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:223)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:107)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:75)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:90)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:494)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1009)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:632)
at
org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:223)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1592)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1550)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
--
Vince Stewart
Re: buffer expand warning in Tomcat (apache-tomcat-8.0.0-RC1-embed)
Posted by Vince Stewart <st...@gmail.com>.
Hi Mark, a bit more follow-up on accessing the servlet InputStream:
You advised "You'd be better off dropping the call to in.ready() and doing
a blocking
read on the socket.....If you remove the call to
in.ready(), I'm fairly sure you'll see the warnings disappear."
I just thought I'd let you know, that prediction was absolutely correct.
On Thu, Aug 29, 2013 at 7:09 AM, Vince Stewart <st...@gmail.com>wrote:
> Appreciate this a lot Mark.
>
> I'm pretty sure my previous code had a short sleep in each loop but
> thankfully, in-coming data rarely exceeds the Servlet InputBuffer size of
> 8192 so looping is rare.
>
> I have been trying to get to grips with the asynchronous stuff but have
> not got it going yet. The concept is very appealing. Thanks for your
> contributions.
>
>
> On Wed, Aug 28, 2013 at 11:00 PM, Mark Thomas <ma...@apache.org> wrote:
>
>> On 28/08/2013 09:41, Mark Thomas wrote:
>> > On 27/08/2013 03:40, Vince Stewart wrote:
>> >> hi all,
>> >> thought I would add some progress on this topic.
>> >> I have changed my method for reading from the HttpServletRequest
>> object but
>> >> the same warning message is thrown for every 8192 bytes read. I no
>> longer
>> >> regard my code to be suspect though am happy to be corrected. The
>> >> application operates completely normally except for the warning
>> message. The
>> >> code I am using to read the input is shown below.
>> >>
>> >> public void doPost(HttpServletRequest httpServletRequest.....etc
>> >> .......other code..............
>> >> char[] cbuf=new char[8192];
>> >> int i=0;
>> >> int requestLength=httpServletRequest.getContentLength();
>> >> BufferedReader in=httpServletRequest.getReader();
>> >> StringBuilder sb=new StringBuilder(requestLength);
>> >> while(sb.length()<requestLength){
>> >> if(in.ready()){
>> >> i=in.read(cbuf,0,reqLen);
>> >> }
>> >> sb.append(cbuf,0,i);
>> >> }
>> >> in.close();
>> >> String message=sb.toString();
>> >> //.....etc
>> >
>> > It looks like there is a possible Tomcat bug but the above code is not
>> > the way to read data from the client as it puts the thread into a tight
>> > loop while it is waiting for more data to arrive.
>>
>> I've just cleaned up the code for Tomcat 8.0.x to fix the bug that your
>> example highlighted. Thanks for such a clear problem statement that made
>> it very easy to track down the root cause of the issue. The fix will be
>> in 8.0.0-RC2 onwards.
>>
>> That said, my comments about there being a better way to do what you are
>> doing stand.
>>
>> Mark
>>
>> >
>> > You'd be better off dropping the call to in.ready() and doing a blocking
>> > read on the socket. The elapsed time should be the same (might even be a
>> > little less) and your CPU consumption during the read when the client is
>> > slow sending data will be significantly lower. If you remove the call to
>> > in.ready(), I'm fairly sure you'll see the warnings disappear.
>> >
>> > Ideally, you should look at non-blocking IO as supported by Servlet 3.1
>> > but that might be too big a change as it fundamentally changes how data
>> > is read and written.
>> >
>> > Mark
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> > For additional commands, e-mail: users-help@tomcat.apache.org
>> >
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
>
> --
> Vince Stewart
>
--
Vince Stewart
Re: buffer expand warning in Tomcat (apache-tomcat-8.0.0-RC1-embed)
Posted by Vince Stewart <st...@gmail.com>.
Appreciate this a lot Mark.
I'm pretty sure my previous code had a short sleep in each loop but
thankfully, in-coming data rarely exceeds the Servlet InputBuffer size of
8192 so looping is rare.
I have been trying to get to grips with the asynchronous stuff but have not
got it going yet. The concept is very appealing. Thanks for your
contributions.
On Wed, Aug 28, 2013 at 11:00 PM, Mark Thomas <ma...@apache.org> wrote:
> On 28/08/2013 09:41, Mark Thomas wrote:
> > On 27/08/2013 03:40, Vince Stewart wrote:
> >> hi all,
> >> thought I would add some progress on this topic.
> >> I have changed my method for reading from the HttpServletRequest object
> but
> >> the same warning message is thrown for every 8192 bytes read. I no
> longer
> >> regard my code to be suspect though am happy to be corrected. The
> >> application operates completely normally except for the warning
> message. The
> >> code I am using to read the input is shown below.
> >>
> >> public void doPost(HttpServletRequest httpServletRequest.....etc
> >> .......other code..............
> >> char[] cbuf=new char[8192];
> >> int i=0;
> >> int requestLength=httpServletRequest.getContentLength();
> >> BufferedReader in=httpServletRequest.getReader();
> >> StringBuilder sb=new StringBuilder(requestLength);
> >> while(sb.length()<requestLength){
> >> if(in.ready()){
> >> i=in.read(cbuf,0,reqLen);
> >> }
> >> sb.append(cbuf,0,i);
> >> }
> >> in.close();
> >> String message=sb.toString();
> >> //.....etc
> >
> > It looks like there is a possible Tomcat bug but the above code is not
> > the way to read data from the client as it puts the thread into a tight
> > loop while it is waiting for more data to arrive.
>
> I've just cleaned up the code for Tomcat 8.0.x to fix the bug that your
> example highlighted. Thanks for such a clear problem statement that made
> it very easy to track down the root cause of the issue. The fix will be
> in 8.0.0-RC2 onwards.
>
> That said, my comments about there being a better way to do what you are
> doing stand.
>
> Mark
>
> >
> > You'd be better off dropping the call to in.ready() and doing a blocking
> > read on the socket. The elapsed time should be the same (might even be a
> > little less) and your CPU consumption during the read when the client is
> > slow sending data will be significantly lower. If you remove the call to
> > in.ready(), I'm fairly sure you'll see the warnings disappear.
> >
> > Ideally, you should look at non-blocking IO as supported by Servlet 3.1
> > but that might be too big a change as it fundamentally changes how data
> > is read and written.
> >
> > Mark
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: users-help@tomcat.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
--
Vince Stewart
Re: buffer expand warning in Tomcat (apache-tomcat-8.0.0-RC1-embed)
Posted by Mark Thomas <ma...@apache.org>.
On 28/08/2013 09:41, Mark Thomas wrote:
> On 27/08/2013 03:40, Vince Stewart wrote:
>> hi all,
>> thought I would add some progress on this topic.
>> I have changed my method for reading from the HttpServletRequest object but
>> the same warning message is thrown for every 8192 bytes read. I no longer
>> regard my code to be suspect though am happy to be corrected. The
>> application operates completely normally except for the warning message. The
>> code I am using to read the input is shown below.
>>
>> public void doPost(HttpServletRequest httpServletRequest.....etc
>> .......other code..............
>> char[] cbuf=new char[8192];
>> int i=0;
>> int requestLength=httpServletRequest.getContentLength();
>> BufferedReader in=httpServletRequest.getReader();
>> StringBuilder sb=new StringBuilder(requestLength);
>> while(sb.length()<requestLength){
>> if(in.ready()){
>> i=in.read(cbuf,0,reqLen);
>> }
>> sb.append(cbuf,0,i);
>> }
>> in.close();
>> String message=sb.toString();
>> //.....etc
>
> It looks like there is a possible Tomcat bug but the above code is not
> the way to read data from the client as it puts the thread into a tight
> loop while it is waiting for more data to arrive.
I've just cleaned up the code for Tomcat 8.0.x to fix the bug that your
example highlighted. Thanks for such a clear problem statement that made
it very easy to track down the root cause of the issue. The fix will be
in 8.0.0-RC2 onwards.
That said, my comments about there being a better way to do what you are
doing stand.
Mark
>
> You'd be better off dropping the call to in.ready() and doing a blocking
> read on the socket. The elapsed time should be the same (might even be a
> little less) and your CPU consumption during the read when the client is
> slow sending data will be significantly lower. If you remove the call to
> in.ready(), I'm fairly sure you'll see the warnings disappear.
>
> Ideally, you should look at non-blocking IO as supported by Servlet 3.1
> but that might be too big a change as it fundamentally changes how data
> is read and written.
>
> Mark
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: buffer expand warning in Tomcat (apache-tomcat-8.0.0-RC1-embed)
Posted by Mark Thomas <ma...@apache.org>.
On 27/08/2013 03:40, Vince Stewart wrote:
> hi all,
> thought I would add some progress on this topic.
> I have changed my method for reading from the HttpServletRequest object but
> the same warning message is thrown for every 8192 bytes read. I no longer
> regard my code to be suspect though am happy to be corrected. The
> application operates completely normally except for the warning message. The
> code I am using to read the input is shown below.
>
> public void doPost(HttpServletRequest httpServletRequest.....etc
> .......other code..............
> char[] cbuf=new char[8192];
> int i=0;
> int requestLength=httpServletRequest.getContentLength();
> BufferedReader in=httpServletRequest.getReader();
> StringBuilder sb=new StringBuilder(requestLength);
> while(sb.length()<requestLength){
> if(in.ready()){
> i=in.read(cbuf,0,reqLen);
> }
> sb.append(cbuf,0,i);
> }
> in.close();
> String message=sb.toString();
> //.....etc
It looks like there is a possible Tomcat bug but the above code is not
the way to read data from the client as it puts the thread into a tight
loop while it is waiting for more data to arrive.
You'd be better off dropping the call to in.ready() and doing a blocking
read on the socket. The elapsed time should be the same (might even be a
little less) and your CPU consumption during the read when the client is
slow sending data will be significantly lower. If you remove the call to
in.ready(), I'm fairly sure you'll see the warnings disappear.
Ideally, you should look at non-blocking IO as supported by Servlet 3.1
but that might be too big a change as it fundamentally changes how data
is read and written.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: buffer expand warning in Tomcat (apache-tomcat-8.0.0-RC1-embed)
Posted by Vince Stewart <st...@gmail.com>.
hi all,
thought I would add some progress on this topic.
I have changed my method for reading from the HttpServletRequest object but
the same warning message is thrown for every 8192 bytes read. I no longer
regard my code to be suspect though am happy to be corrected. The
application operates completely normally except for the warning message. The
code I am using to read the input is shown below.
public void doPost(HttpServletRequest httpServletRequest.....etc
.......other code..............
char[] cbuf=new char[8192];
int i=0;
int requestLength=httpServletRequest.getContentLength();
BufferedReader in=httpServletRequest.getReader();
StringBuilder sb=new StringBuilder(requestLength);
while(sb.length()<requestLength){
if(in.ready()){
i=in.read(cbuf,0,reqLen);
}
sb.append(cbuf,0,i);
}
in.close();
String message=sb.toString();
//.....etc
--
View this message in context: http://tomcat.10.x6.nabble.com/buffer-expand-warning-in-Tomcat-apache-tomcat-8-0-0-RC1-embed-tp5003745p5003777.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org