You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Kumar Pandey <kp...@yahoo.com> on 2006/05/19 08:06:01 UTC
Tomcat 5.0.28, Socket hanging on sending 20KB data for 16 minutes
Hi
I'm using Tomcat 5.0.26 that comes with JBoss 3.2.5
with default configurations
A servlet creates pie/bar charts using JFreeChart.
This is refreshed every 5 seconds via javascript
refresh in the brower. All this has been workign fine
for most customers.
Suddenly for a specific customer sessions coming from
a specific newtwork/georaphical location we see that
the 5 seconds chart refreshes are hanging the thread.
Charts are 20 KB. For a single user in the system
sometimes these requests are talking up 20 ~30 threads
and each stuck for 15~16 minutes before it finally
clears up.
We ran ethereal on the tcp packet and see that when
the threads are hung, browser is sending FIN, ACK
eventhough server hasn't completed sending the data.
Thus we see TCP retranmissions of same packets as its
not receiveing any ack from the client.
How do I troubleshoot this issue.
System is not memory starved and cpu is 98% idle.
Any pointers on trouble shooting this would be greatly
appreciated.
Thanks
Kumar
Here's the thread dump of one of the hung request.
java.net.SocketOutputStream.socketWrite0(Native
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
java.net.SocketOutputStream.write(SocketOutputStream.java:136)
org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:714)
org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:398)
org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:318)
org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:737)
org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:125)
org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:569)
org.apache.coyote.Response.doWrite(Response.java:542)
org.apache.coyote.tomcat5.OutputBuffer.realWriteBytes(OutputBuffer.java:368)
org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:398)
org.apache.coyote.tomcat5.OutputBuffer.doFlush(OutputBuffer.java:318)
org.apache.coyote.tomcat5.OutputBuffer.flush(OutputBuffer.java:297)
org.apache.coyote.tomcat5.CoyoteOutputStream.flush(CoyoteOutputStream.java:85)
sun.awt.image.codec.JPEGImageEncoderImpl.writeJPEGStream(Native
locked
sun.awt.image.codec.JPEGImageEncoderImpl.encode(JPEGImageEncoderImpl.java:472)
sun.awt.image.codec.JPEGImageEncoderImpl.encode(JPEGImageEncoderImpl.java:228)
locked
sun.awt.image.codec.JPEGImageEncoderImpl.encode(JPEGImageEncoderImpl.java:235)
locked
org.jfree.chart.encoders.SunJPEGEncoderAdapter.encode(SunJPEGEncoderAdapter.java:139)
org.jfree.chart.encoders.EncoderUtil.writeBufferedImage(EncoderUtil.java:136)
org.jfree.chart.ChartUtilities.writeChartAsJPEG(ChartUtilities.java:442)
org.jfree.chart.ChartUtilities.writeChartAsJPEG(ChartUtilities.java:388)
com.transerainc.rtmcui.util.ChartUtil.drawChartAsJPEG(ChartUtil.java:46)
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 5.0.28, Socket hanging on sending 20KB data for 16 minutes -Fixed!!
Posted by Kumar Pandey <ku...@yahoo.com>.
OK we finally pinnned the issue down to a client's
proxy server.
We have few ajax refreshes happening on our html
pages. Under certain conditions IE sends a RST when
its unable to serve these requests (you can quickly
verify this by running ethreal while viewing new Yahoo
pages with real time updates on e-mail weather etc).
These RST were not getting propageted by the proxy
server to our server. Thus our server kept sending
re-transmissions on that socket.
Once the proxy was set to just do passthru , eveythign
worked fine as the server received RST sent from the
browser.
Thanks
Kumar Pandey
--- Kumar Pandey <kp...@yahoo.com> wrote:
> If it was synchronization issue at the code level
> then
> I should have seen the issue during load test in our
> env.
>
> We see the issue only for the traffic coming from
> this
> specific network even for just one user in the
> system.
> Cpu is 99% idle.
>
> Also IE just allows 2 socket connection from the
> browser at any given time for that instance of IE.I
> see upto 30 threads tied up to service one user
> session requests for 20KB of data. Each finally
> clears
> up after 16 minutes. Meanhile other users accessing
> from other networks (even from India)
> does not have this issue. Their request for same
> data
> is getting serviced under sub seconds. by remaining
> threads not tied up.
>
> Does anyone have any idea on the re-transmissions of
> the same packets from server tcp layer to broswer
> thus
> hanging the connection for upto 16 minutes in Linux
> FC4, Tomcat 5.0.28. Who is closing the connection
> after ~16 minutes etc.
>
> Thanks
> Kumar
>
> --- Rajeev Jha <jh...@gmail.com> wrote:
>
> > looking at
> http://www.jfree.org/jfreechart/javadoc/
> > ,
> >
> >
>
org.jfree.chart.encoders.EncoderUtil.writeBufferedImage
> > does not look thread-safe.
> > try checking
> > sun.awt.image.codec.JPEGImageEncoderImpl.encode
> API.
> >
> > Try increasing the latency by re-requesting the
> > servlet every 30 secs/ 1 min and see if that works
> > fine or not.
> >
> >
> >
> >
> > Kumar Pandey wrote:
> >
> >
> >
> > >Sorry I forgot to include couple of more stats
> > >
> > >Tomcat is running on linux FC4
> > >Client browser is IE 6.0
> > >
> > >First line of the dump is
> > >"http-0.0.0.0-80-Processor99" daemon prio=1
> > >tid=0x8b09b4b8 nid=0x46ec runnable
> > >[0x84be7000..0x84be91c0]
> > >
> > >Thx
> > >
> > >
> > >
> >
> >
> >
>
---------------------------------------------------------------------
> > To unsubscribe, e-mail:
> > users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail:
> > users-help@tomcat.apache.org
> >
> >
>
>
> __________________________________________________
> Do You Yahoo!?
> Tired of spam? Yahoo! Mail has the best spam
> protection around
> http://mail.yahoo.com
>
>
---------------------------------------------------------------------
> To unsubscribe, e-mail:
> users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail:
> users-help@tomcat.apache.org
>
>
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 5.0.28, Socket hanging on sending 20KB data for 16 minutes
Posted by Kumar Pandey <kp...@yahoo.com>.
If it was synchronization issue at the code level then
I should have seen the issue during load test in our
env.
We see the issue only for the traffic coming from this
specific network even for just one user in the system.
Cpu is 99% idle.
Also IE just allows 2 socket connection from the
browser at any given time for that instance of IE.I
see upto 30 threads tied up to service one user
session requests for 20KB of data. Each finally clears
up after 16 minutes. Meanhile other users accessing
from other networks (even from India)
does not have this issue. Their request for same data
is getting serviced under sub seconds. by remaining
threads not tied up.
Does anyone have any idea on the re-transmissions of
the same packets from server tcp layer to broswer thus
hanging the connection for upto 16 minutes in Linux
FC4, Tomcat 5.0.28. Who is closing the connection
after ~16 minutes etc.
Thanks
Kumar
--- Rajeev Jha <jh...@gmail.com> wrote:
> looking at http://www.jfree.org/jfreechart/javadoc/
> ,
>
>
org.jfree.chart.encoders.EncoderUtil.writeBufferedImage
> does not look thread-safe.
> try checking
> sun.awt.image.codec.JPEGImageEncoderImpl.encode API.
>
> Try increasing the latency by re-requesting the
> servlet every 30 secs/ 1 min and see if that works
> fine or not.
>
>
>
>
> Kumar Pandey wrote:
>
>
>
> >Sorry I forgot to include couple of more stats
> >
> >Tomcat is running on linux FC4
> >Client browser is IE 6.0
> >
> >First line of the dump is
> >"http-0.0.0.0-80-Processor99" daemon prio=1
> >tid=0x8b09b4b8 nid=0x46ec runnable
> >[0x84be7000..0x84be91c0]
> >
> >Thx
> >
> >
> >
>
>
>
---------------------------------------------------------------------
> To unsubscribe, e-mail:
> users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail:
> users-help@tomcat.apache.org
>
>
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 5.0.28, Socket hanging on sending 20KB data for 16 minutes
Posted by Rajeev Jha <jh...@gmail.com>.
looking at http://www.jfree.org/jfreechart/javadoc/ ,
org.jfree.chart.encoders.EncoderUtil.writeBufferedImage does not look thread-safe.
try checking sun.awt.image.codec.JPEGImageEncoderImpl.encode API.
Try increasing the latency by re-requesting the servlet every 30 secs/ 1 min and see if that works fine or not.
Kumar Pandey wrote:
>Sorry I forgot to include couple of more stats
>
>Tomcat is running on linux FC4
>Client browser is IE 6.0
>
>First line of the dump is
>"http-0.0.0.0-80-Processor99" daemon prio=1
>tid=0x8b09b4b8 nid=0x46ec runnable
>[0x84be7000..0x84be91c0]
>
>Thx
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 5.0.28, Socket hanging on sending 20KB data for 16 minutes
Posted by Kumar Pandey <kp...@yahoo.com>.
Sorry I forgot to include couple of more stats
Tomcat is running on linux FC4
Client browser is IE 6.0
First line of the dump is
"http-0.0.0.0-80-Processor99" daemon prio=1
tid=0x8b09b4b8 nid=0x46ec runnable
[0x84be7000..0x84be91c0]
Thx
--- Kumar Pandey <kp...@yahoo.com> wrote:
> Hi
> I'm using Tomcat 5.0.26 that comes with JBoss
> 3.2.5
> with default configurations
>
> A servlet creates pie/bar charts using JFreeChart.
> This is refreshed every 5 seconds via javascript
> refresh in the brower. All this has been workign
> fine
> for most customers.
> Suddenly for a specific customer sessions coming
> from
> a specific newtwork/georaphical location we see that
> the 5 seconds chart refreshes are hanging the
> thread.
> Charts are 20 KB. For a single user in the system
> sometimes these requests are talking up 20 ~30
> threads
> and each stuck for 15~16 minutes before it finally
> clears up.
> We ran ethereal on the tcp packet and see that when
> the threads are hung, browser is sending FIN, ACK
> eventhough server hasn't completed sending the data.
> Thus we see TCP retranmissions of same packets as
> its
> not receiveing any ack from the client.
>
> How do I troubleshoot this issue.
> System is not memory starved and cpu is 98% idle.
>
> Any pointers on trouble shooting this would be
> greatly
> appreciated.
>
> Thanks
> Kumar
>
> Here's the thread dump of one of the hung request.
>
> java.net.SocketOutputStream.socketWrite0(Native
>
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
>
java.net.SocketOutputStream.write(SocketOutputStream.java:136)
>
org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:714)
>
org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:398)
>
org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:318)
>
org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:737)
>
org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:125)
>
org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:569)
>
org.apache.coyote.Response.doWrite(Response.java:542)
>
org.apache.coyote.tomcat5.OutputBuffer.realWriteBytes(OutputBuffer.java:368)
>
org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:398)
>
org.apache.coyote.tomcat5.OutputBuffer.doFlush(OutputBuffer.java:318)
>
org.apache.coyote.tomcat5.OutputBuffer.flush(OutputBuffer.java:297)
>
org.apache.coyote.tomcat5.CoyoteOutputStream.flush(CoyoteOutputStream.java:85)
>
sun.awt.image.codec.JPEGImageEncoderImpl.writeJPEGStream(Native
> locked
>
sun.awt.image.codec.JPEGImageEncoderImpl.encode(JPEGImageEncoderImpl.java:472)
>
sun.awt.image.codec.JPEGImageEncoderImpl.encode(JPEGImageEncoderImpl.java:228)
> locked
>
sun.awt.image.codec.JPEGImageEncoderImpl.encode(JPEGImageEncoderImpl.java:235)
> locked
>
org.jfree.chart.encoders.SunJPEGEncoderAdapter.encode(SunJPEGEncoderAdapter.java:139)
>
org.jfree.chart.encoders.EncoderUtil.writeBufferedImage(EncoderUtil.java:136)
>
org.jfree.chart.ChartUtilities.writeChartAsJPEG(ChartUtilities.java:442)
>
org.jfree.chart.ChartUtilities.writeChartAsJPEG(ChartUtilities.java:388)
>
com.transerainc.rtmcui.util.ChartUtil.drawChartAsJPEG(ChartUtil.java:46)
>
> __________________________________________________
> Do You Yahoo!?
> Tired of spam? Yahoo! Mail has the best spam
> protection around
> http://mail.yahoo.com
>
>
---------------------------------------------------------------------
> To unsubscribe, e-mail:
> users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail:
> users-help@tomcat.apache.org
>
>
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 5.0.28, Socket hanging on sending 20KB data for 16 minutes
Posted by Sameer Acharya <ac...@yahoo.com>.
Try synchronizing the code that uses Jfreechart classes.
-Sameer
Kumar Pandey <kp...@yahoo.com> wrote: Hi
I'm using Tomcat 5.0.26 that comes with JBoss 3.2.5
with default configurations
A servlet creates pie/bar charts using JFreeChart.
This is refreshed every 5 seconds via javascript
refresh in the brower. All this has been workign fine
for most customers.
Suddenly for a specific customer sessions coming from
a specific newtwork/georaphical location we see that
the 5 seconds chart refreshes are hanging the thread.
Charts are 20 KB. For a single user in the system
sometimes these requests are talking up 20 ~30 threads
and each stuck for 15~16 minutes before it finally
clears up.
We ran ethereal on the tcp packet and see that when
the threads are hung, browser is sending FIN, ACK
eventhough server hasn't completed sending the data.
Thus we see TCP retranmissions of same packets as its
not receiveing any ack from the client.
How do I troubleshoot this issue.
System is not memory starved and cpu is 98% idle.
Any pointers on trouble shooting this would be greatly
appreciated.
Thanks
Kumar
Here's the thread dump of one of the hung request.
java.net.SocketOutputStream.socketWrite0(Native
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
java.net.SocketOutputStream.write(SocketOutputStream.java:136)
org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:714)
org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:398)
org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:318)
org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:737)
org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:125)
org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:569)
org.apache.coyote.Response.doWrite(Response.java:542)
org.apache.coyote.tomcat5.OutputBuffer.realWriteBytes(OutputBuffer.java:368)
org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:398)
org.apache.coyote.tomcat5.OutputBuffer.doFlush(OutputBuffer.java:318)
org.apache.coyote.tomcat5.OutputBuffer.flush(OutputBuffer.java:297)
org.apache.coyote.tomcat5.CoyoteOutputStream.flush(CoyoteOutputStream.java:85)
sun.awt.image.codec.JPEGImageEncoderImpl.writeJPEGStream(Native
locked
sun.awt.image.codec.JPEGImageEncoderImpl.encode(JPEGImageEncoderImpl.java:472)
sun.awt.image.codec.JPEGImageEncoderImpl.encode(JPEGImageEncoderImpl.java:228)
locked
sun.awt.image.codec.JPEGImageEncoderImpl.encode(JPEGImageEncoderImpl.java:235)
locked
org.jfree.chart.encoders.SunJPEGEncoderAdapter.encode(SunJPEGEncoderAdapter.java:139)
org.jfree.chart.encoders.EncoderUtil.writeBufferedImage(EncoderUtil.java:136)
org.jfree.chart.ChartUtilities.writeChartAsJPEG(ChartUtilities.java:442)
org.jfree.chart.ChartUtilities.writeChartAsJPEG(ChartUtilities.java:388)
com.transerainc.rtmcui.util.ChartUtil.drawChartAsJPEG(ChartUtil.java:46)
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------
Feel free to call! Free PC-to-PC calls. Low rates on PC-to-Phone. Get Yahoo! Messenger with Voice