You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Martin Perez <mp...@gmail.com> on 2007/05/01 16:52:03 UTC
Re: Exception with Comet in Tomcat with high load
Hi,
Thanks for the tip Sebastian, but it doesn't work for me. I added your
class, wrapped all the output streams and synchronized close events but the
result is the same.
Exception in thread "MessageSender[]" java.lang.NullPointerException
at org.apache.coyote.http11.InternalNioOutputBuffer.addToBB(
InternalNioOutputBuffer.java:607)
at org.apache.coyote.http11.InternalNioOutputBuffer.commit(
InternalNioOutputBuffer.java:600)
at org.apache.coyote.http11.Http11NioProcessor.action(
Http11NioProcessor.java:1032)
at org.apache.coyote.Response.action(Response.java:181)
at org.apache.coyote.http11.InternalNioOutputBuffer.doWrite(
InternalNioOutputBuffer.java:571)
at org.apache.coyote.Response.doWrite(Response.java:560)
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(
OutputBuffer.java:353)
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java
:434)
at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:349)
at org.apache.catalina.connector.OutputBuffer.writeBytes(
OutputBuffer.java:381)
at org.apache.catalina.connector.OutputBuffer.write(
OutputBuffer.java:370)
at org.apache.catalina.connector.CoyoteOutputStream.write(
CoyoteOutputStream.java:89)
at org.apache.catalina.connector.CoyoteOutputStream.write(
CoyoteOutputStream.java:83)
at com.cognotec.streaming.SynchronizedOutputStream.write(
SynchronizedOutputStream.java:25)
at com.cognotec.streaming.CometRateSender.run(CometRateSender.java
:86)
at java.lang.Thread.run(Thread.java:595)
(As I said, I also synchronized all the close method calls either in stream
either in comet event)
So, the question is: Is this a bug or simply I have reached the scalability
limit for my box?
Cheers,
Martin
On 4/30/07, Sebastiaan van Erk <se...@sebster.com> wrote:
>
> Hi,
>
> I had the same problems. It was a synchronization issue with me. What I
> did was make a synchronized output stream wrapper (see attached file)
> which wraps all access to the output stream, and I synchronized on this
> output stream whenever I called event.close() on the Comet event
> (especially in the READ event). When doing serveral IO operations on the
> output stream which should not be interrupted, you can simply
> synchronize on it (since SynchronizedOutputStream locks on itself).
>
> Regards,
> Sebastiaan
>
>
> Martin Perez wrote:
> > Hi all,
> >
> > I have been testing Tomcat comet support during the last days. I've
> > created
> > a test application that simulates data streaming to multiple clients.
> > Clients open a connection to the server and the connection is hold
> > open. In
> > the server there are several "sender" threads that leverage comet
> > support to
> > periodically send stream updates to the subscribed clients. Once a
> client
> > receives a specified amount of data the communication is closed.
> >
> > So, I'm running the test with tomcat svn version from 1 week ago. I have
> > configured 150 tomcat threads, plus 25 acceptor threads. Tomcat runs in
> a
> > Sun-Fire-V240 box with 2 CPU cores. I have 75 sender threads that
> > periodically sleep (100ms) to try to reduce the CPU usage. Once a sender
> > thread wakes up, it will send all the data to the assigned clients.
> >
> > So basically the story is that the box is able to manage about 3500
> > persistent comet connections smoothly. But when I raise the level to
> 4000
> > (80% CPU usage, 1.2G mem.) I start to get these exceptions for some (not
> > all) connections:
> >
> > Exception in thread "MessageSender[]" java.lang.NullPointerException
> > at org.apache.coyote.http11.InternalNioOutputBuffer.addToBB(
> > InternalNioOutputBuffer.java:607)
> > at org.apache.coyote.http11.InternalNioOutputBuffer.commit(
> > InternalNioOutputBuffer.java:600)
> > at org.apache.coyote.http11.Http11NioProcessor.action(
> > Http11NioProcessor.java:1032)
> > at org.apache.coyote.Response.action(Response.java:181)
> > at org.apache.coyote.http11.InternalNioOutputBuffer.doWrite(
> > InternalNioOutputBuffer.java:571)
> > at org.apache.coyote.Response.doWrite(Response.java:560)
> > at org.apache.catalina.connector.OutputBuffer.realWriteBytes(
> > OutputBuffer.java:353)
> > at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(
> ByteChunk.java
> > :434)
> > at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java
> :349)
> > at org.apache.catalina.connector.OutputBuffer.writeBytes(
> > OutputBuffer.java:381)
> > at org.apache.catalina.connector.OutputBuffer.write(
> > OutputBuffer.java:370)
> > at org.apache.catalina.connector.CoyoteOutputStream.write(
> > CoyoteOutputStream.java:89)
> > at org.apache.catalina.connector.CoyoteOutputStream.write(
> > CoyoteOutputStream.java:83)
> > at com.cognotec.streaming.CometRateSender.run(
> CometRateSender.java
> > :86)
> > at java.lang.Thread.run(Thread.java:595)
> >
> > So well, does anybody knows what can be the problem? Any suggestions?
> > It can
> > be also my problem, so I will be happy to provide more information.
> >
> > Cheers,
> > Martin
> >
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Exception with Comet in Tomcat with high load
Posted by Martin Perez <mp...@gmail.com>.
Sebastiaan,
Yes you're right. I forgot to add something like isClosed() to your
SynchronizedOutputStream so I can check if it's closed before writing.
Anyways, I found that after a certain limit (for ex. 6000 -> 8000 users) the
system starts to behave incorrectly. Some clients never get an answer from
the server, some clients only receive partial data, and some of them really
end. I guess I found the limit in the box, but attached are the source code
files if someone wants to see the sample or simply spot something in it.
Cheers,
Martin
On 5/1/07, Sebastiaan van Erk <se...@sebster.com> wrote:
>
> Hi,
>
> Did you synchronize the event.close() with same lock you use to write to
> the output stream (the SynchronizedOuputStream instance)?
>
> I saw exactly the same stack trace you are getting due to (incorrect)
> concurrent access to the CoyoteOutputStream. The NullPointerException
> suggest that the request has already been cleaned up/recycled and yet
> you still write to it. Make sure that when you close the event, either
> the write becomes a null operation, or better, does not get called. It
> is a bug to call write on the response output stream after the event has
> been closed.
>
> With the SynchronizedOutputStream construction I am relying on Tomcat
> not to close or call any methods on the CoyoteOutputStream outside of
> the CometProcessor's event() method. I don't know if this is correct (it
> has worked for me in practice so far, and I've been doing some heavy
> duty testing); however I do not know what else I could synchronize on.
> If anybody knows a better approach, I would like to know. :-)
>
> Regards,
> Sebastiaan
>
> Martin Perez wrote:
> > Hi,
> >
> > Thanks for the tip Sebastian, but it doesn't work for me. I added your
> > class, wrapped all the output streams and synchronized close events
> > but the
> > result is the same.
> >
> > Exception in thread "MessageSender[]" java.lang.NullPointerException
> > at org.apache.coyote.http11.InternalNioOutputBuffer.addToBB(
> > InternalNioOutputBuffer.java:607)
> > at org.apache.coyote.http11.InternalNioOutputBuffer.commit(
> > InternalNioOutputBuffer.java:600)
> > at org.apache.coyote.http11.Http11NioProcessor.action(
> > Http11NioProcessor.java:1032)
> > at org.apache.coyote.Response.action(Response.java:181)
> > at org.apache.coyote.http11.InternalNioOutputBuffer.doWrite(
> > InternalNioOutputBuffer.java:571)
> > at org.apache.coyote.Response.doWrite(Response.java:560)
> > at org.apache.catalina.connector.OutputBuffer.realWriteBytes(
> > OutputBuffer.java:353)
> > at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(
> ByteChunk.java
> > :434)
> > at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java
> :349)
> > at org.apache.catalina.connector.OutputBuffer.writeBytes(
> > OutputBuffer.java:381)
> > at org.apache.catalina.connector.OutputBuffer.write(
> > OutputBuffer.java:370)
> > at org.apache.catalina.connector.CoyoteOutputStream.write(
> > CoyoteOutputStream.java:89)
> > at org.apache.catalina.connector.CoyoteOutputStream.write(
> > CoyoteOutputStream.java:83)
> > at com.cognotec.streaming.SynchronizedOutputStream.write(
> > SynchronizedOutputStream.java:25)
> > at com.cognotec.streaming.CometRateSender.run(
> CometRateSender.java
> > :86)
> > at java.lang.Thread.run(Thread.java:595)
> >
> > (As I said, I also synchronized all the close method calls either in
> > stream
> > either in comet event)
> >
> > So, the question is: Is this a bug or simply I have reached the
> > scalability
> > limit for my box?
> >
> > Cheers,
> > Martin
> >
> > On 4/30/07, Sebastiaan van Erk <se...@sebster.com> wrote:
> >>
> >> Hi,
> >>
> >> I had the same problems. It was a synchronization issue with me. What I
> >> did was make a synchronized output stream wrapper (see attached file)
> >> which wraps all access to the output stream, and I synchronized on this
> >> output stream whenever I called event.close() on the Comet event
> >> (especially in the READ event). When doing serveral IO operations on
> the
> >> output stream which should not be interrupted, you can simply
> >> synchronize on it (since SynchronizedOutputStream locks on itself).
> >>
> >> Regards,
> >> Sebastiaan
> >>
> >>
> >> Martin Perez wrote:
> >> > Hi all,
> >> >
> >> > I have been testing Tomcat comet support during the last days. I've
> >> > created
> >> > a test application that simulates data streaming to multiple clients.
> >> > Clients open a connection to the server and the connection is hold
> >> > open. In
> >> > the server there are several "sender" threads that leverage comet
> >> > support to
> >> > periodically send stream updates to the subscribed clients. Once a
> >> client
> >> > receives a specified amount of data the communication is closed.
> >> >
> >> > So, I'm running the test with tomcat svn version from 1 week ago. I
> >> have
> >> > configured 150 tomcat threads, plus 25 acceptor threads. Tomcat
> >> runs in
> >> a
> >> > Sun-Fire-V240 box with 2 CPU cores. I have 75 sender threads that
> >> > periodically sleep (100ms) to try to reduce the CPU usage. Once a
> >> sender
> >> > thread wakes up, it will send all the data to the assigned clients.
> >> >
> >> > So basically the story is that the box is able to manage about 3500
> >> > persistent comet connections smoothly. But when I raise the level to
> >> 4000
> >> > (80% CPU usage, 1.2G mem.) I start to get these exceptions for some
> >> (not
> >> > all) connections:
> >> >
> >> > Exception in thread "MessageSender[]" java.lang.NullPointerException
> >> > at org.apache.coyote.http11.InternalNioOutputBuffer.addToBB(
> >> > InternalNioOutputBuffer.java:607)
> >> > at org.apache.coyote.http11.InternalNioOutputBuffer.commit(
> >> > InternalNioOutputBuffer.java:600)
> >> > at org.apache.coyote.http11.Http11NioProcessor.action(
> >> > Http11NioProcessor.java:1032)
> >> > at org.apache.coyote.Response.action(Response.java:181)
> >> > at org.apache.coyote.http11.InternalNioOutputBuffer.doWrite(
> >> > InternalNioOutputBuffer.java:571)
> >> > at org.apache.coyote.Response.doWrite(Response.java:560)
> >> > at org.apache.catalina.connector.OutputBuffer.realWriteBytes(
> >> > OutputBuffer.java:353)
> >> > at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(
> >> ByteChunk.java
> >> > :434)
> >> > at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java
> >> :349)
> >> > at org.apache.catalina.connector.OutputBuffer.writeBytes(
> >> > OutputBuffer.java:381)
> >> > at org.apache.catalina.connector.OutputBuffer.write(
> >> > OutputBuffer.java:370)
> >> > at org.apache.catalina.connector.CoyoteOutputStream.write(
> >> > CoyoteOutputStream.java:89)
> >> > at org.apache.catalina.connector.CoyoteOutputStream.write(
> >> > CoyoteOutputStream.java:83)
> >> > at com.cognotec.streaming.CometRateSender.run(
> >> CometRateSender.java
> >> > :86)
> >> > at java.lang.Thread.run(Thread.java:595)
> >> >
> >> > So well, does anybody knows what can be the problem? Any suggestions?
> >> > It can
> >> > be also my problem, so I will be happy to provide more information.
> >> >
> >> > Cheers,
> >> > Martin
> >> >
> >>
> >> ---------------------------------------------------------------------
> >> To start a new topic, e-mail: users@tomcat.apache.org
> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >> For additional commands, e-mail: users-help@tomcat.apache.org
> >>
> >>
> >
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Exception with Comet in Tomcat with high load
Posted by Sebastiaan van Erk <se...@sebster.com>.
Hi,
Did you synchronize the event.close() with same lock you use to write to
the output stream (the SynchronizedOuputStream instance)?
I saw exactly the same stack trace you are getting due to (incorrect)
concurrent access to the CoyoteOutputStream. The NullPointerException
suggest that the request has already been cleaned up/recycled and yet
you still write to it. Make sure that when you close the event, either
the write becomes a null operation, or better, does not get called. It
is a bug to call write on the response output stream after the event has
been closed.
With the SynchronizedOutputStream construction I am relying on Tomcat
not to close or call any methods on the CoyoteOutputStream outside of
the CometProcessor's event() method. I don't know if this is correct (it
has worked for me in practice so far, and I've been doing some heavy
duty testing); however I do not know what else I could synchronize on.
If anybody knows a better approach, I would like to know. :-)
Regards,
Sebastiaan
Martin Perez wrote:
> Hi,
>
> Thanks for the tip Sebastian, but it doesn't work for me. I added your
> class, wrapped all the output streams and synchronized close events
> but the
> result is the same.
>
> Exception in thread "MessageSender[]" java.lang.NullPointerException
> at org.apache.coyote.http11.InternalNioOutputBuffer.addToBB(
> InternalNioOutputBuffer.java:607)
> at org.apache.coyote.http11.InternalNioOutputBuffer.commit(
> InternalNioOutputBuffer.java:600)
> at org.apache.coyote.http11.Http11NioProcessor.action(
> Http11NioProcessor.java:1032)
> at org.apache.coyote.Response.action(Response.java:181)
> at org.apache.coyote.http11.InternalNioOutputBuffer.doWrite(
> InternalNioOutputBuffer.java:571)
> at org.apache.coyote.Response.doWrite(Response.java:560)
> at org.apache.catalina.connector.OutputBuffer.realWriteBytes(
> OutputBuffer.java:353)
> at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java
> :434)
> at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:349)
> at org.apache.catalina.connector.OutputBuffer.writeBytes(
> OutputBuffer.java:381)
> at org.apache.catalina.connector.OutputBuffer.write(
> OutputBuffer.java:370)
> at org.apache.catalina.connector.CoyoteOutputStream.write(
> CoyoteOutputStream.java:89)
> at org.apache.catalina.connector.CoyoteOutputStream.write(
> CoyoteOutputStream.java:83)
> at com.cognotec.streaming.SynchronizedOutputStream.write(
> SynchronizedOutputStream.java:25)
> at com.cognotec.streaming.CometRateSender.run(CometRateSender.java
> :86)
> at java.lang.Thread.run(Thread.java:595)
>
> (As I said, I also synchronized all the close method calls either in
> stream
> either in comet event)
>
> So, the question is: Is this a bug or simply I have reached the
> scalability
> limit for my box?
>
> Cheers,
> Martin
>
> On 4/30/07, Sebastiaan van Erk <se...@sebster.com> wrote:
>>
>> Hi,
>>
>> I had the same problems. It was a synchronization issue with me. What I
>> did was make a synchronized output stream wrapper (see attached file)
>> which wraps all access to the output stream, and I synchronized on this
>> output stream whenever I called event.close() on the Comet event
>> (especially in the READ event). When doing serveral IO operations on the
>> output stream which should not be interrupted, you can simply
>> synchronize on it (since SynchronizedOutputStream locks on itself).
>>
>> Regards,
>> Sebastiaan
>>
>>
>> Martin Perez wrote:
>> > Hi all,
>> >
>> > I have been testing Tomcat comet support during the last days. I've
>> > created
>> > a test application that simulates data streaming to multiple clients.
>> > Clients open a connection to the server and the connection is hold
>> > open. In
>> > the server there are several "sender" threads that leverage comet
>> > support to
>> > periodically send stream updates to the subscribed clients. Once a
>> client
>> > receives a specified amount of data the communication is closed.
>> >
>> > So, I'm running the test with tomcat svn version from 1 week ago. I
>> have
>> > configured 150 tomcat threads, plus 25 acceptor threads. Tomcat
>> runs in
>> a
>> > Sun-Fire-V240 box with 2 CPU cores. I have 75 sender threads that
>> > periodically sleep (100ms) to try to reduce the CPU usage. Once a
>> sender
>> > thread wakes up, it will send all the data to the assigned clients.
>> >
>> > So basically the story is that the box is able to manage about 3500
>> > persistent comet connections smoothly. But when I raise the level to
>> 4000
>> > (80% CPU usage, 1.2G mem.) I start to get these exceptions for some
>> (not
>> > all) connections:
>> >
>> > Exception in thread "MessageSender[]" java.lang.NullPointerException
>> > at org.apache.coyote.http11.InternalNioOutputBuffer.addToBB(
>> > InternalNioOutputBuffer.java:607)
>> > at org.apache.coyote.http11.InternalNioOutputBuffer.commit(
>> > InternalNioOutputBuffer.java:600)
>> > at org.apache.coyote.http11.Http11NioProcessor.action(
>> > Http11NioProcessor.java:1032)
>> > at org.apache.coyote.Response.action(Response.java:181)
>> > at org.apache.coyote.http11.InternalNioOutputBuffer.doWrite(
>> > InternalNioOutputBuffer.java:571)
>> > at org.apache.coyote.Response.doWrite(Response.java:560)
>> > at org.apache.catalina.connector.OutputBuffer.realWriteBytes(
>> > OutputBuffer.java:353)
>> > at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(
>> ByteChunk.java
>> > :434)
>> > at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java
>> :349)
>> > at org.apache.catalina.connector.OutputBuffer.writeBytes(
>> > OutputBuffer.java:381)
>> > at org.apache.catalina.connector.OutputBuffer.write(
>> > OutputBuffer.java:370)
>> > at org.apache.catalina.connector.CoyoteOutputStream.write(
>> > CoyoteOutputStream.java:89)
>> > at org.apache.catalina.connector.CoyoteOutputStream.write(
>> > CoyoteOutputStream.java:83)
>> > at com.cognotec.streaming.CometRateSender.run(
>> CometRateSender.java
>> > :86)
>> > at java.lang.Thread.run(Thread.java:595)
>> >
>> > So well, does anybody knows what can be the problem? Any suggestions?
>> > It can
>> > be also my problem, so I will be happy to provide more information.
>> >
>> > Cheers,
>> > Martin
>> >
>>
>> ---------------------------------------------------------------------
>> To start a new topic, e-mail: users@tomcat.apache.org
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org