You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mina.apache.org by Vinod Panicker <vi...@gmail.com> on 2007/04/25 08:23:50 UTC
Permanent solution for OOM errors
Hi,
I think we've already got a lot of issues on the list regarding OOM
errors. In reality, every server needs to be able to manage I/O
efficiently to avoid high memory utilization, which ultimately results
in an OOM.
What can be done? MINA Connectors and Acceptors need to have the Read
and Write throttle filters by default. Once the specified buffer size
per session / overall has been exhausted, MINA needs to shift into
blocking mode to avoid an OOM on the server.
BTW, the WriteThrottleFilter contributed in DIRMINA-132 has some
issues, which we have fixed and will be contributing soon.
WDYAT?
Re: Permanent solution for OOM errors
Posted by peter royal <pr...@apache.org>.
On Apr 25, 2007, at 12:58 PM, Marcin Waldowski wrote:
> Trustin Lee wrote:
>> Very true. I agree with you absolutely. I think we need to
>> incorporate this feature into the Acceptor / Connector
>> implementations. We could add some hook methods to
>> AbstractIoFilterChain, but I didn't try it yet. (probably after
>> honey
>> moon ;)
>>
>
> I think that this is great idea. But I have second thoughts. Are we
> sure that we are not thinking about something which we will regret
> like ThreadModel?
I would tend to agree with you Marcin.. Perhaps they should be
filters that are in a filter-chain by default, or an easy-to-use
default via a builder. But I think we can safely keep it out of the
core and use our excellent model to layer in the functionality :)
-pete
--
proyal@apache.org - http://fotap.org/~osi
Re: Permanent solution for OOM errors
Posted by mat <fo...@gmail.com>.
But my server is broadcasting the message to all the clients.
2007/4/30, Paul Chen <ch...@gmail.com>:
>
> After curbing the read rate (one quota per acceptor) at a conservative
> rate,
> the write problem is automatically gone.
>
> This is not an approach that leads to optimal performance, but just a way
> to protect our Mina-based server not run into OOM.
>
> Cheers
>
> On 4/28/07, mat <fo...@gmail.com> wrote:
> >
> > Does ReadThrottle filter only solve the "Read" problem? What about
> > "Write"?
> > BTW, what is the max concurrent users you can reach?
> >
> >
> > 2007/4/29, Paul Chen <ch...@gmail.com>:
> > >
> > > No matter how we tune JVM, there's still a certain threshold that when
> > it
> > > is
> > > reached, a server should stop taking new requests or even close the
> > > session
> > > to free up resources, depends on the logic in the server.
> > >
> > > I observed and categorized 2 OOM scenarios in our Mina-based server:
> > >
> > > [1] Incoming reqs from SocketAcceptors are too fast and too many.
> Input
> > > queue blows up. Particularly when the Mina-based server has persistent
> > or
> > > long-lived connections w/ the clients which just keep pumping in reqs
> > w/o
> > > even needing time to establish a connection.
> > >
> > > [2] Outgoing reqs from SocketConnectors to a service provider are too
> > > fast/too many, much faster
> > > than the service provider can handle. Then the output queue
> > > (writeRequestQueue) blows up.
> > >
> > > I reported this problem earlier but I can live with this now by
> tweaking
> > > the
> > > ReadThrottle filter, though
> > > it is very hard to determine the threshold, especially when the
> > > mina-based
> > > server has multiple
> > > Acceptors. I end up assigning a "quota" per Acceptor.
> > >
> > > Regards
> > >
> > > On 4/28/07, Mark Webb <el...@gmail.com> wrote:
> > > >
> > > > have you looked into tweaking your JVM?
> > > >
> > > > http://java.sun.com/performance/reference/whitepapers/tuning.html
> > > >
> > > > On 4/28/07, mat <fo...@gmail.com> wrote:
> > > > > I also observed that the memory increaed way too fast when heavy
> > > > loading.
> > > > >
> > > > > 2007/4/26, Gaston Dombiak <ga...@jivesoftware.com>:
> > > > > >
> > > > > > Hey Mark,
> > > > > >
> > > > > > Based on the statistics I've been collecting from MINA I can
> tell
> > > that
> > > > > > our OOM problems happen when:
> > > > > >
> > > > > > 1) the eventQueue instance variable inside of ExecutorFilter
> fills
> > > up
> > > > > > (that would be the case for incoming traffic) or
> > > > > > 2) the SocketSessionImpl#getScheduledWriteRequests() queue
> starts
> > to
> > > > > > fill up (that would be the case for outgoing traffic)
> > > > > >
> > > > > > Regards,
> > > > > >
> > > > > > -- Gato
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Mark Webb [mailto:elihusmails@gmail.com]
> > > > > > Sent: Wednesday, April 25, 2007 1:53 PM
> > > > > > To: dev@mina.apache.org
> > > > > > Subject: Re: Permanent solution for OOM errors
> > > > > >
> > > > > > Maybe one approach would be to categorize the ways in which
> people
> > > > > > reach OOM conditions. Do these conditions happen only when the
> > > input
> > > > > > and/or output queues get really large, or are there other
> reasons.
> > > > > > Once we can start to categorize the OOM conditions that users
> are
> > > > > > experiencing, we can properly move forward with solutions.
> > > > > >
> > > > > > Basically what I mean is, if the OOM happens when queues fill
> up,
> > > then
> > > > > > throttle filters would be best. If the OOM happen for other
> > > reasons,
> > > > > > people could look at other areas of MINA. Just not sure there
> is
> > > one
> > > > > > solution that will fix all the problems.
> > > > > >
> > > > > > ...just my 2 cents.
> > > > > >
> > > > > > --
> > > > > > ..Cheers
> > > > > > Mark
> > > > > >
> > > > > >
> > > > > > On 4/25/07, Gaston Dombiak <ga...@jivesoftware.com> wrote:
> > > > > > > Hey Marcin,
> > > > > > >
> > > > > > > I see what you mean. I'm really fine with either option as
> long
> > as
> > > > > > MINA
> > > > > > > handles the OOM problem. As I posted a month or so ago the
> > > > throttling
> > > > > > > per connection is not enough. We (Openfire and probably
> others)
> > > also
> > > > > > > need a global throttling.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > -- Gato
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Marcin Waldowski [mailto:M.Waldowski@sulechow.net]
> > > > > > > Sent: Wednesday, April 25, 2007 1:36 PM
> > > > > > > To: dev@mina.apache.org
> > > > > > > Subject: Re: Permanent solution for OOM errors
> > > > > > >
> > > > > > > Hej Gaston,
> > > > > > >
> > > > > > > Gaston Dombiak wrote:
> > > > > > > > Hey Marcin,
> > > > > > > >
> > > > > > > > I'm curious to hear from you why do you think so? I
> absolutely
> > > > agree
> > > > > > > > that MINA has to provide easy ways to handle OOM problems
> due
> > to
> > > > > > heavy
> > > > > > > > incoming or outgoing traffic.
> > > > > > > >
> > > > > > > > This is by far the more common problem people are reporting
> > with
> > > > > > > > Openfire now that we moved to MINA (when under heavy load).
> > > > > > > >
> > > > > > >
> > > > > > > Hmm, my first thought was that it introduce the complexity of
> > > > > > > configuration like ThreadModel. But probably I'm wrong.
> > > > > > >
> > > > > > > I agree definitely that MINA need solution for OOM, but it
> could
> > > be
> > > > > > also
> > > > > > >
> > > > > > > ReadThrottleFilterBuilder and WriteThrottleFilter. They are
> easy
> > > way
> > > > > > to
> > > > > > > handle OOM and (what is important) users are aware that they
> use
> > > it.
> > > > > > >
> > > > > > > But I'm not agains integrate it as default to
> > Acceptor/Connector,
> > > > > > really
> > > > > > >
> > > > > > > :) Meybe I even souldn't post my doubts to this subject
> because
> > I
> > > > > > > haven't contributed to MINA so far...
> > > > > > >
> > > > > > > Regards, Marcin
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > ..Cheers
> > > > Mark
> > > >
> > >
> >
>
Re: Permanent solution for OOM errors
Posted by Trustin Lee <tr...@gmail.com>.
On 4/30/07, mat <fo...@gmail.com> wrote:
> *I only found ReadThrottleFilterBuilder in 1.1.0 API. What about Write
> ThrottleFilter?*
https://issues.apache.org/jira/browse/DIRMINA-302
I just didn't have time to include the patch yet. :)
Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/
--
PGP Key ID: 0x0255ECA6
Re: Permanent solution for OOM errors
Posted by mat <fo...@gmail.com>.
*I only found ReadThrottleFilterBuilder in 1.1.0 API. What about Write
ThrottleFilter?*
2007/4/30, Vinod Panicker <vi...@gmail.com>:
>
> On 4/29/07, Paul Chen <ch...@gmail.com> wrote:
> > After curbing the read rate (one quota per acceptor) at a conservative
> rate,
> > the write problem is automatically gone.
>
> Not really. The read quota can be applied per acceptor, but the
> server might have connectors that could be writing to slower clients,
> leading to another OOM.
>
> > This is not an approach that leads to optimal performance, but just a
> way
> > to protect our Mina-based server not run into OOM.
>
> Using throttling is a solution that leads to better performance
> because the server can provide a good quality of service. Without
> throttling, all clients get affected because the server is not able to
> manage the incoming and outgoing data, resulting in an OOM.
>
> Regards,
> Vinod.
>
> PS: I was referring to DIRMINA-302 in my original post, not 132.
>
Re: Permanent solution for OOM errors
Posted by Paul Chen <ch...@gmail.com>.
Hi, folks,
Here is what I did to solve the OOM problem caused by writeRequest Queues
piling up
when many SocketConnector sessions bombard a service provider thus making
it very slow in responding.
It is the sum of all writeRequest queues that blows up the memory.
The idea is simple, but, alas, it is not that generic. But you get the idea
when reading
the code comments.
Not sure what's the best way/place to modify the Mina src to create global
views
across sessions?
Thanks for comments
package ....
import org.apache.mina.common.IoFilterAdapter;
import org.apache.mina.common.IoSession;
/**
* Prevent out of memory caused by output write queue piling up
*
* This filter is attached on a Mina SocketConnector session to:
*
* - Check outputQ quota of a service provider, the quota implicitly
indicates the max capacity of the
* service provider
*
* This filter complements the Mina ReadThrottleFilter which I attach to
each acceptor opened by my
* Mina-based server, and assign a max quota to each. I preconfigure them to
make sure the
* total of all acceptor quota values don't exceed my system memory limit.
*/
public class WriteThrottleFilter extends IoFilterAdapter {
/** Write a request to a service provider via a SocketConnector */
@Override
public void filterWrite(NextFilter nextFilter, IoSession session,
WriteRequest writeRequest) throws Exception {
// A codec turned it to my Request obj already
Request req = (Request) message;
int reqSize = req.size();
String svc = req.getServiceName();
// Check outputQ quota for the svc
//
// Make sure we don't exceed max capacity pre-configured for this
service provider
// There can be many SocketConnectors sending reqs to the service
provider. When
// it is slow, all write queues connected to this service pile up in
Mina. Thus I
// need a global view instead of a single
session.getScheduledWriteBytes()
if (reqSize + svcProviders.outQueueSize(svc) >
svcProviders.outQueueQuota(svc)) {
// Mark the service as busy, reply an error response to the
client
return;
}
svcProviders.incrOutQueue(svc, reqSize);
nextFilter.filterWrite(session, writeRequest);
}
} // End of class
On 5/3/07, Martin Ritchie <ri...@apache.org> wrote:
>
> That is great. I look forward to using the new filter as we have been
> having a lot more OOM problems on our clients under load testing
> (although that is as much a fault of the test app).
>
> On 03/05/07, Vinod Panicker <vi...@gmail.com> wrote:
> > I've already done that. It's being tested right now. We'll be
> > contributing that as soon as we are happy about the performance under
> > high load.
> >
> > On 5/2/07, Martin Ritchie <ri...@apache.org> wrote:
> > > On 30/04/07, Paul Chen <ch...@gmail.com> wrote:
> > > > I'm refining that to make it more presentable and generic, will
> publish
> > > that
> > > > in 1 week or 2.
> > > >
> > > > On 4/30/07, Gaston Dombiak <ga...@jivesoftware.com> wrote:
> > > > >
> > > > > Hey Paul,
> > > > >
> > > > > That is exactly what I was planning to implement for our case. Can
> you
> > > > > send me your modified version? I would like to include it in
> Openfire.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > -- Gato
> > > > >
> > > > > -----Original Message-----
> > > > > From: Paul Chen [mailto:chenpaul@gmail.com]
> > > > > Sent: Monday, April 30, 2007 1:07 PM
> > > > > To: dev@mina.apache.org
> > > > > Subject: Re: Permanent solution for OOM errors
> > > > >
> > > > > I actually *tweaked* the read throttle to take the output Queue
> size
> > > > > into account to avoid the slow write syndrome. Maybe I should call
> > > > > it traffic control throttle instead.
> > > > >
> > > > > Cheers
> > > > >
> > > > > On 4/29/07, Vinod Panicker <vi...@gmail.com> wrote:
> > > > > >
> > > > > > On 4/29/07, Paul Chen <ch...@gmail.com> wrote:
> > > > > > > After curbing the read rate (one quota per acceptor) at a
> > > > > conservative
> > > > > > rate,
> > > > > > > the write problem is automatically gone.
> > > > > >
> > > > > > Not really. The read quota can be applied per acceptor, but the
> > > > > > server might have connectors that could be writing to slower
> clients,
> > > > > > leading to another OOM.
> > >
> > > Hi,
> > >
> > > I've been watching this thread grow but not had time to contribute
> > > (Sorry about that).
> > >
> > > The Qpid project is hoping to move the latest released version of mina
> > > and so I would be happy to take another look at the
> > > WriteThrottleFilter (on DIRMINA-302) to see how it can be tidied up.
> > > Specifically the filter currently limits by number of requests whereas
> > > the number of bytes on the queue would be far better solution.
> > >
> > > --
> > > Martin Ritchie
> > >
> >
>
>
> --
> Martin Ritchie
>
Re: Permanent solution for OOM errors
Posted by Martin Ritchie <ri...@apache.org>.
That is great. I look forward to using the new filter as we have been
having a lot more OOM problems on our clients under load testing
(although that is as much a fault of the test app).
On 03/05/07, Vinod Panicker <vi...@gmail.com> wrote:
> I've already done that. It's being tested right now. We'll be
> contributing that as soon as we are happy about the performance under
> high load.
>
> On 5/2/07, Martin Ritchie <ri...@apache.org> wrote:
> > On 30/04/07, Paul Chen <ch...@gmail.com> wrote:
> > > I'm refining that to make it more presentable and generic, will publish
> > that
> > > in 1 week or 2.
> > >
> > > On 4/30/07, Gaston Dombiak <ga...@jivesoftware.com> wrote:
> > > >
> > > > Hey Paul,
> > > >
> > > > That is exactly what I was planning to implement for our case. Can you
> > > > send me your modified version? I would like to include it in Openfire.
> > > >
> > > > Thanks,
> > > >
> > > > -- Gato
> > > >
> > > > -----Original Message-----
> > > > From: Paul Chen [mailto:chenpaul@gmail.com]
> > > > Sent: Monday, April 30, 2007 1:07 PM
> > > > To: dev@mina.apache.org
> > > > Subject: Re: Permanent solution for OOM errors
> > > >
> > > > I actually *tweaked* the read throttle to take the output Queue size
> > > > into account to avoid the slow write syndrome. Maybe I should call
> > > > it traffic control throttle instead.
> > > >
> > > > Cheers
> > > >
> > > > On 4/29/07, Vinod Panicker <vi...@gmail.com> wrote:
> > > > >
> > > > > On 4/29/07, Paul Chen <ch...@gmail.com> wrote:
> > > > > > After curbing the read rate (one quota per acceptor) at a
> > > > conservative
> > > > > rate,
> > > > > > the write problem is automatically gone.
> > > > >
> > > > > Not really. The read quota can be applied per acceptor, but the
> > > > > server might have connectors that could be writing to slower clients,
> > > > > leading to another OOM.
> >
> > Hi,
> >
> > I've been watching this thread grow but not had time to contribute
> > (Sorry about that).
> >
> > The Qpid project is hoping to move the latest released version of mina
> > and so I would be happy to take another look at the
> > WriteThrottleFilter (on DIRMINA-302) to see how it can be tidied up.
> > Specifically the filter currently limits by number of requests whereas
> > the number of bytes on the queue would be far better solution.
> >
> > --
> > Martin Ritchie
> >
>
--
Martin Ritchie
Re: Permanent solution for OOM errors
Posted by Vinod Panicker <vi...@gmail.com>.
I've already done that. It's being tested right now. We'll be
contributing that as soon as we are happy about the performance under
high load.
On 5/2/07, Martin Ritchie <ri...@apache.org> wrote:
> On 30/04/07, Paul Chen <ch...@gmail.com> wrote:
> > I'm refining that to make it more presentable and generic, will publish
> that
> > in 1 week or 2.
> >
> > On 4/30/07, Gaston Dombiak <ga...@jivesoftware.com> wrote:
> > >
> > > Hey Paul,
> > >
> > > That is exactly what I was planning to implement for our case. Can you
> > > send me your modified version? I would like to include it in Openfire.
> > >
> > > Thanks,
> > >
> > > -- Gato
> > >
> > > -----Original Message-----
> > > From: Paul Chen [mailto:chenpaul@gmail.com]
> > > Sent: Monday, April 30, 2007 1:07 PM
> > > To: dev@mina.apache.org
> > > Subject: Re: Permanent solution for OOM errors
> > >
> > > I actually *tweaked* the read throttle to take the output Queue size
> > > into account to avoid the slow write syndrome. Maybe I should call
> > > it traffic control throttle instead.
> > >
> > > Cheers
> > >
> > > On 4/29/07, Vinod Panicker <vi...@gmail.com> wrote:
> > > >
> > > > On 4/29/07, Paul Chen <ch...@gmail.com> wrote:
> > > > > After curbing the read rate (one quota per acceptor) at a
> > > conservative
> > > > rate,
> > > > > the write problem is automatically gone.
> > > >
> > > > Not really. The read quota can be applied per acceptor, but the
> > > > server might have connectors that could be writing to slower clients,
> > > > leading to another OOM.
>
> Hi,
>
> I've been watching this thread grow but not had time to contribute
> (Sorry about that).
>
> The Qpid project is hoping to move the latest released version of mina
> and so I would be happy to take another look at the
> WriteThrottleFilter (on DIRMINA-302) to see how it can be tidied up.
> Specifically the filter currently limits by number of requests whereas
> the number of bytes on the queue would be far better solution.
>
> --
> Martin Ritchie
>
Re: Permanent solution for OOM errors
Posted by Martin Ritchie <ri...@apache.org>.
On 30/04/07, Paul Chen <ch...@gmail.com> wrote:
> I'm refining that to make it more presentable and generic, will publish that
> in 1 week or 2.
>
> On 4/30/07, Gaston Dombiak <ga...@jivesoftware.com> wrote:
> >
> > Hey Paul,
> >
> > That is exactly what I was planning to implement for our case. Can you
> > send me your modified version? I would like to include it in Openfire.
> >
> > Thanks,
> >
> > -- Gato
> >
> > -----Original Message-----
> > From: Paul Chen [mailto:chenpaul@gmail.com]
> > Sent: Monday, April 30, 2007 1:07 PM
> > To: dev@mina.apache.org
> > Subject: Re: Permanent solution for OOM errors
> >
> > I actually *tweaked* the read throttle to take the output Queue size
> > into account to avoid the slow write syndrome. Maybe I should call
> > it traffic control throttle instead.
> >
> > Cheers
> >
> > On 4/29/07, Vinod Panicker <vi...@gmail.com> wrote:
> > >
> > > On 4/29/07, Paul Chen <ch...@gmail.com> wrote:
> > > > After curbing the read rate (one quota per acceptor) at a
> > conservative
> > > rate,
> > > > the write problem is automatically gone.
> > >
> > > Not really. The read quota can be applied per acceptor, but the
> > > server might have connectors that could be writing to slower clients,
> > > leading to another OOM.
Hi,
I've been watching this thread grow but not had time to contribute
(Sorry about that).
The Qpid project is hoping to move the latest released version of mina
and so I would be happy to take another look at the
WriteThrottleFilter (on DIRMINA-302) to see how it can be tidied up.
Specifically the filter currently limits by number of requests whereas
the number of bytes on the queue would be far better solution.
--
Martin Ritchie
Re: Permanent solution for OOM errors
Posted by Paul Chen <ch...@gmail.com>.
I'm refining that to make it more presentable and generic, will publish that
in 1 week or 2.
On 4/30/07, Gaston Dombiak <ga...@jivesoftware.com> wrote:
>
> Hey Paul,
>
> That is exactly what I was planning to implement for our case. Can you
> send me your modified version? I would like to include it in Openfire.
>
> Thanks,
>
> -- Gato
>
> -----Original Message-----
> From: Paul Chen [mailto:chenpaul@gmail.com]
> Sent: Monday, April 30, 2007 1:07 PM
> To: dev@mina.apache.org
> Subject: Re: Permanent solution for OOM errors
>
> I actually *tweaked* the read throttle to take the output Queue size
> into account to avoid the slow write syndrome. Maybe I should call
> it traffic control throttle instead.
>
> Cheers
>
> On 4/29/07, Vinod Panicker <vi...@gmail.com> wrote:
> >
> > On 4/29/07, Paul Chen <ch...@gmail.com> wrote:
> > > After curbing the read rate (one quota per acceptor) at a
> conservative
> > rate,
> > > the write problem is automatically gone.
> >
> > Not really. The read quota can be applied per acceptor, but the
> > server might have connectors that could be writing to slower clients,
> > leading to another OOM.
> >
> >
>
RE: Permanent solution for OOM errors
Posted by Gaston Dombiak <ga...@jivesoftware.com>.
Hey Paul,
That is exactly what I was planning to implement for our case. Can you
send me your modified version? I would like to include it in Openfire.
Thanks,
-- Gato
-----Original Message-----
From: Paul Chen [mailto:chenpaul@gmail.com]
Sent: Monday, April 30, 2007 1:07 PM
To: dev@mina.apache.org
Subject: Re: Permanent solution for OOM errors
I actually *tweaked* the read throttle to take the output Queue size
into account to avoid the slow write syndrome. Maybe I should call
it traffic control throttle instead.
Cheers
On 4/29/07, Vinod Panicker <vi...@gmail.com> wrote:
>
> On 4/29/07, Paul Chen <ch...@gmail.com> wrote:
> > After curbing the read rate (one quota per acceptor) at a
conservative
> rate,
> > the write problem is automatically gone.
>
> Not really. The read quota can be applied per acceptor, but the
> server might have connectors that could be writing to slower clients,
> leading to another OOM.
>
>
Re: Permanent solution for OOM errors
Posted by Paul Chen <ch...@gmail.com>.
I actually *tweaked* the read throttle to take the output Queue size
into account to avoid the slow write syndrome. Maybe I should call
it traffic control throttle instead.
Cheers
On 4/29/07, Vinod Panicker <vi...@gmail.com> wrote:
>
> On 4/29/07, Paul Chen <ch...@gmail.com> wrote:
> > After curbing the read rate (one quota per acceptor) at a conservative
> rate,
> > the write problem is automatically gone.
>
> Not really. The read quota can be applied per acceptor, but the
> server might have connectors that could be writing to slower clients,
> leading to another OOM.
>
>
Re: Permanent solution for OOM errors
Posted by Trustin Lee <tr...@gmail.com>.
Sorry for not responding to this important thread actively recently.
On 4/30/07, Vinod Panicker <vi...@gmail.com> wrote:
> On 4/29/07, Paul Chen <ch...@gmail.com> wrote:
> > After curbing the read rate (one quota per acceptor) at a conservative rate,
> > the write problem is automatically gone.
>
> Not really. The read quota can be applied per acceptor, but the
> server might have connectors that could be writing to slower clients,
> leading to another OOM.
True.
> > This is not an approach that leads to optimal performance, but just a way
> > to protect our Mina-based server not run into OOM.
>
> Using throttling is a solution that leads to better performance
> because the server can provide a good quality of service. Without
> throttling, all clients get affected because the server is not able to
> manage the incoming and outgoing data, resulting in an OOM.
To wrap up the discussion so far:
1) MINA must provide a solution that takes care of OOM situation
caused by too fast or slow client/server.
2) Throttling only read opetions won't solve all OOM cases (i.e. slow
client), we need to provide a filter (or filter builder) that can
throttle write operations, too.
Here's what I thought so far:
1) We need to merge ReadThrottleFilterBuilder and
WriteBufferLimitFilterBuffer (DIRMINA-302) to provide a one-stop
solution for the OOM issue.
1-a) We don't want to break backward compatiblity, so this solution
will appear in 2.0.0-M1-SNAPSHOT.
1-b) We could provide WriteBufferLimitFilterBuilder in the next
release of branches/1.0 and branches/1.1. This is a feature addition
strictly speaking, so we might need to create a new branch (i.e. 1.2
and 1.3? :).
2) Using a filter mechanism rather than providing a new parameter is
more elegant, and there's nothing to learn. (I agree.)
Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/
--
PGP Key ID: 0x0255ECA6
Re: Permanent solution for OOM errors
Posted by Vinod Panicker <vi...@gmail.com>.
On 4/29/07, Paul Chen <ch...@gmail.com> wrote:
> After curbing the read rate (one quota per acceptor) at a conservative rate,
> the write problem is automatically gone.
Not really. The read quota can be applied per acceptor, but the
server might have connectors that could be writing to slower clients,
leading to another OOM.
> This is not an approach that leads to optimal performance, but just a way
> to protect our Mina-based server not run into OOM.
Using throttling is a solution that leads to better performance
because the server can provide a good quality of service. Without
throttling, all clients get affected because the server is not able to
manage the incoming and outgoing data, resulting in an OOM.
Regards,
Vinod.
PS: I was referring to DIRMINA-302 in my original post, not 132.
Re: Permanent solution for OOM errors
Posted by Paul Chen <ch...@gmail.com>.
After curbing the read rate (one quota per acceptor) at a conservative rate,
the write problem is automatically gone.
This is not an approach that leads to optimal performance, but just a way
to protect our Mina-based server not run into OOM.
Cheers
On 4/28/07, mat <fo...@gmail.com> wrote:
>
> Does ReadThrottle filter only solve the "Read" problem? What about
> "Write"?
> BTW, what is the max concurrent users you can reach?
>
>
> 2007/4/29, Paul Chen <ch...@gmail.com>:
> >
> > No matter how we tune JVM, there's still a certain threshold that when
> it
> > is
> > reached, a server should stop taking new requests or even close the
> > session
> > to free up resources, depends on the logic in the server.
> >
> > I observed and categorized 2 OOM scenarios in our Mina-based server:
> >
> > [1] Incoming reqs from SocketAcceptors are too fast and too many. Input
> > queue blows up. Particularly when the Mina-based server has persistent
> or
> > long-lived connections w/ the clients which just keep pumping in reqs
> w/o
> > even needing time to establish a connection.
> >
> > [2] Outgoing reqs from SocketConnectors to a service provider are too
> > fast/too many, much faster
> > than the service provider can handle. Then the output queue
> > (writeRequestQueue) blows up.
> >
> > I reported this problem earlier but I can live with this now by tweaking
> > the
> > ReadThrottle filter, though
> > it is very hard to determine the threshold, especially when the
> > mina-based
> > server has multiple
> > Acceptors. I end up assigning a "quota" per Acceptor.
> >
> > Regards
> >
> > On 4/28/07, Mark Webb <el...@gmail.com> wrote:
> > >
> > > have you looked into tweaking your JVM?
> > >
> > > http://java.sun.com/performance/reference/whitepapers/tuning.html
> > >
> > > On 4/28/07, mat <fo...@gmail.com> wrote:
> > > > I also observed that the memory increaed way too fast when heavy
> > > loading.
> > > >
> > > > 2007/4/26, Gaston Dombiak <ga...@jivesoftware.com>:
> > > > >
> > > > > Hey Mark,
> > > > >
> > > > > Based on the statistics I've been collecting from MINA I can tell
> > that
> > > > > our OOM problems happen when:
> > > > >
> > > > > 1) the eventQueue instance variable inside of ExecutorFilter fills
> > up
> > > > > (that would be the case for incoming traffic) or
> > > > > 2) the SocketSessionImpl#getScheduledWriteRequests() queue starts
> to
> > > > > fill up (that would be the case for outgoing traffic)
> > > > >
> > > > > Regards,
> > > > >
> > > > > -- Gato
> > > > >
> > > > > -----Original Message-----
> > > > > From: Mark Webb [mailto:elihusmails@gmail.com]
> > > > > Sent: Wednesday, April 25, 2007 1:53 PM
> > > > > To: dev@mina.apache.org
> > > > > Subject: Re: Permanent solution for OOM errors
> > > > >
> > > > > Maybe one approach would be to categorize the ways in which people
> > > > > reach OOM conditions. Do these conditions happen only when the
> > input
> > > > > and/or output queues get really large, or are there other reasons.
> > > > > Once we can start to categorize the OOM conditions that users are
> > > > > experiencing, we can properly move forward with solutions.
> > > > >
> > > > > Basically what I mean is, if the OOM happens when queues fill up,
> > then
> > > > > throttle filters would be best. If the OOM happen for other
> > reasons,
> > > > > people could look at other areas of MINA. Just not sure there is
> > one
> > > > > solution that will fix all the problems.
> > > > >
> > > > > ...just my 2 cents.
> > > > >
> > > > > --
> > > > > ..Cheers
> > > > > Mark
> > > > >
> > > > >
> > > > > On 4/25/07, Gaston Dombiak <ga...@jivesoftware.com> wrote:
> > > > > > Hey Marcin,
> > > > > >
> > > > > > I see what you mean. I'm really fine with either option as long
> as
> > > > > MINA
> > > > > > handles the OOM problem. As I posted a month or so ago the
> > > throttling
> > > > > > per connection is not enough. We (Openfire and probably others)
> > also
> > > > > > need a global throttling.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > -- Gato
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Marcin Waldowski [mailto:M.Waldowski@sulechow.net]
> > > > > > Sent: Wednesday, April 25, 2007 1:36 PM
> > > > > > To: dev@mina.apache.org
> > > > > > Subject: Re: Permanent solution for OOM errors
> > > > > >
> > > > > > Hej Gaston,
> > > > > >
> > > > > > Gaston Dombiak wrote:
> > > > > > > Hey Marcin,
> > > > > > >
> > > > > > > I'm curious to hear from you why do you think so? I absolutely
> > > agree
> > > > > > > that MINA has to provide easy ways to handle OOM problems due
> to
> > > > > heavy
> > > > > > > incoming or outgoing traffic.
> > > > > > >
> > > > > > > This is by far the more common problem people are reporting
> with
> > > > > > > Openfire now that we moved to MINA (when under heavy load).
> > > > > > >
> > > > > >
> > > > > > Hmm, my first thought was that it introduce the complexity of
> > > > > > configuration like ThreadModel. But probably I'm wrong.
> > > > > >
> > > > > > I agree definitely that MINA need solution for OOM, but it could
> > be
> > > > > also
> > > > > >
> > > > > > ReadThrottleFilterBuilder and WriteThrottleFilter. They are easy
> > way
> > > > > to
> > > > > > handle OOM and (what is important) users are aware that they use
> > it.
> > > > > >
> > > > > > But I'm not agains integrate it as default to
> Acceptor/Connector,
> > > > > really
> > > > > >
> > > > > > :) Meybe I even souldn't post my doubts to this subject because
> I
> > > > > > haven't contributed to MINA so far...
> > > > > >
> > > > > > Regards, Marcin
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > ..Cheers
> > > Mark
> > >
> >
>
Re: Permanent solution for OOM errors
Posted by mat <fo...@gmail.com>.
Does ReadThrottle filter only solve the "Read" problem? What about "Write"?
BTW, what is the max concurrent users you can reach?
2007/4/29, Paul Chen <ch...@gmail.com>:
>
> No matter how we tune JVM, there's still a certain threshold that when it
> is
> reached, a server should stop taking new requests or even close the
> session
> to free up resources, depends on the logic in the server.
>
> I observed and categorized 2 OOM scenarios in our Mina-based server:
>
> [1] Incoming reqs from SocketAcceptors are too fast and too many. Input
> queue blows up. Particularly when the Mina-based server has persistent or
> long-lived connections w/ the clients which just keep pumping in reqs w/o
> even needing time to establish a connection.
>
> [2] Outgoing reqs from SocketConnectors to a service provider are too
> fast/too many, much faster
> than the service provider can handle. Then the output queue
> (writeRequestQueue) blows up.
>
> I reported this problem earlier but I can live with this now by tweaking
> the
> ReadThrottle filter, though
> it is very hard to determine the threshold, especially when the
> mina-based
> server has multiple
> Acceptors. I end up assigning a "quota" per Acceptor.
>
> Regards
>
> On 4/28/07, Mark Webb <el...@gmail.com> wrote:
> >
> > have you looked into tweaking your JVM?
> >
> > http://java.sun.com/performance/reference/whitepapers/tuning.html
> >
> > On 4/28/07, mat <fo...@gmail.com> wrote:
> > > I also observed that the memory increaed way too fast when heavy
> > loading.
> > >
> > > 2007/4/26, Gaston Dombiak <ga...@jivesoftware.com>:
> > > >
> > > > Hey Mark,
> > > >
> > > > Based on the statistics I've been collecting from MINA I can tell
> that
> > > > our OOM problems happen when:
> > > >
> > > > 1) the eventQueue instance variable inside of ExecutorFilter fills
> up
> > > > (that would be the case for incoming traffic) or
> > > > 2) the SocketSessionImpl#getScheduledWriteRequests() queue starts to
> > > > fill up (that would be the case for outgoing traffic)
> > > >
> > > > Regards,
> > > >
> > > > -- Gato
> > > >
> > > > -----Original Message-----
> > > > From: Mark Webb [mailto:elihusmails@gmail.com]
> > > > Sent: Wednesday, April 25, 2007 1:53 PM
> > > > To: dev@mina.apache.org
> > > > Subject: Re: Permanent solution for OOM errors
> > > >
> > > > Maybe one approach would be to categorize the ways in which people
> > > > reach OOM conditions. Do these conditions happen only when the
> input
> > > > and/or output queues get really large, or are there other reasons.
> > > > Once we can start to categorize the OOM conditions that users are
> > > > experiencing, we can properly move forward with solutions.
> > > >
> > > > Basically what I mean is, if the OOM happens when queues fill up,
> then
> > > > throttle filters would be best. If the OOM happen for other
> reasons,
> > > > people could look at other areas of MINA. Just not sure there is
> one
> > > > solution that will fix all the problems.
> > > >
> > > > ...just my 2 cents.
> > > >
> > > > --
> > > > ..Cheers
> > > > Mark
> > > >
> > > >
> > > > On 4/25/07, Gaston Dombiak <ga...@jivesoftware.com> wrote:
> > > > > Hey Marcin,
> > > > >
> > > > > I see what you mean. I'm really fine with either option as long as
> > > > MINA
> > > > > handles the OOM problem. As I posted a month or so ago the
> > throttling
> > > > > per connection is not enough. We (Openfire and probably others)
> also
> > > > > need a global throttling.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > -- Gato
> > > > >
> > > > > -----Original Message-----
> > > > > From: Marcin Waldowski [mailto:M.Waldowski@sulechow.net]
> > > > > Sent: Wednesday, April 25, 2007 1:36 PM
> > > > > To: dev@mina.apache.org
> > > > > Subject: Re: Permanent solution for OOM errors
> > > > >
> > > > > Hej Gaston,
> > > > >
> > > > > Gaston Dombiak wrote:
> > > > > > Hey Marcin,
> > > > > >
> > > > > > I'm curious to hear from you why do you think so? I absolutely
> > agree
> > > > > > that MINA has to provide easy ways to handle OOM problems due to
> > > > heavy
> > > > > > incoming or outgoing traffic.
> > > > > >
> > > > > > This is by far the more common problem people are reporting with
> > > > > > Openfire now that we moved to MINA (when under heavy load).
> > > > > >
> > > > >
> > > > > Hmm, my first thought was that it introduce the complexity of
> > > > > configuration like ThreadModel. But probably I'm wrong.
> > > > >
> > > > > I agree definitely that MINA need solution for OOM, but it could
> be
> > > > also
> > > > >
> > > > > ReadThrottleFilterBuilder and WriteThrottleFilter. They are easy
> way
> > > > to
> > > > > handle OOM and (what is important) users are aware that they use
> it.
> > > > >
> > > > > But I'm not agains integrate it as default to Acceptor/Connector,
> > > > really
> > > > >
> > > > > :) Meybe I even souldn't post my doubts to this subject because I
> > > > > haven't contributed to MINA so far...
> > > > >
> > > > > Regards, Marcin
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> > ..Cheers
> > Mark
> >
>
Re: Permanent solution for OOM errors
Posted by Paul Chen <ch...@gmail.com>.
No matter how we tune JVM, there's still a certain threshold that when it is
reached, a server should stop taking new requests or even close the session
to free up resources, depends on the logic in the server.
I observed and categorized 2 OOM scenarios in our Mina-based server:
[1] Incoming reqs from SocketAcceptors are too fast and too many. Input
queue blows up. Particularly when the Mina-based server has persistent or
long-lived connections w/ the clients which just keep pumping in reqs w/o
even needing time to establish a connection.
[2] Outgoing reqs from SocketConnectors to a service provider are too
fast/too many, much faster
than the service provider can handle. Then the output queue
(writeRequestQueue) blows up.
I reported this problem earlier but I can live with this now by tweaking the
ReadThrottle filter, though
it is very hard to determine the threshold, especially when the mina-based
server has multiple
Acceptors. I end up assigning a "quota" per Acceptor.
Regards
On 4/28/07, Mark Webb <el...@gmail.com> wrote:
>
> have you looked into tweaking your JVM?
>
> http://java.sun.com/performance/reference/whitepapers/tuning.html
>
> On 4/28/07, mat <fo...@gmail.com> wrote:
> > I also observed that the memory increaed way too fast when heavy
> loading.
> >
> > 2007/4/26, Gaston Dombiak <ga...@jivesoftware.com>:
> > >
> > > Hey Mark,
> > >
> > > Based on the statistics I've been collecting from MINA I can tell that
> > > our OOM problems happen when:
> > >
> > > 1) the eventQueue instance variable inside of ExecutorFilter fills up
> > > (that would be the case for incoming traffic) or
> > > 2) the SocketSessionImpl#getScheduledWriteRequests() queue starts to
> > > fill up (that would be the case for outgoing traffic)
> > >
> > > Regards,
> > >
> > > -- Gato
> > >
> > > -----Original Message-----
> > > From: Mark Webb [mailto:elihusmails@gmail.com]
> > > Sent: Wednesday, April 25, 2007 1:53 PM
> > > To: dev@mina.apache.org
> > > Subject: Re: Permanent solution for OOM errors
> > >
> > > Maybe one approach would be to categorize the ways in which people
> > > reach OOM conditions. Do these conditions happen only when the input
> > > and/or output queues get really large, or are there other reasons.
> > > Once we can start to categorize the OOM conditions that users are
> > > experiencing, we can properly move forward with solutions.
> > >
> > > Basically what I mean is, if the OOM happens when queues fill up, then
> > > throttle filters would be best. If the OOM happen for other reasons,
> > > people could look at other areas of MINA. Just not sure there is one
> > > solution that will fix all the problems.
> > >
> > > ...just my 2 cents.
> > >
> > > --
> > > ..Cheers
> > > Mark
> > >
> > >
> > > On 4/25/07, Gaston Dombiak <ga...@jivesoftware.com> wrote:
> > > > Hey Marcin,
> > > >
> > > > I see what you mean. I'm really fine with either option as long as
> > > MINA
> > > > handles the OOM problem. As I posted a month or so ago the
> throttling
> > > > per connection is not enough. We (Openfire and probably others) also
> > > > need a global throttling.
> > > >
> > > > Thanks,
> > > >
> > > > -- Gato
> > > >
> > > > -----Original Message-----
> > > > From: Marcin Waldowski [mailto:M.Waldowski@sulechow.net]
> > > > Sent: Wednesday, April 25, 2007 1:36 PM
> > > > To: dev@mina.apache.org
> > > > Subject: Re: Permanent solution for OOM errors
> > > >
> > > > Hej Gaston,
> > > >
> > > > Gaston Dombiak wrote:
> > > > > Hey Marcin,
> > > > >
> > > > > I'm curious to hear from you why do you think so? I absolutely
> agree
> > > > > that MINA has to provide easy ways to handle OOM problems due to
> > > heavy
> > > > > incoming or outgoing traffic.
> > > > >
> > > > > This is by far the more common problem people are reporting with
> > > > > Openfire now that we moved to MINA (when under heavy load).
> > > > >
> > > >
> > > > Hmm, my first thought was that it introduce the complexity of
> > > > configuration like ThreadModel. But probably I'm wrong.
> > > >
> > > > I agree definitely that MINA need solution for OOM, but it could be
> > > also
> > > >
> > > > ReadThrottleFilterBuilder and WriteThrottleFilter. They are easy way
> > > to
> > > > handle OOM and (what is important) users are aware that they use it.
> > > >
> > > > But I'm not agains integrate it as default to Acceptor/Connector,
> > > really
> > > >
> > > > :) Meybe I even souldn't post my doubts to this subject because I
> > > > haven't contributed to MINA so far...
> > > >
> > > > Regards, Marcin
> > > >
> > > >
> > > >
> > >
> >
>
>
> --
> ..Cheers
> Mark
>
Re: Permanent solution for OOM errors
Posted by Mark Webb <el...@gmail.com>.
have you looked into tweaking your JVM?
http://java.sun.com/performance/reference/whitepapers/tuning.html
On 4/28/07, mat <fo...@gmail.com> wrote:
> I also observed that the memory increaed way too fast when heavy loading.
>
> 2007/4/26, Gaston Dombiak <ga...@jivesoftware.com>:
> >
> > Hey Mark,
> >
> > Based on the statistics I've been collecting from MINA I can tell that
> > our OOM problems happen when:
> >
> > 1) the eventQueue instance variable inside of ExecutorFilter fills up
> > (that would be the case for incoming traffic) or
> > 2) the SocketSessionImpl#getScheduledWriteRequests() queue starts to
> > fill up (that would be the case for outgoing traffic)
> >
> > Regards,
> >
> > -- Gato
> >
> > -----Original Message-----
> > From: Mark Webb [mailto:elihusmails@gmail.com]
> > Sent: Wednesday, April 25, 2007 1:53 PM
> > To: dev@mina.apache.org
> > Subject: Re: Permanent solution for OOM errors
> >
> > Maybe one approach would be to categorize the ways in which people
> > reach OOM conditions. Do these conditions happen only when the input
> > and/or output queues get really large, or are there other reasons.
> > Once we can start to categorize the OOM conditions that users are
> > experiencing, we can properly move forward with solutions.
> >
> > Basically what I mean is, if the OOM happens when queues fill up, then
> > throttle filters would be best. If the OOM happen for other reasons,
> > people could look at other areas of MINA. Just not sure there is one
> > solution that will fix all the problems.
> >
> > ...just my 2 cents.
> >
> > --
> > ..Cheers
> > Mark
> >
> >
> > On 4/25/07, Gaston Dombiak <ga...@jivesoftware.com> wrote:
> > > Hey Marcin,
> > >
> > > I see what you mean. I'm really fine with either option as long as
> > MINA
> > > handles the OOM problem. As I posted a month or so ago the throttling
> > > per connection is not enough. We (Openfire and probably others) also
> > > need a global throttling.
> > >
> > > Thanks,
> > >
> > > -- Gato
> > >
> > > -----Original Message-----
> > > From: Marcin Waldowski [mailto:M.Waldowski@sulechow.net]
> > > Sent: Wednesday, April 25, 2007 1:36 PM
> > > To: dev@mina.apache.org
> > > Subject: Re: Permanent solution for OOM errors
> > >
> > > Hej Gaston,
> > >
> > > Gaston Dombiak wrote:
> > > > Hey Marcin,
> > > >
> > > > I'm curious to hear from you why do you think so? I absolutely agree
> > > > that MINA has to provide easy ways to handle OOM problems due to
> > heavy
> > > > incoming or outgoing traffic.
> > > >
> > > > This is by far the more common problem people are reporting with
> > > > Openfire now that we moved to MINA (when under heavy load).
> > > >
> > >
> > > Hmm, my first thought was that it introduce the complexity of
> > > configuration like ThreadModel. But probably I'm wrong.
> > >
> > > I agree definitely that MINA need solution for OOM, but it could be
> > also
> > >
> > > ReadThrottleFilterBuilder and WriteThrottleFilter. They are easy way
> > to
> > > handle OOM and (what is important) users are aware that they use it.
> > >
> > > But I'm not agains integrate it as default to Acceptor/Connector,
> > really
> > >
> > > :) Meybe I even souldn't post my doubts to this subject because I
> > > haven't contributed to MINA so far...
> > >
> > > Regards, Marcin
> > >
> > >
> > >
> >
>
--
..Cheers
Mark
Re: Permanent solution for OOM errors
Posted by mat <fo...@gmail.com>.
I also observed that the memory increaed way too fast when heavy loading.
2007/4/26, Gaston Dombiak <ga...@jivesoftware.com>:
>
> Hey Mark,
>
> Based on the statistics I've been collecting from MINA I can tell that
> our OOM problems happen when:
>
> 1) the eventQueue instance variable inside of ExecutorFilter fills up
> (that would be the case for incoming traffic) or
> 2) the SocketSessionImpl#getScheduledWriteRequests() queue starts to
> fill up (that would be the case for outgoing traffic)
>
> Regards,
>
> -- Gato
>
> -----Original Message-----
> From: Mark Webb [mailto:elihusmails@gmail.com]
> Sent: Wednesday, April 25, 2007 1:53 PM
> To: dev@mina.apache.org
> Subject: Re: Permanent solution for OOM errors
>
> Maybe one approach would be to categorize the ways in which people
> reach OOM conditions. Do these conditions happen only when the input
> and/or output queues get really large, or are there other reasons.
> Once we can start to categorize the OOM conditions that users are
> experiencing, we can properly move forward with solutions.
>
> Basically what I mean is, if the OOM happens when queues fill up, then
> throttle filters would be best. If the OOM happen for other reasons,
> people could look at other areas of MINA. Just not sure there is one
> solution that will fix all the problems.
>
> ...just my 2 cents.
>
> --
> ..Cheers
> Mark
>
>
> On 4/25/07, Gaston Dombiak <ga...@jivesoftware.com> wrote:
> > Hey Marcin,
> >
> > I see what you mean. I'm really fine with either option as long as
> MINA
> > handles the OOM problem. As I posted a month or so ago the throttling
> > per connection is not enough. We (Openfire and probably others) also
> > need a global throttling.
> >
> > Thanks,
> >
> > -- Gato
> >
> > -----Original Message-----
> > From: Marcin Waldowski [mailto:M.Waldowski@sulechow.net]
> > Sent: Wednesday, April 25, 2007 1:36 PM
> > To: dev@mina.apache.org
> > Subject: Re: Permanent solution for OOM errors
> >
> > Hej Gaston,
> >
> > Gaston Dombiak wrote:
> > > Hey Marcin,
> > >
> > > I'm curious to hear from you why do you think so? I absolutely agree
> > > that MINA has to provide easy ways to handle OOM problems due to
> heavy
> > > incoming or outgoing traffic.
> > >
> > > This is by far the more common problem people are reporting with
> > > Openfire now that we moved to MINA (when under heavy load).
> > >
> >
> > Hmm, my first thought was that it introduce the complexity of
> > configuration like ThreadModel. But probably I'm wrong.
> >
> > I agree definitely that MINA need solution for OOM, but it could be
> also
> >
> > ReadThrottleFilterBuilder and WriteThrottleFilter. They are easy way
> to
> > handle OOM and (what is important) users are aware that they use it.
> >
> > But I'm not agains integrate it as default to Acceptor/Connector,
> really
> >
> > :) Meybe I even souldn't post my doubts to this subject because I
> > haven't contributed to MINA so far...
> >
> > Regards, Marcin
> >
> >
> >
>
RE: Permanent solution for OOM errors
Posted by Gaston Dombiak <ga...@jivesoftware.com>.
Hey Mark,
Based on the statistics I've been collecting from MINA I can tell that
our OOM problems happen when:
1) the eventQueue instance variable inside of ExecutorFilter fills up
(that would be the case for incoming traffic) or
2) the SocketSessionImpl#getScheduledWriteRequests() queue starts to
fill up (that would be the case for outgoing traffic)
Regards,
-- Gato
-----Original Message-----
From: Mark Webb [mailto:elihusmails@gmail.com]
Sent: Wednesday, April 25, 2007 1:53 PM
To: dev@mina.apache.org
Subject: Re: Permanent solution for OOM errors
Maybe one approach would be to categorize the ways in which people
reach OOM conditions. Do these conditions happen only when the input
and/or output queues get really large, or are there other reasons.
Once we can start to categorize the OOM conditions that users are
experiencing, we can properly move forward with solutions.
Basically what I mean is, if the OOM happens when queues fill up, then
throttle filters would be best. If the OOM happen for other reasons,
people could look at other areas of MINA. Just not sure there is one
solution that will fix all the problems.
...just my 2 cents.
--
..Cheers
Mark
On 4/25/07, Gaston Dombiak <ga...@jivesoftware.com> wrote:
> Hey Marcin,
>
> I see what you mean. I'm really fine with either option as long as
MINA
> handles the OOM problem. As I posted a month or so ago the throttling
> per connection is not enough. We (Openfire and probably others) also
> need a global throttling.
>
> Thanks,
>
> -- Gato
>
> -----Original Message-----
> From: Marcin Waldowski [mailto:M.Waldowski@sulechow.net]
> Sent: Wednesday, April 25, 2007 1:36 PM
> To: dev@mina.apache.org
> Subject: Re: Permanent solution for OOM errors
>
> Hej Gaston,
>
> Gaston Dombiak wrote:
> > Hey Marcin,
> >
> > I'm curious to hear from you why do you think so? I absolutely agree
> > that MINA has to provide easy ways to handle OOM problems due to
heavy
> > incoming or outgoing traffic.
> >
> > This is by far the more common problem people are reporting with
> > Openfire now that we moved to MINA (when under heavy load).
> >
>
> Hmm, my first thought was that it introduce the complexity of
> configuration like ThreadModel. But probably I'm wrong.
>
> I agree definitely that MINA need solution for OOM, but it could be
also
>
> ReadThrottleFilterBuilder and WriteThrottleFilter. They are easy way
to
> handle OOM and (what is important) users are aware that they use it.
>
> But I'm not agains integrate it as default to Acceptor/Connector,
really
>
> :) Meybe I even souldn't post my doubts to this subject because I
> haven't contributed to MINA so far...
>
> Regards, Marcin
>
>
>
Re: Permanent solution for OOM errors
Posted by Mark Webb <el...@gmail.com>.
Maybe one approach would be to categorize the ways in which people
reach OOM conditions. Do these conditions happen only when the input
and/or output queues get really large, or are there other reasons.
Once we can start to categorize the OOM conditions that users are
experiencing, we can properly move forward with solutions.
Basically what I mean is, if the OOM happens when queues fill up, then
throttle filters would be best. If the OOM happen for other reasons,
people could look at other areas of MINA. Just not sure there is one
solution that will fix all the problems.
...just my 2 cents.
--
..Cheers
Mark
On 4/25/07, Gaston Dombiak <ga...@jivesoftware.com> wrote:
> Hey Marcin,
>
> I see what you mean. I'm really fine with either option as long as MINA
> handles the OOM problem. As I posted a month or so ago the throttling
> per connection is not enough. We (Openfire and probably others) also
> need a global throttling.
>
> Thanks,
>
> -- Gato
>
> -----Original Message-----
> From: Marcin Waldowski [mailto:M.Waldowski@sulechow.net]
> Sent: Wednesday, April 25, 2007 1:36 PM
> To: dev@mina.apache.org
> Subject: Re: Permanent solution for OOM errors
>
> Hej Gaston,
>
> Gaston Dombiak wrote:
> > Hey Marcin,
> >
> > I'm curious to hear from you why do you think so? I absolutely agree
> > that MINA has to provide easy ways to handle OOM problems due to heavy
> > incoming or outgoing traffic.
> >
> > This is by far the more common problem people are reporting with
> > Openfire now that we moved to MINA (when under heavy load).
> >
>
> Hmm, my first thought was that it introduce the complexity of
> configuration like ThreadModel. But probably I'm wrong.
>
> I agree definitely that MINA need solution for OOM, but it could be also
>
> ReadThrottleFilterBuilder and WriteThrottleFilter. They are easy way to
> handle OOM and (what is important) users are aware that they use it.
>
> But I'm not agains integrate it as default to Acceptor/Connector, really
>
> :) Meybe I even souldn't post my doubts to this subject because I
> haven't contributed to MINA so far...
>
> Regards, Marcin
>
>
>
RE: Permanent solution for OOM errors
Posted by Gaston Dombiak <ga...@jivesoftware.com>.
Hey Marcin,
I see what you mean. I'm really fine with either option as long as MINA
handles the OOM problem. As I posted a month or so ago the throttling
per connection is not enough. We (Openfire and probably others) also
need a global throttling.
Thanks,
-- Gato
-----Original Message-----
From: Marcin Waldowski [mailto:M.Waldowski@sulechow.net]
Sent: Wednesday, April 25, 2007 1:36 PM
To: dev@mina.apache.org
Subject: Re: Permanent solution for OOM errors
Hej Gaston,
Gaston Dombiak wrote:
> Hey Marcin,
>
> I'm curious to hear from you why do you think so? I absolutely agree
> that MINA has to provide easy ways to handle OOM problems due to heavy
> incoming or outgoing traffic.
>
> This is by far the more common problem people are reporting with
> Openfire now that we moved to MINA (when under heavy load).
>
Hmm, my first thought was that it introduce the complexity of
configuration like ThreadModel. But probably I'm wrong.
I agree definitely that MINA need solution for OOM, but it could be also
ReadThrottleFilterBuilder and WriteThrottleFilter. They are easy way to
handle OOM and (what is important) users are aware that they use it.
But I'm not agains integrate it as default to Acceptor/Connector, really
:) Meybe I even souldn't post my doubts to this subject because I
haven't contributed to MINA so far...
Regards, Marcin
Re: Permanent solution for OOM errors
Posted by Marcin Waldowski <M....@sulechow.net>.
Hej Gaston,
Gaston Dombiak wrote:
> Hey Marcin,
>
> I'm curious to hear from you why do you think so? I absolutely agree
> that MINA has to provide easy ways to handle OOM problems due to heavy
> incoming or outgoing traffic.
>
> This is by far the more common problem people are reporting with
> Openfire now that we moved to MINA (when under heavy load).
>
Hmm, my first thought was that it introduce the complexity of
configuration like ThreadModel. But probably I'm wrong.
I agree definitely that MINA need solution for OOM, but it could be also
ReadThrottleFilterBuilder and WriteThrottleFilter. They are easy way to
handle OOM and (what is important) users are aware that they use it.
But I'm not agains integrate it as default to Acceptor/Connector, really
:) Meybe I even souldn't post my doubts to this subject because I
haven't contributed to MINA so far...
Regards, Marcin
RE: Permanent solution for OOM errors
Posted by Gaston Dombiak <ga...@jivesoftware.com>.
Hey Marcin,
I'm curious to hear from you why do you think so? I absolutely agree
that MINA has to provide easy ways to handle OOM problems due to heavy
incoming or outgoing traffic.
This is by far the more common problem people are reporting with
Openfire now that we moved to MINA (when under heavy load).
Regards,
-- Gato
-----Original Message-----
From: Marcin Waldowski [mailto:M.Waldowski@sulechow.net]
Sent: Wednesday, April 25, 2007 12:58 PM
To: dev@mina.apache.org
Subject: Re: Permanent solution for OOM errors
Hi,
Trustin Lee wrote:
> Hi Vinod,
>
> On 4/25/07, Vinod Panicker <vi...@gmail.com> wrote:
>> Hi,
>>
>> I think we've already got a lot of issues on the list regarding OOM
>> errors. In reality, every server needs to be able to manage I/O
>> efficiently to avoid high memory utilization, which ultimately
results
>> in an OOM.
>>
>> What can be done? MINA Connectors and Acceptors need to have the
Read
>> and Write throttle filters by default. Once the specified buffer
size
>> per session / overall has been exhausted, MINA needs to shift into
>> blocking mode to avoid an OOM on the server.
>
> Very true. I agree with you absolutely. I think we need to
> incorporate this feature into the Acceptor / Connector
> implementations. We could add some hook methods to
> AbstractIoFilterChain, but I didn't try it yet. (probably after honey
> moon ;)
>
I think that this is great idea. But I have second thoughts. Are we sure
that we are not thinking about something which we will regret like
ThreadModel?
Regards, Marcin
Re: Permanent solution for OOM errors
Posted by Marcin Waldowski <M....@sulechow.net>.
Hi,
Trustin Lee wrote:
> Hi Vinod,
>
> On 4/25/07, Vinod Panicker <vi...@gmail.com> wrote:
>> Hi,
>>
>> I think we've already got a lot of issues on the list regarding OOM
>> errors. In reality, every server needs to be able to manage I/O
>> efficiently to avoid high memory utilization, which ultimately results
>> in an OOM.
>>
>> What can be done? MINA Connectors and Acceptors need to have the Read
>> and Write throttle filters by default. Once the specified buffer size
>> per session / overall has been exhausted, MINA needs to shift into
>> blocking mode to avoid an OOM on the server.
>
> Very true. I agree with you absolutely. I think we need to
> incorporate this feature into the Acceptor / Connector
> implementations. We could add some hook methods to
> AbstractIoFilterChain, but I didn't try it yet. (probably after honey
> moon ;)
>
I think that this is great idea. But I have second thoughts. Are we sure
that we are not thinking about something which we will regret like
ThreadModel?
Regards, Marcin
Re: Permanent solution for OOM errors
Posted by Trustin Lee <tr...@gmail.com>.
Hi Vinod,
On 4/25/07, Vinod Panicker <vi...@gmail.com> wrote:
> Hi,
>
> I think we've already got a lot of issues on the list regarding OOM
> errors. In reality, every server needs to be able to manage I/O
> efficiently to avoid high memory utilization, which ultimately results
> in an OOM.
>
> What can be done? MINA Connectors and Acceptors need to have the Read
> and Write throttle filters by default. Once the specified buffer size
> per session / overall has been exhausted, MINA needs to shift into
> blocking mode to avoid an OOM on the server.
Very true. I agree with you absolutely. I think we need to
incorporate this feature into the Acceptor / Connector
implementations. We could add some hook methods to
AbstractIoFilterChain, but I didn't try it yet. (probably after honey
moon ;)
> BTW, the WriteThrottleFilter contributed in DIRMINA-132 has some
> issues, which we have fixed and will be contributing soon.
Great!
Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/
--
PGP Key ID: 0x0255ECA6