You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mina.apache.org by Michael Bauroth <mi...@falcom.de> on 2006/04/04 13:02:28 UTC
How can I avoid overblasting a mina server
Hi,
I've just implemented a simple client/server pair. Next I've tried to
run some data transfer (consuming)) performance tests (all happened in 2
JVM's on a single shared server).
The problem is that from a specific load on the data come quicker into
the server then it could handle them. When I'm right, the extra data
will be stored in the incoming pipeline, right? What happens, if this
pipeline is full? And how can I prevent from such situations?
Any suggestions will be appreciated. Thanx.
Michael
Re: How can I avoid overblasting a mina server
Posted by Trustin Lee <tr...@gmail.com>.
On 4/4/06, Trustin Lee <tr...@gmail.com> wrote:
>
> You can control the I/O traffic by calling IoSession.suspend/resumeRead/Write()
> method, so it should be done quite easily.
>
There's also IoSession.getScheduledWriteRequests() and
IoSession.getScheduledWriteBytes() which will be useful for the opposite
case when a server writes too fast.
Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/
--
PGP key fingerprints:
* E167 E6AF E73A CBCE EE41 4A29 544D DE48 FE95 4E7E
* B693 628E 6047 4F8F CFA4 455E 1C62 A7DC 0255 ECA6
Re: How can I avoid overblasting a mina server
Posted by Trustin Lee <tr...@gmail.com>.
On 4/5/06, peter royal <pr...@apache.org> wrote:
>
> On Apr 4, 2006, at 11:18 AM, Trustin Lee wrote:
> >> Write would become synchronous when under load.. The write queue
> >> would be "bounded", attempts to create the future would block when
> >> under load. How to assembly the bits is tbd, but we have all the
> >> necessary primitives to build the behavior.
> >
> >
> > What if a session is running in a single thread model? We cannot
> > block in
> > this case because it's the thread that performs I/O for other sesions.
>
> Good point, but it then may not be an option to do write throttling
> when running with that model...
Yes, we can make it optional then it's fine.
I thought about reads. I think it is not quite easy to implement once
ThreadPoolFilter comes into play. All the counting should take place after
all thread pool filters and before any codecs, and this constraint is
sometimes not applicable. The solution would be:
* To maintain a stateful buffer per session so we can track the read buffer
size easily, but it's not viable because of a memory consumption issue.
* To let users decrease the values by themselves, and it's not reasonable
either.
* To rely on observation on ByteBuffer.release(). There's a risk here that
an IoFilter implementor might forget to release the buffer or doesn't
release intentionally for some implementation issues (e.g. reused buffers).
Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/
--
PGP key fingerprints:
* E167 E6AF E73A CBCE EE41 4A29 544D DE48 FE95 4E7E
* B693 628E 6047 4F8F CFA4 455E 1C62 A7DC 0255 ECA6
Re: How can I avoid overblasting a mina server
Posted by peter royal <pr...@apache.org>.
On Apr 4, 2006, at 3:32 PM, Michael Bauroth wrote:
> peter royal wrote:
>
>> On Apr 4, 2006, at 11:18 AM, Trustin Lee wrote:
>>>> Write would become synchronous when under load.. The write queue
>>>> would be "bounded", attempts to create the future would block when
>>>> under load. How to assembly the bits is tbd, but we have all the
>>>> necessary primitives to build the behavior.
>>>
>>>
>>>
>>> What if a session is running in a single thread model? We
>>> cannot block in
>>> this case because it's the thread that performs I/O for other
>>> sesions.
>> Good point, but it then may not be an option to do write
>> throttling when running with that model...
>> -pete
>
> Just a question: Would it be a solution, when turning on write
> throttling ever turns on a separate internal thread?
I can't think of a scenario where write throttling would cause MINA
to spawn another thread.
-pete
--
proyal@apache.org - http://fotap.org/~osi
Re: How can I avoid overblasting a mina server
Posted by Michael Bauroth <Mi...@falcom.de>.
peter royal wrote:
> On Apr 4, 2006, at 11:18 AM, Trustin Lee wrote:
>
>>> Write would become synchronous when under load.. The write queue
>>> would be "bounded", attempts to create the future would block when
>>> under load. How to assembly the bits is tbd, but we have all the
>>> necessary primitives to build the behavior.
>>
>>
>>
>> What if a session is running in a single thread model? We cannot
>> block in
>> this case because it's the thread that performs I/O for other sesions.
>
>
> Good point, but it then may not be an option to do write throttling
> when running with that model...
> -pete
>
Just a question: Would it be a solution, when turning on write
throttling ever turns on a separate internal thread?
Michael
Re: How can I avoid overblasting a mina server
Posted by peter royal <pr...@apache.org>.
On Apr 4, 2006, at 11:18 AM, Trustin Lee wrote:
>> Write would become synchronous when under load.. The write queue
>> would be "bounded", attempts to create the future would block when
>> under load. How to assembly the bits is tbd, but we have all the
>> necessary primitives to build the behavior.
>
>
> What if a session is running in a single thread model? We cannot
> block in
> this case because it's the thread that performs I/O for other sesions.
Good point, but it then may not be an option to do write throttling
when running with that model...
-pete
--
proyal@apache.org - http://fotap.org/~osi
Re: How can I avoid overblasting a mina server
Posted by Trustin Lee <tr...@gmail.com>.
On 4/5/06, peter royal <pr...@apache.org> wrote:
>
> On Apr 4, 2006, at 11:11 AM, Trustin Lee wrote:
> > But I don't have any clue on writes. Because writes are
> > asynchronous, We
> > cannot block simply. Any idea?
>
> Write would become synchronous when under load.. The write queue
> would be "bounded", attempts to create the future would block when
> under load. How to assembly the bits is tbd, but we have all the
> necessary primitives to build the behavior.
What if a session is running in a single thread model? We cannot block in
this case because it's the thread that performs I/O for other sesions.
Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/
--
PGP key fingerprints:
* E167 E6AF E73A CBCE EE41 4A29 544D DE48 FE95 4E7E
* B693 628E 6047 4F8F CFA4 455E 1C62 A7DC 0255 ECA6
Re: How can I avoid overblasting a mina server
Posted by peter royal <pr...@apache.org>.
On Apr 4, 2006, at 11:11 AM, Trustin Lee wrote:
> But I don't have any clue on writes. Because writes are
> asynchronous, We
> cannot block simply. Any idea?
Write would become synchronous when under load.. The write queue
would be "bounded", attempts to create the future would block when
under load. How to assembly the bits is tbd, but we have all the
necessary primitives to build the behavior.
-pete
--
proyal@apache.org - http://fotap.org/~osi
Re: How can I avoid overblasting a mina server
Posted by Trustin Lee <tr...@gmail.com>.
On 4/4/06, Vinod Panicker <vi...@gmail.com> wrote:
>
> I think there should be a param that allows the read buffer size to be
> configured. An OutOfMemoryError is ghastly! MINA shouldnt read()
> from the session if the buffer is already full. This will cause the
> kernel buffer to get filled up, causing the flow control mechanism to
> kick in. This will cause the sender to send less data, just enough
> that the receiver can consume.
You're right, too. We need to modify the core code in this case to keep
track of the virtual read buffer size, which is actually an acumulated byte
count. The count value will increase when data is read from a channel and
the buffer is conveyed into the filter chain and then decrease when the
buffer is released or passed to messageReceived handler.
But I don't have any clue on writes. Because writes are asynchronous, We
cannot block simply. Any idea?
Thanks,
Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/
--
PGP key fingerprints:
* E167 E6AF E73A CBCE EE41 4A29 544D DE48 FE95 4E7E
* B693 628E 6047 4F8F CFA4 455E 1C62 A7DC 0255 ECA6
Re: How can I avoid overblasting a mina server
Posted by Vinod Panicker <vi...@gmail.com>.
On 4/4/06, Trustin Lee <tr...@gmail.com> wrote:
> On 4/4/06, Michael Bauroth <mi...@falcom.de> wrote:
> >
> > Hi,
> >
> > I've just implemented a simple client/server pair. Next I've tried to
> > run some data transfer (consuming)) performance tests (all happened in 2
> > JVM's on a single shared server).
> > The problem is that from a specific load on the data come quicker into
> > the server then it could handle them. When I'm right, the extra data
> > will be stored in the incoming pipeline, right? What happens, if this
> > pipeline is full? And how can I prevent from such situations?
>
>
> You're right. The server will get broken throwing an OutOfMemoryError. We
> have to implement an IoFilter which prevents this situation. You can
> control the I/O traffic by calling IoSession.suspend/resumeRead/Write()
> method, so it should be done quite easily.
I think there should be a param that allows the read buffer size to be
configured. An OutOfMemoryError is ghastly! MINA shouldnt read()
from the session if the buffer is already full. This will cause the
kernel buffer to get filled up, causing the flow control mechanism to
kick in. This will cause the sender to send less data, just enough
that the receiver can consume.
Regards,
Vinod.
Re: How can I avoid overblasting a mina server
Posted by Trustin Lee <tr...@gmail.com>.
On 4/4/06, Steve Bate <st...@technoetic.com> wrote:
>
> > From: Trustin Lee [mailto:trustin@gmail.com]
> > You're right. The server will get broken throwing an
> > OutOfMemoryError. We have to implement an IoFilter which
> > prevents this situation. You can control the I/O traffic by
> > calling IoSession.suspend/resumeRead/Write()
> > method, so it should be done quite easily.
>
> Is there any equivalent to this capability in 0.8.2 (or
> a workaround)? We're approaching the 1.0 release of QuickFIX/J
> and it's currently using 0.8.2 since its listed as the newest
> stable version of MINA. However, I'm concerned that our users
> will start doing load/performance tests on the engine and
> experience out of memory errors. I have a prototype flow control
> filter for the writer side, but I hadn't considered that the
> same problem could occur in the reader process.
Unfortunately, traffic control is not supported by MINA 0.8, though you can
control the write traffic because IoSession.getScheduledWriteRequests() is
there.
Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/
--
PGP key fingerprints:
* E167 E6AF E73A CBCE EE41 4A29 544D DE48 FE95 4E7E
* B693 628E 6047 4F8F CFA4 455E 1C62 A7DC 0255 ECA6
RE: How can I avoid overblasting a mina server
Posted by Steve Bate <st...@technoetic.com>.
> From: Trustin Lee [mailto:trustin@gmail.com]
> You're right. The server will get broken throwing an
> OutOfMemoryError. We have to implement an IoFilter which
> prevents this situation. You can control the I/O traffic by
> calling IoSession.suspend/resumeRead/Write()
> method, so it should be done quite easily.
Is there any equivalent to this capability in 0.8.2 (or
a workaround)? We're approaching the 1.0 release of QuickFIX/J
and it's currently using 0.8.2 since its listed as the newest
stable version of MINA. However, I'm concerned that our users
will start doing load/performance tests on the engine and
experience out of memory errors. I have a prototype flow control
filter for the writer side, but I hadn't considered that the
same problem could occur in the reader process.
Steve
Re: How can I avoid overblasting a mina server
Posted by Trustin Lee <tr...@gmail.com>.
On 4/4/06, Michael Bauroth <mi...@falcom.de> wrote:
>
> Hi,
>
> I've just implemented a simple client/server pair. Next I've tried to
> run some data transfer (consuming)) performance tests (all happened in 2
> JVM's on a single shared server).
> The problem is that from a specific load on the data come quicker into
> the server then it could handle them. When I'm right, the extra data
> will be stored in the incoming pipeline, right? What happens, if this
> pipeline is full? And how can I prevent from such situations?
You're right. The server will get broken throwing an OutOfMemoryError. We
have to implement an IoFilter which prevents this situation. You can
control the I/O traffic by calling IoSession.suspend/resumeRead/Write()
method, so it should be done quite easily.
HTH,
Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/
--
PGP key fingerprints:
* E167 E6AF E73A CBCE EE41 4A29 544D DE48 FE95 4E7E
* B693 628E 6047 4F8F CFA4 455E 1C62 A7DC 0255 ECA6