You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mina.apache.org by Chris Chalmers <ch...@ecis.co.za> on 2007/07/02 11:42:53 UTC
Re: Blocking using Mina
Hi Peter
I'm now getting something strange:
- my read is throttling fine based on the number of outstanding writes
on a linked session
- the session.resumeRead() method is being called when the linked writes
have dropped
- however, the read is not resumed.
I have limited memory usage to 64Mb, and at the point of session
resumption, 63.8Mb has been read in.
When using 256Mb, the process completes; at the point of resume, it
seems like the allocated memory doubles - I am hazarding a guess that
this could be causing the problem when using 64Mb. However, I don't get
an out-of-memory exception.
Any ideas?
Thanks,
Chris
peter royal wrote:
> On Jun 28, 2007, at 8:00 AM, Chris Chalmers wrote:
>> I have a Mina server that simply pumps data through (based on the
>> Mina Proxy example) -
>> however, when sending large files (500Mb+), the data is read in
>> quickly, but written slowly (the target uses a blocking Socket read);
>> this causes the server's memory usage to increase very quickly.
>> I have tried to use the ReadThrottleFilterBuilder in combination with
>> the WriteBufferLimitFilterBuilder (from
>> http://issues.apache.org/jira/browse/DIRMINA-302), but the best I can
>> get is 226Mb memory usage when setting the read/write
>> setMaximumConnectionBufferSize to 50000.
>>
>> Is there any way that I can force Mina sessions to block incoming
>> messages instead? The source code explicitly sets the SocketChannels
>> to non-blocking and I can't see any way to change this using the API.
>> Alternatively is there a better method that I am overlooking?
>
> IoSession.suspendRead / resumeRead ... you can use that to keep MINA
> from reading the data, just turn it on/off in concert with your writing.
>
> -pete
>
>
Re: Blocking using Mina
Posted by Chris Chalmers <ch...@ecis.co.za>.
Thanks Trustin - I'll try your suggestion and post some code if I don't
have any joy.
Trustin Lee wrote:
> On 7/4/07, Chris Chalmers <ch...@ecis.co.za> wrote:
>> Hi all
>>
>> Checking this again, it seems that an OOM is being thrown after all. The
>> IoFilter throttling doesn't seem to kick in until too late,
>> unfortunately.
>>
>> Looking at the SocketIoProcessor class, if I apply the throttle before
>> the read call, this achieves the desired effect for me: (ie: limits
>> memory usage significantly):
>>
>> private void process( Set<SelectionKey> selectedKeys )
>> {
>> ...
>> if (!throttle(session)) {
>> read( session );
>> }
>> ...
>> }
>>
>> Unfortunately this solution is not clean as I am forced to modify the
>> Mina source code and not use the API.
>> Any better ideas anyone?
>>
>> Chris Chalmers wrote:
>> > Hi Peter
>> >
>> > I'm now getting something strange:
>> > - my read is throttling fine based on the number of outstanding writes
>> > on a linked session
>> > - the session.resumeRead() method is being called when the linked
>> > writes have dropped
>> > - however, the read is not resumed.
>> >
>> > I have limited memory usage to 64Mb, and at the point of session
>> > resumption, 63.8Mb has been read in.
>> > When using 256Mb, the process completes; at the point of resume, it
>> > seems like the allocated memory doubles - I am hazarding a guess that
>> > this could be causing the problem when using 64Mb. However, I don't
>> > get an out-of-memory exception.
>> >
>> > Any ideas?
>
> Did you try to add ReadThrottleFilter and WriteBufferLimitFilter
> *before* ExecutorFilter? I'm not sure this will work, but placing the
> filters before the ExecutorFilter will make the traffic control more
> immediate.
>
> It would be the best if you can provide us some test code that
> reproduces the problem, because it is the easiest way to fix our side.
>
> HTH,
> Trustin
Re: Blocking using Mina
Posted by 向秦贤 <fy...@gmail.com>.
Hi All,
My you guys think grizzly blocking read sytle can be not bad solution.
It do thing with a addition SelectionKey.
http://weblogs.java.net/blog/jfarcand/archive/2006/05/tricks_and_tips_1.html
http://weblogs.java.net/blog/jfarcand/archive/2006/06/tricks_and_tips.html
code framework like this, and it should be usage in mina, or communicate
with mina framework objects.
maybe like grizzly, IoSession could add blockingWrite or blockingRead
methods.
try{
SocketChannel socketChannel = (SocketChannel)key.channel();
while (count > 0){
count = socketChannel.read(byteBuffer);
}
if ( byteRead == 0 ){
readSelector = SelectorFactory.getSelector();
tmpKey = socketChannel
.register(readSelector,SelectionKey.OP_READ);
tmpKey.interestOps(tmpKey.interestOps() |
SelectionKey.OP_READ);
int code = readSelector.select(readTimeout);
tmpKey.interestOps(
tmpKey.interestOps() & (~SelectionKey.OP_READ));
if ( code == 0 ){
return 0;
}
while (count > 0){
count = socketChannel.read(byteBuffer);
}
}
} catch (Throwable t){
....
Regards,
2007/7/5, Trustin Lee <tr...@gmail.com>:
>
> On 7/4/07, Chris Chalmers <ch...@ecis.co.za> wrote:
> > Hi all
> >
> > Checking this again, it seems that an OOM is being thrown after all. The
> > IoFilter throttling doesn't seem to kick in until too late,
> unfortunately.
> >
> > Looking at the SocketIoProcessor class, if I apply the throttle before
> > the read call, this achieves the desired effect for me: (ie: limits
> > memory usage significantly):
> >
> > private void process( Set<SelectionKey> selectedKeys )
> > {
> > ...
> > if (!throttle(session)) {
> > read( session );
> > }
> > ...
> > }
> >
> > Unfortunately this solution is not clean as I am forced to modify the
> > Mina source code and not use the API.
> > Any better ideas anyone?
> >
> > Chris Chalmers wrote:
> > > Hi Peter
> > >
> > > I'm now getting something strange:
> > > - my read is throttling fine based on the number of outstanding writes
> > > on a linked session
> > > - the session.resumeRead() method is being called when the linked
> > > writes have dropped
> > > - however, the read is not resumed.
> > >
> > > I have limited memory usage to 64Mb, and at the point of session
> > > resumption, 63.8Mb has been read in.
> > > When using 256Mb, the process completes; at the point of resume, it
> > > seems like the allocated memory doubles - I am hazarding a guess that
> > > this could be causing the problem when using 64Mb. However, I don't
> > > get an out-of-memory exception.
> > >
> > > Any ideas?
>
> Did you try to add ReadThrottleFilter and WriteBufferLimitFilter
> *before* ExecutorFilter? I'm not sure this will work, but placing the
> filters before the ExecutorFilter will make the traffic control more
> immediate.
>
> It would be the best if you can provide us some test code that
> reproduces the problem, because it is the easiest way to fix our side.
>
> HTH,
> Trustin
> --
> what we call human nature is actually human habit
> --
> http://gleamynode.net/
> --
> PGP Key ID: 0x0255ECA6
>
--
向秦贤
Re: Blocking using Mina
Posted by Trustin Lee <tr...@gmail.com>.
On 7/4/07, Chris Chalmers <ch...@ecis.co.za> wrote:
> Hi all
>
> Checking this again, it seems that an OOM is being thrown after all. The
> IoFilter throttling doesn't seem to kick in until too late, unfortunately.
>
> Looking at the SocketIoProcessor class, if I apply the throttle before
> the read call, this achieves the desired effect for me: (ie: limits
> memory usage significantly):
>
> private void process( Set<SelectionKey> selectedKeys )
> {
> ...
> if (!throttle(session)) {
> read( session );
> }
> ...
> }
>
> Unfortunately this solution is not clean as I am forced to modify the
> Mina source code and not use the API.
> Any better ideas anyone?
>
> Chris Chalmers wrote:
> > Hi Peter
> >
> > I'm now getting something strange:
> > - my read is throttling fine based on the number of outstanding writes
> > on a linked session
> > - the session.resumeRead() method is being called when the linked
> > writes have dropped
> > - however, the read is not resumed.
> >
> > I have limited memory usage to 64Mb, and at the point of session
> > resumption, 63.8Mb has been read in.
> > When using 256Mb, the process completes; at the point of resume, it
> > seems like the allocated memory doubles - I am hazarding a guess that
> > this could be causing the problem when using 64Mb. However, I don't
> > get an out-of-memory exception.
> >
> > Any ideas?
Did you try to add ReadThrottleFilter and WriteBufferLimitFilter
*before* ExecutorFilter? I'm not sure this will work, but placing the
filters before the ExecutorFilter will make the traffic control more
immediate.
It would be the best if you can provide us some test code that
reproduces the problem, because it is the easiest way to fix our side.
HTH,
Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/
--
PGP Key ID: 0x0255ECA6
Re: Blocking using Mina
Posted by Chris Chalmers <ch...@ecis.co.za>.
Hi all
Checking this again, it seems that an OOM is being thrown after all. The
IoFilter throttling doesn't seem to kick in until too late, unfortunately.
Looking at the SocketIoProcessor class, if I apply the throttle before
the read call, this achieves the desired effect for me: (ie: limits
memory usage significantly):
private void process( Set<SelectionKey> selectedKeys )
{
...
if (!throttle(session)) {
read( session );
}
...
}
Unfortunately this solution is not clean as I am forced to modify the
Mina source code and not use the API.
Any better ideas anyone?
Chris Chalmers wrote:
> Hi Peter
>
> I'm now getting something strange:
> - my read is throttling fine based on the number of outstanding writes
> on a linked session
> - the session.resumeRead() method is being called when the linked
> writes have dropped
> - however, the read is not resumed.
>
> I have limited memory usage to 64Mb, and at the point of session
> resumption, 63.8Mb has been read in.
> When using 256Mb, the process completes; at the point of resume, it
> seems like the allocated memory doubles - I am hazarding a guess that
> this could be causing the problem when using 64Mb. However, I don't
> get an out-of-memory exception.
>
> Any ideas?
>
> Thanks,
> Chris
>
> peter royal wrote:
>> On Jun 28, 2007, at 8:00 AM, Chris Chalmers wrote:
>>> I have a Mina server that simply pumps data through (based on the
>>> Mina Proxy example) -
>>> however, when sending large files (500Mb+), the data is read in
>>> quickly, but written slowly (the target uses a blocking Socket
>>> read); this causes the server's memory usage to increase very quickly.
>>> I have tried to use the ReadThrottleFilterBuilder in combination
>>> with the WriteBufferLimitFilterBuilder (from
>>> http://issues.apache.org/jira/browse/DIRMINA-302), but the best I
>>> can get is 226Mb memory usage when setting the read/write
>>> setMaximumConnectionBufferSize to 50000.
>>>
>>> Is there any way that I can force Mina sessions to block incoming
>>> messages instead? The source code explicitly sets the SocketChannels
>>> to non-blocking and I can't see any way to change this using the API.
>>> Alternatively is there a better method that I am overlooking?
>>
>> IoSession.suspendRead / resumeRead ... you can use that to keep MINA
>> from reading the data, just turn it on/off in concert with your writing.
>>
>> -pete
>>
>>
>
>
>