You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@mina.apache.org by Zvika Gart <zv...@gmail.com> on 2009/11/24 19:29:45 UTC

Miserable performance results from MINA 1.1.7

Hello,
Please help, as this is too strange to be true...

I'm comparing two implementations of the most basic socket server - it
listens for a client, and after accepting it just starts reading data from
it.
The first implementation using Java's plain old ServerSocket:

        ServerSocket srv = new ServerSocket(80);
    Socket s = srv.accept();
    InputStream i = s.getInputStream();
    byte[] buf = new byte[1024 * 8];
    while (true) {
     i.read(buf);
    }

The Second implementation using MINA 1.1.7 (which so simple that I can paste
it here):

First the Main class:

package test;

import java.net.InetSocketAddress;

import org.apache.mina.common.ByteBuffer;
import org.apache.mina.common.IoAcceptor;
import org.apache.mina.common.SimpleByteBufferAllocator;
import org.apache.mina.transport.socket.nio.SocketAcceptor;

public class Main {
    public static void main(String[] args) throws Exception {
        // The following two lines don't help
     ByteBuffer.setUseDirectBuffers(false);
        ByteBuffer.setAllocator(new SimpleByteBufferAllocator());

        IoAcceptor acceptor = new SocketAcceptor();
        acceptor.bind(new InetSocketAddress(80), new MyHandler());
    }
}


And the IoHandler:


package test;

import org.apache.mina.common.IoHandlerAdapter;

public class MyHandler extends IoHandlerAdapter {

}


I wrote a simple client that connects to the server and pumps continuous
data.
Both client and server run on Amazon's EC2 on Debian Linux instances
(64-bit, Java 1.6). Client is in Europe and server in USA.


*The results:*
Simple Java implementation - 5 Mbps
MINA implementation - 750 Kbps

Tried removing the setUseDirectBuffers lines; tried a simpler threading
model by running the IoHandler on the IoProcessor threads... nothing helps -
MINA is capped at 750 Kbps.
What is going on here?!

Re: Miserable performance results from MINA 1.1.7

Posted by Emmanuel Lecharny <el...@apache.org>.
Also have a look at this thread :

http://www.mail-archive.com/search?l=dev%40mina.apache.org&q=cloud

Not sure you can have an answer in it, but some people have already used 
MINA on EC2, and it seems to be ok.

-- 
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org



Re: Miserable performance results from MINA 1.1.7

Posted by Niklas Gustavsson <ni...@protocol7.com>.
On Fri, Nov 27, 2009 at 4:22 PM, Emmanuel Lecharny <el...@apache.org> wrote:
> On Fri, Nov 27, 2009 at 4:19 PM, Zvika Gart <zv...@gmail.com> wrote:
>> The way I used MINA is the most basic. It should be performant. The uploaded
>> data can be FTP data, HTTP Form data, etc. The current MINA code is capping
>> the upload bandwidth.
>
> Niklas ? How do you handle such a case on FtpServer ?

We do not currently use MINA for the data channel, only for the
control channel which only handles small messages (the FTP commands
and replies).

/niklas

Re: Miserable performance results from MINA 1.1.7

Posted by Emmanuel Lecharny <el...@apache.org>.
On Fri, Nov 27, 2009 at 4:19 PM, Zvika Gart <zv...@gmail.com> wrote:
>>
>> Keep in mind that the way you used MINA is really very specific. I
>
> don't have such performance simply because I never transfer such big
>
> messages. If you consider that messages that will be exchanged never
>
> exceed a few kilobytes, then performances are also excellent.
>
>
> The way I used MINA is the most basic. It should be performant. The uploaded
> data can be FTP data, HTTP Form data, etc. The current MINA code is capping
> the upload bandwidth.

Niklas ? How do you handle such a case on FtpServer ?


> You don't need to configure anything. The OS itself grows/shrinks the
> receive buffer & window size as needed. We saw that in the simple
> implementations. I didn't configure the receive buffer and it (and the
> receive window) grew to 2MB.

Ok, we have to check the code to see what's going on (ie, if there is
no reason to configure anything, and if we block at 64k, then that
mean we have a default config somewhere)

-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Miserable performance results from MINA 1.1.7

Posted by Zvika Gart <zv...@gmail.com>.
>
> Keep in mind that the way you used MINA is really very specific. I

don't have such performance simply because I never transfer such big

messages. If you consider that messages that will be exchanged never

exceed a few kilobytes, then performances are also excellent.


The way I used MINA is the most basic. It should be performant. The uploaded
data can be FTP data, HTTP Form data, etc. The current MINA code is capping
the upload bandwidth.
In a scenario where the data only flows from client to server, even with
small messages you got a problem - the message rate is capped since the
window size is capped.

AFAIK, this is a configurable option, and if we don't configure it

before connecting the socket, then the RCV buffer size will never grow

bigger than 64K. So what should we do ? Set the buffer size to 2Gb ?.


> Also consider that I'm not 100% sure of the consequecnes of such a

change. Maybe it's totally harmless.


You don't need to configure anything. The OS itself grows/shrinks the
receive buffer & window size as needed. We saw that in the simple
implementations. I didn't configure the receive buffer and it (and the
receive window) grew to 2MB.


On Fri, Nov 27, 2009 at 4:34 PM, Emmanuel Lecharny <el...@apache.org>wrote:

> > First, I really appreciate your work on the framework, it's very easy to
> > work with and has really cool features, but I have to say that it is bad
> > that the out-of-the-box performance of MINA is so poor.
>
> Keep in mind that the way you used MINA is really very specific. I
> don't have such performance simply because I never transfer such big
> messages. If you consider that messages that will be exchanged never
> exceed a few kilobytes, then performances are also excellent.
>
> This simple
> > performance comparison test should always be verified throughout MINA
> > releases.
> > Second, why mass around with the receive buffer anyway if the user didn't
> > explicitly wants to? let the OS manage it...
>
> AFAIK, this is a configurable option, and if we don't configure it
> before connecting the socket, then the RCV buffer size will never grow
> bigger than 64K. So what should we do ? Set the buffer size to 2Gb ?.
>
> Also consider that I'm not 100% sure of the consequecnes of such a
> change. Maybe it's totally harmless.
>
>
> > Can you please release a patched version of MINA 1.1.7 with a fix?
>
> If you can provide a patch for 2.0§RC1, then we can fore sure apply
> it. I'm afraid that 1.1.7 is a dead branch at this point... Anyway,
> feel free to send the patch, I don't know if someone has some time to
> release a 1.1.8
>
>
> --
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com
>

Re: Miserable performance results from MINA 1.1.7

Posted by Emmanuel Lecharny <el...@apache.org>.
> First, I really appreciate your work on the framework, it's very easy to
> work with and has really cool features, but I have to say that it is bad
> that the out-of-the-box performance of MINA is so poor.

Keep in mind that the way you used MINA is really very specific. I
don't have such performance simply because I never transfer such big
messages. If you consider that messages that will be exchanged never
exceed a few kilobytes, then performances are also excellent.

This simple
> performance comparison test should always be verified throughout MINA
> releases.
> Second, why mass around with the receive buffer anyway if the user didn't
> explicitly wants to? let the OS manage it...

AFAIK, this is a configurable option, and if we don't configure it
before connecting the socket, then the RCV buffer size will never grow
bigger than 64K. So what should we do ? Set the buffer size to 2Gb ?.

Also consider that I'm not 100% sure of the consequecnes of such a
change. Maybe it's totally harmless.


> Can you please release a patched version of MINA 1.1.7 with a fix?

If you can provide a patch for 2.0§RC1, then we can fore sure apply
it. I'm afraid that 1.1.7 is a dead branch at this point... Anyway,
feel free to send the patch, I don't know if someone has some time to
release a 1.1.8


-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Miserable performance results from MINA 1.1.7

Posted by Zvika Gart <zv...@gmail.com>.
Emmanuel, you're absolutely right!
I patched the MINA source code (1.1 branch) removing all calls to
Socket.setReceiveBufferSize() (just for the test), and voila - no more
bottlenecks. The MINA implementation reaches the same throughput as the
simple servers.

The strace log indeed shows that there is a call to setsockopt(...SO_RCVBUF...)
after the bind call.

First, I really appreciate your work on the framework, it's very easy to
work with and has really cool features, but I have to say that it is bad
that the out-of-the-box performance of MINA is so poor. This simple
performance comparison test should always be verified throughout MINA
releases.
Second, why mass around with the receive buffer anyway if the user didn't
explicitly wants to? let the OS manage it...

Can you please release a patched version of MINA 1.1.7 with a fix?
Thanks guys.


On Thu, Nov 26, 2009 at 8:10 PM, Emmanuel Lecharny <el...@apache.org>wrote:

> Might be related to http://issues.apache.org/jira/browse/DIRMINA-561
>
> Also read
> http://java.sun.com/j2se/1.5.0/docs/api/java/net/ServerSocket.html#setReceiveBufferSize(int):
>
> "It is possible to change the value subsequently, by calling
> |Socket.setReceiveBufferSize(int)| <
> http://java.sun.com/j2se/1.5.0/docs/api/java/net/Socket.html#setReceiveBufferSize%28int%29>.
> However, if the application wishes to allow a receive window larger than 64K
> bytes, as defined by RFC1323 then the proposed value must be set in the
> ServerSocket *before* it is bound to a local address. This implies, that the
> ServerSocket must be created with the no-argument constructor, then
> setReceiveBufferSize() must be called and lastly the ServerSocket is bound
> to an address by calling bind(). "
>
>
> Zvika Gart wrote:
>
>> And here are the logs
>>
>> On Thu, Nov 26, 2009 at 7:19 PM, Zvika Gart <zv...@gmail.com> wrote:
>>
>>
>>
>>> Attached are the strace logs with the trace=network for the MINA
>>> implementation and the simple NIO implementation. both ran for about 30
>>> seconds.
>>> As you can see the MINA implementation makes lots of getsockopt system
>>> calls to get the receive buffer size (for each receive?).
>>>
>>> In this link http://79.125.32.246/capture.tbz you can find tcp dumps of
>>> the two implementations.
>>> ACK patterns are affected by the capabilities of the receiving
>>> application
>>> - how fast it reads data from the socket. You can see that the window
>>> size
>>> in the MINA implementation doesn't grow and is stuck at about 64KB while
>>> in
>>> the simple NIO it gets to 2MB. This is probably because the MINA
>>> implementation is slow at reading from the socket.
>>>
>>> If you need more logs or want me to run further tests please let me know.
>>>
>>>
>>> On Thu, Nov 26, 2009 at 4:04 AM, Tuure Laurinolli <
>>>
>>>
>>>> tuure.laurinolli@indagon.com> wrote:
>>>>
>>>>
>>>>
>>>>> Zvika Gart wrote:
>>>>>
>>>>>
>>>>>
>>>>>> - Check if there are some differences in the traffic (e.g. TCP ACK
>>>>>>
>>>>>>
>>>>>>> patterns)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> You didn't say anything about this yet.
>>>>>
>>>>>
>>>>>  - Check syscall patterns of the MINA version and try to see if
>>>>> something
>>>>>
>>>>>
>>>>>> is
>>>>>>> taking unreasonably long time (pure NIO implementation might be
>>>>>>> helpful
>>>>>>> here)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>  - As for syscall patterns, I'll need your help as I am a .NET
>>>>> programmer
>>>>>
>>>>>
>>>>>> and
>>>>>> not familiar with Java/Linux profiling...
>>>>>>
>>>>>>
>>>>>>
>>>>> Have you ever done anything with the Sysinternals file monitoring
>>>>> tools?
>>>>> Strace is similar tool for Linux, except its output is textual. You
>>>>> should
>>>>> find and read some tutorial about it to get some idea of how to use it.
>>>>>
>>>>> For an actual test run, something along the lines of "strace
>>>>> -o/tmp/traceout -f -s2048 -tt -e trace=network java -jar myserver.jar"
>>>>> should output all network-related syscalls that your process performs
>>>>> to
>>>>> /tmp/traceout. You'll have to the comparing yourself, though.
>>>>>
>>>>> I'd expect there to be epoll() calls in MINA case, and if there is
>>>>> something to read, recv() calls to read it. The threaded blocking
>>>>> version
>>>>> should probably just sit blocking in recv(). I'd try to minimize other
>>>>> thing
>>>>> going on in the same process for the test, since there will probably be
>>>>> enough noise in any case.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>
>>
>
>
> --
> --
> cordialement, regards,
> Emmanuel Lécharny
> www.iktek.com
> directory.apache.org
>
>
>

Re: Miserable performance results from MINA 1.1.7

Posted by Emmanuel Lecharny <el...@apache.org>.
Might be related to http://issues.apache.org/jira/browse/DIRMINA-561

Also read 
http://java.sun.com/j2se/1.5.0/docs/api/java/net/ServerSocket.html#setReceiveBufferSize(int) 
:

"It is possible to change the value subsequently, by calling 
|Socket.setReceiveBufferSize(int)| 
<http://java.sun.com/j2se/1.5.0/docs/api/java/net/Socket.html#setReceiveBufferSize%28int%29>. 
However, if the application wishes to allow a receive window larger than 
64K bytes, as defined by RFC1323 then the proposed value must be set in 
the ServerSocket *before* it is bound to a local address. This implies, 
that the ServerSocket must be created with the no-argument constructor, 
then setReceiveBufferSize() must be called and lastly the ServerSocket 
is bound to an address by calling bind(). "

Zvika Gart wrote:
> And here are the logs
>
> On Thu, Nov 26, 2009 at 7:19 PM, Zvika Gart <zv...@gmail.com> wrote:
>
>   
>> Attached are the strace logs with the trace=network for the MINA
>> implementation and the simple NIO implementation. both ran for about 30
>> seconds.
>> As you can see the MINA implementation makes lots of getsockopt system
>> calls to get the receive buffer size (for each receive?).
>>
>> In this link http://79.125.32.246/capture.tbz you can find tcp dumps of
>> the two implementations.
>> ACK patterns are affected by the capabilities of the receiving application
>> - how fast it reads data from the socket. You can see that the window size
>> in the MINA implementation doesn't grow and is stuck at about 64KB while in
>> the simple NIO it gets to 2MB. This is probably because the MINA
>> implementation is slow at reading from the socket.
>>
>> If you need more logs or want me to run further tests please let me know.
>>
>>
>> On Thu, Nov 26, 2009 at 4:04 AM, Tuure Laurinolli <
>>     
>>> tuure.laurinolli@indagon.com> wrote:
>>>
>>>       
>>>> Zvika Gart wrote:
>>>>
>>>>         
>>>>> - Check if there are some differences in the traffic (e.g. TCP ACK
>>>>>           
>>>>>> patterns)
>>>>>>
>>>>>>             
>>>> You didn't say anything about this yet.
>>>>
>>>>
>>>>  - Check syscall patterns of the MINA version and try to see if something
>>>>         
>>>>>> is
>>>>>> taking unreasonably long time (pure NIO implementation might be helpful
>>>>>> here)
>>>>>>
>>>>>>             
>>>>  - As for syscall patterns, I'll need your help as I am a .NET programmer
>>>>         
>>>>> and
>>>>> not familiar with Java/Linux profiling...
>>>>>
>>>>>           
>>>> Have you ever done anything with the Sysinternals file monitoring tools?
>>>> Strace is similar tool for Linux, except its output is textual. You should
>>>> find and read some tutorial about it to get some idea of how to use it.
>>>>
>>>> For an actual test run, something along the lines of "strace
>>>> -o/tmp/traceout -f -s2048 -tt -e trace=network java -jar myserver.jar"
>>>> should output all network-related syscalls that your process performs to
>>>> /tmp/traceout. You'll have to the comparing yourself, though.
>>>>
>>>> I'd expect there to be epoll() calls in MINA case, and if there is
>>>> something to read, recv() calls to read it. The threaded blocking version
>>>> should probably just sit blocking in recv(). I'd try to minimize other thing
>>>> going on in the same process for the test, since there will probably be
>>>> enough noise in any case.
>>>>
>>>>
>>>>         
>
>   


-- 
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org



Re: Miserable performance results from MINA 1.1.7

Posted by Zvika Gart <zv...@gmail.com>.
And here are the logs

On Thu, Nov 26, 2009 at 7:19 PM, Zvika Gart <zv...@gmail.com> wrote:

> Attached are the strace logs with the trace=network for the MINA
> implementation and the simple NIO implementation. both ran for about 30
> seconds.
> As you can see the MINA implementation makes lots of getsockopt system
> calls to get the receive buffer size (for each receive?).
>
> In this link http://79.125.32.246/capture.tbz you can find tcp dumps of
> the two implementations.
> ACK patterns are affected by the capabilities of the receiving application
> - how fast it reads data from the socket. You can see that the window size
> in the MINA implementation doesn't grow and is stuck at about 64KB while in
> the simple NIO it gets to 2MB. This is probably because the MINA
> implementation is slow at reading from the socket.
>
> If you need more logs or want me to run further tests please let me know.
>
>
> On Thu, Nov 26, 2009 at 4:04 AM, Tuure Laurinolli <
>> tuure.laurinolli@indagon.com> wrote:
>>
>>> Zvika Gart wrote:
>>>
>>>> - Check if there are some differences in the traffic (e.g. TCP ACK
>>>>> patterns)
>>>>>
>>>>
>>> You didn't say anything about this yet.
>>>
>>>
>>>  - Check syscall patterns of the MINA version and try to see if something
>>>>> is
>>>>> taking unreasonably long time (pure NIO implementation might be helpful
>>>>> here)
>>>>>
>>>>
>>>  - As for syscall patterns, I'll need your help as I am a .NET programmer
>>>> and
>>>> not familiar with Java/Linux profiling...
>>>>
>>>
>>> Have you ever done anything with the Sysinternals file monitoring tools?
>>> Strace is similar tool for Linux, except its output is textual. You should
>>> find and read some tutorial about it to get some idea of how to use it.
>>>
>>> For an actual test run, something along the lines of "strace
>>> -o/tmp/traceout -f -s2048 -tt -e trace=network java -jar myserver.jar"
>>> should output all network-related syscalls that your process performs to
>>> /tmp/traceout. You'll have to the comparing yourself, though.
>>>
>>> I'd expect there to be epoll() calls in MINA case, and if there is
>>> something to read, recv() calls to read it. The threaded blocking version
>>> should probably just sit blocking in recv(). I'd try to minimize other thing
>>> going on in the same process for the test, since there will probably be
>>> enough noise in any case.
>>>
>>>
>>
>

Re: Miserable performance results from MINA 1.1.7

Posted by Zvika Gart <zv...@gmail.com>.
Attached are the strace logs with the trace=network for the MINA
implementation and the simple NIO implementation. both ran for about 30
seconds.
As you can see the MINA implementation makes lots of getsockopt system calls
to get the receive buffer size (for each receive?).

In this link http://79.125.32.246/capture.tbz you can find tcp dumps of the
two implementations.
ACK patterns are affected by the capabilities of the receiving application -
how fast it reads data from the socket. You can see that the window size in
the MINA implementation doesn't grow and is stuck at about 64KB while in the
simple NIO it gets to 2MB. This is probably because the MINA implementation
is slow at reading from the socket.

If you need more logs or want me to run further tests please let me know.


On Thu, Nov 26, 2009 at 4:04 AM, Tuure Laurinolli <
> tuure.laurinolli@indagon.com> wrote:
>
>> Zvika Gart wrote:
>>
>>> - Check if there are some differences in the traffic (e.g. TCP ACK
>>>> patterns)
>>>>
>>>
>> You didn't say anything about this yet.
>>
>>
>>  - Check syscall patterns of the MINA version and try to see if something
>>>> is
>>>> taking unreasonably long time (pure NIO implementation might be helpful
>>>> here)
>>>>
>>>
>>  - As for syscall patterns, I'll need your help as I am a .NET programmer
>>> and
>>> not familiar with Java/Linux profiling...
>>>
>>
>> Have you ever done anything with the Sysinternals file monitoring tools?
>> Strace is similar tool for Linux, except its output is textual. You should
>> find and read some tutorial about it to get some idea of how to use it.
>>
>> For an actual test run, something along the lines of "strace
>> -o/tmp/traceout -f -s2048 -tt -e trace=network java -jar myserver.jar"
>> should output all network-related syscalls that your process performs to
>> /tmp/traceout. You'll have to the comparing yourself, though.
>>
>> I'd expect there to be epoll() calls in MINA case, and if there is
>> something to read, recv() calls to read it. The threaded blocking version
>> should probably just sit blocking in recv(). I'd try to minimize other thing
>> going on in the same process for the test, since there will probably be
>> enough noise in any case.
>>
>>
>

Re: Miserable performance results from MINA 1.1.7

Posted by Tuure Laurinolli <tu...@indagon.com>.
Zvika Gart wrote:
>> - Check if there are some differences in the traffic (e.g. TCP ACK
>> patterns)

You didn't say anything about this yet.

>> - Check syscall patterns of the MINA version and try to see if something is
>> taking unreasonably long time (pure NIO implementation might be helpful
>> here)

> - As for syscall patterns, I'll need your help as I am a .NET programmer and
> not familiar with Java/Linux profiling...

Have you ever done anything with the Sysinternals file monitoring tools? 
Strace is similar tool for Linux, except its output is textual. You 
should find and read some tutorial about it to get some idea of how to 
use it.

For an actual test run, something along the lines of "strace 
-o/tmp/traceout -f -s2048 -tt -e trace=network java -jar myserver.jar" 
should output all network-related syscalls that your process performs to 
/tmp/traceout. You'll have to the comparing yourself, though.

I'd expect there to be epoll() calls in MINA case, and if there is 
something to read, recv() calls to read it. The threaded blocking 
version should probably just sit blocking in recv(). I'd try to minimize 
other thing going on in the same process for the test, since there will 
probably be enough noise in any case.


Re: Miserable performance results from MINA 1.1.7

Posted by Zvika Gart <zv...@gmail.com>.
On Wed, Nov 25, 2009 at 12:46 PM, Tuure Laurinolli <
tuure.laurinolli@indagon.com> wrote:

> Zvika Gart wrote:
>
>  I wrote a simple client that connects to the server and pumps continuous
>> data.
>> Both client and server run on Amazon's EC2 on Debian Linux instances
>> (64-bit, Java 1.6). Client is in Europe and server in USA.
>>
>>
>> *The results:*
>> Simple Java implementation - 5 Mbps
>> MINA implementation - 750 Kbps
>>
>> Tried removing the setUseDirectBuffers lines; tried a simpler threading
>> model by running the IoHandler on the IoProcessor threads... nothing helps
>> -
>> MINA is capped at 750 Kbps.
>> What is going on here?!
>>
>
> Some things that you might want to do (mostly independent of each other):
> - Check what the receive buffer size is for the MINA version
> - Check if changing receive buffer size in MINA change performance?
>  - NIO requires polling
> - Write a simple pure NIO implementation to see if this might be a NIO
> issue
> - Check if there are some differences in the traffic (e.g. TCP ACK
> patterns)
> - Check what the MINA version and the simple version are doing - e.g. is
> MINA version consuming lots of CPU?
> - Check syscall patterns of the MINA version and try to see if something is
> taking unreasonably long time (pure NIO implementation might be helpful
> here)
>


- A simple single threaded NIO server implementation works fine, reaching
the same throughput as the simple ServerSocket implementation.

- All versions, including MINA, hardly take any CPU (1%)

- As for the receive buffer size, this is where the implementations differ:
the 2 simple implementations (simple NIO and ServerSocket) start at 43690
bytes but quickly reach a receive buffer of 2MB. MINA on the other hand is
stuck with a receive buffer of 43728 bytes. When I configured a large
receive buffer in MINA with
acceptor.getDefaultConfig().getSessionConfig().setReceiveBufferSize(2097152);
I see the server using a buffer of 131071 bytes for the client socket.
This slightly improves performance but not anywhere close to that of the
simple implementations.

- As for syscall patterns, I'll need your help as I am a .NET programmer and
not familiar with Java/Linux profiling...

Re: Miserable performance results from MINA 1.1.7

Posted by Tuure Laurinolli <tu...@indagon.com>.
Zvika Gart wrote:

> I wrote a simple client that connects to the server and pumps continuous
> data.
> Both client and server run on Amazon's EC2 on Debian Linux instances
> (64-bit, Java 1.6). Client is in Europe and server in USA.
> 
> 
> *The results:*
> Simple Java implementation - 5 Mbps
> MINA implementation - 750 Kbps
> 
> Tried removing the setUseDirectBuffers lines; tried a simpler threading
> model by running the IoHandler on the IoProcessor threads... nothing helps -
> MINA is capped at 750 Kbps.
> What is going on here?!

Some things that you might want to do (mostly independent of each other):
- Check what the receive buffer size is for the MINA version
- Check if changing receive buffer size in MINA change performance?
  - NIO requires polling
- Write a simple pure NIO implementation to see if this might be a NIO issue
- Check if there are some differences in the traffic (e.g. TCP ACK patterns)
- Check what the MINA version and the simple version are doing - e.g. is 
MINA version consuming lots of CPU?
- Check syscall patterns of the MINA version and try to see if something 
is taking unreasonably long time (pure NIO implementation might be 
helpful here)

Re: MINA on Amazon EC2 : degraded performances observed, why ? (was Re: Miserable performance results from MINA 1.1.7)

Posted by Zvika Gart <zv...@gmail.com>.
Ran the MINA server on rackspace cloud (http://www.rackspacecloud.com/).
Same results as on Amazon EC2.

On Wed, Nov 25, 2009 at 12:53 PM, Emmanuel Lecharny <el...@apache.org>wrote:

> Zvika Gart wrote:
>
>> The default threading model in 1.1.7 is to use the executor filter. I also
>> tried removing it. Both don't help.
>>
>>
> Ok.
>
>> I tested this scenario using two PCs at the office connected with a
>> crossover cable. The MINA server is reaching the 100 Mbps link speed
>> limit.
>> Why running on Amazon's instances yields such a poor performance??
>>
>>
> Interesting. So MINA on a local network does works fine, and in Amazon,
> it's slow as a dog, when a standard ServerSocket is just plain ok.
>
> I changed the subject so that those who have already used MINA on EC2 can
> popup and give use some insight...
>
> --
> --
> cordialement, regards,
> Emmanuel Lécharny
> www.iktek.com
> directory.apache.org
>
>
>

MINA on Amazon EC2 : degraded performances observed, why ? (was Re: Miserable performance results from MINA 1.1.7)

Posted by Emmanuel Lecharny <el...@apache.org>.
Zvika Gart wrote:
> The default threading model in 1.1.7 is to use the executor filter. I also
> tried removing it. Both don't help.
>   
Ok.
> I tested this scenario using two PCs at the office connected with a
> crossover cable. The MINA server is reaching the 100 Mbps link speed limit.
> Why running on Amazon's instances yields such a poor performance??
>   
Interesting. So MINA on a local network does works fine, and in Amazon, 
it's slow as a dog, when a standard ServerSocket is just plain ok.

I changed the subject so that those who have already used MINA on EC2 
can popup and give use some insight...

-- 
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org



Re: Miserable performance results from MINA 1.1.7

Posted by Zvika Gart <zv...@gmail.com>.
The default threading model in 1.1.7 is to use the executor filter. I also
tried removing it. Both don't help.

I tested this scenario using two PCs at the office connected with a
crossover cable. The MINA server is reaching the 100 Mbps link speed limit.
Why running on Amazon's instances yields such a poor performance??


On Wed, Nov 25, 2009 at 12:20 PM, Emmanuel Lecharny <el...@apache.org>wrote:

> Zvika Gart wrote:
>
>> This is the code of my handler. It doesn't do anything. That's the point.
>> The server should just consume the data as fast as it can.
>>
>>
> Interesting. I have to test this code on my computer to see exactly what
> can be the impact.
>
> Can you add an executor in the filter cxhain to see if it improves the
> performance ?
>
>  Strangely enough, Mina 2.0 performs even worse then 1.1.7
>>
>> Is there some problem in using Java nio on Virtualized machines?
>>
>>
> Not that I know about. Anyone else ?
>
>
> --
> --
> cordialement, regards,
> Emmanuel Lécharny
> www.iktek.com
> directory.apache.org
>
>
>

Re: Miserable performance results from MINA 1.1.7

Posted by Emmanuel Lecharny <el...@apache.org>.
Zvika Gart wrote:
> This is the code of my handler. It doesn't do anything. That's the point.
> The server should just consume the data as fast as it can.
>   
Interesting. I have to test this code on my computer to see exactly what 
can be the impact.

Can you add an executor in the filter cxhain to see if it improves the 
performance ?
> Strangely enough, Mina 2.0 performs even worse then 1.1.7
>
> Is there some problem in using Java nio on Virtualized machines?
>   
Not that I know about. Anyone else ?

-- 
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org



Re: Miserable performance results from MINA 1.1.7

Posted by Zvika Gart <zv...@gmail.com>.
This is the code of my handler. It doesn't do anything. That's the point.
The server should just consume the data as fast as it can.

Strangely enough, Mina 2.0 performs even worse then 1.1.7

Is there some problem in using Java nio on Virtualized machines?


On Wed, Nov 25, 2009 at 11:00 AM, Emmanuel Lecharny <el...@apache.org>wrote:

> Zvika Gart wrote:
>
>> The default is false...
>> I tried disabling Nagle by setting TcpNoDelay to true. As expected, this
>> didn't have any effect - the server doesn't send any data to the client,
>> only receives data.
>>
>>
> Assuming that you are sending 8Kb buffers, anyway, it should not make any
> difference.
>
> Now, I don't have the code in MyHandler, so it's difficult to see what you
> do there.
>
> Also, have you tried with MINA 2.0.0-M6?
>
> --
> --
> cordialement, regards,
> Emmanuel Lécharny
> www.iktek.com
> directory.apache.org
>
>
>

Re: Miserable performance results from MINA 1.1.7

Posted by Emmanuel Lecharny <el...@apache.org>.
Zvika Gart wrote:
> The default is false...
> I tried disabling Nagle by setting TcpNoDelay to true. As expected, this
> didn't have any effect - the server doesn't send any data to the client,
> only receives data.
>   
Assuming that you are sending 8Kb buffers, anyway, it should not make 
any difference.

Now, I don't have the code in MyHandler, so it's difficult to see what 
you do there.

Also, have you tried with MINA 2.0.0-M6?

-- 
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org



Re: Miserable performance results from MINA 1.1.7

Posted by Zvika Gart <zv...@gmail.com>.
The default is false...
I tried disabling Nagle by setting TcpNoDelay to true. As expected, this
didn't have any effect - the server doesn't send any data to the client,
only receives data.


On Tue, Nov 24, 2009 at 9:49 PM, Emmanuel Lecharny <el...@apache.org>wrote:

> Can you add this line in your server :
>
> ((SocketSessionConfig) acceptor.getSessionConfig()).setTcpNoDelay(false)
>
> and see if it makes any difference ?
>
>
>
>
> Zvika Gart wrote:
>
>> Sorry, but Nagle doesn't have anything to do with this...
>> Nagle is relevant at the *client side* when sending small buffers over the
>> network.
>> My client is a simple C# program (not MINA, so SocketConnector is
>> irrelevant) that sends large buffers (100KB) at each call to Socket.Send()
>> If Nagle was to blame then I would see the same bandwidth from both server
>> implementations. This is not the case.
>>
>>
>> On Tue, Nov 24, 2009 at 8:41 PM, Emmanuel Lecharny <elecharny@apache.org
>> >wrote:
>>
>>
>>
>>> Damn Nagle ! ( http://en.wikipedia.org/wiki/Nagle_algorithm )
>>>
>>> Check the FAQ : http://mina.apache.org/faq.html (
>>>
>>>
>>>    Why does SocketConnector send several messages as one message?)
>>>
>>>
>>>
>>> Zvika Gart wrote:
>>>
>>>
>>>
>>>> Hello,
>>>> Please help, as this is too strange to be true...
>>>>
>>>> I'm comparing two implementations of the most basic socket server - it
>>>> listens for a client, and after accepting it just starts reading data
>>>> from
>>>> it.
>>>> The first implementation using Java's plain old ServerSocket:
>>>>
>>>>       ServerSocket srv = new ServerSocket(80);
>>>>   Socket s = srv.accept();
>>>>   InputStream i = s.getInputStream();
>>>>   byte[] buf = new byte[1024 * 8];
>>>>   while (true) {
>>>>    i.read(buf);
>>>>   }
>>>>
>>>> The Second implementation using MINA 1.1.7 (which so simple that I can
>>>> paste
>>>> it here):
>>>>
>>>> First the Main class:
>>>>
>>>> package test;
>>>>
>>>> import java.net.InetSocketAddress;
>>>>
>>>> import org.apache.mina.common.ByteBuffer;
>>>> import org.apache.mina.common.IoAcceptor;
>>>> import org.apache.mina.common.SimpleByteBufferAllocator;
>>>> import org.apache.mina.transport.socket.nio.SocketAcceptor;
>>>>
>>>> public class Main {
>>>>   public static void main(String[] args) throws Exception {
>>>>       // The following two lines don't help
>>>>    ByteBuffer.setUseDirectBuffers(false);
>>>>       ByteBuffer.setAllocator(new SimpleByteBufferAllocator());
>>>>
>>>>       IoAcceptor acceptor = new SocketAcceptor();
>>>>       acceptor.bind(new InetSocketAddress(80), new MyHandler());
>>>>   }
>>>> }
>>>>
>>>>
>>>> And the IoHandler:
>>>>
>>>>
>>>> package test;
>>>>
>>>> import org.apache.mina.common.IoHandlerAdapter;
>>>>
>>>> public class MyHandler extends IoHandlerAdapter {
>>>>
>>>> }
>>>>
>>>>
>>>> I wrote a simple client that connects to the server and pumps continuous
>>>> data.
>>>> Both client and server run on Amazon's EC2 on Debian Linux instances
>>>> (64-bit, Java 1.6). Client is in Europe and server in USA.
>>>>
>>>>
>>>> *The results:*
>>>> Simple Java implementation - 5 Mbps
>>>> MINA implementation - 750 Kbps
>>>>
>>>> Tried removing the setUseDirectBuffers lines; tried a simpler threading
>>>> model by running the IoHandler on the IoProcessor threads... nothing
>>>> helps
>>>> -
>>>> MINA is capped at 750 Kbps.
>>>> What is going on here?!
>>>>
>>>>
>>>>
>>>>
>>>>
>>> --
>>> --
>>> cordialement, regards,
>>> Emmanuel Lécharny
>>> www.iktek.com
>>> directory.apache.org
>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>
>
> --
> --
> cordialement, regards,
> Emmanuel Lécharny
> www.iktek.com
> directory.apache.org
>
>
>

Re: Miserable performance results from MINA 1.1.7

Posted by Emmanuel Lecharny <el...@apache.org>.
Can you add this line in your server :

((SocketSessionConfig) acceptor.getSessionConfig()).setTcpNoDelay(false)

and see if it makes any difference ?



Zvika Gart wrote:
> Sorry, but Nagle doesn't have anything to do with this...
> Nagle is relevant at the *client side* when sending small buffers over the
> network.
> My client is a simple C# program (not MINA, so SocketConnector is
> irrelevant) that sends large buffers (100KB) at each call to Socket.Send()
> If Nagle was to blame then I would see the same bandwidth from both server
> implementations. This is not the case.
>
>
> On Tue, Nov 24, 2009 at 8:41 PM, Emmanuel Lecharny <el...@apache.org>wrote:
>
>   
>> Damn Nagle ! ( http://en.wikipedia.org/wiki/Nagle_algorithm )
>>
>> Check the FAQ : http://mina.apache.org/faq.html (
>>
>>
>>     Why does SocketConnector send several messages as one message?)
>>
>>
>>
>> Zvika Gart wrote:
>>
>>     
>>> Hello,
>>> Please help, as this is too strange to be true...
>>>
>>> I'm comparing two implementations of the most basic socket server - it
>>> listens for a client, and after accepting it just starts reading data from
>>> it.
>>> The first implementation using Java's plain old ServerSocket:
>>>
>>>        ServerSocket srv = new ServerSocket(80);
>>>    Socket s = srv.accept();
>>>    InputStream i = s.getInputStream();
>>>    byte[] buf = new byte[1024 * 8];
>>>    while (true) {
>>>     i.read(buf);
>>>    }
>>>
>>> The Second implementation using MINA 1.1.7 (which so simple that I can
>>> paste
>>> it here):
>>>
>>> First the Main class:
>>>
>>> package test;
>>>
>>> import java.net.InetSocketAddress;
>>>
>>> import org.apache.mina.common.ByteBuffer;
>>> import org.apache.mina.common.IoAcceptor;
>>> import org.apache.mina.common.SimpleByteBufferAllocator;
>>> import org.apache.mina.transport.socket.nio.SocketAcceptor;
>>>
>>> public class Main {
>>>    public static void main(String[] args) throws Exception {
>>>        // The following two lines don't help
>>>     ByteBuffer.setUseDirectBuffers(false);
>>>        ByteBuffer.setAllocator(new SimpleByteBufferAllocator());
>>>
>>>        IoAcceptor acceptor = new SocketAcceptor();
>>>        acceptor.bind(new InetSocketAddress(80), new MyHandler());
>>>    }
>>> }
>>>
>>>
>>> And the IoHandler:
>>>
>>>
>>> package test;
>>>
>>> import org.apache.mina.common.IoHandlerAdapter;
>>>
>>> public class MyHandler extends IoHandlerAdapter {
>>>
>>> }
>>>
>>>
>>> I wrote a simple client that connects to the server and pumps continuous
>>> data.
>>> Both client and server run on Amazon's EC2 on Debian Linux instances
>>> (64-bit, Java 1.6). Client is in Europe and server in USA.
>>>
>>>
>>> *The results:*
>>> Simple Java implementation - 5 Mbps
>>> MINA implementation - 750 Kbps
>>>
>>> Tried removing the setUseDirectBuffers lines; tried a simpler threading
>>> model by running the IoHandler on the IoProcessor threads... nothing helps
>>> -
>>> MINA is capped at 750 Kbps.
>>> What is going on here?!
>>>
>>>
>>>
>>>       
>> --
>> --
>> cordialement, regards,
>> Emmanuel Lécharny
>> www.iktek.com
>> directory.apache.org
>>
>>
>>
>>     
>
>   


-- 
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org



Re: Miserable performance results from MINA 1.1.7

Posted by Zvika Gart <zv...@gmail.com>.
Sorry, but Nagle doesn't have anything to do with this...
Nagle is relevant at the *client side* when sending small buffers over the
network.
My client is a simple C# program (not MINA, so SocketConnector is
irrelevant) that sends large buffers (100KB) at each call to Socket.Send()
If Nagle was to blame then I would see the same bandwidth from both server
implementations. This is not the case.


On Tue, Nov 24, 2009 at 8:41 PM, Emmanuel Lecharny <el...@apache.org>wrote:

> Damn Nagle ! ( http://en.wikipedia.org/wiki/Nagle_algorithm )
>
> Check the FAQ : http://mina.apache.org/faq.html (
>
>
>     Why does SocketConnector send several messages as one message?)
>
>
>
> Zvika Gart wrote:
>
>> Hello,
>> Please help, as this is too strange to be true...
>>
>> I'm comparing two implementations of the most basic socket server - it
>> listens for a client, and after accepting it just starts reading data from
>> it.
>> The first implementation using Java's plain old ServerSocket:
>>
>>        ServerSocket srv = new ServerSocket(80);
>>    Socket s = srv.accept();
>>    InputStream i = s.getInputStream();
>>    byte[] buf = new byte[1024 * 8];
>>    while (true) {
>>     i.read(buf);
>>    }
>>
>> The Second implementation using MINA 1.1.7 (which so simple that I can
>> paste
>> it here):
>>
>> First the Main class:
>>
>> package test;
>>
>> import java.net.InetSocketAddress;
>>
>> import org.apache.mina.common.ByteBuffer;
>> import org.apache.mina.common.IoAcceptor;
>> import org.apache.mina.common.SimpleByteBufferAllocator;
>> import org.apache.mina.transport.socket.nio.SocketAcceptor;
>>
>> public class Main {
>>    public static void main(String[] args) throws Exception {
>>        // The following two lines don't help
>>     ByteBuffer.setUseDirectBuffers(false);
>>        ByteBuffer.setAllocator(new SimpleByteBufferAllocator());
>>
>>        IoAcceptor acceptor = new SocketAcceptor();
>>        acceptor.bind(new InetSocketAddress(80), new MyHandler());
>>    }
>> }
>>
>>
>> And the IoHandler:
>>
>>
>> package test;
>>
>> import org.apache.mina.common.IoHandlerAdapter;
>>
>> public class MyHandler extends IoHandlerAdapter {
>>
>> }
>>
>>
>> I wrote a simple client that connects to the server and pumps continuous
>> data.
>> Both client and server run on Amazon's EC2 on Debian Linux instances
>> (64-bit, Java 1.6). Client is in Europe and server in USA.
>>
>>
>> *The results:*
>> Simple Java implementation - 5 Mbps
>> MINA implementation - 750 Kbps
>>
>> Tried removing the setUseDirectBuffers lines; tried a simpler threading
>> model by running the IoHandler on the IoProcessor threads... nothing helps
>> -
>> MINA is capped at 750 Kbps.
>> What is going on here?!
>>
>>
>>
>
>
> --
> --
> cordialement, regards,
> Emmanuel Lécharny
> www.iktek.com
> directory.apache.org
>
>
>

Re: Miserable performance results from MINA 1.1.7

Posted by Emmanuel Lecharny <el...@apache.org>.
Damn Nagle ! ( http://en.wikipedia.org/wiki/Nagle_algorithm )

Check the FAQ : http://mina.apache.org/faq.html (


      Why does SocketConnector send several messages as one message?)


Zvika Gart wrote:
> Hello,
> Please help, as this is too strange to be true...
>
> I'm comparing two implementations of the most basic socket server - it
> listens for a client, and after accepting it just starts reading data from
> it.
> The first implementation using Java's plain old ServerSocket:
>
>         ServerSocket srv = new ServerSocket(80);
>     Socket s = srv.accept();
>     InputStream i = s.getInputStream();
>     byte[] buf = new byte[1024 * 8];
>     while (true) {
>      i.read(buf);
>     }
>
> The Second implementation using MINA 1.1.7 (which so simple that I can paste
> it here):
>
> First the Main class:
>
> package test;
>
> import java.net.InetSocketAddress;
>
> import org.apache.mina.common.ByteBuffer;
> import org.apache.mina.common.IoAcceptor;
> import org.apache.mina.common.SimpleByteBufferAllocator;
> import org.apache.mina.transport.socket.nio.SocketAcceptor;
>
> public class Main {
>     public static void main(String[] args) throws Exception {
>         // The following two lines don't help
>      ByteBuffer.setUseDirectBuffers(false);
>         ByteBuffer.setAllocator(new SimpleByteBufferAllocator());
>
>         IoAcceptor acceptor = new SocketAcceptor();
>         acceptor.bind(new InetSocketAddress(80), new MyHandler());
>     }
> }
>
>
> And the IoHandler:
>
>
> package test;
>
> import org.apache.mina.common.IoHandlerAdapter;
>
> public class MyHandler extends IoHandlerAdapter {
>
> }
>
>
> I wrote a simple client that connects to the server and pumps continuous
> data.
> Both client and server run on Amazon's EC2 on Debian Linux instances
> (64-bit, Java 1.6). Client is in Europe and server in USA.
>
>
> *The results:*
> Simple Java implementation - 5 Mbps
> MINA implementation - 750 Kbps
>
> Tried removing the setUseDirectBuffers lines; tried a simpler threading
> model by running the IoHandler on the IoProcessor threads... nothing helps -
> MINA is capped at 750 Kbps.
> What is going on here?!
>
>   


-- 
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org