You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@directory.apache.org by Vinod Panicker <vi...@gmail.com> on 2005/03/22 06:18:52 UTC
[mina] Performance issues
Hi,
I was just trying to do some tests using mina. I've got a server and
a client, both of which are using mina and the thread pool filter.
The server and the client have been run on multiple OS's - both
Windows (2000 & 2003) and Linux (Fedora Core 3).
Max fd limits have been upped.
The server uses the IoProtocolAcceptor class. The client uses the
SocketConnector class.
The client application is basically just connecting to the server and
maintaining a concurrent connection.
When the client app is run, and told to establish 2000 connections to
the server for eg, the connection rate suddenly (no pattern observed)
drops to about 1 connection per second. This has been tried on all
the OS's mentioned above. The server is apparently not the cause of
this issue because when a different instance of the client is started
(on the same machine, even) the connection rate remains normal. This
behaviour is not consistent, though it happens on every other run of
the client.
There are no sleep()'s in the client / server code.
Any pointers on what is happening?
Also, is there a way using which the server connection accept backlog
(i forget the exact term) can be set?
Regards,
Vinod.
PS: This same test has been carried out using our own code (non-mina)
and works perfectly on all the above OS's (over 20k connections on the
server)
[mina] Performance issues
Posted by Vinod Panicker <vi...@gmail.com>.
Dunno if i've sent this already - was suspicious about the mailing
list working correctly with my old subscription.. sorry for the
inconvenience
OK.. here goes..
I was just trying to do some tests using mina. I've got a server and
a client, both of which are using mina and the thread pool filter.
The server and the client have been run on multiple OS's - both
Windows (2000 & 2003) and Linux (Fedora Core 3).
Max fd limits have been upped.
The server uses the IoProtocolAcceptor class. The client uses the
SocketConnector class.
The client application is basically just connecting to the server and
maintaining a concurrent connection.
When the client app is run, and told to establish 2000 connections to
the server for eg, the connection rate suddenly (no pattern observed)
drops to about 1 connection per second. This has been tried on all
the OS's mentioned above. The server is apparently not the cause of
this issue because when a different instance of the client is started
(on the same machine, even) the connection rate remains normal. This
behaviour is not consistent, though it happens on every other run of
the client.
There are no sleep()'s in the client / server code.
Any pointers on what is happening?
Also, is there a way using which the server connection accept backlog
(i forget the exact term) can be set?
Regards,
Vinod.
PS: This same test has been carried out using our own code (non-mina)
and works perfectly on all the above OS's (over 20k connections on the
server)
Re: [mina] Performance issues
Posted by Vinod Panicker <vi...@gmail.com>.
Good idea. I didnt think of that maybe coz CPU usage was just around
4-5% during these tests.
Regards,
Vinod.
On Tue, 22 Mar 2005 02:00:09 -0500, Alex Karasulu <ao...@bellsouth.net> wrote:
> Might be good to try to look at what is causing the problem using YourKit.
>
> Alex
>
-- snip --
Re: [mina] Performance issues
Posted by Alex Karasulu <ao...@bellsouth.net>.
Might be good to try to look at what is causing the problem using YourKit.
Alex
Trustin Lee wrote:
>Hi,
>
>On Tue, 22 Mar 2005 10:48:52 +0530, Vinod Panicker <vi...@gmail.com> wrote:
>
>
>>Hi,
>>
>>I was just trying to do some tests using mina. I've got a server and
>>a client, both of which are using mina and the thread pool filter.
>>
>>The server and the client have been run on multiple OS's - both
>>Windows (2000 & 2003) and Linux (Fedora Core 3).
>>
>>Max fd limits have been upped.
>>
>>The server uses the IoProtocolAcceptor class. The client uses the
>>SocketConnector class.
>>
>>The client application is basically just connecting to the server and
>>maintaining a concurrent connection.
>>
>>When the client app is run, and told to establish 2000 connections to
>>the server for eg, the connection rate suddenly (no pattern observed)
>>drops to about 1 connection per second. This has been tried on all
>>the OS's mentioned above. The server is apparently not the cause of
>>this issue because when a different instance of the client is started
>>(on the same machine, even) the connection rate remains normal. This
>>behaviour is not consistent, though it happens on every other run of
>>the client.
>>
>>There are no sleep()'s in the client / server code.
>>
>>Any pointers on what is happening?
>>
>>Also, is there a way using which the server connection accept backlog
>>(i forget the exact term) can be set?
>>
>>
>
>There is a 'bind()' method with backlog parameter in SocketAcceptor.
>IoAcceptor doesn't have it, so you'll have to down-cast it.
>
>
>
>>PS: This same test has been carried out using our own code (non-mina)
>>and works perfectly on all the above OS's (over 20k connections on the
>>server)
>>
>>
>
>I don't have any test machines to do that massive test. Can that kind
>of test be reproducable in two PCs? If so, I'll test by myself and
>figure out why.
>
>Regards,
>Trustin.
>
>
Re: [mina] Performance issues
Posted by Vinod Panicker <vi...@gmail.com>.
Hi Emmanuel,
You had forgotten to send to the list, so sending it now.
Regarding the performance tests, what I was planning was a couple of
java apps (client and server) that would allow me to saturate
resources such as CPU, Memory, Network etc.
Its just in its infancy, and would be building it up over time. Would
be honored to contribute code to the project.
Regards,
Vinod.
On Tue, 22 Mar 2005 08:31:34 +0100, Emmanuel Lecharny
<el...@apache.org> wrote:
> > Its not a massive test. The Linux machine i'm using has got 512 MB
> > RAM. The Windows box has 1 GB. You should be able to run both client
> > and server on the same machine without any issues, on a machine with
> > even 256 MB RAM IMO.
> >
> > Do you want me to send you the source files so you can test it out?
>
> Hi !
>
> Just peeking the mails, (I do no have any answer), I think that we can
> have a sub-project dedicated to performance tests.
>
> Your code could be a part of it, may be the first massive test, if you
> don't mind to give it to the community (if you don't or can't, no pb,
> but the sub-project proposition remains ;-).
>
> What do you think (Vinod and Directory guys?)
>
> -- Emmanuel Lécharny
>
> --
> Emmanuel Lecharny <el...@apache.org>
> iktek
>
>
Re: [mina] Performance issues
Posted by Alex Karasulu <ao...@bellsouth.net>.
Vinod Panicker wrote:
>With the heap size for the JVM specified as 500 MB, the server is able
>to have 25K + connections. I need to spruce up the client now to be
>able to establish more connections.
>
>Lets see how many concurrent connections the mina server can support.
>
>
Excellent Vinod these statistics are great to have. Do you mind
compiling these stats and the memory footprints you observe into a Wiki
page.
One of the questions I've always wanted to know (too busy with code
though) was the amount of data stored in the server with the
responsiveness degradation. I want to see just how much data we can put
in a single partition before she starts to buckle.
Thanks,
Alex
>Regards,
>Vinod.
>
>
>On Wed, 23 Mar 2005 12:17:49 +0530, Vinod Panicker <vi...@gmail.com> wrote:
>
>
>>Hi,
>>
>>The Connector seems to be working much better now. Memory consumption
>>is still an issue with the server running out of memory at around 8000
>>concurrent connections.
>>
>>Here's the stack trace -
>>
>>Exception in thread "SocketAcceptor-0" java.lang.OutOfMemoryError:
>>Direct buffer memory
>> at java.nio.Bits.reserveMemory(Unknown Source)
>> at java.nio.DirectByteBuffer.<init>(Unknown Source)
>> at java.nio.ByteBuffer.allocateDirect(Unknown Source)
>> at org.apache.mina.common.ByteBuffer.allocate(ByteBuffer.java:95)
>> at org.apache.mina.io.socket.SocketSession.<init>(SocketSession.java:94)
>> at org.apache.mina.io.socket.SocketAcceptor$Worker.run(SocketAcceptor.java:227)
>>
>>This was using default JVM settings till now. Tinkering out with
>>different heap sizes now.
>>
>>I'm using JDK 1.5.0_01
>>
>>Regards,
>>Vinod.
>>
>>On Wed, 23 Mar 2005 10:29:03 +0530, Vinod Panicker <vi...@gmail.com> wrote:
>>
>>
>>>Hi Trustin,
>>>
>>>Thanks for the fix, checking it out now.
>>>
>>>
>>>On Wed, 23 Mar 2005 00:03:29 +0900, Trustin Lee <tr...@gmail.com> wrote:
>>>
>>>
>>>>Hi Vinod,
>>>>
>>>>
>>>>
>>>>>It seems like the issue is in the Connector. When I used a non-mina
>>>>>based client running in a single thread to connect to the mina based
>>>>>server, it works fine.
>>>>>
>>>>>
>>>>I found a serious flaw that will degrade SocketConnector performance.
>>>>I checked in the fix into trunk. Could you retry?
>>>>
>>>>
>>>>
>>>>>The server runs out of memory after a bit over 8000 connections.
>>>>>
>>>>>
>>>>What was max heap size? Does this mean we have to lessen MINA's
>>>>memory consumption?
>>>>
>>>>
>>>Actually rather than looking at it from that perspective, i'd look at
>>>is as "how much memory should be consumed for maintaining a single
>>>connected socket"
>>>
>>>IMO, on a "standard" linux box, number of connected sockets supported
>>>should be around 50K or even more.
>>>
>>>
>>>
>>>><snip/>
>>>>
>>>>Thanks,
>>>>Trustin.
>>>>--
>>>>what we call human nature is actually human habit
>>>>--
>>>>http://gleamynode.net/
>>>>
>>>>
>>>>
>>>Regards,
>>>Vinod.
>>>
>>>
>>>
>
>
>
Re: [mina] Performance issues
Posted by Trustin Lee <tr...@gmail.com>.
It would be nice if we can test with new snapshot to see if the
changes in filter chains affects performance or not. :)
On Wed, 23 Mar 2005 13:43:21 +0530, Vinod Panicker <vi...@gmail.com> wrote:
> With the heap size for the JVM specified as 500 MB, the server is able
> to have 25K + connections. I need to spruce up the client now to be
> able to establish more connections.
>
> Lets see how many concurrent connections the mina server can support.
>
> Regards,
> Vinod.
>
>
> On Wed, 23 Mar 2005 12:17:49 +0530, Vinod Panicker <vi...@gmail.com> wrote:
> > Hi,
> >
> > The Connector seems to be working much better now. Memory consumption
> > is still an issue with the server running out of memory at around 8000
> > concurrent connections.
> >
> > Here's the stack trace -
> >
> > Exception in thread "SocketAcceptor-0" java.lang.OutOfMemoryError:
> > Direct buffer memory
> > at java.nio.Bits.reserveMemory(Unknown Source)
> > at java.nio.DirectByteBuffer.<init>(Unknown Source)
> > at java.nio.ByteBuffer.allocateDirect(Unknown Source)
> > at org.apache.mina.common.ByteBuffer.allocate(ByteBuffer.java:95)
> > at org.apache.mina.io.socket.SocketSession.<init>(SocketSession.java:94)
> > at org.apache.mina.io.socket.SocketAcceptor$Worker.run(SocketAcceptor.java:227)
> >
> > This was using default JVM settings till now. Tinkering out with
> > different heap sizes now.
> >
> > I'm using JDK 1.5.0_01
> >
> > Regards,
> > Vinod.
> >
> > On Wed, 23 Mar 2005 10:29:03 +0530, Vinod Panicker <vi...@gmail.com> wrote:
> > > Hi Trustin,
> > >
> > > Thanks for the fix, checking it out now.
> > >
> > >
> > > On Wed, 23 Mar 2005 00:03:29 +0900, Trustin Lee <tr...@gmail.com> wrote:
> > > > Hi Vinod,
> > > >
> > > > > It seems like the issue is in the Connector. When I used a non-mina
> > > > > based client running in a single thread to connect to the mina based
> > > > > server, it works fine.
> > > >
> > > > I found a serious flaw that will degrade SocketConnector performance.
> > > > I checked in the fix into trunk. Could you retry?
> > > >
> > > > > The server runs out of memory after a bit over 8000 connections.
> > > >
> > > > What was max heap size? Does this mean we have to lessen MINA's
> > > > memory consumption?
> > >
> > > Actually rather than looking at it from that perspective, i'd look at
> > > is as "how much memory should be consumed for maintaining a single
> > > connected socket"
> > >
> > > IMO, on a "standard" linux box, number of connected sockets supported
> > > should be around 50K or even more.
> > >
> > > >
> > > > <snip/>
> > > >
> > > > Thanks,
> > > > Trustin.
> > > > --
> > > > what we call human nature is actually human habit
> > > > --
> > > > http://gleamynode.net/
> > > >
> > >
> > > Regards,
> > > Vinod.
> > >
> >
>
Thanks,
Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/
Re: [mina] Performance issues
Posted by Vinod Panicker <vi...@gmail.com>.
With the heap size for the JVM specified as 500 MB, the server is able
to have 25K + connections. I need to spruce up the client now to be
able to establish more connections.
Lets see how many concurrent connections the mina server can support.
Regards,
Vinod.
On Wed, 23 Mar 2005 12:17:49 +0530, Vinod Panicker <vi...@gmail.com> wrote:
> Hi,
>
> The Connector seems to be working much better now. Memory consumption
> is still an issue with the server running out of memory at around 8000
> concurrent connections.
>
> Here's the stack trace -
>
> Exception in thread "SocketAcceptor-0" java.lang.OutOfMemoryError:
> Direct buffer memory
> at java.nio.Bits.reserveMemory(Unknown Source)
> at java.nio.DirectByteBuffer.<init>(Unknown Source)
> at java.nio.ByteBuffer.allocateDirect(Unknown Source)
> at org.apache.mina.common.ByteBuffer.allocate(ByteBuffer.java:95)
> at org.apache.mina.io.socket.SocketSession.<init>(SocketSession.java:94)
> at org.apache.mina.io.socket.SocketAcceptor$Worker.run(SocketAcceptor.java:227)
>
> This was using default JVM settings till now. Tinkering out with
> different heap sizes now.
>
> I'm using JDK 1.5.0_01
>
> Regards,
> Vinod.
>
> On Wed, 23 Mar 2005 10:29:03 +0530, Vinod Panicker <vi...@gmail.com> wrote:
> > Hi Trustin,
> >
> > Thanks for the fix, checking it out now.
> >
> >
> > On Wed, 23 Mar 2005 00:03:29 +0900, Trustin Lee <tr...@gmail.com> wrote:
> > > Hi Vinod,
> > >
> > > > It seems like the issue is in the Connector. When I used a non-mina
> > > > based client running in a single thread to connect to the mina based
> > > > server, it works fine.
> > >
> > > I found a serious flaw that will degrade SocketConnector performance.
> > > I checked in the fix into trunk. Could you retry?
> > >
> > > > The server runs out of memory after a bit over 8000 connections.
> > >
> > > What was max heap size? Does this mean we have to lessen MINA's
> > > memory consumption?
> >
> > Actually rather than looking at it from that perspective, i'd look at
> > is as "how much memory should be consumed for maintaining a single
> > connected socket"
> >
> > IMO, on a "standard" linux box, number of connected sockets supported
> > should be around 50K or even more.
> >
> > >
> > > <snip/>
> > >
> > > Thanks,
> > > Trustin.
> > > --
> > > what we call human nature is actually human habit
> > > --
> > > http://gleamynode.net/
> > >
> >
> > Regards,
> > Vinod.
> >
>
Re: [mina] Performance issues
Posted by Vinod Panicker <vi...@gmail.com>.
Hey Niclas,
Thanks for the generous offer :) Will definitely take you up on that
once I have something thats sorted out. There is still quite a long
way to go with this. I also have a dual proc 2 GB RAM server coming
in in a few days.
Regards,
Vinod.
On Thu, 24 Mar 2005 21:52:33 +0800, Niclas Hedhman <ni...@hedhman.org> wrote:
> On Thursday 24 March 2005 21:37, Vinod Panicker wrote:
> > Mem usage over 1 GB (machine only has 512, so becomes unresponsive at
> > times because of paging)
>
> If the test setup is dead easy, I have two 2GB machines available here (1Linux
> + 1 Windows) and could run it for you. Provided instructions are effectively;
> unzip Filename
> cd mina-test
> ./start-server.sh
>
> and same for client.
>
> Cheers
> Niclas
>
Re: [mina] Performance issues
Posted by Niclas Hedhman <ni...@hedhman.org>.
On Thursday 24 March 2005 21:37, Vinod Panicker wrote:
> Mem usage over 1 GB (machine only has 512, so becomes unresponsive at
> times because of paging)
If the test setup is dead easy, I have two 2GB machines available here (1Linux
+ 1 Windows) and could run it for you. Provided instructions are effectively;
unzip Filename
cd mina-test
./start-server.sh
and same for client.
Cheers
Niclas
Re: [mina] Performance issues
Posted by Vinod Panicker <vi...@gmail.com>.
OK... reaching around 85K connections on the server, but there is too
much of CPU usage and disk thrashing due to memory constraints due to
which connections start getting dropped.
Trying this with the existing MINA functionality - idle connections
being processed.
Have reduced the read buffer size to 8 bytes (for less mem usage,
impractical in the real-world)
CPU usage 100% constantly on the server, with 87% in kernel and 13% in
user space.
Mem usage over 1 GB (machine only has 512, so becomes unresponsive at
times because of paging)
The client is currently a blocking IO one (non-mina). Seems like
there is some issue still in the Connector due to which the connection
establishment starts getting bogged down as it progresses (non-linear
performance), and with 100% CPU usage. With this client, the CPU
usage stays at around 7-10% (2 Ghz machine).
We've got an extended weekend here, so will be back at this on Monday.
Will surely hit C100K then :)
Regards,
Vinod.
Re: [mina] Performance issues
Posted by Alex Karasulu <ao...@bellsouth.net>.
David Boreham wrote:
> Vinod Panicker wrote:
>
>> On Thu, 24 Mar 2005 08:26:19 -0800, David Boreham
>> <da...@bozemanpass.com> wrote:
>>
>>> Presumably you guys have read the master work on this
>>> subject ? : http://www.kegel.com/c10k.html
>>
>> Been following this for years. Trying to make his C100K dream come
>> true :)
>>
>> And I'm sure MINA will be figuring over there pretty soon.
>
> Send Dan a link , I'm sure he'd be happy to add it to the page.
+1
-- Alex
Re: [mina] Performance issues
Posted by David Boreham <da...@bozemanpass.com>.
Vinod Panicker wrote:
>On Thu, 24 Mar 2005 08:26:19 -0800, David Boreham <da...@bozemanpass.com> wrote:
>
>
>>Presumably you guys have read the master work on this
>>subject ? : http://www.kegel.com/c10k.html
>>
>>
>
>Been following this for years. Trying to make his C100K dream come true :)
>
>And I'm sure MINA will be figuring over there pretty soon.
>
>
>
Send Dan a link , I'm sure he'd be happy to add it to the page.
Re: [mina] Performance issues
Posted by Vinod Panicker <vi...@gmail.com>.
On Thu, 24 Mar 2005 08:26:19 -0800, David Boreham <da...@bozemanpass.com> wrote:
> Presumably you guys have read the master work on this
> subject ? : http://www.kegel.com/c10k.html
Been following this for years. Trying to make his C100K dream come true :)
And I'm sure MINA will be figuring over there pretty soon.
Regards,
Vinod.
Re: [mina] Performance issues
Posted by David Boreham <da...@bozemanpass.com>.
Presumably you guys have read the master work on this
subject ? : http://www.kegel.com/c10k.html
Re: [mina] Performance issues
Posted by Vinod Panicker <vi...@gmail.com>.
Found something real weird. Without the jvm options of -Xmn100M
-Xms500M -Xmx500M which I was providing regularly for the client and
server, the client (non-mina) drops connections. Anyways, got that
sorted and its working fine now. Sorry for the false alarm.
Regards,
Vinod.
On Mon, 28 Mar 2005 12:32:15 +0530, Vinod Panicker <vi...@gmail.com> wrote:
> Trustin,
>
> The latest version of mina seems to be buggy. Its dropping
> connections. After 200 established connections, it drops 2
> connections and once it reaches the 600 mark it drops over 200
> connections.
>
> Pls do check it out.
>
> Regards,
> Vinod.
>
Re: [mina] Performance issues
Posted by Vinod Panicker <vi...@gmail.com>.
Trustin,
The latest version of mina seems to be buggy. Its dropping
connections. After 200 established connections, it drops 2
connections and once it reaches the 600 mark it drops over 200
connections.
Pls do check it out.
Regards,
Vinod.
Re: [mina] Performance issues
Posted by Berin Loritsch <bl...@d-haven.org>.
Vinod Panicker wrote:
> On Thu, 24 Mar 2005 09:20:53 -0500, Berin Loritsch
> <bl...@d-haven.org> wrote:
>
<snip type="my poor and false understanding">
>>With 30k connections we may be at the practical limits of what the OS
>>will allow. I can guarantee this much, on a properly locked down box,
>>you will have a hard time having any socket under the 1k mark. That
>>means you would be limited to at most 63K connections. Not to mention
>>we would start competing with other network bound services on our
>>Unix/Windows boxes. RPC is considered bad to have running, but it is
>>there by default on Windows and necessary for NFS mounted shares.
>
>
> Didnt quite get what you're saying out here.
Basically, based on "my poor and false understanding", if the server
uses additional ports to process connections (the new socket) you would
be sharing with other OS resources using sockets.
>>Keep in mind that even the original SEDA project with the experimental
>>Haboob HTTP server was targeting 1000 simultaneous connections. We have
>>far acceeded that with no load. Pretty soon we will have to see how the
>>system responds with load, but you've already alluded to that.
>
> The target system for me is an IM system, and I've set the per-box
> concurrent connection limit with required performance at 20,000 :) If
> I finalize on a Shared Nothing architecture, this figure might not be
> achieveable. Looks reasonable to me though.
With MINA it's looking awfully good.
Re: [mina] Performance issues
Posted by Vinod Panicker <vi...@gmail.com>.
On Thu, 24 Mar 2005 09:20:53 -0500, Berin Loritsch
<bl...@d-haven.org> wrote:
> Trustin Lee wrote:
>
> >>I'm expecting that MINA should be able to take 100K connections on the
> >>server. It cant go over 65K for the client due to lack of available
> >>port numbers.
> >
> >
> > Can we accept more than 64K connections on the server side?
> >
>
> Thats a good point! If I recall correctly, you would be limited to 64k
> - 1 connections. The details are fuzzy on this one as its been a while
> since I been familiar with papers on Berkley sockets. Nevertheless the
> ServerSocket will listen on the port its bound to. When the server
> accepts the socket connection a new Socket is created to communicate
> over an aliased port. In other words, the listening can be happening on
> port 80, but the actual communication might happen on port 30231.
Nopes. AFAIK the server doesnt use any ports except the one its
listening on when communicating with multiple clients. netstat says
it doesnt, and so does the number of concurrent sessions I've managed
to establish.
> One of the challenges I faced with creating the denial service on the
> SEDANG project was that the server was listening on port 6666 or
> whatever, but the created SocketChannel was on a completely different
> port. I got around that, but nevertheless it is how sockets work in Java.
Doesnt seem to fit in with my understanding. Would be interested in
seeing it though.
> With 30k connections we may be at the practical limits of what the OS
> will allow. I can guarantee this much, on a properly locked down box,
> you will have a hard time having any socket under the 1k mark. That
> means you would be limited to at most 63K connections. Not to mention
> we would start competing with other network bound services on our
> Unix/Windows boxes. RPC is considered bad to have running, but it is
> there by default on Windows and necessary for NFS mounted shares.
Didnt quite get what you're saying out here.
> Keep in mind that even the original SEDA project with the experimental
> Haboob HTTP server was targeting 1000 simultaneous connections. We have
> far acceeded that with no load. Pretty soon we will have to see how the
> system responds with load, but you've already alluded to that.
The target system for me is an IM system, and I've set the per-box
concurrent connection limit with required performance at 20,000 :) If
I finalize on a Shared Nothing architecture, this figure might not be
achieveable. Looks reasonable to me though.
Regards,
Vinod.
Re: [mina] Performance issues
Posted by Berin Loritsch <bl...@d-haven.org>.
Trustin Lee wrote:
>>I'm expecting that MINA should be able to take 100K connections on the
>>server. It cant go over 65K for the client due to lack of available
>>port numbers.
>
>
> Can we accept more than 64K connections on the server side?
>
Thats a good point! If I recall correctly, you would be limited to 64k
- 1 connections. The details are fuzzy on this one as its been a while
since I been familiar with papers on Berkley sockets. Nevertheless the
ServerSocket will listen on the port its bound to. When the server
accepts the socket connection a new Socket is created to communicate
over an aliased port. In other words, the listening can be happening on
port 80, but the actual communication might happen on port 30231.
One of the challenges I faced with creating the denial service on the
SEDANG project was that the server was listening on port 6666 or
whatever, but the created SocketChannel was on a completely different
port. I got around that, but nevertheless it is how sockets work in Java.
With 30k connections we may be at the practical limits of what the OS
will allow. I can guarantee this much, on a properly locked down box,
you will have a hard time having any socket under the 1k mark. That
means you would be limited to at most 63K connections. Not to mention
we would start competing with other network bound services on our
Unix/Windows boxes. RPC is considered bad to have running, but it is
there by default on Windows and necessary for NFS mounted shares.
Keep in mind that even the original SEDA project with the experimental
Haboob HTTP server was targeting 1000 simultaneous connections. We have
far acceeded that with no load. Pretty soon we will have to see how the
system responds with load, but you've already alluded to that.
Re: [mina] Performance issues
Posted by Vinod Panicker <vi...@gmail.com>.
On Thu, 24 Mar 2005 15:13:23 +0900, Trustin Lee <tr...@gmail.com> wrote:
> Hi Vinod,
>
> On Thu, 24 Mar 2005 10:50:21 +0530, Vinod Panicker <vi...@gmail.com> wrote:
> <snip/>
>
> > Alright... Latest stats are as follows -
> >
-- snip --
Latest stats stand at 50K connections now. Too much CPU usage.
-- snip --
> >
> > I agree with what Berin says, and every Internet server should know
> > its limits. Thats where these benchmarks help - we can estimate what
> > kind of load is MAX, under which the system can still provide expected
> > performance levels.
>
> We'll be able to create a IoHandlerFilter that limits the maximum
> number of connected clients.
I think this Filter should consider most DOS attack scenarios. Would
be excellent to have a Network library that takes care of most DOS
issues out of the box. Makes a lot of sense for servers such as
ApacheDS as well, as a H/W firewall becomes less necessary.
-- snip --
>
> I think we can provide your testing code into separate subproject.
Whatever you guys think is best. Suggestions welcome for the benchmark suite.
-- snip --
>
> You're right. I've just added SocketAcceptor.get/setBacklog() method.
> So you can:
>
> ((SocketAcceptor) ioProtocolAcceptor.getIoAcceptor()).setBacklog( yourBacklog );
Thanks!
-- snip --
>
> I reduced it to 1K and reconfigurable, and added
> SocketSessionConfig.setSessionReceiveBufferSize(...). :)
Alex will get a shock when he sees the performance now :)
>
> > 3. Client/Server CPU usage. The number of connections per second
> > steadily keeps decreasing as the total connections increase. Client &
> > Server both are at 100% CPU. That is what is weird. I'm aiming to
> > isolate the problem. I'm currently a bit doubtful about the way the
> > Connector works, but wont be able to point out anything concrete until
> > more permutations & combinations are tried.
>
> Hmm, interesting. This might be related with
> processIdleSessions(...). Could you try to disable it?
Couldnt find anything like this? Where's it supposed to be?
>
> > I'm expecting that MINA should be able to take 100K connections on the
> > server. It cant go over 65K for the client due to lack of available
> > port numbers.
>
> Can we accept more than 64K connections on the server side?
Yes, of course! The server, unlike the client is not constricted by
the number of ports available, since it is simply using a file
descriptor per connection rather than a port number.
So theoretically, it should only be limited by Hardware and OS limits.
And since OS limits have been recently upped significantly (eg the
2.6 kernel series and Windows 2003 server), this should not be a
problem at all.
>
> <snip/>
>
> Thanks for your consistent contribution! :)
Most welcome. I just hope you will keep responding to valid change
requests. Its high time that Java has a carrier-grade network
library.
Regards,
Vinod.
Re: [mina] Performance issues
Posted by Vinod Panicker <vi...@gmail.com>.
On Thu, 24 Mar 2005 15:13:23 +0900, Trustin Lee <tr...@gmail.com> wrote:
> Hi Vinod,
>
> On Thu, 24 Mar 2005 10:50:21 +0530, Vinod Panicker <vi...@gmail.com> wrote:
> <snip/>
>
> > 3. Client/Server CPU usage. The number of connections per second
> > steadily keeps decreasing as the total connections increase. Client &
> > Server both are at 100% CPU. That is what is weird. I'm aiming to
> > isolate the problem. I'm currently a bit doubtful about the way the
> > Connector works, but wont be able to point out anything concrete until
> > more permutations & combinations are tried.
>
> Hmm, interesting. This might be related with
> processIdleSessions(...). Could you try to disable it?
Found notifyIdleSessions in SocketIoProcessor, which looks to me as a
killer. Its traversing all the keys on the selector. Huge
performance hit when number of connections are more. Any other
approach available to obtain this functionality? Will be trying with
this commented. Should make a whole world of difference IMO.
Regards,
Vinod.
Re: [mina] Performance issues
Posted by Trustin Lee <tr...@gmail.com>.
Hi Vinod,
On Thu, 24 Mar 2005 10:50:21 +0530, Vinod Panicker <vi...@gmail.com> wrote:
<snip/>
> Alright... Latest stats are as follows -
>
> Server - 30K concurrent connections on a Linux box with 512 MB RAM.
> Ran out of Direct Buffer Memory (more on this later)
> Client - 30K concurrent connections from a single Windows 2003 server
> box 1 GB RAM (no problems)
Sounds great.
> Basically the client is going full throttle with connect requests.
> Obviously we will run benchmarks with more sedate clients that
> occasionally cause load spikes.
>
> I agree with what Berin says, and every Internet server should know
> its limits. Thats where these benchmarks help - we can estimate what
> kind of load is MAX, under which the system can still provide expected
> performance levels.
We'll be able to create a IoHandlerFilter that limits the maximum
number of connected clients.
> Alex, there's still lot more to do. Will compile all of this into
> something that Trustin can put someplace. I'm also tinkering around
> with JVMStat to obtain exact figures. Regarding the data limits per
> partition, I wont be able to help out over there (I think you should
> be looking at external storage for such kind of benchmarks)
I think we can provide your testing code into separate subproject.
> Alright, problem areas -
>
> 1. No backlog support currently provided with IoProtocolAcceptor
You're right. I've just added SocketAcceptor.get/setBacklog() method.
So you can:
((SocketAcceptor) ioProtocolAcceptor.getIoAcceptor()).setBacklog( yourBacklog );
> 2. When the server accepts a connection, 8K of Direct Buffer Memory is
> being allocated. I suspect this is too much. Depending on where MINA
> is used, this memory should be tunable. From what I saw in the code,
> this is for the read buffer. Taking an example of a Directory Server
> such as ApacheDS, I doubt incoming data is going to fill the 8K
> buffer. And since its direct buffer, more care should be taken to use
> up this kind of memory. If MINA is being used for an FTP server, this
> figure would make sense. My suggestion would be to make the buffer
> size and kind of buffer (direct, non-direct) values tunable.
I reduced it to 1K and reconfigurable, and added
SocketSessionConfig.setSessionReceiveBufferSize(...). :)
> 3. Client/Server CPU usage. The number of connections per second
> steadily keeps decreasing as the total connections increase. Client &
> Server both are at 100% CPU. That is what is weird. I'm aiming to
> isolate the problem. I'm currently a bit doubtful about the way the
> Connector works, but wont be able to point out anything concrete until
> more permutations & combinations are tried.
Hmm, interesting. This might be related with
processIdleSessions(...). Could you try to disable it?
> I'm expecting that MINA should be able to take 100K connections on the
> server. It cant go over 65K for the client due to lack of available
> port numbers.
Can we accept more than 64K connections on the server side?
<snip/>
Thanks for your consistent contribution! :)
Regards,
Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/
Re: [mina] Performance issues
Posted by Berin Loritsch <bl...@d-haven.org>.
Vinod Panicker wrote:
>>Thanks for the update. Its very encouraging.
>
>
> The real fun will be coming when I start with the throughput tests.
> That will definitely show the real power (or lack thereof) of NIO.
> Which gets me to the question, how can throughput be effectively
> measured?
>
> Guys, repeating the call for suggestions for the benchmark suite for
> MINA to make it as real-world a test as possible.
>
Throughput is always measured in however many tested units per time
measure. What is going to be a better test? What exactly are we trying
to test?
IMO, a world class server spends more effort handling scalability than
raw performance. IOW, as important as raw performance is, being able to
gracefully degrade performance is the most important part of the
picture. To me its not so important that we are limited to ~119MB/s
throughput (the theorhetical maximum of a 1000base-T card) but that the
clients connected have a fair share of that bandwidth.
I.e. at what point does the system start becoming unusable when the
connected clients are using their share of bandwidth.
In order to effectively test that, it is best to avoid the overhead of
serving up files. It would be better to have a simple PING/PONG style
of passing data trough. At that point we are testing the scalability of
the networking API, and not really instroducing a lot of other variables
to the equation. The client can then measure how long it takes for it
to both send the "PING" request and get a "PONG" response. The number
of these simple transactions per second become your throughput per client.
Once you have this data, you can then measure for fairness. Fairness
would be the percentage of the smallest throughput to the largest
throughput. IOW if there were 1000 transactions/s for the largest and
990 transactions/s for the smallest the fairness would be 99%.
Another useful metric would be at what point does the server get
saturated? If you have a 100mb/s network and 100,000 clients with 100%
fairness you would only be able to expect a max of 15.65 transactions
per second (4 bytes out/4 bytes back). If we are getting .01
transactions per second then the server has been saturated a while ago
and we have passed the practical limit.
Re: [mina] Performance issues
Posted by Alex Karasulu <ao...@bellsouth.net>.
Vinod Panicker wrote:
>On Thu, 24 Mar 2005 09:03:41 -0500, Berin Loritsch
><bl...@d-haven.org> wrote:
>-- snip --
>
>
>
<snip/>
>The real fun will be coming when I start with the throughput tests.
>That will definitely show the real power (or lack thereof) of NIO.
>Which gets me to the question, how can throughput be effectively
>measured?
>
>
Yes that will be fun. Well throughput will be application specific
based on the nature of the server. You're testing obviously is centered
around MINA and NIO limits. This boils down to how much data the server
over all can consume and how much it can pump out as we vary the number
of concurrent connections.
(1) Avoid tests with servers that access disk
Minimizing the number of unrelated variables are key, like for example
disk access. I'd recommend testing the echo server which does not hit
disk like say an LDAP server would.
(2) Turn off swap
I would also making sure swapping does not come into the picture. The
tests should be designed not to consume physical memory. Perhaps even
disabling swap might be good idea. Just have the tested apps stop when
the machine runs out of physical memory so you don't get tainted data
with swapping.
(3) Disable as man applications and services as possible
Prevent resource utilization by other running processes. You want to
minimize the overall CPU and memory taken up by other processes. Try
printing out stats to a buffer instead of a log file and dumping them at
the end of the test.
(4) Measure throughput as data in/out per second (or even per 100
millisecond) samples per avg number of connections
There are a couple options here. You can use Ethereal to collect the
information or the server itself with filters to gather your stats into
a buffer. Again the stat buffer should be dumped to a file at the end
of the test (perhaps in a shutdown hook in case you get out of memory
JVM crashes). This however might not give you the kind of resolution
you're looking for.
I would have a MINA filter measure per second samples on the amount of
data going in and coming out. Within MINA this can be correlated with
the number of average concurrent connections for each sample.
These samples can then be used to construct the throughput graph where
total IO per sample is ploted against the avg concurrent connections per
sample on the x-axis.
>Guys, repeating the call for suggestions for the benchmark suite for
>MINA to make it as real-world a test as possible.
>
>
>
(5) [optional] Use a good bell curve distribution for message sizes
It will be fun to generate several throughput curves using constant
message sizes. This might tell us more about how we can deal with
buffer sizes within MINA. This is not very realistic but helps
understand some sweet spots in the MINA implementation. After you run a
few tests using constant message sizes to compare the resultant curves
it might be neat to see how things change when the message sizes are
varied with a realistic distribution. A series of plots can be
generated where the size distribution's maxima is displaced around the
buffer size. This may tell us more about real world dynamics.
Final Notes
These code (which hopefully will become tools) for capturing throughput
will be very useful for analyzing the nature of different servers and
configurations. I think server implementors using MINA will certainly
benefit from them rather than tuning their servers in the dark.
This is really great work you're doing Vinod. Hope some of these
suggestions help and that you're not dismayed by the amount of traffic -
in particular around your research :). We're all very interested in
your results.
Thanks,
-- Alex
Re: [mina] Performance issues
Posted by Vinod Panicker <vi...@gmail.com>.
On Thu, 24 Mar 2005 09:03:41 -0500, Berin Loritsch
<bl...@d-haven.org> wrote:
-- snip --
> > 1. No backlog support currently provided with IoProtocolAcceptor
>
> This is something that the blocking sockets do for us. Essentially it
> amounts to just not responding to a socket until we are done with
> another. There should also be a timeout to simply kill the connection
> if it has been in the backlog too long.
I doubt we can configure a timeout. Dunno if its there in TCP - maybe I forget
-- snip --
> > 3. Client/Server CPU usage. The number of connections per second
> > steadily keeps decreasing as the total connections increase. Client &
> > Server both are at 100% CPU. That is what is weird. I'm aiming to
> > isolate the problem. I'm currently a bit doubtful about the way the
> > Connector works, but wont be able to point out anything concrete until
> > more permutations & combinations are tried.
>
> A couple of thoughts here. First it is normal for the number of
> connections per second to decrease as the total connections increase.
> The key here is how quickly it decreases. Ideally we would have a
> smooth taper off of connections per second instead of an abrupt drop.
> That means the system scales well. Secondly, the 100% CPU cycles can be
> a symptom of how you are running your loops. I have found that calling
> Thread.yeild() will sometimes allow another thread to run, and sometimes
> not--on the same system. No matter what, the processor is still pegged.
> The next question to ask is how often do we need to poll, really? I
> find that even the process of forcing the loop to sleep 1 millisecond
> between iterations will greatly reduce the load on the processor. In
> the life of connection times the loss of a millisecond is nothing, the
> the reduction in stress on the CPU is tremendous. It helps the CPU stay
> busy with real work. Same thing with game loops. If you draw a new
> screen each iteration you might have 200+ FPS but your processor will be
> busy doing work that you will never truly appreciate.
Due to this problem, I've stopped using the MINA based client of mine.
I'm using a normal blocking IO client right now and it doesnt exhibit
this behaviour. On Monday, will be coding a NIO based single-threaded
client that will be a more effective comparison with MINA. Am aware
of all the points you have stated.
> > I'm expecting that MINA should be able to take 100K connections on the
> > server. It cant go over 65K for the client due to lack of available
> > port numbers.
>
> Thats where multiple clients come in to play. Quick question, are you
> checking the number of sustained client connections or the number of
> connection requests?
Yes of course. If you see my latest update, the sustained (concurrent
connections) stayed around 85K, whereas the total number of
connections accepted have even gone up to 95K. And yes, multiple
clients are being used for this due to the 64K limit.
> > When I ran a test for file descriptors, the Linux/FreeBSD boxes could
> > easily allocate up to 262144 FD's (I stopped testing at this number).
> > The Windows box gets limited to a bit over 100K - maybe I can get
> > around that as well.
>
> I'm not sure how you are handling threading, but on normal user accounts
> with Linux there is a limit of 500 threads/processes total for that user.
Can be upped to ridiculous figures :) and have been done. Also, I
remembered that the Windows box fd limit had not been tested. Will do
that on Monday too. But I've seen 100K+ file handles being used on my
box when I was running the initial tests on the client.
> Thanks for the update. Its very encouraging.
The real fun will be coming when I start with the throughput tests.
That will definitely show the real power (or lack thereof) of NIO.
Which gets me to the question, how can throughput be effectively
measured?
Guys, repeating the call for suggestions for the benchmark suite for
MINA to make it as real-world a test as possible.
Regards,
Vinod.
Re: [mina] Performance issues
Posted by Alex Karasulu <ao...@bellsouth.net>.
Berin Loritsch wrote:
> Vinod Panicker wrote:
>
>>
<snip/>
>> 3. Client/Server CPU usage. The number of connections per second
>> steadily keeps decreasing as the total connections increase. Client &
>> Server both are at 100% CPU. That is what is weird. I'm aiming to
>> isolate the problem. I'm currently a bit doubtful about the way the
>> Connector works, but wont be able to point out anything concrete until
>> more permutations & combinations are tried.
>
>
> A couple of thoughts here. First it is normal for the number of
> connections per second to decrease as the total connections increase.
> The key here is how quickly it decreases. Ideally we would have a
> smooth taper off of connections per second instead of an abrupt drop.
> That means the system scales well. Secondly, the 100% CPU cycles can
> be a symptom of how you are running your loops. I have found that
> calling Thread.yeild() will sometimes allow another thread to run, and
> sometimes not--on the same system. No matter what, the processor is
> still pegged. The next question to ask is how often do we need to
> poll, really? I find that even the process of forcing the loop to
> sleep 1 millisecond between iterations will greatly reduce the load on
> the processor. In the life of connection times the loss of a
> millisecond is nothing, the the reduction in stress on the CPU is
> tremendous. It helps the CPU stay busy with real work. Same thing
> with game loops. If you draw a new screen each iteration you might
> have 200+ FPS but your processor will be busy doing work that you will
> never truly appreciate.
Hmm it might help to get an idea of how much effort the CPU is expending
while context switching verses actually executing code in a thread (work
being done). You've made some good points Berin on giving the CPU some
rest while looping. It would be nice to have some raw statistics to
graph the decay with greater concurrency and show these stats on the
MINA pages. If someone provides raw data I can gnuplot it and make it
pretty.
It took me a while too but I've become comfortable taking a back seat to
Trustin. His work here is first class. It was a no brainer tossing apseda.
Alex
Re: [mina] Performance issues
Posted by Berin Loritsch <bl...@d-haven.org>.
Vinod Panicker wrote:
> Guys, please remember that there is absolutely no functionality being
> undertaken by the client and the server apart from just maintaining
> connected sockets. So in real-life performance, these numbers wont
> make much difference.
Yep. That will make a huge difference.
>
> Alright... Latest stats are as follows -
>
> Server - 30K concurrent connections on a Linux box with 512 MB RAM.
> Ran out of Direct Buffer Memory (more on this later)
> Client - 30K concurrent connections from a single Windows 2003 server
> box 1 GB RAM (no problems)
Ok, let me wipe the drool out of the corner of my mouth. Trustin, you
win! We can probably wipe the last remnants of the stuff I started way
back in incubator days. I'm looking at some other fish right now (one
of them is a pretty ambitious catch), so I won't be able to finish what
I started anyway.
>
> Alright, problem areas -
>
> 1. No backlog support currently provided with IoProtocolAcceptor
This is something that the blocking sockets do for us. Essentially it
amounts to just not responding to a socket until we are done with
another. There should also be a timeout to simply kill the connection
if it has been in the backlog too long.
>
> 2. When the server accepts a connection, 8K of Direct Buffer Memory is
> being allocated. I suspect this is too much. Depending on where MINA
> is used, this memory should be tunable. From what I saw in the code,
> this is for the read buffer. Taking an example of a Directory Server
> such as ApacheDS, I doubt incoming data is going to fill the 8K
> buffer. And since its direct buffer, more care should be taken to use
> up this kind of memory. If MINA is being used for an FTP server, this
> figure would make sense. My suggestion would be to make the buffer
> size and kind of buffer (direct, non-direct) values tunable.
Agreed. 8K is two pages of memory on x86 systems. One page should be
plenty for most cases. And I have a feeling that the system will
reserve a whole page even if we are only using 1/4 of it. I have no
tests to prove that though.
>
> 3. Client/Server CPU usage. The number of connections per second
> steadily keeps decreasing as the total connections increase. Client &
> Server both are at 100% CPU. That is what is weird. I'm aiming to
> isolate the problem. I'm currently a bit doubtful about the way the
> Connector works, but wont be able to point out anything concrete until
> more permutations & combinations are tried.
A couple of thoughts here. First it is normal for the number of
connections per second to decrease as the total connections increase.
The key here is how quickly it decreases. Ideally we would have a
smooth taper off of connections per second instead of an abrupt drop.
That means the system scales well. Secondly, the 100% CPU cycles can be
a symptom of how you are running your loops. I have found that calling
Thread.yeild() will sometimes allow another thread to run, and sometimes
not--on the same system. No matter what, the processor is still pegged.
The next question to ask is how often do we need to poll, really? I
find that even the process of forcing the loop to sleep 1 millisecond
between iterations will greatly reduce the load on the processor. In
the life of connection times the loss of a millisecond is nothing, the
the reduction in stress on the CPU is tremendous. It helps the CPU stay
busy with real work. Same thing with game loops. If you draw a new
screen each iteration you might have 200+ FPS but your processor will be
busy doing work that you will never truly appreciate.
>
> I'm expecting that MINA should be able to take 100K connections on the
> server. It cant go over 65K for the client due to lack of available
> port numbers.
Thats where multiple clients come in to play. Quick question, are you
checking the number of sustained client connections or the number of
connection requests?
>
> Currently there are no OS/Kernel issues that should prevent this -
> hopefully I can solve them if I hit them.
>
> When I ran a test for file descriptors, the Linux/FreeBSD boxes could
> easily allocate up to 262144 FD's (I stopped testing at this number).
> The Windows box gets limited to a bit over 100K - maybe I can get
> around that as well.
I'm not sure how you are handling threading, but on normal user accounts
with Linux there is a limit of 500 threads/processes total for that user.
>
> Once I hit the target of 100K concurrent connections, I'll be starting
> with IO on the concurrent connections to measure throughput values.
>
> There have been benchmarks put up that say that a
> thread-per-connection model is much better than NIO for throughput and
> performance (http://www.theserverside.com/discussions/thread.tss?thread_id=26700)
> but the point I'm trying to make is that the level of scalability the
> NIO model can provide can never be provided by a thead-per-connection
> model.
Yep, and sometimes scalability is far more important than raw
performance. The best analogy I can come up with is that of the
transition from traditional manufacturing to assembly line
manufacturing. You might be able to build one car in less time the
traditional way, but you can't build the same number of cars per hour.
>
> Phew.. long mail.. And since it seems that all you guys are in a way
> different time zone, more long mails are likely to come by (I'm in
> IST).
Thanks for the update. Its very encouraging.
Re: [mina] Performance issues
Posted by Vinod Panicker <vi...@gmail.com>.
Phew.. lots of replies.. thought it would be better to reply in a
consolidated mail instead of "carpet bombing" the list.
Guys, please remember that there is absolutely no functionality being
undertaken by the client and the server apart from just maintaining
connected sockets. So in real-life performance, these numbers wont
make much difference.
Alright... Latest stats are as follows -
Server - 30K concurrent connections on a Linux box with 512 MB RAM.
Ran out of Direct Buffer Memory (more on this later)
Client - 30K concurrent connections from a single Windows 2003 server
box 1 GB RAM (no problems)
Basically the client is going full throttle with connect requests.
Obviously we will run benchmarks with more sedate clients that
occasionally cause load spikes.
I agree with what Berin says, and every Internet server should know
its limits. Thats where these benchmarks help - we can estimate what
kind of load is MAX, under which the system can still provide expected
performance levels.
Alex, there's still lot more to do. Will compile all of this into
something that Trustin can put someplace. I'm also tinkering around
with JVMStat to obtain exact figures. Regarding the data limits per
partition, I wont be able to help out over there (I think you should
be looking at external storage for such kind of benchmarks)
Alright, problem areas -
1. No backlog support currently provided with IoProtocolAcceptor
2. When the server accepts a connection, 8K of Direct Buffer Memory is
being allocated. I suspect this is too much. Depending on where MINA
is used, this memory should be tunable. From what I saw in the code,
this is for the read buffer. Taking an example of a Directory Server
such as ApacheDS, I doubt incoming data is going to fill the 8K
buffer. And since its direct buffer, more care should be taken to use
up this kind of memory. If MINA is being used for an FTP server, this
figure would make sense. My suggestion would be to make the buffer
size and kind of buffer (direct, non-direct) values tunable.
3. Client/Server CPU usage. The number of connections per second
steadily keeps decreasing as the total connections increase. Client &
Server both are at 100% CPU. That is what is weird. I'm aiming to
isolate the problem. I'm currently a bit doubtful about the way the
Connector works, but wont be able to point out anything concrete until
more permutations & combinations are tried.
I'm expecting that MINA should be able to take 100K connections on the
server. It cant go over 65K for the client due to lack of available
port numbers.
Currently there are no OS/Kernel issues that should prevent this -
hopefully I can solve them if I hit them.
When I ran a test for file descriptors, the Linux/FreeBSD boxes could
easily allocate up to 262144 FD's (I stopped testing at this number).
The Windows box gets limited to a bit over 100K - maybe I can get
around that as well.
Once I hit the target of 100K concurrent connections, I'll be starting
with IO on the concurrent connections to measure throughput values.
There have been benchmarks put up that say that a
thread-per-connection model is much better than NIO for throughput and
performance (http://www.theserverside.com/discussions/thread.tss?thread_id=26700)
but the point I'm trying to make is that the level of scalability the
NIO model can provide can never be provided by a thead-per-connection
model.
Phew.. long mail.. And since it seems that all you guys are in a way
different time zone, more long mails are likely to come by (I'm in
IST).
Regards,
Vinod.
-- snipped * --
Re: [mina] Performance issues
Posted by Jan Andersson <ja...@minq.se>.
> On 2005-03-23, at 15:59, Trustin Lee wrote:
>
> Hi,
>
> On Wed, 23 Mar 2005 15:05:55 +0100, Jan Andersson <ja...@minq.se>
> wrote:
>> Hi,
>>
>>> [...]
>> When running out of direct buffer memory, try using the
>> -XX:MaxDirectMemorySize
>> option. For example:
>>
>> -XX:MaxDirectMemorySize=128M
>
> Thanks for pointing out a good point. I remember you implemented your
> byte buffer pool to allocate non-direct buffer when OutOfMemoryError
> is caught. Would it be OK to modify byte buffer pool like you did?
> WDYT?
Yes, I think this is a good idea.
But I also beleive that you should write some kind of warning
when switching to non-direct buffer...
/Janne
Re: [mina] Performance issues
Posted by Alex Karasulu <ao...@bellsouth.net>.
Berin Loritsch wrote:
<snip/>
> Can I play devil's advocate here?
>
> Handling over 100 simultaneous connections is good (Tomcat doesn't
> even do that out of the box). 8,000 simultaneous connections is
> incredible for an all Java solution.
Really Tomcat could then gain by using MINA.
> So I guess the big thing here is to know when to say "when". If you
> know that 8,000 simultaneous connections is your limit, then don't
> accept any more. Sure we can squeeze out a few more, and we can play
> around with that. However, to have a robust server we need to be able
> to avoid being in a situation where a DOS attack can kill the server.
> Once you limit the number of connections we deny any more coming in.
Very good points however I think these fellas are having a good time
testing the limits and seeing for sure how far they can push MINA.
We're going to better off for it even if we're not going to get over 8K
concurrent connections.
> Last, but certainly not least, is there any way that we can have
> multiple instances of the system up so that a smart router can send
> connections to the different instances--allowing the system to scale
> up using more machines?
This is a great question to ask. I think this is the begining of some
P2P convos we can have.
I have always dreamed of building an index mesh for LDAP which could
benefit from this. The idea is different servers have different sets of
attributes indexed. When a query comes in and the present server
instance (DSA) does not have the indices to do a fast search it hands it
off to a server that can do the search faster than a full table scan.
That would need some kind of P2P communication for index discovery.
Regardless I have other perhaps more relavent ideas where P2P can come
in handy.
> I think it is stellar that MINA is performing as well as it is.
That's a big +1. The Trustinator will never give up on making his stuff
better ;).
Alex
Re: [mina] Performance issues
Posted by Berin Loritsch <bl...@d-haven.org>.
Trustin Lee wrote:
> Hi,
>
> On Wed, 23 Mar 2005 15:05:55 +0100, Jan Andersson <ja...@minq.se> wrote:
>
>>Hi,
>>
>>
>>>The Connector seems to be working much better now. Memory consumption
>>>is still an issue with the server running out of memory at around 8000
>>>concurrent connections.
>>>
Can I play devil's advocate here?
Handling over 100 simultaneous connections is good (Tomcat doesn't even
do that out of the box). 8,000 simultaneous connections is incredible
for an all Java solution.
So I guess the big thing here is to know when to say "when". If you
know that 8,000 simultaneous connections is your limit, then don't
accept any more. Sure we can squeeze out a few more, and we can play
around with that. However, to have a robust server we need to be able
to avoid being in a situation where a DOS attack can kill the server.
Once you limit the number of connections we deny any more coming in.
Last, but certainly not least, is there any way that we can have
multiple instances of the system up so that a smart router can send
connections to the different instances--allowing the system to scale up
using more machines?
I think it is stellar that MINA is performing as well as it is.
Re: [mina] Performance issues
Posted by Trustin Lee <tr...@gmail.com>.
Hi,
On Wed, 23 Mar 2005 15:05:55 +0100, Jan Andersson <ja...@minq.se> wrote:
> Hi,
>
> > The Connector seems to be working much better now. Memory consumption
> > is still an issue with the server running out of memory at around 8000
> > concurrent connections.
> >
> > Here's the stack trace -
> >
> > Exception in thread "SocketAcceptor-0" java.lang.OutOfMemoryError:
> > Direct buffer memory
> > at java.nio.Bits.reserveMemory(Unknown Source)
> > at java.nio.DirectByteBuffer.<init>(Unknown Source)
> > at java.nio.ByteBuffer.allocateDirect(Unknown Source)
> > at
> > org.apache.mina.common.ByteBuffer.allocate(ByteBuffer.java:95)
> > at
> > org.apache.mina.io.socket.SocketSession.<init>(SocketSession.java:94)
> > at
> > org.apache.mina.io.socket.SocketAcceptor$Worker.run(SocketAcceptor.java
> > :227)
> >
> >
> > This was using default JVM settings till now. Tinkering out with
> > different heap sizes now.
> >
> > I'm using JDK 1.5.0_01
>
> When running out of direct buffer memory, try using the
> -XX:MaxDirectMemorySize
> option. For example:
>
> -XX:MaxDirectMemorySize=128M
Thanks for pointing out a good point. I remember you implemented your
byte buffer pool to allocate non-direct buffer when OutOfMemoryError
is caught. Would it be OK to modify byte buffer pool like you did?
WDYT?
Thanks,
Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/
Re: [mina] Performance issues
Posted by Jan Andersson <ja...@minq.se>.
Hi,
> The Connector seems to be working much better now. Memory consumption
> is still an issue with the server running out of memory at around 8000
> concurrent connections.
>
> Here's the stack trace -
>
> Exception in thread "SocketAcceptor-0" java.lang.OutOfMemoryError:
> Direct buffer memory
> at java.nio.Bits.reserveMemory(Unknown Source)
> at java.nio.DirectByteBuffer.<init>(Unknown Source)
> at java.nio.ByteBuffer.allocateDirect(Unknown Source)
> at
> org.apache.mina.common.ByteBuffer.allocate(ByteBuffer.java:95)
> at
> org.apache.mina.io.socket.SocketSession.<init>(SocketSession.java:94)
> at
> org.apache.mina.io.socket.SocketAcceptor$Worker.run(SocketAcceptor.java
> :227)
>
>
> This was using default JVM settings till now. Tinkering out with
> different heap sizes now.
>
> I'm using JDK 1.5.0_01
When running out of direct buffer memory, try using the
-XX:MaxDirectMemorySize
option. For example:
-XX:MaxDirectMemorySize=128M
/Janne
Re: [mina] Performance issues
Posted by Vinod Panicker <vi...@gmail.com>.
Hi,
The Connector seems to be working much better now. Memory consumption
is still an issue with the server running out of memory at around 8000
concurrent connections.
Here's the stack trace -
Exception in thread "SocketAcceptor-0" java.lang.OutOfMemoryError:
Direct buffer memory
at java.nio.Bits.reserveMemory(Unknown Source)
at java.nio.DirectByteBuffer.<init>(Unknown Source)
at java.nio.ByteBuffer.allocateDirect(Unknown Source)
at org.apache.mina.common.ByteBuffer.allocate(ByteBuffer.java:95)
at org.apache.mina.io.socket.SocketSession.<init>(SocketSession.java:94)
at org.apache.mina.io.socket.SocketAcceptor$Worker.run(SocketAcceptor.java:227)
This was using default JVM settings till now. Tinkering out with
different heap sizes now.
I'm using JDK 1.5.0_01
Regards,
Vinod.
On Wed, 23 Mar 2005 10:29:03 +0530, Vinod Panicker <vi...@gmail.com> wrote:
> Hi Trustin,
>
> Thanks for the fix, checking it out now.
>
>
> On Wed, 23 Mar 2005 00:03:29 +0900, Trustin Lee <tr...@gmail.com> wrote:
> > Hi Vinod,
> >
> > > It seems like the issue is in the Connector. When I used a non-mina
> > > based client running in a single thread to connect to the mina based
> > > server, it works fine.
> >
> > I found a serious flaw that will degrade SocketConnector performance.
> > I checked in the fix into trunk. Could you retry?
> >
> > > The server runs out of memory after a bit over 8000 connections.
> >
> > What was max heap size? Does this mean we have to lessen MINA's
> > memory consumption?
>
> Actually rather than looking at it from that perspective, i'd look at
> is as "how much memory should be consumed for maintaining a single
> connected socket"
>
> IMO, on a "standard" linux box, number of connected sockets supported
> should be around 50K or even more.
>
> >
> > <snip/>
> >
> > Thanks,
> > Trustin.
> > --
> > what we call human nature is actually human habit
> > --
> > http://gleamynode.net/
> >
>
> Regards,
> Vinod.
>
Re: [mina] Performance issues
Posted by Vinod Panicker <vi...@gmail.com>.
Hi Trustin,
Thanks for the fix, checking it out now.
On Wed, 23 Mar 2005 00:03:29 +0900, Trustin Lee <tr...@gmail.com> wrote:
> Hi Vinod,
>
> > It seems like the issue is in the Connector. When I used a non-mina
> > based client running in a single thread to connect to the mina based
> > server, it works fine.
>
> I found a serious flaw that will degrade SocketConnector performance.
> I checked in the fix into trunk. Could you retry?
>
> > The server runs out of memory after a bit over 8000 connections.
>
> What was max heap size? Does this mean we have to lessen MINA's
> memory consumption?
Actually rather than looking at it from that perspective, i'd look at
is as "how much memory should be consumed for maintaining a single
connected socket"
IMO, on a "standard" linux box, number of connected sockets supported
should be around 50K or even more.
>
> <snip/>
>
> Thanks,
> Trustin.
> --
> what we call human nature is actually human habit
> --
> http://gleamynode.net/
>
Regards,
Vinod.
Re: [mina] Performance issues
Posted by Trustin Lee <tr...@gmail.com>.
Hi Vinod,
> It seems like the issue is in the Connector. When I used a non-mina
> based client running in a single thread to connect to the mina based
> server, it works fine.
I found a serious flaw that will degrade SocketConnector performance.
I checked in the fix into trunk. Could you retry?
> The server runs out of memory after a bit over 8000 connections.
What was max heap size? Does this mean we have to lessen MINA's
memory consumption?
<snip/>
Thanks,
Trustin.
--
what we call human nature is actually human habit
--
http://gleamynode.net/
Re: [mina] Performance issues
Posted by Vinod Panicker <vi...@gmail.com>.
Hi,
On Tue, 22 Mar 2005 15:50:19 +0900, Trustin Lee <tr...@gmail.com> wrote:
> Hi,
>
> On Tue, 22 Mar 2005 10:48:52 +0530, Vinod Panicker <vi...@gmail.com> wrote:
> > Hi,
> >
> > I was just trying to do some tests using mina. I've got a server and
> > a client, both of which are using mina and the thread pool filter.
> >
> > The server and the client have been run on multiple OS's - both
> > Windows (2000 & 2003) and Linux (Fedora Core 3).
> >
> > Max fd limits have been upped.
> >
> > The server uses the IoProtocolAcceptor class. The client uses the
> > SocketConnector class.
> >
> > The client application is basically just connecting to the server and
> > maintaining a concurrent connection.
> >
> > When the client app is run, and told to establish 2000 connections to
> > the server for eg, the connection rate suddenly (no pattern observed)
> > drops to about 1 connection per second. This has been tried on all
> > the OS's mentioned above. The server is apparently not the cause of
> > this issue because when a different instance of the client is started
> > (on the same machine, even) the connection rate remains normal. This
> > behaviour is not consistent, though it happens on every other run of
> > the client.
> >
> > There are no sleep()'s in the client / server code.
> >
> > Any pointers on what is happening?
> >
> > Also, is there a way using which the server connection accept backlog
> > (i forget the exact term) can be set?
>
> There is a 'bind()' method with backlog parameter in SocketAcceptor.
> IoAcceptor doesn't have it, so you'll have to down-cast it.
In the code, the IoProtocolAcceptor class is being used for the
server, which doesnt have a bind that has the functionality of
configuring a backlog. Also, since this class is not derived from
SocketAcceptor, no question of downcasting arises.
>
> > PS: This same test has been carried out using our own code (non-mina)
> > and works perfectly on all the above OS's (over 20k connections on the
> > server)
>
> I don't have any test machines to do that massive test. Can that kind
> of test be reproducable in two PCs? If so, I'll test by myself and
> figure out why.
Its not a massive test. The Linux machine i'm using has got 512 MB
RAM. The Windows box has 1 GB. You should be able to run both client
and server on the same machine without any issues, on a machine with
even 256 MB RAM IMO.
Do you want me to send you the source files so you can test it out?
>
> Regards,
> Trustin.
> --
> what we call human nature is actually human habit
> --
> http://gleamynode.net/
>
Regards,
Vinod.
Re: [mina] Performance issues
Posted by Trustin Lee <tr...@gmail.com>.
Hi,
On Tue, 22 Mar 2005 10:48:52 +0530, Vinod Panicker <vi...@gmail.com> wrote:
> Hi,
>
> I was just trying to do some tests using mina. I've got a server and
> a client, both of which are using mina and the thread pool filter.
>
> The server and the client have been run on multiple OS's - both
> Windows (2000 & 2003) and Linux (Fedora Core 3).
>
> Max fd limits have been upped.
>
> The server uses the IoProtocolAcceptor class. The client uses the
> SocketConnector class.
>
> The client application is basically just connecting to the server and
> maintaining a concurrent connection.
>
> When the client app is run, and told to establish 2000 connections to
> the server for eg, the connection rate suddenly (no pattern observed)
> drops to about 1 connection per second. This has been tried on all
> the OS's mentioned above. The server is apparently not the cause of
> this issue because when a different instance of the client is started
> (on the same machine, even) the connection rate remains normal. This
> behaviour is not consistent, though it happens on every other run of
> the client.
>
> There are no sleep()'s in the client / server code.
>
> Any pointers on what is happening?
>
> Also, is there a way using which the server connection accept backlog
> (i forget the exact term) can be set?
There is a 'bind()' method with backlog parameter in SocketAcceptor.
IoAcceptor doesn't have it, so you'll have to down-cast it.
> PS: This same test has been carried out using our own code (non-mina)
> and works perfectly on all the above OS's (over 20k connections on the
> server)
I don't have any test machines to do that massive test. Can that kind
of test be reproducable in two PCs? If so, I'll test by myself and
figure out why.
Regards,
Trustin.
--
what we call human nature is actually human habit
--
http://gleamynode.net/