You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@mina.apache.org by Johan Haleby <jo...@gmail.com> on 2008/08/15 10:10:41 UTC

Help with performance issues

Hi,

I've basically tried to ask this question before (
http://mina.markmail.org/message/bjpiogngdaxf3fyc?q=Buffering+question) but
unfortunately I didn't manage to get much further. So I'll try to refrain my
question a little bit in the hope to get some more answers.

I'm using Mina to broadcast lots of incoming data from external hardware
devices to connected clients. We need a write-throughput of at least 40
Mbit/s and right now we're reaching _at most_ about 10MBits/s (but usually
it's just around 3Mbit/s). The data is received by the Mina server as binary
data but is converted into strings (the data is actually utf-8 encoded
strings). When a new line has been detected, some meta-data is added to the
detected row and the whole row is persisted to disk as well as being sent to
all clients registered to receive data from this device. For each line one
or more regular expression filters are applied (since the clients are only
interested in certain data). Right now I'm using session.write(rowData) for
each new line for each client who's registered as interested. But as soon as
we get many clients (> 30 or so) the performance is declining. In some tests
we've done it takes several minutes for the clients to receive all data
after the hardware device has been shutdown. For encoding I'm using the
TextLineEncoder and I suspect I may be suffering from the same issue as in
http://mina.markmail.org/message/5xtmwbode6cgifox?q=performance+file("[TextLineEncoder]
create a bunch of (small) IoBuffer objects for each
individual line (or message object) which are then put into the writer's
queue. This is quite inefficient since thousand of small IoBuffer objects
will be sent over the wire instead of creating one buffer with a more
appropriate buffer size"). I've tried using the mergeAll method but I don't
see much difference, but mabey I'm not using it correctly. I've also taken a
look at the Mina FTP server project. Here the data channels are made up of
input and outputstreams stored in the session. Perhaps I'm better of using
this approach for sending the data instead of going through the
TextLineEncoder? I'm using Mina 1.1.7.

What do you think?

/Johan

Re: Help with performance issues

Posted by Johan Haleby <jo...@gmail.com>.
Sorry I've been on vacation last week.

Yes I've used a profiler (I've attached the result to this message as well)
and it seems to me like the application spends most time in the doFlush()
method in the SocketIoProcessor class when calling
ByteBufferProxy.hasRemaining().

/Johan



On Fri, Aug 15, 2008 at 11:29 AM, Emmanuel Lecharny <el...@gmail.com>wrote:

> Hi,
>
> have you used a profiler to see if you have some bottleneck, and if so,
> where ?
>
>
> Johan Haleby wrote:
>
>> Hi,
>>
>> I've basically tried to ask this question before (
>> http://mina.markmail.org/message/bjpiogngdaxf3fyc?q=Buffering+question)
>> but
>> unfortunately I didn't manage to get much further. So I'll try to refrain
>> my
>> question a little bit in the hope to get some more answers.
>>
>> I'm using Mina to broadcast lots of incoming data from external hardware
>> devices to connected clients. We need a write-throughput of at least 40
>> Mbit/s and right now we're reaching _at most_ about 10MBits/s (but usually
>> it's just around 3Mbit/s). The data is received by the Mina server as
>> binary
>> data but is converted into strings (the data is actually utf-8 encoded
>> strings). When a new line has been detected, some meta-data is added to
>> the
>> detected row and the whole row is persisted to disk as well as being sent
>> to
>> all clients registered to receive data from this device. For each line one
>> or more regular expression filters are applied (since the clients are only
>> interested in certain data). Right now I'm using session.write(rowData)
>> for
>> each new line for each client who's registered as interested. But as soon
>> as
>> we get many clients (> 30 or so) the performance is declining. In some
>> tests
>> we've done it takes several minutes for the clients to receive all data
>> after the hardware device has been shutdown. For encoding I'm using the
>> TextLineEncoder and I suspect I may be suffering from the same issue as in
>> http://mina.markmail.org/message/5xtmwbode6cgifox?q=performance+file(<http://mina.markmail.org/message/5xtmwbode6cgifox?q=performance+file%28>
>> "[TextLineEncoder]
>> create a bunch of (small) IoBuffer objects for each
>> individual line (or message object) which are then put into the writer's
>> queue. This is quite inefficient since thousand of small IoBuffer objects
>> will be sent over the wire instead of creating one buffer with a more
>> appropriate buffer size"). I've tried using the mergeAll method but I
>> don't
>> see much difference, but mabey I'm not using it correctly. I've also taken
>> a
>> look at the Mina FTP server project. Here the data channels are made up of
>> input and outputstreams stored in the session. Perhaps I'm better of using
>> this approach for sending the data instead of going through the
>> TextLineEncoder? I'm using Mina 1.1.7.
>>
>> What do you think?
>>
>> /Johan
>>
>>
>>
>
>
> --
> --
> cordialement, regards,
> Emmanuel Lécharny
> www.iktek.com
> directory.apache.org
>
>
>

Re: Help with performance issues

Posted by Emmanuel Lecharny <el...@gmail.com>.
Hi,

have you used a profiler to see if you have some bottleneck, and if so, 
where ?

Johan Haleby wrote:
> Hi,
>
> I've basically tried to ask this question before (
> http://mina.markmail.org/message/bjpiogngdaxf3fyc?q=Buffering+question) but
> unfortunately I didn't manage to get much further. So I'll try to refrain my
> question a little bit in the hope to get some more answers.
>
> I'm using Mina to broadcast lots of incoming data from external hardware
> devices to connected clients. We need a write-throughput of at least 40
> Mbit/s and right now we're reaching _at most_ about 10MBits/s (but usually
> it's just around 3Mbit/s). The data is received by the Mina server as binary
> data but is converted into strings (the data is actually utf-8 encoded
> strings). When a new line has been detected, some meta-data is added to the
> detected row and the whole row is persisted to disk as well as being sent to
> all clients registered to receive data from this device. For each line one
> or more regular expression filters are applied (since the clients are only
> interested in certain data). Right now I'm using session.write(rowData) for
> each new line for each client who's registered as interested. But as soon as
> we get many clients (> 30 or so) the performance is declining. In some tests
> we've done it takes several minutes for the clients to receive all data
> after the hardware device has been shutdown. For encoding I'm using the
> TextLineEncoder and I suspect I may be suffering from the same issue as in
> http://mina.markmail.org/message/5xtmwbode6cgifox?q=performance+file("[TextLineEncoder]
> create a bunch of (small) IoBuffer objects for each
> individual line (or message object) which are then put into the writer's
> queue. This is quite inefficient since thousand of small IoBuffer objects
> will be sent over the wire instead of creating one buffer with a more
> appropriate buffer size"). I've tried using the mergeAll method but I don't
> see much difference, but mabey I'm not using it correctly. I've also taken a
> look at the Mina FTP server project. Here the data channels are made up of
> input and outputstreams stored in the session. Perhaps I'm better of using
> this approach for sending the data instead of going through the
> TextLineEncoder? I'm using Mina 1.1.7.
>
> What do you think?
>
> /Johan
>
>   


-- 
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org



Re: Help with performance issues

Posted by Tuure Laurinolli <tu...@indagon.com>.
Johan Haleby wrote:
> The heap size doesn't increase until the data-producing device is shutdown,
> it fluctuates like it should do (after tweaking the GC). 
> The CPU usage goes
> up dangerously close to 100% pretty fast though.

This would seem to indicate that the amount of data buffered does not 
increase with time, ie. you can handle all the traffic produced by the 
device. What happens if do whatever transformations you do to the data, 
and then simply discard it instead of writing it to the recipient 
sessions? Can you try replacing reading from the device with some dummy 
data source with the goal of not using as much resources as actual 
reading, and thus potentially being able to find the bottleneck?

Re: Help with performance issues

Posted by Johan Haleby <jo...@gmail.com>.
The heap size doesn't increase until the data-producing device is shutdown,
it fluctuates like it should do (after tweaking the GC). The CPU usage goes
up dangerously close to 100% pretty fast though.

I've tried experimenting with the TCP_NO_DELAY settings after your tip,
enabling it (i.e. disabling Naggle's algorithm) gives me lower write
throughput but also lower CPU usage. I've not changed the default receive
buffer size (it's not an issue afaik) but I'm using a 64k as write buffer
size. I'll try wireshark to check if the buffers are sent in seperate
packets asap.

/Johan





On Fri, Aug 15, 2008 at 11:48 AM, Tuure Laurinolli <
tuure.laurinolli@indagon.com> wrote:

> Johan Haleby wrote:
>
>> I'm using Mina to broadcast lots of incoming data from external hardware
>> devices to connected clients.
>>
>
>  But as soon as
>> we get many clients (> 30 or so) the performance is declining. In some
>> tests
>> we've done it takes several minutes for the clients to receive all data
>> after the hardware device has been shutdown.
>>
>
> How is the program behaving? Is it CPU-bound? Does heap size increase until
> you shut down the data-producing device?
>
>  This is quite inefficient since thousand of small IoBuffer objects
>> will be sent over the wire instead of creating one buffer with a more
>> appropriate buffer size").
>>
>
> Have you checked if the buffers are actually sent in separate IP packets,
> or combined at some level into larger packets? Have you tuned send/receive
> buffer sizes? Does your socket have TCP_NO_DELAY set?
>
>

Re: Help with performance issues

Posted by Tuure Laurinolli <tu...@indagon.com>.
Johan Haleby wrote:
> I'm using Mina to broadcast lots of incoming data from external hardware
> devices to connected clients. 

> But as soon as
> we get many clients (> 30 or so) the performance is declining. In some tests
> we've done it takes several minutes for the clients to receive all data
> after the hardware device has been shutdown.

How is the program behaving? Is it CPU-bound? Does heap size increase 
until you shut down the data-producing device?

> This is quite inefficient since thousand of small IoBuffer objects
> will be sent over the wire instead of creating one buffer with a more
> appropriate buffer size"). 

Have you checked if the buffers are actually sent in separate IP 
packets, or combined at some level into larger packets? Have you tuned 
send/receive buffer sizes? Does your socket have TCP_NO_DELAY set?