You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@directory.apache.org by Joshua Portway <jo...@stain.org> on 2006/02/15 18:43:30 UTC
[mina] DIRMINA-165 revisited
Hi,
I've been looking at the code submitted for this bug (to allow the
creating of read-only buffers and buffer slices) and reading the
comments, and I'm not sure I understand why the submitted patch is
considered "dangerous". Trustin advises that all the methods assert
that every buffer involved is not pooled, but I'm not sure why this
should be - presumably, assuming that any derived buffers maintain a
reference to their parent buffer, the parent buffer won't be returned
to the pool until all derived buffers have been released, and
therefore released it in turn. Am I missing something here ?
josh
Re: [mina] Socket IO processor, thread problem.
Posted by fe...@gmail.com.
I was wrong using "recollect", I wanted to say:
"So, the non-written bytebuffers are no more put in a single big
bytebuffer before to send them to the SocketChannel?"
Because I remember a feature like that in the 0.8.0.
Tell me if I'm wrong, I think giving to the SocketChannel a big bytebuffer instead of many little buffers make it works better with the underling asynch IO. That's right?
I'm studing for exams now, as soon as I can I'll try to reproduce the strange cpu usage.
by Fedechicco
Fedechicco wrote:
Finally, I must to send many ( the usual number of 1000 ) little
pre-prepared bytebuffers, and I initially wrote them one-by-one, but my
tests went better (not good, but better) since I have done my own cicle that
put all the little buffers in one big buffer, and then write() only the big
bytebuffer.
So, the non-written bytebuffers are no more recollected in a single big
bytebuffer before to send them to the SocketChannel?
Trustin wrote:
What do you mean by 'recollected'? Does it mean 'pooled'? They (small buffers) are pooled if you call ByteBuffer.release() explicitly in case you didn't pass (write) them to MINA.
Re: [mina] Socket IO processor, thread problem.
Posted by Trustin Lee <tr...@gmail.com>.
Hi Fedechicco,
On 2/16/06, fedechicco@gmail.com <fe...@gmail.com> wrote:
>
> I tried to change mina.socket.processor property to a polite value of 10
> (on
> a mono processor, for an average of 1000 sockets), and it was like I
> hoped.
> Actually it is useless to increase the mina.socket.processor, on my mono
> CPU
> obviously.
> Now I tried to set false the SO_KEEP_ALIVE, I thought that on 1000
> sockets
> it can be an enormouse series of little useless packets that make the
> selector work.
> But I don't think that this can be the right solution.
I don't think so, either.
I'm not sure that the selector is the cause of the CPU leak, I only see on
> the profiler that the poll() method of selector take the 99% of the time,
> but I think that my profiler do not analyze the time spent waiting for
> data
> from the time spent in other activities.
> Said that I'm not sure that the fault is of selector, but all the other
> classes take a ridiculus part of the cpu time ( less than 1% ).
Why don't you count the calls? If the count of Selector.select() calls is
exceptionally high, it means that the selector is behaving wrong.
Can you reproduce this problem easily? I experienced this problem only in
Windows very rarely. I found that Selector.selector() returns immediately
returning '0' even though I didn't call any Selector.wakeup(); it was an
apparent JDK bug.
Please use the profiler to cound the number of calls. It will help us to
debug the problem. If it's easy to reproduce it by myself with one or two
commodity PCs, I can test it by myself and track down the problem.
I'd like having a profiler that can distinguish between wait() and used
> time, I'm using M10 integrated in NetBeans, somebody can advice me a
> better
> profiler?
YourKit profiler is the best profiler I have ever used, but I think it also
cannot distinguish it because the Selector implementation doesn't use
Object.wait() at all. You have to count the calls anyway.
Finally, I must to send many ( the usual number of 1000 ) little
> pre-prepared bytebuffers, and I initially wrote them one-by-one, but my
> tests went better (not good, but better) since I have done my own cicle
> that
> put all the little buffers in one big buffer, and then write() only the
> big
> bytebuffer.
> So, the non-written bytebuffers are no more recollected in a single big
> bytebuffer before to send them to the SocketChannel?
What do you mean by 'recollected'? Does it mean 'pooled'? They (small
buffers) are pooled if you call ByteBuffer.release() explicitly in case you
didn't pass (write) them to MINA.
HTH,
Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/
PGP Key ID: 0xFE954E7E
Re: [mina] Socket IO processor, thread problem.
Posted by fe...@gmail.com.
I tried to change mina.socket.processor property to a polite value of 10 (on
a mono processor, for an average of 1000 sockets), and it was like I hoped.
Actually it is useless to increase the mina.socket.processor, on my mono CPU
obviously.
Now I tried to set false the SO_KEEP_ALIVE, I thought that on 1000 sockets
it can be an enormouse series of little useless packets that make the
selector work.
But I don't think that this can be the right solution.
I'm not sure that the selector is the cause of the CPU leak, I only see on
the profiler that the poll() method of selector take the 99% of the time,
but I think that my profiler do not analyze the time spent waiting for data
from the time spent in other activities.
Said that I'm not sure that the fault is of selector, but all the other
classes take a ridiculus part of the cpu time ( less than 1% ).
I'd like having a profiler that can distinguish between wait() and used
time, I'm using M10 integrated in NetBeans, somebody can advice me a better
profiler?
It is a fact that with 850-900 sockets things go well, but with 1000 I reach
teh 100% usage in less than an hour, and then all freeze, on my linux.
Finally, I must to send many ( the usual number of 1000 ) little
pre-prepared bytebuffers, and I initially wrote them one-by-one, but my
tests went better (not good, but better) since I have done my own cicle that
put all the little buffers in one big buffer, and then write() only the big
bytebuffer.
So, the non-written bytebuffers are no more recollected in a single big
bytebuffer before to send them to the SocketChannel?
Re: [mina] Socket IO processor, thread problem.
Posted by Niklas Therning <ni...@trillian.se>.
fedechicco@gmail.com wrote:
> I have some problem with a simple application that does almost only IO
> on tcp sockets, it take me 100% CPU usage when I have 1000-1200
> connections.
> I have met this problem every time I use a single selector thread for
> many channels, and in my own old implementations I have divided the
> channels on different selector threads, and then all worked well.
>
> I thought that I was not able to do well the selector loops, and so I
> have not taken care about it, but now with mina 0.9.0 I have the same
> problem.
> I don't know, do you all have the same problem with many sockets making
> I/O?
>
> I will experiment with the "mina.socket.processor" property, but it will
> be strange if it works.
Have you tried tuning the number of threads in your thread pool
(ThreadPoolFilter)? Are you sure it's the selector thread which uses all
the cpu?
I haven't had any such issues but I haven't had that many active
connections either. Please keep us updated on how it works when using
the mina.socket.processor property.
/Niklas
[mina] Socket IO processor, thread problem.
Posted by fe...@gmail.com.
I have some problem with a simple application that does almost only IO on
tcp sockets, it take me 100% CPU usage when I have 1000-1200 connections.
I have met this problem every time I use a single selector thread for many
channels, and in my own old implementations I have divided the channels on
different selector threads, and then all worked well.
I thought that I was not able to do well the selector loops, and so I have
not taken care about it, but now with mina 0.9.0 I have the same problem.
I don't know, do you all have the same problem with many sockets making I/O?
I will experiment with the "mina.socket.processor" property, but it will be
strange if it works.
by Fed
Re: [mina] DIRMINA-165 revisited
Posted by Trustin Lee <tr...@gmail.com>.
On 2/16/06, joshua portway <jo...@stain.org> wrote:
>
> thanks for the warning - is anyone actively working on this issue ?
Not yet unfortunately. Any contribution is welcome. ;)
It's important to my project to be able to slice buffers ( actually,
> I have one permanent buffer, with permanent "slice" buffers pointing
> to bits of it and when the contents of the buffer change I send a
> load of network messages - one for each of the slice buffers - this
> avoids any copying or reallocating buffers ).
We understand your point clearly. We'll resolve this issue with careful
discussion.
If no one else is working on this I'll probably give it a go myself,
> but I would appreciate some advice about why it was considered a bad
> idea for view buffer "containers" (mina ByteBuffer) to be pooled -
> can anyone help me out here ?
It is because it is very hard to keep track of NIO buffers. It becomes very
error-prone for a user to release the buffer while its duplicate or slice is
being used by others. There should be a trade-off somewhere to achieve
duplication and slicing of buffers. Let's keep discussing on this issue in
the JIRA:
http://issues.apache.org/jira/browse/DIRMINA-165
Thanks,
Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/
--
PGP key fingerprints:
* E167 E6AF E73A CBCE EE41 4A29 544D DE48 FE95 4E7E
* B693 628E 6047 4F8F CFA4 455E 1C62 A7DC 0255 ECA6
Re: [mina] DIRMINA-165 revisited
Posted by Trustin Lee <tr...@gmail.com>.
On 2/16/06, joshua portway <jo...@stain.org> wrote:
>
> thanks for the warning - is anyone actively working on this issue ?
Not yet unfortunately. Any contribution is welcome. ;)
It's important to my project to be able to slice buffers ( actually,
> I have one permanent buffer, with permanent "slice" buffers pointing
> to bits of it and when the contents of the buffer change I send a
> load of network messages - one for each of the slice buffers - this
> avoids any copying or reallocating buffers ).
We understand your point clearly. We'll resolve this issue with careful
discussion.
If no one else is working on this I'll probably give it a go myself,
> but I would appreciate some advice about why it was considered a bad
> idea for view buffer "containers" (mina ByteBuffer) to be pooled -
> can anyone help me out here ?
It is because it is very hard to keep track of NIO buffers. It becomes very
error-prone for a user to release the buffer while its duplicate or slice is
being used by others. There should be a trade-off somewhere to achieve
duplication and slicing of buffers. Let's keep discussing on this issue in
the JIRA:
http://issues.apache.org/jira/browse/DIRMINA-165
Thanks,
Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/
--
PGP key fingerprints:
* E167 E6AF E73A CBCE EE41 4A29 544D DE48 FE95 4E7E
* B693 628E 6047 4F8F CFA4 455E 1C62 A7DC 0255 ECA6
Re: [mina] DIRMINA-165 revisited
Posted by joshua portway <jo...@stain.org>.
thanks for the warning - is anyone actively working on this issue ?
It's important to my project to be able to slice buffers ( actually,
I have one permanent buffer, with permanent "slice" buffers pointing
to bits of it and when the contents of the buffer change I send a
load of network messages - one for each of the slice buffers - this
avoids any copying or reallocating buffers ).
If no one else is working on this I'll probably give it a go myself,
but I would appreciate some advice about why it was considered a bad
idea for view buffer "containers" (mina ByteBuffer) to be pooled -
can anyone help me out here ?
josh
On 15 Feb 2006, at 20:05, <fe...@gmail.com> wrote:
> I tested my own patch to DIRMINA-165 and I admit that it is very
> very bad, please don't use it.
> I'm not sure of the why of the bad-working of those classes, but
> I'm sure that they simply don't work.
> For the reason please wait for the answer of someone else.
>
> by Federico Bonelli
>
> ----- Original Message ----- From: "Joshua Portway" <jo...@stain.org>
> To: <de...@directory.apache.org>
> Sent: Wednesday, February 15, 2006 6:43 PM
> Subject: [mina] DIRMINA-165 revisited
>
>
>> Hi,
>> I've been looking at the code submitted for this bug (to allow the
>> creating of read-only buffers and buffer slices) and reading the
>> comments, and I'm not sure I understand why the submitted patch is
>> considered "dangerous". Trustin advises that all the methods
>> assert that every buffer involved is not pooled, but I'm not sure
>> why this should be - presumably, assuming that any derived
>> buffers maintain a reference to their parent buffer, the parent
>> buffer won't be returned to the pool until all derived buffers
>> have been released, and therefore released it in turn. Am I
>> missing something here ?
>>
>> josh
>
>
Re: [mina] DIRMINA-165 revisited
Posted by fe...@gmail.com.
I tested my own patch to DIRMINA-165 and I admit that it is very very bad,
please don't use it.
I'm not sure of the why of the bad-working of those classes, but I'm sure
that they simply don't work.
For the reason please wait for the answer of someone else.
by Federico Bonelli
----- Original Message -----
From: "Joshua Portway" <jo...@stain.org>
To: <de...@directory.apache.org>
Sent: Wednesday, February 15, 2006 6:43 PM
Subject: [mina] DIRMINA-165 revisited
> Hi,
> I've been looking at the code submitted for this bug (to allow the
> creating of read-only buffers and buffer slices) and reading the
> comments, and I'm not sure I understand why the submitted patch is
> considered "dangerous". Trustin advises that all the methods assert that
> every buffer involved is not pooled, but I'm not sure why this should
> be - presumably, assuming that any derived buffers maintain a reference
> to their parent buffer, the parent buffer won't be returned to the pool
> until all derived buffers have been released, and therefore released it
> in turn. Am I missing something here ?
>
> josh