You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mina.apache.org by "Craig L. Ching" <cc...@mqsoftware.com> on 2008/03/13 20:41:12 UTC
IoBuffer and auto expand
Hi,
I've changed over to using the latest milestone release, 2.0.0-M1 and I
have a question about IoBuffer.autoExpand(true). I'm basically doing
the following:
IoBuffer buf = IoBuffer.allocate(1024);
buf.setAutoExpand(true);
[various buf.putXXX calls]
buf.flip();
[write buffer]
I note that the documentation for autoExpand(true) says this:
"The underlying ByteBuffer is reallocated by IoBuffer behind the scene
if the encoded data is larger than 16 bytes in the example above. Its
capacity will double, and its limit will increase to the last position
the string is written."
For most of my usage, a small buffer is all I need. However, there are
specific instances where that IoBuffer will need to grow to 1MB+. The
problem I'm having is that when it needs to grow, it appears to be
growing very slowly, contrary to what the documentation says. For
instance, I printed out IoBuffer.capacity() at various points and see
this:
Capacity: 892736
Capacity: 892958
Capacity: 893070
Capacity: 893292
Capacity: 893410
Capacity: 893632
Which tells me that it's growing very slowly which jibes with how I'm
seeing the machine behave (using 100% CPU for a few minutes). Not only
that, but we had the same problem with our own buffers (coincidentally
named IoBuffer about 10 years ago, feels like I'm right at home :-P )
and we implemented basically the algorithm you have (double the
capacity), but we also allowed the user to specify how to grow the
buffer (a simple "growSize" parameter). The point of this latter
comment is that we had the same behavior I'm seeing with MINA's
IoBuffers before we fixed our growth algorithms.
I haven't gotten down into the MINA code yet, I just wanted to see if
anyone had any obvious comments on what I might be doing wrong.
BTW, thanks for MINA, I was going to write something equivalent to it
for our own products, but I was very, very happy to see someone else
doing it and saving me quite a bit of time ;-)
Cheers,
Craig
RE: IoBuffer and auto expand
Posted by "Craig L. Ching" <cc...@mqsoftware.com>.
Awesome, fix works great, thanks Trustin!
Cheers,
Craig
> -----Original Message-----
> From: Craig L. Ching [mailto:cching@mqsoftware.com]
> Sent: Monday, March 17, 2008 9:19 AM
> To: dev@mina.apache.org
> Subject: RE: IoBuffer and auto expand
>
> Hi Trustin,
>
> Thanks, I'll definitely be trying that out today! As for the grow size, I
> don't know that it's necessary at all, I was probably describing my
> problem too deeply ;-) If I do feel I need it, I'll definitely contribute
> something, but for now the interface is fine barring this defect. I'll
> give it a try and let you know.
>
> Cheers,
> Craig
>
> > -----Original Message-----
> > From: 이희승 (Trustin Lee) [mailto:trustin@gmail.com]
> > Sent: Sunday, March 16, 2008 10:22 PM
> > To: dev@mina.apache.org
> > Subject: Re: IoBuffer and auto expand
> >
> > Hi Craig,
> >
> > Thanks for reporting a bug. I've just fixed it. Please give
> > 2.0.0-M2-SNAPSHOT a try.
> >
> > I'm not sure growSize parameter will be meaningful in most cases. You
> > could come up with some patch if you are interested in implementing such
> a
> > feature. I'd suggest expansionRate (float, default = 2.0) property in
> > IoBuffer.
> >
> > Please also note that you can manually expand the buffer by calling
> > IoBuffer.expand(...) method.
> >
> > HTH,
> >
> > On Fri, 14 Mar 2008 04:41:12 +0900, Craig L. Ching
> <cc...@mqsoftware.com>
> > wrote:
> >
> > > Hi,
> > >
> > > I've changed over to using the latest milestone release, 2.0.0-M1 and
> I
> > > have a question about IoBuffer.autoExpand(true). I'm basically doing
> > > the following:
> > >
> > > IoBuffer buf = IoBuffer.allocate(1024);
> > > buf.setAutoExpand(true);
> > >
> > > [various buf.putXXX calls]
> > >
> > > buf.flip();
> > >
> > > [write buffer]
> > >
> > > I note that the documentation for autoExpand(true) says this:
> > >
> > > "The underlying ByteBuffer is reallocated by IoBuffer behind the scene
> > > if the encoded data is larger than 16 bytes in the example above. Its
> > > capacity will double, and its limit will increase to the last position
> > > the string is written."
> > >
> > > For most of my usage, a small buffer is all I need. However, there
> are
> > > specific instances where that IoBuffer will need to grow to 1MB+. The
> > > problem I'm having is that when it needs to grow, it appears to be
> > > growing very slowly, contrary to what the documentation says. For
> > > instance, I printed out IoBuffer.capacity() at various points and see
> > > this:
> > >
> > > Capacity: 892736
> > > Capacity: 892958
> > > Capacity: 893070
> > > Capacity: 893292
> > > Capacity: 893410
> > > Capacity: 893632
> > >
> > > Which tells me that it's growing very slowly which jibes with how I'm
> > > seeing the machine behave (using 100% CPU for a few minutes). Not
> only
> > > that, but we had the same problem with our own buffers (coincidentally
> > > named IoBuffer about 10 years ago, feels like I'm right at home :-P )
> > > and we implemented basically the algorithm you have (double the
> > > capacity), but we also allowed the user to specify how to grow the
> > > buffer (a simple "growSize" parameter). The point of this latter
> > > comment is that we had the same behavior I'm seeing with MINA's
> > > IoBuffers before we fixed our growth algorithms.
> > >
> > > I haven't gotten down into the MINA code yet, I just wanted to see if
> > > anyone had any obvious comments on what I might be doing wrong.
> > >
> > > BTW, thanks for MINA, I was going to write something equivalent to it
> > > for our own products, but I was very, very happy to see someone else
> > > doing it and saving me quite a bit of time ;-)
> > >
> > > Cheers,
> > > Craig
> >
> >
> >
> > --
> > Trustin Lee - Principal Software Engineer, JBoss, Red Hat
> > --
> > what we call human nature is actually human habit
> > --
> > http://gleamynode.net/
RE: IoBuffer and auto expand
Posted by "Craig L. Ching" <cc...@mqsoftware.com>.
Hi Trustin,
Thanks, I'll definitely be trying that out today! As for the grow size, I don't know that it's necessary at all, I was probably describing my problem too deeply ;-) If I do feel I need it, I'll definitely contribute something, but for now the interface is fine barring this defect. I'll give it a try and let you know.
Cheers,
Craig
> -----Original Message-----
> From: 이희승 (Trustin Lee) [mailto:trustin@gmail.com]
> Sent: Sunday, March 16, 2008 10:22 PM
> To: dev@mina.apache.org
> Subject: Re: IoBuffer and auto expand
>
> Hi Craig,
>
> Thanks for reporting a bug. I've just fixed it. Please give
> 2.0.0-M2-SNAPSHOT a try.
>
> I'm not sure growSize parameter will be meaningful in most cases. You
> could come up with some patch if you are interested in implementing such a
> feature. I'd suggest expansionRate (float, default = 2.0) property in
> IoBuffer.
>
> Please also note that you can manually expand the buffer by calling
> IoBuffer.expand(...) method.
>
> HTH,
>
> On Fri, 14 Mar 2008 04:41:12 +0900, Craig L. Ching <cc...@mqsoftware.com>
> wrote:
>
> > Hi,
> >
> > I've changed over to using the latest milestone release, 2.0.0-M1 and I
> > have a question about IoBuffer.autoExpand(true). I'm basically doing
> > the following:
> >
> > IoBuffer buf = IoBuffer.allocate(1024);
> > buf.setAutoExpand(true);
> >
> > [various buf.putXXX calls]
> >
> > buf.flip();
> >
> > [write buffer]
> >
> > I note that the documentation for autoExpand(true) says this:
> >
> > "The underlying ByteBuffer is reallocated by IoBuffer behind the scene
> > if the encoded data is larger than 16 bytes in the example above. Its
> > capacity will double, and its limit will increase to the last position
> > the string is written."
> >
> > For most of my usage, a small buffer is all I need. However, there are
> > specific instances where that IoBuffer will need to grow to 1MB+. The
> > problem I'm having is that when it needs to grow, it appears to be
> > growing very slowly, contrary to what the documentation says. For
> > instance, I printed out IoBuffer.capacity() at various points and see
> > this:
> >
> > Capacity: 892736
> > Capacity: 892958
> > Capacity: 893070
> > Capacity: 893292
> > Capacity: 893410
> > Capacity: 893632
> >
> > Which tells me that it's growing very slowly which jibes with how I'm
> > seeing the machine behave (using 100% CPU for a few minutes). Not only
> > that, but we had the same problem with our own buffers (coincidentally
> > named IoBuffer about 10 years ago, feels like I'm right at home :-P )
> > and we implemented basically the algorithm you have (double the
> > capacity), but we also allowed the user to specify how to grow the
> > buffer (a simple "growSize" parameter). The point of this latter
> > comment is that we had the same behavior I'm seeing with MINA's
> > IoBuffers before we fixed our growth algorithms.
> >
> > I haven't gotten down into the MINA code yet, I just wanted to see if
> > anyone had any obvious comments on what I might be doing wrong.
> >
> > BTW, thanks for MINA, I was going to write something equivalent to it
> > for our own products, but I was very, very happy to see someone else
> > doing it and saving me quite a bit of time ;-)
> >
> > Cheers,
> > Craig
>
>
>
> --
> Trustin Lee - Principal Software Engineer, JBoss, Red Hat
> --
> what we call human nature is actually human habit
> --
> http://gleamynode.net/
Re: IoBuffer and auto expand
Posted by "이희승 (Trustin Lee)" <tr...@gmail.com>.
Hi Craig,
Thanks for reporting a bug. I've just fixed it. Please give
2.0.0-M2-SNAPSHOT a try.
I'm not sure growSize parameter will be meaningful in most cases. You
could come up with some patch if you are interested in implementing such a
feature. I'd suggest expansionRate (float, default = 2.0) property in
IoBuffer.
Please also note that you can manually expand the buffer by calling
IoBuffer.expand(...) method.
HTH,
On Fri, 14 Mar 2008 04:41:12 +0900, Craig L. Ching <cc...@mqsoftware.com>
wrote:
> Hi,
>
> I've changed over to using the latest milestone release, 2.0.0-M1 and I
> have a question about IoBuffer.autoExpand(true). I'm basically doing
> the following:
>
> IoBuffer buf = IoBuffer.allocate(1024);
> buf.setAutoExpand(true);
>
> [various buf.putXXX calls]
>
> buf.flip();
>
> [write buffer]
>
> I note that the documentation for autoExpand(true) says this:
>
> "The underlying ByteBuffer is reallocated by IoBuffer behind the scene
> if the encoded data is larger than 16 bytes in the example above. Its
> capacity will double, and its limit will increase to the last position
> the string is written."
>
> For most of my usage, a small buffer is all I need. However, there are
> specific instances where that IoBuffer will need to grow to 1MB+. The
> problem I'm having is that when it needs to grow, it appears to be
> growing very slowly, contrary to what the documentation says. For
> instance, I printed out IoBuffer.capacity() at various points and see
> this:
>
> Capacity: 892736
> Capacity: 892958
> Capacity: 893070
> Capacity: 893292
> Capacity: 893410
> Capacity: 893632
>
> Which tells me that it's growing very slowly which jibes with how I'm
> seeing the machine behave (using 100% CPU for a few minutes). Not only
> that, but we had the same problem with our own buffers (coincidentally
> named IoBuffer about 10 years ago, feels like I'm right at home :-P )
> and we implemented basically the algorithm you have (double the
> capacity), but we also allowed the user to specify how to grow the
> buffer (a simple "growSize" parameter). The point of this latter
> comment is that we had the same behavior I'm seeing with MINA's
> IoBuffers before we fixed our growth algorithms.
>
> I haven't gotten down into the MINA code yet, I just wanted to see if
> anyone had any obvious comments on what I might be doing wrong.
>
> BTW, thanks for MINA, I was going to write something equivalent to it
> for our own products, but I was very, very happy to see someone else
> doing it and saving me quite a bit of time ;-)
>
> Cheers,
> Craig
--
Trustin Lee - Principal Software Engineer, JBoss, Red Hat
--
what we call human nature is actually human habit
--
http://gleamynode.net/