You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mina.apache.org by Rich Dougherty <ri...@rd.gen.nz> on 2008/04/30 22:54:29 UTC
Streams and disposal of ByteBuffers [was Re: Redesigning IoBuffer...]
On Wed, Apr 30, 2008 at 1:29 PM, Daniel Wirtz <da...@virtunity.com> wrote:
> Another advantage is, that the coherent composite stream could be ... [snip] Also
> completely consumed byte arrays could be freed automatically by removing the
> reference from the stream.
I thought it might be worth picking up on Daniel's point here. After
reading a discussion a while ago about direct ByteBuffers, I've been
thinking about approaches for automatically handling their allocation
and deallocation. Daniel's suggestion - that a stream can handle the
deallocation automatically - seems like a great solution to this
problem.
The good thing about a severely restricted interface like a stream is that:
(a) we can prevent references to ByteBuffers from escaping
(b) the consumer can only read forward, not back, so we know when a
ByteBuffer is finished with.
This means we do not need to rely on automatic garbage collection for
disposal. We know when a ByteBuffer is garbage, so we can do it
ourself. This means we can use a potentially more efficient scheme,
like pooling.
I'm pretty new to MINA, so this may already have been thought about,
or implemented, or perhaps it's not even a problem! Anyway, I thought
I'd mention it, since it seemed like a possible benefit of a stream
interface. (And this does not in any way detract from the current
Iteratable<BufferBuffer>/ByteBuffer proposals floating around either -
streams may not be appropriate or necessary for all IoFilters.)
Cheers
Rich
--
Rich Dougherty
http://blog.richdougherty.com/
Re: Streams and disposal of ByteBuffers [Was: Re: Redesigning IoBuffer...]
Posted by Rich Dougherty <ri...@rd.gen.nz>.
Patch is now added to
ticket<https://issues.apache.org/jira/browse/DIRMINA-489>,
and correctly licensed.
Cheers
Rich
On Fri, May 16, 2008 at 8:40 AM, 이희승 (Trustin Lee) <tr...@gmail.com>
wrote:
> Oh sure. I know you spent great deal of time to resolve this issue. :)
>
> It would be great if you can attach the patch again to our JIRA page just
> to make sure it's correctly licensed.
>
--
http://www.richdougherty.com/
Re: Streams and disposal of ByteBuffers [Was: Re: Redesigning
IoBuffer...]
Posted by "이희승 (Trustin Lee)" <tr...@gmail.com>.
Hey Rich,
Oh sure. I know you spent great deal of time to resolve this issue. :)
It would be great if you can attach the patch again to our JIRA page just
to make sure it's correctly licensed.
Thanks again!
On Thu, 15 May 2008 21:19:26 +0900, Rich Dougherty <ri...@rd.gen.nz> wrote:
> Hi Trustin
>
> Not just discussion! Since my last email I've spent a bit of time playing
> around with a set of classes and interfaces that (hopefully) provide the
> sort of functionality that we talk about. They're not quite ready, but I
> think it might be worth sharing them, so you can consider them for your
> new
> branch.
>
> I've attached the source for these files. They're not complete yet or
> very
> well tested (e.g. flushing doesn't work properly), but I think I'm
> reasonably happy with the design. Notably there's no mark/reset
> functionality, nor is there support for reading/writing a wide variety of
> types (just byte/ByteBuffer/int). These features should be easy to add. I
> just wanted to keep it as simple as possible initially.
>
> The key thing I've done is factored the design into multiple classes and
> interfaces. This move away from a single, monolithic class is
> intentional.
> Having multiple classes allows users to pick the implementation that
> suits a
> particular usage. It also dramatically simplifies implementation since we
> can use lower-level classes to build our higher-level ones, rather than
> trying to do everything within a single class. e.g. Stream classes are
> much
> easier to write once a good composite buffer class has been written.
>
> Here's a summary of the classes: (Names are just placeholders.)
> - *BufferByteArray:* A class that wraps ByteBuffer, providing simple
> utility
> methods and especially a *free* method to support pooling.
> - *CompositeByteArray:* There is a class with the same interface that
> supports multiple buffers with O(1) adding and removing.
> - **ByteArray.Cursor: *Stores position information for a *ByteArray*.
> Keeping this information separate makes the classes simpler, and gives
> users
> more flexibility (e.g. reading and writing at separate positions at the
> same
> time).
> - *CompositeByteArrayRelativeReader/Writer: *Restrictive,
> relative-access-only stream interfaces, backed by a *CompositeByteArray*.
> The benefit of these stream interfaces is that they control access to the
> underlying buffers, and so can do certain things automatically for the
> user
> (e.g. freeing buffers).
>
> Anyway the design should allow all the big features we've been talking
> about:
> - zero-copy reads
> - gathering writes
> - optional asynchronous stream interface with
> auto-freeing/auto-allocating/auto-flushing of ByteBuffers
>
> Please forgive me for not providing any more detail in this email - it's
> past midnight here, so I've had to be brief. :-) I've at least gone
> through
> the classes and given them some light javadoc, so hopefully that will be
> good enough for the moment.
>
> Cheers
> Rich
>
> On Thu, May 15, 2008 at 8:35 PM, 이희승 (Trustin Lee) <tr...@gmail.com>
> wrote:
>
>> If there's no objection or no better alternative, let me create a branch
>> and go forward. The implementation should be straightforward. I will
>> also
>> give you updates whenever meaningful commit is made.
>>
>> A bunch of thanks to Daniel, Rich and Emmanuel for all the related
>> discussion! :)
>>
>> Cheers,
>>
>>
>> On Wed, 14 May 2008 23:31:00 +0900, 이희승 (Trustin Lee)
>> <tr...@gmail.com>
>> wrote:
>>
>> Daniel and folks,
>>>
>>> It seems like we are getting close to consensus.
>>>
>>> I concur with you that the stream-like implementation makes a perfect
>>> sense. I'd prefer to roll out our own stream-like real non-blocking
>>> stream
>>> and provide a wrapper to InputStream and OutputStream. I think we can
>>> use
>>> java.io.RandomAccessFile as a good starting point:
>>>
>>> * http://java.sun.com/javase/6/docs/api/java/io/RandomAccessFile.html
>>>
>>> What we need with the new interface are:
>>>
>>> * stream-like sequential read and write operations for primitive
>>> types,
>>> arrays of primitive type and ByteBuffer
>>> * seek operations - mark, reset, position (let's keep this minimal
>>> because it's not used often)
>>> * discard operation for efficient memory utilization (+ discardAll)
>>> * no flip / no compact
>>> * no convenience read / write methods such as readString, readUTF8,
>>> readEnum, ... (will be provided in a utility class)
>>>
>>> For instance, it could look like this:
>>>
>>> public interface NameToBeDetermined extends Iterable<ByteBuffer> {
>>> ByteBuffer read();
>>> byte readByte();
>>> int readInt();
>>> int read(ByteBuffer dst);
>>> ...
>>>
>>> void write(ByteBuffer src);
>>> void writeByte(byte value);
>>> void writeInt(int value);
>>> ...
>>>
>>> int available();
>>> int position();
>>> void position(int position);
>>> int skip(int bytes);
>>> void mark();
>>> void reset();
>>>
>>> void discard();
>>> void discardAll();
>>>
>>> // Some extra methods which might help:
>>> ByteOrder order();
>>> void order(ByteOrder order);
>>> boolean isAutoDiscard();
>>> // true by default, which is OK for most users.
>>> void setAutoDiscard(boolean autoDiscard);
>>>
>>> // Conversion methods...
>>> ByteBuffer[] asByteBuffers();
>>> InputStream asInputStream();
>>> OutputStream asOutputStream();
>>> }
>>>
>>> I introduced autoDiscard so any read buffer is discarded automatically.
>>> Some users would want this happen manually, then they can call
>>> setAutoDiscard(false).
>>>
>>> Another good thing to have is a bounded stream, which has limited
>>> capacity
>>> to prevent OutOfMemoryError. It should be simple to implement because
>>> all
>>> it needs to is implement the interface.
>>>
>>> The last (?) thing to consider is to use get and put instead of read
>>> and
>>> write to ease migration from IoBuffer, and it is easier to type.
>>>
>>> WDYT?
>>>
>>> On Sat, 03 May 2008 23:49:41 +0900, Daniel Wirtz <da...@virtunity.com>
>>> wrote:
>>>
>>>> Hello Trusin,
>>>>
>>>> you pointed out some very interesting thoughts. In fact it will
>>>> result in
>>>> unexpected behaviour, when common stream wrappers are used on a
>>>> non-blocking
>>>> stream without proper orevious checking. I see two possibilities:
>>>> Using
>>>> streams that emulate a blocking mode and throw some sort of
>>>> WouldBlockException if blocking is disabled (by default) or rolling
>>>> out
>>>> our
>>>> own stream-like version of a real non-blocking stream.
>>>>
>>>> I see the following general advantages of streams over buffers:
>>>> - Very easy to use because the stream API is very clear and known by
>>>> most
>>>> developers
>>>> - Automatic dispose of already consumed underlying buffers
>>>> - No exposure of unnecessary low level functionality like flip(),
>>>> limit(), iterate over buffers etc.
>>>> - Possibility to reuse the same stream when some new data is
>>>> available so
>>>> that protocol developers don`t need to handle buffering on their own
>>>> (just
>>>> leave unused bytes inside the stream and process them later when more
>>>> data
>>>> is available / the decoder is notified)
>>>> - Zero-Copy without a need for users to know (much) about the
>>>> underlying
>>>> buffer architecture (just append buffers to a stream)
>>>> - Not an advantage, but also: Possibility for random access by using
>>>> mark(), skip() and reset() in sequence, maybe with an additional
>>>> moveTo()
>>>> relative to the marked position implemented by combining the previous
>>>> three
>>>> - however, random access is not required for most protocols
>>>> - Ability to use common wrappers for OutputStream
>>>> - Ability to use common wrappers for InputStream (but only if data
>>>> availability is previously checkable/checked to be valid)
>>>>
>>>> Disadvantages of extending java.io.InputStream:
>>>> - Unexpected behaviour when using common wrappers on a stream in
>>>> non-blocking mode with no proper previous checking of available data
>>>>
>>>> So, do we really need to extend InputStream / use common input
>>>> wrappers?
>>>> Maybe it makes sence to roll out our own stream-like real non-blocking
>>>> inputstream that cannot be used with common input wrappers but
>>>> implementing
>>>> all the required put- and get-Methods. However, I still think that a
>>>> stream-like implementation of the whole buffer thing makes it much
>>>> easier
>>>> for the end user to understand and use.
>>>>
>>>> regards
>>>> Daniel
>>>>
>>>
>>>
>>>
>>>
>>
>>
>> --
>> Trustin Lee - Principal Software Engineer, JBoss, Red Hat
>> --
>> what we call human nature is actually human habit
>> --
>> http://gleamynode.net/
>>
>
> --
> http://www.richdougherty.com/
--
Trustin Lee - Principal Software Engineer, JBoss, Red Hat
--
what we call human nature is actually human habit
--
http://gleamynode.net/
Re: Streams and disposal of ByteBuffers [Was: Re: Redesigning IoBuffer...]
Posted by Rich Dougherty <ri...@rd.gen.nz>.
Hi Trustin
Not just discussion! Since my last email I've spent a bit of time playing
around with a set of classes and interfaces that (hopefully) provide the
sort of functionality that we talk about. They're not quite ready, but I
think it might be worth sharing them, so you can consider them for your new
branch.
I've attached the source for these files. They're not complete yet or very
well tested (e.g. flushing doesn't work properly), but I think I'm
reasonably happy with the design. Notably there's no mark/reset
functionality, nor is there support for reading/writing a wide variety of
types (just byte/ByteBuffer/int). These features should be easy to add. I
just wanted to keep it as simple as possible initially.
The key thing I've done is factored the design into multiple classes and
interfaces. This move away from a single, monolithic class is intentional.
Having multiple classes allows users to pick the implementation that suits a
particular usage. It also dramatically simplifies implementation since we
can use lower-level classes to build our higher-level ones, rather than
trying to do everything within a single class. e.g. Stream classes are much
easier to write once a good composite buffer class has been written.
Here's a summary of the classes: (Names are just placeholders.)
- *BufferByteArray:* A class that wraps ByteBuffer, providing simple utility
methods and especially a *free* method to support pooling.
- *CompositeByteArray:* There is a class with the same interface that
supports multiple buffers with O(1) adding and removing.
- **ByteArray.Cursor: *Stores position information for a *ByteArray*.
Keeping this information separate makes the classes simpler, and gives users
more flexibility (e.g. reading and writing at separate positions at the same
time).
- *CompositeByteArrayRelativeReader/Writer: *Restrictive,
relative-access-only stream interfaces, backed by a *CompositeByteArray*.
The benefit of these stream interfaces is that they control access to the
underlying buffers, and so can do certain things automatically for the user
(e.g. freeing buffers).
Anyway the design should allow all the big features we've been talking
about:
- zero-copy reads
- gathering writes
- optional asynchronous stream interface with
auto-freeing/auto-allocating/auto-flushing of ByteBuffers
Please forgive me for not providing any more detail in this email - it's
past midnight here, so I've had to be brief. :-) I've at least gone through
the classes and given them some light javadoc, so hopefully that will be
good enough for the moment.
Cheers
Rich
On Thu, May 15, 2008 at 8:35 PM, 이희승 (Trustin Lee) <tr...@gmail.com>
wrote:
> If there's no objection or no better alternative, let me create a branch
> and go forward. The implementation should be straightforward. I will also
> give you updates whenever meaningful commit is made.
>
> A bunch of thanks to Daniel, Rich and Emmanuel for all the related
> discussion! :)
>
> Cheers,
>
>
> On Wed, 14 May 2008 23:31:00 +0900, 이희승 (Trustin Lee) <tr...@gmail.com>
> wrote:
>
> Daniel and folks,
>>
>> It seems like we are getting close to consensus.
>>
>> I concur with you that the stream-like implementation makes a perfect
>> sense. I'd prefer to roll out our own stream-like real non-blocking stream
>> and provide a wrapper to InputStream and OutputStream. I think we can use
>> java.io.RandomAccessFile as a good starting point:
>>
>> * http://java.sun.com/javase/6/docs/api/java/io/RandomAccessFile.html
>>
>> What we need with the new interface are:
>>
>> * stream-like sequential read and write operations for primitive types,
>> arrays of primitive type and ByteBuffer
>> * seek operations - mark, reset, position (let's keep this minimal
>> because it's not used often)
>> * discard operation for efficient memory utilization (+ discardAll)
>> * no flip / no compact
>> * no convenience read / write methods such as readString, readUTF8,
>> readEnum, ... (will be provided in a utility class)
>>
>> For instance, it could look like this:
>>
>> public interface NameToBeDetermined extends Iterable<ByteBuffer> {
>> ByteBuffer read();
>> byte readByte();
>> int readInt();
>> int read(ByteBuffer dst);
>> ...
>>
>> void write(ByteBuffer src);
>> void writeByte(byte value);
>> void writeInt(int value);
>> ...
>>
>> int available();
>> int position();
>> void position(int position);
>> int skip(int bytes);
>> void mark();
>> void reset();
>>
>> void discard();
>> void discardAll();
>>
>> // Some extra methods which might help:
>> ByteOrder order();
>> void order(ByteOrder order);
>> boolean isAutoDiscard();
>> // true by default, which is OK for most users.
>> void setAutoDiscard(boolean autoDiscard);
>>
>> // Conversion methods...
>> ByteBuffer[] asByteBuffers();
>> InputStream asInputStream();
>> OutputStream asOutputStream();
>> }
>>
>> I introduced autoDiscard so any read buffer is discarded automatically.
>> Some users would want this happen manually, then they can call
>> setAutoDiscard(false).
>>
>> Another good thing to have is a bounded stream, which has limited capacity
>> to prevent OutOfMemoryError. It should be simple to implement because all
>> it needs to is implement the interface.
>>
>> The last (?) thing to consider is to use get and put instead of read and
>> write to ease migration from IoBuffer, and it is easier to type.
>>
>> WDYT?
>>
>> On Sat, 03 May 2008 23:49:41 +0900, Daniel Wirtz <da...@virtunity.com>
>> wrote:
>>
>>> Hello Trusin,
>>>
>>> you pointed out some very interesting thoughts. In fact it will result in
>>> unexpected behaviour, when common stream wrappers are used on a
>>> non-blocking
>>> stream without proper orevious checking. I see two possibilities: Using
>>> streams that emulate a blocking mode and throw some sort of
>>> WouldBlockException if blocking is disabled (by default) or rolling out
>>> our
>>> own stream-like version of a real non-blocking stream.
>>>
>>> I see the following general advantages of streams over buffers:
>>> - Very easy to use because the stream API is very clear and known by most
>>> developers
>>> - Automatic dispose of already consumed underlying buffers
>>> - No exposure of unnecessary low level functionality like flip(),
>>> limit(), iterate over buffers etc.
>>> - Possibility to reuse the same stream when some new data is available so
>>> that protocol developers don`t need to handle buffering on their own
>>> (just
>>> leave unused bytes inside the stream and process them later when more
>>> data
>>> is available / the decoder is notified)
>>> - Zero-Copy without a need for users to know (much) about the underlying
>>> buffer architecture (just append buffers to a stream)
>>> - Not an advantage, but also: Possibility for random access by using
>>> mark(), skip() and reset() in sequence, maybe with an additional moveTo()
>>> relative to the marked position implemented by combining the previous
>>> three
>>> - however, random access is not required for most protocols
>>> - Ability to use common wrappers for OutputStream
>>> - Ability to use common wrappers for InputStream (but only if data
>>> availability is previously checkable/checked to be valid)
>>>
>>> Disadvantages of extending java.io.InputStream:
>>> - Unexpected behaviour when using common wrappers on a stream in
>>> non-blocking mode with no proper previous checking of available data
>>>
>>> So, do we really need to extend InputStream / use common input wrappers?
>>> Maybe it makes sence to roll out our own stream-like real non-blocking
>>> inputstream that cannot be used with common input wrappers but
>>> implementing
>>> all the required put- and get-Methods. However, I still think that a
>>> stream-like implementation of the whole buffer thing makes it much easier
>>> for the end user to understand and use.
>>>
>>> regards
>>> Daniel
>>>
>>
>>
>>
>>
>
>
> --
> Trustin Lee - Principal Software Engineer, JBoss, Red Hat
> --
> what we call human nature is actually human habit
> --
> http://gleamynode.net/
>
--
http://www.richdougherty.com/
Re: Streams and disposal of ByteBuffers [Was: Re: Redesigning IoBuffer...]
Posted by Emmanuel Lecharny <el...@apache.org>.
이희승 (Trustin Lee) wrote:
> If there's no objection or no better alternative, let me create a
> branch and go forward. The implementation should be straightforward.
> I will also give you updates whenever meaningful commit is made.
Go for it !
--
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org
Re: Streams and disposal of ByteBuffers [Was: Re: Redesigning
IoBuffer...]
Posted by "이희승 (Trustin Lee)" <tr...@gmail.com>.
If there's no objection or no better alternative, let me create a branch
and go forward. The implementation should be straightforward. I will
also give you updates whenever meaningful commit is made.
A bunch of thanks to Daniel, Rich and Emmanuel for all the related
discussion! :)
Cheers,
On Wed, 14 May 2008 23:31:00 +0900, 이희승 (Trustin Lee)
<tr...@gmail.com> wrote:
> Daniel and folks,
>
> It seems like we are getting close to consensus.
>
> I concur with you that the stream-like implementation makes a perfect
> sense. I'd prefer to roll out our own stream-like real non-blocking
> stream and provide a wrapper to InputStream and OutputStream. I think
> we can use java.io.RandomAccessFile as a good starting point:
>
> * http://java.sun.com/javase/6/docs/api/java/io/RandomAccessFile.html
>
> What we need with the new interface are:
>
> * stream-like sequential read and write operations for primitive
> types, arrays of primitive type and ByteBuffer
> * seek operations - mark, reset, position (let's keep this minimal
> because it's not used often)
> * discard operation for efficient memory utilization (+ discardAll)
> * no flip / no compact
> * no convenience read / write methods such as readString, readUTF8,
> readEnum, ... (will be provided in a utility class)
>
> For instance, it could look like this:
>
> public interface NameToBeDetermined extends Iterable<ByteBuffer> {
> ByteBuffer read();
> byte readByte();
> int readInt();
> int read(ByteBuffer dst);
> ...
>
> void write(ByteBuffer src);
> void writeByte(byte value);
> void writeInt(int value);
> ...
>
> int available();
> int position();
> void position(int position);
> int skip(int bytes);
> void mark();
> void reset();
>
> void discard();
> void discardAll();
>
> // Some extra methods which might help:
> ByteOrder order();
> void order(ByteOrder order);
> boolean isAutoDiscard();
> // true by default, which is OK for most users.
> void setAutoDiscard(boolean autoDiscard);
>
> // Conversion methods...
> ByteBuffer[] asByteBuffers();
> InputStream asInputStream();
> OutputStream asOutputStream();
> }
>
> I introduced autoDiscard so any read buffer is discarded automatically.
> Some users would want this happen manually, then they can call
> setAutoDiscard(false).
>
> Another good thing to have is a bounded stream, which has limited
> capacity to prevent OutOfMemoryError. It should be simple to implement
> because all it needs to is implement the interface.
>
> The last (?) thing to consider is to use get and put instead of read and
> write to ease migration from IoBuffer, and it is easier to type.
>
> WDYT?
>
> On Sat, 03 May 2008 23:49:41 +0900, Daniel Wirtz <da...@virtunity.com>
> wrote:
>> Hello Trusin,
>>
>> you pointed out some very interesting thoughts. In fact it will result
>> in
>> unexpected behaviour, when common stream wrappers are used on a
>> non-blocking
>> stream without proper orevious checking. I see two possibilities: Using
>> streams that emulate a blocking mode and throw some sort of
>> WouldBlockException if blocking is disabled (by default) or rolling out
>> our
>> own stream-like version of a real non-blocking stream.
>>
>> I see the following general advantages of streams over buffers:
>> - Very easy to use because the stream API is very clear and known by
>> most
>> developers
>> - Automatic dispose of already consumed underlying buffers
>> - No exposure of unnecessary low level functionality like flip(),
>> limit(), iterate over buffers etc.
>> - Possibility to reuse the same stream when some new data is available
>> so
>> that protocol developers don`t need to handle buffering on their own
>> (just
>> leave unused bytes inside the stream and process them later when more
>> data
>> is available / the decoder is notified)
>> - Zero-Copy without a need for users to know (much) about the underlying
>> buffer architecture (just append buffers to a stream)
>> - Not an advantage, but also: Possibility for random access by using
>> mark(), skip() and reset() in sequence, maybe with an additional
>> moveTo()
>> relative to the marked position implemented by combining the previous
>> three
>> - however, random access is not required for most protocols
>> - Ability to use common wrappers for OutputStream
>> - Ability to use common wrappers for InputStream (but only if data
>> availability is previously checkable/checked to be valid)
>>
>> Disadvantages of extending java.io.InputStream:
>> - Unexpected behaviour when using common wrappers on a stream in
>> non-blocking mode with no proper previous checking of available data
>>
>> So, do we really need to extend InputStream / use common input wrappers?
>> Maybe it makes sence to roll out our own stream-like real non-blocking
>> inputstream that cannot be used with common input wrappers but
>> implementing
>> all the required put- and get-Methods. However, I still think that a
>> stream-like implementation of the whole buffer thing makes it much
>> easier
>> for the end user to understand and use.
>>
>> regards
>> Daniel
>
>
>
--
Trustin Lee - Principal Software Engineer, JBoss, Red Hat
--
what we call human nature is actually human habit
--
http://gleamynode.net/
Re: Streams and disposal of ByteBuffers [Was: Re: Redesigning
IoBuffer...]
Posted by "이희승 (Trustin Lee)" <tr...@gmail.com>.
Daniel and folks,
It seems like we are getting close to consensus.
I concur with you that the stream-like implementation makes a perfect
sense. I'd prefer to roll out our own stream-like real non-blocking
stream and provide a wrapper to InputStream and OutputStream. I think we
can use java.io.RandomAccessFile as a good starting point:
* http://java.sun.com/javase/6/docs/api/java/io/RandomAccessFile.html
What we need with the new interface are:
* stream-like sequential read and write operations for primitive types,
arrays of primitive type and ByteBuffer
* seek operations - mark, reset, position (let's keep this minimal
because it's not used often)
* discard operation for efficient memory utilization (+ discardAll)
* no flip / no compact
* no convenience read / write methods such as readString, readUTF8,
readEnum, ... (will be provided in a utility class)
For instance, it could look like this:
public interface NameToBeDetermined extends Iterable<ByteBuffer> {
ByteBuffer read();
byte readByte();
int readInt();
int read(ByteBuffer dst);
...
void write(ByteBuffer src);
void writeByte(byte value);
void writeInt(int value);
...
int available();
int position();
void position(int position);
int skip(int bytes);
void mark();
void reset();
void discard();
void discardAll();
// Some extra methods which might help:
ByteOrder order();
void order(ByteOrder order);
boolean isAutoDiscard();
// true by default, which is OK for most users.
void setAutoDiscard(boolean autoDiscard);
// Conversion methods...
ByteBuffer[] asByteBuffers();
InputStream asInputStream();
OutputStream asOutputStream();
}
I introduced autoDiscard so any read buffer is discarded automatically.
Some users would want this happen manually, then they can call
setAutoDiscard(false).
Another good thing to have is a bounded stream, which has limited capacity
to prevent OutOfMemoryError. It should be simple to implement because all
it needs to is implement the interface.
The last (?) thing to consider is to use get and put instead of read and
write to ease migration from IoBuffer, and it is easier to type.
WDYT?
On Sat, 03 May 2008 23:49:41 +0900, Daniel Wirtz <da...@virtunity.com>
wrote:
> Hello Trusin,
>
> you pointed out some very interesting thoughts. In fact it will result in
> unexpected behaviour, when common stream wrappers are used on a
> non-blocking
> stream without proper orevious checking. I see two possibilities: Using
> streams that emulate a blocking mode and throw some sort of
> WouldBlockException if blocking is disabled (by default) or rolling out
> our
> own stream-like version of a real non-blocking stream.
>
> I see the following general advantages of streams over buffers:
> - Very easy to use because the stream API is very clear and known by most
> developers
> - Automatic dispose of already consumed underlying buffers
> - No exposure of unnecessary low level functionality like flip(),
> limit(), iterate over buffers etc.
> - Possibility to reuse the same stream when some new data is available so
> that protocol developers don`t need to handle buffering on their own
> (just
> leave unused bytes inside the stream and process them later when more
> data
> is available / the decoder is notified)
> - Zero-Copy without a need for users to know (much) about the underlying
> buffer architecture (just append buffers to a stream)
> - Not an advantage, but also: Possibility for random access by using
> mark(), skip() and reset() in sequence, maybe with an additional moveTo()
> relative to the marked position implemented by combining the previous
> three
> - however, random access is not required for most protocols
> - Ability to use common wrappers for OutputStream
> - Ability to use common wrappers for InputStream (but only if data
> availability is previously checkable/checked to be valid)
>
> Disadvantages of extending java.io.InputStream:
> - Unexpected behaviour when using common wrappers on a stream in
> non-blocking mode with no proper previous checking of available data
>
> So, do we really need to extend InputStream / use common input wrappers?
> Maybe it makes sence to roll out our own stream-like real non-blocking
> inputstream that cannot be used with common input wrappers but
> implementing
> all the required put- and get-Methods. However, I still think that a
> stream-like implementation of the whole buffer thing makes it much easier
> for the end user to understand and use.
>
> regards
> Daniel
--
Trustin Lee - Principal Software Engineer, JBoss, Red Hat
--
what we call human nature is actually human habit
--
http://gleamynode.net/
Re: Streams and disposal of ByteBuffers [Was: Re: Redesigning IoBuffer...]
Posted by Daniel Wirtz <da...@virtunity.com>.
Hello Trusin,
you pointed out some very interesting thoughts. In fact it will result in
unexpected behaviour, when common stream wrappers are used on a non-blocking
stream without proper orevious checking. I see two possibilities: Using
streams that emulate a blocking mode and throw some sort of
WouldBlockException if blocking is disabled (by default) or rolling out our
own stream-like version of a real non-blocking stream.
I see the following general advantages of streams over buffers:
- Very easy to use because the stream API is very clear and known by most
developers
- Automatic dispose of already consumed underlying buffers
- No exposure of unnecessary low level functionality like flip(),
limit(), iterate over buffers etc.
- Possibility to reuse the same stream when some new data is available so
that protocol developers don`t need to handle buffering on their own (just
leave unused bytes inside the stream and process them later when more data
is available / the decoder is notified)
- Zero-Copy without a need for users to know (much) about the underlying
buffer architecture (just append buffers to a stream)
- Not an advantage, but also: Possibility for random access by using
mark(), skip() and reset() in sequence, maybe with an additional moveTo()
relative to the marked position implemented by combining the previous three
- however, random access is not required for most protocols
- Ability to use common wrappers for OutputStream
- Ability to use common wrappers for InputStream (but only if data
availability is previously checkable/checked to be valid)
Disadvantages of extending java.io.InputStream:
- Unexpected behaviour when using common wrappers on a stream in
non-blocking mode with no proper previous checking of available data
So, do we really need to extend InputStream / use common input wrappers?
Maybe it makes sence to roll out our own stream-like real non-blocking
inputstream that cannot be used with common input wrappers but implementing
all the required put- and get-Methods. However, I still think that a
stream-like implementation of the whole buffer thing makes it much easier
for the end user to understand and use.
regards
Daniel
Re: Streams and disposal of ByteBuffers [Was: Re: Redesigning IoBuffer...]
Posted by 이희승 (Trustin Lee) <t...@gmail.com>.
Hi Daniel,
Sorry for the late response. Just got connected to the Internet after
landing down to SFO.
IIUC, you are suggesting an InputStream which allows movable position to
address the question #2. Doesn't it effectively make InputStream
random-accessible? If it's non-blocking and random-accessible, what
would be the benefit of using InputStream? I am asking these questions
because it sounds like the current IoBuffer without limit() and some
complications.
The unique advantages of using InputStream instead of our custom
interface I can think of right now are:
* Users already know InputStream very well.
* We can reuse all the existing stream wrappers around the world.
Ans possible disadvantages are:
* We violate the default blocking behavior of InputStream, so existing
wrappers might not work as expected unless the whole message is read,
and reading the whole message might lead to OOM so the wrappers
available might not be that helpful.
* The change is pretty big comparing to what we gain? (we don't care
much about how big the change is though ;)
Daniel Wirtz wrote:
> On Thu, May 1, 2008 at 5:05 AM, "이희승 (Trustin Lee) <trustin@gmail.com
> <ma...@gmail.com>> wrote:
>
> 1) How do we create a sliceqd?
>
>
> input.slice(count). Same as skip(count) but by reusing the relevant
> buffers / arrays and returning them as a new stream. Even overlapping
> buffers at the beginning/end are no problem, assumed that every buffer
> is wrapped by a wrapper instance that holds the valid range.
>
> Buffer: AAAAAABB
> Wrapper at the end of Stream A: (Buffer,0,6)
> Wrapper at the beginning of Stream B: (Buffer,6,8)
>
> Also new buffers can be appended to both streams without influencing
> each other.
>
>
> 2) How do we slice an InputStream if we only know the size of the slice
> after we read something and the slice should begin from the first byte
> of the stream?
>
>
>
> One possible solution to Q2 is to support mark() and reset() operation
> in InputStream, but it means we can't release the related resources even
> if the stream has reached to its end, because reset() can be called at
> any time. We might need something better?
>
>
> The only thing is, that no automatic dispose() can happen to buffers
> that are relevant to the marked position if set, which is completely ok
> because this data is of importance when using it. If it is not set
> explicitly, the stream should just free buffers automatically. Maybe
> there should be an additional method to discard the marked position
> without resetting:
>
> IoInputStream.mark(readlimit); // readlimit is very useful for us (see
> InputStream API)
> IoInputStream.reset(); // Reset to the marked position and clear it;
> throw IoException if mark has not been set
> + IoInputStream.discard(); // Discard/free everything before the current
> position (marked position = null, buffer references not used any longer)
>
> When supporting mark, even basic random access is possible, by
> subsequent calls to reset() and mark(). Some sort of
> IoInputStream.moveTo(int) relative to the marked position could
> make this even easier and also make sure that nobody needs to switch
> back to the lower level ByteBuffers / byte arrays for random access.
> However, there are only very few cases where random access is really
> required.
>
> regards
> Daniel
--
Trustin Lee - Principal Software Engineer, JBoss, Red Hat
--
what we call human nature is actually human habit
--
http://gleamynode.net/
Re: Streams and disposal of ByteBuffers [Was: Re: Redesigning IoBuffer...]
Posted by Daniel Wirtz <da...@virtunity.com>.
On Thu, May 1, 2008 at 5:05 AM, "이희승 (Trustin Lee) <tr...@gmail.com>
wrote:
> 1) How do we create a sliced?
input.slice(count). Same as skip(count) but by reusing the relevant buffers
/ arrays and returning them as a new stream. Even overlapping buffers at the
beginning/end are no problem, assumed that every buffer is wrapped by a
wrapper instance that holds the valid range.
Buffer: AAAAAABB
Wrapper at the end of Stream A: (Buffer,0,6)
Wrapper at the beginning of Stream B: (Buffer,6,8)
Also new buffers can be appended to both streams without influencing each
other.
> 2) How do we slice an InputStream if we only know the size of the slice
> after we read something and the slice should begin from the first byte
> of the stream?
> One possible solution to Q2 is to support mark() and reset() operation
> in InputStream, but it means we can't release the related resources even
> if the stream has reached to its end, because reset() can be called at
> any time. We might need something better?
The only thing is, that no automatic dispose() can happen to buffers that
are relevant to the marked position if set, which is completely ok
because this data is of importance when using it. If it is not set
explicitly, the stream should just free buffers automatically. Maybe there
should be an additional method to discard the marked position without
resetting:
IoInputStream.mark(readlimit); // readlimit is very useful for us (see
InputStream API)
IoInputStream.reset(); // Reset to the marked position and clear it; throw
IoException if mark has not been set
+ IoInputStream.discard(); // Discard/free everything before the current
position (marked position = null, buffer references not used any longer)
When supporting mark, even basic random access is possible, by subsequent
calls to reset() and mark(). Some sort of IoInputStream.moveTo(int) relative
to the marked position could make this even easier and also make sure that
nobody needs to switch back to the lower level ByteBuffers / byte arrays for
random access. However, there are only very few cases where random access is
really required.
regards
Daniel
Re: Streams and disposal of ByteBuffers [Was: Re: Redesigning IoBuffer...]
Posted by 이희승 (Trustin Lee) <t...@gmail.com>.
Hi Rich,
Rich Dougherty wrote:
> On Wed, Apr 30, 2008 at 1:29 PM, Daniel Wirtz <da...@virtunity.com> wrote:
>> Another advantage is, that the coherent composite stream could be ... [snip] Also
>> completely consumed byte arrays could be freed automatically by removing the
>> reference from the stream.
>
> I thought it might be worth picking up on Daniel's point here. After
> reading a discussion a while ago about direct ByteBuffers, I've been
> thinking about approaches for automatically handling their allocation
> and deallocation. Daniel's suggestion - that a stream can handle the
> deallocation automatically - seems like a great solution to this
> problem.
>
> The good thing about a severely restricted interface like a stream is that:
> (a) we can prevent references to ByteBuffers from escaping
> (b) the consumer can only read forward, not back, so we know when a
> ByteBuffer is finished with.
So far, the idea sounds very cool. I started to be convinced. :)
Probably the remaining questions are..
1) How do we create a sliced?
2) How do we slice an InputStream if we only know the size of the slice
after we read something and the slice should begin from the first byte
of the stream?
One possible solution to Q2 is to support mark() and reset() operation
in InputStream, but it means we can't release the related resources even
if the stream has reached to its end, because reset() can be called at
any time. We might need something better?
> This means we do not need to rely on automatic garbage collection for
> disposal. We know when a ByteBuffer is garbage, so we can do it
> ourself. This means we can use a potentially more efficient scheme,
> like pooling.
>
> I'm pretty new to MINA, so this may already have been thought about,
> or implemented, or perhaps it's not even a problem! Anyway, I thought
> I'd mention it, since it seemed like a possible benefit of a stream
> interface. (And this does not in any way detract from the current
> Iteratable<BufferBuffer>/ByteBuffer proposals floating around either -
> streams may not be appropriate or necessary for all IoFilters.)
I'd love to hear from someone new to MINA. Fresh blood comes with fresh
ideas!
Cheers,
--
Trustin Lee - Principal Software Engineer, JBoss, Red Hat
--
what we call human nature is actually human habit
--
http://gleamynode.net/