You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mina.apache.org by "David M. Lloyd" <da...@redhat.com> on 2008/01/04 17:10:12 UTC
Better support for "framing"
I've written an IoFilter that is intended to run over TCP, which
"frames" data into discrete messages in order to make TCP appear to
preserve message boundaries (like UDP) while still retaining the other
advantages of TCP (guaranteed delivery, firewall interoperability, etc).
My filter definitely *works* but it's not very good. It requires that
the message length be prepended to each message. I've got a hack in
such that if the provided outbound buffer has enough space "before" the
current position, it backs up the position and writes the length there.
Otherwise it has to copy the whole buffer. In order to make this work
efficiently, whenever I allocate or flip a buffer I have to "skip" a set
amount of bytes to "make room" for this data.
On receive, if the incoming buffer contains the whole message, the
filter can just forward a slice of this buffer to the upper layers.
Otherwise it has to copy the data into a new, properly-sized buffer.
So I'd like to ask your thoughts on the following:
1) Support sending multiple buffers in one "go" using gathering-writes
(the message type could be e.g. Collection<IoBuffer> in this case, or
perhaps using a composite buffer [see DIRMINA-489 [1]]).
2) Add support to IoBuffer for reserving "space" at the start of the
buffer. The semantics would need some thought here though. For
example, should the buffer capacity include the "reserved" space?
Probably not I guess. Other questions like this would need to be
answered as well. I guess I should open a JIRA ticket for this to
provide a point of reference.
Anyway, let me know what you guys think about this stuff.
- DML
[1] https://issues.apache.org/jira/browse/DIRMINA-489
Re: Better support for "framing"
Posted by "David M. Lloyd" <da...@redhat.com>.
On Fri, 4 Jan 2008 17:38:23 +0100
"Maarten Bosteels" <mb...@gmail.com> wrote:
> Hello David,
>
> On Jan 4, 2008 5:10 PM, David M. Lloyd <da...@redhat.com> wrote:
> > I've written an IoFilter that is intended to run over TCP, which
> > "frames" data into discrete messages in order to make TCP appear to
> > preserve message boundaries (like UDP) while still retaining the other
> > advantages of TCP (guaranteed delivery, firewall interoperability, etc).
>
> Isn't this the exact purpose of the ProtocolCodecFilter ?
> Have you had a look at
> http://mina.apache.org/tutorial-on-protocolcodecfilter-for-mina-2x.html
> ?
Yes - but more generally applied, rather than being protocol-specific.
So for example, if/when SCTP ever becomes available for MINA I can
simply remove the TCP+Filter and replace it with SCTP and get similar
behavior. Basically I can use an IoHandler that expects discrete
messages regardless of the underlying protocol.
> I guess you are looking for a ByteArrayCodecFactory that can
> encode/decode byte arrays ?
> Have a look at http://www.mail-archive.com/dev@mina.apache.org/msg06323.html
No, my point is that copying is evil, and using a byte array would
require that copying be done. Also the existing solutions all require
potentially a great deal of extra copying.
FWIW, I opened a feature request for the "framing" business here:
https://issues.apache.org/jira/browse/DIRMINA-507
- DML
> > My filter definitely *works* but it's not very good. It requires that
> > the message length be prepended to each message. I've got a hack in
> > such that if the provided outbound buffer has enough space "before" the
> > current position, it backs up the position and writes the length there.
> > Otherwise it has to copy the whole buffer. In order to make this work
> > efficiently, whenever I allocate or flip a buffer I have to "skip" a set
> > amount of bytes to "make room" for this data.
> >
> > On receive, if the incoming buffer contains the whole message, the
> > filter can just forward a slice of this buffer to the upper layers.
> > Otherwise it has to copy the data into a new, properly-sized buffer.
> >
> > So I'd like to ask your thoughts on the following:
> >
> > 1) Support sending multiple buffers in one "go" using gathering-writes
> > (the message type could be e.g. Collection<IoBuffer> in this case, or
> > perhaps using a composite buffer [see DIRMINA-489 [1]]).
> >
> > 2) Add support to IoBuffer for reserving "space" at the start of the
> > buffer. The semantics would need some thought here though. For
> > example, should the buffer capacity include the "reserved" space?
> > Probably not I guess. Other questions like this would need to be
> > answered as well. I guess I should open a JIRA ticket for this to
> > provide a point of reference.
> >
> > Anyway, let me know what you guys think about this stuff.
> >
> > - DML
> >
> > [1] https://issues.apache.org/jira/browse/DIRMINA-489
> >
Re: Better support for "framing"
Posted by Daniel Cheng <gm...@sdiz.net>.
David M. Lloyd wrote:
> On Fri, 4 Jan 2008 17:38:23 +0100
> "Maarten Bosteels" <mb...@gmail.com> wrote:
>
>> Have you had a look at
>> http://mina.apache.org/tutorial-on-protocolcodecfilter-for-mina-2x.html
>> ?
>
> A minor issue with the article - the fragmentation link in the first
> section ("Why use a ProtocolCodecFilter?") is completely irrelevant to
> how reads and writes are done. IP fragmentation has *no effect* on the
> reads and writes - one sent IP packet on the sending side equals zero
> or one IP packets on the receiving side, regardless of how it is
> fragmented along the way.
If you consider packet lost, SACK and congestion control,
there is no grantee these "frame"s would keep intact.
some simple case:
1. consider RWIN < "frame" size.
2. consider a slow reader on receiver side..
All of these are handled transparently at the OS (or maybe even a TCP
Offload engine). There are no way to get these info in a platform
independent way.
Never mention there are tons of non-standard TCP congestion control
algorithm that interact wildly with QoS routers.
TCP is intended to be used as stream, not "frame"s ...
> Probably a better link to reference here is "Nagle's Algorithm" [1].
> - DML
>
> [1] http://en.wikipedia.org/wiki/Nagle%27s_algorithm
>
--
This space was intended to be left blank.
Re: Better support for "framing"
Posted by "David M. Lloyd" <da...@redhat.com>.
On Fri, 4 Jan 2008 17:38:23 +0100
"Maarten Bosteels" <mb...@gmail.com> wrote:
> Have you had a look at
> http://mina.apache.org/tutorial-on-protocolcodecfilter-for-mina-2x.html
> ?
A minor issue with the article - the fragmentation link in the first
section ("Why use a ProtocolCodecFilter?") is completely irrelevant to
how reads and writes are done. IP fragmentation has *no effect* on the
reads and writes - one sent IP packet on the sending side equals zero
or one IP packets on the receiving side, regardless of how it is
fragmented along the way.
Probably a better link to reference here is "Nagle's Algorithm" [1].
- DML
[1] http://en.wikipedia.org/wiki/Nagle%27s_algorithm
Re: Better support for "framing"
Posted by Maarten Bosteels <mb...@gmail.com>.
Hello David,
On Jan 4, 2008 5:10 PM, David M. Lloyd <da...@redhat.com> wrote:
> I've written an IoFilter that is intended to run over TCP, which
> "frames" data into discrete messages in order to make TCP appear to
> preserve message boundaries (like UDP) while still retaining the other
> advantages of TCP (guaranteed delivery, firewall interoperability, etc).
Isn't this the exact purpose of the ProtocolCodecFilter ?
Have you had a look at
http://mina.apache.org/tutorial-on-protocolcodecfilter-for-mina-2x.html
?
I guess you are looking for a ByteArrayCodecFactory that can
encode/decode byte arrays ?
Have a look at http://www.mail-archive.com/dev@mina.apache.org/msg06323.html
We could certainly include such a ByteArrayCodecFactory in mina.
Trustin, any news on your idea expressed here :
http://www.mail-archive.com/dev@mina.apache.org/msg06290.html ?
regards,
Maarten
>
> My filter definitely *works* but it's not very good. It requires that
> the message length be prepended to each message. I've got a hack in
> such that if the provided outbound buffer has enough space "before" the
> current position, it backs up the position and writes the length there.
> Otherwise it has to copy the whole buffer. In order to make this work
> efficiently, whenever I allocate or flip a buffer I have to "skip" a set
> amount of bytes to "make room" for this data.
>
> On receive, if the incoming buffer contains the whole message, the
> filter can just forward a slice of this buffer to the upper layers.
> Otherwise it has to copy the data into a new, properly-sized buffer.
>
> So I'd like to ask your thoughts on the following:
>
> 1) Support sending multiple buffers in one "go" using gathering-writes
> (the message type could be e.g. Collection<IoBuffer> in this case, or
> perhaps using a composite buffer [see DIRMINA-489 [1]]).
>
> 2) Add support to IoBuffer for reserving "space" at the start of the
> buffer. The semantics would need some thought here though. For
> example, should the buffer capacity include the "reserved" space?
> Probably not I guess. Other questions like this would need to be
> answered as well. I guess I should open a JIRA ticket for this to
> provide a point of reference.
>
> Anyway, let me know what you guys think about this stuff.
>
> - DML
>
> [1] https://issues.apache.org/jira/browse/DIRMINA-489
>