You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@mina.apache.org by Louis Naber <lo...@gmail.com> on 2008/08/01 17:50:31 UTC
CumulativeProtocolDecoder
I have an unfortunate scenario where I am dealing with legacy output
peripherals that do not have nicely designed messaging built into them. The
messages are variable length, unterminated, with no header information.
So I have encountered a scenario, without a decoder, where the response
message (for example) is 70 bytes in length. I've confirmed this with a
packet analyzer and it is delivered as a single TCP message.
A portion of the message is delivered to the IO handler as a 64 byte buffer
where the lim=64 and the cap=64, followed immediately by the remainder of
the message in a 6 byte buffer.
To handle this situation I've implemented a CumulativeProtocolDecoder that
checks to see if the buffer lim = cap in which case I make a big assumption
that the message is incomplete, returning for more data.
This solves the immediate problem but still has me concerned for a number of
reasons.
1. Is MINA buffering splitting the message to fit into buffers.
2. What if the message is exactly the buffer size, my algorithm would cause
the message to "hang" so to speak.
The question I have, is it possible to find out if more data is available
for the session when the lim=capacity just to make sure that the incomplete
message assumption is correct?
Best Regards
Louis
Re: CumulativeProtocolDecoder
Posted by Emmanuel Lecharny <el...@gmail.com>.
jclabat wrote:
>>> Which kind of printer is it ?
>>>
>
>
>>> --
>>> --
>>> cordialement, regards,
>>> Emmanuel Lécharny
>>> www.iktek.com
>>> directory.apache.org
>>>
>
> They are actually a variety of printers - airline data specific printers -
> Unimark ET6000, IER BPR640, TI1600, among others. The firmware I am
> referring to, was built years ago to talk to a Unisys DCP (1980's
> technology) which was used as a frontend to talk to a IBM TPF host. The
> legacy of the DCP was that it offered IP connectivity to the printer, and
> could do the necessary character translations to go from ASCII to EBCDIC.
> These are all issues that your typical developer shouldn't need to worry
> about these days...
>
yep :) I was just asking because I was used to manage such printers, 20
years ago (not exactly the same, but close...). You think it's there for
a couple of years, and it stays for ever ;)
> In the end we will replace all of these printers, but we need an interim
> solution.
>
Makes sense.
--
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org
Re: CumulativeProtocolDecoder
Posted by jclabat <Je...@nwa.com>.
>>Which kind of printer is it ?
>>--
>>--
>>cordialement, regards,
>>Emmanuel Lécharny
>>www.iktek.com
>>directory.apache.org
They are actually a variety of printers - airline data specific printers -
Unimark ET6000, IER BPR640, TI1600, among others. The firmware I am
referring to, was built years ago to talk to a Unisys DCP (1980's
technology) which was used as a frontend to talk to a IBM TPF host. The
legacy of the DCP was that it offered IP connectivity to the printer, and
could do the necessary character translations to go from ASCII to EBCDIC.
These are all issues that your typical developer shouldn't need to worry
about these days...
In the end we will replace all of these printers, but we need an interim
solution.
Jerry
--
View this message in context: http://www.nabble.com/CumulativeProtocolDecoder-tp18777497p19316816.html
Sent from the Apache MINA User Forum mailing list archive at Nabble.com.
Re: CumulativeProtocolDecoder
Posted by Emmanuel Lecharny <el...@gmail.com>.
jclabat wrote:
> Emmanuel Lecharny-3 wrote:
>
>>
>>> Anyone see any obvious issues with this?
>>>
>>>
>> How long will you have to wait to be sure that you have received a full
>> message ? And how do you know that it's full ? Seems to be a very losy
>> protocol you are dealing with :)
>>
>>> Jerry
>>>
>>>
>> --
>> --
>> cordialement, regards,
>> Emmanuel Lécharny
>> www.iktek.com
>> directory.apache.org
>>
>>
>>
>>
>>
>
> When Mina splits the messages apart into multiple buffers, they arrive
> within milliseconds of each other, so a timeout of a second or two easily
> covers it.
>
> Yes it is a lousy protocol, but I can't do anything about it because it is
> an embedded protocol in the firmware of printers that no longer have a
> manufacturing vendor to support.
>
Which kind of printer is it ?
--
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org
Re: CumulativeProtocolDecoder
Posted by jclabat <Je...@nwa.com>.
Emmanuel Lecharny-3 wrote:
>
>
>>
>> Anyone see any obvious issues with this?
>>
> How long will you have to wait to be sure that you have received a full
> message ? And how do you know that it's full ? Seems to be a very losy
> protocol you are dealing with :)
>>
>> Jerry
>>
>
>
> --
> --
> cordialement, regards,
> Emmanuel Lécharny
> www.iktek.com
> directory.apache.org
>
>
>
>
When Mina splits the messages apart into multiple buffers, they arrive
within milliseconds of each other, so a timeout of a second or two easily
covers it.
Yes it is a lousy protocol, but I can't do anything about it because it is
an embedded protocol in the firmware of printers that no longer have a
manufacturing vendor to support.
Jerry
--
View this message in context: http://www.nabble.com/CumulativeProtocolDecoder-tp18777497p19308797.html
Sent from the Apache MINA User Forum mailing list archive at Nabble.com.
Re: CumulativeProtocolDecoder
Posted by Emmanuel Lecharny <el...@gmail.com>.
>
> Anyone see any obvious issues with this?
>
How long will you have to wait to be sure that you have received a full
message ? And how do you know that it's full ? Seems to be a very losy
protocol you are dealing with :)
>
> Jerry
>
--
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org
Re: CumulativeProtocolDecoder
Posted by jclabat <Je...@nwa.com>.
I solved this by setting a read idle timer to a small value when I think I
have a partial message. I capture the previous idle timer value and a copy
of the input Mina ByteBuffer into an object that I attach to the session,
and return false. If I come back through the decoder with more data before
timing out, and I believe it is the end of the data (in my case lim < cap),
I get rid of the object attached to session, reset the idle timer, write it
and return true. Because the CumulativeProtocolDecoder hangs of off a Mina
filter, I implement the SessionIdle method on this particular filter, in the
SessionIdle method (this is also the first SessionIdle method implemented in
my filter chain), I copy the message again into a new ByteBuffer, call the
nextFilter.messageReceived with my new ByteBuffer, reset my idle timer to
the saved amount, and remove my temporary object from the session. I ended
up copying to a new ByteBuffer, because when I didn't I got a buffer
released multiple times exception.
I am not positive that this solves all of my issues, but initial testing
seemed to go well. I was surprised that I didn't have to do anything to
avoid concatenating my current message with the next, since I returned false
in my CumulativeProtocolDecoder implementation, but it seemed to go away on
its own when I called the nextFilter.messageReceived. I am sure this
implementation causes a performance hit, but I have lots of connections that
don't send many messages, so causing a connection to wait a second or two
isn't really a problem.
I also wasn't sure why I saw exceptions for multiple released buffers when I
used the ByteBuffer I had attached to the object I created in the session,
I realize that I can't do this with buffers/messages passed into the various
filter/encoder/decoder methods. Perhaps that just shows that I don't quite
know what I am doing yet...
Anyone see any obvious issues with this?
Jerry
--
View this message in context: http://www.nabble.com/CumulativeProtocolDecoder-tp18777497p19301228.html
Sent from the Apache MINA User Forum mailing list archive at Nabble.com.
Re: CumulativeProtocolDecoder
Posted by "W.B. Garvelink" <ba...@gmail.com>.
If there's no message terminator whatsoever, you have to work
time-based (can't think of any other way, but maybe that's just me).
In that case you're looking at is basically a two-state state machine
with a "wait for first data" state and an "accumulate until timeout
then reset" state. If I understand your problem correctly, I think
something along these lines might do the trick. The pseudocode is a
bit nasty, but you get the idea.
// used instead of CumulativeProtocolDecoder, not alongside it
public class MyFilter extends IoFilterAdapter {
/** Helper class for MyFilter, an instance of this class is bound to
every IoSession */
private static class Accumulator {
// synchronize all access to these fields, as we'll be using two threads!
private final IoBuffer accumulator = newBuffer();
private boolean setTimeout = true;
private static IoBuffer newBuffer() {
// return a new IoBuffer of reasonable size and maybe setAutoExpand(true)
}
public synchronized boolean receive(IoBuffer message) {
accumulator.put(message);
if (setTimeout) {
setTimeout = false;
return true;
}
return false;
}
public synchronized IoBuffer flush() {
setTimeout = true;
IoBuffer returnval = accumulator;
accumulator = newBuffer();
returnval.flip();
return returnval;
}
}
private final ScheduledExecutorService scheduler =
Executors.newScheduledThreadPool();
private final static long TIMEOUT = 250L; //Whatever is reasonable
@Override public void sessionCreated() {
// insert a new Accumulator into the session
}
@Override public void messageReceived(final NextFilter nf, final
Session ssn, Object msg) {
final Accumulator accu = // retrieve accumulator from session
if (accu.receive( (IoBuffer)msg )) {
scheduler.schedule( new Runnable() {
public void run() {
nf.messageReceived(ssn, accu.flush());
}
}, TIMEOUT, TimeUnit.MILLISECONDS);
}
// done
}
// override sessionClosed() to cancel any scheduled tasks
}
HTH,
Barend
On Tue, Aug 5, 2008 at 1:32 PM, lnaber <lo...@gmail.com> wrote:
>>>>Are there *any* rules that allow detection of message boundaries? For
>>>>example, is there a time interval between messages?
>>
>> The application is conversational in that a request-response is expected
>> so yes, there is timing available.
>>
Re: CumulativeProtocolDecoder
Posted by lnaber <lo...@gmail.com>.
lnaber wrote:
>
>
> Tuure Laurinolli-2 wrote:
>>
>>
>>>Are there *any* rules that allow detection of message boundaries? For
>>>example, is there a time interval between messages? Is a message always
>>>sent in a single IP packet, and is there enough knowledge of the
>>>underlying protocol stack to assume that there will be no fragmentation?
>>
>>>If there is a time interval between messages, you could probably start a
>>>timer when data is received, buffering the data received until the timer
>>>fires, and then pass the buffered data onwards.
>>
>>
>
> The application is conversational in that a request-response is expected
> so yes, there is timing available.
>
> I like the time interval idea but I'm not sure how to insert that into the
> MINA objects (session, filter) life cycle events. Potentially the
> session's
> idle timer could be used to detect message boundaries but how does
> one insert that back into the CumulativeProtocolDecoder's event to
> process and complete the message flow through the filter stack?
>
Is it possible to set & reset the session idle timer during operation? For
detecting a subset of messages that fall into the scenario described earlier
it would be good to timeout very quickly (100 ms) but that would not be the
optimal setting for the overall operation. So the CumulativeProtocolDecoder
subclass could, when it detected the fragmented message, set the timer very
short until the fragment is located, then set the timer back to a more
reasonable value.
--
View this message in context: http://www.nabble.com/CumulativeProtocolDecoder-tp18777497p18829198.html
Sent from the Apache MINA User Forum mailing list archive at Nabble.com.
Re: CumulativeProtocolDecoder
Posted by lnaber <lo...@gmail.com>.
Tuure Laurinolli-2 wrote:
>
>
>>Are there *any* rules that allow detection of message boundaries? For
>>example, is there a time interval between messages? Is a message always
>>sent in a single IP packet, and is there enough knowledge of the
>>underlying protocol stack to assume that there will be no fragmentation?
>
>>If there is a time interval between messages, you could probably start a
>>timer when data is received, buffering the data received until the timer
>>fires, and then pass the buffered data onwards.
>
>
The application is conversational in that a request-response is expected
so yes, there is timing available.
I like the time interval idea but I'm not sure how to insert that into the
MINA objects (session, filter) life cycle events. Potentially the session's
idle timer could be used to detect message boundaries but how does
one insert that back into the CumulativeProtocolDecoder's event to
process and complete the message flow through the filter stack?
--
View this message in context: http://www.nabble.com/CumulativeProtocolDecoder-tp18777497p18828914.html
Sent from the Apache MINA User Forum mailing list archive at Nabble.com.
RE: Mina Throttling
Posted by Arv Mistry <ar...@kindsight.net>.
Geoff/Guo
That's great, thanks.
Cheers Arv
-----Original Message-----
From: Wenrui Guo [mailto:wenrui.guo@ericsson.com]
Sent: Friday, August 08, 2008 12:30 AM
To: users@mina.apache.org; geoff@thecadiens.org
Subject: RE: Mina Throttling
Yes, Mina has a very useful IoFilter named ReadThrottleFilter, who
relies on IoSession.suspendRead() to take advantage of TCP flow control.
You can use following code to achieve such objective:
Connector = new NioSocketConnector();
Connector.getFilterChain().addLast("exec", new ExecutorFilter());
Connector.getFilterChain().addLast("throttle", new
ReadThrottleFilter(throttleExecutor, ReadThrottlePolicy.BLOCK));
Remember, throttleExecutor is a ScheduledExecutorService instance,
programmer must be reponsible for such executor's lifecycle. In other
words, you should shutdown it later when application is going to
shutdown.
BR
anderson
-----Original Message-----
From: Geoff Cadien [mailto:gcadien@gmail.com]
Sent: Thursday, August 07, 2008 12:02 AM
To: users@mina.apache.org
Subject: Re: Mina Throttling
On Wed, Aug 6, 2008 at 6:11 AM, Arv Mistry <ar...@kindsight.net> wrote:
>
> Hi,
>
> I have a system which sends a very high throughput of messages. So one
> thing I wanted to be able to do during high loads is to take advantage
> of the TCP flow control i.e be able to throttle back on the TCP socket
> so as not to read off it until the processing catches up.
>
> Does anyone know is there way with Mina to tell it not to read from
> the TCP socket? If I return false in the doDecode of the
> ProtocolDecoder will that do it?
IoSession.suspendRead() will accomplish what you're asking for a single
tcp connection.
-geoff
RE: Mina Throttling
Posted by Wenrui Guo <we...@ericsson.com>.
Yes, Mina has a very useful IoFilter named ReadThrottleFilter, who
relies on IoSession.suspendRead() to take advantage of TCP flow control.
You can use following code to achieve such objective:
Connector = new NioSocketConnector();
Connector.getFilterChain().addLast("exec", new ExecutorFilter());
Connector.getFilterChain().addLast("throttle", new
ReadThrottleFilter(throttleExecutor, ReadThrottlePolicy.BLOCK));
Remember, throttleExecutor is a ScheduledExecutorService instance,
programmer must be reponsible for such executor's lifecycle. In other
words, you should shutdown it later when application is going to
shutdown.
BR
anderson
-----Original Message-----
From: Geoff Cadien [mailto:gcadien@gmail.com]
Sent: Thursday, August 07, 2008 12:02 AM
To: users@mina.apache.org
Subject: Re: Mina Throttling
On Wed, Aug 6, 2008 at 6:11 AM, Arv Mistry <ar...@kindsight.net> wrote:
>
> Hi,
>
> I have a system which sends a very high throughput of messages. So one
> thing I wanted to be able to do during high loads is to take advantage
> of the TCP flow control i.e be able to throttle back on the TCP socket
> so as not to read off it until the processing catches up.
>
> Does anyone know is there way with Mina to tell it not to read from
> the TCP socket? If I return false in the doDecode of the
> ProtocolDecoder will that do it?
IoSession.suspendRead() will accomplish what you're asking for a single
tcp connection.
-geoff
Re: Mina Throttling
Posted by Geoff Cadien <gc...@gmail.com>.
On Wed, Aug 6, 2008 at 6:11 AM, Arv Mistry <ar...@kindsight.net> wrote:
>
> Hi,
>
> I have a system which sends a very high throughput of messages. So one
> thing I wanted to be able to do during high loads is to take advantage
> of the TCP flow control i.e be able to throttle back on the TCP socket
> so as not to read off it until the processing catches up.
>
> Does anyone know is there way with Mina to tell it not to read from the
> TCP socket? If I return false in the doDecode of the ProtocolDecoder
> will that do it?
IoSession.suspendRead() will accomplish what you're asking for a single tcp
connection.
-geoff
Mina Throttling
Posted by Arv Mistry <ar...@kindsight.net>.
Hi,
I have a system which sends a very high throughput of messages. So one
thing I wanted to be able to do during high loads is to take advantage
of the TCP flow control i.e be able to throttle back on the TCP socket
so as not to read off it until the processing catches up.
Does anyone know is there way with Mina to tell it not to read from the
TCP socket? If I return false in the doDecode of the ProtocolDecoder
will that do it?
Cheers Arv
-----Original Message-----
From: Tuure Laurinolli [mailto:tuure.laurinolli@indagon.com]
Sent: Friday, August 01, 2008 3:39 PM
To: users@mina.apache.org
Subject: Re: CumulativeProtocolDecoder
Louis Naber wrote:
> I have an unfortunate scenario where I am dealing with legacy output
> peripherals that do not have nicely designed messaging built into
> them. The messages are variable length, unterminated, with no header
information.
Are there *any* rules that allow detection of message boundaries? For
example, is there a time interval between messages? Is a message always
sent in a single IP packet, and is there enough knowledge of the
underlying protocol stack to assume that there will be no fragmentation?
> The question I have, is it possible to find out if more data is
> available for the session when the lim=capacity just to make sure that
> the incomplete message assumption is correct?
If there is a time interval between messages, you could probably start a
timer when data is received, buffering the data received until the timer
fires, and then pass the buffered data onwards.
Re: CumulativeProtocolDecoder
Posted by Tuure Laurinolli <tu...@indagon.com>.
Louis Naber wrote:
> I have an unfortunate scenario where I am dealing with legacy output
> peripherals that do not have nicely designed messaging built into them. The
> messages are variable length, unterminated, with no header information.
Are there *any* rules that allow detection of message boundaries? For
example, is there a time interval between messages? Is a message always
sent in a single IP packet, and is there enough knowledge of the
underlying protocol stack to assume that there will be no fragmentation?
> The question I have, is it possible to find out if more data is available
> for the session when the lim=capacity just to make sure that the incomplete
> message assumption is correct?
If there is a time interval between messages, you could probably start a
timer when data is received, buffering the data received until the timer
fires, and then pass the buffered data onwards.