What is a best way of decoding variable size records?

[Andrei Palskoi <a_...@hotmail.com>]

Hi, I am new to this forum. I have some code that converts COBOL copybook
records, which can have variable length, into Java objects. For some reasons
the byte length of the record cannot be passed up front. So my existing code
uses blocking I/O (buffered InputStream) and it works pretty well.
I have a server around it that was built on top of Apache Phoenix code. The
Phoenix project is no longer supported so I've decided to move to Apache
MINA which looks pretty cool too. However, I am not sure what is the best
way of integrating my code into it.

As far as I can see, there are two ways:

1) Implement a subclass of StreamIoHandler and get back to good old blocking
I/O. But I can see that experts here recommend not doing so unless
absolutely necessary. Also, why Handler and not Filter?
2) Implement a subclass of CumulativeProtocolDecoder that will try to decode
message and if there is not enough bytes available yet, return false so that
MINA comes back with more data. Well, that seems highly ineffective,
especially when working with large records - you have to throw away all the
work and redo it again when more data becomes available.

Suggestions?
Thanks, Andrei
-- 
View this message in context: http://www.nabble.com/What-is-a-best-way-of-decoding-variable-size-records--tf4518931s16868.html#a12890468
Sent from the Apache MINA Support Forum mailing list archive at Nabble.com.

Re: What is a best way of decoding variable size records?

[Andrei Palskoi <a_...@hotmail.com>]

Well, the problem is that my parser uses recursive calls (records can have
subrecords which can be collections of subrecords...) and if it hits the end
of available data, it is pretty hard to bookmark its state and make it jump
back to the same point once more data is available. Or so I think - it's
been a while since I wrote it...

The way how the code knows how much data is needed is pretty tricky.
Basically, at any node in a record tree it can tell how many children will
follow, and for each child it will be able to tell whether its byte size is
fixed or dynamic (i.e. it can have variable number of descendant nodes
itself).

It's the same as parsing a huge XML file using streaming API. If you have a
state engine, you can come back to the same point with more data. But my
parser simply tries to read as much data from the InputStream as needed and
throws EOFException if SO_TIMEOUT on read() has been exceeded.

I guess I better add some statefulness to it.


Maarten Bosteels-4 wrote:
> 
> Hello Andrei,
> 
> I think the CumulativeProtocolDecoder is the way to go.
> You don't necessarily have to "throw away all the work", since your
> decoder could be stateful. When it has partially decoded a record, it
> can store this data
> and when more data becomes available, continue from there.
> 
> How does your existing code determine that you have enough data
> to decode a record ?
> 
> Maarten
> 
> On 9/26/07, Andrei Palskoi <a_...@hotmail.com> wrote:
>>
>> Hi, I am new to this forum. I have some code that converts COBOL copybook
>> records, which can have variable length, into Java objects. For some
>> reasons
>> the byte length of the record cannot be passed up front. So my existing
>> code
>> uses blocking I/O (buffered InputStream) and it works pretty well.
>> I have a server around it that was built on top of Apache Phoenix code.
>> The
>> Phoenix project is no longer supported so I've decided to move to Apache
>> MINA which looks pretty cool too. However, I am not sure what is the best
>> way of integrating my code into it.
>>
>> As far as I can see, there are two ways:
>>
>> 1) Implement a subclass of StreamIoHandler and get back to good old
>> blocking
>> I/O. But I can see that experts here recommend not doing so unless
>> absolutely necessary. Also, why Handler and not Filter?
>> 2) Implement a subclass of CumulativeProtocolDecoder that will try to
>> decode
>> message and if there is not enough bytes available yet, return false so
>> that
>> MINA comes back with more data. Well, that seems highly ineffective,
>> especially when working with large records - you have to throw away all
>> the
>> work and redo it again when more data becomes available.
>>
>> Suggestions?
>> Thanks, Andrei
>> --
>> View this message in context:
>> http://www.nabble.com/What-is-a-best-way-of-decoding-variable-size-records--tf4518931s16868.html#a12890468
>> Sent from the Apache MINA Support Forum mailing list archive at
>> Nabble.com.
>>
>>
> 
> 

-- 
View this message in context: http://www.nabble.com/What-is-a-best-way-of-decoding-variable-size-records--tf4518931s16868.html#a12903011
Sent from the Apache MINA Support Forum mailing list archive at Nabble.com.

Re: What is a best way of decoding variable size records?

[Maarten Bosteels <mb...@gmail.com>]

Hello Andrei,

I think the CumulativeProtocolDecoder is the way to go.
You don't necessarily have to "throw away all the work", since your
decoder could be stateful. When it has partially decoded a record, it
can store this data
and when more data becomes available, continue from there.

How does your existing code determine that you have enough data
to decode a record ?

Maarten

On 9/26/07, Andrei Palskoi <a_...@hotmail.com> wrote:
>
> Hi, I am new to this forum. I have some code that converts COBOL copybook
> records, which can have variable length, into Java objects. For some reasons
> the byte length of the record cannot be passed up front. So my existing code
> uses blocking I/O (buffered InputStream) and it works pretty well.
> I have a server around it that was built on top of Apache Phoenix code. The
> Phoenix project is no longer supported so I've decided to move to Apache
> MINA which looks pretty cool too. However, I am not sure what is the best
> way of integrating my code into it.
>
> As far as I can see, there are two ways:
>
> 1) Implement a subclass of StreamIoHandler and get back to good old blocking
> I/O. But I can see that experts here recommend not doing so unless
> absolutely necessary. Also, why Handler and not Filter?
> 2) Implement a subclass of CumulativeProtocolDecoder that will try to decode
> message and if there is not enough bytes available yet, return false so that
> MINA comes back with more data. Well, that seems highly ineffective,
> especially when working with large records - you have to throw away all the
> work and redo it again when more data becomes available.
>
> Suggestions?
> Thanks, Andrei
> --
> View this message in context: http://www.nabble.com/What-is-a-best-way-of-decoding-variable-size-records--tf4518931s16868.html#a12890468
> Sent from the Apache MINA Support Forum mailing list archive at Nabble.com.
>
>