CQL3 Frame Length

[Ben Hood <0x...@gmail.com>]

Hi,

I've read the CQL wire specification and naively, I can't see how the frame
length length header is used.

To me, it looks like on the read side, you know which type of structures to
expect based on the opcode and each structure is TLV encoded.

On the write side, you need to encode TLV structures as well, but you don't
know the overall frame length until you've encoded it. So it would seem
that you either need to pre-calculate the cumulative TLV size before you
serialize the frame body, or you serialize the frame body to a buffer which
you can then get the size of and then write to the socket, after having
first written the count out.

Is there potentially an implicit assumption that the reader will want to
pre-buffer the entire frame before decoding it?

Cheers,

Ben

Re: CQL3 Frame Length

[Ben Hood <0x...@gmail.com>]

Hey Sylvain,

Thanks for explaining the rationale. When you look at from the perspective
of the use cases you mention, it makes sense to be able to supply the
reader with the frame size up front.

I've opted to go for serializing the frame into a buffer. Although this
could materialize an arbitrarily large amount of memory, ultimately the
driving application has control of the degree to which this can occur, so
in the grander scheme of things, you can still maintain streaming semantics.

Thanks for the heads up.

Cheers,

Ben


On Tue, Jan 8, 2013 at 4:08 PM, Sylvain Lebresne <sy...@datastax.com>wrote:

> Mostly this is because having the frame length is convenient to have in
> practice.
>
> Without pretending that there is only one way to write a server, it is
> common
> to separate the phase "read a frame from the network" from the phase
> "decode
> the frame" which is often simpler if you can read the frame upfront. Also,
> if
> you don't have the frame size, it means you need to decode the whole frame
> before being able to decode the next one, and so you can't parallelize the
> decoding.
>
> It is true however that it means for the write side that you need to
> either be
> able to either pre-compute the frame body size or to serialize it in memory
> first. That's a trade of for making it easier on the read side. But if you
> want
> my opinion, on the write side too it's probably worth parallelizing the
> message
> encoding (which require you encode it in memory first) since it's an
> asynchronous protocol and so there will likely be multiple writer
> simultaneously.
>
> --
> Sylvain
>
>
>
> On Tue, Jan 8, 2013 at 12:48 PM, Ben Hood <0x...@gmail.com> wrote:
>
>> Hi,
>>
>> I've read the CQL wire specification and naively, I can't see how the
>> frame length length header is used.
>>
>> To me, it looks like on the read side, you know which type of structures
>> to expect based on the opcode and each structure is TLV encoded.
>>
>> On the write side, you need to encode TLV structures as well, but you
>> don't know the overall frame length until you've encoded it. So it would
>> seem that you either need to pre-calculate the cumulative TLV size before
>> you serialize the frame body, or you serialize the frame body to a buffer
>> which you can then get the size of and then write to the socket, after
>> having first written the count out.
>>
>> Is there potentially an implicit assumption that the reader will want to
>> pre-buffer the entire frame before decoding it?
>>
>> Cheers,
>>
>> Ben
>>
>
>

Re: CQL3 Frame Length

[Sylvain Lebresne <sy...@datastax.com>]

Mostly this is because having the frame length is convenient to have in
practice.

Without pretending that there is only one way to write a server, it is
common
to separate the phase "read a frame from the network" from the phase "decode
the frame" which is often simpler if you can read the frame upfront. Also,
if
you don't have the frame size, it means you need to decode the whole frame
before being able to decode the next one, and so you can't parallelize the
decoding.

It is true however that it means for the write side that you need to either
be
able to either pre-compute the frame body size or to serialize it in memory
first. That's a trade of for making it easier on the read side. But if you
want
my opinion, on the write side too it's probably worth parallelizing the
message
encoding (which require you encode it in memory first) since it's an
asynchronous protocol and so there will likely be multiple writer
simultaneously.

--
Sylvain



On Tue, Jan 8, 2013 at 12:48 PM, Ben Hood <0x...@gmail.com> wrote:

> Hi,
>
> I've read the CQL wire specification and naively, I can't see how the
> frame length length header is used.
>
> To me, it looks like on the read side, you know which type of structures
> to expect based on the opcode and each structure is TLV encoded.
>
> On the write side, you need to encode TLV structures as well, but you
> don't know the overall frame length until you've encoded it. So it would
> seem that you either need to pre-calculate the cumulative TLV size before
> you serialize the frame body, or you serialize the frame body to a buffer
> which you can then get the size of and then write to the socket, after
> having first written the count out.
>
> Is there potentially an implicit assumption that the reader will want to
> pre-buffer the entire frame before decoding it?
>
> Cheers,
>
> Ben
>