Re: heterogeneous collections

[ma...@bestweb.net]

Some thoughts on Thrift & its encoding style:
- Thrift, at the wire level, encodes values with an accompanying type.
- The thrift RPC demarshallers are told the type they are expecting
- The received message may not be the expected type.
- What happens if there is an incompatibility? Thrift appears to only
define two situations:
  1. a structure expects less fields than are actually sent; expected
behavior is to discard the fields
  2. a structure expects more fields than are actually sent; expected
behavior is to leave the unsent fields uninitialized (or set to some
default value).
- Cases which might have been defined but are not are things such as:
expect a i32 but was on the wire as i16.

In some sense, at the wire level, Thrift can be thought of as sending
messages that are
* scalars (bool,byte,i16,i32,i64,double,strings?)
* structs that are maps from field numbers to values
* lists (&sets) that are sequences of values
* maps that are sequences of tuples.

On the wire, Thrift is permissive enough to send a value of the type
LIST 2 STRUCT 1 I16 2 I32 STOP 1 DOUBLE 5 STRING STOP
or
list { struct {1: I16, 2:I32}, struct{ 1:double, 5:string}}
In that sense, it will support list<any>.

However, in a typed language (and probably in any language other than one
with duck-typing), it will be difficult (or impossible) to convert the
received object into an acceptable object.

[[ This also suggests that there is a possible compression available here:
specify complete types as arguments to containers, as opposed to repeating
them for each element. Taken to its conclusion, however, this requires
sending the type signature in advance of the actual message, as opposed to
interleaved with it.

Note that, with that one exception, it seems unclear where you could save
logical space in the type-encoding overhead, and still retain the ability
to change the number of fields & their types and then detect that there is
an incompatiblity.

You may, of course, be able to compress the logical type information, as
well as the values transmitted.]]

So, the problem with list<any> is not necessarily the wire encoding, but
specifying the expected type -- what should the demarshaller turn it into?


> Hi,
>
> I'd like to use an heterogeneous collection, e.g. list<any>, in a service.
>
> I see there's been some work/discussion on this earlier,
> https://issues.apache.org/jira/browse/THRIFT-122
> https://issues.apache.org/jira/browse/THRIFT-110
>
> Now THRIFT-110 is committed and claims to incorporate THRIFT-122.
>
> I've looked around but couldn't find the answer... how do I declare an
> heterogeneous collection in a .thrift definition?
>
> alex
>

Re: heterogeneous collections

[Alex Boisvert <al...@bizo.com>]

On Sun, May 2, 2010 at 10:28 PM, <ma...@bestweb.net> wrote:

> On the wire, Thrift is permissive enough to send a value of the type
> LIST 2 STRUCT 1 I16 2 I32 STOP 1 DOUBLE 5 STRING STOP
> or
> list { struct {1: I16, 2:I32}, struct{ 1:double, 5:string}}
> In that sense, it will support list<any>.
>
> However, in a typed language (and probably in any language other than one
> with duck-typing), it will be difficult (or impossible) to convert the
> received object into an acceptable object.
>

Hmmm... why do you think so?  It's usually pretty trivial (though not always
pretty) to do these types of raw-data to typed-objects conversions.

[[ This also suggests that there is a possible compression available here:
> specify complete types as arguments to containers, as opposed to repeating
> them for each element. Taken to its conclusion, however, this requires
> sending the type signature in advance of the actual message, as opposed to
> interleaved with it.
>

If the collections are homogeneous, yes.   If not, then the current encoding
is just fine.  I'd argue that  the current encoding is just fine since I
need support for heterogeneous collections ;)

Note that, with that one exception, it seems unclear where you could save
> logical space in the type-encoding overhead, and still retain the ability
> to change the number of fields & their types and then detect that there is
> an incompatiblity.
>
> You may, of course, be able to compress the logical type information, as
> well as the values transmitted.]]
>
> So, the problem with list<any> is not necessarily the wire encoding, but
> specifying the expected type -- what should the demarshaller turn it into?
>

A generic list (or more generally, a generic collection/container).   As far
as I know, all languages supported by Thrift also support generic +
heterogeneous collections.

alex

Re: heterogeneous collections

[Alex Boisvert <al...@bizo.com>]

On Sun, May 2, 2010 at 10:28 PM, <ma...@bestweb.net> wrote:

> On the wire, Thrift is permissive enough to send a value of the type
> LIST 2 STRUCT 1 I16 2 I32 STOP 1 DOUBLE 5 STRING STOP
> or
> list { struct {1: I16, 2:I32}, struct{ 1:double, 5:string}}
> In that sense, it will support list<any>.
>
> However, in a typed language (and probably in any language other than one
> with duck-typing), it will be difficult (or impossible) to convert the
> received object into an acceptable object.
>

Hmmm... why do you think so?  It's usually pretty trivial (though not always
pretty) to do these types of raw-data to typed-objects conversions.

[[ This also suggests that there is a possible compression available here:
> specify complete types as arguments to containers, as opposed to repeating
> them for each element. Taken to its conclusion, however, this requires
> sending the type signature in advance of the actual message, as opposed to
> interleaved with it.
>

If the collections are homogeneous, yes.   If not, then the current encoding
is just fine.  I'd argue that  the current encoding is just fine since I
need support for heterogeneous collections ;)

Note that, with that one exception, it seems unclear where you could save
> logical space in the type-encoding overhead, and still retain the ability
> to change the number of fields & their types and then detect that there is
> an incompatiblity.
>
> You may, of course, be able to compress the logical type information, as
> well as the values transmitted.]]
>
> So, the problem with list<any> is not necessarily the wire encoding, but
> specifying the expected type -- what should the demarshaller turn it into?
>

A generic list (or more generally, a generic collection/container).   As far
as I know, all languages supported by Thrift also support generic +
heterogeneous collections.

alex