[Avatica] Separating wire message from service implementation

[Josh Elser <jo...@gmail.com>]

Hi,

I have some ideas which I'd like to try to working on to improve 
Avatica. My direct interest is in support of the Phoenix Query Server 
which builds on top of Avatica.

One of the big areas of improvement that stands out to me is the 
Jackson-annotated Response/Request classes. First, we'd want to make 
sure these classes are stable (in terms of fields/attributes) so that, 
as each object evolves, we prevent existing clients from breaking. 
Second, it's desirable that we have some representation of these objects 
in a manner that's accessible to non-Java clients.

Ideally, preserving JSON as the serialized representation of these 
classes is also desirable as it's always pleasant to actually be able to 
read the data going over the wire when necessary.

Given these goals, two tools stood out to me as candidates that would 
solve this problem well without extra effort in the transport: Thrift 
and Protobuf v3.

Using Thrift only for object generation is a bit overkill, but this is 
an area of Thrift that has been rock-solid in the years that I've used 
it. Classpath issues with Thrift are always a concern as multiple 
versions of Thrift tend to break terribly (at least when performing 
Thrift RPCs -- I wouldn't necessarily trust it to be safe for only 
object serialization). It's easy to serialize Thrift objects to JSON and 
we can generate these classes for other languages.

I would call Protobuf v3 very young, although it does appear to fit the 
bill. Protobuf v2 has had a track record of being a pain when different 
versions of the library exist. For example, the Java classes generated 
by protobuf 2.x would fail to work with protobuf 2.y. I assume there's 
not enough evidence on protobuf v3 to say whether this will continue to 
be a problem. JSON-representation of objects and cross-language support 
are both natively supported (best as I can tell). Hypothetically, 
adopting protobufv3 could also have some extra benefits down the road 
such as leveraging grpc to replace Jetty, but that is an entirely 
different discussion for later :)

If you've gotten this far, I'd humbly ask for any opinions on the 
technology choices. Any opinions for or against one or the other? Any 
similar experiences?

Thanks!

- Josh

Re: [Avatica] Separating wire message from service implementation

[Josh Elser <jo...@gmail.com>]

Thanks, Julian! I meant to make one, just hadn't gotten around to it.

Julian Hyde wrote:
> I have logged https://issues.apache.org/jira/browse/CALCITE-839<https://issues.apache.org/jira/browse/CALCITE-839>  for this task.
>
>> On Aug 10, 2015, at 8:27 PM, Josh Elser<jo...@gmail.com>  wrote:
>>
>> Julian Hyde wrote:
>>> If my understand is correct, this decision is an easy one to make, because whatever we choose, we still comply with the specification.
>>>
>>> Avatica specifies its API at two levels: (1) the Java API (Meta), (2) the JSON requests/responses going over the wire.
>>>
>>> I would accept any transport that complies with both of those levels. In principle we could write a JSON-over-HTTP-marshalled-via-Jackson transport, specifying the mappings elsewhere than in the Meta classes, and someone else could use another transport.
>> Assuming by "specifying the mappings elsewhere" you mean "by a means other than POJOs", yes, I'd agree. I feel like the POJOs are bound to eventually cause troubles across versions (as it's very easy to accidentally make some change that breaks something you forgot you did before).
>>
>> Any serialization library isn't foolproof obviously, but they are certainly much easier to work around than doing it all by hand.
>>
>>> Is it feasible to achieve this?
>>>
>>> Julian
>>>
>
>

Re: [Avatica] Separating wire message from service implementation

[Julian Hyde <jh...@apache.org>]

I have logged https://issues.apache.org/jira/browse/CALCITE-839 <https://issues.apache.org/jira/browse/CALCITE-839> for this task.

> On Aug 10, 2015, at 8:27 PM, Josh Elser <jo...@gmail.com> wrote:
> 
> Julian Hyde wrote:
>> If my understand is correct, this decision is an easy one to make, because whatever we choose, we still comply with the specification.
>> 
>> Avatica specifies its API at two levels: (1) the Java API (Meta), (2) the JSON requests/responses going over the wire.
>> 
>> I would accept any transport that complies with both of those levels. In principle we could write a JSON-over-HTTP-marshalled-via-Jackson transport, specifying the mappings elsewhere than in the Meta classes, and someone else could use another transport.
> 
> Assuming by "specifying the mappings elsewhere" you mean "by a means other than POJOs", yes, I'd agree. I feel like the POJOs are bound to eventually cause troubles across versions (as it's very easy to accidentally make some change that breaks something you forgot you did before).
> 
> Any serialization library isn't foolproof obviously, but they are certainly much easier to work around than doing it all by hand.
> 
>> Is it feasible to achieve this?
>> 
>> Julian
>> 

Re: [Avatica] Separating wire message from service implementation

[Josh Elser <jo...@gmail.com>]

Julian Hyde wrote:
> If my understand is correct, this decision is an easy one to make, because whatever we choose, we still comply with the specification.
>
> Avatica specifies its API at two levels: (1) the Java API (Meta), (2) the JSON requests/responses going over the wire.
>
> I would accept any transport that complies with both of those levels. In principle we could write a JSON-over-HTTP-marshalled-via-Jackson transport, specifying the mappings elsewhere than in the Meta classes, and someone else could use another transport.

Assuming by "specifying the mappings elsewhere" you mean "by a means 
other than POJOs", yes, I'd agree. I feel like the POJOs are bound to 
eventually cause troubles across versions (as it's very easy to 
accidentally make some change that breaks something you forgot you did 
before).

Any serialization library isn't foolproof obviously, but they are 
certainly much easier to work around than doing it all by hand.

> Is it feasible to achieve this?
>
> Julian
>

Re: [Avatica] Separating wire message from service implementation

[Julian Hyde <jh...@apache.org>]

If my understand is correct, this decision is an easy one to make, because whatever we choose, we still comply with the specification.

Avatica specifies its API at two levels: (1) the Java API (Meta), (2) the JSON requests/responses going over the wire.

I would accept any transport that complies with both of those levels. In principle we could write a JSON-over-HTTP-marshalled-via-Jackson transport, specifying the mappings elsewhere than in the Meta classes, and someone else could use another transport.

Is it feasible to achieve this? 

Julian

> On Aug 7, 2015, at 5:40 PM, Josh Elser <jo...@gmail.com> wrote:
> 
> Hrmph. Well, fun fact: that JSON serialization for pb3 apparently isn't implemented yet..
> 
> Josh Elser wrote:
>> :) I hear you. I don't think I have anything putting me in one camp or
>> the other aside from potential worries.
>> 
>> I can respect the Thrift "doing too much" notion -- kind of half of my
>> concern in using something like Thrift just to get object serialization
>> stuff (now I'm thinking I should help separate that out within thrift!)
>> I've used Thrift since the 0.6 timeframe and I don't remember once
>> coming across an issue where object serialization wasn't handled correctly.
>> 
>> I'll have to poke around pb3 docs some more to see if I can find some
>> talk on compatibility. I think at this point I'm leaning towards
>> protobuf, but we'll see.
>> 
>> Thanks again for taking the time to write up your thoughts, Jesse!
>> 
>> Jesse Yates wrote:
>>> Yup, totally get it. Honestly, I have no idea what the compat story is
>>> supposed to be for proto3. At the same time, I've got no hands on
>>> experience w/ Thrift either, beyond dabbling a little bit years ago, so I
>>> don't know what their on-the-wire-compat story is either :) I'm not as
>>> huge
>>> of a fan of thrift in general though because it tries to do too much IMO
>>> 
>>> If we do go the thrift route, but want to support grpc, we could do the
>>> work to make a thrift adapter, so picking one or the other now doesn't
>>> really preclude doing cool stuff later.
>>> 
>>> In short, it doesn't really matter? At least not to me.
>>> 
>>> On Thu, Aug 6, 2015 at 3:57 PM Josh Elser<jo...@gmail.com> wrote:
>>> 
>>>> Thanks for the feedback, Jesse!
>>>> 
>>>> To clarify, the compatibility I'm most worried about is backwards compat
>>>> in future versions of protobuf3. I hadn't considered v2 support into the
>>>> equation -- figured v3 is the starting point like you say. I know the
>>>> protobuf 2.4 to 2.5 upgrade in Hadoop/HBase land was rather scary that
>>>> required a lot of lock-step movement across many big projects. This is
>>>> my biggest fear with protobuf.
>>>> 
>>>> Despite fear of derailing conversation, grpc definitely has some nice
>>>> features built in (and implementing a service is definitely slick). I
>>>> think it's definitely a "later" since the single endpoint presently used
>>>> doesn't really require a complex service definition (with evolving args,
>>>> etc). My gut reaction is to wait to adopt such software until there is a
>>>> definite understanding of what the service would look like (which I
>>>> assume will become clearer as development continues).
>>>> 
>>>> Jesse Yates wrote:
>>>>> +1 for proto3. mostly for the later move to using grpc, which lets us
>>>>> leverage a battle tested framework for doing real protobuf services
>>>>> with
>>>>> fast streaming (how nice!).
>>>>> 
>>>>> proto3 can be backwards compat in some ways (eg. map implementation
>>>>> done
>>>>> manually in proto2). I don't think its a huge issue since there are
>>>>> currently no backwards compat guarantees anyways AFAIK.
>>>>> 
>>>>> 
>>>>> On Thu, Aug 6, 2015 at 3:11 PM Josh Elser<jo...@gmail.com> wrote:
>>>>> 
>>>>>> Hi,
>>>>>> 
>>>>>> I have some ideas which I'd like to try to working on to improve
>>>>>> Avatica. My direct interest is in support of the Phoenix Query Server
>>>>>> which builds on top of Avatica.
>>>>>> 
>>>>>> One of the big areas of improvement that stands out to me is the
>>>>>> Jackson-annotated Response/Request classes. First, we'd want to make
>>>>>> sure these classes are stable (in terms of fields/attributes) so that,
>>>>>> as each object evolves, we prevent existing clients from breaking.
>>>>>> Second, it's desirable that we have some representation of these
>>>>>> objects
>>>>>> in a manner that's accessible to non-Java clients.
>>>>>> 
>>>>>> Ideally, preserving JSON as the serialized representation of these
>>>>>> classes is also desirable as it's always pleasant to actually be
>>>>>> able to
>>>>>> read the data going over the wire when necessary.
>>>>>> 
>>>>>> Given these goals, two tools stood out to me as candidates that would
>>>>>> solve this problem well without extra effort in the transport: Thrift
>>>>>> and Protobuf v3.
>>>>>> 
>>>>>> Using Thrift only for object generation is a bit overkill, but this is
>>>>>> an area of Thrift that has been rock-solid in the years that I've used
>>>>>> it. Classpath issues with Thrift are always a concern as multiple
>>>>>> versions of Thrift tend to break terribly (at least when performing
>>>>>> Thrift RPCs -- I wouldn't necessarily trust it to be safe for only
>>>>>> object serialization). It's easy to serialize Thrift objects to
>>>>>> JSON and
>>>>>> we can generate these classes for other languages.
>>>>>> 
>>>>>> I would call Protobuf v3 very young, although it does appear to fit
>>>>>> the
>>>>>> bill. Protobuf v2 has had a track record of being a pain when
>>>>>> different
>>>>>> versions of the library exist. For example, the Java classes generated
>>>>>> by protobuf 2.x would fail to work with protobuf 2.y. I assume there's
>>>>>> not enough evidence on protobuf v3 to say whether this will
>>>>>> continue to
>>>>>> be a problem. JSON-representation of objects and cross-language
>>>>>> support
>>>>>> are both natively supported (best as I can tell). Hypothetically,
>>>>>> adopting protobufv3 could also have some extra benefits down the road
>>>>>> such as leveraging grpc to replace Jetty, but that is an entirely
>>>>>> different discussion for later :)
>>>>>> 
>>>>>> If you've gotten this far, I'd humbly ask for any opinions on the
>>>>>> technology choices. Any opinions for or against one or the other? Any
>>>>>> similar experiences?
>>>>>> 
>>>>>> Thanks!
>>>>>> 
>>>>>> - Josh
>>>>>> 
>>> 

Re: [Avatica] Separating wire message from service implementation

[Josh Elser <jo...@gmail.com>]

Hrmph. Well, fun fact: that JSON serialization for pb3 apparently isn't 
implemented yet..

Josh Elser wrote:
> :) I hear you. I don't think I have anything putting me in one camp or
> the other aside from potential worries.
>
> I can respect the Thrift "doing too much" notion -- kind of half of my
> concern in using something like Thrift just to get object serialization
> stuff (now I'm thinking I should help separate that out within thrift!)
> I've used Thrift since the 0.6 timeframe and I don't remember once
> coming across an issue where object serialization wasn't handled correctly.
>
> I'll have to poke around pb3 docs some more to see if I can find some
> talk on compatibility. I think at this point I'm leaning towards
> protobuf, but we'll see.
>
> Thanks again for taking the time to write up your thoughts, Jesse!
>
> Jesse Yates wrote:
>> Yup, totally get it. Honestly, I have no idea what the compat story is
>> supposed to be for proto3. At the same time, I've got no hands on
>> experience w/ Thrift either, beyond dabbling a little bit years ago, so I
>> don't know what their on-the-wire-compat story is either :) I'm not as
>> huge
>> of a fan of thrift in general though because it tries to do too much IMO
>>
>> If we do go the thrift route, but want to support grpc, we could do the
>> work to make a thrift adapter, so picking one or the other now doesn't
>> really preclude doing cool stuff later.
>>
>> In short, it doesn't really matter? At least not to me.
>>
>> On Thu, Aug 6, 2015 at 3:57 PM Josh Elser<jo...@gmail.com> wrote:
>>
>>> Thanks for the feedback, Jesse!
>>>
>>> To clarify, the compatibility I'm most worried about is backwards compat
>>> in future versions of protobuf3. I hadn't considered v2 support into the
>>> equation -- figured v3 is the starting point like you say. I know the
>>> protobuf 2.4 to 2.5 upgrade in Hadoop/HBase land was rather scary that
>>> required a lot of lock-step movement across many big projects. This is
>>> my biggest fear with protobuf.
>>>
>>> Despite fear of derailing conversation, grpc definitely has some nice
>>> features built in (and implementing a service is definitely slick). I
>>> think it's definitely a "later" since the single endpoint presently used
>>> doesn't really require a complex service definition (with evolving args,
>>> etc). My gut reaction is to wait to adopt such software until there is a
>>> definite understanding of what the service would look like (which I
>>> assume will become clearer as development continues).
>>>
>>> Jesse Yates wrote:
>>>> +1 for proto3. mostly for the later move to using grpc, which lets us
>>>> leverage a battle tested framework for doing real protobuf services
>>>> with
>>>> fast streaming (how nice!).
>>>>
>>>> proto3 can be backwards compat in some ways (eg. map implementation
>>>> done
>>>> manually in proto2). I don't think its a huge issue since there are
>>>> currently no backwards compat guarantees anyways AFAIK.
>>>>
>>>>
>>>> On Thu, Aug 6, 2015 at 3:11 PM Josh Elser<jo...@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I have some ideas which I'd like to try to working on to improve
>>>>> Avatica. My direct interest is in support of the Phoenix Query Server
>>>>> which builds on top of Avatica.
>>>>>
>>>>> One of the big areas of improvement that stands out to me is the
>>>>> Jackson-annotated Response/Request classes. First, we'd want to make
>>>>> sure these classes are stable (in terms of fields/attributes) so that,
>>>>> as each object evolves, we prevent existing clients from breaking.
>>>>> Second, it's desirable that we have some representation of these
>>>>> objects
>>>>> in a manner that's accessible to non-Java clients.
>>>>>
>>>>> Ideally, preserving JSON as the serialized representation of these
>>>>> classes is also desirable as it's always pleasant to actually be
>>>>> able to
>>>>> read the data going over the wire when necessary.
>>>>>
>>>>> Given these goals, two tools stood out to me as candidates that would
>>>>> solve this problem well without extra effort in the transport: Thrift
>>>>> and Protobuf v3.
>>>>>
>>>>> Using Thrift only for object generation is a bit overkill, but this is
>>>>> an area of Thrift that has been rock-solid in the years that I've used
>>>>> it. Classpath issues with Thrift are always a concern as multiple
>>>>> versions of Thrift tend to break terribly (at least when performing
>>>>> Thrift RPCs -- I wouldn't necessarily trust it to be safe for only
>>>>> object serialization). It's easy to serialize Thrift objects to
>>>>> JSON and
>>>>> we can generate these classes for other languages.
>>>>>
>>>>> I would call Protobuf v3 very young, although it does appear to fit
>>>>> the
>>>>> bill. Protobuf v2 has had a track record of being a pain when
>>>>> different
>>>>> versions of the library exist. For example, the Java classes generated
>>>>> by protobuf 2.x would fail to work with protobuf 2.y. I assume there's
>>>>> not enough evidence on protobuf v3 to say whether this will
>>>>> continue to
>>>>> be a problem. JSON-representation of objects and cross-language
>>>>> support
>>>>> are both natively supported (best as I can tell). Hypothetically,
>>>>> adopting protobufv3 could also have some extra benefits down the road
>>>>> such as leveraging grpc to replace Jetty, but that is an entirely
>>>>> different discussion for later :)
>>>>>
>>>>> If you've gotten this far, I'd humbly ask for any opinions on the
>>>>> technology choices. Any opinions for or against one or the other? Any
>>>>> similar experiences?
>>>>>
>>>>> Thanks!
>>>>>
>>>>> - Josh
>>>>>
>>

Re: [Avatica] Separating wire message from service implementation

[Josh Elser <jo...@gmail.com>]

:) I hear you. I don't think I have anything putting me in one camp or 
the other aside from potential worries.

I can respect the Thrift "doing too much" notion -- kind of half of my 
concern in using something like Thrift just to get object serialization 
stuff (now I'm thinking I should help separate that out within thrift!) 
I've used Thrift since the 0.6 timeframe and I don't remember once 
coming across an issue where object serialization wasn't handled correctly.

I'll have to poke around pb3 docs some more to see if I can find some 
talk on compatibility. I think at this point I'm leaning towards 
protobuf, but we'll see.

Thanks again for taking the time to write up your thoughts, Jesse!

Jesse Yates wrote:
> Yup, totally get it. Honestly, I have no idea what the compat story is
> supposed to be for proto3. At the same time, I've got no hands on
> experience w/ Thrift either, beyond dabbling a little bit years ago, so I
> don't know what their on-the-wire-compat story is either :) I'm not as huge
> of a fan of thrift in general though because it tries to do too much IMO
>
> If we do go the thrift route, but want to support grpc, we could do the
> work to make a thrift adapter, so picking one or the other now doesn't
> really preclude doing cool stuff later.
>
> In short, it doesn't really matter? At least not to me.
>
> On Thu, Aug 6, 2015 at 3:57 PM Josh Elser<jo...@gmail.com>  wrote:
>
>> Thanks for the feedback, Jesse!
>>
>> To clarify, the compatibility I'm most worried about is backwards compat
>> in future versions of protobuf3. I hadn't considered v2 support into the
>> equation -- figured v3 is the starting point like you say. I know the
>> protobuf 2.4 to 2.5 upgrade in Hadoop/HBase land was rather scary that
>> required a lot of lock-step movement across many big projects. This is
>> my biggest fear with protobuf.
>>
>> Despite fear of derailing conversation, grpc definitely has some nice
>> features built in (and implementing a service is definitely slick). I
>> think it's definitely a "later" since the single endpoint presently used
>> doesn't really require a complex service definition (with evolving args,
>> etc). My gut reaction is to wait to adopt such software until there is a
>> definite understanding of what the service would look like (which I
>> assume will become clearer as development continues).
>>
>> Jesse Yates wrote:
>>> +1 for proto3. mostly for the later move to using grpc, which lets us
>>> leverage a battle tested framework for doing real protobuf services with
>>> fast streaming (how nice!).
>>>
>>> proto3 can be backwards compat in some ways (eg. map implementation done
>>> manually in proto2). I don't think its a huge issue since there are
>>> currently no backwards compat guarantees anyways AFAIK.
>>>
>>>
>>> On Thu, Aug 6, 2015 at 3:11 PM Josh Elser<jo...@gmail.com>   wrote:
>>>
>>>> Hi,
>>>>
>>>> I have some ideas which I'd like to try to working on to improve
>>>> Avatica. My direct interest is in support of the Phoenix Query Server
>>>> which builds on top of Avatica.
>>>>
>>>> One of the big areas of improvement that stands out to me is the
>>>> Jackson-annotated Response/Request classes. First, we'd want to make
>>>> sure these classes are stable (in terms of fields/attributes) so that,
>>>> as each object evolves, we prevent existing clients from breaking.
>>>> Second, it's desirable that we have some representation of these objects
>>>> in a manner that's accessible to non-Java clients.
>>>>
>>>> Ideally, preserving JSON as the serialized representation of these
>>>> classes is also desirable as it's always pleasant to actually be able to
>>>> read the data going over the wire when necessary.
>>>>
>>>> Given these goals, two tools stood out to me as candidates that would
>>>> solve this problem well without extra effort in the transport: Thrift
>>>> and Protobuf v3.
>>>>
>>>> Using Thrift only for object generation is a bit overkill, but this is
>>>> an area of Thrift that has been rock-solid in the years that I've used
>>>> it. Classpath issues with Thrift are always a concern as multiple
>>>> versions of Thrift tend to break terribly (at least when performing
>>>> Thrift RPCs -- I wouldn't necessarily trust it to be safe for only
>>>> object serialization). It's easy to serialize Thrift objects to JSON and
>>>> we can generate these classes for other languages.
>>>>
>>>> I would call Protobuf v3 very young, although it does appear to fit the
>>>> bill. Protobuf v2 has had a track record of being a pain when different
>>>> versions of the library exist. For example, the Java classes generated
>>>> by protobuf 2.x would fail to work with protobuf 2.y. I assume there's
>>>> not enough evidence on protobuf v3 to say whether this will continue to
>>>> be a problem. JSON-representation of objects and cross-language support
>>>> are both natively supported (best as I can tell). Hypothetically,
>>>> adopting protobufv3 could also have some extra benefits down the road
>>>> such as leveraging grpc to replace Jetty, but that is an entirely
>>>> different discussion for later :)
>>>>
>>>> If you've gotten this far, I'd humbly ask for any opinions on the
>>>> technology choices. Any opinions for or against one or the other? Any
>>>> similar experiences?
>>>>
>>>> Thanks!
>>>>
>>>> - Josh
>>>>
>

Re: [Avatica] Separating wire message from service implementation

[Jesse Yates <je...@gmail.com>]

Yup, totally get it. Honestly, I have no idea what the compat story is
supposed to be for proto3. At the same time, I've got no hands on
experience w/ Thrift either, beyond dabbling a little bit years ago, so I
don't know what their on-the-wire-compat story is either :) I'm not as huge
of a fan of thrift in general though because it tries to do too much IMO

If we do go the thrift route, but want to support grpc, we could do the
work to make a thrift adapter, so picking one or the other now doesn't
really preclude doing cool stuff later.

In short, it doesn't really matter? At least not to me.

On Thu, Aug 6, 2015 at 3:57 PM Josh Elser <jo...@gmail.com> wrote:

> Thanks for the feedback, Jesse!
>
> To clarify, the compatibility I'm most worried about is backwards compat
> in future versions of protobuf3. I hadn't considered v2 support into the
> equation -- figured v3 is the starting point like you say. I know the
> protobuf 2.4 to 2.5 upgrade in Hadoop/HBase land was rather scary that
> required a lot of lock-step movement across many big projects. This is
> my biggest fear with protobuf.
>
> Despite fear of derailing conversation, grpc definitely has some nice
> features built in (and implementing a service is definitely slick). I
> think it's definitely a "later" since the single endpoint presently used
> doesn't really require a complex service definition (with evolving args,
> etc). My gut reaction is to wait to adopt such software until there is a
> definite understanding of what the service would look like (which I
> assume will become clearer as development continues).
>
> Jesse Yates wrote:
> > +1 for proto3. mostly for the later move to using grpc, which lets us
> > leverage a battle tested framework for doing real protobuf services with
> > fast streaming (how nice!).
> >
> > proto3 can be backwards compat in some ways (eg. map implementation done
> > manually in proto2). I don't think its a huge issue since there are
> > currently no backwards compat guarantees anyways AFAIK.
> >
> >
> > On Thu, Aug 6, 2015 at 3:11 PM Josh Elser<jo...@gmail.com>  wrote:
> >
> >> Hi,
> >>
> >> I have some ideas which I'd like to try to working on to improve
> >> Avatica. My direct interest is in support of the Phoenix Query Server
> >> which builds on top of Avatica.
> >>
> >> One of the big areas of improvement that stands out to me is the
> >> Jackson-annotated Response/Request classes. First, we'd want to make
> >> sure these classes are stable (in terms of fields/attributes) so that,
> >> as each object evolves, we prevent existing clients from breaking.
> >> Second, it's desirable that we have some representation of these objects
> >> in a manner that's accessible to non-Java clients.
> >>
> >> Ideally, preserving JSON as the serialized representation of these
> >> classes is also desirable as it's always pleasant to actually be able to
> >> read the data going over the wire when necessary.
> >>
> >> Given these goals, two tools stood out to me as candidates that would
> >> solve this problem well without extra effort in the transport: Thrift
> >> and Protobuf v3.
> >>
> >> Using Thrift only for object generation is a bit overkill, but this is
> >> an area of Thrift that has been rock-solid in the years that I've used
> >> it. Classpath issues with Thrift are always a concern as multiple
> >> versions of Thrift tend to break terribly (at least when performing
> >> Thrift RPCs -- I wouldn't necessarily trust it to be safe for only
> >> object serialization). It's easy to serialize Thrift objects to JSON and
> >> we can generate these classes for other languages.
> >>
> >> I would call Protobuf v3 very young, although it does appear to fit the
> >> bill. Protobuf v2 has had a track record of being a pain when different
> >> versions of the library exist. For example, the Java classes generated
> >> by protobuf 2.x would fail to work with protobuf 2.y. I assume there's
> >> not enough evidence on protobuf v3 to say whether this will continue to
> >> be a problem. JSON-representation of objects and cross-language support
> >> are both natively supported (best as I can tell). Hypothetically,
> >> adopting protobufv3 could also have some extra benefits down the road
> >> such as leveraging grpc to replace Jetty, but that is an entirely
> >> different discussion for later :)
> >>
> >> If you've gotten this far, I'd humbly ask for any opinions on the
> >> technology choices. Any opinions for or against one or the other? Any
> >> similar experiences?
> >>
> >> Thanks!
> >>
> >> - Josh
> >>
> >
>

Re: [Avatica] Separating wire message from service implementation

[Josh Elser <jo...@gmail.com>]

Thanks for the feedback, Jesse!

To clarify, the compatibility I'm most worried about is backwards compat 
in future versions of protobuf3. I hadn't considered v2 support into the 
equation -- figured v3 is the starting point like you say. I know the 
protobuf 2.4 to 2.5 upgrade in Hadoop/HBase land was rather scary that 
required a lot of lock-step movement across many big projects. This is 
my biggest fear with protobuf.

Despite fear of derailing conversation, grpc definitely has some nice 
features built in (and implementing a service is definitely slick). I 
think it's definitely a "later" since the single endpoint presently used 
doesn't really require a complex service definition (with evolving args, 
etc). My gut reaction is to wait to adopt such software until there is a 
definite understanding of what the service would look like (which I 
assume will become clearer as development continues).

Jesse Yates wrote:
> +1 for proto3. mostly for the later move to using grpc, which lets us
> leverage a battle tested framework for doing real protobuf services with
> fast streaming (how nice!).
>
> proto3 can be backwards compat in some ways (eg. map implementation done
> manually in proto2). I don't think its a huge issue since there are
> currently no backwards compat guarantees anyways AFAIK.
>
>
> On Thu, Aug 6, 2015 at 3:11 PM Josh Elser<jo...@gmail.com>  wrote:
>
>> Hi,
>>
>> I have some ideas which I'd like to try to working on to improve
>> Avatica. My direct interest is in support of the Phoenix Query Server
>> which builds on top of Avatica.
>>
>> One of the big areas of improvement that stands out to me is the
>> Jackson-annotated Response/Request classes. First, we'd want to make
>> sure these classes are stable (in terms of fields/attributes) so that,
>> as each object evolves, we prevent existing clients from breaking.
>> Second, it's desirable that we have some representation of these objects
>> in a manner that's accessible to non-Java clients.
>>
>> Ideally, preserving JSON as the serialized representation of these
>> classes is also desirable as it's always pleasant to actually be able to
>> read the data going over the wire when necessary.
>>
>> Given these goals, two tools stood out to me as candidates that would
>> solve this problem well without extra effort in the transport: Thrift
>> and Protobuf v3.
>>
>> Using Thrift only for object generation is a bit overkill, but this is
>> an area of Thrift that has been rock-solid in the years that I've used
>> it. Classpath issues with Thrift are always a concern as multiple
>> versions of Thrift tend to break terribly (at least when performing
>> Thrift RPCs -- I wouldn't necessarily trust it to be safe for only
>> object serialization). It's easy to serialize Thrift objects to JSON and
>> we can generate these classes for other languages.
>>
>> I would call Protobuf v3 very young, although it does appear to fit the
>> bill. Protobuf v2 has had a track record of being a pain when different
>> versions of the library exist. For example, the Java classes generated
>> by protobuf 2.x would fail to work with protobuf 2.y. I assume there's
>> not enough evidence on protobuf v3 to say whether this will continue to
>> be a problem. JSON-representation of objects and cross-language support
>> are both natively supported (best as I can tell). Hypothetically,
>> adopting protobufv3 could also have some extra benefits down the road
>> such as leveraging grpc to replace Jetty, but that is an entirely
>> different discussion for later :)
>>
>> If you've gotten this far, I'd humbly ask for any opinions on the
>> technology choices. Any opinions for or against one or the other? Any
>> similar experiences?
>>
>> Thanks!
>>
>> - Josh
>>
>

Re: [Avatica] Separating wire message from service implementation

[Jesse Yates <je...@gmail.com>]

+1 for proto3. mostly for the later move to using grpc, which lets us
leverage a battle tested framework for doing real protobuf services with
fast streaming (how nice!).

proto3 can be backwards compat in some ways (eg. map implementation done
manually in proto2). I don't think its a huge issue since there are
currently no backwards compat guarantees anyways AFAIK.


On Thu, Aug 6, 2015 at 3:11 PM Josh Elser <jo...@gmail.com> wrote:

> Hi,
>
> I have some ideas which I'd like to try to working on to improve
> Avatica. My direct interest is in support of the Phoenix Query Server
> which builds on top of Avatica.
>
> One of the big areas of improvement that stands out to me is the
> Jackson-annotated Response/Request classes. First, we'd want to make
> sure these classes are stable (in terms of fields/attributes) so that,
> as each object evolves, we prevent existing clients from breaking.
> Second, it's desirable that we have some representation of these objects
> in a manner that's accessible to non-Java clients.
>
> Ideally, preserving JSON as the serialized representation of these
> classes is also desirable as it's always pleasant to actually be able to
> read the data going over the wire when necessary.
>
> Given these goals, two tools stood out to me as candidates that would
> solve this problem well without extra effort in the transport: Thrift
> and Protobuf v3.
>
> Using Thrift only for object generation is a bit overkill, but this is
> an area of Thrift that has been rock-solid in the years that I've used
> it. Classpath issues with Thrift are always a concern as multiple
> versions of Thrift tend to break terribly (at least when performing
> Thrift RPCs -- I wouldn't necessarily trust it to be safe for only
> object serialization). It's easy to serialize Thrift objects to JSON and
> we can generate these classes for other languages.
>
> I would call Protobuf v3 very young, although it does appear to fit the
> bill. Protobuf v2 has had a track record of being a pain when different
> versions of the library exist. For example, the Java classes generated
> by protobuf 2.x would fail to work with protobuf 2.y. I assume there's
> not enough evidence on protobuf v3 to say whether this will continue to
> be a problem. JSON-representation of objects and cross-language support
> are both natively supported (best as I can tell). Hypothetically,
> adopting protobufv3 could also have some extra benefits down the road
> such as leveraging grpc to replace Jetty, but that is an entirely
> different discussion for later :)
>
> If you've gotten this far, I'd humbly ask for any opinions on the
> technology choices. Any opinions for or against one or the other? Any
> similar experiences?
>
> Thanks!
>
> - Josh
>