You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@accumulo.apache.org by Josh Elser <jo...@gmail.com> on 2015/12/01 18:49:26 UTC
State of our RPCs
Hi --
My adventures in Thrift as a part of ACCUMULO-4065 are finally coming to
a close, it seems. The briefest summary I can give is that our hack to
work around an 0.9.0->0.9.1 compatibility issue ended up creating a bug
in a very obtuse case (when a server answering a oneway Thrift call
threw an RTE or an Error).
Given some other recent chatter in the project, I'm left wondering: what
next?
We've long considered Thrift to be a very useful tool, but extremely
scary to upgrade. I think this is just another sign of this. This leaves
me asking, how do we fix this?
Best as I understand it, Thrift is still a relatively active project (at
least their mailing list archives shows it). My impression is that the
Java library is much less-so. Most of our issues to me that they
ultimately stem from incompatibilities between libthrift versions and
uncaught performance regressions.
Assuming that to be true, do we need to make a coordinated effort to
improve the upstream libthrift code? Become a part of their community,
focusing on preventing these sorts of issues from ever filtering down to
us? Help them generate and follow compatibility guidelines?
I feel like our strategy over the past few years has been to "avert your
eyes" -- if we don't touch it, it'll hopefully be ok. Perhaps we need to
try something new. Thoughts?
- Josh
Re: State of our RPCs
Posted by Josh Elser <jo...@gmail.com>.
IMO, Thrift provides a lot out of the box, and, all things considered,
reduces our complexity greatly. For example:
* THsHaServer is great for our execution model
* We can inherit things like SSL and SASL (Kerberos)
* An _implemented_ RPC service (stuff like Hadoop RPC and Protobuf just
provide the Service definition but not the actual transport).
I don't think the problems we have with Thrift are so great that we
should throw it away.
dlmarion@comcast.net wrote:
> What was it about Thrift that drove us to use it? Was it the bindings for multiple languages? Should this decision be revisited?
>
> ----- Original Message -----
>
> From: "Josh Elser"<jo...@gmail.com>
> To: "dev"<de...@accumulo.apache.org>
> Sent: Tuesday, December 1, 2015 12:49:26 PM
> Subject: State of our RPCs
>
> Hi --
>
> My adventures in Thrift as a part of ACCUMULO-4065 are finally coming to
> a close, it seems. The briefest summary I can give is that our hack to
> work around an 0.9.0->0.9.1 compatibility issue ended up creating a bug
> in a very obtuse case (when a server answering a oneway Thrift call
> threw an RTE or an Error).
>
> Given some other recent chatter in the project, I'm left wondering: what
> next?
>
> We've long considered Thrift to be a very useful tool, but extremely
> scary to upgrade. I think this is just another sign of this. This leaves
> me asking, how do we fix this?
>
> Best as I understand it, Thrift is still a relatively active project (at
> least their mailing list archives shows it). My impression is that the
> Java library is much less-so. Most of our issues to me that they
> ultimately stem from incompatibilities between libthrift versions and
> uncaught performance regressions.
>
> Assuming that to be true, do we need to make a coordinated effort to
> improve the upstream libthrift code? Become a part of their community,
> focusing on preventing these sorts of issues from ever filtering down to
> us? Help them generate and follow compatibility guidelines?
>
> I feel like our strategy over the past few years has been to "avert your
> eyes" -- if we don't touch it, it'll hopefully be ok. Perhaps we need to
> try something new. Thoughts?
>
> - Josh
>
>
Re: State of our RPCs
Posted by Josh Elser <jo...@gmail.com>.
Gotcha. Billie just noted in IRC that, when Accumulo has custom RPC code
a long time ago, Keith's experiment showed Thrift 500x faster. That's
probably how we got started.
Since then, I think it's just been a good solution that works well in
the end (development work can be hard). The lack of an integrated
serialization and transport system is what kept up with Thrift. I could
only come up with gRPC that could be a true replacement, but I may be
missing others.
dlmarion@comcast.net wrote:
> I'm not suggesting that we replace Thrift (nor am I signing up to do it), just asking for the basis of the decision and if its time to revisit. I'm totally ok with a 'no' answer.
>
> ----- Original Message -----
>
> From: "Josh Elser"<jo...@gmail.com>
> To: dev@accumulo.apache.org
> Sent: Tuesday, December 1, 2015 2:36:18 PM
> Subject: Re: State of our RPCs
>
> To play devil's advocate: I'm not sure if it's quite that simple. For
> example, Avro has been around since 2009, but I don't think it'd be fair
> to consider Avro circa 2009 to Avro circa 2015.
>
> David Medinets wrote:
>> What new protocols have been introduced since the Thrift decisions? Can
>> someone provide pros and cons for that limited set of protocols?
>>
>> On Tue, Dec 1, 2015 at 1:02 PM,<dl...@comcast.net> wrote:
>>
>>> What was it about Thrift that drove us to use it? Was it the bindings for
>>> multiple languages? Should this decision be revisited?
>>>
>>> ----- Original Message -----
>>>
>>> From: "Josh Elser"<jo...@gmail.com>
>>> To: "dev"<de...@accumulo.apache.org>
>>> Sent: Tuesday, December 1, 2015 12:49:26 PM
>>> Subject: State of our RPCs
>>>
>>> Hi --
>>>
>>> My adventures in Thrift as a part of ACCUMULO-4065 are finally coming to
>>> a close, it seems. The briefest summary I can give is that our hack to
>>> work around an 0.9.0->0.9.1 compatibility issue ended up creating a bug
>>> in a very obtuse case (when a server answering a oneway Thrift call
>>> threw an RTE or an Error).
>>>
>>> Given some other recent chatter in the project, I'm left wondering: what
>>> next?
>>>
>>> We've long considered Thrift to be a very useful tool, but extremely
>>> scary to upgrade. I think this is just another sign of this. This leaves
>>> me asking, how do we fix this?
>>>
>>> Best as I understand it, Thrift is still a relatively active project (at
>>> least their mailing list archives shows it). My impression is that the
>>> Java library is much less-so. Most of our issues to me that they
>>> ultimately stem from incompatibilities between libthrift versions and
>>> uncaught performance regressions.
>>>
>>> Assuming that to be true, do we need to make a coordinated effort to
>>> improve the upstream libthrift code? Become a part of their community,
>>> focusing on preventing these sorts of issues from ever filtering down to
>>> us? Help them generate and follow compatibility guidelines?
>>>
>>> I feel like our strategy over the past few years has been to "avert your
>>> eyes" -- if we don't touch it, it'll hopefully be ok. Perhaps we need to
>>> try something new. Thoughts?
>>>
>>> - Josh
>>>
>>>
>
>
Re: State of our RPCs
Posted by dl...@comcast.net.
I'm not suggesting that we replace Thrift (nor am I signing up to do it), just asking for the basis of the decision and if its time to revisit. I'm totally ok with a 'no' answer.
----- Original Message -----
From: "Josh Elser" <jo...@gmail.com>
To: dev@accumulo.apache.org
Sent: Tuesday, December 1, 2015 2:36:18 PM
Subject: Re: State of our RPCs
To play devil's advocate: I'm not sure if it's quite that simple. For
example, Avro has been around since 2009, but I don't think it'd be fair
to consider Avro circa 2009 to Avro circa 2015.
David Medinets wrote:
> What new protocols have been introduced since the Thrift decisions? Can
> someone provide pros and cons for that limited set of protocols?
>
> On Tue, Dec 1, 2015 at 1:02 PM,<dl...@comcast.net> wrote:
>
>> What was it about Thrift that drove us to use it? Was it the bindings for
>> multiple languages? Should this decision be revisited?
>>
>> ----- Original Message -----
>>
>> From: "Josh Elser"<jo...@gmail.com>
>> To: "dev"<de...@accumulo.apache.org>
>> Sent: Tuesday, December 1, 2015 12:49:26 PM
>> Subject: State of our RPCs
>>
>> Hi --
>>
>> My adventures in Thrift as a part of ACCUMULO-4065 are finally coming to
>> a close, it seems. The briefest summary I can give is that our hack to
>> work around an 0.9.0->0.9.1 compatibility issue ended up creating a bug
>> in a very obtuse case (when a server answering a oneway Thrift call
>> threw an RTE or an Error).
>>
>> Given some other recent chatter in the project, I'm left wondering: what
>> next?
>>
>> We've long considered Thrift to be a very useful tool, but extremely
>> scary to upgrade. I think this is just another sign of this. This leaves
>> me asking, how do we fix this?
>>
>> Best as I understand it, Thrift is still a relatively active project (at
>> least their mailing list archives shows it). My impression is that the
>> Java library is much less-so. Most of our issues to me that they
>> ultimately stem from incompatibilities between libthrift versions and
>> uncaught performance regressions.
>>
>> Assuming that to be true, do we need to make a coordinated effort to
>> improve the upstream libthrift code? Become a part of their community,
>> focusing on preventing these sorts of issues from ever filtering down to
>> us? Help them generate and follow compatibility guidelines?
>>
>> I feel like our strategy over the past few years has been to "avert your
>> eyes" -- if we don't touch it, it'll hopefully be ok. Perhaps we need to
>> try something new. Thoughts?
>>
>> - Josh
>>
>>
>
Re: State of our RPCs
Posted by Josh Elser <jo...@gmail.com>.
To play devil's advocate: I'm not sure if it's quite that simple. For
example, Avro has been around since 2009, but I don't think it'd be fair
to consider Avro circa 2009 to Avro circa 2015.
David Medinets wrote:
> What new protocols have been introduced since the Thrift decisions? Can
> someone provide pros and cons for that limited set of protocols?
>
> On Tue, Dec 1, 2015 at 1:02 PM,<dl...@comcast.net> wrote:
>
>> What was it about Thrift that drove us to use it? Was it the bindings for
>> multiple languages? Should this decision be revisited?
>>
>> ----- Original Message -----
>>
>> From: "Josh Elser"<jo...@gmail.com>
>> To: "dev"<de...@accumulo.apache.org>
>> Sent: Tuesday, December 1, 2015 12:49:26 PM
>> Subject: State of our RPCs
>>
>> Hi --
>>
>> My adventures in Thrift as a part of ACCUMULO-4065 are finally coming to
>> a close, it seems. The briefest summary I can give is that our hack to
>> work around an 0.9.0->0.9.1 compatibility issue ended up creating a bug
>> in a very obtuse case (when a server answering a oneway Thrift call
>> threw an RTE or an Error).
>>
>> Given some other recent chatter in the project, I'm left wondering: what
>> next?
>>
>> We've long considered Thrift to be a very useful tool, but extremely
>> scary to upgrade. I think this is just another sign of this. This leaves
>> me asking, how do we fix this?
>>
>> Best as I understand it, Thrift is still a relatively active project (at
>> least their mailing list archives shows it). My impression is that the
>> Java library is much less-so. Most of our issues to me that they
>> ultimately stem from incompatibilities between libthrift versions and
>> uncaught performance regressions.
>>
>> Assuming that to be true, do we need to make a coordinated effort to
>> improve the upstream libthrift code? Become a part of their community,
>> focusing on preventing these sorts of issues from ever filtering down to
>> us? Help them generate and follow compatibility guidelines?
>>
>> I feel like our strategy over the past few years has been to "avert your
>> eyes" -- if we don't touch it, it'll hopefully be ok. Perhaps we need to
>> try something new. Thoughts?
>>
>> - Josh
>>
>>
>
Re: State of our RPCs
Posted by Josh Elser <jo...@gmail.com>.
At a glance, it looks like this doesn't provide any transport, just the
marshaling of data. I really don't want to own that logic (for example,
I've seen the thousands of lines of code in HBase just for setting up
their RPC server).
John R. Frank wrote:
>
> It might be worth considering CBOR http://cbor.io/
>
> jrf
>
Re: State of our RPCs
Posted by "John R. Frank" <jr...@diffeo.com>.
It might be worth considering CBOR http://cbor.io/
jrf
Re: State of our RPCs
Posted by David Medinets <da...@gmail.com>.
What new protocols have been introduced since the Thrift decisions? Can
someone provide pros and cons for that limited set of protocols?
On Tue, Dec 1, 2015 at 1:02 PM, <dl...@comcast.net> wrote:
>
> What was it about Thrift that drove us to use it? Was it the bindings for
> multiple languages? Should this decision be revisited?
>
> ----- Original Message -----
>
> From: "Josh Elser" <jo...@gmail.com>
> To: "dev" <de...@accumulo.apache.org>
> Sent: Tuesday, December 1, 2015 12:49:26 PM
> Subject: State of our RPCs
>
> Hi --
>
> My adventures in Thrift as a part of ACCUMULO-4065 are finally coming to
> a close, it seems. The briefest summary I can give is that our hack to
> work around an 0.9.0->0.9.1 compatibility issue ended up creating a bug
> in a very obtuse case (when a server answering a oneway Thrift call
> threw an RTE or an Error).
>
> Given some other recent chatter in the project, I'm left wondering: what
> next?
>
> We've long considered Thrift to be a very useful tool, but extremely
> scary to upgrade. I think this is just another sign of this. This leaves
> me asking, how do we fix this?
>
> Best as I understand it, Thrift is still a relatively active project (at
> least their mailing list archives shows it). My impression is that the
> Java library is much less-so. Most of our issues to me that they
> ultimately stem from incompatibilities between libthrift versions and
> uncaught performance regressions.
>
> Assuming that to be true, do we need to make a coordinated effort to
> improve the upstream libthrift code? Become a part of their community,
> focusing on preventing these sorts of issues from ever filtering down to
> us? Help them generate and follow compatibility guidelines?
>
> I feel like our strategy over the past few years has been to "avert your
> eyes" -- if we don't touch it, it'll hopefully be ok. Perhaps we need to
> try something new. Thoughts?
>
> - Josh
>
>
Re: State of our RPCs
Posted by dl...@comcast.net.
What was it about Thrift that drove us to use it? Was it the bindings for multiple languages? Should this decision be revisited?
----- Original Message -----
From: "Josh Elser" <jo...@gmail.com>
To: "dev" <de...@accumulo.apache.org>
Sent: Tuesday, December 1, 2015 12:49:26 PM
Subject: State of our RPCs
Hi --
My adventures in Thrift as a part of ACCUMULO-4065 are finally coming to
a close, it seems. The briefest summary I can give is that our hack to
work around an 0.9.0->0.9.1 compatibility issue ended up creating a bug
in a very obtuse case (when a server answering a oneway Thrift call
threw an RTE or an Error).
Given some other recent chatter in the project, I'm left wondering: what
next?
We've long considered Thrift to be a very useful tool, but extremely
scary to upgrade. I think this is just another sign of this. This leaves
me asking, how do we fix this?
Best as I understand it, Thrift is still a relatively active project (at
least their mailing list archives shows it). My impression is that the
Java library is much less-so. Most of our issues to me that they
ultimately stem from incompatibilities between libthrift versions and
uncaught performance regressions.
Assuming that to be true, do we need to make a coordinated effort to
improve the upstream libthrift code? Become a part of their community,
focusing on preventing these sorts of issues from ever filtering down to
us? Help them generate and follow compatibility guidelines?
I feel like our strategy over the past few years has been to "avert your
eyes" -- if we don't touch it, it'll hopefully be ok. Perhaps we need to
try something new. Thoughts?
- Josh
Re: State of our RPCs
Posted by Josh Elser <jo...@gmail.com>.
Well, this doesn't seem to have gone anywhere.
Oh well. For those who are still interested:
https://issues.apache.org/jira/browse/THRIFT-3479
Josh Elser wrote:
> (replying to myself since this was a common sentiment)
>
> I want to be clear that I am most definitely _not_ advocating that we
> rip out Thrift. I am still extremely far removed from that decision.
>
> My question was more how can we make Thrift work better for us, than
> look for a replacement.
>
> Josh Elser wrote:
>> Hi --
>>
>> My adventures in Thrift as a part of ACCUMULO-4065 are finally coming to
>> a close, it seems. The briefest summary I can give is that our hack to
>> work around an 0.9.0->0.9.1 compatibility issue ended up creating a bug
>> in a very obtuse case (when a server answering a oneway Thrift call
>> threw an RTE or an Error).
>>
>> Given some other recent chatter in the project, I'm left wondering: what
>> next?
>>
>> We've long considered Thrift to be a very useful tool, but extremely
>> scary to upgrade. I think this is just another sign of this. This leaves
>> me asking, how do we fix this?
>>
>> Best as I understand it, Thrift is still a relatively active project (at
>> least their mailing list archives shows it). My impression is that the
>> Java library is much less-so. Most of our issues to me that they
>> ultimately stem from incompatibilities between libthrift versions and
>> uncaught performance regressions.
>>
>> Assuming that to be true, do we need to make a coordinated effort to
>> improve the upstream libthrift code? Become a part of their community,
>> focusing on preventing these sorts of issues from ever filtering down to
>> us? Help them generate and follow compatibility guidelines?
>>
>> I feel like our strategy over the past few years has been to "avert your
>> eyes" -- if we don't touch it, it'll hopefully be ok. Perhaps we need to
>> try something new. Thoughts?
>>
>> - Josh
Re: State of our RPCs
Posted by Josh Elser <jo...@gmail.com>.
(replying to myself since this was a common sentiment)
I want to be clear that I am most definitely _not_ advocating that we
rip out Thrift. I am still extremely far removed from that decision.
My question was more how can we make Thrift work better for us, than
look for a replacement.
Josh Elser wrote:
> Hi --
>
> My adventures in Thrift as a part of ACCUMULO-4065 are finally coming to
> a close, it seems. The briefest summary I can give is that our hack to
> work around an 0.9.0->0.9.1 compatibility issue ended up creating a bug
> in a very obtuse case (when a server answering a oneway Thrift call
> threw an RTE or an Error).
>
> Given some other recent chatter in the project, I'm left wondering: what
> next?
>
> We've long considered Thrift to be a very useful tool, but extremely
> scary to upgrade. I think this is just another sign of this. This leaves
> me asking, how do we fix this?
>
> Best as I understand it, Thrift is still a relatively active project (at
> least their mailing list archives shows it). My impression is that the
> Java library is much less-so. Most of our issues to me that they
> ultimately stem from incompatibilities between libthrift versions and
> uncaught performance regressions.
>
> Assuming that to be true, do we need to make a coordinated effort to
> improve the upstream libthrift code? Become a part of their community,
> focusing on preventing these sorts of issues from ever filtering down to
> us? Help them generate and follow compatibility guidelines?
>
> I feel like our strategy over the past few years has been to "avert your
> eyes" -- if we don't touch it, it'll hopefully be ok. Perhaps we need to
> try something new. Thoughts?
>
> - Josh