You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mesatee.apache.org by Yu Ding <di...@apache.org> on 2019/10/26 18:11:31 UTC
RPC framework
Hi folks,
MesaTEE is supporting more and more services. In the current RPC design, we
need several handcrafted Rust struct for each service message. It is not
very scalable and will cause maintainability issues. We need a better RPC
framework.
Seems that protobuf-styled RPC is the best choice. We have the following
choices for the software stack
(1) rust-protobuf + grpc-rust + tokio
(2) prost + tonic (or alternatives) + tokio
(1) is more stable, but the Rust codes generated from proto files are too
much big, which finally leads to a super large TCB and bad for TEEs. (2)
benefits from prost for its tiny codebase but it's changing rapidly. I like
(2) better because it's better for TEE from the security/safety perspective.
Any ideas?
Best,
Yu
Re: RPC framework
Posted by Tongxin Li <li...@apache.org>.
I have tried prost. With prost, MesaTEE can not only use protobuf, but also
generate almost same RPC messages as before.
All the features we used can be kept, including
serialization/deserialization with serde_json, enumeration types and
encoding bytes with base64.
To use these features, Prost needs to be configured in build.rs.
Following are some generated rust structures and a serialized RPC message
in json.
* rust structures
```rust
#[derive(serde_derive::Serialize, serde_derive::Deserialize)]
#[serde(tag = "type")]
pub enum KmsResponse {
#[prost(message, tag="1")]
Get(super::GetKeyResponse),
#[prost(message, tag="2")]
Delete(super::DeleteKeyResponse),
}
pub struct GetKeyResponse {
#[prost(message, optional, tag="1")]
pub config: ::std::option::Option<AeadConfig>,
}
#[derive(Clone, PartialEq, ::prost::Message)]
#[derive(serde_derive::Serialize, serde_derive::Deserialize)]
pub struct AeadConfig {
#[prost(bytes, tag="1")]
#[serde(with = "crate::base64_coder")]
pub key: std::vec::Vec<u8>,
#[prost(bytes, tag="2")]
#[serde(with = "crate::base64_coder")]
pub nonce: std::vec::Vec<u8>,
#[prost(bytes, tag="3")]
#[serde(with = "crate::base64_coder")]
pub ad: std::vec::Vec<u8>,
}
```
* A serialized RPC message in json
```json
{"type":"Get","config":{"key":"MjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjI=","nonce":"MjIyMjIyMjIyMjIy","ad":"MjIyMjI="}}
```
Best,
Tongxin
On Mon, Oct 28, 2019 at 8:01 AM Matt Sicker <bo...@gmail.com> wrote:
> There are some Apache projects you can consider for serialization or RPC:
>
> * https://avro.apache.org/
> * https://dubbo.apache.org/
> * https://thrift.apache.org/
> * https://github.com/apache/incubator-brpc
>
> Avro is fairly popular in big data spaces thanks to Kafka. Thrift has
> a more well defined RPC API and is more similar to protobuf than avro.
> I like Avro for its schemas which can help reduce message size a bit
> with common schemas.
>
> On Sat, 26 Oct 2019 at 19:12, Mingshen Sun <ms...@apache.org> wrote:
> >
> > Yes, this topic is worth to consider. I have looked at the prost
> library. I
> > prefer the second solution.
> >
> > To make the discussion more clear, can you give us an example for the KMS
> > protocol? (
> >
> https://github.com/mesalock-linux/mesatee/blob/master/mesatee_services/kms/proto/src/proto.rs
> )
> > Thanks.
> >
> >
> > On Sat, Oct 26, 2019 at 11:12 AM Yu Ding <di...@apache.org> wrote:
> >
> > > Hi folks,
> > >
> > > MesaTEE is supporting more and more services. In the current RPC
> design, we
> > > need several handcrafted Rust struct for each service message. It is
> not
> > > very scalable and will cause maintainability issues. We need a better
> RPC
> > > framework.
> > >
> > > Seems that protobuf-styled RPC is the best choice. We have the
> following
> > > choices for the software stack
> > > (1) rust-protobuf + grpc-rust + tokio
> > > (2) prost + tonic (or alternatives) + tokio
> > >
> > > (1) is more stable, but the Rust codes generated from proto files are
> too
> > > much big, which finally leads to a super large TCB and bad for TEEs.
> (2)
> > > benefits from prost for its tiny codebase but it's changing rapidly. I
> like
> > > (2) better because it's better for TEE from the security/safety
> > > perspective.
> > >
> > > Any ideas?
> > >
> > > Best,
> > > Yu
> > >
>
>
>
> --
> Matt Sicker <bo...@gmail.com>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@mesatee.apache.org
> For additional commands, e-mail: dev-help@mesatee.apache.org
>
>
Re: RPC framework
Posted by Matt Sicker <bo...@gmail.com>.
There are some Apache projects you can consider for serialization or RPC:
* https://avro.apache.org/
* https://dubbo.apache.org/
* https://thrift.apache.org/
* https://github.com/apache/incubator-brpc
Avro is fairly popular in big data spaces thanks to Kafka. Thrift has
a more well defined RPC API and is more similar to protobuf than avro.
I like Avro for its schemas which can help reduce message size a bit
with common schemas.
On Sat, 26 Oct 2019 at 19:12, Mingshen Sun <ms...@apache.org> wrote:
>
> Yes, this topic is worth to consider. I have looked at the prost library. I
> prefer the second solution.
>
> To make the discussion more clear, can you give us an example for the KMS
> protocol? (
> https://github.com/mesalock-linux/mesatee/blob/master/mesatee_services/kms/proto/src/proto.rs)
> Thanks.
>
>
> On Sat, Oct 26, 2019 at 11:12 AM Yu Ding <di...@apache.org> wrote:
>
> > Hi folks,
> >
> > MesaTEE is supporting more and more services. In the current RPC design, we
> > need several handcrafted Rust struct for each service message. It is not
> > very scalable and will cause maintainability issues. We need a better RPC
> > framework.
> >
> > Seems that protobuf-styled RPC is the best choice. We have the following
> > choices for the software stack
> > (1) rust-protobuf + grpc-rust + tokio
> > (2) prost + tonic (or alternatives) + tokio
> >
> > (1) is more stable, but the Rust codes generated from proto files are too
> > much big, which finally leads to a super large TCB and bad for TEEs. (2)
> > benefits from prost for its tiny codebase but it's changing rapidly. I like
> > (2) better because it's better for TEE from the security/safety
> > perspective.
> >
> > Any ideas?
> >
> > Best,
> > Yu
> >
--
Matt Sicker <bo...@gmail.com>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@mesatee.apache.org
For additional commands, e-mail: dev-help@mesatee.apache.org
Re: RPC framework
Posted by Mingshen Sun <ms...@apache.org>.
Yes, this topic is worth to consider. I have looked at the prost library. I
prefer the second solution.
To make the discussion more clear, can you give us an example for the KMS
protocol? (
https://github.com/mesalock-linux/mesatee/blob/master/mesatee_services/kms/proto/src/proto.rs)
Thanks.
On Sat, Oct 26, 2019 at 11:12 AM Yu Ding <di...@apache.org> wrote:
> Hi folks,
>
> MesaTEE is supporting more and more services. In the current RPC design, we
> need several handcrafted Rust struct for each service message. It is not
> very scalable and will cause maintainability issues. We need a better RPC
> framework.
>
> Seems that protobuf-styled RPC is the best choice. We have the following
> choices for the software stack
> (1) rust-protobuf + grpc-rust + tokio
> (2) prost + tonic (or alternatives) + tokio
>
> (1) is more stable, but the Rust codes generated from proto files are too
> much big, which finally leads to a super large TCB and bad for TEEs. (2)
> benefits from prost for its tiny codebase but it's changing rapidly. I like
> (2) better because it's better for TEE from the security/safety
> perspective.
>
> Any ideas?
>
> Best,
> Yu
>