You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@brpc.apache.org by GitBox <gi...@apache.org> on 2021/10/29 07:47:48 UTC
[GitHub] [incubator-brpc] KaneVV1 opened a new issue #1589: BRPC兼容GRPC stream
KaneVV1 opened a new issue #1589:
URL: https://github.com/apache/incubator-brpc/issues/1589
**Is your feature request related to a problem? (你需要的功能是否与某个问题有关?)**
stream是rpc框架使用中的常用功能，虽然BRPC有streaming rpc，但是兼容grpc stream能给框架的这部分功能带来更大的泛用性。
经过百度内部Service Mesh实践，出于定制化需求方向的考虑，我们希望在proxyless模式下，BRPC能够直连istio，逐渐减弱对envoy的依赖。istio下发配置使用的是双向GRPC stream，所以我们需要完成BRPC兼容GRPC stream的适配。
主要需求如下：
1、兼容GRPC steam的同步/异步或订阅推送式的客户端；
2、一如既往简单便捷的客户端API，不希望定制protobuf插件。
后续事项：*服务端API设计、直达底层h2协议的性能调优
**Describe the solution you'd like (描述你期望的解决方法)**
以GRPC stream中Demo（https://github.com/grpc/grpc/blob/fd3bd70939fb4239639fbd26143ec416366e4157/examples/python/data_transmission/demo.proto）为例，可以通过以下API来对grpc stream服务端进行访问。
`demo::Request req;
demo::Response res;
Controller cntl;
demo::GRPCDemo_Stub stub(&channel);
using namespace brpc;
// 客户端流模式（在一次调用中, 客户端可以多次向服务器传输数据, 但是服务器只能返回一次响应）
// rpc ClientStreamingMethod (stream Request) returns (Response);
auto writer = CreateStreamVisitor<demo::Request, GrpcWriter>(
&demo::GRPCDemo_Stub::ClientStreamingMethod, &stub, &res, &cntl);
writer->Write(req);
writer->WriteDone();
writer->Finish();
// 服务端流模式（在一次调用中, 客户端只能一次向服务器传输数据, 但是服务器可以多次返回响应）
// rpc ServerStreamingMethod (Request) returns (stream Response);
auto reader = CreateStreamVisitor<demo::Response, GrpcReader>(
&demo::GRPCDemo_Stub::ServerStreamingMethod, &stub, &req, &cntl);
while (reader->Read(&res)) {
std::cout << ... << std::endl;
}
reader->Finish();
// 双向流模式 (在一次调用中, 客户端和服务器都可以向对方多次收发数据)
// rpc BidirectionalStreamingMethod (stream Request) returns (stream Response);
auto stream = CreateStreamVisitor<demo::Request, demo::Response, GrpcReaderWriter>(
&demo::GRPCDemo_Stub::BidirectionalStreamingMethod, &stub, &cntl);
std::thread writer([stream]() {
for (int i = 0; i < 3; ++i) {
stream->Write(req);
}
stream->WritesDone();
});
while (stream->Read(&res)) {
std::cout << ... << std::endl;
}
writer.join();
stream->Finish();`
**Describe alternatives you've considered (描述你想到的折衷方案)**
简单地直接引入GRPC依赖
**Additional context/screenshots (更多上下文/截图)**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] KaneVV1 commented on issue #1589: BRPC兼容GRPC stream
Posted by GitBox <gi...@apache.org>.
KaneVV1 commented on issue #1589:
URL: https://github.com/apache/incubator-brpc/issues/1589#issuecomment-968476424
> grpc stream最大的问题就是没有异步接口。 另外上面这个demo中，好像看不到错误处理的部分? stream不能保证是可靠的。
异步接口后面会有计划，至于错误处理，暂时打算放在controller里
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] KaneVV1 removed a comment on issue #1589: BRPC兼容GRPC stream
Posted by GitBox <gi...@apache.org>.
KaneVV1 removed a comment on issue #1589:
URL: https://github.com/apache/incubator-brpc/issues/1589#issuecomment-968592825
> **Is your feature request related to a problem? (你需要的功能是否与某个问题有关?)**
>
> stream是rpc框架使用中的常用功能，虽然brpc有streaming rpc，但是兼容grpc stream能给框架的这部分功能带来更大的泛用性。 经过百度内部Service Mesh实践，出于定制化需求方向的考虑，我们希望在proxyless模式下，brpc能够直连istio，逐渐减弱对envoy的依赖。istio下发配置使用的是双向grpc stream，所以我们需要完成brpc兼容grpc stream的适配。 主要需求如下： 1、兼容grpc steam的同步/异步或订阅推送式的客户端； 2、一如既往简单便捷的客户端API，不希望定制protobuf插件。 后续事项：*服务端API设计、直达底层h2协议的性能调优
>
> **Describe the solution you'd like (描述你期望的解决方法)**
>
> 以grpc stream Demo的消息格式 https://github.com/grpc/grpc/blob/fd3bd70939fb4239639fbd26143ec416366e4157/examples/python/data_transmission/demo.proto 为例，可以通过以下API来对grpc stream服务端进行访问。
>
> ```c++
> using namespace brpc;
> demo::Request req;
> demo::Response res;
> Controller cntl;
>
> demo::GRPCDemo_Stub stub(&channel);
>
> // 客户端流模式（在一次调用中, 客户端可以多次向服务器传输数据, 但是服务器只能返回一次响应）
> // rpc ClientStreamingMethod (stream Request) returns (Response);
> auto writer = CreateStreamVisitor<demo::Request, GrpcWriter>(
> &demo::GRPCDemo_Stub::ClientStreamingMethod, &stub, &res, &cntl);
> writer->Write(req);
> writer->WritesDone();
> writer->Finish();
>
> // 服务端流模式（在一次调用中, 客户端只能一次向服务器传输数据, 但是服务器可以多次返回响应）
> // rpc ServerStreamingMethod (Request) returns (stream Response);
> auto reader = CreateStreamVisitor<demo::Response, GrpcReader>(
> &demo::GRPCDemo_Stub::ServerStreamingMethod, &stub, &req, &cntl);
> while (reader->Read(&res)) {
> std::cout << ... << std::endl;
> }
> reader->Finish();
>
> // 双向流模式 (在一次调用中, 客户端和服务器都可以向对方多次收发数据)
> // rpc BidirectionalStreamingMethod (stream Request) returns (stream Response);
> auto stream = CreateStreamVisitor<demo::Request, demo::Response, GrpcReaderWriter>(
> &demo::GRPCDemo_Stub::BidirectionalStreamingMethod, &stub, &cntl);
> std::thread writer([stream]() {
> for (int i = 0; i < 3; ++i) {
> stream->Write(req);
> }
> stream->WritesDone();
> });
> while (stream->Read(&res)) {
> std::cout << ... << std::endl;
> }
> writer.join();
> stream->Finish();
> ```
>
> 具体设计 TBD
>
> **Describe alternatives you've considered (描述你想到的折衷方案)** 简单地直接引入GRPC依赖
>
> **Additional context/screenshots (更多上下文/截图)**
> 接口好像有点多。 WritesDone和Finish是什么关系？可不可以合为一个。 writer->Finish()和stream->Finisher()又是什么关系? 没发送完毕，业务要中途退出是要如何处理(writer/reader端都可能发生)? 本身rpc接口中的done这里要如何处理？ 这个case并没有看到。 能已echo为例子写一个实际的包含错误处理的例子么。
已更新case
1、2 最好不要合一个，WritesDone用来保证发送端数据完整性，Finish来关闭整个流；
3、done还未考虑
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] KaneVV1 commented on issue #1589: BRPC兼容GRPC stream
Posted by GitBox <gi...@apache.org>.
KaneVV1 commented on issue #1589:
URL: https://github.com/apache/incubator-brpc/issues/1589#issuecomment-968592825
> **Is your feature request related to a problem? (你需要的功能是否与某个问题有关?)**
>
> stream是rpc框架使用中的常用功能，虽然brpc有streaming rpc，但是兼容grpc stream能给框架的这部分功能带来更大的泛用性。 经过百度内部Service Mesh实践，出于定制化需求方向的考虑，我们希望在proxyless模式下，brpc能够直连istio，逐渐减弱对envoy的依赖。istio下发配置使用的是双向grpc stream，所以我们需要完成brpc兼容grpc stream的适配。 主要需求如下： 1、兼容grpc steam的同步/异步或订阅推送式的客户端； 2、一如既往简单便捷的客户端API，不希望定制protobuf插件。 后续事项：*服务端API设计、直达底层h2协议的性能调优
>
> **Describe the solution you'd like (描述你期望的解决方法)**
>
> 以grpc stream Demo的消息格式 https://github.com/grpc/grpc/blob/fd3bd70939fb4239639fbd26143ec416366e4157/examples/python/data_transmission/demo.proto 为例，可以通过以下API来对grpc stream服务端进行访问。
>
> ```c++
> using namespace brpc;
> demo::Request req;
> demo::Response res;
> Controller cntl;
>
> demo::GRPCDemo_Stub stub(&channel);
>
> // 客户端流模式（在一次调用中, 客户端可以多次向服务器传输数据, 但是服务器只能返回一次响应）
> // rpc ClientStreamingMethod (stream Request) returns (Response);
> auto writer = CreateStreamVisitor<demo::Request, GrpcWriter>(
> &demo::GRPCDemo_Stub::ClientStreamingMethod, &stub, &res, &cntl);
> writer->Write(req);
> writer->WritesDone();
> writer->Finish();
>
> // 服务端流模式（在一次调用中, 客户端只能一次向服务器传输数据, 但是服务器可以多次返回响应）
> // rpc ServerStreamingMethod (Request) returns (stream Response);
> auto reader = CreateStreamVisitor<demo::Response, GrpcReader>(
> &demo::GRPCDemo_Stub::ServerStreamingMethod, &stub, &req, &cntl);
> while (reader->Read(&res)) {
> std::cout << ... << std::endl;
> }
> reader->Finish();
>
> // 双向流模式 (在一次调用中, 客户端和服务器都可以向对方多次收发数据)
> // rpc BidirectionalStreamingMethod (stream Request) returns (stream Response);
> auto stream = CreateStreamVisitor<demo::Request, demo::Response, GrpcReaderWriter>(
> &demo::GRPCDemo_Stub::BidirectionalStreamingMethod, &stub, &cntl);
> std::thread writer([stream]() {
> for (int i = 0; i < 3; ++i) {
> stream->Write(req);
> }
> stream->WritesDone();
> });
> while (stream->Read(&res)) {
> std::cout << ... << std::endl;
> }
> writer.join();
> stream->Finish();
> ```
>
> 具体设计 TBD
>
> **Describe alternatives you've considered (描述你想到的折衷方案)** 简单地直接引入GRPC依赖
>
> **Additional context/screenshots (更多上下文/截图)**
> 接口好像有点多。 WritesDone和Finish是什么关系？可不可以合为一个。 writer->Finish()和stream->Finisher()又是什么关系? 没发送完毕，业务要中途退出是要如何处理(writer/reader端都可能发生)? 本身rpc接口中的done这里要如何处理？ 这个case并没有看到。 能已echo为例子写一个实际的包含错误处理的例子么。
已更新case
1、2 最好不要合一个，WritesDone用来保证发送端数据完整性，Finish来关闭整个流；
3、done还未考虑
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] chenzhangyi commented on issue #1589: BRPC兼容GRPC stream
Posted by GitBox <gi...@apache.org>.
chenzhangyi commented on issue #1589:
URL: https://github.com/apache/incubator-brpc/issues/1589#issuecomment-968614244
> > 接口好像有点多。 WritesDone和Finish是什么关系？可不可以合为一个。 writer->Finish()和stream->Finisher()又是什么关系? 没发送完毕，业务要中途退出是要如何处理(writer/reader端都可能发生)? 本身rpc接口中的done这里要如何处理？ 这个case并没有看到。 能已echo为例子写一个实际的包含错误处理的例子么。
>
> 已更新case 1、2 最好不要合一个，WritesDone用来保证发送端数据完整性，Finish来关闭整个流； 3、done还未考虑
调用done->Run()也可以关闭整个流？ 看起来一件事情似乎N个接口都可以做，这个可能已经不是很好的迹象了。
参考tcp的接口，只有close, 没有reset. 甚至连half close都没提供。因为对于调用者来说，这几个接口一般都是同时调用的。 这可能已经意味着只需要保留一个。除非能找到常见的case这两个需要分别调用。否则最好是只保留一个。有遇到特殊情况再提供能分开调用的函数。比如默认就是Close(). 有特殊case再提供CloseRead(), CloseWrite().
另外一个问题， 这个stream接口双工/单工的是如何设置的？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] KaneVV1 commented on issue #1589: BRPC兼容GRPC stream
Posted by GitBox <gi...@apache.org>.
KaneVV1 commented on issue #1589:
URL: https://github.com/apache/incubator-brpc/issues/1589#issuecomment-968593041
> 接口好像有点多。 WritesDone和Finish是什么关系？可不可以合为一个。 writer->Finish()和stream->Finisher()又是什么关系? 没发送完毕，业务要中途退出是要如何处理(writer/reader端都可能发生)? 本身rpc接口中的done这里要如何处理？ 这个case并没有看到。 能已echo为例子写一个实际的包含错误处理的例子么。
已更新case
1、2 最好不要合一个，WritesDone用来保证发送端数据完整性，Finish来关闭整个流；
3、done还未考虑
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] liyichao commented on issue #1589: BRPC兼容GRPC stream
Posted by GitBox <gi...@apache.org>.
liyichao commented on issue #1589:
URL: https://github.com/apache/incubator-brpc/issues/1589#issuecomment-1064933128
这个目前状态如何？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] KaneVV1 commented on issue #1589: BRPC兼容GRPC stream
Posted by GitBox <gi...@apache.org>.
KaneVV1 commented on issue #1589:
URL: https://github.com/apache/incubator-brpc/issues/1589#issuecomment-1068929236
> 这个目前状态如何？
正在开发
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] chenzhangyi commented on issue #1589: BRPC兼容GRPC stream
Posted by GitBox <gi...@apache.org>.
chenzhangyi commented on issue #1589:
URL: https://github.com/apache/incubator-brpc/issues/1589#issuecomment-964882060
grpc stream最大的问题就是没有异步接口。 另外上面这个demo中，好像看不到错误处理的部分? stream不能保证是可靠的。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] KaneVV1 removed a comment on issue #1589: BRPC兼容GRPC stream
Posted by GitBox <gi...@apache.org>.
KaneVV1 removed a comment on issue #1589:
URL: https://github.com/apache/incubator-brpc/issues/1589#issuecomment-976177293
>
目前我们正在开发中，对于超出我们内部的、更加普遍的应用场景，需要更加详细的设计，到时候再与您商讨。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] KaneVV1 commented on issue #1589: BRPC兼容GRPC stream
Posted by GitBox <gi...@apache.org>.
KaneVV1 commented on issue #1589:
URL: https://github.com/apache/incubator-brpc/issues/1589#issuecomment-976177372
> > > 接口好像有点多。 WritesDone和Finish是什么关系？可不可以合为一个。 writer->Finish()和stream->Finisher()又是什么关系? 没发送完毕，业务要中途退出是要如何处理(writer/reader端都可能发生)? 本身rpc接口中的done这里要如何处理？ 这个case并没有看到。 能已echo为例子写一个实际的包含错误处理的例子么。
> >
> >
> > 已更新case 1、2 最好不要合一个，WritesDone用来保证发送端数据完整性，Finish来关闭整个流； 3、done还未考虑
>
> 调用done->Run()也可以关闭整个流？ 看起来一件事情似乎N个接口都可以做，这个可能已经不是很好的迹象了。
>
> 参考tcp的接口，只有close, 没有reset. 甚至连half close都没提供。因为对于调用者来说，这几个接口一般都是同时调用的。 这可能已经意味着只需要保留一个。除非能找到常见的case这两个需要分别调用。否则最好是只保留一个。有遇到特殊情况再提供能分开调用的函数。比如默认就是Close(). 有特殊case再提供CloseRead(), CloseWrite().
>
> 另外一个问题， 这个stream接口双工/单工的是如何设置的？
目前我们正在开发中，对于超出我们内部的、更加普遍的应用场景，需要更加详细的设计，到时候再与您商讨。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] KaneVV1 commented on issue #1589: BRPC兼容GRPC stream
Posted by GitBox <gi...@apache.org>.
KaneVV1 commented on issue #1589:
URL: https://github.com/apache/incubator-brpc/issues/1589#issuecomment-976177293
>
目前我们正在开发中，对于超出我们内部的、更加普遍的应用场景，需要更加详细的设计，到时候再与您商讨。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] chenzhangyi commented on issue #1589: BRPC兼容GRPC stream
Posted by GitBox <gi...@apache.org>.
chenzhangyi commented on issue #1589:
URL: https://github.com/apache/incubator-brpc/issues/1589#issuecomment-968514632
接口好像有点多。
WritesDone和Finish是什么关系？可不可以合为一个。 writer->Finish()和stream->Finisher()又是什么关系?
没发送完毕，业务要中途退出是要如何处理(writer/reader端都可能发生)?
本身rpc接口中的done这里要如何处理？ 这个case并没有看到。
能已echo为例子写一个实际的包含错误处理的例子么。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org