You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by Zike Yang <zi...@apache.org> on 2023/11/01 02:10:41 UTC
Re: Question about Pulsar gRPC client(s)
Another point is that there are many features implemented on the
client side, including batching, chunking, DLQ, etc. This makes it
hard to replace the existing pulsar clients completely.
Zike Yang
On Wed, Nov 1, 2023 at 4:43 AM Christophe Bornet <bo...@gmail.com> wrote:
>
> Hi Kiryl,
>
> Thanks for mentioning pulsar-grpc.
> Indeed, using gRPC simplifies the implementation of the networking logic
> (keep-alive, reconnection, flow control,…). On the other hand, the Java
> gRPC implementation makes a lot of buffer copies to cleanly separate the
> network and app layers but that takes a toll on performance. Compared to
> that, the broker Pulsar protocol impl is optimized to not do copies between
> the consumer/producer endpoints and the bookkeeper client.
> So I think we could not replace completely the Pulsar protocol by gRPC.
> We could maybe support both but it’s a lot of work to maintain both
> protocols. (I kind of gave up maintaining pulsar-grpc because of the amount
> of work compared to the number of users, but if there’s interest I can
> reconsider).
> Another possibility would be to do a proxy instead of a low-level protocol
> handler. A bit like the WebSocket proxy. This would be far less work to
> maintain as it would use the Pulsar protocol to communicate with the
> brokers. It could be done as a Proxy extension. Compared to the WS proxy,
> this would provide type safety, discovery, and so on…
> As for the Admin, it’s a bit the same. It would be a bunch of work to
> support both gRPC and REST. You have some kind of type hinting with the
> OpenAPI spec that you can use to generate client SDKs (eg. with
> openapi-generator.
> I wonder what others have to say.
>
> Christophe
>
>
> Le mar. 31 oct. 2023 à 19:57, Kiryl Valkovich <ki...@teal.tools>
> a écrit :
>
> > Hi! Am I understanding it right, that if this project
> > https://github.com/cbornet/pulsar-grpc is merged to the apache/pulsar
> > repo, we could easily cover non-mainstream platforms that are supported by
> > gRPC, but don't have ready-to-use Pulsar clients?
> >
> > https://github.com/apache/pulsar/wiki/PIP-59:-gPRC-Protocol-Handler
> >
> > Similar to already supported language-agnostic client interfaces - REST
> > and WebSocket.
> >
> > Actively maintained gRPC libraries I found (19, or 15 if considering JVM
> > languages and web as duplicates):
> > - [C# / .NET](https://grpc.io/docs/languages/csharp/)
> > - [C++](https://grpc.io/docs/languages/cpp/)
> > - [Dart](https://grpc.io/docs/languages/dart/)
> > - [Go](https://grpc.io/docs/languages/go/)
> > - [Java](https://grpc.io/docs/languages/java/)
> > - [Kotlin](https://grpc.io/docs/languages/kotlin/)
> > - [Node](https://grpc.io/docs/languages/node/)
> > - [Objective-C](https://grpc.io/docs/languages/objective-c/)
> > - [PHP](https://grpc.io/docs/languages/php/)
> > - [Python](https://grpc.io/docs/languages/python/)
> > - [Ruby](https://grpc.io/docs/languages/ruby/)
> > - [OCaml](https://github.com/dialohq/ocaml-grpc)
> > - [Haskell](https://github.com/awakesecurity/gRPC-haskell)
> > - [Elixir](https://github.com/elixir-grpc/grpc)
> > - [Rust](https://github.com/hyperium/tonic)
> > - [Scala](https://scalapb.github.io/)
> > - [Swift](https://github.com/grpc/grpc-swift)
> > - Web client: https://github.com/grpc/grpc-web
> > - Web client 2: https://connectrpc.com/docs/web/getting-started
> >
> > Actively maintained Pulsar libraries (8):
> > - Java
> > - C++
> > - Python
> > - Go
> > - Node.js
> > - C#
> > - PHP
> > - Rust
> >
> > Is there any reason for not merging it to the apache/pulsar?
> >
> > I would definitely prefer to work with a statically typed gRPC client
> > instead of REST or WebSocket.
> >
> > By the way, the same can work for the Pulsar Admin API. Implement the gRPC
> > server once in Java, and we have full-featured native statically typed
> > (where applicable :)) Pulsar Admin clients for any platform.
> >
> > Best,
> > Kiryl
> >
Re: Question about Pulsar gRPC client(s)
Posted by Dave Fisher <wa...@comcast.net>.
I think that Christophe suggested the protocol handler approach like KoP (also forked as DataStax Starlight for Kafka). Due to the several people on the PMC contributing to that the performance gains were huge and it nearly as performant as Pulsar.
Best,
Dave
Sent from my iPhone
> On Oct 31, 2023, at 8:18 PM, Yunze Xu <xy...@apache.org> wrote:
>
> It's a protocol handler like https://github.com/streamnative/kop that
> supports the Kafka protocol. The protocol handler should not be
> maintained in the core repo because there are a few people maintaining
> the plugin.
>
> And just like Christophe and Zike said, compared with the native
> Pulsar protocol, there is actually some overhead so it cannot fully
> replace the native Pulsar protocol.
>
> Thanks,
> Yunze
>
>> On Wed, Nov 1, 2023 at 10:11 AM Zike Yang <zi...@apache.org> wrote:
>>
>> Another point is that there are many features implemented on the
>> client side, including batching, chunking, DLQ, etc. This makes it
>> hard to replace the existing pulsar clients completely.
>>
>> Zike Yang
>>
>>
>>> On Wed, Nov 1, 2023 at 4:43 AM Christophe Bornet <bo...@gmail.com> wrote:
>>>
>>> Hi Kiryl,
>>>
>>> Thanks for mentioning pulsar-grpc.
>>> Indeed, using gRPC simplifies the implementation of the networking logic
>>> (keep-alive, reconnection, flow control,…). On the other hand, the Java
>>> gRPC implementation makes a lot of buffer copies to cleanly separate the
>>> network and app layers but that takes a toll on performance. Compared to
>>> that, the broker Pulsar protocol impl is optimized to not do copies between
>>> the consumer/producer endpoints and the bookkeeper client.
>>> So I think we could not replace completely the Pulsar protocol by gRPC.
>>> We could maybe support both but it’s a lot of work to maintain both
>>> protocols. (I kind of gave up maintaining pulsar-grpc because of the amount
>>> of work compared to the number of users, but if there’s interest I can
>>> reconsider).
>>> Another possibility would be to do a proxy instead of a low-level protocol
>>> handler. A bit like the WebSocket proxy. This would be far less work to
>>> maintain as it would use the Pulsar protocol to communicate with the
>>> brokers. It could be done as a Proxy extension. Compared to the WS proxy,
>>> this would provide type safety, discovery, and so on…
>>> As for the Admin, it’s a bit the same. It would be a bunch of work to
>>> support both gRPC and REST. You have some kind of type hinting with the
>>> OpenAPI spec that you can use to generate client SDKs (eg. with
>>> openapi-generator.
>>> I wonder what others have to say.
>>>
>>> Christophe
>>>
>>>
>>> Le mar. 31 oct. 2023 à 19:57, Kiryl Valkovich <ki...@teal.tools>
>>> a écrit :
>>>
>>>> Hi! Am I understanding it right, that if this project
>>>> https://github.com/cbornet/pulsar-grpc is merged to the apache/pulsar
>>>> repo, we could easily cover non-mainstream platforms that are supported by
>>>> gRPC, but don't have ready-to-use Pulsar clients?
>>>>
>>>> https://github.com/apache/pulsar/wiki/PIP-59:-gPRC-Protocol-Handler
>>>>
>>>> Similar to already supported language-agnostic client interfaces - REST
>>>> and WebSocket.
>>>>
>>>> Actively maintained gRPC libraries I found (19, or 15 if considering JVM
>>>> languages and web as duplicates):
>>>> - [C# / .NET](https://grpc.io/docs/languages/csharp/)
>>>> - [C++](https://grpc.io/docs/languages/cpp/)
>>>> - [Dart](https://grpc.io/docs/languages/dart/)
>>>> - [Go](https://grpc.io/docs/languages/go/)
>>>> - [Java](https://grpc.io/docs/languages/java/)
>>>> - [Kotlin](https://grpc.io/docs/languages/kotlin/)
>>>> - [Node](https://grpc.io/docs/languages/node/)
>>>> - [Objective-C](https://grpc.io/docs/languages/objective-c/)
>>>> - [PHP](https://grpc.io/docs/languages/php/)
>>>> - [Python](https://grpc.io/docs/languages/python/)
>>>> - [Ruby](https://grpc.io/docs/languages/ruby/)
>>>> - [OCaml](https://github.com/dialohq/ocaml-grpc)
>>>> - [Haskell](https://github.com/awakesecurity/gRPC-haskell)
>>>> - [Elixir](https://github.com/elixir-grpc/grpc)
>>>> - [Rust](https://github.com/hyperium/tonic)
>>>> - [Scala](https://scalapb.github.io/)
>>>> - [Swift](https://github.com/grpc/grpc-swift)
>>>> - Web client: https://github.com/grpc/grpc-web
>>>> - Web client 2: https://connectrpc.com/docs/web/getting-started
>>>>
>>>> Actively maintained Pulsar libraries (8):
>>>> - Java
>>>> - C++
>>>> - Python
>>>> - Go
>>>> - Node.js
>>>> - C#
>>>> - PHP
>>>> - Rust
>>>>
>>>> Is there any reason for not merging it to the apache/pulsar?
>>>>
>>>> I would definitely prefer to work with a statically typed gRPC client
>>>> instead of REST or WebSocket.
>>>>
>>>> By the way, the same can work for the Pulsar Admin API. Implement the gRPC
>>>> server once in Java, and we have full-featured native statically typed
>>>> (where applicable :)) Pulsar Admin clients for any platform.
>>>>
>>>> Best,
>>>> Kiryl
>>>>
Re: Question about Pulsar gRPC client(s)
Posted by Yunze Xu <xy...@apache.org>.
It's a protocol handler like https://github.com/streamnative/kop that
supports the Kafka protocol. The protocol handler should not be
maintained in the core repo because there are a few people maintaining
the plugin.
And just like Christophe and Zike said, compared with the native
Pulsar protocol, there is actually some overhead so it cannot fully
replace the native Pulsar protocol.
Thanks,
Yunze
On Wed, Nov 1, 2023 at 10:11 AM Zike Yang <zi...@apache.org> wrote:
>
> Another point is that there are many features implemented on the
> client side, including batching, chunking, DLQ, etc. This makes it
> hard to replace the existing pulsar clients completely.
>
> Zike Yang
>
>
> On Wed, Nov 1, 2023 at 4:43 AM Christophe Bornet <bo...@gmail.com> wrote:
> >
> > Hi Kiryl,
> >
> > Thanks for mentioning pulsar-grpc.
> > Indeed, using gRPC simplifies the implementation of the networking logic
> > (keep-alive, reconnection, flow control,…). On the other hand, the Java
> > gRPC implementation makes a lot of buffer copies to cleanly separate the
> > network and app layers but that takes a toll on performance. Compared to
> > that, the broker Pulsar protocol impl is optimized to not do copies between
> > the consumer/producer endpoints and the bookkeeper client.
> > So I think we could not replace completely the Pulsar protocol by gRPC.
> > We could maybe support both but it’s a lot of work to maintain both
> > protocols. (I kind of gave up maintaining pulsar-grpc because of the amount
> > of work compared to the number of users, but if there’s interest I can
> > reconsider).
> > Another possibility would be to do a proxy instead of a low-level protocol
> > handler. A bit like the WebSocket proxy. This would be far less work to
> > maintain as it would use the Pulsar protocol to communicate with the
> > brokers. It could be done as a Proxy extension. Compared to the WS proxy,
> > this would provide type safety, discovery, and so on…
> > As for the Admin, it’s a bit the same. It would be a bunch of work to
> > support both gRPC and REST. You have some kind of type hinting with the
> > OpenAPI spec that you can use to generate client SDKs (eg. with
> > openapi-generator.
> > I wonder what others have to say.
> >
> > Christophe
> >
> >
> > Le mar. 31 oct. 2023 à 19:57, Kiryl Valkovich <ki...@teal.tools>
> > a écrit :
> >
> > > Hi! Am I understanding it right, that if this project
> > > https://github.com/cbornet/pulsar-grpc is merged to the apache/pulsar
> > > repo, we could easily cover non-mainstream platforms that are supported by
> > > gRPC, but don't have ready-to-use Pulsar clients?
> > >
> > > https://github.com/apache/pulsar/wiki/PIP-59:-gPRC-Protocol-Handler
> > >
> > > Similar to already supported language-agnostic client interfaces - REST
> > > and WebSocket.
> > >
> > > Actively maintained gRPC libraries I found (19, or 15 if considering JVM
> > > languages and web as duplicates):
> > > - [C# / .NET](https://grpc.io/docs/languages/csharp/)
> > > - [C++](https://grpc.io/docs/languages/cpp/)
> > > - [Dart](https://grpc.io/docs/languages/dart/)
> > > - [Go](https://grpc.io/docs/languages/go/)
> > > - [Java](https://grpc.io/docs/languages/java/)
> > > - [Kotlin](https://grpc.io/docs/languages/kotlin/)
> > > - [Node](https://grpc.io/docs/languages/node/)
> > > - [Objective-C](https://grpc.io/docs/languages/objective-c/)
> > > - [PHP](https://grpc.io/docs/languages/php/)
> > > - [Python](https://grpc.io/docs/languages/python/)
> > > - [Ruby](https://grpc.io/docs/languages/ruby/)
> > > - [OCaml](https://github.com/dialohq/ocaml-grpc)
> > > - [Haskell](https://github.com/awakesecurity/gRPC-haskell)
> > > - [Elixir](https://github.com/elixir-grpc/grpc)
> > > - [Rust](https://github.com/hyperium/tonic)
> > > - [Scala](https://scalapb.github.io/)
> > > - [Swift](https://github.com/grpc/grpc-swift)
> > > - Web client: https://github.com/grpc/grpc-web
> > > - Web client 2: https://connectrpc.com/docs/web/getting-started
> > >
> > > Actively maintained Pulsar libraries (8):
> > > - Java
> > > - C++
> > > - Python
> > > - Go
> > > - Node.js
> > > - C#
> > > - PHP
> > > - Rust
> > >
> > > Is there any reason for not merging it to the apache/pulsar?
> > >
> > > I would definitely prefer to work with a statically typed gRPC client
> > > instead of REST or WebSocket.
> > >
> > > By the way, the same can work for the Pulsar Admin API. Implement the gRPC
> > > server once in Java, and we have full-featured native statically typed
> > > (where applicable :)) Pulsar Admin clients for any platform.
> > >
> > > Best,
> > > Kiryl
> > >