Re: gRPC protocol handler

[Jonathan Ellis <jb...@gmail.com>]

Resurrecting this thread.  Two points to make:

First, it's completely fine for Apache projects to add someone as a
committer to maintain a specific part of the tree, and I believe Pulsar has
already done this for the .NET driver.  So if we agree in principle that a
protocol handler or other piece of code should be in-tree, maintainership
shouldn't be an obstacle.

Second, I think the right approach is to have a mix -- popular protocols
that play well with Pulsar's semantics should be maintained and distributed
with Pulsar to provide the least friction possible.

In particular, DataStax has seen a lot of interest in our Fast JMS
implementation https://github.com/datastax/pulsar-jms and we would like to
donate it to Apache Pulsar.


On Tue, Jul 27, 2021 at 11:31 PM Sijie Guo <gu...@gmail.com> wrote:

> Here are my two cents.
>
> It is great to have more protocol handlers. I'd encourage people to
> maintain the protocol handler themselves rather than pushing to the
> upstream.
>
> 1. It is very hard for the community to maintain all protocol handlers
> upstream and keep them updated. Similar situations happen to
> connectors. It is also commonly seen in other projects like Spark and
> Flink. The creator of the protocol handler is the best person to
> maintain the protocol handler. You can grow the community for a given
> protocol handler as well. It is how we build and maintain KoP, AoP,
> and MoP.
>
> 2. gRPC isn't a zero-copy protocol. It is not well suited for
> implementing high-throughput streaming services.
>
> 3. Unlike KoP, MoP and AoP which are implementing well-known messaging
> protocols, the gRPC protocol handler is not an extension to support a
> different protocol. It is essentially re-implementing Pulsar's
> protocol in gRPC. This leads to a serious problem as it is competing
> with the original protocol.  It will confuse the community on what to
> use. Instead of introducing another protocol handler to re-implement
> Pulsar's protocol, I would suggest discussing whether gRPC is suitable
> for Pulsar or how we can involve Pulsar's current protocol.
>
> 4. If you want to implement a gRPC based service, an alternative would
> be adding a Google Pub/Sub protocol handler to support the protocol of
> Google Pub/Sub.
>
> -  Sijie
>
> On Tue, Jul 27, 2021 at 6:56 AM Christophe Bornet
> <bo...@gmail.com> wrote:
> >
> > Hi Apache Pulsar community,
> >
> > I’m a huge fan of the potential that the pluggable protocol handlers
> > introduced in Pulsar 2.5 unlock for increasing Pulsar popularity by
> > reducing the friction to adopt it. This includes both general-purpose
> APIs
> > like REST and gRPC, and compatibility layers for other streaming products
> > like Kafka and AMQP.
> >
> > I’ve created a PIP
> > <https://github.com/apache/pulsar/wiki/PIP-59%3A-gPRC-Protocol-Handler>
> and
> > an implementation of a gRPC protocol handler, but review of the pull
> > request stalled out <https://github.com/apache/pulsar/pull/6512> so I
> moved
> > my implementation to my own repository
> > <https://github.com/cbornet/pulsar-grpc> while I finished it.
> >
> > With development and testing effectively complete, I’d like to reopen the
> > discussion:
> >
> >
> >    1.
> >
> >    How can I move forward with contributing my gRPC protocol handler to
> >    Apache Pulsar? Is this something that is interesting and useful to the
> >    broader community?
> >    2.
> >
> >    More broadly, what policy do we want to establish for Pulsar protocol
> >    handlers and other compatibility layers?  As indicated above, I think
> that
> >    we should provide Pulsar with “batteries included” to make it as easy
> as
> >    possible for new users to get started.
> >
> >
> > Best regards.
> >
> > Christophe
>


-- 
Jonathan Ellis
co-founder, http://www.datastax.com
@spyced

Re: gRPC protocol handler

[Sijie Guo <gu...@gmail.com>]

Supporting other popular messaging protocols is good in general.

However, I think adding a similar protocol that is similar to the current
Pulsar protocol is confusing.

When you introduce a new protocol, you need to provide the client
implementations for this protocol.

The idea of protocol handler is to allow Pulsar to support existing
protocols where the client implementations are already available.

However, if you are introducing a protocol that is similar to the current
Pulsar protocol, you have to rebuild all the existing Pulsar clients. I see
this is a huge flag to the Pulsar project.

- Sijie



On Wed, Sep 8, 2021 at 10:08 AM Jonathan Ellis <jb...@gmail.com> wrote:

> Resurrecting this thread.  Two points to make:
>
> First, it's completely fine for Apache projects to add someone as a
> committer to maintain a specific part of the tree, and I believe Pulsar has
> already done this for the .NET driver.  So if we agree in principle that a
> protocol handler or other piece of code should be in-tree, maintainership
> shouldn't be an obstacle.
>
> Second, I think the right approach is to have a mix -- popular protocols
> that play well with Pulsar's semantics should be maintained and distributed
> with Pulsar to provide the least friction possible.
>
> In particular, DataStax has seen a lot of interest in our Fast JMS
> implementation https://github.com/datastax/pulsar-jms and we would like to
> donate it to Apache Pulsar.
>
>
> On Tue, Jul 27, 2021 at 11:31 PM Sijie Guo <gu...@gmail.com> wrote:
>
> > Here are my two cents.
> >
> > It is great to have more protocol handlers. I'd encourage people to
> > maintain the protocol handler themselves rather than pushing to the
> > upstream.
> >
> > 1. It is very hard for the community to maintain all protocol handlers
> > upstream and keep them updated. Similar situations happen to
> > connectors. It is also commonly seen in other projects like Spark and
> > Flink. The creator of the protocol handler is the best person to
> > maintain the protocol handler. You can grow the community for a given
> > protocol handler as well. It is how we build and maintain KoP, AoP,
> > and MoP.
> >
> > 2. gRPC isn't a zero-copy protocol. It is not well suited for
> > implementing high-throughput streaming services.
> >
> > 3. Unlike KoP, MoP and AoP which are implementing well-known messaging
> > protocols, the gRPC protocol handler is not an extension to support a
> > different protocol. It is essentially re-implementing Pulsar's
> > protocol in gRPC. This leads to a serious problem as it is competing
> > with the original protocol.  It will confuse the community on what to
> > use. Instead of introducing another protocol handler to re-implement
> > Pulsar's protocol, I would suggest discussing whether gRPC is suitable
> > for Pulsar or how we can involve Pulsar's current protocol.
> >
> > 4. If you want to implement a gRPC based service, an alternative would
> > be adding a Google Pub/Sub protocol handler to support the protocol of
> > Google Pub/Sub.
> >
> > -  Sijie
> >
> > On Tue, Jul 27, 2021 at 6:56 AM Christophe Bornet
> > <bo...@gmail.com> wrote:
> > >
> > > Hi Apache Pulsar community,
> > >
> > > I’m a huge fan of the potential that the pluggable protocol handlers
> > > introduced in Pulsar 2.5 unlock for increasing Pulsar popularity by
> > > reducing the friction to adopt it. This includes both general-purpose
> > APIs
> > > like REST and gRPC, and compatibility layers for other streaming
> products
> > > like Kafka and AMQP.
> > >
> > > I’ve created a PIP
> > > <https://github.com/apache/pulsar/wiki/PIP-59%3A-gPRC-Protocol-Handler
> >
> > and
> > > an implementation of a gRPC protocol handler, but review of the pull
> > > request stalled out <https://github.com/apache/pulsar/pull/6512> so I
> > moved
> > > my implementation to my own repository
> > > <https://github.com/cbornet/pulsar-grpc> while I finished it.
> > >
> > > With development and testing effectively complete, I’d like to reopen
> the
> > > discussion:
> > >
> > >
> > >    1.
> > >
> > >    How can I move forward with contributing my gRPC protocol handler to
> > >    Apache Pulsar? Is this something that is interesting and useful to
> the
> > >    broader community?
> > >    2.
> > >
> > >    More broadly, what policy do we want to establish for Pulsar
> protocol
> > >    handlers and other compatibility layers?  As indicated above, I
> think
> > that
> > >    we should provide Pulsar with “batteries included” to make it as
> easy
> > as
> > >    possible for new users to get started.
> > >
> > >
> > > Best regards.
> > >
> > > Christophe
> >
>
>
> --
> Jonathan Ellis
> co-founder, http://www.datastax.com
> @spyced
>