Protocol Handlers in Pulsar Proxy

[Enrico Olivelli <eo...@gmail.com>]

Hello,
Currently we have the ability to add Protocol Handlers to the Pulsar
Broker, this is great, because you can add your code that uses internal
Pulsar APIs and implement your own protocols.

When you run Pulsar in k8s (and this is happening more and more) you need
to run the Pulsar proxy.
The Pulsar proxy is put in front of a Pulsar Cluster and allows clients
outside of the cluster to access with a single endpoint (the proxy may be
replicated, but let's not enter too much into the details).

As we are doing for the Pulsar Broker I would like to add support for
adding ProtocolHandlers to the Pulsar Proxy service.
The API will be the same, apart from the fact that you have access to the
PulsarProxy object instead of the PulsarBroker.

It would be great to see this feature, if you have ever come to create your
own Pulsar proxy in front of a ProtocolHandler you had to deal with:
- Broker Discovery
- Authentication
- Authorization

Reimplementing this, using APIs that are not officially exported by the
internals of Pulsar code, is very error prone and also it is very difficult
to follow Pulsar evolution.

PHs for the Proxy will ease the Deployment of Pulsar with PHs as you do not
need to add other Services/Pods to your cluster.

If this idea sounds good to you I will be happy to write up a PIP and send
the implementation.

Enrico

Re: Protocol Handlers in Pulsar Proxy

[Yuva raj <uv...@gmail.com>]

For K8s Should we use something like this
https://www.envoyproxy.io/docs/envoy/latest/configuration/listeners/network_filters/kafka_broker_filter#config-network-filters-kafka-broker
A pulsar filter in envoy would be great!

On Sat, 28 Aug 2021 at 23:10, Enrico Olivelli <eo...@gmail.com> wrote:

> Joe,
> Thanks for sharing your knowledge
>
>
> Il Sab 28 Ago 2021, 18:34 Joe F <jo...@gmail.com> ha scritto:
>
> >
> > To give some history and context, Pulsar proxy was meant to be a
> barebones
> > TCP proxy when it was built.   It's sole reason to exist was to forward
> > network traffic to the right host.  Discovery/authn/z was a dependency .
> > The way it came around, it was for a narrow use case  (and it was not for
> > k8s), and a quick and dirty solution
> >
> > I am all for  making the life of protocol developers easier. I'm just
> > concerned that something that was hastily done  is now evolving into a
> > full-fledged service in a piece-meal, spaghetti style.  (There was
> another
> > proposal a few days ago for  dynamic proxy roles.)
> >
> > We can let the Proxy evolve like the proverbial ball of mud, or put some
> > thought into it
> >
>
> I am going to post a PIP, a full design document on which we can discuss
> the details
>
> I am sure that now the community will be able to discuss and get to adding
> another future proof feature
>
>
> Regards
> Enrico
>
>
> > -j
> >
> > On Fri, Aug 27, 2021 at 2:04 PM Michael Marshall <mi...@gmail.com>
> > wrote:
> >
> >> +1 Thanks for your proposal, Enrico.
> >>
> >> I completely agree that the Pulsar Proxy is an integral component in a
> >> Pulsar cluster running on k8s. Further, considering that the proxy
> >> interacts with clients as if it were a broker and that we already
> support
> >> protocol handlers in the broker, I think it is a logical next step to
> add
> >> support for protocol handlers in the proxy.
> >>
> >> I look forward to reviewing the PIP.
> >>
> >> - Michael
> >>
> >> On Fri, Aug 27, 2021 at 8:21 AM Enrico Olivelli <eo...@gmail.com>
> >> wrote:
> >>
> >> > Hello,
> >> > Currently we have the ability to add Protocol Handlers to the Pulsar
> >> > Broker, this is great, because you can add your code that uses
> internal
> >> > Pulsar APIs and implement your own protocols.
> >> >
> >> > When you run Pulsar in k8s (and this is happening more and more) you
> >> need
> >> > to run the Pulsar proxy.
> >> > The Pulsar proxy is put in front of a Pulsar Cluster and allows
> clients
> >> > outside of the cluster to access with a single endpoint (the proxy may
> >> be
> >> > replicated, but let's not enter too much into the details).
> >> >
> >> > As we are doing for the Pulsar Broker I would like to add support for
> >> > adding ProtocolHandlers to the Pulsar Proxy service.
> >> > The API will be the same, apart from the fact that you have access to
> >> the
> >> > PulsarProxy object instead of the PulsarBroker.
> >> >
> >> > It would be great to see this feature, if you have ever come to create
> >> your
> >> > own Pulsar proxy in front of a ProtocolHandler you had to deal with:
> >> > - Broker Discovery
> >> > - Authentication
> >> > - Authorization
> >> >
> >> > Reimplementing this, using APIs that are not officially exported by
> the
> >> > internals of Pulsar code, is very error prone and also it is very
> >> difficult
> >> > to follow Pulsar evolution.
> >> >
> >> > PHs for the Proxy will ease the Deployment of Pulsar with PHs as you
> do
> >> not
> >> > need to add other Services/Pods to your cluster.
> >> >
> >> > If this idea sounds good to you I will be happy to write up a PIP and
> >> send
> >> > the implementation.
> >> >
> >> > Enrico
> >> >
> >>
> >
>


-- 
*Thanks*

*Yuvaraj L*

Re: Protocol Handlers in Pulsar Proxy

[Enrico Olivelli <eo...@gmail.com>]

Joe,
Thanks for sharing your knowledge


Il Sab 28 Ago 2021, 18:34 Joe F <jo...@gmail.com> ha scritto:

>
> To give some history and context, Pulsar proxy was meant to be a barebones
> TCP proxy when it was built.   It's sole reason to exist was to forward
> network traffic to the right host.  Discovery/authn/z was a dependency .
> The way it came around, it was for a narrow use case  (and it was not for
> k8s), and a quick and dirty solution
>
> I am all for  making the life of protocol developers easier. I'm just
> concerned that something that was hastily done  is now evolving into a
> full-fledged service in a piece-meal, spaghetti style.  (There was another
> proposal a few days ago for  dynamic proxy roles.)
>
> We can let the Proxy evolve like the proverbial ball of mud, or put some
> thought into it
>

I am going to post a PIP, a full design document on which we can discuss
the details

I am sure that now the community will be able to discuss and get to adding
another future proof feature


Regards
Enrico


> -j
>
> On Fri, Aug 27, 2021 at 2:04 PM Michael Marshall <mi...@gmail.com>
> wrote:
>
>> +1 Thanks for your proposal, Enrico.
>>
>> I completely agree that the Pulsar Proxy is an integral component in a
>> Pulsar cluster running on k8s. Further, considering that the proxy
>> interacts with clients as if it were a broker and that we already support
>> protocol handlers in the broker, I think it is a logical next step to add
>> support for protocol handlers in the proxy.
>>
>> I look forward to reviewing the PIP.
>>
>> - Michael
>>
>> On Fri, Aug 27, 2021 at 8:21 AM Enrico Olivelli <eo...@gmail.com>
>> wrote:
>>
>> > Hello,
>> > Currently we have the ability to add Protocol Handlers to the Pulsar
>> > Broker, this is great, because you can add your code that uses internal
>> > Pulsar APIs and implement your own protocols.
>> >
>> > When you run Pulsar in k8s (and this is happening more and more) you
>> need
>> > to run the Pulsar proxy.
>> > The Pulsar proxy is put in front of a Pulsar Cluster and allows clients
>> > outside of the cluster to access with a single endpoint (the proxy may
>> be
>> > replicated, but let's not enter too much into the details).
>> >
>> > As we are doing for the Pulsar Broker I would like to add support for
>> > adding ProtocolHandlers to the Pulsar Proxy service.
>> > The API will be the same, apart from the fact that you have access to
>> the
>> > PulsarProxy object instead of the PulsarBroker.
>> >
>> > It would be great to see this feature, if you have ever come to create
>> your
>> > own Pulsar proxy in front of a ProtocolHandler you had to deal with:
>> > - Broker Discovery
>> > - Authentication
>> > - Authorization
>> >
>> > Reimplementing this, using APIs that are not officially exported by the
>> > internals of Pulsar code, is very error prone and also it is very
>> difficult
>> > to follow Pulsar evolution.
>> >
>> > PHs for the Proxy will ease the Deployment of Pulsar with PHs as you do
>> not
>> > need to add other Services/Pods to your cluster.
>> >
>> > If this idea sounds good to you I will be happy to write up a PIP and
>> send
>> > the implementation.
>> >
>> > Enrico
>> >
>>
>

Re: Protocol Handlers in Pulsar Proxy

[Joe F <jo...@gmail.com>]

To give some history and context, Pulsar proxy was meant to be a barebones
TCP proxy when it was built.   It's sole reason to exist was to forward
network traffic to the right host.  Discovery/authn/z was a dependency .
The way it came around, it was for a narrow use case  (and it was not for
k8s), and a quick and dirty solution

I am all for  making the life of protocol developers easier. I'm just
concerned that something that was hastily done  is now evolving into a
full-fledged service in a piece-meal, spaghetti style.  (There was another
proposal a few days ago for  dynamic proxy roles.)

We can let the Proxy evolve like the proverbial ball of mud, or put some
thought into it

-j

On Fri, Aug 27, 2021 at 2:04 PM Michael Marshall <mi...@gmail.com>
wrote:

> +1 Thanks for your proposal, Enrico.
>
> I completely agree that the Pulsar Proxy is an integral component in a
> Pulsar cluster running on k8s. Further, considering that the proxy
> interacts with clients as if it were a broker and that we already support
> protocol handlers in the broker, I think it is a logical next step to add
> support for protocol handlers in the proxy.
>
> I look forward to reviewing the PIP.
>
> - Michael
>
> On Fri, Aug 27, 2021 at 8:21 AM Enrico Olivelli <eo...@gmail.com>
> wrote:
>
> > Hello,
> > Currently we have the ability to add Protocol Handlers to the Pulsar
> > Broker, this is great, because you can add your code that uses internal
> > Pulsar APIs and implement your own protocols.
> >
> > When you run Pulsar in k8s (and this is happening more and more) you need
> > to run the Pulsar proxy.
> > The Pulsar proxy is put in front of a Pulsar Cluster and allows clients
> > outside of the cluster to access with a single endpoint (the proxy may be
> > replicated, but let's not enter too much into the details).
> >
> > As we are doing for the Pulsar Broker I would like to add support for
> > adding ProtocolHandlers to the Pulsar Proxy service.
> > The API will be the same, apart from the fact that you have access to the
> > PulsarProxy object instead of the PulsarBroker.
> >
> > It would be great to see this feature, if you have ever come to create
> your
> > own Pulsar proxy in front of a ProtocolHandler you had to deal with:
> > - Broker Discovery
> > - Authentication
> > - Authorization
> >
> > Reimplementing this, using APIs that are not officially exported by the
> > internals of Pulsar code, is very error prone and also it is very
> difficult
> > to follow Pulsar evolution.
> >
> > PHs for the Proxy will ease the Deployment of Pulsar with PHs as you do
> not
> > need to add other Services/Pods to your cluster.
> >
> > If this idea sounds good to you I will be happy to write up a PIP and
> send
> > the implementation.
> >
> > Enrico
> >
>

Re: Protocol Handlers in Pulsar Proxy

[Michael Marshall <mi...@gmail.com>]

+1 Thanks for your proposal, Enrico.

I completely agree that the Pulsar Proxy is an integral component in a
Pulsar cluster running on k8s. Further, considering that the proxy
interacts with clients as if it were a broker and that we already support
protocol handlers in the broker, I think it is a logical next step to add
support for protocol handlers in the proxy.

I look forward to reviewing the PIP.

- Michael

On Fri, Aug 27, 2021 at 8:21 AM Enrico Olivelli <eo...@gmail.com> wrote:

> Hello,
> Currently we have the ability to add Protocol Handlers to the Pulsar
> Broker, this is great, because you can add your code that uses internal
> Pulsar APIs and implement your own protocols.
>
> When you run Pulsar in k8s (and this is happening more and more) you need
> to run the Pulsar proxy.
> The Pulsar proxy is put in front of a Pulsar Cluster and allows clients
> outside of the cluster to access with a single endpoint (the proxy may be
> replicated, but let's not enter too much into the details).
>
> As we are doing for the Pulsar Broker I would like to add support for
> adding ProtocolHandlers to the Pulsar Proxy service.
> The API will be the same, apart from the fact that you have access to the
> PulsarProxy object instead of the PulsarBroker.
>
> It would be great to see this feature, if you have ever come to create your
> own Pulsar proxy in front of a ProtocolHandler you had to deal with:
> - Broker Discovery
> - Authentication
> - Authorization
>
> Reimplementing this, using APIs that are not officially exported by the
> internals of Pulsar code, is very error prone and also it is very difficult
> to follow Pulsar evolution.
>
> PHs for the Proxy will ease the Deployment of Pulsar with PHs as you do not
> need to add other Services/Pods to your cluster.
>
> If this idea sounds good to you I will be happy to write up a PIP and send
> the implementation.
>
> Enrico
>