[PIP] Broker extensions to provide operators of enterprise-wide clusters better control and flexibility

["Narayanan, Madhavan" <Ma...@intuit.com.INVALID>]

Hi All,

   I request your help to review, discuss and resolve the problem and solution approach outlined in the PIP entry https://github.com/apache/pulsar/issues/12858

Regards,
Madhavan

Re: [PIP] Broker extensions to provide operators of enterprise-wide clusters better control and flexibility

[PengHui Li <pe...@apache.org>]

+1 on this proposal.

For the details, I think we should consider avoiding the breaking change
for the existing broker interceptor(make sure the old interceptor can still
work),

Penghui

On Wed, Dec 8, 2021 at 12:56 PM Sijie Guo <gu...@gmail.com> wrote:

> Agree with Matteo.
>
> +1 on this proposal.
>
> On Tue, Dec 7, 2021 at 4:51 PM Matteo Merli <mm...@apache.org> wrote:
>
> > I'm +1 on this proposal.
> >
> > > It looks interesting, but I see a major problem with this approach.
> > > Basically we would be adding a way to tweak everything in Pulsar, from
> > > Connections to what we are reading and writing to storage.
> >
> > > This feature will become very hard to maintain for users, as Pulsar
> > changes
> > > and there are things that may be different in the future.
> >
> > That is true even today. The interceptor is by its very nature subject
> > to have access to these internal details and thus is more prone to API
> > breaks, either intended or inadvertent.
> > Perhaps it is something that should be communicated more clearly, so
> > that interceptor maintainers are aware of the stability of the APIs.
> >
> > Though, to be clear: I don't think that this proposal increases such
> risk.
> >
> >
> > > I am not clear on how this (allow interception of write and read
> > operations
> > > of a managed ledger and modify the payload)  would work with e2e
> > > encryption.
> > > That is literally a MITM proposal.
> >
> > Yes, it allows you to modify the payload before it gets stored. People
> > can use that for multiple reasons:
> >  * Enforcing data encryption
> >  * Attaching headers or tracing information to the messages
> >
> > I don't see any problem with this approach as it builds on top of the
> > "interceptor" interface to extend its capabilities.
> >
> > This is only used if you supply a custom interceptor in your cluster
> > deployment. Otherwise, there will be no impact to
> > performance/stability.
> >
> >
> >
> >  --
> > Matteo Merli
> > <mm...@apache.org>
> >
> >
> > On Thu, Nov 18, 2021 at 9:16 AM Joe F <jo...@gmail.com> wrote:
> > >
> > > Agree with Enrico.
> > >
> > > I am not clear on how this (allow interception of write and read
> > operations
> > > of a managed ledger and modify the payload)  would work with e2e
> > > encryption.
> > > That is literally a MITM proposal.
> > >
> > > Joe
> > >
> > > On Thu, Nov 18, 2021 at 9:04 AM Enrico Olivelli <eo...@gmail.com>
> > wrote:
> > >
> > > > Madhavan,
> > > > Thanks for sharing your PIP.
> > > > It looks interesting, but I see a major problem with this approach.
> > > > Basically we would be adding a way to tweak everything in Pulsar,
> from
> > > > Connections to what we are reading and writing to storage.
> > > >
> > > > This feature will become very hard to maintain for users, as Pulsar
> > changes
> > > > and there are things that may be different in the future.
> > > >
> > > > We recently had other PIPs that try to add more flexibility and add
> > code
> > > > into Pulsar.
> > > >
> > > > It is not clear to me the kind of operations that you want to cover,
> > > > perhaps we could provide dedicated extensibility points to fulfill
> your
> > > > needs with specific APIs, that we can maintain and for which we can
> > > > guarantee
> > > > the compatibility in the future
> > > >
> > > >
> > > > Enrico
> > > >
> > > > Il giorno gio 18 nov 2021 alle ore 16:31 Narayanan, Madhavan
> > > > <Ma...@intuit.com.invalid> ha scritto:
> > > >
> > > > > Hi All,
> > > > >
> > > > >    I request your help to review, discuss and resolve the problem
> and
> > > > > solution approach outlined in the PIP entry
> > > > > https://github.com/apache/pulsar/issues/12858
> > > > >
> > > > > Regards,
> > > > > Madhavan
> > > > >
> > > >
> >
>

Re: [PIP] Broker extensions to provide operators of enterprise-wide clusters better control and flexibility

[Sijie Guo <gu...@gmail.com>]

Agree with Matteo.

+1 on this proposal.

On Tue, Dec 7, 2021 at 4:51 PM Matteo Merli <mm...@apache.org> wrote:

> I'm +1 on this proposal.
>
> > It looks interesting, but I see a major problem with this approach.
> > Basically we would be adding a way to tweak everything in Pulsar, from
> > Connections to what we are reading and writing to storage.
>
> > This feature will become very hard to maintain for users, as Pulsar
> changes
> > and there are things that may be different in the future.
>
> That is true even today. The interceptor is by its very nature subject
> to have access to these internal details and thus is more prone to API
> breaks, either intended or inadvertent.
> Perhaps it is something that should be communicated more clearly, so
> that interceptor maintainers are aware of the stability of the APIs.
>
> Though, to be clear: I don't think that this proposal increases such risk.
>
>
> > I am not clear on how this (allow interception of write and read
> operations
> > of a managed ledger and modify the payload)  would work with e2e
> > encryption.
> > That is literally a MITM proposal.
>
> Yes, it allows you to modify the payload before it gets stored. People
> can use that for multiple reasons:
>  * Enforcing data encryption
>  * Attaching headers or tracing information to the messages
>
> I don't see any problem with this approach as it builds on top of the
> "interceptor" interface to extend its capabilities.
>
> This is only used if you supply a custom interceptor in your cluster
> deployment. Otherwise, there will be no impact to
> performance/stability.
>
>
>
>  --
> Matteo Merli
> <mm...@apache.org>
>
>
> On Thu, Nov 18, 2021 at 9:16 AM Joe F <jo...@gmail.com> wrote:
> >
> > Agree with Enrico.
> >
> > I am not clear on how this (allow interception of write and read
> operations
> > of a managed ledger and modify the payload)  would work with e2e
> > encryption.
> > That is literally a MITM proposal.
> >
> > Joe
> >
> > On Thu, Nov 18, 2021 at 9:04 AM Enrico Olivelli <eo...@gmail.com>
> wrote:
> >
> > > Madhavan,
> > > Thanks for sharing your PIP.
> > > It looks interesting, but I see a major problem with this approach.
> > > Basically we would be adding a way to tweak everything in Pulsar, from
> > > Connections to what we are reading and writing to storage.
> > >
> > > This feature will become very hard to maintain for users, as Pulsar
> changes
> > > and there are things that may be different in the future.
> > >
> > > We recently had other PIPs that try to add more flexibility and add
> code
> > > into Pulsar.
> > >
> > > It is not clear to me the kind of operations that you want to cover,
> > > perhaps we could provide dedicated extensibility points to fulfill your
> > > needs with specific APIs, that we can maintain and for which we can
> > > guarantee
> > > the compatibility in the future
> > >
> > >
> > > Enrico
> > >
> > > Il giorno gio 18 nov 2021 alle ore 16:31 Narayanan, Madhavan
> > > <Ma...@intuit.com.invalid> ha scritto:
> > >
> > > > Hi All,
> > > >
> > > >    I request your help to review, discuss and resolve the problem and
> > > > solution approach outlined in the PIP entry
> > > > https://github.com/apache/pulsar/issues/12858
> > > >
> > > > Regards,
> > > > Madhavan
> > > >
> > >
>

Re: [PIP] Broker extensions to provide operators of enterprise-wide clusters better control and flexibility

[Matteo Merli <mm...@apache.org>]

I'm +1 on this proposal.

> It looks interesting, but I see a major problem with this approach.
> Basically we would be adding a way to tweak everything in Pulsar, from
> Connections to what we are reading and writing to storage.

> This feature will become very hard to maintain for users, as Pulsar changes
> and there are things that may be different in the future.

That is true even today. The interceptor is by its very nature subject
to have access to these internal details and thus is more prone to API
breaks, either intended or inadvertent.
Perhaps it is something that should be communicated more clearly, so
that interceptor maintainers are aware of the stability of the APIs.

Though, to be clear: I don't think that this proposal increases such risk.


> I am not clear on how this (allow interception of write and read operations
> of a managed ledger and modify the payload)  would work with e2e
> encryption.
> That is literally a MITM proposal.

Yes, it allows you to modify the payload before it gets stored. People
can use that for multiple reasons:
 * Enforcing data encryption
 * Attaching headers or tracing information to the messages

I don't see any problem with this approach as it builds on top of the
"interceptor" interface to extend its capabilities.

This is only used if you supply a custom interceptor in your cluster
deployment. Otherwise, there will be no impact to
performance/stability.



 --
Matteo Merli
<mm...@apache.org>


On Thu, Nov 18, 2021 at 9:16 AM Joe F <jo...@gmail.com> wrote:
>
> Agree with Enrico.
>
> I am not clear on how this (allow interception of write and read operations
> of a managed ledger and modify the payload)  would work with e2e
> encryption.
> That is literally a MITM proposal.
>
> Joe
>
> On Thu, Nov 18, 2021 at 9:04 AM Enrico Olivelli <eo...@gmail.com> wrote:
>
> > Madhavan,
> > Thanks for sharing your PIP.
> > It looks interesting, but I see a major problem with this approach.
> > Basically we would be adding a way to tweak everything in Pulsar, from
> > Connections to what we are reading and writing to storage.
> >
> > This feature will become very hard to maintain for users, as Pulsar changes
> > and there are things that may be different in the future.
> >
> > We recently had other PIPs that try to add more flexibility and add code
> > into Pulsar.
> >
> > It is not clear to me the kind of operations that you want to cover,
> > perhaps we could provide dedicated extensibility points to fulfill your
> > needs with specific APIs, that we can maintain and for which we can
> > guarantee
> > the compatibility in the future
> >
> >
> > Enrico
> >
> > Il giorno gio 18 nov 2021 alle ore 16:31 Narayanan, Madhavan
> > <Ma...@intuit.com.invalid> ha scritto:
> >
> > > Hi All,
> > >
> > >    I request your help to review, discuss and resolve the problem and
> > > solution approach outlined in the PIP entry
> > > https://github.com/apache/pulsar/issues/12858
> > >
> > > Regards,
> > > Madhavan
> > >
> >

Re: [PIP] Broker extensions to provide operators of enterprise-wide clusters better control and flexibility

[Joe F <jo...@gmail.com>]

Agree with Enrico.

I am not clear on how this (allow interception of write and read operations
of a managed ledger and modify the payload)  would work with e2e
encryption.
That is literally a MITM proposal.

Joe

On Thu, Nov 18, 2021 at 9:04 AM Enrico Olivelli <eo...@gmail.com> wrote:

> Madhavan,
> Thanks for sharing your PIP.
> It looks interesting, but I see a major problem with this approach.
> Basically we would be adding a way to tweak everything in Pulsar, from
> Connections to what we are reading and writing to storage.
>
> This feature will become very hard to maintain for users, as Pulsar changes
> and there are things that may be different in the future.
>
> We recently had other PIPs that try to add more flexibility and add code
> into Pulsar.
>
> It is not clear to me the kind of operations that you want to cover,
> perhaps we could provide dedicated extensibility points to fulfill your
> needs with specific APIs, that we can maintain and for which we can
> guarantee
> the compatibility in the future
>
>
> Enrico
>
> Il giorno gio 18 nov 2021 alle ore 16:31 Narayanan, Madhavan
> <Ma...@intuit.com.invalid> ha scritto:
>
> > Hi All,
> >
> >    I request your help to review, discuss and resolve the problem and
> > solution approach outlined in the PIP entry
> > https://github.com/apache/pulsar/issues/12858
> >
> > Regards,
> > Madhavan
> >
>

Re: [PIP] Broker extensions to provide operators of enterprise-wide clusters better control and flexibility

[Enrico Olivelli <eo...@gmail.com>]

Madhavan,
Thanks for sharing your PIP.
It looks interesting, but I see a major problem with this approach.
Basically we would be adding a way to tweak everything in Pulsar, from
Connections to what we are reading and writing to storage.

This feature will become very hard to maintain for users, as Pulsar changes
and there are things that may be different in the future.

We recently had other PIPs that try to add more flexibility and add code
into Pulsar.

It is not clear to me the kind of operations that you want to cover,
perhaps we could provide dedicated extensibility points to fulfill your
needs with specific APIs, that we can maintain and for which we can
guarantee
the compatibility in the future


Enrico

Il giorno gio 18 nov 2021 alle ore 16:31 Narayanan, Madhavan
<Ma...@intuit.com.invalid> ha scritto:

> Hi All,
>
>    I request your help to review, discuss and resolve the problem and
> solution approach outlined in the PIP entry
> https://github.com/apache/pulsar/issues/12858
>
> Regards,
> Madhavan
>