You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by Michael Marshall <mm...@apache.org> on 2023/03/09 22:34:49 UTC
[DISCUSS] PIP-257: Add Open ID Connect Support to Server Components
Hi Pulsar Community,
I would like to contribute Open ID Connect support to the server
components in Pulsar. Here is a link to the PIP:
https://github.com/apache/pulsar/issues/19771. I plan to start working
on the implementation next week. I look forward to your feedback.
Thanks,
Michael
### Motivation
Apache Pulsar does not yet support a server side
`AuthenticationProvider` that implements the Open ID Connect spec for
a relying party as defined by https://openid.net/connect/. The only
token based authentication is provided via the
`AuthenticationProviderToken` class. Given that we already have
clients that implement the OAuth2.0 protocol, which integrates easily
with an Open ID Connect `AuthenticationProvider`, it would be very
helpful to add this support to the Pulsar Server components.
### Goal
In implementing the OIDC spec, we will fulfill both the core
(https://openid.net/specs/openid-connect-core-1_0.html) and the
discovery (https://openid.net/specs/openid-connect-discovery-1_0.html)
portions of the spec in the `AuthenticationProvider` implementation.
The end result will be a plugin that:
* supports multiple token issuers
* retrieves the JWKS uri for each issuer from the token issuer's
`/.well-known/openid-configuration` endpoint
* retrieves and caches the JKWS when a client attempts to connect
using a token issued by one of the trusted issuers
* refreshes the JWKS after a configured amount of time, which allows
for seamless key rotation without needing to restart the proxy,
broker, function worker, websocket proxy. (Restarts are still needed
to mitigate problems like leaked private keys.)
* verifies that a token's signature and claims are valid
### API Changes
There will be two new public classes:
1. Implement `AuthenticationProvider` with
`AuthenticationProviderOpenIDConnect`.
2. Implement `KubernetesFunctionAuthProvider` with
`KubernetesSecretsAuthProvider`.
### Implementation
Add a module to the apache/pulsar repo named `pulsar-openid-connect`
where we will implement the logic for
`AuthenticationProviderOpenIDConnect`. The core additions will include
the ability to discovery the JWKS URI for each token issuer, following
the Open ID Connect Discovery protocol, then retrieving and caching
the JWKS.
Create a class named `KubernetesSecretsAuthProvider` that mounts a
pre-existing secret into the Pulsar Function pod. This class serves
two purposes. First, the function worker does not need to have the
credentials used by the function pod. Second, the current
authentication flow in the function worker is to forward the
authentication data used to create the function. Because OpenID
Connect supports short lived tokens, it is not valid to assume that
the function pod can operate with the supplied authentication data.
Instead, a user will be able to configure the function pod with the
client id and the client secret necessary to retrieve the
authentication token from the OAuth 2 Authorization Server.
### Alternatives
There was an initial attempt to implement this feature here:
https://github.com/apache/pulsar/pull/11794. The PR was never
completed, so it is closed now.
### Anything else?
We will need to address the proxy authentication handling in order for
users to avoid encountering some of the issues documented here
https://github.com/apache/pulsar/issues/10816. I plan to follow up
with a fix for this issue.
Re: [DISCUSS] PIP-257: Add Open ID Connect Support to Server Components
Posted by Xiangying Meng <xi...@apache.org>.
Thanks for initiating the proposal, Michael.
I strongly support this addition, which will greatly enhance Pulsar's
security and reliability.
> I am new to this security component, but how do we support token revoke?
Regarding Heesung Sohn's question about token revocation,
I believe that we can initially rely on short-lived tokens and periodic
refreshes to ensure security.
If the community has a clear demand for token revocation in the future, we
can consider implementing this feature at that time.
I'm really looking forward to the successful implementation of this
proposal.
Best regards,
Xiangying
On Fri, Apr 7, 2023 at 1:57 AM Heesung Sohn
<he...@streamnative.io.invalid> wrote:
> Hi,
>
> I am new to this security component, but how do we support token revoke?
>
> Thanks,
> Heesung
>
> On Mon, Mar 27, 2023 at 12:31 PM Michael Marshall <mm...@apache.org>
> wrote:
>
> > Here is the K8s integration: https://github.com/apache/pulsar/pull/19888
> .
> >
> > That PR makes it possible to configure a function running in
> > kubernetes to use the Service Account Token provided by kubernetes.
> >
> > Future work suggested by Eron Wright is to add support for the
> > function worker to mount tokens for any requested audience. I think
> > that feature will be very valuable, but in hopes of completing this
> > PIP by 3.0.0, I would like to defer that feature.
> >
> > Thanks,
> > Michael
> >
> > On Mon, Mar 20, 2023 at 5:59 PM Michael Marshall <mm...@apache.org>
> > wrote:
> > >
> > > Update: the PR [0] to add the OIDC authentication provider module is
> > > ready for review.
> > >
> > > I plan to start looking at the function worker integration with k8s
> > tomorrow.
> > >
> > > I hope to start the vote later this week.
> > >
> > > Thanks,
> > > Michael
> > >
> > > [0] https://github.com/apache/pulsar/pull/19849
> > >
> > > On Mon, Mar 20, 2023 at 5:56 PM Michael Marshall <mmarshall@apache.org
> >
> > wrote:
> > > >
> > > > >> 2. Implement `KubernetesFunctionAuthProvider` with
> > > > >`KubernetesSecretsAuthProvider`.
> > > >
> > > > >It looks like we add an authentication provider for the Kubernetes
> > > > >environment. Is the OIDC authentication provider?
> > > >
> > > > The current KubernetesSecretsTokenAuthProvider [0] mounts the auth
> > > > data used to create a function. Because OIDC often has short lived
> > > > tokens, it won't work to copy the token from the call used to create
> > > > the function. Instead, my initial proposal was to let a user specify
> a
> > > > pre-existing k8s secret that will have the correct authentication
> > > > data. Because anything can be in the secret, there isn't a reason to
> > > > require this secret to have the client id and client secret.
> > > >
> > > > Eron suggested in the PIP issue [1] that we make it possible to
> easily
> > > > integrate with the Kubernetes Service Account. I'll be looking into
> > > > that integration this week.
> > > >
> > > > Thanks,
> > > > Michael
> > > >
> > > > [0]
> >
> https://github.com/apache/pulsar/blob/82237d3684fe506bcb6426b3b23f413422e6e4fb/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/auth/KubernetesSecretsTokenAuthProvider.java
> > > > [1]
> > https://github.com/apache/pulsar/issues/19771#issuecomment-1463029346
> > > >
> > > >
> > > > On Mon, Mar 13, 2023 at 3:03 AM Zixuan Liu <no...@gmail.com>
> wrote:
> > > > >
> > > > > Hi Michael,
> > > > >
> > > > > +1, Thank you for your PIP! That's important for modern
> > authentication!
> > > > >
> > > > > I have a question:
> > > > >
> > > > > > 2. Implement `KubernetesFunctionAuthProvider` with
> > > > > `KubernetesSecretsAuthProvider`.
> > > > >
> > > > > It looks like we add an authentication provider for the Kubernetes
> > > > > environment. Is the OIDC authentication provider?
> > > > >
> > > > >
> > > > > Thanks,
> > > > > Zixuan
> > > > >
> > > > >
> > > > >
> > > > > Lari Hotari <lh...@apache.org> 于2023年3月10日周五 14:56写道：
> > > > >
> > > > > > Thanks for starting this PIP, Michael.
> > > > > > This is really important in improving Pulsar's security and
> > reducing
> > > > > > certain attack surfaces and eliminating certain attack vectors.
> > I'm looking
> > > > > > forward to having Open ID connect (OIDC) supported in Pulsar
> server
> > > > > > components so that Pulsar could be operated without the use of
> > static JWT
> > > > > > tokens such as the superuser token.
> > > > > >
> > > > > > -Lari
> > > > > >
> > > > > > On 2023/03/09 22:34:49 Michael Marshall wrote:
> > > > > > > Hi Pulsar Community,
> > > > > > >
> > > > > > > I would like to contribute Open ID Connect support to the
> server
> > > > > > > components in Pulsar. Here is a link to the PIP:
> > > > > > > https://github.com/apache/pulsar/issues/19771. I plan to start
> > working
> > > > > > > on the implementation next week. I look forward to your
> feedback.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Michael
> > > > > > >
> > > > > > > ### Motivation
> > > > > > >
> > > > > > > Apache Pulsar does not yet support a server side
> > > > > > > `AuthenticationProvider` that implements the Open ID Connect
> > spec for
> > > > > > > a relying party as defined by https://openid.net/connect/. The
> > only
> > > > > > > token based authentication is provided via the
> > > > > > > `AuthenticationProviderToken` class. Given that we already have
> > > > > > > clients that implement the OAuth2.0 protocol, which integrates
> > easily
> > > > > > > with an Open ID Connect `AuthenticationProvider`, it would be
> > very
> > > > > > > helpful to add this support to the Pulsar Server components.
> > > > > > >
> > > > > > > ### Goal
> > > > > > >
> > > > > > > In implementing the OIDC spec, we will fulfill both the core
> > > > > > > (https://openid.net/specs/openid-connect-core-1_0.html) and
> the
> > > > > > > discovery (
> > https://openid.net/specs/openid-connect-discovery-1_0.html)
> > > > > > > portions of the spec in the `AuthenticationProvider`
> > implementation.
> > > > > > >
> > > > > > > The end result will be a plugin that:
> > > > > > >
> > > > > > > * supports multiple token issuers
> > > > > > >
> > > > > > > * retrieves the JWKS uri for each issuer from the token
> issuer's
> > > > > > > `/.well-known/openid-configuration` endpoint
> > > > > > >
> > > > > > > * retrieves and caches the JKWS when a client attempts to
> connect
> > > > > > > using a token issued by one of the trusted issuers
> > > > > > >
> > > > > > > * refreshes the JWKS after a configured amount of time, which
> > allows
> > > > > > > for seamless key rotation without needing to restart the proxy,
> > > > > > > broker, function worker, websocket proxy. (Restarts are still
> > needed
> > > > > > > to mitigate problems like leaked private keys.)
> > > > > > >
> > > > > > > * verifies that a token's signature and claims are valid
> > > > > > >
> > > > > > > ### API Changes
> > > > > > >
> > > > > > > There will be two new public classes:
> > > > > > >
> > > > > > > 1. Implement `AuthenticationProvider` with
> > > > > > > `AuthenticationProviderOpenIDConnect`.
> > > > > > > 2. Implement `KubernetesFunctionAuthProvider` with
> > > > > > > `KubernetesSecretsAuthProvider`.
> > > > > > >
> > > > > > > ### Implementation
> > > > > > >
> > > > > > > Add a module to the apache/pulsar repo named
> > `pulsar-openid-connect`
> > > > > > > where we will implement the logic for
> > > > > > > `AuthenticationProviderOpenIDConnect`. The core additions will
> > include
> > > > > > > the ability to discovery the JWKS URI for each token issuer,
> > following
> > > > > > > the Open ID Connect Discovery protocol, then retrieving and
> > caching
> > > > > > > the JWKS.
> > > > > > >
> > > > > > > Create a class named `KubernetesSecretsAuthProvider` that
> mounts
> > a
> > > > > > > pre-existing secret into the Pulsar Function pod. This class
> > serves
> > > > > > > two purposes. First, the function worker does not need to have
> > the
> > > > > > > credentials used by the function pod. Second, the current
> > > > > > > authentication flow in the function worker is to forward the
> > > > > > > authentication data used to create the function. Because OpenID
> > > > > > > Connect supports short lived tokens, it is not valid to assume
> > that
> > > > > > > the function pod can operate with the supplied authentication
> > data.
> > > > > > > Instead, a user will be able to configure the function pod with
> > the
> > > > > > > client id and the client secret necessary to retrieve the
> > > > > > > authentication token from the OAuth 2 Authorization Server.
> > > > > > >
> > > > > > > ### Alternatives
> > > > > > >
> > > > > > > There was an initial attempt to implement this feature here:
> > > > > > > https://github.com/apache/pulsar/pull/11794. The PR was never
> > > > > > > completed, so it is closed now.
> > > > > > >
> > > > > > > ### Anything else?
> > > > > > >
> > > > > > > We will need to address the proxy authentication handling in
> > order for
> > > > > > > users to avoid encountering some of the issues documented here
> > > > > > > https://github.com/apache/pulsar/issues/10816. I plan to
> follow
> > up
> > > > > > > with a fix for this issue.
> > > > > > >
> > > > > >
> >
>
Re: [DISCUSS] PIP-257: Add Open ID Connect Support to Server Components
Posted by Heesung Sohn <he...@streamnative.io.INVALID>.
Hi,
I am new to this security component, but how do we support token revoke?
Thanks,
Heesung
On Mon, Mar 27, 2023 at 12:31 PM Michael Marshall <mm...@apache.org>
wrote:
> Here is the K8s integration: https://github.com/apache/pulsar/pull/19888.
>
> That PR makes it possible to configure a function running in
> kubernetes to use the Service Account Token provided by kubernetes.
>
> Future work suggested by Eron Wright is to add support for the
> function worker to mount tokens for any requested audience. I think
> that feature will be very valuable, but in hopes of completing this
> PIP by 3.0.0, I would like to defer that feature.
>
> Thanks,
> Michael
>
> On Mon, Mar 20, 2023 at 5:59 PM Michael Marshall <mm...@apache.org>
> wrote:
> >
> > Update: the PR [0] to add the OIDC authentication provider module is
> > ready for review.
> >
> > I plan to start looking at the function worker integration with k8s
> tomorrow.
> >
> > I hope to start the vote later this week.
> >
> > Thanks,
> > Michael
> >
> > [0] https://github.com/apache/pulsar/pull/19849
> >
> > On Mon, Mar 20, 2023 at 5:56 PM Michael Marshall <mm...@apache.org>
> wrote:
> > >
> > > >> 2. Implement `KubernetesFunctionAuthProvider` with
> > > >`KubernetesSecretsAuthProvider`.
> > >
> > > >It looks like we add an authentication provider for the Kubernetes
> > > >environment. Is the OIDC authentication provider?
> > >
> > > The current KubernetesSecretsTokenAuthProvider [0] mounts the auth
> > > data used to create a function. Because OIDC often has short lived
> > > tokens, it won't work to copy the token from the call used to create
> > > the function. Instead, my initial proposal was to let a user specify a
> > > pre-existing k8s secret that will have the correct authentication
> > > data. Because anything can be in the secret, there isn't a reason to
> > > require this secret to have the client id and client secret.
> > >
> > > Eron suggested in the PIP issue [1] that we make it possible to easily
> > > integrate with the Kubernetes Service Account. I'll be looking into
> > > that integration this week.
> > >
> > > Thanks,
> > > Michael
> > >
> > > [0]
> https://github.com/apache/pulsar/blob/82237d3684fe506bcb6426b3b23f413422e6e4fb/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/auth/KubernetesSecretsTokenAuthProvider.java
> > > [1]
> https://github.com/apache/pulsar/issues/19771#issuecomment-1463029346
> > >
> > >
> > > On Mon, Mar 13, 2023 at 3:03 AM Zixuan Liu <no...@gmail.com> wrote:
> > > >
> > > > Hi Michael,
> > > >
> > > > +1, Thank you for your PIP! That's important for modern
> authentication!
> > > >
> > > > I have a question:
> > > >
> > > > > 2. Implement `KubernetesFunctionAuthProvider` with
> > > > `KubernetesSecretsAuthProvider`.
> > > >
> > > > It looks like we add an authentication provider for the Kubernetes
> > > > environment. Is the OIDC authentication provider?
> > > >
> > > >
> > > > Thanks,
> > > > Zixuan
> > > >
> > > >
> > > >
> > > > Lari Hotari <lh...@apache.org> 于2023年3月10日周五 14:56写道：
> > > >
> > > > > Thanks for starting this PIP, Michael.
> > > > > This is really important in improving Pulsar's security and
> reducing
> > > > > certain attack surfaces and eliminating certain attack vectors.
> I'm looking
> > > > > forward to having Open ID connect (OIDC) supported in Pulsar server
> > > > > components so that Pulsar could be operated without the use of
> static JWT
> > > > > tokens such as the superuser token.
> > > > >
> > > > > -Lari
> > > > >
> > > > > On 2023/03/09 22:34:49 Michael Marshall wrote:
> > > > > > Hi Pulsar Community,
> > > > > >
> > > > > > I would like to contribute Open ID Connect support to the server
> > > > > > components in Pulsar. Here is a link to the PIP:
> > > > > > https://github.com/apache/pulsar/issues/19771. I plan to start
> working
> > > > > > on the implementation next week. I look forward to your feedback.
> > > > > >
> > > > > > Thanks,
> > > > > > Michael
> > > > > >
> > > > > > ### Motivation
> > > > > >
> > > > > > Apache Pulsar does not yet support a server side
> > > > > > `AuthenticationProvider` that implements the Open ID Connect
> spec for
> > > > > > a relying party as defined by https://openid.net/connect/. The
> only
> > > > > > token based authentication is provided via the
> > > > > > `AuthenticationProviderToken` class. Given that we already have
> > > > > > clients that implement the OAuth2.0 protocol, which integrates
> easily
> > > > > > with an Open ID Connect `AuthenticationProvider`, it would be
> very
> > > > > > helpful to add this support to the Pulsar Server components.
> > > > > >
> > > > > > ### Goal
> > > > > >
> > > > > > In implementing the OIDC spec, we will fulfill both the core
> > > > > > (https://openid.net/specs/openid-connect-core-1_0.html) and the
> > > > > > discovery (
> https://openid.net/specs/openid-connect-discovery-1_0.html)
> > > > > > portions of the spec in the `AuthenticationProvider`
> implementation.
> > > > > >
> > > > > > The end result will be a plugin that:
> > > > > >
> > > > > > * supports multiple token issuers
> > > > > >
> > > > > > * retrieves the JWKS uri for each issuer from the token issuer's
> > > > > > `/.well-known/openid-configuration` endpoint
> > > > > >
> > > > > > * retrieves and caches the JKWS when a client attempts to connect
> > > > > > using a token issued by one of the trusted issuers
> > > > > >
> > > > > > * refreshes the JWKS after a configured amount of time, which
> allows
> > > > > > for seamless key rotation without needing to restart the proxy,
> > > > > > broker, function worker, websocket proxy. (Restarts are still
> needed
> > > > > > to mitigate problems like leaked private keys.)
> > > > > >
> > > > > > * verifies that a token's signature and claims are valid
> > > > > >
> > > > > > ### API Changes
> > > > > >
> > > > > > There will be two new public classes:
> > > > > >
> > > > > > 1. Implement `AuthenticationProvider` with
> > > > > > `AuthenticationProviderOpenIDConnect`.
> > > > > > 2. Implement `KubernetesFunctionAuthProvider` with
> > > > > > `KubernetesSecretsAuthProvider`.
> > > > > >
> > > > > > ### Implementation
> > > > > >
> > > > > > Add a module to the apache/pulsar repo named
> `pulsar-openid-connect`
> > > > > > where we will implement the logic for
> > > > > > `AuthenticationProviderOpenIDConnect`. The core additions will
> include
> > > > > > the ability to discovery the JWKS URI for each token issuer,
> following
> > > > > > the Open ID Connect Discovery protocol, then retrieving and
> caching
> > > > > > the JWKS.
> > > > > >
> > > > > > Create a class named `KubernetesSecretsAuthProvider` that mounts
> a
> > > > > > pre-existing secret into the Pulsar Function pod. This class
> serves
> > > > > > two purposes. First, the function worker does not need to have
> the
> > > > > > credentials used by the function pod. Second, the current
> > > > > > authentication flow in the function worker is to forward the
> > > > > > authentication data used to create the function. Because OpenID
> > > > > > Connect supports short lived tokens, it is not valid to assume
> that
> > > > > > the function pod can operate with the supplied authentication
> data.
> > > > > > Instead, a user will be able to configure the function pod with
> the
> > > > > > client id and the client secret necessary to retrieve the
> > > > > > authentication token from the OAuth 2 Authorization Server.
> > > > > >
> > > > > > ### Alternatives
> > > > > >
> > > > > > There was an initial attempt to implement this feature here:
> > > > > > https://github.com/apache/pulsar/pull/11794. The PR was never
> > > > > > completed, so it is closed now.
> > > > > >
> > > > > > ### Anything else?
> > > > > >
> > > > > > We will need to address the proxy authentication handling in
> order for
> > > > > > users to avoid encountering some of the issues documented here
> > > > > > https://github.com/apache/pulsar/issues/10816. I plan to follow
> up
> > > > > > with a fix for this issue.
> > > > > >
> > > > >
>
Re: [DISCUSS] PIP-257: Add Open ID Connect Support to Server Components
Posted by Michael Marshall <mm...@apache.org>.
Here is the K8s integration: https://github.com/apache/pulsar/pull/19888.
That PR makes it possible to configure a function running in
kubernetes to use the Service Account Token provided by kubernetes.
Future work suggested by Eron Wright is to add support for the
function worker to mount tokens for any requested audience. I think
that feature will be very valuable, but in hopes of completing this
PIP by 3.0.0, I would like to defer that feature.
Thanks,
Michael
On Mon, Mar 20, 2023 at 5:59 PM Michael Marshall <mm...@apache.org> wrote:
>
> Update: the PR [0] to add the OIDC authentication provider module is
> ready for review.
>
> I plan to start looking at the function worker integration with k8s tomorrow.
>
> I hope to start the vote later this week.
>
> Thanks,
> Michael
>
> [0] https://github.com/apache/pulsar/pull/19849
>
> On Mon, Mar 20, 2023 at 5:56 PM Michael Marshall <mm...@apache.org> wrote:
> >
> > >> 2. Implement `KubernetesFunctionAuthProvider` with
> > >`KubernetesSecretsAuthProvider`.
> >
> > >It looks like we add an authentication provider for the Kubernetes
> > >environment. Is the OIDC authentication provider?
> >
> > The current KubernetesSecretsTokenAuthProvider [0] mounts the auth
> > data used to create a function. Because OIDC often has short lived
> > tokens, it won't work to copy the token from the call used to create
> > the function. Instead, my initial proposal was to let a user specify a
> > pre-existing k8s secret that will have the correct authentication
> > data. Because anything can be in the secret, there isn't a reason to
> > require this secret to have the client id and client secret.
> >
> > Eron suggested in the PIP issue [1] that we make it possible to easily
> > integrate with the Kubernetes Service Account. I'll be looking into
> > that integration this week.
> >
> > Thanks,
> > Michael
> >
> > [0] https://github.com/apache/pulsar/blob/82237d3684fe506bcb6426b3b23f413422e6e4fb/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/auth/KubernetesSecretsTokenAuthProvider.java
> > [1] https://github.com/apache/pulsar/issues/19771#issuecomment-1463029346
> >
> >
> > On Mon, Mar 13, 2023 at 3:03 AM Zixuan Liu <no...@gmail.com> wrote:
> > >
> > > Hi Michael,
> > >
> > > +1, Thank you for your PIP! That's important for modern authentication!
> > >
> > > I have a question:
> > >
> > > > 2. Implement `KubernetesFunctionAuthProvider` with
> > > `KubernetesSecretsAuthProvider`.
> > >
> > > It looks like we add an authentication provider for the Kubernetes
> > > environment. Is the OIDC authentication provider?
> > >
> > >
> > > Thanks,
> > > Zixuan
> > >
> > >
> > >
> > > Lari Hotari <lh...@apache.org> 于2023年3月10日周五 14:56写道：
> > >
> > > > Thanks for starting this PIP, Michael.
> > > > This is really important in improving Pulsar's security and reducing
> > > > certain attack surfaces and eliminating certain attack vectors. I'm looking
> > > > forward to having Open ID connect (OIDC) supported in Pulsar server
> > > > components so that Pulsar could be operated without the use of static JWT
> > > > tokens such as the superuser token.
> > > >
> > > > -Lari
> > > >
> > > > On 2023/03/09 22:34:49 Michael Marshall wrote:
> > > > > Hi Pulsar Community,
> > > > >
> > > > > I would like to contribute Open ID Connect support to the server
> > > > > components in Pulsar. Here is a link to the PIP:
> > > > > https://github.com/apache/pulsar/issues/19771. I plan to start working
> > > > > on the implementation next week. I look forward to your feedback.
> > > > >
> > > > > Thanks,
> > > > > Michael
> > > > >
> > > > > ### Motivation
> > > > >
> > > > > Apache Pulsar does not yet support a server side
> > > > > `AuthenticationProvider` that implements the Open ID Connect spec for
> > > > > a relying party as defined by https://openid.net/connect/. The only
> > > > > token based authentication is provided via the
> > > > > `AuthenticationProviderToken` class. Given that we already have
> > > > > clients that implement the OAuth2.0 protocol, which integrates easily
> > > > > with an Open ID Connect `AuthenticationProvider`, it would be very
> > > > > helpful to add this support to the Pulsar Server components.
> > > > >
> > > > > ### Goal
> > > > >
> > > > > In implementing the OIDC spec, we will fulfill both the core
> > > > > (https://openid.net/specs/openid-connect-core-1_0.html) and the
> > > > > discovery (https://openid.net/specs/openid-connect-discovery-1_0.html)
> > > > > portions of the spec in the `AuthenticationProvider` implementation.
> > > > >
> > > > > The end result will be a plugin that:
> > > > >
> > > > > * supports multiple token issuers
> > > > >
> > > > > * retrieves the JWKS uri for each issuer from the token issuer's
> > > > > `/.well-known/openid-configuration` endpoint
> > > > >
> > > > > * retrieves and caches the JKWS when a client attempts to connect
> > > > > using a token issued by one of the trusted issuers
> > > > >
> > > > > * refreshes the JWKS after a configured amount of time, which allows
> > > > > for seamless key rotation without needing to restart the proxy,
> > > > > broker, function worker, websocket proxy. (Restarts are still needed
> > > > > to mitigate problems like leaked private keys.)
> > > > >
> > > > > * verifies that a token's signature and claims are valid
> > > > >
> > > > > ### API Changes
> > > > >
> > > > > There will be two new public classes:
> > > > >
> > > > > 1. Implement `AuthenticationProvider` with
> > > > > `AuthenticationProviderOpenIDConnect`.
> > > > > 2. Implement `KubernetesFunctionAuthProvider` with
> > > > > `KubernetesSecretsAuthProvider`.
> > > > >
> > > > > ### Implementation
> > > > >
> > > > > Add a module to the apache/pulsar repo named `pulsar-openid-connect`
> > > > > where we will implement the logic for
> > > > > `AuthenticationProviderOpenIDConnect`. The core additions will include
> > > > > the ability to discovery the JWKS URI for each token issuer, following
> > > > > the Open ID Connect Discovery protocol, then retrieving and caching
> > > > > the JWKS.
> > > > >
> > > > > Create a class named `KubernetesSecretsAuthProvider` that mounts a
> > > > > pre-existing secret into the Pulsar Function pod. This class serves
> > > > > two purposes. First, the function worker does not need to have the
> > > > > credentials used by the function pod. Second, the current
> > > > > authentication flow in the function worker is to forward the
> > > > > authentication data used to create the function. Because OpenID
> > > > > Connect supports short lived tokens, it is not valid to assume that
> > > > > the function pod can operate with the supplied authentication data.
> > > > > Instead, a user will be able to configure the function pod with the
> > > > > client id and the client secret necessary to retrieve the
> > > > > authentication token from the OAuth 2 Authorization Server.
> > > > >
> > > > > ### Alternatives
> > > > >
> > > > > There was an initial attempt to implement this feature here:
> > > > > https://github.com/apache/pulsar/pull/11794. The PR was never
> > > > > completed, so it is closed now.
> > > > >
> > > > > ### Anything else?
> > > > >
> > > > > We will need to address the proxy authentication handling in order for
> > > > > users to avoid encountering some of the issues documented here
> > > > > https://github.com/apache/pulsar/issues/10816. I plan to follow up
> > > > > with a fix for this issue.
> > > > >
> > > >
Re: [DISCUSS] PIP-257: Add Open ID Connect Support to Server Components
Posted by Michael Marshall <mm...@apache.org>.
Update: the PR [0] to add the OIDC authentication provider module is
ready for review.
I plan to start looking at the function worker integration with k8s tomorrow.
I hope to start the vote later this week.
Thanks,
Michael
[0] https://github.com/apache/pulsar/pull/19849
On Mon, Mar 20, 2023 at 5:56 PM Michael Marshall <mm...@apache.org> wrote:
>
> >> 2. Implement `KubernetesFunctionAuthProvider` with
> >`KubernetesSecretsAuthProvider`.
>
> >It looks like we add an authentication provider for the Kubernetes
> >environment. Is the OIDC authentication provider?
>
> The current KubernetesSecretsTokenAuthProvider [0] mounts the auth
> data used to create a function. Because OIDC often has short lived
> tokens, it won't work to copy the token from the call used to create
> the function. Instead, my initial proposal was to let a user specify a
> pre-existing k8s secret that will have the correct authentication
> data. Because anything can be in the secret, there isn't a reason to
> require this secret to have the client id and client secret.
>
> Eron suggested in the PIP issue [1] that we make it possible to easily
> integrate with the Kubernetes Service Account. I'll be looking into
> that integration this week.
>
> Thanks,
> Michael
>
> [0] https://github.com/apache/pulsar/blob/82237d3684fe506bcb6426b3b23f413422e6e4fb/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/auth/KubernetesSecretsTokenAuthProvider.java
> [1] https://github.com/apache/pulsar/issues/19771#issuecomment-1463029346
>
>
> On Mon, Mar 13, 2023 at 3:03 AM Zixuan Liu <no...@gmail.com> wrote:
> >
> > Hi Michael,
> >
> > +1, Thank you for your PIP! That's important for modern authentication!
> >
> > I have a question:
> >
> > > 2. Implement `KubernetesFunctionAuthProvider` with
> > `KubernetesSecretsAuthProvider`.
> >
> > It looks like we add an authentication provider for the Kubernetes
> > environment. Is the OIDC authentication provider?
> >
> >
> > Thanks,
> > Zixuan
> >
> >
> >
> > Lari Hotari <lh...@apache.org> 于2023年3月10日周五 14:56写道：
> >
> > > Thanks for starting this PIP, Michael.
> > > This is really important in improving Pulsar's security and reducing
> > > certain attack surfaces and eliminating certain attack vectors. I'm looking
> > > forward to having Open ID connect (OIDC) supported in Pulsar server
> > > components so that Pulsar could be operated without the use of static JWT
> > > tokens such as the superuser token.
> > >
> > > -Lari
> > >
> > > On 2023/03/09 22:34:49 Michael Marshall wrote:
> > > > Hi Pulsar Community,
> > > >
> > > > I would like to contribute Open ID Connect support to the server
> > > > components in Pulsar. Here is a link to the PIP:
> > > > https://github.com/apache/pulsar/issues/19771. I plan to start working
> > > > on the implementation next week. I look forward to your feedback.
> > > >
> > > > Thanks,
> > > > Michael
> > > >
> > > > ### Motivation
> > > >
> > > > Apache Pulsar does not yet support a server side
> > > > `AuthenticationProvider` that implements the Open ID Connect spec for
> > > > a relying party as defined by https://openid.net/connect/. The only
> > > > token based authentication is provided via the
> > > > `AuthenticationProviderToken` class. Given that we already have
> > > > clients that implement the OAuth2.0 protocol, which integrates easily
> > > > with an Open ID Connect `AuthenticationProvider`, it would be very
> > > > helpful to add this support to the Pulsar Server components.
> > > >
> > > > ### Goal
> > > >
> > > > In implementing the OIDC spec, we will fulfill both the core
> > > > (https://openid.net/specs/openid-connect-core-1_0.html) and the
> > > > discovery (https://openid.net/specs/openid-connect-discovery-1_0.html)
> > > > portions of the spec in the `AuthenticationProvider` implementation.
> > > >
> > > > The end result will be a plugin that:
> > > >
> > > > * supports multiple token issuers
> > > >
> > > > * retrieves the JWKS uri for each issuer from the token issuer's
> > > > `/.well-known/openid-configuration` endpoint
> > > >
> > > > * retrieves and caches the JKWS when a client attempts to connect
> > > > using a token issued by one of the trusted issuers
> > > >
> > > > * refreshes the JWKS after a configured amount of time, which allows
> > > > for seamless key rotation without needing to restart the proxy,
> > > > broker, function worker, websocket proxy. (Restarts are still needed
> > > > to mitigate problems like leaked private keys.)
> > > >
> > > > * verifies that a token's signature and claims are valid
> > > >
> > > > ### API Changes
> > > >
> > > > There will be two new public classes:
> > > >
> > > > 1. Implement `AuthenticationProvider` with
> > > > `AuthenticationProviderOpenIDConnect`.
> > > > 2. Implement `KubernetesFunctionAuthProvider` with
> > > > `KubernetesSecretsAuthProvider`.
> > > >
> > > > ### Implementation
> > > >
> > > > Add a module to the apache/pulsar repo named `pulsar-openid-connect`
> > > > where we will implement the logic for
> > > > `AuthenticationProviderOpenIDConnect`. The core additions will include
> > > > the ability to discovery the JWKS URI for each token issuer, following
> > > > the Open ID Connect Discovery protocol, then retrieving and caching
> > > > the JWKS.
> > > >
> > > > Create a class named `KubernetesSecretsAuthProvider` that mounts a
> > > > pre-existing secret into the Pulsar Function pod. This class serves
> > > > two purposes. First, the function worker does not need to have the
> > > > credentials used by the function pod. Second, the current
> > > > authentication flow in the function worker is to forward the
> > > > authentication data used to create the function. Because OpenID
> > > > Connect supports short lived tokens, it is not valid to assume that
> > > > the function pod can operate with the supplied authentication data.
> > > > Instead, a user will be able to configure the function pod with the
> > > > client id and the client secret necessary to retrieve the
> > > > authentication token from the OAuth 2 Authorization Server.
> > > >
> > > > ### Alternatives
> > > >
> > > > There was an initial attempt to implement this feature here:
> > > > https://github.com/apache/pulsar/pull/11794. The PR was never
> > > > completed, so it is closed now.
> > > >
> > > > ### Anything else?
> > > >
> > > > We will need to address the proxy authentication handling in order for
> > > > users to avoid encountering some of the issues documented here
> > > > https://github.com/apache/pulsar/issues/10816. I plan to follow up
> > > > with a fix for this issue.
> > > >
> > >
Re: [DISCUSS] PIP-257: Add Open ID Connect Support to Server Components
Posted by Michael Marshall <mm...@apache.org>.
>> 2. Implement `KubernetesFunctionAuthProvider` with
>`KubernetesSecretsAuthProvider`.
>It looks like we add an authentication provider for the Kubernetes
>environment. Is the OIDC authentication provider?
The current KubernetesSecretsTokenAuthProvider [0] mounts the auth
data used to create a function. Because OIDC often has short lived
tokens, it won't work to copy the token from the call used to create
the function. Instead, my initial proposal was to let a user specify a
pre-existing k8s secret that will have the correct authentication
data. Because anything can be in the secret, there isn't a reason to
require this secret to have the client id and client secret.
Eron suggested in the PIP issue [1] that we make it possible to easily
integrate with the Kubernetes Service Account. I'll be looking into
that integration this week.
Thanks,
Michael
[0] https://github.com/apache/pulsar/blob/82237d3684fe506bcb6426b3b23f413422e6e4fb/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/auth/KubernetesSecretsTokenAuthProvider.java
[1] https://github.com/apache/pulsar/issues/19771#issuecomment-1463029346
On Mon, Mar 13, 2023 at 3:03 AM Zixuan Liu <no...@gmail.com> wrote:
>
> Hi Michael,
>
> +1, Thank you for your PIP! That's important for modern authentication!
>
> I have a question:
>
> > 2. Implement `KubernetesFunctionAuthProvider` with
> `KubernetesSecretsAuthProvider`.
>
> It looks like we add an authentication provider for the Kubernetes
> environment. Is the OIDC authentication provider?
>
>
> Thanks,
> Zixuan
>
>
>
> Lari Hotari <lh...@apache.org> 于2023年3月10日周五 14:56写道：
>
> > Thanks for starting this PIP, Michael.
> > This is really important in improving Pulsar's security and reducing
> > certain attack surfaces and eliminating certain attack vectors. I'm looking
> > forward to having Open ID connect (OIDC) supported in Pulsar server
> > components so that Pulsar could be operated without the use of static JWT
> > tokens such as the superuser token.
> >
> > -Lari
> >
> > On 2023/03/09 22:34:49 Michael Marshall wrote:
> > > Hi Pulsar Community,
> > >
> > > I would like to contribute Open ID Connect support to the server
> > > components in Pulsar. Here is a link to the PIP:
> > > https://github.com/apache/pulsar/issues/19771. I plan to start working
> > > on the implementation next week. I look forward to your feedback.
> > >
> > > Thanks,
> > > Michael
> > >
> > > ### Motivation
> > >
> > > Apache Pulsar does not yet support a server side
> > > `AuthenticationProvider` that implements the Open ID Connect spec for
> > > a relying party as defined by https://openid.net/connect/. The only
> > > token based authentication is provided via the
> > > `AuthenticationProviderToken` class. Given that we already have
> > > clients that implement the OAuth2.0 protocol, which integrates easily
> > > with an Open ID Connect `AuthenticationProvider`, it would be very
> > > helpful to add this support to the Pulsar Server components.
> > >
> > > ### Goal
> > >
> > > In implementing the OIDC spec, we will fulfill both the core
> > > (https://openid.net/specs/openid-connect-core-1_0.html) and the
> > > discovery (https://openid.net/specs/openid-connect-discovery-1_0.html)
> > > portions of the spec in the `AuthenticationProvider` implementation.
> > >
> > > The end result will be a plugin that:
> > >
> > > * supports multiple token issuers
> > >
> > > * retrieves the JWKS uri for each issuer from the token issuer's
> > > `/.well-known/openid-configuration` endpoint
> > >
> > > * retrieves and caches the JKWS when a client attempts to connect
> > > using a token issued by one of the trusted issuers
> > >
> > > * refreshes the JWKS after a configured amount of time, which allows
> > > for seamless key rotation without needing to restart the proxy,
> > > broker, function worker, websocket proxy. (Restarts are still needed
> > > to mitigate problems like leaked private keys.)
> > >
> > > * verifies that a token's signature and claims are valid
> > >
> > > ### API Changes
> > >
> > > There will be two new public classes:
> > >
> > > 1. Implement `AuthenticationProvider` with
> > > `AuthenticationProviderOpenIDConnect`.
> > > 2. Implement `KubernetesFunctionAuthProvider` with
> > > `KubernetesSecretsAuthProvider`.
> > >
> > > ### Implementation
> > >
> > > Add a module to the apache/pulsar repo named `pulsar-openid-connect`
> > > where we will implement the logic for
> > > `AuthenticationProviderOpenIDConnect`. The core additions will include
> > > the ability to discovery the JWKS URI for each token issuer, following
> > > the Open ID Connect Discovery protocol, then retrieving and caching
> > > the JWKS.
> > >
> > > Create a class named `KubernetesSecretsAuthProvider` that mounts a
> > > pre-existing secret into the Pulsar Function pod. This class serves
> > > two purposes. First, the function worker does not need to have the
> > > credentials used by the function pod. Second, the current
> > > authentication flow in the function worker is to forward the
> > > authentication data used to create the function. Because OpenID
> > > Connect supports short lived tokens, it is not valid to assume that
> > > the function pod can operate with the supplied authentication data.
> > > Instead, a user will be able to configure the function pod with the
> > > client id and the client secret necessary to retrieve the
> > > authentication token from the OAuth 2 Authorization Server.
> > >
> > > ### Alternatives
> > >
> > > There was an initial attempt to implement this feature here:
> > > https://github.com/apache/pulsar/pull/11794. The PR was never
> > > completed, so it is closed now.
> > >
> > > ### Anything else?
> > >
> > > We will need to address the proxy authentication handling in order for
> > > users to avoid encountering some of the issues documented here
> > > https://github.com/apache/pulsar/issues/10816. I plan to follow up
> > > with a fix for this issue.
> > >
> >
Re: [DISCUSS] PIP-257: Add Open ID Connect Support to Server Components
Posted by Zixuan Liu <no...@gmail.com>.
Hi Michael,
+1, Thank you for your PIP! That's important for modern authentication!
I have a question:
> 2. Implement `KubernetesFunctionAuthProvider` with
`KubernetesSecretsAuthProvider`.
It looks like we add an authentication provider for the Kubernetes
environment. Is the OIDC authentication provider?
Thanks,
Zixuan
Lari Hotari <lh...@apache.org> 于2023年3月10日周五 14:56写道：
> Thanks for starting this PIP, Michael.
> This is really important in improving Pulsar's security and reducing
> certain attack surfaces and eliminating certain attack vectors. I'm looking
> forward to having Open ID connect (OIDC) supported in Pulsar server
> components so that Pulsar could be operated without the use of static JWT
> tokens such as the superuser token.
>
> -Lari
>
> On 2023/03/09 22:34:49 Michael Marshall wrote:
> > Hi Pulsar Community,
> >
> > I would like to contribute Open ID Connect support to the server
> > components in Pulsar. Here is a link to the PIP:
> > https://github.com/apache/pulsar/issues/19771. I plan to start working
> > on the implementation next week. I look forward to your feedback.
> >
> > Thanks,
> > Michael
> >
> > ### Motivation
> >
> > Apache Pulsar does not yet support a server side
> > `AuthenticationProvider` that implements the Open ID Connect spec for
> > a relying party as defined by https://openid.net/connect/. The only
> > token based authentication is provided via the
> > `AuthenticationProviderToken` class. Given that we already have
> > clients that implement the OAuth2.0 protocol, which integrates easily
> > with an Open ID Connect `AuthenticationProvider`, it would be very
> > helpful to add this support to the Pulsar Server components.
> >
> > ### Goal
> >
> > In implementing the OIDC spec, we will fulfill both the core
> > (https://openid.net/specs/openid-connect-core-1_0.html) and the
> > discovery (https://openid.net/specs/openid-connect-discovery-1_0.html)
> > portions of the spec in the `AuthenticationProvider` implementation.
> >
> > The end result will be a plugin that:
> >
> > * supports multiple token issuers
> >
> > * retrieves the JWKS uri for each issuer from the token issuer's
> > `/.well-known/openid-configuration` endpoint
> >
> > * retrieves and caches the JKWS when a client attempts to connect
> > using a token issued by one of the trusted issuers
> >
> > * refreshes the JWKS after a configured amount of time, which allows
> > for seamless key rotation without needing to restart the proxy,
> > broker, function worker, websocket proxy. (Restarts are still needed
> > to mitigate problems like leaked private keys.)
> >
> > * verifies that a token's signature and claims are valid
> >
> > ### API Changes
> >
> > There will be two new public classes:
> >
> > 1. Implement `AuthenticationProvider` with
> > `AuthenticationProviderOpenIDConnect`.
> > 2. Implement `KubernetesFunctionAuthProvider` with
> > `KubernetesSecretsAuthProvider`.
> >
> > ### Implementation
> >
> > Add a module to the apache/pulsar repo named `pulsar-openid-connect`
> > where we will implement the logic for
> > `AuthenticationProviderOpenIDConnect`. The core additions will include
> > the ability to discovery the JWKS URI for each token issuer, following
> > the Open ID Connect Discovery protocol, then retrieving and caching
> > the JWKS.
> >
> > Create a class named `KubernetesSecretsAuthProvider` that mounts a
> > pre-existing secret into the Pulsar Function pod. This class serves
> > two purposes. First, the function worker does not need to have the
> > credentials used by the function pod. Second, the current
> > authentication flow in the function worker is to forward the
> > authentication data used to create the function. Because OpenID
> > Connect supports short lived tokens, it is not valid to assume that
> > the function pod can operate with the supplied authentication data.
> > Instead, a user will be able to configure the function pod with the
> > client id and the client secret necessary to retrieve the
> > authentication token from the OAuth 2 Authorization Server.
> >
> > ### Alternatives
> >
> > There was an initial attempt to implement this feature here:
> > https://github.com/apache/pulsar/pull/11794. The PR was never
> > completed, so it is closed now.
> >
> > ### Anything else?
> >
> > We will need to address the proxy authentication handling in order for
> > users to avoid encountering some of the issues documented here
> > https://github.com/apache/pulsar/issues/10816. I plan to follow up
> > with a fix for this issue.
> >
>
Re: [DISCUSS] PIP-257: Add Open ID Connect Support to Server Components
Posted by Lari Hotari <lh...@apache.org>.
Thanks for starting this PIP, Michael.
This is really important in improving Pulsar's security and reducing certain attack surfaces and eliminating certain attack vectors. I'm looking forward to having Open ID connect (OIDC) supported in Pulsar server components so that Pulsar could be operated without the use of static JWT tokens such as the superuser token.
-Lari
On 2023/03/09 22:34:49 Michael Marshall wrote:
> Hi Pulsar Community,
>
> I would like to contribute Open ID Connect support to the server
> components in Pulsar. Here is a link to the PIP:
> https://github.com/apache/pulsar/issues/19771. I plan to start working
> on the implementation next week. I look forward to your feedback.
>
> Thanks,
> Michael
>
> ### Motivation
>
> Apache Pulsar does not yet support a server side
> `AuthenticationProvider` that implements the Open ID Connect spec for
> a relying party as defined by https://openid.net/connect/. The only
> token based authentication is provided via the
> `AuthenticationProviderToken` class. Given that we already have
> clients that implement the OAuth2.0 protocol, which integrates easily
> with an Open ID Connect `AuthenticationProvider`, it would be very
> helpful to add this support to the Pulsar Server components.
>
> ### Goal
>
> In implementing the OIDC spec, we will fulfill both the core
> (https://openid.net/specs/openid-connect-core-1_0.html) and the
> discovery (https://openid.net/specs/openid-connect-discovery-1_0.html)
> portions of the spec in the `AuthenticationProvider` implementation.
>
> The end result will be a plugin that:
>
> * supports multiple token issuers
>
> * retrieves the JWKS uri for each issuer from the token issuer's
> `/.well-known/openid-configuration` endpoint
>
> * retrieves and caches the JKWS when a client attempts to connect
> using a token issued by one of the trusted issuers
>
> * refreshes the JWKS after a configured amount of time, which allows
> for seamless key rotation without needing to restart the proxy,
> broker, function worker, websocket proxy. (Restarts are still needed
> to mitigate problems like leaked private keys.)
>
> * verifies that a token's signature and claims are valid
>
> ### API Changes
>
> There will be two new public classes:
>
> 1. Implement `AuthenticationProvider` with
> `AuthenticationProviderOpenIDConnect`.
> 2. Implement `KubernetesFunctionAuthProvider` with
> `KubernetesSecretsAuthProvider`.
>
> ### Implementation
>
> Add a module to the apache/pulsar repo named `pulsar-openid-connect`
> where we will implement the logic for
> `AuthenticationProviderOpenIDConnect`. The core additions will include
> the ability to discovery the JWKS URI for each token issuer, following
> the Open ID Connect Discovery protocol, then retrieving and caching
> the JWKS.
>
> Create a class named `KubernetesSecretsAuthProvider` that mounts a
> pre-existing secret into the Pulsar Function pod. This class serves
> two purposes. First, the function worker does not need to have the
> credentials used by the function pod. Second, the current
> authentication flow in the function worker is to forward the
> authentication data used to create the function. Because OpenID
> Connect supports short lived tokens, it is not valid to assume that
> the function pod can operate with the supplied authentication data.
> Instead, a user will be able to configure the function pod with the
> client id and the client secret necessary to retrieve the
> authentication token from the OAuth 2 Authorization Server.
>
> ### Alternatives
>
> There was an initial attempt to implement this feature here:
> https://github.com/apache/pulsar/pull/11794. The PR was never
> completed, so it is closed now.
>
> ### Anything else?
>
> We will need to address the proxy authentication handling in order for
> users to avoid encountering some of the issues documented here
> https://github.com/apache/pulsar/issues/10816. I plan to follow up
> with a fix for this issue.
>