You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "syun64 (via GitHub)" <gi...@apache.org> on 2023/10/18 17:59:31 UTC
[I] Iceberg Rest Catalog Support for Separate Authorization Server [iceberg]
syun64 opened a new issue, #8869:
URL: https://github.com/apache/iceberg/issues/8869
### Feature Request / Improvement
The design of OAuth2 enables [Separation of Roles](https://www.oauth.com/oauth2-servers/differences-between-oauth-1-2/separation-of-roles/) between the API server and an authorization server. The two roles can be on physically separate servers, and even be on different domain names. In this design, the authorization server needs to know about the app's **client_id** and **client_secret**, but the API server would only ever need to accept access tokens.
The current implementation of the Rest Catalog (in Spark, and in PyIceberg) assumes that the resource server is also the authorization server. Unfortunately this requires the Rest Catalog Server to also take on the responsibility of credentials management. Adding support for an optional authorization server url parameter in the catalog configs would help separate out the responsibility of user/machine client authentication based authorization grant to a dedicated authorization server, and allow the Rest Catalog Server to only be responsible to parsing and validating the access_token that had already been retrieved by the querying engine (Spark/PyIceberg).
The existing Rest Catalog Open API spec already supports Separation of Roles, and no change may be needed to support this feature enhancement. It accepts the Bearer Authorization provided in the header of the requests sent to its non-auth endpoints, and uses that to grant access to the users. Currently, the access_token is provided into these requests by Spark and in PyIceberg in two ways:
1. It is fetched from the Rest Catalog Server from the v1/oauth/tokens endpoint using the Client credentials that is provided in the config
2. It uses a static access_token that was fetched through prior means and provided in the config instead
Approach (2) sort of enables Separation of Roles and allows us to fetch the access token from an external authorization server, but it is an incomplete solution due to the fact that access tokens may have short lifespans. When a token expires, the engine is required to use the existing Client Credentials (or the expired access token?) to re-request a valid access_token through [token exchange](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/rest/auth/OAuth2Util.java#L467). In the case the Rest Catalog Server isn't an authorization server, this is not possible.
I would like to enhance the existing [RestSessionCatalog](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java#L135), [HTTPClient](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/rest/HTTPClient.java#L81) and [OAuth2Util](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/rest/auth/OAuth2Util.java#L190) implementations to optionally take an _authorization_server_url_ config parameter to direct the auth token requests to the separate endpoint instead of the resource server's v1/oauth/tokens endpoint in order to achieve Separation of Roles.
If that sounds good, I'd be happy to put up a PR that implements the above proposal, and also do the same on [iceberg-python](https://github.com/apache/iceberg-python).
[Previous discussion on PyIceberg](https://github.com/apache/iceberg/pull/8400) on a similar issue.
### Query engine
Spark
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Iceberg Rest Catalog Support for a Separate OIDC Authorization Server URI [iceberg]
Posted by "syun64 (via GitHub)" <gi...@apache.org>.
syun64 commented on issue #8869:
URL: https://github.com/apache/iceberg/issues/8869#issuecomment-1771122573
> @syun64 In my org, we have very similar situation where we, unfortunately, can only use an internal procedure to grab auth token (that is quite different from OIDC flow). Based on what you talked about, I believe we are doing quite similar things as you mentioned, that we setup a different auth server, and has to reimplement our own internal version of RestSessionCatalog and OAuth2Util so that our Spark/Flink Iceberg users can leverage our internal process to refresh token on the fly.
>
> That being said, I do see the value of separating out the interface of auth away from the interface of resources and define an auth protocol (interface) to allow more degrees of freedom for auth implementation.
Hi @haizhou-zhao - that's interesting that we share somewhat of a similar concern, that is also quite different.
I think there's two layers of customization here and I would like to note them down just so that we don't lose track of each one:
1. Support for a separate OIDC Authorization Server that is a separate entity from the Rest Catalog Server
2. Support for custom authorization methods that do not follow the OIDC flow
I think the implementation details of the solutions to the two issues will be quite different as well, where (1) will be resolved through introducing an optional parameter support to override the full path of token endpoint to the already existing OAuth2Util class.
On the other hand, (2) will require a considerable amount of refactoring to OAuth2Util to create a BaseAuth interface that requires users to implement core auth methods like fetch_token, and break down many of the assumptions RestSessionCatalog is making about the auth mechanism. And I think solving for (2) at scale will be quite challenging - because there's no single spec for how a custom authorization flow would work that we can base our abstractions on.
I'd like to advocate for this issue to be focused on solving for (1) which is a valid OIDC flow, which is just not supported yet with the Java and Python RestCatalog implementations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Iceberg Rest Catalog Support for Separate Authorization Server [iceberg]
Posted by "haizhou-zhao (via GitHub)" <gi...@apache.org>.
haizhou-zhao commented on issue #8869:
URL: https://github.com/apache/iceberg/issues/8869#issuecomment-1769293387
@syun64 In my org, we have very similar situation where we, unfortunately, can only use an internal procedure to grab auth token (that is quite different from OIDC flow). Based on what you talked about, I believe we are doing quite similar things as you mentioned, that we setup a different auth server, and has to reimplement our own internal version of RestSessionCatalog and OAuth2Util so that our Spark/Flink Iceberg users can leverage our internal process to refresh token on the fly.
That being said, I do see the value of separating out the interface of auth away from the interface of resources and define an auth protocol (interface) to allow more degrees of freedom for auth implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Iceberg Rest Catalog Support for a Separate OIDC Authorization Server URI [iceberg]
Posted by "syun64 (via GitHub)" <gi...@apache.org>.
syun64 closed issue #8869: Iceberg Rest Catalog Support for a Separate OIDC Authorization Server URI
URL: https://github.com/apache/iceberg/issues/8869
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Iceberg Rest Catalog Support for Separate Authorization Server [iceberg]
Posted by "syun64 (via GitHub)" <gi...@apache.org>.
syun64 commented on issue #8869:
URL: https://github.com/apache/iceberg/issues/8869#issuecomment-1769215467
@danielcweeks and @Fokko
I've raised this issue to continue our discussion on the previously closed PyIceberg PR on this parent repository. I've done some follow up research regarding the authorization flows since then, and I still believe that introducing an enhancement that allows our tools to fetch access tokens from a separate authorization server would encourage the adoption of the Rest Catalog even more.
Please let me know if you think this is a good idea for the Java (Spark/Trino) RestCatalog as well as for PyIceberg
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org