You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "betodealmeida (via GitHub)" <gi...@apache.org> on 2024/04/03 15:40:58 UTC
[PR] chore(OAuth2): refactor for custom OAuth2 clients [superset]
betodealmeida opened a new pull request, #27880:
URL: https://github.com/apache/superset/pull/27880
<!---
Please write the PR title following the conventions at https://www.conventionalcommits.org/en/v1.0.0/
Example:
fix(dashboard): load charts correctly
-->
### SUMMARY
<!--- Describe the change below, including rationale and design decisions -->
This PR is a small refactor of https://github.com/apache/superset/pull/27631, addressing a different workflow for OAuth2 suggested by @villebro, where any number of OAuth2 clients can be configured and assigned manually to databases.
The main change is that the DB engine spec no longer inspects the application config to get the OAuth2 client details (ID, secret, etc.). Instead, the OAuth2 related methods in the DB engine spec now receive a configuration object:
```python
class OAuth2ClientConfig(TypedDict):
id: str
secret: str
scope: str
redirect_uri: str
authorization_request_uri: str
token_request_uri: str
```
The configuration is read from the **database model**:
```python
oauth2_config = database.get_oauth2_config()
```
The current implementation simply builds the config object from `superset_config.py`, but in the future the database could check first for a custom OAuth2 client associated with it and use that instead. Similarly, to check if OAuth2 is enabled in a given database we now do:
```python
if database.is_oauth2_enabled():
pass
```
The current method also simply checks `superset_config.py` for now.
Having a standard configuration being passed to the DB engine spec has helped simplify the OAuth2 related methods. I was able to move the methods from GSheets to the base DB engine spec by adding a few new attributes. All that is needed to support OAuth2 in Gsheets now is:
```python
from shillelagh.adapters.api.gsheets.lib import SCOPES
class GSheetsEngineSpec(ShillelaghEngineSpec):
supports_oauth2 = True
oauth2_scope = " ".join(SCOPES)
oauth2_authorization_request_uri = "https://accounts.google.com/o/oauth2/v2/auth"
oauth2_token_request_uri = "https://oauth2.googleapis.com/token"
oauth2_exception = UnauthenticatedError
@classmethod
def get_url_for_impersonation(
cls,
url: URL,
impersonate_user: bool,
username: str | None,
access_token: str | None,
) -> URL:
if impersonate_user and access_token:
url = url.update_query_dict({"access_token": access_token})
return url
```
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
<!--- Skip this if not applicable -->
### TESTING INSTRUCTIONS
<!--- Required! What steps can be taken to manually verify the changes? -->
### ADDITIONAL INFORMATION
<!--- Check any relevant boxes with "x" -->
<!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->
- [ ] Has associated issue:
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] chore(OAuth2): refactor for custom OAuth2 clients [superset]
Posted by "betodealmeida (via GitHub)" <gi...@apache.org>.
betodealmeida commented on code in PR #27880:
URL: https://github.com/apache/superset/pull/27880#discussion_r1550124554
##########
superset/db_engine_specs/README.md:
##########
@@ -547,65 +547,53 @@ Alternatively, it's also possible to impersonate users by implementing the `upda
Support for authenticating to a database using personal OAuth2 access tokens was introduced in [SIP-85](https://github.com/apache/superset/issues/20300). The Google Sheets DB engine spec is the reference implementation.
-To add support for OAuth2 to a DB engine spec, the following attribute and methods are needed:
+To add support for OAuth2 to a DB engine spec, the following attributes are needed:
Review Comment:
Ah, good point, I'll a note — though I am hoping we can stabilize the API in the next couple months.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] chore(OAuth2): refactor for custom OAuth2 clients [superset]
Posted by "betodealmeida (via GitHub)" <gi...@apache.org>.
betodealmeida commented on code in PR #27880:
URL: https://github.com/apache/superset/pull/27880#discussion_r1550225555
##########
superset/db_engine_specs/README.md:
##########
@@ -547,65 +547,53 @@ Alternatively, it's also possible to impersonate users by implementing the `upda
Support for authenticating to a database using personal OAuth2 access tokens was introduced in [SIP-85](https://github.com/apache/superset/issues/20300). The Google Sheets DB engine spec is the reference implementation.
-To add support for OAuth2 to a DB engine spec, the following attribute and methods are needed:
+To add support for OAuth2 to a DB engine spec, the following attributes are needed:
```python
class BaseEngineSpec:
+ supports_oauth2 = True
oauth2_exception = OAuth2RedirectError
- @staticmethod
- def is_oauth2_enabled() -> bool:
- return False
-
- @staticmethod
- def get_oauth2_authorization_uri(state: OAuth2State) -> str:
- raise NotImplementedError()
-
- @staticmethod
- def get_oauth2_token(code: str, state: OAuth2State) -> OAuth2TokenResponse:
- raise NotImplementedError()
-
- @staticmethod
- def get_oauth2_fresh_token(refresh_token: str) -> OAuth2TokenResponse:
- raise NotImplementedError()
+ oauth2_scope = " ".join([
+ "https://example.org/scope1",
+ "https://example.org/scope2",
+ ])
+ oauth2_authorization_request_uri = "https://example.org/authorize"
+ oauth2_token_request_uri = "https://example.org/token"
```
The `oauth2_exception` is an exception that is raised by `cursor.execute` when OAuth2 is needed. This will start the OAuth2 dance when `BaseEngineSpec.execute` is called, by returning the custom error `OAUTH2_REDIRECT` to the frontend. If the database driver doesn't have a specific exception, it might be necessary to overload the `execute` method in the DB engine spec, so that the `BaseEngineSpec.start_oauth2_dance` method gets called whenever OAuth2 is needed.
-The first method, `is_oauth2_enabled`, is used to inform if the database supports OAuth2. This can be dynamic; for example, the Google Sheets DB engine spec checks if the Superset configuration has the necessary section:
-
-```python
-from flask import current_app
-
+The DB engine should implement logic in either `get_url_for_impersonation` or `update_impersonation_config` to update the connection with the personal access token. See the Google Sheets DB engine spec for a reference implementation.
Review Comment:
Ah, I see.
Interestingly, I always understood it is "_Superset_ is impersonating a user", while your perception is that "the _service account_ is impersonating a user". I agree those are two very different scenarios — OAuth2 starts from zero permissions and increases them; traditional impersonation starts with full permissions and lowers them.
But it just confirms your point, the naming is confusing, and we should definitely clarify it and probably make these two separate things.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] chore(OAuth2): refactor for custom OAuth2 clients [superset]
Posted by "villebro (via GitHub)" <gi...@apache.org>.
villebro commented on code in PR #27880:
URL: https://github.com/apache/superset/pull/27880#discussion_r1550163063
##########
superset/db_engine_specs/README.md:
##########
@@ -547,65 +547,53 @@ Alternatively, it's also possible to impersonate users by implementing the `upda
Support for authenticating to a database using personal OAuth2 access tokens was introduced in [SIP-85](https://github.com/apache/superset/issues/20300). The Google Sheets DB engine spec is the reference implementation.
-To add support for OAuth2 to a DB engine spec, the following attribute and methods are needed:
+To add support for OAuth2 to a DB engine spec, the following attributes are needed:
```python
class BaseEngineSpec:
+ supports_oauth2 = True
oauth2_exception = OAuth2RedirectError
- @staticmethod
- def is_oauth2_enabled() -> bool:
- return False
-
- @staticmethod
- def get_oauth2_authorization_uri(state: OAuth2State) -> str:
- raise NotImplementedError()
-
- @staticmethod
- def get_oauth2_token(code: str, state: OAuth2State) -> OAuth2TokenResponse:
- raise NotImplementedError()
-
- @staticmethod
- def get_oauth2_fresh_token(refresh_token: str) -> OAuth2TokenResponse:
- raise NotImplementedError()
+ oauth2_scope = " ".join([
+ "https://example.org/scope1",
+ "https://example.org/scope2",
+ ])
+ oauth2_authorization_request_uri = "https://example.org/authorize"
+ oauth2_token_request_uri = "https://example.org/token"
```
The `oauth2_exception` is an exception that is raised by `cursor.execute` when OAuth2 is needed. This will start the OAuth2 dance when `BaseEngineSpec.execute` is called, by returning the custom error `OAUTH2_REDIRECT` to the frontend. If the database driver doesn't have a specific exception, it might be necessary to overload the `execute` method in the DB engine spec, so that the `BaseEngineSpec.start_oauth2_dance` method gets called whenever OAuth2 is needed.
-The first method, `is_oauth2_enabled`, is used to inform if the database supports OAuth2. This can be dynamic; for example, the Google Sheets DB engine spec checks if the Superset configuration has the necessary section:
-
-```python
-from flask import current_app
-
+The DB engine should implement logic in either `get_url_for_impersonation` or `update_impersonation_config` to update the connection with the personal access token. See the Google Sheets DB engine spec for a reference implementation.
Review Comment:
This may be splitting hairs, but I perceive user impersonation to be a very specific execution form, which means "query is executed using a service account, but is executed as if a particular user us executing it". In the case of OAuth2, the query is in fact executed by the user, not the service account, and makes it more secure.
I also agree that consolidating these methods could make sense, as having separate methods for mutating the URL and the other for params to `create_engine` often causes confusion for me, and are IMO really the same thing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] chore(OAuth2): refactor for custom OAuth2 clients [superset]
Posted by "betodealmeida (via GitHub)" <gi...@apache.org>.
betodealmeida merged PR #27880:
URL: https://github.com/apache/superset/pull/27880
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] chore(OAuth2): refactor for custom OAuth2 clients [superset]
Posted by "betodealmeida (via GitHub)" <gi...@apache.org>.
betodealmeida commented on code in PR #27880:
URL: https://github.com/apache/superset/pull/27880#discussion_r1550124554
##########
superset/db_engine_specs/README.md:
##########
@@ -547,65 +547,53 @@ Alternatively, it's also possible to impersonate users by implementing the `upda
Support for authenticating to a database using personal OAuth2 access tokens was introduced in [SIP-85](https://github.com/apache/superset/issues/20300). The Google Sheets DB engine spec is the reference implementation.
-To add support for OAuth2 to a DB engine spec, the following attribute and methods are needed:
+To add support for OAuth2 to a DB engine spec, the following attributes are needed:
Review Comment:
Ah, good point, I'll add a note — though I am hoping we can stabilize the API in the next couple months.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] chore(OAuth2): refactor for custom OAuth2 clients [superset]
Posted by "villebro (via GitHub)" <gi...@apache.org>.
villebro commented on code in PR #27880:
URL: https://github.com/apache/superset/pull/27880#discussion_r1550100365
##########
superset/db_engine_specs/README.md:
##########
@@ -547,65 +547,53 @@ Alternatively, it's also possible to impersonate users by implementing the `upda
Support for authenticating to a database using personal OAuth2 access tokens was introduced in [SIP-85](https://github.com/apache/superset/issues/20300). The Google Sheets DB engine spec is the reference implementation.
-To add support for OAuth2 to a DB engine spec, the following attribute and methods are needed:
+To add support for OAuth2 to a DB engine spec, the following attributes are needed:
Review Comment:
As the API is still evolving, I think it could be a god idea to clearly state that this functionality is still experimental, and subject to breaking changes.
##########
superset/db_engine_specs/README.md:
##########
@@ -547,65 +547,53 @@ Alternatively, it's also possible to impersonate users by implementing the `upda
Support for authenticating to a database using personal OAuth2 access tokens was introduced in [SIP-85](https://github.com/apache/superset/issues/20300). The Google Sheets DB engine spec is the reference implementation.
-To add support for OAuth2 to a DB engine spec, the following attribute and methods are needed:
+To add support for OAuth2 to a DB engine spec, the following attributes are needed:
```python
class BaseEngineSpec:
+ supports_oauth2 = True
oauth2_exception = OAuth2RedirectError
- @staticmethod
- def is_oauth2_enabled() -> bool:
- return False
-
- @staticmethod
- def get_oauth2_authorization_uri(state: OAuth2State) -> str:
- raise NotImplementedError()
-
- @staticmethod
- def get_oauth2_token(code: str, state: OAuth2State) -> OAuth2TokenResponse:
- raise NotImplementedError()
-
- @staticmethod
- def get_oauth2_fresh_token(refresh_token: str) -> OAuth2TokenResponse:
- raise NotImplementedError()
+ oauth2_scope = " ".join([
+ "https://example.org/scope1",
+ "https://example.org/scope2",
+ ])
+ oauth2_authorization_request_uri = "https://example.org/authorize"
+ oauth2_token_request_uri = "https://example.org/token"
```
The `oauth2_exception` is an exception that is raised by `cursor.execute` when OAuth2 is needed. This will start the OAuth2 dance when `BaseEngineSpec.execute` is called, by returning the custom error `OAUTH2_REDIRECT` to the frontend. If the database driver doesn't have a specific exception, it might be necessary to overload the `execute` method in the DB engine spec, so that the `BaseEngineSpec.start_oauth2_dance` method gets called whenever OAuth2 is needed.
-The first method, `is_oauth2_enabled`, is used to inform if the database supports OAuth2. This can be dynamic; for example, the Google Sheets DB engine spec checks if the Superset configuration has the necessary section:
-
-```python
-from flask import current_app
-
+The DB engine should implement logic in either `get_url_for_impersonation` or `update_impersonation_config` to update the connection with the personal access token. See the Google Sheets DB engine spec for a reference implementation.
Review Comment:
For some forthcoming major version we might want to consider renaming these methods to make it clear they don't only encapsulate impersonation logic. Maybe call it something that clarifies that it updates the URL to include user information (=JWT in the case of OAuth2 or user impersonation in the more traditional approach).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] chore(OAuth2): refactor for custom OAuth2 clients [superset]
Posted by "betodealmeida (via GitHub)" <gi...@apache.org>.
betodealmeida commented on code in PR #27880:
URL: https://github.com/apache/superset/pull/27880#discussion_r1553962130
##########
superset/common/query_object.py:
##########
@@ -108,7 +108,7 @@ class QueryObject: # pylint: disable=too-many-instance-attributes
time_range: str | None
to_dttm: datetime | None
- def __init__( # pylint: disable=too-many-arguments,too-many-locals
+ def __init__( # pylint: disable=too-many-locals, too-many-arguments
Review Comment:
No, I think what happened is that `pylint` got bumped. When that happens, the new rules are not enforced in the same PR, since it only runs in files that have been modified. So only in the next PR that the rules will be checked. I think @mistercrunch is working on fixing this.
Here I fixed the lint manually, then someone else fixed in master, but using a different order.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] chore(OAuth2): refactor for custom OAuth2 clients [superset]
Posted by "betodealmeida (via GitHub)" <gi...@apache.org>.
betodealmeida commented on code in PR #27880:
URL: https://github.com/apache/superset/pull/27880#discussion_r1550127325
##########
superset/db_engine_specs/README.md:
##########
@@ -547,65 +547,53 @@ Alternatively, it's also possible to impersonate users by implementing the `upda
Support for authenticating to a database using personal OAuth2 access tokens was introduced in [SIP-85](https://github.com/apache/superset/issues/20300). The Google Sheets DB engine spec is the reference implementation.
-To add support for OAuth2 to a DB engine spec, the following attribute and methods are needed:
+To add support for OAuth2 to a DB engine spec, the following attributes are needed:
```python
class BaseEngineSpec:
+ supports_oauth2 = True
oauth2_exception = OAuth2RedirectError
- @staticmethod
- def is_oauth2_enabled() -> bool:
- return False
-
- @staticmethod
- def get_oauth2_authorization_uri(state: OAuth2State) -> str:
- raise NotImplementedError()
-
- @staticmethod
- def get_oauth2_token(code: str, state: OAuth2State) -> OAuth2TokenResponse:
- raise NotImplementedError()
-
- @staticmethod
- def get_oauth2_fresh_token(refresh_token: str) -> OAuth2TokenResponse:
- raise NotImplementedError()
+ oauth2_scope = " ".join([
+ "https://example.org/scope1",
+ "https://example.org/scope2",
+ ])
+ oauth2_authorization_request_uri = "https://example.org/authorize"
+ oauth2_token_request_uri = "https://example.org/token"
```
The `oauth2_exception` is an exception that is raised by `cursor.execute` when OAuth2 is needed. This will start the OAuth2 dance when `BaseEngineSpec.execute` is called, by returning the custom error `OAUTH2_REDIRECT` to the frontend. If the database driver doesn't have a specific exception, it might be necessary to overload the `execute` method in the DB engine spec, so that the `BaseEngineSpec.start_oauth2_dance` method gets called whenever OAuth2 is needed.
-The first method, `is_oauth2_enabled`, is used to inform if the database supports OAuth2. This can be dynamic; for example, the Google Sheets DB engine spec checks if the Superset configuration has the necessary section:
-
-```python
-from flask import current_app
-
+The DB engine should implement logic in either `get_url_for_impersonation` or `update_impersonation_config` to update the connection with the personal access token. See the Google Sheets DB engine spec for a reference implementation.
Review Comment:
But OAuth2 is still a form of user impersonation, we're running a query on a connection that is built on behalf of the user. Putting this under user impersonation means that the cache is not shared between users. This is what we want most of the time, but not always, which is why I think we should decouple the "user-level cache" from "user-impersonation" and communicate better what's happening in these flows.
Having said that, I agree that the naming of these methods is confusing, and personally I think they could be combined in a single one. Currently the former modifies the URL and the latter modifies the params passed to `create_engine`, but they could be consolidated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] chore(OAuth2): refactor for custom OAuth2 clients [superset]
Posted by "villebro (via GitHub)" <gi...@apache.org>.
villebro commented on code in PR #27880:
URL: https://github.com/apache/superset/pull/27880#discussion_r1553760713
##########
superset/common/query_object.py:
##########
@@ -108,7 +108,7 @@ class QueryObject: # pylint: disable=too-many-instance-attributes
time_range: str | None
to_dttm: datetime | None
- def __init__( # pylint: disable=too-many-arguments,too-many-locals
+ def __init__( # pylint: disable=too-many-locals, too-many-arguments
Review Comment:
Out of curiosity, was this change automatically applied by one of our linters?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org