You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "betodealmeida (via GitHub)" <gi...@apache.org> on 2023/05/15 22:32:11 UTC
[GitHub] [superset] betodealmeida opened a new pull request, #24067: feat: split database information
betodealmeida opened a new pull request, #24067:
URL: https://github.com/apache/superset/pull/24067
<!---
Please write the PR title following the conventions at https://www.conventionalcommits.org/en/v1.0.0/
Example:
fix(dashboard): load charts correctly
-->
### SUMMARY
<!--- Describe the change below, including rationale and design decisions -->
Move potentially sensitive database information (SQLAlchemy URI, eg) to a separate endpoint that can be only accessed by the admin (`can_write` on `Database`).
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
<!--- Skip this if not applicable -->
N/A
### TESTING INSTRUCTIONS
<!--- Required! What steps can be taken to manually verify the changes? -->
Updating tests.
### ADDITIONAL INFORMATION
<!--- Check any relevant boxes with "x" -->
<!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->
- [ ] Has associated issue:
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] betodealmeida commented on a diff in pull request #24067: feat: split database information
Posted by "betodealmeida (via GitHub)" <gi...@apache.org>.
betodealmeida commented on code in PR #24067:
URL: https://github.com/apache/superset/pull/24067#discussion_r1199220822
##########
superset/databases/api.py:
##########
@@ -144,12 +146,8 @@ class DatabaseRestApi(BaseSupersetModelRestApi):
"driver",
"force_ctas_schema",
"impersonate_user",
- "masked_encrypted_extra",
"extra",
- "parameters",
- "parameters_schema",
"server_cert",
Review Comment:
Good point!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] dpgaspar commented on a diff in pull request #24067: feat: split database information
Posted by "dpgaspar (via GitHub)" <gi...@apache.org>.
dpgaspar commented on code in PR #24067:
URL: https://github.com/apache/superset/pull/24067#discussion_r1198672844
##########
superset/databases/api.py:
##########
@@ -237,6 +236,50 @@ class DatabaseRestApi(BaseSupersetModelRestApi):
ValidateSQLResponse,
)
+ @expose("/<int:pk>/connection", methods=("GET",))
+ @protect()
+ @safe
+ def get_connection(self, pk: int) -> Response:
+ """Get database connection info.
+ ---
+ get:
+ description: >-
Review Comment:
nit: I'm currently switching the description for summary, looks better on swagger
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] betodealmeida merged pull request #24067: feat: split database information
Posted by "betodealmeida (via GitHub)" <gi...@apache.org>.
betodealmeida merged PR #24067:
URL: https://github.com/apache/superset/pull/24067
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] codecov[bot] commented on pull request #24067: feat: split database information
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #24067:
URL: https://github.com/apache/superset/pull/24067#issuecomment-1553695719
## [Codecov](https://app.codecov.io/gh/apache/superset/pull/24067?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#24067](https://app.codecov.io/gh/apache/superset/pull/24067?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (2cdd6f1) into [master](https://app.codecov.io/gh/apache/superset/commit/d96b72d46fe6e7b8a0a4749032c13322396633e1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (d96b72d) will **decrease** coverage by `11.22%`.
> The diff coverage is `74.63%`.
> :exclamation: Current head 2cdd6f1 differs from pull request most recent head f5cd423. Consider uploading reports for the commit f5cd423 to get more accurate results
```diff
@@ Coverage Diff @@
## master #24067 +/- ##
===========================================
- Coverage 68.18% 56.96% -11.22%
===========================================
Files 1941 1952 +11
Lines 75287 75470 +183
Branches 8177 8191 +14
===========================================
- Hits 51332 42992 -8340
- Misses 21860 30374 +8514
- Partials 2095 2104 +9
```
| Flag | Coverage Δ | |
|---|---|---|
| hive | `?` | |
| postgres | `?` | |
| presto | `53.12% <48.00%> (+0.02%)` | :arrow_up: |
| python | `59.37% <66.66%> (-23.34%)` | :arrow_down: |
| sqlite | `?` | |
| unit | `53.14% <62.66%> (+0.15%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://app.codecov.io/gh/apache/superset/pull/24067?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...chart-controls/src/shared-controls/dndControls.tsx](https://app.codecov.io/gh/apache/superset/pull/24067?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQtZnJvbnRlbmQvcGFja2FnZXMvc3VwZXJzZXQtdWktY2hhcnQtY29udHJvbHMvc3JjL3NoYXJlZC1jb250cm9scy9kbmRDb250cm9scy50c3g=) | `58.33% <ø> (ø)` | |
| [...set-ui-core/src/ui-overrides/ExtensionsRegistry.ts](https://app.codecov.io/gh/apache/superset/pull/24067?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQtZnJvbnRlbmQvcGFja2FnZXMvc3VwZXJzZXQtdWktY29yZS9zcmMvdWktb3ZlcnJpZGVzL0V4dGVuc2lvbnNSZWdpc3RyeS50cw==) | `100.00% <ø> (ø)` | |
| [...ackages/superset-ui-core/src/utils/featureFlags.ts](https://app.codecov.io/gh/apache/superset/pull/24067?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQtZnJvbnRlbmQvcGFja2FnZXMvc3VwZXJzZXQtdWktY29yZS9zcmMvdXRpbHMvZmVhdHVyZUZsYWdzLnRz) | `100.00% <ø> (ø)` | |
| [...hart-echarts/src/MixedTimeseries/transformProps.ts](https://app.codecov.io/gh/apache/superset/pull/24067?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQtZnJvbnRlbmQvcGx1Z2lucy9wbHVnaW4tY2hhcnQtZWNoYXJ0cy9zcmMvTWl4ZWRUaW1lc2VyaWVzL3RyYW5zZm9ybVByb3BzLnRz) | `0.00% <0.00%> (ø)` | |
| [...lugin-chart-echarts/src/Timeseries/transformers.ts](https://app.codecov.io/gh/apache/superset/pull/24067?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQtZnJvbnRlbmQvcGx1Z2lucy9wbHVnaW4tY2hhcnQtZWNoYXJ0cy9zcmMvVGltZXNlcmllcy90cmFuc2Zvcm1lcnMudHM=) | `56.95% <ø> (-0.29%)` | :arrow_down: |
| [...n-chart-handlebars/src/plugin/controls/columns.tsx](https://app.codecov.io/gh/apache/superset/pull/24067?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQtZnJvbnRlbmQvcGx1Z2lucy9wbHVnaW4tY2hhcnQtaGFuZGxlYmFycy9zcmMvcGx1Z2luL2NvbnRyb2xzL2NvbHVtbnMudHN4) | `15.78% <0.00%> (+0.78%)` | :arrow_up: |
| [superset-frontend/src/SqlLab/actions/sqlLab.js](https://app.codecov.io/gh/apache/superset/pull/24067?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL1NxbExhYi9hY3Rpb25zL3NxbExhYi5qcw==) | `66.41% <ø> (-2.35%)` | :arrow_down: |
| [...frontend/src/SqlLab/components/ResultSet/index.tsx](https://app.codecov.io/gh/apache/superset/pull/24067?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL1NxbExhYi9jb21wb25lbnRzL1Jlc3VsdFNldC9pbmRleC50c3g=) | `63.05% <ø> (+0.39%)` | :arrow_up: |
| [superset-frontend/src/components/Modal/Modal.tsx](https://app.codecov.io/gh/apache/superset/pull/24067?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL2NvbXBvbmVudHMvTW9kYWwvTW9kYWwudHN4) | `85.89% <ø> (ø)` | |
| [...et-frontend/src/components/TableSelector/index.tsx](https://app.codecov.io/gh/apache/superset/pull/24067?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQtZnJvbnRlbmQvc3JjL2NvbXBvbmVudHMvVGFibGVTZWxlY3Rvci9pbmRleC50c3g=) | `81.08% <0.00%> (ø)` | |
| ... and [92 more](https://app.codecov.io/gh/apache/superset/pull/24067?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | |
... and [283 files with indirect coverage changes](https://app.codecov.io/gh/apache/superset/pull/24067/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on a diff in pull request #24067: feat: split database information
Posted by "eschutho (via GitHub)" <gi...@apache.org>.
eschutho commented on code in PR #24067:
URL: https://github.com/apache/superset/pull/24067#discussion_r1198416991
##########
tests/unit_tests/databases/api_test.py:
##########
@@ -103,14 +104,163 @@ def test_password_mask(
mocker.patch("sqlalchemy.engine.URL.get_driver_name", return_value="gsheets")
mocker.patch("superset.utils.log.DBEventLogger.log")
- response = client.get("/api/v1/database/1")
+ response = client.get("/api/v1/database/1/connection")
+
+ # check that private key is masked
assert (
response.json["result"]["parameters"]["service_account_info"]["private_key"]
== "XXXXXXXXXX"
)
assert "encrypted_extra" not in response.json["result"]
+def test_database_connection(
+ mocker: MockFixture,
+ app: Any,
+ session: Session,
+ client: Any,
+ full_api_access: None,
+) -> None:
+ """
+ Test that connection info is only returned in ``api/v1/database/${id}/connection``.
+ """
+ from superset.databases.api import DatabaseRestApi
+ from superset.models.core import Database
+
+ DatabaseRestApi.datamodel.session = session
+
+ # create table for databases
+ Database.metadata.create_all(session.get_bind()) # pylint: disable=no-member
+
+ database = Database(
+ uuid=UUID("02feae18-2dd6-4bb4-a9c0-49e9d4f29d58"),
+ database_name="my_database",
+ sqlalchemy_uri="gsheets://",
+ encrypted_extra=json.dumps(
+ {
+ "service_account_info": {
+ "type": "service_account",
+ "project_id": "black-sanctum-314419",
+ "private_key_id": "259b0d419a8f840056158763ff54d8b08f7b8173",
+ "private_key": "SECRET",
+ "client_email": "google-spreadsheets-demo-servi@black-sanctum-314419.iam.gserviceaccount.com",
+ "client_id": "114567578578109757129",
+ "auth_uri": "https://accounts.google.com/o/oauth2/auth",
+ "token_uri": "https://oauth2.googleapis.com/token",
+ "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
+ "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/google-spreadsheets-demo-servi%40black-sanctum-314419.iam.gserviceaccount.com",
+ },
Review Comment:
is this a real google account?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] dpgaspar commented on a diff in pull request #24067: feat: split database information
Posted by "dpgaspar (via GitHub)" <gi...@apache.org>.
dpgaspar commented on code in PR #24067:
URL: https://github.com/apache/superset/pull/24067#discussion_r1198655946
##########
superset/databases/api.py:
##########
@@ -144,12 +146,8 @@ class DatabaseRestApi(BaseSupersetModelRestApi):
"driver",
"force_ctas_schema",
"impersonate_user",
- "masked_encrypted_extra",
"extra",
- "parameters",
- "parameters_schema",
"server_cert",
Review Comment:
This should be a CA bundle so it's public info, but we are encrypting it's content.
But I agree a non admin user does not need this info
##########
tests/unit_tests/databases/api_test.py:
##########
@@ -103,14 +104,163 @@ def test_password_mask(
mocker.patch("sqlalchemy.engine.URL.get_driver_name", return_value="gsheets")
mocker.patch("superset.utils.log.DBEventLogger.log")
- response = client.get("/api/v1/database/1")
+ response = client.get("/api/v1/database/1/connection")
+
+ # check that private key is masked
assert (
response.json["result"]["parameters"]["service_account_info"]["private_key"]
== "XXXXXXXXXX"
)
assert "encrypted_extra" not in response.json["result"]
+def test_database_connection(
+ mocker: MockFixture,
+ app: Any,
+ session: Session,
+ client: Any,
+ full_api_access: None,
+) -> None:
+ """
+ Test that connection info is only returned in ``api/v1/database/${id}/connection``.
+ """
+ from superset.databases.api import DatabaseRestApi
+ from superset.models.core import Database
+
+ DatabaseRestApi.datamodel.session = session
+
+ # create table for databases
+ Database.metadata.create_all(session.get_bind()) # pylint: disable=no-member
+
+ database = Database(
+ uuid=UUID("02feae18-2dd6-4bb4-a9c0-49e9d4f29d58"),
+ database_name="my_database",
+ sqlalchemy_uri="gsheets://",
+ encrypted_extra=json.dumps(
+ {
+ "service_account_info": {
+ "type": "service_account",
+ "project_id": "black-sanctum-314419",
+ "private_key_id": "259b0d419a8f840056158763ff54d8b08f7b8173",
+ "private_key": "SECRET",
+ "client_email": "google-spreadsheets-demo-servi@black-sanctum-314419.iam.gserviceaccount.com",
+ "client_id": "114567578578109757129",
+ "auth_uri": "https://accounts.google.com/o/oauth2/auth",
+ "token_uri": "https://oauth2.googleapis.com/token",
+ "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
+ "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/google-spreadsheets-demo-servi%40black-sanctum-314419.iam.gserviceaccount.com",
+ },
+ }
+ ),
+ )
+ session.add(database)
+ session.commit()
+
+ # mock the lookup so that we don't need to include the driver
+ mocker.patch("sqlalchemy.engine.URL.get_driver_name", return_value="gsheets")
+ mocker.patch("superset.utils.log.DBEventLogger.log")
+
+ response = client.get("/api/v1/database/1/connection")
+ assert response.json == {
+ "id": 1,
+ "result": {
+ "allow_ctas": False,
+ "allow_cvas": False,
+ "allow_dml": False,
+ "allow_file_upload": False,
+ "allow_run_async": False,
+ "backend": "gsheets",
+ "cache_timeout": None,
+ "configuration_method": "sqlalchemy_form",
+ "database_name": "my_database",
+ "driver": "gsheets",
+ "engine_information": {
+ "disable_ssh_tunneling": True,
+ "supports_file_upload": False,
+ },
+ "expose_in_sqllab": True,
+ "extra": '{\n "metadata_params": {},\n "engine_params": {},\n "metadata_cache_timeout": {},\n "schemas_allowed_for_file_upload": []\n}\n',
+ "force_ctas_schema": None,
+ "id": 1,
+ "impersonate_user": False,
+ "is_managed_externally": False,
+ "masked_encrypted_extra": json.dumps(
+ {
+ "service_account_info": {
+ "type": "service_account",
+ "project_id": "black-sanctum-314419",
+ "private_key_id": "259b0d419a8f840056158763ff54d8b08f7b8173",
+ "private_key": "XXXXXXXXXX",
+ "client_email": "google-spreadsheets-demo-servi@black-sanctum-314419.iam.gserviceaccount.com",
+ "client_id": "114567578578109757129",
+ "auth_uri": "https://accounts.google.com/o/oauth2/auth",
+ "token_uri": "https://oauth2.googleapis.com/token",
+ "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
+ "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/google-spreadsheets-demo-servi%40black-sanctum-314419.iam.gserviceaccount.com",
+ }
+ }
+ ),
+ "parameters": {
+ "service_account_info": {
+ "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
+ "auth_uri": "https://accounts.google.com/o/oauth2/auth",
+ "client_email": "google-spreadsheets-demo-servi@black-sanctum-314419.iam.gserviceaccount.com",
+ "client_id": "114567578578109757129",
+ "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/google-spreadsheets-demo-servi%40black-sanctum-314419.iam.gserviceaccount.com",
+ "private_key": "XXXXXXXXXX",
+ "private_key_id": "259b0d419a8f840056158763ff54d8b08f7b8173",
+ "project_id": "black-sanctum-314419",
+ "token_uri": "https://oauth2.googleapis.com/token",
+ "type": "service_account",
+ }
+ },
+ "parameters_schema": {
+ "properties": {
+ "catalog": {"type": "object"},
+ "service_account_info": {
+ "description": "Contents of GSheets JSON credentials.",
+ "type": "string",
+ "x-encrypted-extra": True,
+ },
+ },
+ "type": "object",
+ },
+ "server_cert": None,
+ "sqlalchemy_uri": "gsheets://",
+ "uuid": "02feae18-2dd6-4bb4-a9c0-49e9d4f29d58",
+ },
+ }
+
+ response = client.get("/api/v1/database/1")
+ assert response.json == {
+ "id": 1,
+ "result": {
+ "allow_ctas": False,
+ "allow_cvas": False,
+ "allow_dml": False,
+ "allow_file_upload": False,
+ "allow_run_async": False,
+ "backend": "gsheets",
+ "cache_timeout": None,
+ "configuration_method": "sqlalchemy_form",
+ "database_name": "my_database",
+ "driver": "gsheets",
+ "engine_information": {
+ "disable_ssh_tunneling": True,
+ "supports_file_upload": False,
+ },
+ "expose_in_sqllab": True,
+ "extra": '{\n "metadata_params": {},\n "engine_params": {},\n "metadata_cache_timeout": {},\n "schemas_allowed_for_file_upload": []\n}\n',
Review Comment:
extra also feels unecessary
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] betodealmeida commented on a diff in pull request #24067: feat: split database information
Posted by "betodealmeida (via GitHub)" <gi...@apache.org>.
betodealmeida commented on code in PR #24067:
URL: https://github.com/apache/superset/pull/24067#discussion_r1199225746
##########
tests/unit_tests/databases/api_test.py:
##########
@@ -103,14 +104,163 @@ def test_password_mask(
mocker.patch("sqlalchemy.engine.URL.get_driver_name", return_value="gsheets")
mocker.patch("superset.utils.log.DBEventLogger.log")
- response = client.get("/api/v1/database/1")
+ response = client.get("/api/v1/database/1/connection")
+
+ # check that private key is masked
assert (
response.json["result"]["parameters"]["service_account_info"]["private_key"]
== "XXXXXXXXXX"
)
assert "encrypted_extra" not in response.json["result"]
+def test_database_connection(
+ mocker: MockFixture,
+ app: Any,
+ session: Session,
+ client: Any,
+ full_api_access: None,
+) -> None:
+ """
+ Test that connection info is only returned in ``api/v1/database/${id}/connection``.
+ """
+ from superset.databases.api import DatabaseRestApi
+ from superset.models.core import Database
+
+ DatabaseRestApi.datamodel.session = session
+
+ # create table for databases
+ Database.metadata.create_all(session.get_bind()) # pylint: disable=no-member
+
+ database = Database(
+ uuid=UUID("02feae18-2dd6-4bb4-a9c0-49e9d4f29d58"),
+ database_name="my_database",
+ sqlalchemy_uri="gsheets://",
+ encrypted_extra=json.dumps(
+ {
+ "service_account_info": {
+ "type": "service_account",
+ "project_id": "black-sanctum-314419",
+ "private_key_id": "259b0d419a8f840056158763ff54d8b08f7b8173",
+ "private_key": "SECRET",
+ "client_email": "google-spreadsheets-demo-servi@black-sanctum-314419.iam.gserviceaccount.com",
+ "client_id": "114567578578109757129",
+ "auth_uri": "https://accounts.google.com/o/oauth2/auth",
+ "token_uri": "https://oauth2.googleapis.com/token",
+ "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
+ "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/google-spreadsheets-demo-servi%40black-sanctum-314419.iam.gserviceaccount.com",
+ },
Review Comment:
It is, with the private key omitted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] jfrag1 commented on a diff in pull request #24067: feat: split database information
Posted by "jfrag1 (via GitHub)" <gi...@apache.org>.
jfrag1 commented on code in PR #24067:
URL: https://github.com/apache/superset/pull/24067#discussion_r1198350774
##########
superset/databases/api.py:
##########
@@ -144,12 +146,8 @@ class DatabaseRestApi(BaseSupersetModelRestApi):
"driver",
"force_ctas_schema",
"impersonate_user",
- "masked_encrypted_extra",
"extra",
- "parameters",
- "parameters_schema",
"server_cert",
Review Comment:
could `server_cert` also be sensitive?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] betodealmeida commented on pull request #24067: feat: split database information
Posted by "betodealmeida (via GitHub)" <gi...@apache.org>.
betodealmeida commented on PR #24067:
URL: https://github.com/apache/superset/pull/24067#issuecomment-1555109210
> Thanks for doing this!
>
> Do you think non Admin users still need access to `/api/v1/database/<id>` ?
Yeah, SQL Lab broke when I removed the access.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on a diff in pull request #24067: feat: split database information
Posted by "eschutho (via GitHub)" <gi...@apache.org>.
eschutho commented on code in PR #24067:
URL: https://github.com/apache/superset/pull/24067#discussion_r1198416209
##########
superset-frontend/src/views/CRUD/hooks.ts:
##########
@@ -242,8 +243,12 @@ export function useSingleViewResource<D extends object = any>(
loading: true,
});
+ const endpoint =
+ path_suffix !== ''
+ ? `/api/v1/${resourceName}/${resourceID}/${path_suffix}`
+ : `/api/v1/${resourceName}/${resourceID}`;
Review Comment:
nit, but maybe something like this may be a bit dryer:
```
const endpoint = `/api/v1/${resourceName}/${resourceID}${path_suffix ? `/${path_suffix}` : '' }`
or
const endpoint = `/api/v1/${resourceName}/${resourceID}${path_suffix && `/${path_suffix}` || ''}`
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] betodealmeida commented on a diff in pull request #24067: feat: split database information
Posted by "betodealmeida (via GitHub)" <gi...@apache.org>.
betodealmeida commented on code in PR #24067:
URL: https://github.com/apache/superset/pull/24067#discussion_r1199225891
##########
tests/unit_tests/databases/api_test.py:
##########
@@ -103,14 +104,163 @@ def test_password_mask(
mocker.patch("sqlalchemy.engine.URL.get_driver_name", return_value="gsheets")
mocker.patch("superset.utils.log.DBEventLogger.log")
- response = client.get("/api/v1/database/1")
+ response = client.get("/api/v1/database/1/connection")
+
+ # check that private key is masked
assert (
response.json["result"]["parameters"]["service_account_info"]["private_key"]
== "XXXXXXXXXX"
)
assert "encrypted_extra" not in response.json["result"]
+def test_database_connection(
+ mocker: MockFixture,
+ app: Any,
+ session: Session,
+ client: Any,
+ full_api_access: None,
+) -> None:
+ """
+ Test that connection info is only returned in ``api/v1/database/${id}/connection``.
+ """
+ from superset.databases.api import DatabaseRestApi
+ from superset.models.core import Database
+
+ DatabaseRestApi.datamodel.session = session
+
+ # create table for databases
+ Database.metadata.create_all(session.get_bind()) # pylint: disable=no-member
+
+ database = Database(
+ uuid=UUID("02feae18-2dd6-4bb4-a9c0-49e9d4f29d58"),
+ database_name="my_database",
+ sqlalchemy_uri="gsheets://",
+ encrypted_extra=json.dumps(
+ {
+ "service_account_info": {
+ "type": "service_account",
+ "project_id": "black-sanctum-314419",
+ "private_key_id": "259b0d419a8f840056158763ff54d8b08f7b8173",
+ "private_key": "SECRET",
+ "client_email": "google-spreadsheets-demo-servi@black-sanctum-314419.iam.gserviceaccount.com",
+ "client_id": "114567578578109757129",
+ "auth_uri": "https://accounts.google.com/o/oauth2/auth",
+ "token_uri": "https://oauth2.googleapis.com/token",
+ "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
+ "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/google-spreadsheets-demo-servi%40black-sanctum-314419.iam.gserviceaccount.com",
+ },
+ }
+ ),
+ )
+ session.add(database)
+ session.commit()
+
+ # mock the lookup so that we don't need to include the driver
+ mocker.patch("sqlalchemy.engine.URL.get_driver_name", return_value="gsheets")
+ mocker.patch("superset.utils.log.DBEventLogger.log")
+
+ response = client.get("/api/v1/database/1/connection")
+ assert response.json == {
+ "id": 1,
+ "result": {
+ "allow_ctas": False,
+ "allow_cvas": False,
+ "allow_dml": False,
+ "allow_file_upload": False,
+ "allow_run_async": False,
+ "backend": "gsheets",
+ "cache_timeout": None,
+ "configuration_method": "sqlalchemy_form",
+ "database_name": "my_database",
+ "driver": "gsheets",
+ "engine_information": {
+ "disable_ssh_tunneling": True,
+ "supports_file_upload": False,
+ },
+ "expose_in_sqllab": True,
+ "extra": '{\n "metadata_params": {},\n "engine_params": {},\n "metadata_cache_timeout": {},\n "schemas_allowed_for_file_upload": []\n}\n',
+ "force_ctas_schema": None,
+ "id": 1,
+ "impersonate_user": False,
+ "is_managed_externally": False,
+ "masked_encrypted_extra": json.dumps(
+ {
+ "service_account_info": {
+ "type": "service_account",
+ "project_id": "black-sanctum-314419",
+ "private_key_id": "259b0d419a8f840056158763ff54d8b08f7b8173",
+ "private_key": "XXXXXXXXXX",
+ "client_email": "google-spreadsheets-demo-servi@black-sanctum-314419.iam.gserviceaccount.com",
+ "client_id": "114567578578109757129",
+ "auth_uri": "https://accounts.google.com/o/oauth2/auth",
+ "token_uri": "https://oauth2.googleapis.com/token",
+ "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
+ "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/google-spreadsheets-demo-servi%40black-sanctum-314419.iam.gserviceaccount.com",
+ }
+ }
+ ),
+ "parameters": {
+ "service_account_info": {
+ "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
+ "auth_uri": "https://accounts.google.com/o/oauth2/auth",
+ "client_email": "google-spreadsheets-demo-servi@black-sanctum-314419.iam.gserviceaccount.com",
+ "client_id": "114567578578109757129",
+ "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/google-spreadsheets-demo-servi%40black-sanctum-314419.iam.gserviceaccount.com",
+ "private_key": "XXXXXXXXXX",
+ "private_key_id": "259b0d419a8f840056158763ff54d8b08f7b8173",
+ "project_id": "black-sanctum-314419",
+ "token_uri": "https://oauth2.googleapis.com/token",
+ "type": "service_account",
+ }
+ },
+ "parameters_schema": {
+ "properties": {
+ "catalog": {"type": "object"},
+ "service_account_info": {
+ "description": "Contents of GSheets JSON credentials.",
+ "type": "string",
+ "x-encrypted-extra": True,
+ },
+ },
+ "type": "object",
+ },
+ "server_cert": None,
+ "sqlalchemy_uri": "gsheets://",
+ "uuid": "02feae18-2dd6-4bb4-a9c0-49e9d4f29d58",
+ },
+ }
+
+ response = client.get("/api/v1/database/1")
+ assert response.json == {
+ "id": 1,
+ "result": {
+ "allow_ctas": False,
+ "allow_cvas": False,
+ "allow_dml": False,
+ "allow_file_upload": False,
+ "allow_run_async": False,
+ "backend": "gsheets",
+ "cache_timeout": None,
+ "configuration_method": "sqlalchemy_form",
+ "database_name": "my_database",
+ "driver": "gsheets",
+ "engine_information": {
+ "disable_ssh_tunneling": True,
+ "supports_file_upload": False,
+ },
+ "expose_in_sqllab": True,
+ "extra": '{\n "metadata_params": {},\n "engine_params": {},\n "metadata_cache_timeout": {},\n "schemas_allowed_for_file_upload": []\n}\n',
Review Comment:
Will remove!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat: split database information [superset]
Posted by "eschutho (via GitHub)" <gi...@apache.org>.
eschutho commented on PR #24067:
URL: https://github.com/apache/superset/pull/24067#issuecomment-1743874014
@betodealmeida I added a label for breaking-change on this PR since it removes some properties from the API response. LMK if you think otherwise, though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org