You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "withnale (via GitHub)" <gi...@apache.org> on 2024/12/13 15:11:03 UTC
[I] Unable to import datasources with `encrypted_extra` params via REST [superset]
withnale opened a new issue, #31438:
URL: https://github.com/apache/superset/issues/31438
### Bug description
At present it is possible to create a bigquery datasource which uses an explicit key by creating an `import_datasources.yaml` file that contains something similar to the block below:
```
databases:
-
database_name: mydatabase
sqlalchemy_uri: bigquery://gcp-project-id/
cache_timeout: 0
expose_in_sqllab: true
allow_run_async: false
allow_ctas: false
allow_cvas: false
allow_dml: false
allow_file_upload: false
encrypted_extra: |
{"credentials_info": { "type": "service_account", "project_id": "gcp-project-id", "private_key_id": "aca359ef9828567aa1f031f367ba547fe26bf54c", "private_key": "-----BEGIN PRIVATE KEY-----VERY_BIG_PRIVATE_KEY_STRING\n-----END PRIVATE KEY-----\n", "client_email": "service-account@gcp-project-id.iam.gserviceaccount.com", "client_id": "1149993100000000000", "auth_uri": "https://accounts.google.com/o/oauth2/auth", "token_uri": "https://oauth2.googleapis.com/token", "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/service-account%40gcp-project-id.iam.gserviceaccount.com", "universe_domain": "googleapis.com"} }
extra: |
{ "allows_virtual_table_explore": true }
uuid: 00000000-0000-0000-0000-123456789001
```
This can be imported using `superset import_datasources -p filename.yaml` from a running instance.
However, if I wish to import the same data structure using the REST API `/api/v1/database/import` it will fail.
It seems that the REST version applies additional schema checks on the bundled zipfile and fails because `encrypted_extra` is not part of `ImportV1DatabaseSchema`
### Screenshots/recordings
_No response_
### Superset version
master / latest-dev
### Python version
3.11
### Node version
16
### Browser
Chrome
### Additional context
_No response_
### Checklist
- [X] I have searched Superset docs and Slack and didn't find a solution to my problem.
- [X] I have searched the GitHub issue tracker and didn't find a similar bug report.
- [X] I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] Unable to import datasources with `encrypted_extra` params via REST [superset]
Posted by "betodealmeida (via GitHub)" <gi...@apache.org>.
betodealmeida commented on issue #31438:
URL: https://github.com/apache/superset/issues/31438#issuecomment-2673240144
> Hello. Did this ever get addressed?
https://github.com/apache/superset/pull/32339
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] Unable to import datasources with `encrypted_extra` params via REST [superset]
Posted by "withnale (via GitHub)" <gi...@apache.org>.
withnale commented on issue #31438:
URL: https://github.com/apache/superset/issues/31438#issuecomment-2541678785
To comment on the bot notes above. I've seen the `masked_encypted_extra` section and also tried that. This is another entry that isn't present in `ImportV1DatabaseSchema`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] Unable to import datasources with `encrypted_extra` params via REST [superset]
Posted by "betodealmeida (via GitHub)" <gi...@apache.org>.
betodealmeida commented on issue #31438:
URL: https://github.com/apache/superset/issues/31438#issuecomment-2541938249
Ah, looks like we need to update the schema.
I think the reasoning is that when we export the database we strip passwords and encrypted extra, so the import doesn't expect it. But people can add it manually, and the import should accept it and use it if present.
I'll fix it, it should be quick.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] Unable to import datasources with `encrypted_extra` params via REST [superset]
Posted by "withnale (via GitHub)" <gi...@apache.org>.
withnale commented on issue #31438:
URL: https://github.com/apache/superset/issues/31438#issuecomment-2541995754
Does it make sense to ensure the same validations take place on the CLI and REST invocations? It seems strange that an object with encrypted_extra can sneak through the schema validation on the CLI version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] Unable to import datasources with `encrypted_extra` params via REST [superset]
Posted by "betodealmeida (via GitHub)" <gi...@apache.org>.
betodealmeida closed issue #31438: Unable to import datasources with `encrypted_extra` params via REST
URL: https://github.com/apache/superset/issues/31438
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] Unable to import datasources with `encrypted_extra` params via REST [superset]
Posted by "geido (via GitHub)" <gi...@apache.org>.
geido commented on issue #31438:
URL: https://github.com/apache/superset/issues/31438#issuecomment-2541842205
@fisjac @betodealmeida wondering if you might have some ideas about this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] Unable to import datasources with `encrypted_extra` params via REST [superset]
Posted by "betodealmeida (via GitHub)" <gi...@apache.org>.
betodealmeida commented on issue #31438:
URL: https://github.com/apache/superset/issues/31438#issuecomment-2673100834
> Does it make sense to ensure the same validations take place on the CLI and REST invocations? It seems strange that an object with encrypted_extra can sneak through the schema validation on the CLI version.
I think it's OK to have a different validations for CLI and REST, since using the CLI implies direct access to the database. Because of this, allowing more powerful operations than the ones supported by the API makes sense, IMO. That said, I think the API should accept the encrypted extra, since it's not different than sending credentials when you create a DB.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] Unable to import datasources with `encrypted_extra` params via REST [superset]
Posted by "withnale (via GitHub)" <gi...@apache.org>.
withnale commented on issue #31438:
URL: https://github.com/apache/superset/issues/31438#issuecomment-2621392258
Hello. Did this ever get addressed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] Unable to import datasources with `encrypted_extra` params via REST [superset]
Posted by "dosubot[bot] (via GitHub)" <gi...@apache.org>.
dosubot[bot] commented on issue #31438:
URL: https://github.com/apache/superset/issues/31438#issuecomment-2541669924
<!-- Greeting -->
Hey @withnale! I'm here to help you with your Superset issue. I'm a bot designed to assist with bugs, questions, and contributions. Let's tackle this together!
<!-- Answer -->
A similar issue was previously reported and closed. The issue was about importing a Snowflake database with the `encrypted_extra` field, which was resolved by a pull request that allowed the `encrypted_extra` field to be imported <sup>[[1]](https://github.com/apache/superset/issues/22348)</sup>.
In your case, the failure when importing a BigQuery datasource with `encrypted_extra` parameters via the REST API `/api/v1/database/import` is likely due to the schema validation that renames `encrypted_extra` to `masked_encrypted_extra`. The schema expects `encrypted_extra` to be renamed to `masked_encrypted_extra`, which is automatically handled in the `superset import_datasources -p filename.yaml` command. However, when using the REST API, this renaming might not occur, leading to a failure if `encrypted_extra` is not manually renamed to `masked_encrypted_extra` in the payload <sup>[[2]](https://github.com/apache/superset/blob/master/tests/unit_tests/databases/schema_tests.py)</sup>.
<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
---
<sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/ace9b6c5-2b6b-44d3-9eb8-37bfc14b57d4?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/ace9b6c5-2b6b-44d3-9eb8-37bfc14b57d4?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/ace9b6c5-2b6b-44d3-9eb8-37bfc14b57d4?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/ace9b6c5-2b6b-44d3-9eb8-37bfc14b57d4?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/ace9b6c5-2b6b-44d3-9eb8-37bfc14b57d4?feedback_type=hallucination) | [Other](https://app.dosu.dev/response-feedback/ace9b6c5-2b6b-44d3-9eb8-37bfc14b57d4?feedback_type=other)</sup>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org