You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "ilsaloving (via GitHub)" <gi...@apache.org> on 2024/09/23 19:14:34 UTC
[I] superset cli export_datasource_schema does not exist [superset]
ilsaloving opened a new issue, #30363:
URL: https://github.com/apache/superset/issues/30363
### Bug description
I am trying to figure out how to get data sources created in superset in an automated way. There does not appear to be a CLI command for adding a data source, but it does support importing definitions from yaml files.
According to [this](https://superset.apache.org/docs/configuration/importing-exporting-datasources/#exporting-datasources-to-yaml) doc page, it should be possible to do `superset export_datasource_schema` to get a comprehensive list of yaml directives.
As of 4.0.2, this command does not exist:
```
superset@0ffc84ec25b5:/app$ superset --help
Loading custom config...
Loaded your LOCAL configuration at [/app/superset_config.py]
logging was configured successfully
2024-09-23 19:08:03,111:INFO:superset.utils.logging_configurator:logging was configured successfully
2024-09-23 19:08:03,119:INFO:root:Configured event logger of type <class 'superset.utils.log.DBEventLogger'>
/usr/local/lib/python3.10/site-packages/flask_limiter/extension.py:293: UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend.
warnings.warn(
Usage: superset [OPTIONS] COMMAND [ARGS]...
This is a management script for the Superset application.
Options:
-e, --env-file FILE Load environment variables from this file. python-
dotenv must be installed.
-A, --app IMPORT The Flask application or factory function to load, in
the form 'module:name'. Module can be a dotted import
or file path. Name is not required if it is 'app',
'application', 'create_app', or 'make_app', and can be
'name(args)' to pass arguments.
--debug / --no-debug Set debug mode.
--version Show the Flask version.
--help Show this message and exit.
Commands:
compute-thumbnails Compute thumbnails
db Perform database migrations.
export-dashboards Export dashboards to ZIP file
export-datasources Export datasources to ZIP file
fab FAB flask group commands
import-dashboards Import dashboards from ZIP file
import-datasources Import datasources from ZIP file
import-directory Imports configs from a given directory
init Inits the Superset application
legacy-export-dashboards Export dashboards to JSON
legacy-export-datasource-schema
Export datasource YAML schema to stdout
legacy-export-datasources Export datasources to YAML
legacy-import-dashboards Import dashboards from JSON file
legacy-import-datasources Import datasources from YAML
limiter Flask-Limiter maintenance & utility...
load-examples Loads a set of Slices and Dashboards...
load-test-users Loads admin, alpha, and gamma user for...
migrate-viz Migrate a viz from one type to another.
re-encrypt-secrets
routes Show the routes for the app.
run Run a development server.
set-database-uri Updates a database connection URI
shell Run a shell in the app context.
superset This is a management script for the...
sync-tags Rebuilds special tags (owner, type,...
test-db Run a series of tests against an...
update-api-docs Regenerate the openapi.json file in docs
version Prints the current version number
```
Additionally, if I try to do `superset legacy-export-datasource-schema`, I get the following error:
```
superset@0ffc84ec25b5:/app$ superset legacy-export-datasource-schema
Loading custom config...
Loaded your LOCAL configuration at [/app/superset_config.py]
logging was configured successfully
2024-09-23 19:09:09,331:INFO:superset.utils.logging_configurator:logging was configured successfully
2024-09-23 19:09:09,339:INFO:root:Configured event logger of type <class 'superset.utils.log.DBEventLogger'>
/usr/local/lib/python3.10/site-packages/flask_limiter/extension.py:293: UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend.
warnings.warn(
Traceback (most recent call last):
File "/usr/local/bin/superset", line 33, in <module>
sys.exit(load_entry_point('apache-superset', 'console_scripts', 'superset')())
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/flask/cli.py", line 357, in decorator
return __ctx.invoke(f, *args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/app/superset/cli/importexport.py", line 404, in legacy_export_datasource_schema
data = dict_import_export.export_schema_to_dict(back_references=back_references)
File "/app/superset/utils/dict_import_export.py", line 31, in export_schema_to_dict
Database.export_schema(recursive=True, include_parent_ref=back_references)
File "/app/superset/models/helpers.py", line 240, in export_schema
child_class.export_schema(
File "/app/superset/models/helpers.py", line 238, in export_schema
child_class = cls.__mapper__.relationships[column].argument.class_
AttributeError: type object 'SqlMetric' has no attribute 'class_'. Did you mean: '__class__'?
```
### How to reproduce the bug
1. Spin up a superset docker container
2. Configure it as desired
3. Try to obtain the yaml schema definition so I can construct a new yaml file
### Screenshots/recordings
_No response_
### Superset version
4.0.2
### Python version
Not applicable
### Node version
Not applicable
### Browser
Not applicable
### Additional context
2024-09-23 19:09:09,339:INFO:root:Configured event logger of type <class 'superset.utils.log.DBEventLogger'>
/usr/local/lib/python3.10/site-packages/flask_limiter/extension.py:293: UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend.
warnings.warn(
Traceback (most recent call last):
File "/usr/local/bin/superset", line 33, in <module>
sys.exit(load_entry_point('apache-superset', 'console_scripts', 'superset')())
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/flask/cli.py", line 357, in decorator
return __ctx.invoke(f, *args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/app/superset/cli/importexport.py", line 404, in legacy_export_datasource_schema
data = dict_import_export.export_schema_to_dict(back_references=back_references)
File "/app/superset/utils/dict_import_export.py", line 31, in export_schema_to_dict
Database.export_schema(recursive=True, include_parent_ref=back_references)
File "/app/superset/models/helpers.py", line 240, in export_schema
child_class.export_schema(
File "/app/superset/models/helpers.py", line 238, in export_schema
child_class = cls.__mapper__.relationships[column].argument.class_
AttributeError: type object 'SqlMetric' has no attribute 'class_'. Did you mean: '__class__'?
### Checklist
- [X] I have searched Superset docs and Slack and didn't find a solution to my problem.
- [X] I have searched the GitHub issue tracker and didn't find a similar bug report.
- [X] I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] superset cli export_datasource_schema does not exist [superset]
Posted by "betodealmeida (via GitHub)" <gi...@apache.org>.
betodealmeida commented on issue #30363:
URL: https://github.com/apache/superset/issues/30363#issuecomment-3129330558
> export-datasources does not export data sources at all. It only exports datasets.
Sorry, can you clarify what's the difference for you between a data source and a dataset? We use the term somewhat as synonyms in Superset, for historical reasons. But the export only works for datasets. What else do you want to export?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] superset cli export_datasource_schema does not exist [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on issue #30363:
URL: https://github.com/apache/superset/issues/30363#issuecomment-2369232693
It looks like `export_datasource_schema` has been renamed to `legacy_export_datasource_schema` - I'm not sure if it's on the path to deprecation or if there's another reason for the rename. @betodealmeida might know.
You may also want to take a look at the Preset CLI (not maintained by the Superset project, but it is built to be compatible with Superset).
https://preset.io/blog/version-control-superset-charts-dashboards-superset/
We probably need to update our docs to fix the `export_datasource_schema` reference, but I'd like to know more before making a change myself. PRs welcome, of course :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] superset cli export_datasource_schema does not exist [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on issue #30363:
URL: https://github.com/apache/superset/issues/30363#issuecomment-4597102974
I think this one is out of date, with the CLI command renamed to legacy_export_datasource_schema, and some docs staleness issues... Closing as not planned, but feel free to reopen or start this anew if you disagree :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] superset cli export_datasource_schema does not exist [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on issue #30363:
URL: https://github.com/apache/superset/issues/30363#issuecomment-2810910467
Hi there. Sorry this has gone cold for > 6 months, and is thus at risk of being closed as stale/not-planned. We'd welcome a PR adding/updating API endpoints to fit your use case. My understanding is that the CLI more or less conjures API functionality, so I'm not sure if that warrants further investigation. Let us know if you're still facing this in newer versions (4.1.2 or newer), and we can try to act accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] superset cli export_datasource_schema does not exist [superset]
Posted by "dosubot[bot] (via GitHub)" <gi...@apache.org>.
dosubot[bot] commented on issue #30363:
URL: https://github.com/apache/superset/issues/30363#issuecomment-2369158486
<!-- Greeting -->
Hey @ilsaloving, I'm here to help you with any bugs, questions, or contributions you have regarding Superset. Let's tackle this together!
<!-- Answer -->
To obtain the YAML schema definition for data sources in Superset 4.0.2 using the CLI, you should use the `export_datasource_schema` command. Here is the command you need to run:
```sh
superset export_datasource_schema
```
This command will export the datasource YAML schema to stdout. If you want to include parent back references, you can add the `-b` flag:
```sh
superset export_datasource_schema -b
```
Regarding the `AttributeError` you encountered with the `legacy-export-datasource-schema` command, it seems to be related to a change in the `export_schema_to_dict` function in `dict_import_export.py`. The function was modified to call `Database.export_schema` with the parameters `recursive=True` and `include_parent_ref=back_references`. This change might cause an `AttributeError` if the `Database` class does not have an `export_schema` method.
Here is the relevant code snippet:
```python
def export_schema_to_dict(back_references: bool) -> dict[str, Any]:
"""Exports the supported import/export schema to a dictionary"""
databases = [
Database.export_schema(recursive=True, include_parent_ref=back_references)
]
data = {}
if databases:
data[DATABASES_KEY] = databases
return data
```
If `Database` does not have an `export_schema` method, attempting to call it will result in an `AttributeError` <sup>[[1]](https://github.com/apache/superset/blob/master/superset/cli/importexport.py)</sup><sup>[[2]](https://github.com/apache/superset/blob/master/superset/utils/dict_import_export.py)</sup>.
For the `SqlMetric` class, the `class_` attribute is not defined. The `SqlMetric` class includes attributes such as `id`, `metric_name`, `verbose_name`, `metric_type`, `description`, `d3format`, `currency`, `warning_text`, `table_id`, `expression`, and `extra`. It also includes relationships, methods, and properties to handle various functionalities related to SQL metrics <sup>[[3]](https://github.com/apache/superset/blob/master/superset/connectors/sqla/models.py)</sup><sup>[[4]](https://github.com/apache/superset/blob/master/tests/unit_tests/datasets/conftest.py)</sup>.
To resolve the issue, you may need to ensure that the `Database` class has the `export_schema` method implemented or adjust the code to handle the absence of this method.
<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] superset cli export_datasource_schema does not exist [superset]
Posted by "ilsaloving (via GitHub)" <gi...@apache.org>.
ilsaloving commented on issue #30363:
URL: https://github.com/apache/superset/issues/30363#issuecomment-2402894589
Is there any update to this?
- legacy-export-datasources works but doesn't include the security information (credentials, impersonation, etc)
- export-datasources does not export data sources at all. It only exports datasets.
Short of using the REST API, there does not appear to be any way to export/import superset data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] superset cli export_datasource_schema does not exist [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas closed issue #30363: superset cli export_datasource_schema does not exist
URL: https://github.com/apache/superset/issues/30363
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org