You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by GitBox <gi...@apache.org> on 2021/04/01 13:44:30 UTC
[GitHub] [superset] krsnik93 opened a new issue #13914: CLI commands import_* and export_* are completely broken
krsnik93 opened a new issue #13914:
URL: https://github.com/apache/superset/issues/13914
I am trying to develop a procedure for moving charts, dashboards and other resources from one instance to another.
I started with the simplest attempt: export dashboards from one superset instance and import them back in.
Running `superset export-dashboards` creates a file called `dashboard_export_YYYYMMDDTHHMMSS`. The file is a .zip archive, which is difficult to know because the extension is missing. The `YYYYMMDDTHHMMSS` part of it is redundant and should be dropped. Upon unzipping the file, the contents are directories `charts`, `dashboards`, `databases`, `datasets` and a file called `metadata.yaml`. So even though the command is called `export-dashboards`, it actually exports other resource types as well. Not sure it's needed, certainly is unclear.
Importing back a dashboard with:
`superset import-dashboards --path dashboard_export_20210401T140536/dashboards/USA_Births_Names.yaml`
gives:
```
File "/home/fedora/superset/venv/lib64/python3.8/site-packages/superset/cli.py", line 318, in import_dashboards
ImportDashboardsCommand(contents).run()
File "/home/fedora/superset/venv/lib64/python3.8/site-packages/superset/dashboards/commands/importers/dispatcher.py", line 69, in run
raise exc
File "/home/fedora/superset/venv/lib64/python3.8/site-packages/superset/dashboards/commands/importers/dispatcher.py", line 58, in run
command.run()
File "/home/fedora/superset/venv/lib64/python3.8/site-packages/superset/dashboards/commands/importers/v0.py", line 325, in run
self.validate()
File "/home/fedora/superset/venv/lib64/python3.8/site-packages/superset/dashboards/commands/importers/v0.py", line 335, in validate
json.loads(content)
File "/usr/lib64/python3.8/json/__init__.py", line 357, in loads
return _default_decoder.decode(s)
File "/usr/lib64/python3.8/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python3.8/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
```
So it expects a .json but receives a .yaml as exported.
Then I tried a similar approach with `export-datasources` and `import-datasources`. Exporting also creates a .zip archive with the extension missing. Unzipping gives a bunch of .yaml files. Importing back with:
`superset import-datasources --path dataset_export_20210401T141602/datasets/examples/random_time_series.yaml`
gives:
```
Traceback (most recent call last):
File "/home/fedora/superset/venv/bin/superset", line 11, in <module>
load_entry_point('apache-superset==1.0.1', 'console_scripts', 'superset')()
File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/home/fedora/superset/venv/lib64/python3.8/site-packages/flask/cli.py", line 586, in main
return super(FlaskGroup, self).main(*args, **kwargs)
File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/decorators.py", line 21, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/fedora/superset/venv/lib64/python3.8/site-packages/flask/cli.py", line 426, in decorator
return __ctx.invoke(f, *args, **kwargs)
File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/decorators.py", line 21, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/fedora/superset/venv/lib64/python3.8/site-packages/flask/cli.py", line 426, in decorator
return __ctx.invoke(f, *args, **kwargs)
File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
TypeError: import_datasources() got an unexpected keyword argument 'sync'
```
### Expected results
Expected to be able to import/export unchanged files smoothly.
### Actual results
I might be using it wrong, but there seem to be many problems with this. File naming etc. is minor but the elementary .yaml/.json mismatch is really weird.
#### How to reproduce the bug
Described above in great detail.
### Environment
- superset version: 1.0.1
- python version: 3.8.7
### Checklist
Make sure to follow these steps before submitting your issue - thank you!
- [x] I have checked the superset logs for python stacktraces and included it here as text if there are any.
- [x] I have reproduced the issue with at least the latest released version of superset.
- [x] I have checked the issue tracker for the same issue and I haven't found one similar.
### Additional context
I have encountered a similar problem when using REST API instead of the CLI commands:
https://github.com/apache/superset/issues/13239
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] krsnik93 commented on issue #13914: [Question]CLI commands import_* and export_*
Posted by GitBox <gi...@apache.org>.
krsnik93 commented on issue #13914:
URL: https://github.com/apache/superset/issues/13914#issuecomment-812056358
> The reason why the filename includes the date is to help differentiate between exports. Exports can contain multiple objects (you could export 100 dashboards with all the associated charts, datasets and databases, and it would still be a single file), so we can't name the file based on the exported object.
I don't have a problem with differentiating exported files with a timestamp included in the filename. However, the filename does not have an actual timestamp but a placeholder `YYYYMMDDTHHMMSS`. Only after unzipping it does the resulting root directory have an actual timestamp in its filename. This is most likely another bug then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] betodealmeida closed issue #13914: [Question]CLI commands import_* and export_*
Posted by GitBox <gi...@apache.org>.
betodealmeida closed issue #13914:
URL: https://github.com/apache/superset/issues/13914
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] junlincc commented on issue #13914: CLI commands import_* and export_* are completely broken
Posted by GitBox <gi...@apache.org>.
junlincc commented on issue #13914:
URL: https://github.com/apache/superset/issues/13914#issuecomment-812033174
@betodealmeida Hi Beto, I believe this is in you domain. 🙏
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] krsnik93 commented on issue #13914: [Question]CLI commands import_* and export_*
Posted by GitBox <gi...@apache.org>.
krsnik93 commented on issue #13914:
URL: https://github.com/apache/superset/issues/13914#issuecomment-824346282
@betodealmeida thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] krsnik93 edited a comment on issue #13914: [Question]CLI commands import_* and export_*
Posted by GitBox <gi...@apache.org>.
krsnik93 edited a comment on issue #13914:
URL: https://github.com/apache/superset/issues/13914#issuecomment-812056358
> The reason why the filename includes the date is to help differentiate between exports. Exports can contain multiple objects (you could export 100 dashboards with all the associated charts, datasets and databases, and it would still be a single file), so we can't name the file based on the exported object.
I don't have a problem with differentiating exported files with a timestamp included in the filename. However, the filename does not have an actual timestamp but a placeholder `YYYYMMDDTHHMMSS`. Only after unzipping it does the resulting root directory have an actual timestamp in its filename. This is most likely a bug as it makes differentiating impossible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] betodealmeida commented on issue #13914: [Question]CLI commands import_* and export_*
Posted by GitBox <gi...@apache.org>.
betodealmeida commented on issue #13914:
URL: https://github.com/apache/superset/issues/13914#issuecomment-812095374
> I don't have a problem with differentiating exported files with a timestamp included in the filename. However, the filename does not have an actual timestamp but a placeholder `YYYYMMDDTHHMMSS`. Only after unzipping it does the resulting root directory have an actual timestamp in its filename. This is most likely another bug then.
Oh, yeah, that's a bug for sure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] krsnik93 commented on issue #13914: CLI commands import_* and export_* are completely broken
Posted by GitBox <gi...@apache.org>.
krsnik93 commented on issue #13914:
URL: https://github.com/apache/superset/issues/13914#issuecomment-812051549
Sorry for not including in the description as I assumed it was not the way to use it, but I did try to import the whole archive first.
For `superset import-dashboards --path dashboard_export_YYYYMMDDTHHMMSS` (so using the archive "untouched"):
```
Traceback (most recent call last):
File "/home/fedora/superset/venv/bin/superset", line 11, in <module>
load_entry_point('apache-superset==1.0.1', 'console_scripts', 'superset')()
File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/home/fedora/superset/venv/lib64/python3.8/site-packages/flask/cli.py", line 586, in main
return super(FlaskGroup, self).main(*args, **kwargs)
File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/decorators.py", line 21, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/fedora/superset/venv/lib64/python3.8/site-packages/flask/cli.py", line 426, in decorator
return __ctx.invoke(f, *args, **kwargs)
File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/decorators.py", line 21, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/fedora/superset/venv/lib64/python3.8/site-packages/flask/cli.py", line 426, in decorator
return __ctx.invoke(f, *args, **kwargs)
File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/home/fedora/superset/venv/lib64/python3.8/site-packages/superset/cli.py", line 316, in import_dashboards
contents = {path: open(path).read()}
File "/usr/lib64/python3.8/codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x97 in position 14: invalid start byte
```
And for `superset import-datasources --path dataset_export_YYYYMMDDTHHMMSS`:
```
Traceback (most recent call last):
File "/home/fedora/superset/venv/bin/superset", line 11, in <module>
load_entry_point('apache-superset==1.0.1', 'console_scripts', 'superset')()
File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/home/fedora/superset/venv/lib64/python3.8/site-packages/flask/cli.py", line 586, in main
return super(FlaskGroup, self).main(*args, **kwargs)
File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/decorators.py", line 21, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/fedora/superset/venv/lib64/python3.8/site-packages/flask/cli.py", line 426, in decorator
return __ctx.invoke(f, *args, **kwargs)
File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/decorators.py", line 21, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/fedora/superset/venv/lib64/python3.8/site-packages/flask/cli.py", line 426, in decorator
return __ctx.invoke(f, *args, **kwargs)
File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
TypeError: import_datasources() got an unexpected keyword argument 'sync'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] betodealmeida commented on issue #13914: CLI commands import_* and export_* are completely broken
Posted by GitBox <gi...@apache.org>.
betodealmeida commented on issue #13914:
URL: https://github.com/apache/superset/issues/13914#issuecomment-812047635
@krsnik93, this is not "completely broken", and I feel that that language is disrespectful.
The new export produces a ZIP file. The contents of the ZIP file include related objects needed to fully import what was exported. For a dashboard, that means charts, datasets and databases. This is described in https://github.com/apache/superset/issues/11167.
The reason why the filename includes the date is to help differentiate between exports. Exports can contain multiple objects (you could export 100 dashboards with all the associated charts, datasets and databases, and it would still be a single file), so we can't name the file based on the exported object.
The ZIP is an atomic bundle. You're not supposed to unzip it and import individual YAML files. You have to import the whole ZIP either in the UI or in the command line.
The reason why you can import YAML in the command line and UI is because in the original export/import we would use JSON for dashboards, and YAML for datasets.
This is the expected behavior:
1. The filename should have the `.zip` extension, so that's a bug.
2. If you export a ZIP file from the command line you should be able to import that unmodified file, either from the UI or from the CLI. If it doesn't work, it's a bug.
3. If you try to import an old JSON or YAML file it should work, both from the UI as well as CLI. If it doesn't work it's a bug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] betodealmeida commented on issue #13914: [Question]CLI commands import_* and export_*
Posted by GitBox <gi...@apache.org>.
betodealmeida commented on issue #13914:
URL: https://github.com/apache/superset/issues/13914#issuecomment-812166670
Thanks for filing the issue, @krsnik93. I have a PR out fixing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] betodealmeida commented on issue #13914: [Question]CLI commands import_* and export_*
Posted by GitBox <gi...@apache.org>.
betodealmeida commented on issue #13914:
URL: https://github.com/apache/superset/issues/13914#issuecomment-824310349
This has been fixed in https://github.com/apache/superset/pull/13921
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] krsnik93 edited a comment on issue #13914: [Question]CLI commands import_* and export_*
Posted by GitBox <gi...@apache.org>.
krsnik93 edited a comment on issue #13914:
URL: https://github.com/apache/superset/issues/13914#issuecomment-812056358
> The reason why the filename includes the date is to help differentiate between exports. Exports can contain multiple objects (you could export 100 dashboards with all the associated charts, datasets and databases, and it would still be a single file), so we can't name the file based on the exported object.
I don't have a problem with differentiating exported files with a timestamp included in the filename. However, the filename does not have an actual timestamp but a placeholder `YYYYMMDDTHHMMSS`. Only after unzipping it does the resulting root directory have an actual timestamp in its filename. This is most likely another bug then as it makes differentiating impossible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] betodealmeida commented on issue #13914: [Question]CLI commands import_* and export_*
Posted by GitBox <gi...@apache.org>.
betodealmeida commented on issue #13914:
URL: https://github.com/apache/superset/issues/13914#issuecomment-812056090
@krsnik93 thanks, looks like the CLI has a bug when importing the ZIP, we need to implement the logic that's used in the API to load the contents of the file. I can fix it in the next few days and test to make sure it works. We've been using this feature only in the UI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org