You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by GitBox <gi...@apache.org> on 2021/10/09 13:46:10 UTC
[GitHub] [superset] maxcapraro opened a new issue #17049: Import of datasources (`superset import-datasources`) fails
maxcapraro opened a new issue #17049:
URL: https://github.com/apache/superset/issues/17049
The import of datasources (e.g. the connection to a PostgreSQL database) fails when using `superset import-datasources`.
#### How to reproduce the bug
First, I performed an export of a database connection:
1. Spin up a new docker container (from my own image which is basically `superset/latest` + my config file)
1. Log into the application and navigate to `databaseview/list`
1. Create a new database by clicking on `+ DATABASE` (for reproduction, I created a new PostgreSQL connection named `db-for-debugging-import`)
1. Export the database by Clicking the Export-Icon in the database list (resulting in this ZIP file: [database_export_20211009T125106.zip](https://github.com/apache/superset/files/7315592/database_export_20211009T125106.zip))
Subsequently, I attempted to import the file:
1. Unzip the exported datasource, replace the `xxxxxxx...` with the actual password (in this case a throw-away password), make a new zip archive (resulting in this ZIP file: [database_export_20211009T125106.zip](https://github.com/apache/superset/files/7315599/database_export_20211009T125106.zip))
1. Spin up a new docker container
1. Import the ZIP file by executing `superset import-datasources -p ...` in the container's shell
### Expected results
I expected the database to show up under `databaseview/list` and e.g. being able to use it when creating a new dataset (selecting the database, a schema, and a table).
### Actual results
When importing via `superset import-datasources -p`, the logs contained the following:
````
superset@kolabri-dashboard-deployment-dd8dd7754-p5sq5:/app$ superset import_datasources -p /kolabri-assets/database_export_20211009T125106.zip
Loaded your LOCAL configuration at [/kolabri-assets/superset_config.py]
logging was configured successfully
2021-10-09 13:27:15,936:INFO:superset.utils.logging_configurator:logging was configured successfully
2021-10-09 13:27:15,946:INFO:root:Configured event logger of type <class 'superset.utils.log.DBEventLogger'>
/usr/local/lib/python3.8/site-packages/flask_caching/__init__.py:201: UserWarning: Flask-Caching: CACHE_TYPE is set to null, caching is effectively disabled.
warnings.warn(
Command failed validation
2021-10-09 13:27:17,396:INFO:superset.datasets.commands.importers.dispatcher:Command failed validation
There was an error when importing the dataset(s), please check the exception traceback in the log
Traceback (most recent call last):
File "/app/superset/cli.py", line 384, in import_datasources
ImportDatasetsCommand(contents, overwrite=True).run()
File "/app/superset/datasets/commands/importers/dispatcher.py", line 64, in run
raise exc
File "/app/superset/datasets/commands/importers/dispatcher.py", line 57, in run
command.run()
File "/app/superset/commands/importers/v1/__init__.py", line 63, in run
self.validate()
File "/app/superset/commands/importers/v1/__init__.py", line 146, in validate
raise exception
superset.commands.exceptions.CommandInvalidError: Error importing dataset
2021-10-09 13:27:17,396:ERROR:superset.cli:There was an error when importing the dataset(s), please check the exception traceback in the log
Traceback (most recent call last):
File "/app/superset/cli.py", line 384, in import_datasources
ImportDatasetsCommand(contents, overwrite=True).run()
File "/app/superset/datasets/commands/importers/dispatcher.py", line 64, in run
raise exc
File "/app/superset/datasets/commands/importers/dispatcher.py", line 57, in run
command.run()
File "/app/superset/commands/importers/v1/__init__.py", line 63, in run
self.validate()
File "/app/superset/commands/importers/v1/__init__.py", line 146, in validate
raise exception
superset.commands.exceptions.CommandInvalidError: Error importing dataset
````
#### Screenshots
If applicable, add screenshots to help explain your problem.
### Environment
(please complete the following information):
- browser type and version: `Version 94.0.4606.71 (x86_64)`
- superset version: `0.0.0dev` (last pull of the base image was a on October 4th)
- python version: `Python 3.8.12`
- node.js version: No node.js installed
- any feature flags active:
- `'VERSIONED_EXPORT' : True`
### Checklist
Make sure to follow these steps before submitting your issue - thank you!
- [x] I have checked the superset logs for python stacktraces and included it here as text if there are any.
- [ ] I have reproduced the issue with at least the latest released version of superset.
- [x] I have checked the issue tracker for the same issue and I haven't found one similar.
Regarding the second checkbox: I am happy to reproduce this with another "proper" version if this helps, but from [this](https://hub.docker.com/r/apache/superset) I am not 100% sure this is possible yet using the docker images :-)
### Additional context
- When importing the ZIP-file via the UI, the import succeeds. Though, an error message appears when trying to edit the database connection: `ERROR: Please re-enter the password`. I am not sure if this is the expected behaviour or part of the issue.
- It appears to me (from my outsider's perspective) that this bug is not identical to #16858 and #16848
I am happy to help if any additional info / debugging is useful and thanks in advance for the help :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] andrzejmartyna commented on issue #17049: Import of datasources (`superset import-datasources`) fails
Posted by GitBox <gi...@apache.org>.
andrzejmartyna commented on issue #17049:
URL: https://github.com/apache/superset/issues/17049#issuecomment-999729061
I experienced the same inconsistency between importing from file and entering from UI.
"Please re-enter the password" is especially irritating because:
- the connection does not work in dashboards unless you... re-enter the password
- you cannot enter the password just by modifying connection string because it says again "re-enter the password"
- workaround 1 [failed]: trying to change the password, save, and then change to the correct password ("save" not allowed as it is not a valid collection - should it be allowed? I believe a very valid situation may be that the database is not available at the time of configuration)
- workaround 2 (succeeded): enter valid, **_but different_**, connection string, save, go back and enter correct connection string, save
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] BFergerson commented on issue #17049: Import of datasources (`superset import-datasources`) fails
Posted by GitBox <gi...@apache.org>.
BFergerson commented on issue #17049:
URL: https://github.com/apache/superset/issues/17049#issuecomment-997880612
I believe I'm experience the same issue. My setup looks like:
### docker-compose.yml
```yml
version: '3.3'
services:
superset:
container_name: superset
hostname: superset
ports:
- "8888:8088"
depends_on:
- postgres
links:
- postgres
build:
context: .
dockerfile: Dockerfile-superset
postgres:
image: postgres:9.6.21
container_name: postgres
hostname: postgres
environment:
- POSTGRES_PASSWORD=123456
- POSTGRES_USER=postgres
- POSTGRES_DB=postgres
- POSTGRES_HOST_AUTH_METHOD=md5
```
### datasources.yml
```yml
databases:
- database_name: postgres
extra: '{"metadata_params":{},"engine_params":{},"metadata_cache_timeout":{},"schemas_allowed_for_file_upload":[]}'
sqlalchemy_uri: postgresql+psycopg2://postgres:123456@postgres:5432/postgres
```
#### setup-superset.sh
```sh
docker exec -it superset superset fab create-admin \
--username admin \
--firstname Superset \
--lastname Admin \
--email admin@superset.com \
--password admin
docker exec -it superset superset db upgrade
docker exec -it superset superset init
docker exec -it superset superset import-datasources -p datasources.yml
```
---
If I input `postgresql+psycopg2://postgres:123456@postgres:5432/postgres` through the UI, everything works fine. It is only when I use `import-datasources` that I get `ERROR: Please re-enter the password`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] BFergerson edited a comment on issue #17049: Import of datasources (`superset import-datasources`) fails
Posted by GitBox <gi...@apache.org>.
BFergerson edited a comment on issue #17049:
URL: https://github.com/apache/superset/issues/17049#issuecomment-997880612
I believe I'm experiencing the same issue. My setup looks like:
### docker-compose.yml
```yml
version: '3.3'
services:
superset:
container_name: superset
hostname: superset
ports:
- "8888:8088"
depends_on:
- postgres
links:
- postgres
build:
context: .
dockerfile: Dockerfile-superset
postgres:
image: postgres:9.6.21
container_name: postgres
hostname: postgres
environment:
- POSTGRES_PASSWORD=123456
- POSTGRES_USER=postgres
- POSTGRES_DB=postgres
- POSTGRES_HOST_AUTH_METHOD=md5
```
### Dockerfile-superset
```
FROM apache/superset:latest
USER root
RUN pip install psycopg2-binary
USER superset
ADD datasources.yml ./
```
### datasources.yml
```yml
databases:
- database_name: postgres
extra: '{"metadata_params":{},"engine_params":{},"metadata_cache_timeout":{},"schemas_allowed_for_file_upload":[]}'
sqlalchemy_uri: postgresql+psycopg2://postgres:123456@postgres:5432/postgres
```
#### setup-superset.sh
```sh
docker exec -it superset superset fab create-admin \
--username admin \
--firstname Superset \
--lastname Admin \
--email admin@superset.com \
--password admin
docker exec -it superset superset db upgrade
docker exec -it superset superset init
docker exec -it superset superset import-datasources -p datasources.yml
```
---
If I input `postgresql+psycopg2://postgres:123456@postgres:5432/postgres` through the UI, everything works fine. It is only when I use `import-datasources` that I get `ERROR: Please re-enter the password`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] BFergerson edited a comment on issue #17049: Import of datasources (`superset import-datasources`) fails
Posted by GitBox <gi...@apache.org>.
BFergerson edited a comment on issue #17049:
URL: https://github.com/apache/superset/issues/17049#issuecomment-997880612
I believe I'm experiencing the same issue. My setup looks like:
### docker-compose.yml
```yml
version: '3.3'
services:
superset:
container_name: superset
hostname: superset
ports:
- "8888:8088"
depends_on:
- postgres
links:
- postgres
build:
context: .
dockerfile: Dockerfile-superset
postgres:
image: postgres:9.6.21
container_name: postgres
hostname: postgres
environment:
- POSTGRES_PASSWORD=123456
- POSTGRES_USER=postgres
- POSTGRES_DB=postgres
- POSTGRES_HOST_AUTH_METHOD=md5
```
### Dockerfile-superset
```
FROM apache/superset:latest
USER root
RUN pip install psycopg2-binary
USER superset
ADD datasources.yml ./
```
### datasources.yml
```yml
databases:
- database_name: postgres
extra: '{"metadata_params":{},"engine_params":{},"metadata_cache_timeout":{},"schemas_allowed_for_file_upload":[]}'
sqlalchemy_uri: postgresql+psycopg2://postgres:123456@postgres:5432/postgres
```
#### setup-superset.sh
```sh
docker exec -it superset superset fab create-admin \
--username admin \
--firstname Superset \
--lastname Admin \
--email admin@superset.com \
--password admin
docker exec -it superset superset db upgrade
docker exec -it superset superset init
docker exec -it superset superset import-datasources -p datasources.yml
```
---
If I input `postgresql+psycopg2://postgres:123456@postgres:5432/postgres` through the UI, everything works fine. It is only when I use `import-datasources` that I get `ERROR: Please re-enter the password`.
---
Note: If you update the connection via the UI things work again. What's interesting is when you add the datasource through `import-datasource` you can see the database password in the UI. Whereas if you do this via UI, the password you see is `XXXXXXXXXX`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] BFergerson edited a comment on issue #17049: Import of datasources (`superset import-datasources`) fails
Posted by GitBox <gi...@apache.org>.
BFergerson edited a comment on issue #17049:
URL: https://github.com/apache/superset/issues/17049#issuecomment-997880612
I believe I'm experience the same issue. My setup looks like:
### docker-compose.yml
```yml
version: '3.3'
services:
superset:
container_name: superset
hostname: superset
ports:
- "8888:8088"
depends_on:
- postgres
links:
- postgres
build:
context: .
dockerfile: Dockerfile-superset
postgres:
image: postgres:9.6.21
container_name: postgres
hostname: postgres
environment:
- POSTGRES_PASSWORD=123456
- POSTGRES_USER=postgres
- POSTGRES_DB=postgres
- POSTGRES_HOST_AUTH_METHOD=md5
```
### Dockerfile-superset
```
FROM apache/superset:latest
USER root
RUN pip install psycopg2-binary
USER superset
ADD datasources.yml ./
```
### datasources.yml
```yml
databases:
- database_name: postgres
extra: '{"metadata_params":{},"engine_params":{},"metadata_cache_timeout":{},"schemas_allowed_for_file_upload":[]}'
sqlalchemy_uri: postgresql+psycopg2://postgres:123456@postgres:5432/postgres
```
#### setup-superset.sh
```sh
docker exec -it superset superset fab create-admin \
--username admin \
--firstname Superset \
--lastname Admin \
--email admin@superset.com \
--password admin
docker exec -it superset superset db upgrade
docker exec -it superset superset init
docker exec -it superset superset import-datasources -p datasources.yml
```
---
If I input `postgresql+psycopg2://postgres:123456@postgres:5432/postgres` through the UI, everything works fine. It is only when I use `import-datasources` that I get `ERROR: Please re-enter the password`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org