You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by GitBox <gi...@apache.org> on 2022/12/08 13:50:45 UTC
[GitHub] [superset] sapcode commented on issue #20708: Unable to read S3 parquet files using DuckDB as Database Connector
sapcode commented on issue #20708:
URL: https://github.com/apache/superset/issues/20708#issuecomment-1342768294
Dear @Mause,
we have installed the superset with the current helm chart and also installed the DuckDB engine, it shows now in the Databases Connections dropdown. The superset version in the right menu is shown as "Version: 0.0.0-dev".
When we create a new DuckDB database without engine parameter but with "Allow DML" and call SQL-Lab we are able to use the connection settings when executing as DML statements and the data is delivered correctly:
install 'httpfs';
load 'httpfs';
SET s3_endpoint='argo-artifacts:9000';
SET s3_access_key_id='XXX';
SET s3_secret_access_key='XXX';
SET s3_url_style = 'path';
SET s3_use_ssl=false;
select count(*) from 's3://raw/*.parquet';
However we don't want to provide the credentials in the SQL-Lab but would like to store them in the DuckDB Connector Advanced ENGINE PARAMETER dialog, we have tested the following parameter JSON strings but none of them seems to work:
Test1:
`{
"engine_params": {
"connect_args":{
"preload_extensions": ["httpfs"],
"config": {
"s3_endpoint": "argo-artifacts:9000",
"s3_access_key_id": "XXX",
"s3_secret_access_key": "XXX",
"s3_url_style": "path",
"s3_use_ssl": "False"
}
}
}
}`
Test2:
`{
"connect_args":{
"preload_extensions": ["httpfs"],
"config": {
"s3_endpoint": "argo-artifacts:9000",
"s3_access_key_id": "XXX",
"s3_secret_access_key": "XXX",
"s3_url_style": "path",
"s3_use_ssl": "False"
}
}
}`
We always get the same error:
**An error occurred while fetching databases: "Connection failed, please check your connection settings"**
In chrome debug we see:
PUT http://superset.mini.kube/api/v1/database/1 422 (UNPROCESSABLE ENTITY)
Response {type: 'basic', url: 'http://superset.mini.kube/api/v1/database/1'
When looking at the core.py line 394 - 405 it looks like that "engine_params" is searched first, then "connect_args"
https://github.com/apache/superset/blob/master/superset/models/core.py
params = extra.get("engine_params", {})
if nullpool:
params["poolclass"] = NullPool
connect_args = params.get("connect_args", {})
if self.impersonate_user:
self.db_engine_spec.update_impersonation_config(
connect_args, str(sqlalchemy_url), effective_username
)
if connect_args:
params["connect_args"] = connect_args
In Connector Advanced ENGINE PARAMETER dialog, what is the specific connection string which need to be passed ?
Are the any restrictions in usage of double " or single ' quotes in the JSON string ?
Thank you very much!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org