You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by GitBox <gi...@apache.org> on 2022/06/17 20:02:29 UTC
[GitHub] [superset] Thelin90 opened a new issue, #20428: Presto-Trino Does Not Work Superset Version 1.5.1 - Solution Found
Thelin90 opened a new issue, #20428:
URL: https://github.com/apache/superset/issues/20428
I am currently deploying the latest version of superset in my local k8s (helm). I have added trino package, but also tried sqlalchemy-trino . And I can see it gets installed, all fine.
Superset boots up and all look fine.
I have a presto - trino deployment running in local k8s (coordinator + hive metastore). For the presto-trino deployment I am running, I am not using any user/password, since I just want to verify the functionality.
When I connect to trino, the database connection is OK! Works fine no worries.
I can then extract metadata information around catalog and schema. But when I try to display data, I get this weird error logs from superset:
```bash
Traceback (most recent call last):
File "/app/superset/sql_lab.py", line 275, in execute_sql_statement
db_engine_spec.handle_cursor(cursor, query, session)
File "/app/superset/db_engine_specs/presto.py", line 970, in handle_cursor
polled = cursor.poll()
AttributeError: 'Cursor' object has no attribute 'poll'
```
#### How to reproduce the bug
1. Run a `presto-trino` cluster with version >= `385 `. Then install `superset` via docs: https://superset.apache.org/docs/installation/running-on-kubernetes/
2. Use official `superset helm` with `image tag`, and modify to install `sqlalchemy-trino`, I have used following `values.yaml`:
```yaml
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Default values for superset.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
replicaCount: 1
# User ID directive. This user must have enough permissions to run the bootstrap script
# Runn containers as root is not recommended in production. Change this to another UID - e.g. 1000 to be more secure
runAsUser: 0
# Create custom service account for Superset. If create: true and name is not provided, superset.fullname will be used.
# serviceAccountName: superset
serviceAccount:
create: false
# Install additional packages and do any other bootstrap configuration in this script
# For production clusters it's recommended to build own image with this step done in CI
bootstrapScript: |
#!/bin/bash
rm -rf /var/lib/apt/lists/* && \
pip install \
psycopg2-binary==2.9.1 \
--upgrade trino \
--upgrade pyhive \
redis==3.5.3 && \
if [ ! -f ~/bootstrap ]; then echo "Running Superset with uid {{ .Values.runAsUser }}" > ~/bootstrap; fi
## The name of the secret which we will use to generate a superset_config.py file
## Note: this secret must have the key superset_config.py in it and can include other files as well
##
configFromSecret: '{{ template "superset.fullname" . }}-config'
## The name of the secret which we will use to populate env vars in deployed pods
## This can be useful for secret keys, etc.
##
envFromSecret: '{{ template "superset.fullname" . }}-env'
## This can be a list of template strings
envFromSecrets: []
## Extra environment variables that will be passed into pods
##
extraEnv: {}
# Extend timeout to allow long running queries.
# GUNICORN_TIMEOUT: 300
# OAUTH_HOME_DOMAIN: ..
# # If a whitelist is not set, any address that can use your OAuth2 endpoint will be able to login.
# # this includes any random Gmail address if your OAuth2 Web App is set to External.
# OAUTH_WHITELIST_REGEX: ...
## Extra environment variables in RAW format that will be passed into pods
##
extraEnvRaw: []
# Load DB password from other secret (e.g. for zalando operator)
# - name: DB_PASS
# valueFrom:
# secretKeyRef:
# name: superset.superset-postgres.credentials.postgresql.acid.zalan.do
# key: password
## Extra environment variables to pass as secrets
##
extraSecretEnv: {}
# MAPBOX_API_KEY: ...
# # Google API Keys: https://console.cloud.google.com/apis/credentials
# GOOGLE_KEY: ...
# GOOGLE_SECRET: ...
extraConfigs: {}
# import_datasources.yaml: |
# databases:
# - allow_file_upload: true
# allow_ctas: true
# allow_cvas: true
# database_name: example-db
# extra: "{\r\n \"metadata_params\": {},\r\n \"engine_params\": {},\r\n \"\
# metadata_cache_timeout\": {},\r\n \"schemas_allowed_for_file_upload\": []\r\n\
# }"
# sqlalchemy_uri: example://example-db.local
# tables: []
extraSecrets: {}
extraVolumes: []
# - name: customConfig
# configMap:
# name: '{{ template "superset.fullname" . }}-custom-config'
# - name: additionalSecret
# secret:
# secretName: my-secret
# defaultMode: 0600
extraVolumeMounts: []
# - name: customConfig
# mountPath: /mnt/config
# readOnly: true
# - name: additionalSecret:
# mountPath: /mnt/secret
# A dictionary of overrides to append at the end of superset_config.py - the name does not matter
# WARNING: the order is not guaranteed
configOverrides: {}
# extend_timeout: |
# # Extend timeout to allow long running queries.
# SUPERSET_WEBSERVER_TIMEOUT = ...
# enable_oauth: |
# from flask_appbuilder.security.manager import (AUTH_DB, AUTH_OAUTH)
# AUTH_TYPE = AUTH_OAUTH
# OAUTH_PROVIDERS = [
# {
# "name": "google",
# "whitelist": [ os.getenv("OAUTH_WHITELIST_REGEX", "") ],
# "icon": "fa-google",
# "token_key": "access_token",
# "remote_app": {
# "client_id": os.environ.get("GOOGLE_KEY"),
# "client_secret": os.environ.get("GOOGLE_SECRET"),
# "api_base_url": "https://www.googleapis.com/oauth2/v2/",
# "client_kwargs": {"scope": "email profile"},
# "request_token_url": None,
# "access_token_url": "https://accounts.google.com/o/oauth2/token",
# "authorize_url": "https://accounts.google.com/o/oauth2/auth",
# "authorize_params": {"hd": os.getenv("OAUTH_HOME_DOMAIN", "")}
# }
# }
# ]
# # Map Authlib roles to superset roles
# AUTH_ROLE_ADMIN = 'Admin'
# AUTH_ROLE_PUBLIC = 'Public'
# # Will allow user self registration, allowing to create Flask users from Authorized User
# AUTH_USER_REGISTRATION = True
# # The default user self registration role
# AUTH_USER_REGISTRATION_ROLE = "Admin"
# secret: |
# # Generate your own secret key for encryption. Use openssl rand -base64 42 to generate a good key
# SECRET_KEY = 'YOUR_OWN_RANDOM_GENERATED_SECRET_KEY'
# Same as above but the values are files
configOverridesFiles: {}
# extend_timeout: extend_timeout.py
# enable_oauth: enable_oauth.py
configMountPath: "/app/pythonpath"
extraConfigMountPath: "/app/configs"
image:
repository: apache/superset
tag: latest
pullPolicy: IfNotPresent
imagePullSecrets: []
initImage:
repository: busybox
tag: latest
pullPolicy: IfNotPresent
service:
type: ClusterIP
port: 8088
annotations: {}
# cloud.google.com/load-balancer-type: "Internal"
loadBalancerIP: null
ingress:
enabled: false
# ingressClassName: nginx
annotations: {}
# kubernetes.io/tls-acme: "true"
## Extend timeout to allow long running queries.
# nginx.ingress.kubernetes.io/proxy-connect-timeout: "300"
# nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
# nginx.ingress.kubernetes.io/proxy-send-timeout: "300"
path: /
pathType: ImplementationSpecific
hosts:
- chart-example.local
tls: []
# - secretName: chart-example-tls
# hosts:
# - chart-example.local
resources: {}
# We usually recommend not to specify default resources and to leave this as a conscious
# choice for the user. This also increases chances charts run on environments with little
# resources, such as Minikube. If you do want to specify resources, uncomment the following
# lines, adjust them as necessary, and remove the curly braces after 'resources:'.
# The limits below will apply to all Superset components. To set individual resource limitations refer to the pod specific values below.
# The pod specific values will overwrite anything that is set here.
# limits:
# cpu: 100m
# memory: 128Mi
# requests:
# cpu: 100m
# memory: 128Mi
##
## Custom hostAliases for all superset pods
## https://kubernetes.io/docs/tasks/network/customize-hosts-file-for-pods/
hostAliases: []
# - hostnames:
# - nodns.my.lan
# ip: 18.27.36.45
##
## Superset node configuration
supersetNode:
command:
- "/bin/sh"
- "-c"
- ". {{ .Values.configMountPath }}/superset_bootstrap.sh; /usr/bin/run-server.sh"
connections:
# Change in case of bringing your own redis and then also set redis.enabled:false
redis_host: '{{ template "superset.fullname" . }}-redis-headless'
# redis_password: superset
redis_port: "6379"
# You need to change below configuration incase bringing own PostgresSQL instance and also set postgresql.enabled:false
db_host: '{{ template "superset.fullname" . }}-postgresql'
db_port: "5432"
db_user: superset
db_pass: superset
db_name: superset
env: {}
forceReload: false # If true, forces deployment to reload on each upgrade
initContainers:
- name: wait-for-postgres
image: "{{ .Values.initImage.repository }}:{{ .Values.initImage.tag }}"
imagePullPolicy: "{{ .Values.initImage.pullPolicy }}"
envFrom:
- secretRef:
name: '{{ tpl .Values.envFromSecret . }}'
command: [ "/bin/sh", "-c", "until nc -zv $DB_HOST $DB_PORT -w1; do echo 'waiting for db'; sleep 1; done" ]
## Annotations to be added to supersetNode deployment
deploymentAnnotations: {}
## Annotations to be added to supersetNode pods
podAnnotations: {}
## Labels to be added to supersetNode pods
podLabels: {}
# Resource settings for the supersetNode pods - these settings overwrite might existing values from the global resources object defined above.
resources: {}
# limits:
# cpu: 100m
# memory: 128Mi
# requests:
# cpu: 100m
# memory: 128Mi
##
## Superset worker configuration
supersetWorker:
command:
- "/bin/sh"
- "-c"
- ". {{ .Values.configMountPath }}/superset_bootstrap.sh; celery --app=superset.tasks.celery_app:app worker"
forceReload: false # If true, forces deployment to reload on each upgrade
initContainers:
- name: wait-for-postgres
image: "{{ .Values.initImage.repository }}:{{ .Values.initImage.tag }}"
imagePullPolicy: "{{ .Values.initImage.pullPolicy }}"
envFrom:
- secretRef:
name: '{{ tpl .Values.envFromSecret . }}'
command: [ "/bin/sh", "-c", "until nc -zv $DB_HOST $DB_PORT -w1; do echo 'waiting for db'; sleep 1; done" ]
## Annotations to be added to supersetWorker deployment
deploymentAnnotations: {}
## Annotations to be added to supersetWorker pods
podAnnotations: {}
## Labels to be added to supersetWorker pods
podLabels: {}
# Resource settings for the supersetWorker pods - these settings overwrite might existing values from the global resources object defined above.
resources: {}
# limits:
# cpu: 100m
# memory: 128Mi
# requests:
# cpu: 100m
# memory: 128Mi
##
## Superset beat configuration (to trigger scheduled jobs like reports)
supersetCeleryBeat:
# This is only required if you intend to use alerts and reports
enabled: false
command:
- "/bin/sh"
- "-c"
- ". {{ .Values.configMountPath }}/superset_bootstrap.sh; celery --app=superset.tasks.celery_app:app beat --pidfile /tmp/celerybeat.pid --schedule /tmp/celerybeat-schedule"
forceReload: false # If true, forces deployment to reload on each upgrade
initContainers:
- name: wait-for-postgres
image: "{{ .Values.initImage.repository }}:{{ .Values.initImage.tag }}"
imagePullPolicy: "{{ .Values.initImage.pullPolicy }}"
envFrom:
- secretRef:
name: '{{ tpl .Values.envFromSecret . }}'
command: [ "/bin/sh", "-c", "until nc -zv $DB_HOST $DB_PORT -w1; do echo 'waiting for db'; sleep 1; done" ]
## Annotations to be added to supersetCeleryBeat deployment
deploymentAnnotations: {}
## Annotations to be added to supersetCeleryBeat pods
podAnnotations: {}
## Labels to be added to supersetCeleryBeat pods
podLabels: {}
# Resource settings for the CeleryBeat pods - these settings overwrite might existing values from the global resources object defined above.
resources: {}
# limits:
# cpu: 100m
# memory: 128Mi
# requests:
# cpu: 100m
# memory: 128Mi
##
## Init job configuration
init:
# Configure resources
# Warning: fab command consumes a lot of ram and can
# cause the process to be killed due to OOM if it exceeds limit
# Make sure you are giving a strong password for the admin user creation( else make sure you are changing after setup)
# Also change the admin email to your own custom email.
resources: {}
# limits:
# cpu:
# memory:
# requests:
# cpu:
# memory:
command:
- "/bin/sh"
- "-c"
- ". {{ .Values.configMountPath }}/superset_bootstrap.sh; . {{ .Values.configMountPath }}/superset_init.sh"
enabled: true
loadExamples: false
createAdmin: true
adminUser:
username: admin
firstname: Superset
lastname: Admin
email: admin@superset.com
password: admin
initContainers:
- name: wait-for-postgres
image: "{{ .Values.initImage.repository }}:{{ .Values.initImage.tag }}"
imagePullPolicy: "{{ .Values.initImage.pullPolicy }}"
envFrom:
- secretRef:
name: '{{ tpl .Values.envFromSecret . }}'
command: [ "/bin/sh", "-c", "until nc -zv $DB_HOST $DB_PORT -w1; do echo 'waiting for db'; sleep 1; done" ]
initscript: |-
#!/bin/sh
set -eu
echo "Upgrading DB schema..."
superset db upgrade
echo "Initializing roles..."
superset init
{{ if .Values.init.createAdmin }}
echo "Creating admin user..."
superset fab create-admin \
--username {{ .Values.init.adminUser.username }} \
--firstname {{ .Values.init.adminUser.firstname }} \
--lastname {{ .Values.init.adminUser.lastname }} \
--email {{ .Values.init.adminUser.email }} \
--password {{ .Values.init.adminUser.password }} \
|| true
{{- end }}
{{ if .Values.init.loadExamples }}
echo "Loading examples..."
superset load_examples
{{- end }}
if [ -f "{{ .Values.extraConfigMountPath }}/import_datasources.yaml" ]; then
echo "Importing database connections.... "
superset import_datasources -p {{ .Values.extraConfigMountPath }}/import_datasources.yaml
fi
## Annotations to be added to init job pods
podAnnotations: {}
##
## Configuration values for the postgresql dependency.
## ref: https://github.com/kubernetes/charts/blob/master/stable/postgresql/README.md
postgresql:
##
## Use the PostgreSQL chart dependency.
## Set to false if bringing your own PostgreSQL.
enabled: true
## Authentication parameters
auth:
## The name of an existing secret that contains the postgres password.
existingSecret:
## PostgreSQL name for a custom user to create
username: superset
## PostgreSQL password for the custom user to create. Ignored if `auth.existingSecret` with key `password` is provided
password: superset
## PostgreSQL name for a custom database to create
database: superset
## PostgreSQL Primary parameters
primary:
##
## Persistent Volume Storage configuration.
## ref: https://kubernetes.io/docs/user-guide/persistent-volumes
persistence:
##
## Enable PostgreSQL persistence using Persistent Volume Claims.
enabled: true
##
## Persistant class
# storageClass: classname
##
## Access modes:
accessModes:
- ReadWriteOnce
## PostgreSQL port
service:
ports:
postgresql: "5432"
## Configuration values for the Redis dependency.
## ref: https://github.com/bitnami/charts/blob/master/bitnami/redis
## More documentation can be found here: https://artifacthub.io/packages/helm/bitnami/redis
redis:
##
## Use the redis chart dependency.
##
## If you are bringing your own redis, you can set the host in supersetNode.connections.redis_host
##
## Set to false if bringing your own redis.
enabled: true
##
## Set architecture to standalone/replication
architecture: standalone
##
## Auth configuration:
##
auth:
## Enable password authentication
enabled: false
## The name of an existing secret that contains the redis password.
existingSecret: ""
## Name of the key containing the secret.
existingSecretKey: ""
## Redis password
password: superset
##
## Master configuration
##
master:
##
## Image configuration
# image:
##
## docker registry secret names (list)
# pullSecrets: nil
##
## Configure persistance
persistence:
##
## Use a PVC to persist data.
enabled: false
##
## Persistant class
# storageClass: classname
##
## Access mode:
accessModes:
- ReadWriteOnce
nodeSelector: {}
tolerations: []
affinity: {}
```
### Expected results
Being able to query `delta.io` table via `presto-trino` deployment.
### Actual results
No output.
### Root Cause To Problem
There was a chance to the codebase via this commit:
https://github.com/apache/superset/commit/b08e21efd906d13994414b39bfa7f6e98466d4cb
### Solution
After downgrading to `1.5.0`, querying works, so the change from:
`BaseEngineSpec` -> `PrestoEngineSpec` is causing the issue.
#### Screenshots
When troubles:
<img width="526" alt="image (3)" src="https://user-images.githubusercontent.com/19892351/174393160-e07de6f9-20d8-4991-8fa8-de568e1c1225.png">
<img width="941" alt="image (2)" src="https://user-images.githubusercontent.com/19892351/174393166-893c0145-74f9-4d04-b16a-04baffcc0242.png">
<img width="1413" alt="image (1)" src="https://user-images.githubusercontent.com/19892351/174393170-a148a914-71e6-4038-ad4a-b52a50612760.png">
When downgrading to `1.5.0`
<img width="949" alt="image" src="https://user-images.githubusercontent.com/19892351/174393758-bde585b6-f49d-4c1e-8724-dd399ec8101c.png">
### Environment
- browser type and version:
- superset version: `1.5.1`
- python version: `Python 3.8.12`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] Thelin90 commented on issue #20428: Presto-Trino Does Not Work Superset Version 1.5.1 - Solution Found
Posted by GitBox <gi...@apache.org>.
Thelin90 commented on issue #20428:
URL: https://github.com/apache/superset/issues/20428#issuecomment-1159668567
@quguangyu0321 slack conversation here: https://apache-superset.slack.com/archives/C015WAZL0KH/p1655449901353689
I will be trying to raise a `PR` to resolve this now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] Thelin90 commented on issue #20428: Presto-Trino Does Not Work Superset Version 1.5.1 - Solution Found
Posted by GitBox <gi...@apache.org>.
Thelin90 commented on issue #20428:
URL: https://github.com/apache/superset/issues/20428#issuecomment-1160052285
Closing issue now resolved!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] quguangyu0321 commented on issue #20428: Presto-Trino Does Not Work Superset Version 1.5.1 - Solution Found
Posted by GitBox <gi...@apache.org>.
quguangyu0321 commented on issue #20428:
URL: https://github.com/apache/superset/issues/20428#issuecomment-1159475545
thank you, I also had the same problem in latest version superset. although I can read metadata of Trino, but I can not query it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] Thelin90 closed issue #20428: Presto-Trino Does Not Work Superset Version 1.5.1 - Solution Found
Posted by GitBox <gi...@apache.org>.
Thelin90 closed issue #20428: Presto-Trino Does Not Work Superset Version 1.5.1 - Solution Found
URL: https://github.com/apache/superset/issues/20428
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org