You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "aavanzyl (via GitHub)" <gi...@apache.org> on 2024/04/02 07:29:12 UTC

[I] Sqllab stuck executing a query [superset]

aavanzyl opened a new issue, #27850:
URL: https://github.com/apache/superset/issues/27850

   ### Bug description
   
   When using SQL Lab to execute a query such as `SELECT * from claims.claim limit 1`, the UI shows `Pending` and never starts increasing the execution time. 
   
   The following is logged in the browser:
   
   Execute: 
   `
   {
       "query": {
           "changed_on": "2024-04-02T07:16:18.493787",
           "dbId": 1,
           "db": "Amazon Redshift (Prod)",
           "endDttm": null,
           "errorMessage": null,
           "executedSql": null,
           "id": "9E6eWGcfB",
           "queryId": 26748,
           "limit": 1,
           "limitingFactor": "QUERY",
           "progress": 0,
           "rows": null,
           "schema": null,
           "ctas": false,
           "serverId": 26748,
           "sql": "SELECT * from claims.claim limit 1\n",
           "sqlEditorId": "1773",
           "startDttm": 1712042178240.671000,
           "state": "pending",
           "tab": "Untitled Query 1",
           "tempSchema": null,
           "tempTable": null,
           "userId": 1,
           "user": "Anton van Zyl",
           "resultsKey": null,
           "trackingUrl": null,
           "extra": {}
       }
   }
   `
   
   We get multiple updated since messages back such as:
   `{
     "result": [
       {
         "changed_on": "2024-04-02T07:16:18.242144",
         "ctas": false,
         "db": "Amazon Redshift (Prod)",
         "dbId": 1,
         "endDttm": null,
         "errorMessage": null,
         "executedSql": null,
         "extra": {},
         "id": "9E6eWGcfB",
         "limit": 1000,
         "limitingFactor": "UNKNOWN",
         "progress": 0,
         "queryId": 26748,
         "resultsKey": null,
         "rows": null,
         "schema": null,
         "serverId": 26748,
         "sql": "SELECT * from claims.claim limit 1\n",
         "sqlEditorId": "1773",
         "startDttm": "1712042178240.671000",
         "state": "pending",
         "tab": "Untitled Query 1",
         "tempSchema": null,
         "tempTable": null,
         "trackingUrl": null,
         "user": "Anton van Zyl",
         "userId": 1
       }
     ]
   }
   `
   
   But never receive and `/results` call back. 
   My current remedy is to restart the instance and cache, which seems to intermittently solve the issue. 
   This issue only started recently, with us updating to 3.1.0 and then 3.1.1. 
   
   I have checked the logs, nothing pops up as out of the ordinary. 
   
   ### How to reproduce the bug
   
   Go to SQL Lab and run any query will result in the query being stuck in `Pending` 
   
   ### Screenshots/recordings
   
   The current UI state being shown to the end user:
   ![Screenshot 2024-04-02 at 07 46 55](https://github.com/apache/superset/assets/23451086/7480b0af-ba84-4b58-a77b-10256fe54c4d)
   
   
   ### Superset version
   
   3.1.1
   
   ### Python version
   
   I don't know
   
   ### Node version
   
   I don't know
   
   ### Browser
   
   Chrome
   
   ### Additional context
   
   We are running the latest 3.1.1 superset from a docker container hosted on ECS in AWS. The instance size has 12gigs of memory and connects to a Redshift database. 
   Superset Caching is set up with Redis 7.1.0 running on elasticache in AWS. 
   
   ### Checklist
   
   - [X] I have searched Superset docs and Slack and didn't find a solution to my problem.
   - [X] I have searched the GitHub issue tracker and didn't find a similar bug report.
   - [X] I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org

Re: [I] Sqllab stuck executing a query [superset]

Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on issue #27850:
URL: https://github.com/apache/superset/issues/27850#issuecomment-2168294353

   @aavanzyl glad to see the issue closed, but curious what happened!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org

Re: [I] Sqllab stuck executing a query [superset]

Posted by "aavanzyl (via GitHub)" <gi...@apache.org>.
aavanzyl commented on issue #27850:
URL: https://github.com/apache/superset/issues/27850#issuecomment-2060557307

   Hi guys, unfortunately after 6 days of stability, the issue has returned. This might be related to load in some way. 
   The behaviour explained [above](https://github.com/apache/superset/issues/27850#issuecomment-2044267335) seems better however, I no longer see empty execute being returned
   
   What can I provide from my side to assist in solving the issue? 
   
   superset_config.py:
   ```
   import logging
   import os
   from datetime import timedelta
   from typing import Optional
   
   from cachelib.file import FileSystemCache
   from celery.schedules import crontab
   from flask_caching.backends.rediscache import RedisCache
   from flask_appbuilder.security.manager import AUTH_OAUTH
   from s3cache.s3cache import S3Cache
   
   
   logging.basicConfig()
   logging.getLogger().setLevel(logging.INFO)
   logger = logging.getLogger()
   
   
   def get_env_variable(var_name: str, default: Optional[str] = None) -> str:
       """Get the environment variable or raise exception."""
       try:
           return os.environ[var_name]
       except KeyError:
           if default is not None:
               return default
           else:
               error_msg = "The environment variable {} was missing, abort...".format(
                   var_name
               )
               raise EnvironmentError(error_msg)
   
   
   SECRET_KEY = get_env_variable("SECRET_KEY")
   
   DATABASE_DIALECT = get_env_variable("DATABASE_DIALECT")
   DATABASE_USER = get_env_variable("DATABASE_USER")
   DATABASE_PASSWORD = get_env_variable("DATABASE_PASSWORD")
   DATABASE_HOST = get_env_variable("DATABASE_HOST")
   DATABASE_PORT = get_env_variable("DATABASE_PORT")
   DATABASE_DB = get_env_variable("DATABASE_DB")
   
   # The SQLAlchemy connection string.
   SQLALCHEMY_DATABASE_URI = "%s://%s:%s@%s:%s/%s" % (
       DATABASE_DIALECT,
       DATABASE_USER,
       DATABASE_PASSWORD,
       DATABASE_HOST,
       DATABASE_PORT,
       DATABASE_DB,
   )
   
   REDIS_HOST = get_env_variable("REDIS_HOST")
   REDIS_PORT = get_env_variable("REDIS_PORT")
   REDIS_CELERY_DB = get_env_variable("REDIS_CELERY_DB", "0")
   REDIS_RESULTS_DB = get_env_variable("REDIS_RESULTS_DB", "1")
   FILTER_STATE_CACHE_DB = get_env_variable("FILTER_STATE_CACHE_DB", "2")
   EXPLORE_FORM_DATA_CACHE_DB = get_env_variable("EXPLORE_FORM_DATA_CACHE_DB", "3")
   RATE_LIMIT_STORAGE_CACHE_DB = get_env_variable("RATE_LIMIT_STORAGE_CACHE_DB", "4")
   
   
   
   class CeleryConfig(object):
       broker_url = f"redis://{REDIS_HOST}:{REDIS_PORT}/{REDIS_CELERY_DB}"
       imports = (
           "superset.sql_lab",
           "superset.tasks",
           "superset.tasks.scheduler",
       )
       result_backend = f"redis://{REDIS_HOST}:{REDIS_PORT}/{REDIS_RESULTS_DB}"
       worker_log_level = "INFO"
       CELERYD_LOG_LEVEL = "INFO"
       # worker_prefetch_multiplier = 2
       worker_max_tasks_per_child = get_env_variable("MAX_TASKS_PER_CHILD")
       worker_max_memory_per_child = get_env_variable("MAX_MEMORY_PER_CHILD")
       task_acks_late = False
       task_annotations = {
           "sql_lab.get_sql_results": {
               "rate_limit": "100/s",
           },
           "email_reports.send": {
               "rate_limit": "1/s",
               "time_limit": 120,
               "soft_time_limit": 150,
               "ignore_result": True,
           },
       }
       beat_schedule = {
           "reports.scheduler": {
               "task": "reports.scheduler",
               "schedule": crontab(minute="*", hour="*"),
           }
       }
   
   
   CELERY_CONFIG = CeleryConfig
   
   # Caching
   S3_CACHE_BUCKET = get_env_variable("S3_CACHE_BUCKET")
   RESULTS_BACKEND = S3Cache(S3_CACHE_BUCKET, 'sql_lab_result')
   
   FILTER_STATE_CACHE_CONFIG = {
       "CACHE_TYPE": "RedisCache",
       "CACHE_DEFAULT_TIMEOUT": 86400,
       "CACHE_KEY_PREFIX": "superset_filter_cache",
       "CACHE_REDIS_URL": f"redis://{REDIS_HOST}:{REDIS_PORT}/{FILTER_STATE_CACHE_DB}",
   }
   
   EXPLORE_FORM_DATA_CACHE_CONFIG = {
       "CACHE_TYPE": "RedisCache",
       "CACHE_DEFAULT_TIMEOUT": 86400,
       "CACHE_KEY_PREFIX": "superset_form_data_cache",
       "CACHE_REDIS_URL": f"redis://{REDIS_HOST}:{REDIS_PORT}/{EXPLORE_FORM_DATA_CACHE_DB}",
   }
   
   RATELIMIT_STORAGE_URL = f"redis://{REDIS_HOST}:{REDIS_PORT}/{RATE_LIMIT_STORAGE_CACHE_DB}",
   
   FEATURE_FLAGS = {"ALERT_REPORTS": True, "ENABLE_TEMPLATE_PROCESSING": True}
   SLACK_API_TOKEN = get_env_variable("SLACK_API_TOKEN")
   ALERT_REPORTS_NOTIFICATION_DRY_RUN = False
   WEBDRIVER_BASEURL = "http://superset-app:8088"
   
   # The base URL for the email report hyperlinks.
   WEBDRIVER_BASEURL_USER_FRIENDLY = "https://superset-dw-ib.naked.insure"
   
   WEBDRIVER_TYPE = "firefox"
   WEBDRIVER_OPTION_ARGS = ["--headless"]
   
   MAPBOX_API_KEY = get_env_variable("MAPBOX_API_KEY")
   
   # EXTRA_CATEGORICAL_COLOR_SCHEMES is used for adding custom categorical color schemes
   EXTRA_CATEGORICAL_COLOR_SCHEMES = [
       {
           "id": "nakedPaletteCategorical",
           "description": "Naked colour palette used for categorical variables",
           "label": "Naked Colour Palette (Categorical)",
           "isDefault": False,
           "colors": [
               "#9cffc4",
               "#46d281",
               "#126556",
               "#004047",
               "#c2d5ff",
               "#547eed",
               "#384ba2",
               "#231ef8",
               "#a178f0",
               "#624ba6",
               "#fccaff",
               "#d263c0",
               "#692b63",
               "#44153b",
               "#ffd8c7",
               "#ffa15a",
               "#a95745",
               "#733843",
           ],
       }
   ]
   
   # EXTRA_SEQUENTIAL_COLOR_SCHEMES is used for adding custom sequential colour schemes
   EXTRA_SEQUENTIAL_COLOR_SCHEMES = [
       {
           "id": "nakedPaletteSequential",
           "description": "Naked colour palette used for sequential scales",
           "isDiverging": True,
           "label": "Naked Colour Palette (Sequential)",
           "isDefault": False,
           "colors": [
               "#9cffc4",
               "#46d281",
               "#35ae72",
               "#238964",
               "#126556",
               "#004047",
           ],
       }
   ]
   
   
   # SQL Lab settings
   SQLLAB_TIMEOUT = 600 # This is the limit after which queries will timeout (10 minutes)
   SQLLAB_CTAS_NO_LIMIT = False # A boolean flag that, when set to True, removes the row limit for CREATE TABLE AS queries.
   SQL_MAX_ROW = 50000 # The maximum number of rows that can be returned by any single SQL Lab query.
   
   SUPERSET_WEBSERVER_TIMEOUT = 180
   
   ENABLE_PROXY_FIX = True
   
   #
   # Optionally import superset_config_docker.py (which will have been included on
   # the PYTHONPATH) in order to allow for local settings to be overridden
   #
   try:
       import superset_config_docker
       from superset_config_docker import *  # noqa
   
       logger.info(
           f"Loaded your Docker configuration at " f"[{superset_config_docker.__file__}]"
       )
   except ImportError:
       logger.info("Using default Docker config...")
   
   ```
   
   requirement.txt:
   ```
   sqlalchemy-redshift==0.8.14
   snowflake-sqlalchemy==1.5.1
   psycopg2==2.9.9
   cryptography==42.0.5
   Authlib==1.3.0
   s3werkzeugcache==0.2.1
   gevent==24.2.1
   ```
   
   Dockerfile:
   ```
   FROM apache/superset:3.1.2-dev
   # We switch to root
   USER root
   
   ENV TINI_VERSION v0.19.0
   RUN curl --show-error --location --output /tini https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini-amd64
   RUN chmod +x /tini
   
   # Update the current image
   RUN 	apt-get update \
           && apt-get install -qq -y --no-install-recommends \
           sudo \
           make \
           unzip \
           curl \
           jq \
           wget \
           firefox-esr
           
   # Download aws dependencies
   RUN 	curl --silent --show-error https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip -o /tmp/awscliv2.zip && \
           curl --silent --show-error --location --output /tmp/amazon-ssm-agent.deb https://s3.eu-west-1.amazonaws.com/amazon-ssm-eu-west-1/latest/debian_amd64/amazon-ssm-agent.deb
   
   RUN 	unzip /tmp/awscliv2.zip
   RUN 	dpkg -i /tmp/amazon-ssm-agent.deb && \
           ./aws/install && \
           rm -rf /tmp/awscliv2.zip && \
           set -ex \
           && rm -rf /var/lib/apt/lists/* \
           && usermod -aG sudo superset \
           && echo "superset ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers
   
   # We install the Python interface for Redis
   COPY requirements.txt .
   RUN pip install -r requirements.txt
   
   # We add the superset_config.py file to the container
   COPY superset_config.py /app/
   
   # We tell Superset where to find it
   ENV SUPERSET_CONFIG_PATH /app/superset_config.py
   COPY /scripts/superset-entrypoint.sh /app/scripts/
   COPY /scripts/docker-bootstrap.sh /app/scripts/
   COPY /scripts/docker-init.sh /app/scripts/
   COPY /scripts/docker-entrypoint.sh /app/scripts/
   
   # We give permissions to different files
   RUN chmod +x /app/scripts/superset-entrypoint.sh
   RUN chmod +x /app/scripts/docker-entrypoint.sh
   RUN chmod +x /app/scripts/docker-init.sh
   RUN chmod +x /app/scripts/docker-bootstrap.sh
   
   # We switch back to the `superset` user
   USER superset
   ENTRYPOINT ["/tini", "-g", "--","/app/scripts/docker-entrypoint.sh"]
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org

Re: [I] Sqllab stuck executing a query [superset]

Posted by "aavanzyl (via GitHub)" <gi...@apache.org>.
aavanzyl commented on issue #27850:
URL: https://github.com/apache/superset/issues/27850#issuecomment-2049024184

   We have had 48 hours of stability with the issue not poping up. Its safe to say its solved. Thank you @michael-s-molina and team for the swift assistance. I appreciate the effort on your side. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org

Re: [I] Sqllab stuck executing a query [superset]

Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on issue #27850:
URL: https://github.com/apache/superset/issues/27850#issuecomment-2061189724

   @justinpark could you help?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org

Re: [I] Sqllab stuck executing a query [superset]

Posted by "aavanzyl (via GitHub)" <gi...@apache.org>.
aavanzyl commented on issue #27850:
URL: https://github.com/apache/superset/issues/27850#issuecomment-2044267335

   @michael-s-molina I have deployed 3.1.2 as asked. The issue currently is not showing, will monitor for a couple of days and let you know. Thank you for the response on my issue. 
   
   For further information, I can see that when clicking on run, it calls the `execute` API. This in turn takes a while to respond and was an empty response when broken. When things are working as expected, the `execute` API responds with a json object. 
   
   I also noticed that the `updated_since` API call started before the `execute` had a chance to receive a response, leading me to believe that this might contribute to the issue. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org

Re: [I] Sqllab stuck executing a query [superset]

Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on issue #27850:
URL: https://github.com/apache/superset/issues/27850#issuecomment-2044774982

   Thanks @aavanzyl. Let us know if things are working after your tests so we can close the issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org

Re: [I] Sqllab stuck executing a query [superset]

Posted by "aavanzyl (via GitHub)" <gi...@apache.org>.
aavanzyl closed issue #27850: Sqllab stuck executing a query
URL: https://github.com/apache/superset/issues/27850


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org

Re: [I] Sqllab stuck executing a query [superset]

Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on issue #27850:
URL: https://github.com/apache/superset/issues/27850#issuecomment-2034709023

   Hi @aavanzyl. Could you check if this problem happens in 3.1.2 which was released today? @justinpark for awareness.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org

Re: [I] Sqllab stuck executing a query [superset]

Posted by "aavanzyl (via GitHub)" <gi...@apache.org>.
aavanzyl closed issue #27850: Sqllab stuck executing a query
URL: https://github.com/apache/superset/issues/27850


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org