[GitHub] [incubator-superset] Stephan3555 opened a new issue #8171: Celery worker support for Druid Queries

[GitBox <gi...@apache.org>]

Stephan3555 opened a new issue #8171: Celery worker support for Druid Queries
URL: https://github.com/apache/incubator-superset/issues/8171
 
 
   I set up a superset cluster consisting of superset web server, redis (as broker, result backend and cache) and celery workers. Everything works fine within SQL Lab: The celery worker receive the tasks, write in the result backend and fill the cache.
   
   On the other hand Druid queries via "Sources" --> "Datasources" are handled by the superset web server and not by the celery workers. Ist this normal behavior?
   
   Did i miss a configuration that forces this kind of queries to use the celery workers as well?
   
   ### Expected results
   
   Druid queries via "Sources" --> "Datasources" are handled by the celery workers
   
   ### Actual results
   
   Druid queries via "Sources" --> "Datasources" are handled by the superset server
   
   ### Environment
   
   (please complete the following information):
   
   - superset version: `0.28.1`
   - python version: `3.6.8`
   - doker image: amancevice/superset:0.28.1
   
   superset_config.py:
   
   ------------------------------
   import os
   
   from werkzeug.contrib.cache import RedisCache
   
   if 'SUPERSET_HOME' in os.environ:
       DATA_DIR = os.environ['SUPERSET_HOME']
   else:
       DATA_DIR = os.path.join(os.path.expanduser('~'), '.superset')
   
   
   LOG_FORMAT = "%(asctime)s:%(levelname)s:%(name)s:%(message)s"
   LOG_LEVEL = "DEBUG"
   
   ENABLE_TIME_ROTATE = False
   TIME_ROTATE_LOG_LEVEL = "DEBUG"
   FILENAME = os.path.join(DATA_DIR, "superset.log")
   ROLLOVER = "midnight"
   INTERVAL = 1
   BACKUP_COUNT = 30
   
   ROW_LIMIT = 50000
   VIZ_ROW_LIMIT = 5000
   FILTER_SELECT_ROW_LIMIT = 1000
   SQLALCHEMY_TRACK_MODIFICATIONS = True
   SUPERSET_WEBSERVER_TIMEOUT = 60
   
   QUERY_SEARCH_LIMIT = 1000
   
   POSTGRES_SERVER_URL = os.getenv('POSTGRES_SERVER_URL', '')
   POSTGRES_DB = os.getenv('POSTGRES_DB', '')
   POSTGRES_USER = os.getenv('POSTGRES_USER', '')
   POSTGRES_PASSWORD = os.getenv('POSTGRES_PASSWORD', '')
   
   SUPERSET_SQLALCHEMY_DATABASE_URI = "".join(['postgresql+psycopg2://', POSTGRES_USER, ':', POSTGRES_PASSWORD, '@', POSTGRES_SERVER_URL, '/', POSTGRES_DB])
   
   SQLALCHEMY_DATABASE_URI = SUPERSET_SQLALCHEMY_DATABASE_URI
   
   CACHE_CONFIG =  {
                      'CACHE_TYPE': 'redis',
                      'CACHE_DEFAULT_TIMEOUT': os.getenv('CACHE_DEFAULT_TIMEOUT', ''),
                      'CACHE_KEY_PREFIX': 'superset_cache',
                      'CACHE_REDIS_URL': os.getenv('CACHE_REDIS_URL', '')
                    }
   
   MAPBOX_API_KEY = os.environ.get('MAPBOX_API_KEY', '')
   
   
   class CeleryConfig(object):
       BROKER_URL = os.getenv('BROKER_URL', '')
       CELERY_IMPORTS = ('superset.sql_lab')
       CELERY_RESULT_BACKEND = os.getenv('CELERY_RESULT_BACKEND', '')
       CELERY_ANNOTATIONS = {'tasks.add': {'rate_limit': '10/s'}}
       CELERYD_TASK_SOFT_TIME_LIMIT = os.getenv('CELERYD_TASK_SOFT_TIME_LIMIT', '') 
       CELERYD_TASK_TIME_LIMIT = os.getenv('CELERYD_TASK_TIME_LIMIT', '') # 30 min
       CELERYD_MAX_TASKS_PER_CHILD = os.getenv('CELERYD_MAX_TASKS_PER_CHILD', '') 
       CELERYD_LOG_LEVEL = os.getenv('CELERYD_LOG_LEVEL', '')
       CELERYD_PREFETCH_MULTIPLIER = os.getenv('CELERYD_PREFETCH_MULTIPLIER', '')
       CELERY_ACKS_LATE = True
       CELERY_SEND_EVENTS = True
   CELERY_CONFIG = CeleryConfig
   RESULTS_BACKEND = RedisCache(
       host=os.getenv('RESULTS_BACKEND_HOST', ''),
       port=os.getenv('RESULTS_BACKEND_PORT', ''),
       key_prefix='superset_results'
   )
   
   ------------------------------
   
   Command to start
   
   - Server:
     gunicorn \
           -w 10 \
           -k gevent \
           --timeout 60 \
           -b  0.0.0.0:8088 \
           --limit-request-line 0 \
           --limit-request-field_size 0 \
           superset:app
   - worker:
     celery worker --app=superset.sql_lab:celery_app --pool=gevent -Ofair --task-events
   
   
   ### Checklist
   
   Make sure these boxes are checked before submitting your issue - thank you!
   
   - [x] I have checked the superset logs for python stacktraces and included it here as text if there are any.
   - [x] I have reproduced the issue with at least the latest released version of superset.
   - [x] I have checked the issue tracker for the same issue and I haven't found one similar.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org