[GitHub] abrenaut commented on issue #5447: Gunicorn workers running blocking calls

[GitBox <gi...@apache.org>]

abrenaut commented on issue #5447: Gunicorn workers running blocking calls
URL: https://github.com/apache/incubator-superset/issues/5447#issuecomment-407216994
 
 
   I reproduced locally on the master branch running superset with the following command
   
   ```gunicorn -w 1 -k gevent --timeout 120 -b 0.0.0.0:8088 --limit-request-line 0 --limit-request-field_size 0 superset:app```
   
   If I load a chart based on a query that takes a long time to run, all the request I try to make while the chart is loading are hanging.
   
   I managed to work around that by configuring the pg driver to run in a non-blocking style in superset_config.py (see http://initd.org/psycopg/docs/advanced.html#support-for-coroutine-libraries).
   
   ```import psycopg2
   from psycopg2 import extensions
   
   from gevent.socket import wait_read, wait_write
   
   def make_psycopg_green():
       """Configure Psycopg to be used with gevent in non-blocking way."""
       if not hasattr(extensions, 'set_wait_callback'):
           raise ImportError(
               "support for coroutines not available in this Psycopg version (%s)"
               % psycopg2.__version__)
   
       extensions.set_wait_callback(gevent_wait_callback)
   
   def gevent_wait_callback(conn, timeout=None):
       """A wait callback useful to allow gevent to work with Psycopg."""
       while 1:
           state = conn.poll()
           if state == extensions.POLL_OK:
               break
           elif state == extensions.POLL_READ:
               wait_read(conn.fileno(), timeout=timeout)
           elif state == extensions.POLL_WRITE:
               wait_write(conn.fileno(), timeout=timeout)
           else:
               raise psycopg2.OperationalError(
                   "Bad result from poll: %r" % state)
   
   make_psycopg_green()```
   
   I still have an issue when loading multiple charts (6+) with long running queries at the same time and I'm not quite sure what's causing the workers to be blocked.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org