[GitHub] [incubator-superset] mistercrunch commented on issue #8574: [SIP-26] Proposal for Implementing Connection Pooling for Analytics Database Connections

[GitBox <gi...@apache.org>]

mistercrunch commented on issue #8574: [SIP-26] Proposal for Implementing Connection Pooling for Analytics Database Connections
URL: https://github.com/apache/incubator-superset/issues/8574#issuecomment-554241405
 
 
   There's a challenge here around the fact that each subprocess (gunicorn worker and celery worker) gets its own pool, and that each one of those can connect to multiple databases. Depending on whether you configure these things to use threads or workers (subprocesses), you can end up with a lot of connections very quickly that is vastly bigger than the number of **active** connections. One problem is that while you probably want to cap the number of connections to your db, you want do have a dynamic number of workers as you need more capacity. 
   
   There's also a challenge around the fact that threading and SQLAlchemy pools have intricate issues. There are endless stackoverflows documenting this.
   
   Another thought is that we may want to limit concurrency to analytics databases, but this approach is not achieving that in any way as there's no global state shared across server/workers. This would need to get handled as some sort of environment global variable (redis?) that would keep track of the number of active connections.
   
   What's the cost of spawning / destroying a connection? On the server / on the client? Milliseconds of CPU time? Worth it?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org