You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by GitBox <gi...@apache.org> on 2021/04/29 13:38:52 UTC

[GitHub] [superset] thammanenitcs opened a new issue #14414: Connection Pool implementation to create DB sessions in superset

thammanenitcs opened a new issue #14414:
URL: https://github.com/apache/superset/issues/14414


   Hi Team,
   
   i am using superset0.36 version, have observed superset is creating more no.of db sessions when dashboards/reports are running continuously. Sometimes due to high load/no.of schedule jobs are running db sessions are breaking. I thought to see whether connection pool creation for superset will improve db sessions failures. i have followed below SIP as a reference.
   
   https://github.com/apache/superset/issues/8574 
   
   i have followed the sqlalchemy documentation and used QueuePool to created connection pool when superset is establishing db session with help of "create_engine" method. Below are the connection pool parameters and file locations where i have updated in superset.
   
   Documentation: https://docs.sqlalchemy.org/en/13/core/pooling.html#sqlalchemy.pool.QueuePool
   
   Connection pool parameters:
   filepath: superset/models/core.py [ function: get_sqla_engine ]
           params["poolclass"] = QueuePool
           params["pool_size"] = 20
           params["max_overflow"] = 10
           params["pool_recycle"]= 3600
           params["pool_pre_ping"] = True
           params["pool_use_lifo"] = True
   -------------------------------------------------------------------
   filepath: superset/migrations/env.py
   engine = engine_from_config(
           config.get_section(config.config_ini_section),
           prefix="sqlalchemy.",
           poolclass=pool.QueuePool,
       )
   
   I can these are the main functions where superset initiating for db sessions creations. there are couple of places where i have override nullpool parameter to false, since nullpoll is for disabling connection pool.
   
   i also tried to see by varying the pool size but i can't see any difference after applying connection pool code. I need suggestion to whether i am going in right direction or not  OR is there a way to test/see the connection pool usage at application level to make sure the new code customizations that i have done is working.
   
   Please give some suggestions/ if there are any implementation done w.r.to this requirement in superset before.
   
   Thanks for the attention and time. 
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org