You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by GitBox <gi...@apache.org> on 2021/04/29 13:38:52 UTC
[GitHub] [superset] thammanenitcs opened a new issue #14414: Connection Pool implementation to create DB sessions in superset
thammanenitcs opened a new issue #14414:
URL: https://github.com/apache/superset/issues/14414
Hi Team,
i am using superset0.36 version, have observed superset is creating more no.of db sessions when dashboards/reports are running continuously. Sometimes due to high load/no.of schedule jobs are running db sessions are breaking. I thought to see whether connection pool creation for superset will improve db sessions failures. i have followed below SIP as a reference.
https://github.com/apache/superset/issues/8574
i have followed the sqlalchemy documentation and used QueuePool to created connection pool when superset is establishing db session with help of "create_engine" method. Below are the connection pool parameters and file locations where i have updated in superset.
Documentation: https://docs.sqlalchemy.org/en/13/core/pooling.html#sqlalchemy.pool.QueuePool
Connection pool parameters:
filepath: superset/models/core.py [ function: get_sqla_engine ]
params["poolclass"] = QueuePool
params["pool_size"] = 20
params["max_overflow"] = 10
params["pool_recycle"]= 3600
params["pool_pre_ping"] = True
params["pool_use_lifo"] = True
-------------------------------------------------------------------
filepath: superset/migrations/env.py
engine = engine_from_config(
config.get_section(config.config_ini_section),
prefix="sqlalchemy.",
poolclass=pool.QueuePool,
)
I can these are the main functions where superset initiating for db sessions creations. there are couple of places where i have override nullpool parameter to false, since nullpoll is for disabling connection pool.
i also tried to see by varying the pool size but i can't see any difference after applying connection pool code. I need suggestion to whether i am going in right direction or not OR is there a way to test/see the connection pool usage at application level to make sure the new code customizations that i have done is working.
Please give some suggestions/ if there are any implementation done w.r.to this requirement in superset before.
Thanks for the attention and time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org