You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Eugene (JIRA)" <ji...@apache.org> on 2019/06/28 17:34:00 UTC
[jira] [Commented] (AIRFLOW-3548) Tasks instances doesn't reuse
mysql connection
[ https://issues.apache.org/jira/browse/AIRFLOW-3548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16875086#comment-16875086 ]
Eugene commented on AIRFLOW-3548:
---------------------------------
We you able to find anything? We are seeing replication lags because of I/O.
However, active connections look like this:
[root@***** netcount]# cat monitor-netstat-connect-3115.txt
1 airflow.worker.com
1 airflow.worker.com
2 airflow.worker.com
3 airflow.worker.com
1 airflow.worker.com
65 airflow.worker.com
105 airflow.webserver.com
117airflow.webserver.com
13 airflow.worker.com
9 airflow.worker.com
14airflow.worker.com
9 airflow.worker.com
10 airflow.worker.com
10 airflow.worker.com
and then when new set of tasks submites spikes to 200-300 per worker
> Tasks instances doesn't reuse mysql connection
> ----------------------------------------------
>
> Key: AIRFLOW-3548
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3548
> Project: Apache Airflow
> Issue Type: Improvement
> Components: worker
> Affects Versions: 1.10.1
> Reporter: Rami Darwish
> Priority: Minor
>
> Task instances keep opening a new connection to mysql every "job_heartbeat_sec" interval. Ideally, it should open 1 connection for the life span of the task until it finished executing. It seems to ignore sql_alchemy_pool_enabled = True
> We're using airflow 1.10.1, mysql 5.7, SQLAlchemy 1.1.18, Python 2.7.12
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)