[jira] [Commented] (AIRFLOW-5642) sqlalchemy.exc.OperationalError happens when sql is too long, which cause scheduler processor will be exited

["Ash Berlin-Taylor (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-5642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16950955#comment-16950955 ] 

Ash Berlin-Taylor commented on AIRFLOW-5642:
--------------------------------------------

{noformat}
# This changes the batch size of queries in the scheduling main loop.
# If this is too high, SQL query performance may be impacted by one
# or more of the following:
#  - reversion to full table scan
#  - complexity of query predicate
#  - excessive locking
#
# Additionally, you may hit the maximum allowable query length for your db.
#
# Set this to 0 for no limit (not advised)
max_tis_per_query = 512
{noformat}

Perhaps you need to lower this config setting setting? (It's under the {{scheduler}} section)

> sqlalchemy.exc.OperationalError happens when sql is too long, which cause scheduler processor will be exited
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: AIRFLOW-5642
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-5642
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: scheduler
>    Affects Versions: 1.10.5
>            Reporter: chen xianxin
>            Priority: Minor
>
> h2. code
> https://github.com/apache/airflow/blob/v1-10-stable/airflow/jobs/scheduler_job.py#L1060-L1069
> h2. Error description
> {{if there is too many queued_tasks, sqlalchemy.exc.OperationalError will occur. details are as follow:}}
> {{<}}
> {{sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2006, "MySQL server has gone away (BrokenPipeError(32, 'Broken pipe'))")}}
>  [SQL: SELECT ...]
> (Background on this error at: [http://sqlalche.me/e/e3q8])
> {{> }}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)