You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/10/26 13:10:32 UTC
[GitHub] [airflow] vchiapaikeo opened a new issue, #27296: Task completes work but ends up failing due to a lock wait timeout exceeded error and does not honor retries
vchiapaikeo opened a new issue, #27296:
URL: https://github.com/apache/airflow/issues/27296
### Apache Airflow version
2.4.2
### What happened
We are load testing our Airflow environment with the KubernetesExecutor and will observe lock wait timeouts happening every so often. This happens after the job itself completes (in this case, sleeping for a random amount of time) and marks the task as a failure without performing any retries.
```
[2022-10-26, 00:04:07 UTC] {taskinstance.py:1165} INFO - Dependencies all met for <TaskInstance: factory_load_test_v53_31.longrun2 scheduled__2022-10-25T00:00:00+00:00 [queued]>
[2022-10-26, 00:04:07 UTC] {taskinstance.py:1165} INFO - Dependencies all met for <TaskInstance: factory_load_test_v53_31.longrun2 scheduled__2022-10-25T00:00:00+00:00 [queued]>
[2022-10-26, 00:04:07 UTC] {taskinstance.py:1362} INFO -
--------------------------------------------------------------------------------
[2022-10-26, 00:04:07 UTC] {taskinstance.py:1363} INFO - Starting attempt 1 of 2
[2022-10-26, 00:04:07 UTC] {taskinstance.py:1364} INFO -
--------------------------------------------------------------------------------
[2022-10-26, 00:04:07 UTC] {taskinstance.py:1383} INFO - Executing <Task(BashOperator): longrun2> on 2022-10-25 00:00:00+00:00
[2022-10-26, 00:04:07 UTC] {standard_task_runner.py:55} INFO - Started process 162 to run task
[2022-10-26, 00:04:07 UTC] {standard_task_runner.py:82} INFO - Running: ['airflow', 'tasks', 'run', 'factory_load_test_v53_31', 'longrun2', 'scheduled__2022-10-25T00:00:00+00:00', '--job-id', '56234', '--raw', '--subdir', 'DAGS_FOLDER/factory_load_test_v53.py', '--cfg-path', '/tmp/tmp82sxf8im']
[2022-10-26, 00:04:07 UTC] {standard_task_runner.py:83} INFO - Job 56234: Subtask longrun2
[2022-10-26, 00:04:08 UTC] {task_command.py:376} INFO - Running <TaskInstance: factory_load_test_v53_31.longrun2 scheduled__2022-10-25T00:00:00+00:00 [running]> on host factoryloadtestv5331longrun2-be2fa24c168a4614aace786ac9cb6e78
[2022-10-26, 00:04:09 UTC] {taskinstance.py:1590} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=gcp-data-platform
AIRFLOW_CTX_DAG_ID=factory_load_test_v53_31
AIRFLOW_CTX_TASK_ID=longrun2
AIRFLOW_CTX_EXECUTION_DATE=2022-10-25T00:00:00+00:00
AIRFLOW_CTX_TRY_NUMBER=1
AIRFLOW_CTX_DAG_RUN_ID=scheduled__2022-10-25T00:00:00+00:00
[2022-10-26, 00:04:09 UTC] {subprocess.py:63} INFO - Tmp dir root location:
/tmp
[2022-10-26, 00:04:09 UTC] {subprocess.py:75} INFO - Running command: ['/bin/bash', '-c', 'sleep 777']
[2022-10-26, 00:04:09 UTC] {subprocess.py:86} INFO - Output:
[2022-10-26, 00:17:06 UTC] {subprocess.py:97} INFO - Command exited with return code 0
[2022-10-26, 00:17:06 UTC] {datahub.py:122} ERROR - 'BashOperator' object has no attribute '_inlets'
[2022-10-26, 00:17:06 UTC] {datahub.py:123} INFO - Supressing error because graceful_exceptions is set
[2022-10-26, 00:17:06 UTC] {taskinstance.py:1401} INFO - Marking task as SUCCESS. dag_id=factory_load_test_v53_31, task_id=longrun2, execution_date=20221025T000000, start_date=20221026T000407, end_date=20221026T001706
[2022-10-26, 00:17:58 UTC] {standard_task_runner.py:100} ERROR - Failed to execute job 56234 for task longrun2 ((MySQLdb.OperationalError) (1205, 'Lock wait timeout exceeded; try restarting transaction')
[SQL: UPDATE dag_run SET last_scheduling_decision=%s WHERE dag_run.id = %s]
[parameters: (datetime.datetime(2022, 10, 26, 0, 3, 41, 951161), 8265)]
(Background on this error at: https://sqlalche.me/e/14/e3q8); 162)
[2022-10-26, 00:17:58 UTC] {local_task_job.py:164} INFO - Task exited with return code 1
```
<img width="1436" alt="image" src="https://user-images.githubusercontent.com/9200263/198032136-2f9c3e12-6ecd-4147-bd8b-2848e3689af3.png">
### What you think should happen instead
Updates to the dag_run table's last_scheduling_decision should be part of a db retry strategy (like `run_with_db_retries`):
https://github.com/apache/airflow/blob/777b57f0c6a8ca16df2b96fd17c26eab56b3f268/airflow/models/dagrun.py#L553
Additionally, an Airflow retry is not attempted, even though the task itself has retries.
### How to reproduce
Using MySQL 8 deployed via GCP CloudSQL with transaction isolation level set to READ_COMMITTED and a default innodb_lock_wait_timeout of 50s.
```
from airflow import DAG
from airflow.operators.bash import BashOperator
from airflow.operators.empty import EmptyOperator
from random import randint
DEFAULT_TASK_ARGS = {
"owner": "gcp-data-platform",
"start_date": "2022-09-26",
"retries": 1,
"retry_delay": 300,
}
dag_id_prefix = "factory_load_test_v53_"
def generate_dag(key):
with DAG(
schedule_interval="@daily",
max_active_runs=1,
max_active_tasks=5,
catchup=False,
dag_id=f"{dag_id_prefix}{key}",
default_args=DEFAULT_TASK_ARGS,
) as dag:
start = EmptyOperator(task_id="start")
long_running_task_1 = BashOperator(
task_id="longrun1",
bash_command=f"sleep {randint(0, 1800)}",
)
long_running_task_2 = BashOperator(
task_id="longrun2",
bash_command=f"sleep {randint(0, 1200)}",
)
start >> [long_running_task_1, long_running_task_2]
return dag
for key in range(0, 1000):
globals()[f"{dag_id_prefix}{key}"] = generate_dag(key)
```
### Operating System
Debian 11
### Versions of Apache Airflow Providers
apache-airflow-providers-google==8.4.0
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
KubernetesExecutor with the following configurations:
```
AIRFLOW__CORE__DEFAULT_POOL_TASK_SLOT_COUNT: "100000"
# DAG factories can take a long time to process. Bump from default 50s to
# 90s to ensure that we aren't stuck in timeout dag processor loops
AIRFLOW__CORE__DAG_FILE_PROCESSOR_TIMEOUT: "90"
# Reduces load on db with less frequent updates; we double health check
# threshold as well in line with fewer heartbeats
AIRFLOW__SCHEDULER__SCHEDULER_HEARTBEAT_SEC: "10"
# Need to keep job heartbeat at a high rate to avoid jobs being mistakened
# in queued state (even though they are actually running); 5s is the default
AIRFLOW__SCHEDULER__JOB_HEARTBEAT_SEC: "5"
# Lower from default 512 to 100 to reduce the load on the DB
AIRFLOW__SCHEDULER__MAX_TIS_PER_QUERY: "100"
# Bump from 2 to 4 to speed up dag processing per scheduler loop
AIRFLOW__SCHEDULER__PARSING_PROCESSES: "4"
# Interval in which to run adopt_or_reset_orphaned_tasks in scheduler_job.py
# which will externally mark SchedulerJobs as failed or LocalJob tasks to None
# Default is 300s. Increase to avoid orphaning tasks
AIRFLOW__SCHEDULER__ORPHANED_TASKS_CHECK_INTERVAL: "600"
AIRFLOW__KUBERNETES__WORKER_PODS_CREATION_BATCH_SIZE: "5"
# Allow pods up to 20 mins to schedule themselves - defaults to 5 mins
# During periods of high load (like midnight UTC), pods may be slow to schedule so allow
# them more time to do so
AIRFLOW__KUBERNETES__WORKER_PODS_PENDING_TIMEOUT: "1200"
AIRFLOW__KUBERNETES__WORKER_PODS_PENDING_TIMEOUT_BATCH_SIZE: "200"
# Num seconds to check for task instances stuck in `pending` status without
# a pod. Default is 120s but during periods of high load (like midnight UTC),
# this can actually be a long time so increase to 480s
AIRFLOW__KUBERNETES__WORKER_PODS_PENDING_TIMEOUT_CHECK_INTERVAL: "480"
# Num seconds to check for task instances stuck in `queued` status without
# a pod. Default is 60s but during periods of high load (like midnight UTC),
# this can actually be a long time so increase to 900s. Note, this check will
# attempt to put pods in queued state to scheduled state
AIRFLOW__KUBERNETES__WORKER_PODS_QUEUED_CHECK_INTERVAL: "900"
# Note that if no _request_timeout is specified, the kubernetes client will wait
# indefinitely for kubernetes api responses, which will cause the scheduler to hang.
# The timeout is specified as [connect timeout, read timeout]
AIRFLOW__KUBERNETES__KUBE_CLIENT_REQUEST_ARGS: '{"_request_timeout": [60,60]}'
```
### Anything else
With 1k DAGs starting a midnight, this happens approximately 2 every night.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on issue #27296: Task completes work but ends up failing due to a lock wait timeout exceeded error and does not honor retries
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on issue #27296:
URL: https://github.com/apache/airflow/issues/27296#issuecomment-1455101957
> The other thing I’d suggest is changing from REPEATABLE READ to READ COMMITTED. This change helped us significantly as well as the number of locks taken on select are greatly reduced.
It should be already set to READ COMMITTED, but only for mysql and mssql, and assume that user [do not change default PostgreSQL isolation](https://www.postgresql.org/docs/current/runtime-config-client.html#GUC-DEFAULT-TRANSACTION-ISOLATION) to other than READ COMMITED
https://github.com/apache/airflow/blob/c4f8e22227a3f445d47386799be504388513ce3f/airflow/settings.py#L337-L340
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vchiapaikeo commented on issue #27296: Task completes work but ends up failing due to a lock wait timeout exceeded error and does not honor retries
Posted by GitBox <gi...@apache.org>.
vchiapaikeo commented on issue #27296:
URL: https://github.com/apache/airflow/issues/27296#issuecomment-1297480661
Hey @potiuk - thanks for the response. I forgot to mention that we've already disabled this configuration:
```
AIRFLOW__SCHEDULER__SCHEDULE_AFTER_TASK_EXECUTION: "False"
```
That said, we have been starting to see more stability with our scheduler lately. Another config that I failed to provide on our end was this:
```
AIRFLOW__CORE__PARALLELISM: "0"
```
This seems to have caused the schedulers to take on more load than they could handle and also likely contributed to a greater likelihood of locks on the db.
We've adjusted parallelism to 500. The default of 32 seems too low for us and we've also noticed that more schedulers can also contribute to a higher likelihood of locking. Our plan is to launch with 3 schedulers (each with a parallelism of 500).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tanvn commented on issue #27296: Task completes work but ends up failing due to a lock wait timeout exceeded error and does not honor retries
Posted by "tanvn (via GitHub)" <gi...@apache.org>.
tanvn commented on issue #27296:
URL: https://github.com/apache/airflow/issues/27296#issuecomment-1455059928
Hi, may I know if there is anything else that should be taken into account regarding this issue?
Actually we are running our Airflow (2.5.0) environment with the KubernetesExecutor and lock wait timeouts are happening quite often too.
We are setting:
- parallelism = 1024
- max_active_tasks_per_dag = 16 (for some DAG, it is 32)
- max_active_runs_per_dag = 8
- max_tis_per_query = 1024
If setting parallelism to a high value is considered an issue, may I know the reason why?
We are using MySQL 8, transaction_isolation = REPEATABLE-READ, innodb_lock_wait_timeout=50s.
One difference is that after failing, the tasks are being retried.
We have not tried disabling `schedule_after_task_execution` yet, but as far as I notice, lock wait timeouts are happening in various cases, not only when running mini scheduler, but also when it tries deleting old records of `rendered_task_instance_fields` table. A related issue: https://github.com/apache/airflow/issues/29687
(however, we are not hitting deadlock errors but timeouts instead
```
sqlalchemy.exc.OperationalError: (MySQLdb.OperationalError) (1205, 'Lock wait timeout exceeded; try restarting transaction')
[SQL: DELETE FROM rendered_task_instance_fields WHERE rendered_task_instance_fields.dag_id = %s AND rendered_task_instance_fields.task_id = %s AND ((rendered_task_instance_fields.dag_id, rendered_task_instance_fields.task_id, rendered_task_instance_fields.run_id) NOT IN (SELECT anon_1.dag_id, anon_1.task_id, anon_1.run_id
FROM (SELECT DISTINCT rendered_task_instance_fields.dag_id AS dag_id, rendered_task_instance_fields.task_id AS task_id, rendered_task_instance_fields.run_id AS run_id, dag_run.execution_date AS execution_date
FROM rendered_task_instance_fields INNER JOIN dag_run ON rendered_task_instance_fields.dag_id = dag_run.dag_id AND rendered_task_instance_fields.run_id = dag_run.run_id
WHERE rendered_task_instance_fields.dag_id = %s AND rendered_task_instance_fields.task_id = %s ORDER BY dag_run.execution_date DESC
LIMIT %s) AS anon_1))]
[parameters: ('my_dag', 'my_task', 'my_dag', 'my_task', 30)]
```
Currently we have been running a crontab to kill long-sleeping connections to MySQL from Airflow and it seems to work but we have not found the root cause yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vchiapaikeo commented on issue #27296: Task completes work but ends up failing due to a lock wait timeout exceeded error and does not honor retries
Posted by GitBox <gi...@apache.org>.
vchiapaikeo commented on issue #27296:
URL: https://github.com/apache/airflow/issues/27296#issuecomment-1303703814
This stopped being an issue for us after reducing parallelism on our schedulers. Closing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tanvn commented on issue #27296: Task completes work but ends up failing due to a lock wait timeout exceeded error and does not honor retries
Posted by "tanvn (via GitHub)" <gi...@apache.org>.
tanvn commented on issue #27296:
URL: https://github.com/apache/airflow/issues/27296#issuecomment-1455083768
@potiuk @vchiapaikeo
Thank you both so much!
Let me discuss with my team and get back to you later.
By the way, IIRC, `airflow db clean` command currently does not work well with MySQL Db,
so I am considering using this https://github.com/teamclairvoyant/airflow-maintenance-dags/tree/master/db-cleanup instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vchiapaikeo commented on issue #27296: Task completes work but ends up failing due to a lock wait timeout exceeded error and does not honor retries
Posted by GitBox <gi...@apache.org>.
vchiapaikeo commented on issue #27296:
URL: https://github.com/apache/airflow/issues/27296#issuecomment-1292782722
Adding a bit more analysis here. I'm noticing the only place where a query like this (SQL: UPDATE dag_run SET last_scheduling_decision=%s WHERE dag_run.id = %s) would be run is here in `DagRun.update_state`:
https://github.com/apache/airflow/blob/2.4.2/airflow/models/dagrun.py#L516-L518
Specifically, `last_scheduling_decision` gets set here:
https://github.com/apache/airflow/blob/2.4.2/airflow/models/dagrun.py#L552
And I think the most likely place that `DagRun.update_state` is being called from is here in `SchedulerJob._schedule_dag_run`:
https://github.com/apache/airflow/blob/2.4.2/airflow/jobs/scheduler_job.py#L1242-L1246
https://github.com/apache/airflow/blob/2.4.2/airflow/jobs/scheduler_job.py#L1301
What I don't quite understand is if this is a call from SchedulerJob to update the dagrun, why are we seeing these logs on the worker pod? Is that because we're using KubernetesExecutor and the airflow worker pod itself is actually run as LocalExecutor? I also wonder what could be holding a lock on this same record for >50s...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #27296: Task completes work but ends up failing due to a lock wait timeout exceeded error and does not honor retries
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #27296:
URL: https://github.com/apache/airflow/issues/27296#issuecomment-1455062224
> Hi, may I know if there is anything else that should be taken into account regarding this issue? Actually we are running our Airflow (2.5.0) environment with the KubernetesExecutor and lock wait timeouts are happening quite often too. We are setting:
>
> * parallelism = 1024
> * max_active_tasks_per_dag = 16 (for some DAG, it is 32)
> * max_active_runs_per_dag = 8
> * max_tis_per_query = 1024
>
> If setting parallelism to a high value is considered an issue, may I know the reason why? We are using MySQL 8, transaction_isolation = REPEATABLE-READ, innodb_lock_wait_timeout=50s. One difference is that after failing, the tasks are being retried. We have not tried disabling `schedule_after_task_execution` yet, but as far as I notice, lock wait timeouts are happening in various cases, not only when running mini scheduler, but also when it tries deleting old records of `rendered_task_instance_fields` table. A related issue: #29687 (however, we are not hitting deadlock errors but timeouts instead
>
> ```
> sqlalchemy.exc.OperationalError: (MySQLdb.OperationalError) (1205, 'Lock wait timeout exceeded; try restarting transaction')
> [SQL: DELETE FROM rendered_task_instance_fields WHERE rendered_task_instance_fields.dag_id = %s AND rendered_task_instance_fields.task_id = %s AND ((rendered_task_instance_fields.dag_id, rendered_task_instance_fields.task_id, rendered_task_instance_fields.run_id) NOT IN (SELECT anon_1.dag_id, anon_1.task_id, anon_1.run_id
> FROM (SELECT DISTINCT rendered_task_instance_fields.dag_id AS dag_id, rendered_task_instance_fields.task_id AS task_id, rendered_task_instance_fields.run_id AS run_id, dag_run.execution_date AS execution_date
> FROM rendered_task_instance_fields INNER JOIN dag_run ON rendered_task_instance_fields.dag_id = dag_run.dag_id AND rendered_task_instance_fields.run_id = dag_run.run_id
> WHERE rendered_task_instance_fields.dag_id = %s AND rendered_task_instance_fields.task_id = %s ORDER BY dag_run.execution_date DESC
> LIMIT %s) AS anon_1))]
> [parameters: ('my_dag', 'my_task', 'my_dag', 'my_task', 30)]
> ```
>
> Currently we have been running a crontab to kill long-sleeping connections to MySQL from Airflow and it seems to work but we have not found the root cause yet.
@tanvn Yes I tihnk #29687 is the right one. I think what could help temporarily is to set the `max_num_rendered_ti_fields_per_task` to 0 . We realize that this query might be problematic on mysql especially and we are discussing with @Taragolis and @kaxil the right way of solving it.
The side effect of setting this value to 0 is that if you have a lot of rendered task instance fields and their size is significant, you might end up with quickly growing size of the Database. But you can in modern airflow to periodically run `airflow db clean` cleaning that table.
Can you please set it and observe your DB and let us know - ideally in #29687 the db size growth and whether it fixed your problems ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #27296: Task completes work but ends up failing due to a lock wait timeout exceeded error and does not honor retries
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #27296:
URL: https://github.com/apache/airflow/issues/27296#issuecomment-1455080894
> The other thing I’d suggest is changing from REPEATABLE READ to READ COMMITTED. This change helped us significantly as well as the number of locks taken on select are greatly reduced.
TIL. Thanks @vchiapaikeo !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vchiapaikeo commented on issue #27296: Task completes work but ends up failing due to a lock wait timeout exceeded error and does not honor retries
Posted by "vchiapaikeo (via GitHub)" <gi...@apache.org>.
vchiapaikeo commented on issue #27296:
URL: https://github.com/apache/airflow/issues/27296#issuecomment-1455077615
You can read more about this here:
https://airflow.apache.org/docs/apache-airflow/2.1.0/concepts/scheduler.html#scheduler-ha-db-requirements
and here:
https://stackoverflow.com/questions/46230555/why-i-shouldnt-use-repeatable-read-with-locking-reading-select-for-update
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vchiapaikeo closed issue #27296: Task completes work but ends up failing due to a lock wait timeout exceeded error and does not honor retries
Posted by GitBox <gi...@apache.org>.
vchiapaikeo closed issue #27296: Task completes work but ends up failing due to a lock wait timeout exceeded error and does not honor retries
URL: https://github.com/apache/airflow/issues/27296
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vchiapaikeo commented on issue #27296: Task completes work but ends up failing due to a lock wait timeout exceeded error and does not honor retries
Posted by "vchiapaikeo (via GitHub)" <gi...@apache.org>.
vchiapaikeo commented on issue #27296:
URL: https://github.com/apache/airflow/issues/27296#issuecomment-1455076684
The other thing I’d suggest is changing from REPEATABLE READ to READ COMMITTED. This change helped us significantly as well as the number of locks taken on select are greatly reduced.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tanvn commented on issue #27296: Task completes work but ends up failing due to a lock wait timeout exceeded error and does not honor retries
Posted by "tanvn (via GitHub)" <gi...@apache.org>.
tanvn commented on issue #27296:
URL: https://github.com/apache/airflow/issues/27296#issuecomment-1455108659
Thanks for the information!
And one thing I should have mentioned here is that we have a hourly DAG that has about 130 independent tasks (but the DAG's Concurrency is set to 32. And I think somehow that is why we are hitting the error like below quite often.
```
[2023-03-05, 23:07:38 JST] {standard_task_runner.py:100} ERROR - Failed to execute job 2083194 for task my_task ((MySQLdb.OperationalError) (1205, 'Lock wait timeout exceeded; try restarting transaction')
[SQL: UPDATE dag_run SET last_scheduling_decision=%s, updated_at=%s WHERE dag_run.id = %s]
```
(however, the tasks are retried and the DAG would be finished in the end)
But I am not sure if this has anything to do with the alert of `Too many (more than 40) concurrent queries have been running.` I got from MySQL server.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #27296: Task completes work but ends up failing due to a lock wait timeout exceeded error and does not honor retries
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #27296:
URL: https://github.com/apache/airflow/issues/27296#issuecomment-1296526559
I think this might be due to https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#schedule-after-task-execution - you can disable it and see whether things will be better (side effect is that there will be potentially bigger latency between directly dependent tasks in the same DAG. Also highly recommend to take a look at @ashb talk about scheduler https://airflowsummit.org/sessions/2021/deep-dive-in-to-the-airflow-scheduler/ - it might explain a lot of your observations.
Please take a look and see if it will be enlightening.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org