You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/07/27 23:34:36 UTC
[GitHub] [airflow] AetherUnbound opened a new issue, #25353: Backfill stalls with certain combination of skipped tasks & trigger rules
AetherUnbound opened a new issue, #25353:
URL: https://github.com/apache/airflow/issues/25353
### Apache Airflow version
2.3.0
### What happened
While trying to run a backfill for one of our DAGs, we noticed that the backfill stalled after completing all the tasks for a given DAG. The `max_active_runs` for this DAG was set to `1`, so the entire backfill stalled even though all the tasks in the last DAG it ran completed successfully.
### What you think should happen instead
I would assume that once all the tasks are complete in a DAG (whether succeeded, skipped, or failed) during a backfill, the backfill should mark the DAG with the proper state and proceed on with the rest of the tasks.
### How to reproduce
Here is simulacrum of our DAG with all the actual logic stripped out:
```python
from datetime import datetime
from airflow.decorators import dag
from airflow.exceptions import AirflowSkipException
from airflow.operators.python import PythonOperator
from airflow.utils.task_group import TaskGroup
from airflow.utils.trigger_rule import TriggerRule
def skipme():
raise AirflowSkipException("Skip")
def run():
return
@dag(
schedule_interval="@daily",
start_date=datetime(2022, 7, 14),
catchup=False,
max_active_runs=1,
)
def sample_dag_with_skip():
a = PythonOperator(
task_id="first",
python_callable=run,
)
with TaskGroup(group_id="subgroup") as tg:
b = PythonOperator(
task_id="run_and_skip",
trigger_rule=TriggerRule.NONE_SKIPPED,
python_callable=skipme,
)
c = PythonOperator(
task_id="run_fine",
trigger_rule=TriggerRule.NONE_SKIPPED,
python_callable=skipme,
)
d = PythonOperator(
task_id="gather",
python_callable=run,
)
e = PythonOperator(
task_id="always_succeed",
trigger_rule=TriggerRule.ALL_DONE,
python_callable=run,
)
[b, c] >> d >> e
f = PythonOperator(
task_id="final",
trigger_rule=TriggerRule.ALL_DONE,
python_callable=run,
)
a >> tg >> f
skip_dag = sample_dag_with_skip()
```
Here's a screenshot of the DAG:
Note that the DAG is still shown as "running" even though the last task ended several minutes ago:
Here's the backfill command I ran for this DAG: `airflow dags backfill -s 2022-07-25 -e 2022-07-26 sample_dag_with_skip --reset-dagruns -y`
And here are the logs from the backfill process:
<details> <summary>Backfill logs</summary>
```
airflow@42a81ed08a3d:~$ airflow dags backfill -s 2022-07-25 -e 2022-07-26 sample_dag_with_skip --reset-dagruns -y
/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/cli/commands/dag_command.py:57 PendingDeprecationWarning: --ignore-first-depends-on-past is deprecated as the value is always set to True
[2022-07-27 23:15:44,655] {dagbag.py:507} INFO - Filling up the DagBag from /usr/local/airflow/openverse_catalog/dags
[2022-07-27 23:15:44,937] {urls.py:74} INFO - https://creativecommons.org/publicdomain/zero/1.0 was rewritten to https://creativecommons.org/publicdomain/zero/1.0/
[2022-07-27 23:15:44,948] {media.py:63} INFO - Initialized image MediaStore with provider brooklynmuseum
[2022-07-27 23:15:44,948] {media.py:168} INFO - No given output directory. Using OUTPUT_DIR from environment.
[2022-07-27 23:15:44,948] {media.py:186} INFO - Output path: /var/workflow_output/brooklynmuseum_image_v001_20220727231544.tsv
[2022-07-27 23:15:44,952] {media.py:63} INFO - Initialized image MediaStore with provider europeana
[2022-07-27 23:15:44,952] {media.py:168} INFO - No given output directory. Using OUTPUT_DIR from environment.
[2022-07-27 23:15:44,952] {media.py:186} INFO - Output path: /var/workflow_output/europeana_image_v001_20220727231544.tsv
[2022-07-27 23:15:44,953] {media.py:63} INFO - Initialized image MediaStore with provider finnishmuseums
[2022-07-27 23:15:44,953] {media.py:168} INFO - No given output directory. Using OUTPUT_DIR from environment.
[2022-07-27 23:15:44,953] {media.py:186} INFO - Output path: /var/workflow_output/finnishmuseums_image_v001_20220727231544.tsv
[2022-07-27 23:15:44,955] {media.py:63} INFO - Initialized image MediaStore with provider flickr
[2022-07-27 23:15:44,955] {media.py:168} INFO - No given output directory. Using OUTPUT_DIR from environment.
[2022-07-27 23:15:44,955] {media.py:186} INFO - Output path: /var/workflow_output/flickr_image_v001_20220727231544.tsv
[2022-07-27 23:15:44,957] {media.py:63} INFO - Initialized audio MediaStore with provider freesound
[2022-07-27 23:15:44,957] {media.py:168} INFO - No given output directory. Using OUTPUT_DIR from environment.
[2022-07-27 23:15:44,957] {media.py:186} INFO - Output path: /var/workflow_output/freesound_audio_v001_20220727231544.tsv
[2022-07-27 23:15:44,959] {media.py:63} INFO - Initialized audio MediaStore with provider jamendo
[2022-07-27 23:15:44,959] {media.py:168} INFO - No given output directory. Using OUTPUT_DIR from environment.
[2022-07-27 23:15:44,959] {media.py:186} INFO - Output path: /var/workflow_output/jamendo_audio_v001_20220727231544.tsv
[2022-07-27 23:15:44,961] {media.py:63} INFO - Initialized image MediaStore with provider met
[2022-07-27 23:15:44,961] {media.py:168} INFO - No given output directory. Using OUTPUT_DIR from environment.
[2022-07-27 23:15:44,961] {media.py:186} INFO - Output path: /var/workflow_output/met_image_v001_20220727231544.tsv
[2022-07-27 23:15:44,962] {media.py:63} INFO - Initialized image MediaStore with provider museumsvictoria
[2022-07-27 23:15:44,962] {media.py:168} INFO - No given output directory. Using OUTPUT_DIR from environment.
[2022-07-27 23:15:44,962] {media.py:186} INFO - Output path: /var/workflow_output/museumsvictoria_image_v001_20220727231544.tsv
[2022-07-27 23:15:44,964] {media.py:63} INFO - Initialized image MediaStore with provider nypl
[2022-07-27 23:15:44,964] {media.py:168} INFO - No given output directory. Using OUTPUT_DIR from environment.
[2022-07-27 23:15:44,964] {media.py:186} INFO - Output path: /var/workflow_output/nypl_image_v001_20220727231544.tsv
[2022-07-27 23:15:44,965] {media.py:63} INFO - Initialized image MediaStore with provider phylopic
[2022-07-27 23:15:44,965] {media.py:168} INFO - No given output directory. Using OUTPUT_DIR from environment.
[2022-07-27 23:15:44,965] {media.py:186} INFO - Output path: /var/workflow_output/phylopic_image_v001_20220727231544.tsv
[2022-07-27 23:15:44,967] {media.py:63} INFO - Initialized image MediaStore with provider rawpixel
[2022-07-27 23:15:44,967] {media.py:168} INFO - No given output directory. Using OUTPUT_DIR from environment.
[2022-07-27 23:15:44,967] {media.py:186} INFO - Output path: /var/workflow_output/rawpixel_image_v001_20220727231544.tsv
[2022-07-27 23:15:44,968] {media.py:63} INFO - Initialized image MediaStore with provider sciencemuseum
[2022-07-27 23:15:44,968] {media.py:168} INFO - No given output directory. Using OUTPUT_DIR from environment.
[2022-07-27 23:15:44,968] {media.py:186} INFO - Output path: /var/workflow_output/sciencemuseum_image_v001_20220727231544.tsv
[2022-07-27 23:15:44,970] {media.py:63} INFO - Initialized image MediaStore with provider smithsonian
[2022-07-27 23:15:44,970] {media.py:168} INFO - No given output directory. Using OUTPUT_DIR from environment.
[2022-07-27 23:15:44,970] {media.py:186} INFO - Output path: /var/workflow_output/smithsonian_image_v001_20220727231544.tsv
[2022-07-27 23:15:44,971] {media.py:63} INFO - Initialized image MediaStore with provider smk
[2022-07-27 23:15:44,972] {media.py:168} INFO - No given output directory. Using OUTPUT_DIR from environment.
[2022-07-27 23:15:44,972] {media.py:186} INFO - Output path: /var/workflow_output/smk_image_v001_20220727231544.tsv
[2022-07-27 23:15:44,974] {media.py:63} INFO - Initialized image MediaStore with provider waltersartmuseum
[2022-07-27 23:15:44,974] {media.py:168} INFO - No given output directory. Using OUTPUT_DIR from environment.
[2022-07-27 23:15:44,974] {media.py:186} INFO - Output path: /var/workflow_output/waltersartmuseum_image_v001_20220727231544.tsv
[2022-07-27 23:15:44,976] {media.py:63} INFO - Initialized audio MediaStore with provider wikimedia_audio
[2022-07-27 23:15:44,976] {media.py:168} INFO - No given output directory. Using OUTPUT_DIR from environment.
[2022-07-27 23:15:44,976] {media.py:186} INFO - Output path: /var/workflow_output/wikimedia_audio_audio_v001_20220727231544.tsv
[2022-07-27 23:15:44,976] {media.py:63} INFO - Initialized image MediaStore with provider wikimedia
[2022-07-27 23:15:44,976] {media.py:168} INFO - No given output directory. Using OUTPUT_DIR from environment.
[2022-07-27 23:15:44,977] {media.py:186} INFO - Output path: /var/workflow_output/wikimedia_image_v001_20220727231544.tsv
[2022-07-27 23:15:44,980] {media.py:63} INFO - Initialized image MediaStore with provider wordpress
[2022-07-27 23:15:44,980] {media.py:168} INFO - No given output directory. Using OUTPUT_DIR from environment.
[2022-07-27 23:15:44,980] {media.py:186} INFO - Output path: /var/workflow_output/wordpress_image_v001_20220727231544.tsv
[2022-07-27 23:15:45,043] {executor_loader.py:106} INFO - Loaded executor: LocalExecutor
[2022-07-27 23:15:45,176] {base_executor.py:91} INFO - Adding to queue: ['airflow', 'tasks', 'run', 'sample_dag_with_skip', 'first', 'backfill__2022-07-25T00:00:00+00:00', '--ignore-depends-on-past', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/simple_backfill_example.py', '--cfg-path', '/tmp/tmpfowbb78c']
[2022-07-27 23:15:50,050] {local_executor.py:79} INFO - QueuedLocalWorker running ['airflow', 'tasks', 'run', 'sample_dag_with_skip', 'first', 'backfill__2022-07-25T00:00:00+00:00', '--ignore-depends-on-past', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/simple_backfill_example.py', '--cfg-path', '/tmp/tmpfowbb78c']
[2022-07-27 23:15:50,060] {dagrun.py:647} WARNING - Failed to get task '<TaskInstance: sample_dag_with_skip.skipme backfill__2022-07-25T00:00:00+00:00 [skipped]>' for dag 'sample_dag_with_skip'. Marking it as removed.
[2022-07-27 23:15:50,061] {dagrun.py:647} WARNING - Failed to get task '<TaskInstance: sample_dag_with_skip.always_run backfill__2022-07-25T00:00:00+00:00 [success]>' for dag 'sample_dag_with_skip'. Marking it as removed.
[2022-07-27 23:15:50,063] {backfill_job.py:367} INFO - [backfill progress] | finished run 0 of 2 | tasks waiting: 7 | succeeded: 0 | running: 1 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 5
[2022-07-27 23:15:50,071] {dagbag.py:507} INFO - Filling up the DagBag from /usr/local/***/openverse_catalog/dags/simple_backfill_example.py
[2022-07-27 23:15:50,089] {task_command.py:369} INFO - Running <TaskInstance: sample_dag_with_skip.first backfill__2022-07-25T00:00:00+00:00 [queued]> on host 42a81ed08a3d
[2022-07-27 23:15:50,486] {base.py:68} INFO - Using connection ID 'aws_default' for task execution.
[2022-07-27 23:15:50,487] {base.py:68} INFO - Using connection ID 'aws_default' for task execution.
[2022-07-27 23:15:50,488] {base_aws.py:206} INFO - Credentials retrieved from login
[2022-07-27 23:15:50,488] {base_aws.py:100} INFO - Retrieving region_name from Connection.extra_config['region_name']
[2022-07-27 23:15:50,525] {base.py:68} INFO - Using connection ID 'aws_default' for task execution.
[2022-07-27 23:15:55,064] {dagrun.py:647} WARNING - Failed to get task '<TaskInstance: sample_dag_with_skip.skipme backfill__2022-07-25T00:00:00+00:00 [removed]>' for dag 'sample_dag_with_skip'. Marking it as removed.
[2022-07-27 23:15:55,065] {dagrun.py:647} WARNING - Failed to get task '<TaskInstance: sample_dag_with_skip.always_run backfill__2022-07-25T00:00:00+00:00 [removed]>' for dag 'sample_dag_with_skip'. Marking it as removed.
[2022-07-27 23:15:55,067] {backfill_job.py:367} INFO - [backfill progress] | finished run 0 of 2 | tasks waiting: 7 | succeeded: 1 | running: 0 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 5
[2022-07-27 23:15:55,081] {base_executor.py:91} INFO - Adding to queue: ['airflow', 'tasks', 'run', 'sample_dag_with_skip', 'subgroup.run_and_skip', 'backfill__2022-07-25T00:00:00+00:00', '--ignore-depends-on-past', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/simple_backfill_example.py', '--cfg-path', '/tmp/tmp1uesref4']
[2022-07-27 23:15:55,170] {base_executor.py:91} INFO - Adding to queue: ['airflow', 'tasks', 'run', 'sample_dag_with_skip', 'subgroup.run_fine', 'backfill__2022-07-25T00:00:00+00:00', '--ignore-depends-on-past', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/simple_backfill_example.py', '--cfg-path', '/tmp/tmpib__7p5u']
[2022-07-27 23:16:00,058] {local_executor.py:79} INFO - QueuedLocalWorker running ['airflow', 'tasks', 'run', 'sample_dag_with_skip', 'subgroup.run_fine', 'backfill__2022-07-25T00:00:00+00:00', '--ignore-depends-on-past', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/simple_backfill_example.py', '--cfg-path', '/tmp/tmpib__7p5u']
[2022-07-27 23:16:00,058] {local_executor.py:79} INFO - QueuedLocalWorker running ['airflow', 'tasks', 'run', 'sample_dag_with_skip', 'subgroup.run_and_skip', 'backfill__2022-07-25T00:00:00+00:00', '--ignore-depends-on-past', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/simple_backfill_example.py', '--cfg-path', '/tmp/tmp1uesref4']
[2022-07-27 23:16:00,063] {dagrun.py:647} WARNING - Failed to get task '<TaskInstance: sample_dag_with_skip.skipme backfill__2022-07-25T00:00:00+00:00 [removed]>' for dag 'sample_dag_with_skip'. Marking it as removed.
[2022-07-27 23:16:00,064] {dagrun.py:647} WARNING - Failed to get task '<TaskInstance: sample_dag_with_skip.always_run backfill__2022-07-25T00:00:00+00:00 [removed]>' for dag 'sample_dag_with_skip'. Marking it as removed.
[2022-07-27 23:16:00,065] {backfill_job.py:367} INFO - [backfill progress] | finished run 0 of 2 | tasks waiting: 5 | succeeded: 1 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 3
[2022-07-27 23:16:00,077] {dagbag.py:507} INFO - Filling up the DagBag from /usr/local/***/openverse_catalog/dags/simple_backfill_example.py
[2022-07-27 23:16:00,080] {dagbag.py:507} INFO - Filling up the DagBag from /usr/local/***/openverse_catalog/dags/simple_backfill_example.py
[2022-07-27 23:16:00,096] {task_command.py:369} INFO - Running <TaskInstance: sample_dag_with_skip.subgroup.run_fine backfill__2022-07-25T00:00:00+00:00 [queued]> on host 42a81ed08a3d
[2022-07-27 23:16:00,097] {task_command.py:369} INFO - Running <TaskInstance: sample_dag_with_skip.subgroup.run_and_skip backfill__2022-07-25T00:00:00+00:00 [queued]> on host 42a81ed08a3d
[2022-07-27 23:16:00,504] {base.py:68} INFO - Using connection ID 'aws_default' for task execution.
[2022-07-27 23:16:00,505] {base.py:68} INFO - Using connection ID 'aws_default' for task execution.
[2022-07-27 23:16:00,506] {base_aws.py:206} INFO - Credentials retrieved from login
[2022-07-27 23:16:00,506] {base_aws.py:100} INFO - Retrieving region_name from Connection.extra_config['region_name']
[2022-07-27 23:16:00,514] {base.py:68} INFO - Using connection ID 'aws_default' for task execution.
[2022-07-27 23:16:00,515] {base.py:68} INFO - Using connection ID 'aws_default' for task execution.
[2022-07-27 23:16:00,516] {base_aws.py:206} INFO - Credentials retrieved from login
[2022-07-27 23:16:00,516] {base_aws.py:100} INFO - Retrieving region_name from Connection.extra_config['region_name']
[2022-07-27 23:16:00,541] {base.py:68} INFO - Using connection ID 'aws_default' for task execution.
[2022-07-27 23:16:00,559] {base.py:68} INFO - Using connection ID 'aws_default' for task execution.
[2022-07-27 23:16:05,071] {dagrun.py:647} WARNING - Failed to get task '<TaskInstance: sample_dag_with_skip.skipme backfill__2022-07-25T00:00:00+00:00 [removed]>' for dag 'sample_dag_with_skip'. Marking it as removed.
[2022-07-27 23:16:05,072] {dagrun.py:647} WARNING - Failed to get task '<TaskInstance: sample_dag_with_skip.always_run backfill__2022-07-25T00:00:00+00:00 [removed]>' for dag 'sample_dag_with_skip'. Marking it as removed.
[2022-07-27 23:16:05,073] {dagrun.py:583} ERROR - Deadlock; marking run <DagRun sample_dag_with_skip @ 2022-07-25 00:00:00+00:00: backfill__2022-07-25T00:00:00+00:00, externally triggered: False> failed
[2022-07-27 23:16:05,073] {dagrun.py:607} INFO - DagRun Finished: dag_id=sample_dag_with_skip, execution_date=2022-07-25 00:00:00+00:00, run_id=backfill__2022-07-25T00:00:00+00:00, run_start_date=None, run_end_date=2022-07-27 23:16:05.073628+00:00, run_duration=None, state=failed, external_trigger=False, run_type=backfill, data_interval_start=2022-07-25 00:00:00+00:00, data_interval_end=2022-07-26 00:00:00+00:00, dag_hash=None
[2022-07-27 23:16:05,075] {dagrun.py:799} WARNING - Failed to record duration of <DagRun sample_dag_with_skip @ 2022-07-25 00:00:00+00:00: backfill__2022-07-25T00:00:00+00:00, externally triggered: False>: start_date is not set.
[2022-07-27 23:16:05,075] {backfill_job.py:367} INFO - [backfill progress] | finished run 1 of 2 | tasks waiting: 5 | succeeded: 1 | running: 0 | failed: 0 | skipped: 2 | deadlocked: 0 | not ready: 3
[2022-07-27 23:16:05,096] {base_executor.py:91} INFO - Adding to queue: ['airflow', 'tasks', 'run', 'sample_dag_with_skip', 'subgroup.always_succeed', 'backfill__2022-07-25T00:00:00+00:00', '--ignore-depends-on-past', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/simple_backfill_example.py', '--cfg-path', '/tmp/tmpulo4p958']
[2022-07-27 23:16:10,065] {local_executor.py:79} INFO - QueuedLocalWorker running ['airflow', 'tasks', 'run', 'sample_dag_with_skip', 'subgroup.always_succeed', 'backfill__2022-07-25T00:00:00+00:00', '--ignore-depends-on-past', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/simple_backfill_example.py', '--cfg-path', '/tmp/tmpulo4p958']
[2022-07-27 23:16:10,067] {backfill_job.py:367} INFO - [backfill progress] | finished run 1 of 2 | tasks waiting: 3 | succeeded: 1 | running: 1 | failed: 0 | skipped: 3 | deadlocked: 0 | not ready: 1
[2022-07-27 23:16:10,084] {dagbag.py:507} INFO - Filling up the DagBag from /usr/local/***/openverse_catalog/dags/simple_backfill_example.py
[2022-07-27 23:16:10,104] {task_command.py:369} INFO - Running <TaskInstance: sample_dag_with_skip.subgroup.always_succeed backfill__2022-07-25T00:00:00+00:00 [queued]> on host 42a81ed08a3d
[2022-07-27 23:16:10,500] {base.py:68} INFO - Using connection ID 'aws_default' for task execution.
[2022-07-27 23:16:10,501] {base.py:68} INFO - Using connection ID 'aws_default' for task execution.
[2022-07-27 23:16:10,502] {base_aws.py:206} INFO - Credentials retrieved from login
[2022-07-27 23:16:10,502] {base_aws.py:100} INFO - Retrieving region_name from Connection.extra_config['region_name']
[2022-07-27 23:16:10,537] {base.py:68} INFO - Using connection ID 'aws_default' for task execution.
[2022-07-27 23:16:15,074] {backfill_job.py:367} INFO - [backfill progress] | finished run 1 of 2 | tasks waiting: 3 | succeeded: 2 | running: 0 | failed: 0 | skipped: 3 | deadlocked: 0 | not ready: 1
[2022-07-27 23:16:15,091] {base_executor.py:91} INFO - Adding to queue: ['airflow', 'tasks', 'run', 'sample_dag_with_skip', 'final', 'backfill__2022-07-25T00:00:00+00:00', '--ignore-depends-on-past', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/simple_backfill_example.py', '--cfg-path', '/tmp/tmp7ifr68s2']
[2022-07-27 23:16:20,073] {local_executor.py:79} INFO - QueuedLocalWorker running ['airflow', 'tasks', 'run', 'sample_dag_with_skip', 'final', 'backfill__2022-07-25T00:00:00+00:00', '--ignore-depends-on-past', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/simple_backfill_example.py', '--cfg-path', '/tmp/tmp7ifr68s2']
[2022-07-27 23:16:20,075] {backfill_job.py:367} INFO - [backfill progress] | finished run 1 of 2 | tasks waiting: 2 | succeeded: 2 | running: 1 | failed: 0 | skipped: 3 | deadlocked: 0 | not ready: 0
[2022-07-27 23:16:20,094] {dagbag.py:507} INFO - Filling up the DagBag from /usr/local/***/openverse_catalog/dags/simple_backfill_example.py
[2022-07-27 23:16:20,114] {task_command.py:369} INFO - Running <TaskInstance: sample_dag_with_skip.final backfill__2022-07-25T00:00:00+00:00 [queued]> on host 42a81ed08a3d
[2022-07-27 23:16:20,522] {base.py:68} INFO - Using connection ID 'aws_default' for task execution.
[2022-07-27 23:16:20,523] {base.py:68} INFO - Using connection ID 'aws_default' for task execution.
[2022-07-27 23:16:20,524] {base_aws.py:206} INFO - Credentials retrieved from login
[2022-07-27 23:16:20,524] {base_aws.py:100} INFO - Retrieving region_name from Connection.extra_config['region_name']
[2022-07-27 23:16:20,561] {base.py:68} INFO - Using connection ID 'aws_default' for task execution.
[2022-07-27 23:16:25,082] {backfill_job.py:367} INFO - [backfill progress] | finished run 1 of 2 | tasks waiting: 2 | succeeded: 3 | running: 0 | failed: 0 | skipped: 3 | deadlocked: 0 | not ready: 0
[2022-07-27 23:16:30,083] {backfill_job.py:367} INFO - [backfill progress] | finished run 1 of 2 | tasks waiting: 2 | succeeded: 3 | running: 0 | failed: 0 | skipped: 3 | deadlocked: 0 | not ready: 0
[2022-07-27 23:16:35,089] {backfill_job.py:367} INFO - [backfill progress] | finished run 1 of 2 | tasks waiting: 2 | succeeded: 3 | running: 0 | failed: 0 | skipped: 3 | deadlocked: 0 | not ready: 0
[2022-07-27 23:16:40,093] {backfill_job.py:367} INFO - [backfill progress] | finished run 1 of 2 | tasks waiting: 2 | succeeded: 3 | running: 0 | failed: 0 | skipped: 3 | deadlocked: 0 | not ready: 0
[2022-07-27 23:16:45,099] {backfill_job.py:367} INFO - [backfill progress] | finished run 1 of 2 | tasks waiting: 2 | succeeded: 3 | running: 0 | failed: 0 | skipped: 3 | deadlocked: 0 | not ready: 0
[2022-07-27 23:16:50,105] {backfill_job.py:367} INFO - [backfill progress] | finished run 1 of 2 | tasks waiting: 2 | succeeded: 3 | running: 0 | failed: 0 | skipped: 3 | deadlocked: 0 | not ready: 0
[2022-07-27 23:16:55,112] {backfill_job.py:367} INFO - [backfill progress] | finished run 1 of 2 | tasks waiting: 2 | succeeded: 3 | running: 0 | failed: 0 | skipped: 3 | deadlocked: 0 | not ready: 0
[2022-07-27 23:17:00,117] {backfill_job.py:367} INFO - [backfill progress] | finished run 1 of 2 | tasks waiting: 2 | succeeded: 3 | running: 0 | failed: 0 | skipped: 3 | deadlocked: 0 | not ready: 0
[2022-07-27 23:17:05,122] {backfill_job.py:367} INFO - [backfill progress] | finished run 1 of 2 | tasks waiting: 2 | succeeded: 3 | running: 0 | failed: 0 | skipped: 3 | deadlocked: 0 | not ready: 0
[2022-07-27 23:17:10,128] {backfill_job.py:367} INFO - [backfill progress] | finished run 1 of 2 | tasks waiting: 2 | succeeded: 3 | running: 0 | failed: 0 | skipped: 3 | deadlocked: 0 | not ready: 0
[2022-07-27 23:17:15,133] {backfill_job.py:367} INFO - [backfill progress] | finished run 1 of 2 | tasks waiting: 2 | succeeded: 3 | running: 0 | failed: 0 | skipped: 3 | deadlocked: 0 | not ready: 0
[2022-07-27 23:17:20,137] {backfill_job.py:367} INFO - [backfill progress] | finished run 1 of 2 | tasks waiting: 2 | succeeded: 3 | running: 0 | failed: 0 | skipped: 3 | deadlocked: 0 | not ready: 0
[2022-07-27 23:17:25,141] {backfill_job.py:367} INFO - [backfill progress] | finished run 1 of 2 | tasks waiting: 2 | succeeded: 3 | running: 0 | failed: 0 | skipped: 3 | deadlocked: 0 | not ready: 0
[2022-07-27 23:17:30,145] {backfill_job.py:367} INFO - [backfill progress] | finished run 1 of 2 | tasks waiting: 2 | succeeded: 3 | running: 0 | failed: 0 | skipped: 3 | deadlocked: 0 | not ready: 0
[2022-07-27 23:17:35,149] {backfill_job.py:367} INFO - [backfill progress] | finished run 1 of 2 | tasks waiting: 2 | succeeded: 3 | running: 0 | failed: 0 | skipped: 3 | deadlocked: 0 | not ready: 0
[2022-07-27 23:17:40,153] {backfill_job.py:367} INFO - [backfill progress] | finished run 1 of 2 | tasks waiting: 2 | succeeded: 3 | running: 0 | failed: 0 | skipped: 3 | deadlocked: 0 | not ready: 0
[2022-07-27 23:17:45,157] {backfill_job.py:367} INFO - [backfill progress] | finished run 1 of 2 | tasks waiting: 2 | succeeded: 3 | running: 0 | failed: 0 | skipped: 3 | deadlocked: 0 | not ready: 0
[2022-07-27 23:17:50,161] {backfill_job.py:367} INFO - [backfill progress] | finished run 1 of 2 | tasks waiting: 2 | succeeded: 3 | running: 0 | failed: 0 | skipped: 3 | deadlocked: 0 | not ready: 0
[2022-07-27 23:17:55,166] {backfill_job.py:367} INFO - [backfill progress] | finished run 1 of 2 | tasks waiting: 2 | succeeded: 3 | running: 0 | failed: 0 | skipped: 3 | deadlocked: 0 | not ready: 0
[2022-07-27 23:18:00,169] {backfill_job.py:367} INFO - [backfill progress] | finished run 1 of 2 | tasks waiting: 2 | succeeded: 3 | running: 0 | failed: 0 | skipped: 3 | deadlocked: 0 | not ready: 0
[2022-07-27 23:18:05,174] {backfill_job.py:367} INFO - [backfill progress] | finished run 1 of 2 | tasks waiting: 2 | succeeded: 3 | running: 0 | failed: 0 | skipped: 3 | deadlocked: 0 | not ready: 0
[2022-07-27 23:18:10,177] {backfill_job.py:367} INFO - [backfill progress] | finished run 1 of 2 | tasks waiting: 2 | succeeded: 3 | running: 0 | failed: 0 | skipped: 3 | deadlocked: 0 | not ready: 0
^C[2022-07-27 23:18:11,199] {backfill_job.py:870} WARNING - Backfill terminated by user.
[2022-07-27 23:18:11,199] {local_executor.py:390} INFO - Shutting down LocalExecutor; waiting for running tasks to finish. Signal again if you don't want to wait.
```
</details>
---
It's worth noting that I tried to replicate this with a DAG that only had a single skip task, and a DAG with skip -> run -> skip, and both succeeded with the backfill. So my guess would be that there's an odd interaction with the TaskGroup, skipped tasks, trigger rules, and possibly `max_active_runs`.
### Operating System
Debian GNU/Linux 11 (bullseye)
### Versions of Apache Airflow Providers
```
apache-airflow-providers-amazon==3.3.0
apache-airflow-providers-ftp==2.1.2
apache-airflow-providers-http==2.1.2
apache-airflow-providers-imap==2.2.3
apache-airflow-providers-postgres==4.1.0
apache-airflow-providers-sqlite==2.1.3
```
### Deployment
Docker-Compose
### Deployment details
This is a custom configured Docker image, but doesn't deviate too much from a standard deployment: https://github.com/WordPress/openverse-catalog/blob/main/docker/airflow/Dockerfile
### Anything else
I'll try to see if I can continue to pare down the DAG to see if there are pieces I can throw out and still replicate the error. I don't think I'd be comfortable submitting a PR for this one because my gut says it's probably deep in the bowels of the codebase 😅 If it's something clear or straightforward though, I'd be happy to take a stab at it! I'd just need to be pointed in the right direction within the codebase 🙂
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #25353: Backfill stalls with certain combination of skipped tasks & trigger rules
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #25353:
URL: https://github.com/apache/airflow/issues/25353#issuecomment-1198553241
```
[2022-07-27 23:16:05,073] {dagrun.py:583} ERROR - Deadlock; marking run <DagRun sample_dag_with_skip @ 2022-07-25 00:00:00+00:00: backfill__2022-07-25T00:00:00+00:00, externally triggered: False> failed
```
@ashb - that might be another manifestation of the same deadlock case we saw recently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] wolfier commented on issue #25353: Backfill stalls with certain combination of skipped tasks & trigger rules
Posted by GitBox <gi...@apache.org>.
wolfier commented on issue #25353:
URL: https://github.com/apache/airflow/issues/25353#issuecomment-1230716931
@potiuk Is there an issue for the deadlock issue?
I made an issue (#26044) in regards to a similar/same behaviour.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] AetherUnbound commented on issue #25353: Backfill stalls with certain combination of skipped tasks & trigger rules
Posted by GitBox <gi...@apache.org>.
AetherUnbound commented on issue #25353:
URL: https://github.com/apache/airflow/issues/25353#issuecomment-1197493866
Ha, I could have sworn I've submitted other issues in the past! 😄 Looks like it was with my old @madison-ookla account 🙂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #25353: Backfill stalls with certain combination of skipped tasks & trigger rules
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #25353:
URL: https://github.com/apache/airflow/issues/25353#issuecomment-1197487907
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr closed issue #25353: Backfill stalls with certain combination of skipped tasks & trigger rules
Posted by GitBox <gi...@apache.org>.
uranusjr closed issue #25353: Backfill stalls with certain combination of skipped tasks & trigger rules
URL: https://github.com/apache/airflow/issues/25353
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org