You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/08/15 21:48:13 UTC
[GitHub] [airflow] Gollum999 commented on issue #25730: Deadlock when backfilling mapped tasks
Gollum999 commented on issue #25730:
URL: https://github.com/apache/airflow/issues/25730#issuecomment-1215890856
More log context from my second example, since my comment was too long:
<details>
<summary>Expand</summary>
```
2022-08-15 16:25:05.855 | INFO | airflow.models.dagbag.DagBag | Filling up the DagBag from /opt/tradebot/scripts/dags
2022-08-15 16:25:07.025 | INFO | luigi-interface | Loaded ['/etc/luigi/luigi.cfg']
2022-08-15 16:25:07.582 | INFO | airflow.executors.executor_loader | Loaded executor: CeleryExecutor
2022-08-15 16:25:07.710 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 'get_tasks', 'backfill__2022-01-03T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py']
2022-08-15 16:25:07.717 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 'get_tasks', 'backfill__2022-01-04T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py']
2022-08-15 16:25:07.724 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 'get_tasks', 'backfill__2022-01-05T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py']
2022-08-15 16:25:07.730 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 'get_tasks', 'backfill__2022-01-06T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py']
2022-08-15 16:25:07.736 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 'get_tasks', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py']
2022-08-15 16:25:12.741 | INFO | airflow.jobs.backfill_job.BackfillJob | [backfill progress] | finished run 0 of 5 | tasks waiting: 10 | succeeded: 0 | running: 5 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 10
2022-08-15 16:25:18.324 | INFO | airflow.jobs.backfill_job.BackfillJob | [backfill progress] | finished run 0 of 5 | tasks waiting: 255 | succeeded: 5 | running: 0 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 10
2022-08-15 16:25:18.338 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't1', 'backfill__2022-01-03T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '1']
2022-08-15 16:25:18.345 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't1', 'backfill__2022-01-03T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '2']
2022-08-15 16:25:18.352 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't1', 'backfill__2022-01-03T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '3']
2022-08-15 16:25:18.360 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't1', 'backfill__2022-01-03T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '4']
2022-08-15 16:25:18.367 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't1', 'backfill__2022-01-03T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '5']
2022-08-15 16:25:18.374 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't1', 'backfill__2022-01-03T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '6']
...
2022-08-15 16:25:18.681 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't1', 'backfill__2022-01-03T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '49']
2022-08-15 16:25:18.693 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't1', 'backfill__2022-01-04T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '1']
2022-08-15 16:25:18.700 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't1', 'backfill__2022-01-04T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '2']
2022-08-15 16:25:18.707 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't1', 'backfill__2022-01-04T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '3']
...
2022-08-15 16:25:19.112 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't1', 'backfill__2022-01-05T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '10']
2022-08-15 16:25:28.124 | INFO | airflow.models.dagrun.DagRun | Hit IntegrityError while creating the TIs for test_backfill_races- backfill__2022-01-04T00:00:00-06:00
Traceback (most recent call last):
File "/opt/conda/envs/production/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1799, in _execute_context
self.dialect.do_executemany(
File "/opt/conda/envs/production/lib/python3.9/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 953, in do_executemany
context._psycopg2_fetched_rows = xtras.execute_values(
File "/opt/conda/envs/production/lib/python3.9/site-packages/psycopg2/extras.py", line 1270, in execute_values
cur.execute(b''.join(parts))
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "task_instance_pkey"
DETAIL: Key (dag_id, task_id, run_id, map_index)=(test_backfill_races, t2, backfill__2022-01-04T00:00:00-06:00, 24) already exists.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/conda/envs/production/lib/python3.9/site-packages/airflow/models/dagrun.py", line 1048, in _create_task_instances
session.bulk_save_objects(tasks)
File "/opt/conda/envs/production/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 3627, in bulk_save_objects
self._bulk_save_mappings(
File "/opt/conda/envs/production/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 3843, in _bulk_save_mappings
transaction.rollback(_capture_exception=True)
File "/opt/conda/envs/production/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
compat.raise_(
File "/opt/conda/envs/production/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/opt/conda/envs/production/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 3831, in _bulk_save_mappings
persistence._bulk_insert(
File "/opt/conda/envs/production/lib/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 107, in _bulk_insert
_emit_insert_statements(
File "/opt/conda/envs/production/lib/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements
c = connection._execute_20(
File "/opt/conda/envs/production/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20
return meth(self, args_10style, kwargs_10style, execution_options)
File "/opt/conda/envs/production/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 325, in _execute_on_connection
return connection._execute_clauseelement(
File "/opt/conda/envs/production/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement
ret = self._execute_context(
File "/opt/conda/envs/production/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context
self._handle_dbapi_exception(
File "/opt/conda/envs/production/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2043, in _handle_dbapi_exception
util.raise_(
File "/opt/conda/envs/production/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/opt/conda/envs/production/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1799, in _execute_context
self.dialect.do_executemany(
File "/opt/conda/envs/production/lib/python3.9/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 953, in do_executemany
context._psycopg2_fetched_rows = xtras.execute_values(
File "/opt/conda/envs/production/lib/python3.9/site-packages/psycopg2/extras.py", line 1270, in execute_values
cur.execute(b''.join(parts))
sqlalchemy.exc.IntegrityError: (psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "task_instance_pkey"
DETAIL: Key (dag_id, task_id, run_id, map_index)=(test_backfill_races, t2, backfill__2022-01-04T00:00:00-06:00, 24) already exists.
[SQL: INSERT INTO task_instance (task_id, dag_id, run_id, map_index, try_number, max_tries, hostname, unixname, pool, pool_slots, queue, priority_weight, operator, executor_config) VALUES (%(task_id)s, %(dag_id)s, %(run_id)s, %(map_index)s, %(try_number)s, %(max_tries)s, %(hostname)s, %(unixname)s, %(pool)s, %(pool_slots)s, %(queue)s, %(priority_weight)s, %(operator)s, %(executor_config)s)]
[parameters: ({'task_id': 't2', 'dag_id': 'test_backfill_races', 'run_id': 'backfill__2022-01-04T00:00:00-06:00', 'map_index': 24, 'try_number': 0, 'max_tries': 0, 'hostname': '', 'unixname': 'tradebot', 'pool': 'default_pool', 'pool_slots': 1, 'queue': 'default', 'priority_weight': 1, 'operator': '_PythonDecoratedOperator', 'executor_config': <psycopg2.extensions.Binary object at 0x7f2522faf2d0>}, {'task_id': 't2', 'dag_id': 'test_backfill_races', 'run_id': 'backfill__2022-01-04T00:00:00-06:00', 'map_index': 25, 'try_number': 0, 'max_tries': 0, 'hostname': '', 'unixname': 'tradebot', 'pool': 'default_pool', 'pool_slots': 1, 'queue': 'default', 'priority_weight': 1, 'operator': '_PythonDecoratedOperator', 'executor_config': <psycopg2.extensions.Binary object at 0x7f2522faf0f0>}, {'task_id': 't2', 'dag_id': 'test_backfill_races', 'run_id': 'backfill__2022-01-04T00:00:00-06:00', 'map_index': 26, 'try_number': 0, 'max_tries': 0, 'hostname': '', 'unixname': 'tradebot', 'pool': 'defaul
t_pool', 'pool_slots': 1, 'queue': 'default', 'priority_weight': 1, 'operator': '_PythonDecoratedOperator', 'executor_config': <psycopg2.extensions.Binary object at 0x7f2522faf8d0>}, {'task_id': 't2', 'dag_id': 'test_backfill_races', 'run_id': 'backfill__2022-01-04T00:00:00-06:00', 'map_index': 27, 'try_number': 0, 'max_tries': 0, 'hostname': '', 'unixname': 'tradebot', 'pool': 'default_pool', 'pool_slots': 1, 'queue': 'default', 'priority_weight': 1, 'operator': '_PythonDecoratedOperator', 'executor_config': <psycopg2.extensions.Binary object at 0x7f2522fafbd0>})]
(Background on this error at: https://sqlalche.me/e/14/gkpj)
2022-08-15 16:25:28.127 | INFO | airflow.models.dagrun.DagRun | Doing session rollback.
2022-08-15 16:25:29.537 | INFO | airflow.jobs.backfill_job.BackfillJob | [backfill progress] | finished run 0 of 5 | tasks waiting: 179 | succeeded: 88 | running: 25 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 3
2022-08-15 16:25:36.734 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't1', 'backfill__2022-01-05T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '11']
2022-08-15 16:25:36.742 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't1', 'backfill__2022-01-05T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '12']
2022-08-15 16:25:36.749 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't1', 'backfill__2022-01-05T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '13']
...
2022-08-15 16:25:37.227 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't1', 'backfill__2022-01-06T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '25']
2022-08-15 16:25:38.088 | WARNING | airflow.jobs.backfill_job.BackfillJob | TaskInstanceKey(dag_id='test_backfill_races', task_id='t1', run_id='backfill__2022-01-03T00:00:00-06:00', try_number=1, map_index=1) state success not in running=dict_values([<TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=1 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=12 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=13 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=14 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=15 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=16 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=17 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04
T00:00:00-06:00 map_index=18 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=19 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=20 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=21 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=36 [running]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=37 [running]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=38 [running]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=39 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=40 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=41 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:0
0 map_index=42 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=43 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=44 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=45 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=46 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=47 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=48 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=49 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=11 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=12 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=13 [q
ueued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=14 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=15 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=16 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=17 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=18 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=19 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=20 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=21 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=22 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=23 [running]>, <TaskIn
stance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=24 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=25 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=26 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=27 [running]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=28 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=29 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=30 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=31 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=32 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=33 [queued]>, <TaskInstance: test_bac
kfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=34 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=35 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=36 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=37 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=38 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=39 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=40 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=41 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=42 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=43 [queued]>, <TaskInstance: test_backfill_races.t1 ba
ckfill__2022-01-05T00:00:00-06:00 map_index=44 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=45 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=46 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=47 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=48 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=49 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=1 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=2 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=3 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=4 [running]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T0
0:00:00-06:00 map_index=5 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=6 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=7 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=8 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=9 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=10 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=11 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=12 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=13 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=14 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_inde
x=15 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=16 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=17 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=18 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=19 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=20 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=21 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=22 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=23 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=24 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=25 [queued]>])
2022-08-15 16:25:38.089 | WARNING | airflow.jobs.backfill_job.BackfillJob | TaskInstanceKey(dag_id='test_backfill_races', task_id='t1', run_id='backfill__2022-01-03T00:00:00-06:00', try_number=1, map_index=2) state success not in running=dict_values([<TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=1 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=12 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=13 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=14 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=15 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=16 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=17 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04
T00:00:00-06:00 map_index=18 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=19 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=20 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=21 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=36 [running]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=37 [running]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=38 [running]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=39 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=40 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=41 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:0
0 map_index=42 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=43 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=44 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=45 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=46 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=47 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=48 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=49 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=11 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=12 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=13 [q
ueued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=14 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=15 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=16 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=17 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=18 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=19 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=20 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=21 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=22 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=23 [running]>, <TaskIn
stance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=24 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=25 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=26 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=27 [running]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=28 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=29 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=30 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=31 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=32 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=33 [queued]>, <TaskInstance: test_bac
kfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=34 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=35 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=36 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=37 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=38 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=39 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=40 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=41 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=42 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=43 [queued]>, <TaskInstance: test_backfill_races.t1 ba
ckfill__2022-01-05T00:00:00-06:00 map_index=44 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=45 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=46 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=47 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=48 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=49 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=1 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=2 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=3 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=4 [running]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T0
0:00:00-06:00 map_index=5 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=6 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=7 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=8 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=9 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=10 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=11 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=12 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=13 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=14 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_inde
x=15 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=16 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=17 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=18 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=19 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=20 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=21 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=22 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=23 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=24 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=25 [queued]>])
...
2022-08-15 16:25:38.237 | WARNING | airflow.jobs.backfill_job.BackfillJob | TaskInstanceKey(dag_id='test_backfill_races', task_id='t1', run_id='backfill__2022-01-05T00:00:00-06:00', try_number=1, map_index=10) state success not in running=dict_values([<TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=1 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=12 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=13 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=14 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=15 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=16 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=17 [success]>, <TaskInstance: test_backfill_races.t1 backfill__20
22-01-04T00:00:00-06:00 map_index=18 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=19 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=20 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=21 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=36 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=37 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=38 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=39 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=40 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=41 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-0
4T00:00:00-06:00 map_index=42 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=43 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=44 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=45 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=46 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=47 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=48 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=49 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=11 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=12 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:0
0-06:00 map_index=13 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=14 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=15 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=16 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=17 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=18 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=19 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=20 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=21 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=22 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index
=23 [running]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=24 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=25 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=26 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=27 [running]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=28 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=29 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=30 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=31 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=32 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=33 [queued]>,
<TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=34 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=35 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=36 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=37 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=38 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=39 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=40 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=41 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=42 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=43 [queued]>, <TaskInstance: te
st_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=44 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=45 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=46 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=47 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=48 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=49 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=1 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=2 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=3 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=4 [running]>, <TaskInstance: test_backfill_races.t1
backfill__2022-01-06T00:00:00-06:00 map_index=5 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=6 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=7 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=8 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=9 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=10 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=11 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=12 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=13 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=14 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T
00:00:00-06:00 map_index=15 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=16 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=17 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=18 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=19 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=20 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=21 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=22 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=23 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=24 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 ma
p_index=25 [queued]>])
2022-08-15 16:25:39.135 | INFO | airflow.jobs.backfill_job.BackfillJob | [backfill progress] | finished run 0 of 5 | tasks waiting: 115 | succeeded: 114 | running: 63 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 4
2022-08-15 16:25:44.692 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't1', 'backfill__2022-01-06T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '26']
2022-08-15 16:25:44.701 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't1', 'backfill__2022-01-06T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '27']
2022-08-15 16:25:44.709 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't1', 'backfill__2022-01-06T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '28']
...
2022-08-15 16:25:45.232 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't1', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '48']
2022-08-15 16:25:45.239 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't1', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '49']
2022-08-15 16:25:52.343 | WARNING | airflow.jobs.backfill_job.BackfillJob | TaskInstanceKey(dag_id='test_backfill_races', task_id='t1', run_id='backfill__2022-01-06T00:00:00-06:00', try_number=1, map_index=4) state success not in running=dict_values([<TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=11 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=12 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=13 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=14 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=15 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=16 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=17 [success]>, <TaskInstance: test_backfill_races.t1 backfill__20
22-01-05T00:00:00-06:00 map_index=18 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=19 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=20 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=21 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=22 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=23 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=24 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=25 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=26 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=27 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-0
5T00:00:00-06:00 map_index=28 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=29 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=30 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=31 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=32 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=33 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=34 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=35 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=36 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=37 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00
:00-06:00 map_index=38 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=39 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=40 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=41 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=42 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=43 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=44 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=45 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=46 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=47 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:
00 map_index=48 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=49 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=1 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=2 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=3 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=5 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=6 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=7 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=8 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=9 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=10 [que
ued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=11 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=12 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=13 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=14 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=15 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=16 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=17 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=18 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=19 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=20 [queued]>, <TaskInsta
nce: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=21 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=22 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=23 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=24 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=25 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=26 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=27 [running]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=28 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=29 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=30 [queued]>, <TaskInstance: test_backfi
ll_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=31 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=32 [running]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=33 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=34 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=35 [running]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=36 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=37 [running]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=38 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=39 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=40 [queued]>, <TaskInstance: test_backfill_races.t1 ba
ckfill__2022-01-06T00:00:00-06:00 map_index=41 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=42 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=43 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=44 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=45 [running]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=46 [running]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=47 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=48 [running]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=49 [running]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=0 [running]>, <TaskInstance: test_backfill_races.t1 backfill__2022-
01-07T00:00:00-06:00 map_index=1 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=2 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=3 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=4 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=5 [running]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=6 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=7 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=8 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=9 [running]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=10 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map
_index=11 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=12 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=13 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=14 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=15 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=16 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=17 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=18 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=19 [running]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=20 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=21 [queue
d]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=22 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=23 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=24 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=25 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=26 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=27 [running]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=28 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=29 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=30 [running]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=31 [queued]>, <TaskInsta
nce: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=32 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=33 [running]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=34 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=35 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=36 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=37 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=38 [running]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=39 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=40 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=41 [queued]>, <TaskInstance: test_backf
ill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=42 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=43 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=44 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=45 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=46 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=47 [running]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=48 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=49 [queued]>])
2022-08-15 16:25:57.668 | INFO | airflow.jobs.backfill_job.BackfillJob | [backfill progress] | finished run 0 of 5 | tasks waiting: 78 | succeeded: 197 | running: 54 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 41
...
2022-08-15 16:26:01.984 | WARNING | airflow.jobs.backfill_job.BackfillJob | TaskInstanceKey(dag_id='test_backfill_races', task_id='t1', run_id='backfill__2022-01-06T00:00:00-06:00', try_number=1, map_index=49) state success not in running=dict_values([<TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=29 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=34 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=39 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=40 [success]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=0 [running]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=1 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=2 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-0
1-07T00:00:00-06:00 map_index=3 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=4 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=5 [running]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=6 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=7 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=8 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=9 [running]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=10 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=11 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=12 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 ma
p_index=13 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=14 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=15 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=16 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=17 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=18 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=19 [running]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=20 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=21 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=22 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=23 [queu
ed]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=24 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=25 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=26 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=27 [running]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=28 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=29 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=30 [running]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=31 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=32 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=33 [running]>, <TaskIns
tance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=34 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=35 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=36 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=37 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=38 [running]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=39 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=40 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=41 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=42 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=43 [queued]>, <TaskInstance: test_back
fill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=44 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=45 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=46 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=47 [running]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=48 [queued]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-07T00:00:00-06:00 map_index=49 [queued]>])
2022-08-15 16:26:02.857 | INFO | airflow.jobs.backfill_job.BackfillJob | [backfill progress] | finished run 0 of 5 | tasks waiting: 127 | succeeded: 251 | running: 0 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 78
2022-08-15 16:26:03.496 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '0']
2022-08-15 16:26:03.506 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '1']
2022-08-15 16:26:03.516 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '2']
2022-08-15 16:26:03.526 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '3']
2022-08-15 16:26:03.537 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '4']
2022-08-15 16:26:03.547 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '5']
2022-08-15 16:26:03.557 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '6']
2022-08-15 16:26:03.567 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '7']
2022-08-15 16:26:03.578 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '8']
2022-08-15 16:26:03.588 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '9']
2022-08-15 16:26:03.598 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '10']
2022-08-15 16:26:03.608 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '11']
2022-08-15 16:26:03.619 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '12']
2022-08-15 16:26:03.629 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '13']
2022-08-15 16:26:03.639 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '14']
2022-08-15 16:26:03.650 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '15']
2022-08-15 16:26:03.660 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '16']
2022-08-15 16:26:03.670 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '17']
2022-08-15 16:26:03.680 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '18']
2022-08-15 16:26:03.691 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '19']
2022-08-15 16:26:03.701 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '20']
2022-08-15 16:26:03.711 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '21']
2022-08-15 16:26:03.721 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '22']
2022-08-15 16:26:03.732 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '23']
2022-08-15 16:26:03.742 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '24']
2022-08-15 16:26:03.752 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '25']
2022-08-15 16:26:03.762 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '26']
2022-08-15 16:26:03.773 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '27']
2022-08-15 16:26:03.783 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '28']
2022-08-15 16:26:03.793 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '29']
2022-08-15 16:26:03.803 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '30']
2022-08-15 16:26:03.814 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '31']
2022-08-15 16:26:03.824 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '32']
2022-08-15 16:26:03.834 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '33']
2022-08-15 16:26:03.845 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '34']
2022-08-15 16:26:03.855 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '35']
2022-08-15 16:26:03.865 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '36']
2022-08-15 16:26:03.876 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '37']
2022-08-15 16:26:03.886 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '38']
2022-08-15 16:26:03.896 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '39']
2022-08-15 16:26:03.907 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '40']
2022-08-15 16:26:03.917 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '41']
2022-08-15 16:26:03.927 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '42']
2022-08-15 16:26:03.937 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '43']
2022-08-15 16:26:03.948 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '44']
2022-08-15 16:26:03.958 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '45']
2022-08-15 16:26:03.968 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '46']
2022-08-15 16:26:03.979 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '47']
2022-08-15 16:26:03.989 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '48']
2022-08-15 16:26:03.999 | INFO | airflow.executors.celery_executor.CeleryExecutor | Adding to queue: ['airflow', 'tasks', 'run', 'test_backfill_races', 't2', 'backfill__2022-01-07T00:00:00-06:00', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/test_backfill_races.py', '--map-index', '49']
2022-08-15 16:26:08.214 | INFO | airflow.jobs.backfill_job.BackfillJob | [backfill progress] | finished run 0 of 5 | tasks waiting: 77 | succeeded: 251 | running: 50 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 77
2022-08-15 16:26:12.635 | INFO | airflow.models.dagrun.DagRun | Marking run <DagRun test_backfill_races @ 2022-01-07T00:00:00-06:00: backfill__2022-01-07T00:00:00-06:00, externally triggered: False> successful
2022-08-15 16:26:12.636 | INFO | airflow.models.dagrun.DagRun | DagRun Finished: dag_id=test_backfill_races, execution_date=2022-01-07T00:00:00-06:00, run_id=backfill__2022-01-07T00:00:00-06:00, run_start_date=2022-08-15 21:25:07.685057+00:00, run_end_date=2022-08-15 21:26:12.636822+00:00, run_duration=64.951765, state=success, external_trigger=False, run_type=backfill, data_interval_start=2022-01-07T00:00:00-06:00, data_interval_end=2022-01-08T00:00:00-06:00, dag_hash=None
2022-08-15 16:26:12.637 | INFO | airflow.jobs.backfill_job.BackfillJob | [backfill progress] | finished run 1 of 5 | tasks waiting: 77 | succeeded: 301 | running: 0 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 77
2022-08-15 16:26:16.971 | WARNING | airflow.jobs.backfill_job.BackfillJob | Deadlock discovered for ti_status.to_run=dict_values([<TaskInstance: test_backfill_races.t2 backfill__2022-01-03T00:00:00-06:00 [scheduled]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-03T00:00:00-06:00 map_index=0 [scheduled]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-04T00:00:00-06:00 map_index=0 [scheduled]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-05T00:00:00-06:00 map_index=0 [scheduled]>, <TaskInstance: test_backfill_races.t1 backfill__2022-01-06T00:00:00-06:00 map_index=0 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-04T00:00:00-06:00 map_index=0 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-04T00:00:00-06:00 map_index=1 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-04T00:00:00-06:00 map_index=2 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-04T00:00:00-06:00 ma
p_index=3 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-04T00:00:00-06:00 map_index=4 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-04T00:00:00-06:00 map_index=5 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-04T00:00:00-06:00 map_index=6 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-04T00:00:00-06:00 map_index=7 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-04T00:00:00-06:00 map_index=8 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-04T00:00:00-06:00 map_index=9 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-04T00:00:00-06:00 map_index=10 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-04T00:00:00-06:00 map_index=11 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-04T00:00:00-06:00 map_index=12 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-04T00:00:00-06
:00 map_index=13 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-04T00:00:00-06:00 map_index=14 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-04T00:00:00-06:00 map_index=15 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-04T00:00:00-06:00 map_index=16 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-04T00:00:00-06:00 map_index=17 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-04T00:00:00-06:00 map_index=18 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-04T00:00:00-06:00 map_index=19 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-04T00:00:00-06:00 map_index=20 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-04T00:00:00-06:00 map_index=21 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-04T00:00:00-06:00 map_index=22 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-0
4T00:00:00-06:00 map_index=23 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-05T00:00:00-06:00 map_index=0 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-05T00:00:00-06:00 map_index=1 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-05T00:00:00-06:00 map_index=2 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-05T00:00:00-06:00 map_index=3 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-05T00:00:00-06:00 map_index=4 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-05T00:00:00-06:00 map_index=5 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-05T00:00:00-06:00 map_index=6 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-05T00:00:00-06:00 map_index=7 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-05T00:00:00-06:00 map_index=8 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-
01-05T00:00:00-06:00 map_index=9 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-06T00:00:00-06:00 map_index=0 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-06T00:00:00-06:00 map_index=1 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-06T00:00:00-06:00 map_index=2 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-06T00:00:00-06:00 map_index=3 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-06T00:00:00-06:00 map_index=4 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-06T00:00:00-06:00 map_index=5 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-06T00:00:00-06:00 map_index=6 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-06T00:00:00-06:00 map_index=7 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-06T00:00:00-06:00 map_index=8 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__20
22-01-06T00:00:00-06:00 map_index=9 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-06T00:00:00-06:00 map_index=10 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-06T00:00:00-06:00 map_index=11 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-06T00:00:00-06:00 map_index=12 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-06T00:00:00-06:00 map_index=13 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-06T00:00:00-06:00 map_index=14 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-06T00:00:00-06:00 map_index=15 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-06T00:00:00-06:00 map_index=16 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-06T00:00:00-06:00 map_index=17 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-06T00:00:00-06:00 map_index=18 [scheduled]>, <TaskInstance: test_backfill_races.t2
backfill__2022-01-06T00:00:00-06:00 map_index=19 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-06T00:00:00-06:00 map_index=20 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-06T00:00:00-06:00 map_index=21 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-06T00:00:00-06:00 map_index=22 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-06T00:00:00-06:00 map_index=23 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-06T00:00:00-06:00 map_index=24 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-06T00:00:00-06:00 map_index=25 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-06T00:00:00-06:00 map_index=26 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-06T00:00:00-06:00 map_index=27 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-06T00:00:00-06:00 map_index=28 [scheduled]>, <TaskInstance: test_backf
ill_races.t2 backfill__2022-01-06T00:00:00-06:00 map_index=29 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-06T00:00:00-06:00 map_index=30 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-06T00:00:00-06:00 map_index=31 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-06T00:00:00-06:00 map_index=32 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-06T00:00:00-06:00 map_index=33 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-06T00:00:00-06:00 map_index=34 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-06T00:00:00-06:00 map_index=35 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-06T00:00:00-06:00 map_index=36 [scheduled]>, <TaskInstance: test_backfill_races.t2 backfill__2022-01-06T00:00:00-06:00 map_index=37 [scheduled]>])
2022-08-15 16:26:16.973 | WARNING | airflow.jobs.backfill_job.BackfillJob | TaskInstanceKey(dag_id='test_backfill_races', task_id='t2', run_id='backfill__2022-01-07T00:00:00-06:00', try_number=1, map_index=13) state success not in running=dict_values([])
2022-08-15 16:26:16.973 | WARNING | airflow.jobs.backfill_job.BackfillJob | TaskInstanceKey(dag_id='test_backfill_races', task_id='t2', run_id='backfill__2022-01-07T00:00:00-06:00', try_number=1, map_index=28) state success not in running=dict_values([])
2022-08-15 16:26:16.973 | WARNING | airflow.jobs.backfill_job.BackfillJob | TaskInstanceKey(dag_id='test_backfill_races', task_id='t2', run_id='backfill__2022-01-07T00:00:00-06:00', try_number=1, map_index=31) state success not in running=dict_values([])
2022-08-15 16:26:16.973 | WARNING | airflow.jobs.backfill_job.BackfillJob | TaskInstanceKey(dag_id='test_backfill_races', task_id='t2', run_id='backfill__2022-01-07T00:00:00-06:00', try_number=1, map_index=33) state success not in running=dict_values([])
2022-08-15 16:26:16.973 | WARNING | airflow.jobs.backfill_job.BackfillJob | TaskInstanceKey(dag_id='test_backfill_races', task_id='t2', run_id='backfill__2022-01-07T00:00:00-06:00', try_number=1, map_index=34) state success not in running=dict_values([])
2022-08-15 16:26:16.973 | WARNING | airflow.jobs.backfill_job.BackfillJob | TaskInstanceKey(dag_id='test_backfill_races', task_id='t2', run_id='backfill__2022-01-07T00:00:00-06:00', try_number=1, map_index=45) state success not in running=dict_values([])
2022-08-15 16:26:16.973 | WARNING | airflow.jobs.backfill_job.BackfillJob | TaskInstanceKey(dag_id='test_backfill_races', task_id='t2', run_id='backfill__2022-01-07T00:00:00-06:00', try_number=1, map_index=46) state success not in running=dict_values([])
2022-08-15 16:26:16.973 | WARNING | airflow.jobs.backfill_job.BackfillJob | TaskInstanceKey(dag_id='test_backfill_races', task_id='t2', run_id='backfill__2022-01-07T00:00:00-06:00', try_number=1, map_index=48) state success not in running=dict_values([])
2022-08-15 16:26:16.973 | WARNING | airflow.jobs.backfill_job.BackfillJob | TaskInstanceKey(dag_id='test_backfill_races', task_id='t2', run_id='backfill__2022-01-07T00:00:00-06:00', try_number=1, map_index=49) state success not in running=dict_values([])
2022-08-15 16:26:17.450 | INFO | airflow.jobs.backfill_job.BackfillJob | [backfill progress] | finished run 1 of 5 | tasks waiting: 0 | succeeded: 301 | running: 0 | failed: 0 | skipped: 0 | deadlocked: 77 | not ready: 77
Traceback (most recent call last):
File "/opt/conda/envs/production/bin/airflow", line 11, in <module>
sys.exit(main())
File "/opt/conda/envs/production/lib/python3.9/site-packages/airflow/__main__.py", line 38, in main
args.func(args)
File "/opt/conda/envs/production/lib/python3.9/site-packages/airflow/cli/cli_parser.py", line 51, in command
return func(*args, **kwargs)
File "/opt/conda/envs/production/lib/python3.9/site-packages/airflow/utils/cli.py", line 99, in wrapper
return f(*args, **kwargs)
File "/opt/conda/envs/production/lib/python3.9/site-packages/airflow/cli/commands/dag_command.py", line 107, in dag_backfill
dag.run(
File "/opt/conda/envs/production/lib/python3.9/site-packages/airflow/models/dag.py", line 2288, in run
job.run()
File "/opt/conda/envs/production/lib/python3.9/site-packages/airflow/jobs/base_job.py", line 244, in run
self._execute()
File "/opt/conda/envs/production/lib/python3.9/site-packages/airflow/utils/session.py", line 71, in wrapper
return func(*args, session=session, **kwargs)
File "/opt/conda/envs/production/lib/python3.9/site-packages/airflow/jobs/backfill_job.py", line 860, in _execute
raise BackfillUnfinished(err, ti_status)
airflow.exceptions.BackfillUnfinished: BackfillJob is deadlocked.
These tasks have succeeded:
DAG ID Task ID Run ID Map Index Try number
------------------- --------- ----------------------------------- ----------- ------------
test_backfill_races get_tasks backfill__2022-01-03T00:00:00-06:00 1 -1
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 1
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 2
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 3
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 4
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 5
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 6
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 7
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 8
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 9
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 10
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 11
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 12
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 13
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 14
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 15
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 16
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 17
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 18
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 19
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 20
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 21
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 22
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 23
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 24
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 25
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 26
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 27
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 28
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 29
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 30
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 31
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 32
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 33
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 34
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 35
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 36
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 37
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 38
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 39
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 40
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 41
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 42
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 43
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 44
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 45
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 46
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 47
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 48
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 49
test_backfill_races get_tasks backfill__2022-01-04T00:00:00-06:00 1 -1
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 1
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 2
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 3
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 4
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 5
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 6
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 7
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 8
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 9
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 10
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 11
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 12
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 13
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 14
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 15
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 16
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 17
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 18
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 19
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 20
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 21
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 22
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 23
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 24
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 25
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 26
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 27
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 28
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 29
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 30
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 31
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 32
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 33
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 34
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 35
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 36
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 37
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 38
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 39
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 40
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 41
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 42
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 43
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 44
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 45
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 46
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 47
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 48
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 49
test_backfill_races get_tasks backfill__2022-01-05T00:00:00-06:00 1 -1
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 1
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 2
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 3
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 4
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 5
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 6
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 7
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 8
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 9
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 10
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 11
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 12
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 13
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 14
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 15
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 16
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 17
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 18
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 19
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 20
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 21
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 22
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 23
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 24
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 25
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 26
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 27
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 28
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 29
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 30
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 31
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 32
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 33
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 34
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 35
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 36
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 37
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 38
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 39
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 40
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 41
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 42
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 43
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 44
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 45
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 46
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 47
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 48
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 49
test_backfill_races get_tasks backfill__2022-01-06T00:00:00-06:00 1 -1
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 1
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 2
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 3
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 4
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 5
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 6
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 7
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 8
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 9
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 10
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 11
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 12
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 13
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 14
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 15
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 16
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 17
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 18
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 19
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 20
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 21
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 22
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 23
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 24
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 25
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 26
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 27
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 28
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 29
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 30
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 31
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 32
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 33
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 34
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 35
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 36
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 37
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 38
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 39
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 40
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 41
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 42
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 43
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 44
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 45
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 46
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 47
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 48
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 49
test_backfill_races get_tasks backfill__2022-01-07T00:00:00-06:00 1 -1
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 0
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 1
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 2
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 3
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 4
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 5
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 6
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 7
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 8
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 9
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 10
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 11
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 12
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 13
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 14
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 15
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 16
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 17
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 18
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 19
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 20
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 21
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 22
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 23
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 24
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 25
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 26
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 27
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 28
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 29
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 30
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 31
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 32
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 33
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 34
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 35
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 36
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 37
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 38
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 39
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 40
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 41
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 42
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 43
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 44
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 45
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 46
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 47
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 48
test_backfill_races t1 backfill__2022-01-07T00:00:00-06:00 1 49
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 0
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 1
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 2
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 3
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 4
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 5
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 6
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 7
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 8
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 9
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 10
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 11
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 12
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 13
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 14
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 15
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 16
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 17
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 18
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 19
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 20
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 21
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 22
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 23
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 24
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 25
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 26
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 27
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 28
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 29
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 30
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 31
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 32
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 33
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 34
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 35
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 36
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 37
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 38
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 39
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 40
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 41
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 42
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 43
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 44
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 45
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 46
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 47
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 48
test_backfill_races t2 backfill__2022-01-07T00:00:00-06:00 1 49
These tasks are running:
DAG ID Task ID Run ID Try number
-------- --------- -------- ------------
These tasks have failed:
DAG ID Task ID Run ID Try number
-------- --------- -------- ------------
These tasks are skipped:
DAG ID Task ID Run ID Try number
-------- --------- -------- ------------
These tasks are deadlocked:
DAG ID Task ID Run ID Map Index Try number
------------------- --------- ----------------------------------- ----------- ------------
test_backfill_races t1 backfill__2022-01-03T00:00:00-06:00 1 0
test_backfill_races t2 backfill__2022-01-03T00:00:00-06:00 1 -1
test_backfill_races t1 backfill__2022-01-04T00:00:00-06:00 1 0
test_backfill_races t2 backfill__2022-01-04T00:00:00-06:00 1 0
test_backfill_races t2 backfill__2022-01-04T00:00:00-06:00 1 1
test_backfill_races t2 backfill__2022-01-04T00:00:00-06:00 1 2
test_backfill_races t2 backfill__2022-01-04T00:00:00-06:00 1 3
test_backfill_races t2 backfill__2022-01-04T00:00:00-06:00 1 4
test_backfill_races t2 backfill__2022-01-04T00:00:00-06:00 1 5
test_backfill_races t2 backfill__2022-01-04T00:00:00-06:00 1 6
test_backfill_races t2 backfill__2022-01-04T00:00:00-06:00 1 7
test_backfill_races t2 backfill__2022-01-04T00:00:00-06:00 1 8
test_backfill_races t2 backfill__2022-01-04T00:00:00-06:00 1 9
test_backfill_races t2 backfill__2022-01-04T00:00:00-06:00 1 10
test_backfill_races t2 backfill__2022-01-04T00:00:00-06:00 1 11
test_backfill_races t2 backfill__2022-01-04T00:00:00-06:00 1 12
test_backfill_races t2 backfill__2022-01-04T00:00:00-06:00 1 13
test_backfill_races t2 backfill__2022-01-04T00:00:00-06:00 1 14
test_backfill_races t2 backfill__2022-01-04T00:00:00-06:00 1 15
test_backfill_races t2 backfill__2022-01-04T00:00:00-06:00 1 16
test_backfill_races t2 backfill__2022-01-04T00:00:00-06:00 1 17
test_backfill_races t2 backfill__2022-01-04T00:00:00-06:00 1 18
test_backfill_races t2 backfill__2022-01-04T00:00:00-06:00 1 19
test_backfill_races t2 backfill__2022-01-04T00:00:00-06:00 1 20
test_backfill_races t2 backfill__2022-01-04T00:00:00-06:00 1 21
test_backfill_races t2 backfill__2022-01-04T00:00:00-06:00 1 22
test_backfill_races t2 backfill__2022-01-04T00:00:00-06:00 1 23
test_backfill_races t1 backfill__2022-01-05T00:00:00-06:00 1 0
test_backfill_races t2 backfill__2022-01-05T00:00:00-06:00 1 0
test_backfill_races t2 backfill__2022-01-05T00:00:00-06:00 1 1
test_backfill_races t2 backfill__2022-01-05T00:00:00-06:00 1 2
test_backfill_races t2 backfill__2022-01-05T00:00:00-06:00 1 3
test_backfill_races t2 backfill__2022-01-05T00:00:00-06:00 1 4
test_backfill_races t2 backfill__2022-01-05T00:00:00-06:00 1 5
test_backfill_races t2 backfill__2022-01-05T00:00:00-06:00 1 6
test_backfill_races t2 backfill__2022-01-05T00:00:00-06:00 1 7
test_backfill_races t2 backfill__2022-01-05T00:00:00-06:00 1 8
test_backfill_races t2 backfill__2022-01-05T00:00:00-06:00 1 9
test_backfill_races t1 backfill__2022-01-06T00:00:00-06:00 1 0
test_backfill_races t2 backfill__2022-01-06T00:00:00-06:00 1 0
test_backfill_races t2 backfill__2022-01-06T00:00:00-06:00 1 1
test_backfill_races t2 backfill__2022-01-06T00:00:00-06:00 1 2
test_backfill_races t2 backfill__2022-01-06T00:00:00-06:00 1 3
test_backfill_races t2 backfill__2022-01-06T00:00:00-06:00 1 4
test_backfill_races t2 backfill__2022-01-06T00:00:00-06:00 1 5
test_backfill_races t2 backfill__2022-01-06T00:00:00-06:00 1 6
test_backfill_races t2 backfill__2022-01-06T00:00:00-06:00 1 7
test_backfill_races t2 backfill__2022-01-06T00:00:00-06:00 1 8
test_backfill_races t2 backfill__2022-01-06T00:00:00-06:00 1 9
test_backfill_races t2 backfill__2022-01-06T00:00:00-06:00 1 10
test_backfill_races t2 backfill__2022-01-06T00:00:00-06:00 1 11
test_backfill_races t2 backfill__2022-01-06T00:00:00-06:00 1 12
test_backfill_races t2 backfill__2022-01-06T00:00:00-06:00 1 13
test_backfill_races t2 backfill__2022-01-06T00:00:00-06:00 1 14
test_backfill_races t2 backfill__2022-01-06T00:00:00-06:00 1 15
test_backfill_races t2 backfill__2022-01-06T00:00:00-06:00 1 16
test_backfill_races t2 backfill__2022-01-06T00:00:00-06:00 1 17
test_backfill_races t2 backfill__2022-01-06T00:00:00-06:00 1 18
test_backfill_races t2 backfill__2022-01-06T00:00:00-06:00 1 19
test_backfill_races t2 backfill__2022-01-06T00:00:00-06:00 1 20
test_backfill_races t2 backfill__2022-01-06T00:00:00-06:00 1 21
test_backfill_races t2 backfill__2022-01-06T00:00:00-06:00 1 22
test_backfill_races t2 backfill__2022-01-06T00:00:00-06:00 1 23
test_backfill_races t2 backfill__2022-01-06T00:00:00-06:00 1 24
test_backfill_races t2 backfill__2022-01-06T00:00:00-06:00 1 25
test_backfill_races t2 backfill__2022-01-06T00:00:00-06:00 1 26
test_backfill_races t2 backfill__2022-01-06T00:00:00-06:00 1 27
test_backfill_races t2 backfill__2022-01-06T00:00:00-06:00 1 28
test_backfill_races t2 backfill__2022-01-06T00:00:00-06:00 1 29
test_backfill_races t2 backfill__2022-01-06T00:00:00-06:00 1 30
test_backfill_races t2 backfill__2022-01-06T00:00:00-06:00 1 31
test_backfill_races t2 backfill__2022-01-06T00:00:00-06:00 1 32
test_backfill_races t2 backfill__2022-01-06T00:00:00-06:00 1 33
test_backfill_races t2 backfill__2022-01-06T00:00:00-06:00 1 34
test_backfill_races t2 backfill__2022-01-06T00:00:00-06:00 1 35
test_backfill_races t2 backfill__2022-01-06T00:00:00-06:00 1 36
test_backfill_races t2 backfill__2022-01-06T00:00:00-06:00 1 37
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org