You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/01/25 18:00:24 UTC
[GitHub] [airflow] roldugin opened a new issue #12959: Skipped tasks deadlock when cleared
roldugin opened a new issue #12959:
URL: https://github.com/apache/airflow/issues/12959
**Apache Airflow version**: 1.10.12 (regression, *NOT* reproducible on 1.10.5)
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`): N/A
**Environment**: local with SequentialExecutor + deployed with CeleryExecutor on Kubernetes
- **Cloud provider or hardware configuration**:
- **OS** (e.g. from /etc/os-release): macOS 10.15
- **Kernel** (e.g. `uname -a`): Darwin macmac 19.6.0 Darwin Kernel Version 19.6.0: Sun Jul 5 00:43:10 PDT 2020; root:xnu-6153.141.1~9/RELEASE_X86_64 x86_64
- **Install tools**:
- **Others**:
**What happened**:
Clearing the `SKIPPED` "dummy" task in the following DAG deadlocks it and returns it to the `SKIPPED` state.
<img width="298" alt="dag" src="https://user-images.githubusercontent.com/749698/101660840-e8ef0680-3a47-11eb-81e4-689c27a62113.png">
**What you expected to happen**:
I expect the "dummy" task to run and succeed. This is the behaviour in Airflow 1.10.5 and before.
**How to reproduce it**:
```python
from datetime import datetime
import airflow
from airflow.operators.dummy_operator import DummyOperator
from airflow.operators.python_operator import ShortCircuitOperator
with airflow.DAG(
dag_id='test_dag',
start_date=datetime(2020, 1, 1),
schedule_interval="@monthly",
catchup=False,
) as dag:
short_circuit_returning_false = ShortCircuitOperator(
task_id='short_circuit_returning_false',
python_callable=lambda **kwargs: False)
dummy = DummyOperator(task_id="dummy")
short_circuit_returning_false >> dummy
```
```
airflow backfill test_dag -s 2020-01-01 -e 2020-01-01 # Backfill the DAG - dummy task will be skipped as expected
airflow clear test_dag -t dummy -s 2020-01-01 -e 2020-01-01 # Crear the dummy task
airflow backfill test_dag -s 2020-01-01 -e 2020-01-01 # Backfill the DAG again - dummy task deadlocks
```
**Detailed logs**:
<details><summary>deadlock.log</summary>
$ airflow backfill test_dag -s 2020-01-01 -e 2020-01-01
[2020-12-09 17:36:03,580] {__init__.py:50} INFO - Using executor SequentialExecutor
[2020-12-09 17:36:03,580] {dagbag.py:417} INFO - Filling up the DagBag from /Users/grol/dev/sc-airflow-dags/dags
[2020-12-09 17:36:21,409] {base_executor.py:58} INFO - Adding to queue: ['airflow', 'run', 'test_dag', 'short_circuit_returning_false', '2020-01-01T00:00:00+00:00', '--local', '--pool', 'default_pool', '-sd', '/Users/grol/dev/sc-airflow-dags/dags/rnr/test_dag.py', '--cfg_path', '/var/folders/1z/s9zsm2hx4sj5c674rvkn6q9c0000gp/T/tmppn6x5bv3']
[2020-12-09 17:36:26,342] {sequential_executor.py:54} INFO - Executing command: ['airflow', 'run', 'test_dag', 'short_circuit_returning_false', '2020-01-01T00:00:00+00:00', '--local', '--pool', 'default_pool', '-sd', '/Users/grol/dev/sc-airflow-dags/dags/rnr/test_dag.py', '--cfg_path', '/var/folders/1z/s9zsm2hx4sj5c674rvkn6q9c0000gp/T/tmppn6x5bv3']
[2020-12-09 17:36:28,616] {__init__.py:50} INFO - Using executor SequentialExecutor
[2020-12-09 17:36:28,617] {dagbag.py:417} INFO - Filling up the DagBag from /Users/grol/dev/sc-airflow-dags/dags/rnr/test_dag.py
Running <TaskInstance: test_dag.short_circuit_returning_false 2020-01-01T00:00:00+00:00 [queued]> on host macmac
[2020-12-09 17:36:33,847] {dagrun.py:320} INFO - Marking run <DagRun test_dag @ 2020-01-01 00:00:00+00:00: backfill_2020-01-01T00:00:00+00:00, externally triggered: False> successful
[2020-12-09 17:36:33,852] {backfill_job.py:359} INFO - [backfill progress] | finished run 1 of 1 | tasks waiting: 1 | succeeded: 1 | running: 0 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 1
[2020-12-09 17:36:33,864] {backfill_job.py:359} INFO - [backfill progress] | finished run 1 of 1 | tasks waiting: 0 | succeeded: 1 | running: 0 | failed: 0 | skipped: 1 | deadlocked: 0 | not ready: 0
[2020-12-09 17:36:33,864] {backfill_job.py:813} INFO - Backfill done. Exiting.
$ airflow clear test_dag -t dummy -s 2020-01-01 -e 2020-01-01
[2020-12-09 17:36:45,965] {__init__.py:50} INFO - Using executor SequentialExecutor
[2020-12-09 17:36:45,965] {dagbag.py:417} INFO - Filling up the DagBag from /Users/grol/dev/sc-airflow-dags/dags
You are about to delete these 1 tasks:
<TaskInstance: test_dag.dummy 2020-01-01 00:00:00+00:00 [skipped]>
Are you sure? (yes/no):
yes
$ airflow backfill test_dag -s 2020-01-01 -e 2020-01-01
[2020-12-09 17:37:14,115] {__init__.py:50} INFO - Using executor SequentialExecutor
[2020-12-09 17:37:14,116] {dagbag.py:417} INFO - Filling up the DagBag from /Users/grol/dev/sc-airflow-dags/dags
[2020-12-09 17:37:34,878] {backfill_job.py:600} WARNING - Deadlock discovered for ti_status.to_run=odict_values([<TaskInstance: test_dag.dummy 2020-01-01 00:00:00+00:00 [skipped]>])
[2020-12-09 17:37:34,881] {dagrun.py:320} INFO - Marking run <DagRun test_dag @ 2020-01-01 00:00:00+00:00: backfill_2020-01-01T00:00:00+00:00, externally triggered: False> successful
[2020-12-09 17:37:34,884] {backfill_job.py:359} INFO - [backfill progress] | finished run 1 of 1 | tasks waiting: 0 | succeeded: 1 | running: 0 | failed: 0 | skipped: 0 | deadlocked: 1 | not ready: 1
Traceback (most recent call last):
File "/Users/grol/dev/airflow-dev/venv/bin/airflow", line 37, in <module>
args.func(args)
File "/Users/grol/dev/airflow-dev/venv/lib/python3.8/site-packages/airflow/utils/cli.py", line 76, in wrapper
return f(*args, **kwargs)
File "/Users/grol/dev/airflow-dev/venv/lib/python3.8/site-packages/airflow/bin/cli.py", line 222, in backfill
dag.run(
File "/Users/grol/dev/airflow-dev/venv/lib/python3.8/site-packages/airflow/models/dag.py", line 1432, in run
job.run()
File "/Users/grol/dev/airflow-dev/venv/lib/python3.8/site-packages/airflow/jobs/base_job.py", line 218, in run
self._execute()
File "/Users/grol/dev/airflow-dev/venv/lib/python3.8/site-packages/airflow/utils/db.py", line 74, in wrapper
return func(*args, **kwargs)
File "/Users/grol/dev/airflow-dev/venv/lib/python3.8/site-packages/airflow/jobs/backfill_job.py", line 794, in _execute
raise AirflowException(err)
airflow.exceptions.AirflowException: BackfillJob is deadlocked.
These tasks have succeeded:
DAG ID Task ID Execution date Try number
-------- ----------------------------- ------------------------- ------------
test_dag short_circuit_returning_false 2020-01-01 00:00:00+00:00 5
These tasks are running:
DAG ID Task ID Execution date Try number
-------- --------- ---------------- ------------
These tasks have failed:
DAG ID Task ID Execution date Try number
-------- --------- ---------------- ------------
These tasks are skipped:
DAG ID Task ID Execution date Try number
-------- --------- ---------------- ------------
These tasks are deadlocked:
DAG ID Task ID Execution date Try number
-------- --------- ------------------------- ------------
test_dag dummy 2020-01-01 00:00:00+00:00 1
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #12959: Skipped tasks deadlock when cleared
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #12959:
URL: https://github.com/apache/airflow/issues/12959#issuecomment-851727541
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] roldugin commented on issue #12959: Skipped tasks deadlock when cleared
Posted by GitBox <gi...@apache.org>.
roldugin commented on issue #12959:
URL: https://github.com/apache/airflow/issues/12959#issuecomment-767014395
> This is not a regression. It's by design.
> See 1.10.12 [change log](https://github.com/apache/airflow/blob/fef9173ac4ba02ff5fd8f4dfdd2bfe5cd483bd25/CHANGELOG.txt#L176) and [updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md#clearing-tasks-skipped-by-skipmixin-will-skip-them)
>
> Pull request: #8992
@eladkal thanks, I think this explains the issue I was having. It's not very obvious from the log that it's intended behaviour.
We use `ShortCircuitOperator` to model a pause in the process, waiting for a user to decide when to continue. Can you think of an alternative solution that you'd recommend?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Mofef edited a comment on issue #12959: Skipped tasks deadlock when cleared
Posted by GitBox <gi...@apache.org>.
Mofef edited a comment on issue #12959:
URL: https://github.com/apache/airflow/issues/12959#issuecomment-766958303
@eladkal did you read the error log, that @roldugin posted? Is this also intended (quoting): `airflow.exceptions.AirflowException: BackfillJob is deadlocked.` because it sounds like its not.
I agree that skipped jobs should be skipped again after clearing.
(However, the issue I had, produced the same message and was introduced in 1.10.10 not only in 1.10.12, but i can't prove it's the same issue so thats just a side note)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #12959: Skipped tasks deadlock when cleared
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #12959:
URL: https://github.com/apache/airflow/issues/12959#issuecomment-835583380
This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ernest-kr commented on issue #12959: Skipped tasks deadlock when cleared
Posted by GitBox <gi...@apache.org>.
ernest-kr commented on issue #12959:
URL: https://github.com/apache/airflow/issues/12959#issuecomment-766981324
..
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Mofef edited a comment on issue #12959: Skipped tasks deadlock when cleared
Posted by GitBox <gi...@apache.org>.
Mofef edited a comment on issue #12959:
URL: https://github.com/apache/airflow/issues/12959#issuecomment-766958303
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal edited a comment on issue #12959: Skipped tasks deadlock when cleared
Posted by GitBox <gi...@apache.org>.
eladkal edited a comment on issue #12959:
URL: https://github.com/apache/airflow/issues/12959#issuecomment-766982930
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ernest-kr removed a comment on issue #12959: Skipped tasks deadlock when cleared
Posted by GitBox <gi...@apache.org>.
ernest-kr removed a comment on issue #12959:
URL: https://github.com/apache/airflow/issues/12959#issuecomment-766981324
..
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ernest-kr removed a comment on issue #12959: Skipped tasks deadlock when cleared
Posted by GitBox <gi...@apache.org>.
ernest-kr removed a comment on issue #12959:
URL: https://github.com/apache/airflow/issues/12959#issuecomment-766981324
..
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Mofef commented on issue #12959: Skipped tasks deadlock when cleared
Posted by GitBox <gi...@apache.org>.
Mofef commented on issue #12959:
URL: https://github.com/apache/airflow/issues/12959#issuecomment-766958303
@eladkal did you read the error log, that @roldugin posted? Is this also intended (quoting): `airflow.exceptions.AirflowException: BackfillJob is deadlocked.` because it sounds like its not.
I agree that skipped jobs should be skipped again after clearing.
(However, the issue I had produced the same message and was introduced in 1.10.10 not only in 1.10.12, but i can't prove its related so thats just a side note)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Mofef edited a comment on issue #12959: Skipped tasks deadlock when cleared
Posted by GitBox <gi...@apache.org>.
Mofef edited a comment on issue #12959:
URL: https://github.com/apache/airflow/issues/12959#issuecomment-758083601
I had an issue with deadlocks of subdags (that contain skipped tasks) on airflow 1.10.12 on composer-1.13.2.
Thanks to your post I tried 1.10.9. and there the problem is gone. Did you test any other versions between 1.10.5 and 1.10.12?
Edit: I might have been a bit too fast with drawing a connection between the issues, because I actually can't reproduce your issue with 1.10.12 :/
Edit2: Fact is, my issue is also causing deadlocks and occurs on 1.10.10 and 1.10.12 but not on 1.10.9. I will open a separate ticket, when I have a minimal reproducible example
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Mofef edited a comment on issue #12959: Skipped tasks deadlock when cleared
Posted by GitBox <gi...@apache.org>.
Mofef edited a comment on issue #12959:
URL: https://github.com/apache/airflow/issues/12959#issuecomment-758083601
I had an issue with deadlocks of subdags (that contain skipped tasks) on airflow 1.10.12 on composer-1.13.2.
Thanks to your post I tried 1.10.9. and there the problem is gone. Did you test any other versions between 1.10.5 and 1.10.12?
Edit: I might have been a bit too fast with drawing a connection between the issues, because I actually can't reproduce your issue with 1.10.12 :/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #12959: Skipped tasks deadlock when cleared
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #12959:
URL: https://github.com/apache/airflow/issues/12959#issuecomment-766982930
Hi @Mofef you commented that you opened a separate issue for your report which is a different problem.
If you believe this issue is relevant please explain further the case. We need something to work with otherwise it's impossible to verify the issue
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #12959: Skipped tasks deadlock when cleared
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #12959:
URL: https://github.com/apache/airflow/issues/12959#issuecomment-766753175
This is not a regression. It's by design.
See 1.10.12 [change log](https://github.com/apache/airflow/blob/fef9173ac4ba02ff5fd8f4dfdd2bfe5cd483bd25/CHANGELOG.txt#L176) and [updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md#clearing-tasks-skipped-by-skipmixin-will-skip-them)
Pull request: https://github.com/apache/airflow/pull/8992
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal edited a comment on issue #12959: Skipped tasks deadlock when cleared
Posted by GitBox <gi...@apache.org>.
eladkal edited a comment on issue #12959:
URL: https://github.com/apache/airflow/issues/12959#issuecomment-766982930
Hi @Mofef you commented that you opened a separate issue for your report, didn't you?
Can you please see if you get the same error when testing against 2.0? it may give indication if we still have a problem even if no exact reproduce steps given
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #12959: Skipped tasks deadlock when cleared
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #12959:
URL: https://github.com/apache/airflow/issues/12959#issuecomment-812945496
This issue has been automatically marked as stale because it has been open for 15 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ernest-kr commented on issue #12959: Skipped tasks deadlock when cleared
Posted by GitBox <gi...@apache.org>.
ernest-kr commented on issue #12959:
URL: https://github.com/apache/airflow/issues/12959#issuecomment-766981324
..
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bicepjai commented on issue #12959: Skipped tasks deadlock when cleared
Posted by GitBox <gi...@apache.org>.
bicepjai commented on issue #12959:
URL: https://github.com/apache/airflow/issues/12959#issuecomment-815211072
I face the same issue. solutions suggested [here](https://godatadriven.com/blog/the-zen-of-python-and-apache-airflow/) for skipping execution of tasks; creates
```
airflow.exceptions.AirflowException: BackfillJob is deadlocked.
```
using airflow version `1.10.10`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Mofef edited a comment on issue #12959: Skipped tasks deadlock when cleared
Posted by GitBox <gi...@apache.org>.
Mofef edited a comment on issue #12959:
URL: https://github.com/apache/airflow/issues/12959#issuecomment-766958303
@eladkal did you read the error log, that @roldugin posted? Is this also intended (quoting): `airflow.exceptions.AirflowException: BackfillJob is deadlocked.` because it sounds like its not.
I agree that skipped jobs should be skipped again after clearing.
(However, the issue I had, produced the same message and was introduced in 1.10.10 not only in 1.10.12, but i can't prove its related so thats just a side note)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] roldugin commented on issue #12959: Skipped tasks deadlock when cleared
Posted by GitBox <gi...@apache.org>.
roldugin commented on issue #12959:
URL: https://github.com/apache/airflow/issues/12959#issuecomment-767014395
> This is not a regression. It's by design.
> See 1.10.12 [change log](https://github.com/apache/airflow/blob/fef9173ac4ba02ff5fd8f4dfdd2bfe5cd483bd25/CHANGELOG.txt#L176) and [updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md#clearing-tasks-skipped-by-skipmixin-will-skip-them)
>
> Pull request: #8992
@eladkal thanks, I think this explains the issue I was having. It's not very obvious from the log that it's intended behaviour.
We use `ShortCircuitOperator` to model a pause in the process, waiting for a user to decide when to continue. Can you think of an alternative solution that you'd recommend?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #12959: Skipped tasks deadlock when cleared
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #12959:
URL: https://github.com/apache/airflow/issues/12959#issuecomment-741908776
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal closed issue #12959: Skipped tasks deadlock when cleared
Posted by GitBox <gi...@apache.org>.
eladkal closed issue #12959:
URL: https://github.com/apache/airflow/issues/12959
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] closed issue #12959: Skipped tasks deadlock when cleared
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed issue #12959:
URL: https://github.com/apache/airflow/issues/12959
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #12959: Skipped tasks deadlock when cleared
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #12959:
URL: https://github.com/apache/airflow/issues/12959#issuecomment-766753175
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Mofef commented on issue #12959: Skipped tasks deadlock when cleared
Posted by GitBox <gi...@apache.org>.
Mofef commented on issue #12959:
URL: https://github.com/apache/airflow/issues/12959#issuecomment-766958303
@eladkal did you read the error log, that @roldugin posted? Is this also intended (quoting): `airflow.exceptions.AirflowException: BackfillJob is deadlocked.` because it sounds like its not.
I agree that skipped jobs should be skipped again after clearing.
(However, the issue I had produced the same message and was introduced in 1.10.10 not only in 1.10.12, but i can't prove its related so thats just a side note)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Mofef commented on issue #12959: Skipped tasks deadlock when cleared
Posted by GitBox <gi...@apache.org>.
Mofef commented on issue #12959:
URL: https://github.com/apache/airflow/issues/12959#issuecomment-758083601
I had an issue with deadlocks of subdags (that contain skipped tasks) on airflow 1.10.12 on composer-1.13.2.
Thanks to your post I tried 1.10.9. and there the problem is gone. Did you test any other versions between 1.10.5 and 1.10.12?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal closed issue #12959: Skipped tasks deadlock when cleared
Posted by GitBox <gi...@apache.org>.
eladkal closed issue #12959:
URL: https://github.com/apache/airflow/issues/12959
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal edited a comment on issue #12959: Skipped tasks deadlock when cleared
Posted by GitBox <gi...@apache.org>.
eladkal edited a comment on issue #12959:
URL: https://github.com/apache/airflow/issues/12959#issuecomment-766982930
Hi @Mofef you commented that you opened a separate issue for your report, didn't you?
Can you please see if you get the same error when testing against 2.0?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org