You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "tanelk (via GitHub)" <gi...@apache.org> on 2023/08/07 07:02:21 UTC
[GitHub] [airflow] tanelk opened a new issue, #33164: Removed mapped tasks block all_success trigger rule
tanelk opened a new issue, #33164:
URL: https://github.com/apache/airflow/issues/33164
### Apache Airflow version
2.6.3
### What happened
When rerunning a DAG run with dynamically mapped tasks and the number of mapped task instances degreases, then downstream tasks with all_success trigger rule (likely some others as well) will not get scheduled, because removed status is not considered to be successful. When a regular task gets removed, then this does not happen, because it will get removed from the DAG structure.
Task dependency message
```Task's trigger rule 'all_success' requires all upstream tasks to have succeeded, but found 2 non-success(es). upstream_states=_UpstreamTIStates(success=22, skipped=0, failed=0, upstream_failed=0, removed=2, done=24), upstream_task_ids={'upstream_task'}```
Previously you could forcefully run it with the "run" button in UI, but it has been removed in one of the resent releases.
Luckily I could delete the removed TIs and then the downstream tasks got rescheduled, but this is not "scalable".
### What you think should happen instead
_No response_
### How to reproduce
-
### Operating System
-
### Versions of Apache Airflow Providers
_No response_
### Deployment
Virtualenv installation
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tanelk commented on issue #33164: Removed mapped tasks block all_success trigger rule
Posted by "tanelk (via GitHub)" <gi...@apache.org>.
tanelk commented on issue #33164:
URL: https://github.com/apache/airflow/issues/33164#issuecomment-1671103047
Example DAG
```import pendulum
from airflow import DAG
from airflow.decorators import task
from airflow.operators.empty import EmptyOperator
with DAG(
'removed_mapped_tasks',
schedule='@daily',
start_date=pendulum.DateTime(2023, 8, 7),
) as dag:
@task
def gen_elements():
return [1, 2, 3]
@task
def mapped_task(element):
return element * 2
mapped_task.expand(element=gen_elements()) >> EmptyOperator(task_id='end')
```
Let it complete and then return one less element from the `gen_elements` task. Then clear the last DAG run.
The `end` task will not get scheduled because `Task's trigger rule 'all_success' requires all upstream tasks to have succeeded, but found 1 non-success(es). upstream_states=_UpstreamTIStates(success=2, skipped=0, failed=0, upstream_failed=0, removed=1, done=3), upstream_task_ids={'mapped_task'}`
In this very simple DAG, the run will be failed with ` scheduler | [2023-08-09T13:49:56.448+0300] {dagrun.py:651} ERROR - Task deadlock (no runnable tasks); marking run <DagRun removed_mapped_tasks @ 2023-08-06 21:00:00+00:00: scheduled__2023-08-06T21:00:00+00:00, state:running, queued_at: 2023-08-09 10:49:44.110567+00:00. externally triggered: False> failed`
On more complex structures the deadlock detection might not kick in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Removed mapped tasks block all_success trigger rule [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 closed issue #33164: Removed mapped tasks block all_success trigger rule
URL: https://github.com/apache/airflow/issues/33164
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Removed mapped tasks block all_success trigger rule [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on issue #33164:
URL: https://github.com/apache/airflow/issues/33164#issuecomment-2254513051
> Example DAG
>
> ```
> from airflow import DAG
> from airflow.decorators import task
> from airflow.operators.empty import EmptyOperator
>
> with DAG(
> 'removed_mapped_tasks',
> schedule='@daily',
> start_date=pendulum.DateTime(2023, 8, 7),
> ) as dag:
>
> @task
> def gen_elements():
> return [1, 2, 3]
>
> @task
> def mapped_task(element):
> return element * 2
>
> mapped_task.expand(element=gen_elements()) >> EmptyOperator(task_id='end')
> ```
>
> Let it complete and then return one less element from the `gen_elements` task. Then clear the last DAG run.
>
> The `end` task will not get scheduled because `Task's trigger rule 'all_success' requires all upstream tasks to have succeeded, but found 1 non-success(es). upstream_states=_UpstreamTIStates(success=2, skipped=0, failed=0, upstream_failed=0, removed=1, done=3), upstream_task_ids={'mapped_task'}`
>
> In this very simple DAG, the run will be failed with ` scheduler | [2023-08-09T13:49:56.448+0300] {dagrun.py:651} ERROR - Task deadlock (no runnable tasks); marking run <DagRun removed_mapped_tasks @ 2023-08-06 21:00:00+00:00: scheduled__2023-08-06T21:00:00+00:00, state:running, queued_at: 2023-08-09 10:49:44.110567+00:00. externally triggered: False> failed`
>
> On more complex structures the deadlock detection might not kick in.
> Example DAG
>
> ```
> from airflow import DAG
> from airflow.decorators import task
> from airflow.operators.empty import EmptyOperator
>
> with DAG(
> 'removed_mapped_tasks',
> schedule='@daily',
> start_date=pendulum.DateTime(2023, 8, 7),
> ) as dag:
>
> @task
> def gen_elements():
> return [1, 2, 3]
>
> @task
> def mapped_task(element):
> return element * 2
>
> mapped_task.expand(element=gen_elements()) >> EmptyOperator(task_id='end')
> ```
>
> Let it complete and then return one less element from the `gen_elements` task. Then clear the last DAG run.
>
> The `end` task will not get scheduled because `Task's trigger rule 'all_success' requires all upstream tasks to have succeeded, but found 1 non-success(es). upstream_states=_UpstreamTIStates(success=2, skipped=0, failed=0, upstream_failed=0, removed=1, done=3), upstream_task_ids={'mapped_task'}`
>
> In this very simple DAG, the run will be failed with ` scheduler | [2023-08-09T13:49:56.448+0300] {dagrun.py:651} ERROR - Task deadlock (no runnable tasks); marking run <DagRun removed_mapped_tasks @ 2023-08-06 21:00:00+00:00: scheduled__2023-08-06T21:00:00+00:00, state:running, queued_at: 2023-08-09 10:49:44.110567+00:00. externally triggered: False> failed`
>
> On more complex structures the deadlock detection might not kick in.
I tried to run this example with the v2.9.3 (first run with original DAG, remove one element, and clear one DAG run) and I didn't manage to reproduce it - all tasks succeeded.
I'm closing this issue, please create a new one if you encounter this again in versions 2.9.3+ (while providing reproducible examples).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RNHTTR commented on issue #33164: Removed mapped tasks block all_success trigger rule
Posted by "RNHTTR (via GitHub)" <gi...@apache.org>.
RNHTTR commented on issue #33164:
URL: https://github.com/apache/airflow/issues/33164#issuecomment-1669754223
Can you please provide reproduction steps?
Also, what do you mean by the following:
> the number of mapped task instances decreases
How are the number of mapped task instances decreasing during a DAG run?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Removed mapped tasks block all_success trigger rule [airflow]
Posted by "nick-brady (via GitHub)" <gi...@apache.org>.
nick-brady commented on issue #33164:
URL: https://github.com/apache/airflow/issues/33164#issuecomment-2313219422
I don't have time to reproduce, but this happened to me on Airflow 2.6.2. I manually delete the removed task for the time being until I can try this on a newer version
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dzhigimont commented on issue #33164: Removed mapped tasks block all_success trigger rule
Posted by "dzhigimont (via GitHub)" <gi...@apache.org>.
dzhigimont commented on issue #33164:
URL: https://github.com/apache/airflow/issues/33164#issuecomment-1668155728
@tanelk Could you please provide an example of DAG?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org