You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "vlieven (via GitHub)" <gi...@apache.org> on 2024/01/16 11:44:19 UTC
[I] ShortCircuitOperator does not properly retry when python_callable exits unexpectedly [airflow]
vlieven opened a new issue, #36809:
URL: https://github.com/apache/airflow/issues/36809
### Apache Airflow version
Other Airflow 2 version (please specify below)
### If "Other Airflow 2 version" selected, which one?
2.7.3
### What happened?
If the `python_callable` used by a ShortCircuitOperator does not properly return a boolean, but instead crashes, the retry and downstream behavior is not deterministic. Either:
- the task properly retries, succeeds and downstream tasks are triggered
- the task does not properly retry, downstream tasks are marked as upstream_failed
- the task does retry, is not marked as success, but downstream tasks are triggered
### What you think should happen instead?
In case the callable crashes, it should be treated as a failure, and retry logic should be properly executed until the task completes successfully.
### How to reproduce
As the behavior is not deterministic, I've created 10 parallel flows to demonstrate the issue. A crash is simulated by `exit(-1)`.
This DAG file should demonstrate the broken behavior:
```python
from airflow import DAG
from airflow.models.taskinstance import TaskInstance
from airflow.operators.empty import EmptyOperator
from airflow.operators.python import ShortCircuitOperator
from datetime import datetime, timedelta
default_args = {
"owner": "Test",
"depends_on_past": False,
"start_date": datetime.now() - timedelta(days=2),
"retries": 1,
"retry_delay": timedelta(seconds=5),
}
def python_callable(ti: TaskInstance):
if ti.try_number % 2 == 1:
exit(-1)
else:
return True
with DAG(
"test_shortcircuit",
default_args=default_args,
schedule_interval="@daily",
) as test_dag:
for i in range(10):
ShortCircuitOperator(
task_id=f"short-{i}", python_callable=python_callable,
) >> EmptyOperator(
task_id=f"empty-{i}"
)
```
The resulting grid view is the following:
<img width="514" alt="Screenshot 2024-01-16 at 09 14 08" src="https://github.com/apache/airflow/assets/16362535/69fa9e39-ec0f-4daf-a7e7-163b670fc65c">
### Operating System
Amazon Linux 2
### Versions of Apache Airflow Providers
This is using only the built-in operators
### Deployment
Other Docker-based deployment
### Deployment details
Running on a Kubernetes cluster
### Anything else?
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ShortCircuitOperator does not properly retry when python_callable exits unexpectedly [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on issue #36809:
URL: https://github.com/apache/airflow/issues/36809#issuecomment-1895499822
I check on the `main` and this kind of bug exists.
However it not only affects `ShortCircuitOperator`, I reproduced on regular task flow operators
```python
from __future__ import annotations
from typing import TYPE_CHECKING
from airflow import DAG
from airflow.decorators import task
from airflow.utils.task_group import TaskGroup
if TYPE_CHECKING:
from airflow.models import TaskInstance
with DAG(dag_id="issue_36809", schedule=None, tags=["issue", "36809", "zombie?"]):
@task(retries=1, retry_delay=5)
def flakey_failure(ti: TaskInstance = None) -> int:
if (try_number := ti.try_number) == 1:
raise SystemExit(-1)
return try_number
@task
def my_task(i: int):
print(f"Upstream try number: {i}")
for ix in range(1, 10 + 1):
with TaskGroup(group_id=f"group_{ix:02d}"):
my_task(flakey_failure())
```
Seems like the problem with zombie detection/resolution. In addition found interesting bug, on next DagRun when task failed it also might marks (with high probability) as failed even if it previously run successfully
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ShortCircuitOperator does not properly retry when python_callable exits unexpectedly [airflow]
Posted by "vlieven (via GitHub)" <gi...@apache.org>.
vlieven commented on issue #36809:
URL: https://github.com/apache/airflow/issues/36809#issuecomment-1912157884
Hi @avkirilishin, thanks for your work on the PR, it looks like this will definitely improve the behavior!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ShortCircuitOperator does not properly retry when python_callable exits unexpectedly [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #36809: ShortCircuitOperator does not properly retry when python_callable exits unexpectedly
URL: https://github.com/apache/airflow/issues/36809
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ShortCircuitOperator does not properly retry when python_callable exits unexpectedly [airflow]
Posted by "avkirilishin (via GitHub)" <gi...@apache.org>.
avkirilishin commented on issue #36809:
URL: https://github.com/apache/airflow/issues/36809#issuecomment-1907030668
@vlieven, there is an issue with using exit() to simulate a crash. I hope it will be fixed in #36986, but I'm not certain if this fix will resolve your original problem. Could you please check it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org