You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "shahar1 (via GitHub)" <gi...@apache.org> on 2024/11/21 12:11:20 UTC
[I] Stats of a dynamically expended task disappear after an automatic retry of a failed task [airflow]
shahar1 opened a new issue, #44245:
URL: https://github.com/apache/airflow/issues/44245
### Apache Airflow version
2.10.3
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
At least in the classic UI, when a task gets expanded [dynamically](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/dynamic-task-mapping.html), any retried failed task makes its other stats disappear.
### What you think should happen instead?
_No response_
### How to reproduce
```python
import random
from datetime import datetime
from airflow import DAG
from airflow.operators.python import PythonOperator
def random_fail_task(task_id):
if random.random() < 0.1: # 10% chance of failure
raise Exception(f"Task {task_id} failed")
print(f"Task {task_id} succeeded")
def all_success(task_id):
print(f"Task {task_id} succeeded")
with DAG(
dag_id='dynamic_task_expansion',
start_date=datetime(2023, 1, 1),
schedule_interval=None,
catchup=False,
) as dag:
tasks = PythonOperator.partial(
task_id='task',
## Uncoment below for the second coherence check
# retries=0,
python_callable=random_fail_task, # <- Change to all_success for first coherence checks
).expand(op_args=[[i] for i in range(100)])
```
**Coherence checks**
1. When replacing `random_fail_task` with `random_all_success_task `, we get the exact number of tasks, all succesful (100):
2. When we apply `retries=0` with `random_fail_task` the total number of tasks (succesful+failed) is once again 100:
** Reproduction**
1. When running the task - about 10% of the mapped tasks fail and are up for retry:
2. When the failed tasks are triggered for re-run, the other tasks previously marked as successful/failed disappear:
3. And finally we're left with only 1 task:
---
### Operating System
Linux
### Versions of Apache Airflow Providers
_No response_
### Deployment
Google Cloud Composer
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Stats of a dynamically expended task disappear after an automatic retry of a failed task [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on issue #44245:
URL: https://github.com/apache/airflow/issues/44245#issuecomment-2494458751
> First idea: look at the network inspector and see if the issue is in the API response or just in the presentation of the data
Thanks for the tip! It seems like the issue is in the API response.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Stats of a dynamic mapped tasks disappear after an automatic retry of a failed task [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #44245: Stats of a dynamic mapped tasks disappear after an automatic retry of a failed task
URL: https://github.com/apache/airflow/issues/44245
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Stats of a dynamically expended task disappear after an automatic retry of a failed task [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on issue #44245:
URL: https://github.com/apache/airflow/issues/44245#issuecomment-2491575328
First idea: look at the network inspector and see if the issue is in the API response or just in the presentation of the data
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org