You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "pavelpi (via GitHub)" <gi...@apache.org> on 2025/01/22 10:27:15 UTC
[I] Tasks finish despite marked as failed [airflow]
pavelpi opened a new issue, #45925:
URL: https://github.com/apache/airflow/issues/45925
### Apache Airflow version
2.10.4
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
Having following DAG:
```python
from datetime import datetime
from airflow.operators.python import PythonOperator
from airflow import DAG
import time
def wait_callable():
for i in range(1, 30):
print(f'sleep #{0}', i)
time.sleep(1)
default_args = {
'owner': 'airflow',
'start_date': datetime(2020, 3, 4)
}
dag = DAG("wait_dag_python",
default_args=default_args,
schedule=None)
wait_task = PythonOperator(
task_id='wait_task',
python_callable=wait_callable,
dag=dag
)
```
This task is made to finish after 30s and it prints output every 1s.
On Airflow UI when I mark the task as failed, it is marked as red and the DAG finishes in failed state. However when I refresh the page later (after 30s), the DAG is green and the logfile shows that the task finished successfully with all logs (one per second)
scheduler log:
[scheduler_log.txt](https://github.com/user-attachments/files/18503709/scheduler_log.txt)
same happens in 2.7.3 and 2.10.4
### What you think should happen instead?
UI should be more consistent if this is as designed. Now it is confusing.
The user should have an option to terminate the task if it got stuck. This is especially needed for KubernetesPodOperator tasks where this could happen easily.
### How to reproduce
Run the DAG above. Mark the task as failed while it is running. Refresh the page in 30s
### Operating System
Ubuntu
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==9.0.0
apache-airflow-providers-celery==3.8.3
apache-airflow-providers-cncf-kubernetes==9.0.1
apache-airflow-providers-common-compat==1.2.1
apache-airflow-providers-common-io==1.4.2
apache-airflow-providers-common-sql==1.19.0
apache-airflow-providers-elasticsearch==5.5.2
apache-airflow-providers-fab==1.5.0
apache-airflow-providers-ftp==3.11.1
apache-airflow-providers-google==10.25.0
apache-airflow-providers-grpc==3.6.0
apache-airflow-providers-hashicorp==3.8.0
apache-airflow-providers-http==4.13.2
apache-airflow-providers-imap==3.7.0
apache-airflow-providers-microsoft-azure==11.0.0
apache-airflow-providers-microsoft-mssql==3.9.1
apache-airflow-providers-mysql==5.7.3
apache-airflow-providers-odbc==4.8.0
apache-airflow-providers-opsgenie==5.7.0
apache-airflow-providers-oracle==3.12.0
apache-airflow-providers-postgres==5.13.1
apache-airflow-providers-redis==3.8.0
apache-airflow-providers-sftp==4.11.1
apache-airflow-providers-smtp==1.8.0
apache-airflow-providers-sqlite==3.9.0
apache-airflow-providers-ssh==3.14.0
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
default executor is CeleryKubernetesExecutor
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Tasks finish despite marked as failed [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb closed issue #45925: Tasks finish despite marked as failed
URL: https://github.com/apache/airflow/issues/45925
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Tasks finish despite marked as failed [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on issue #45925:
URL: https://github.com/apache/airflow/issues/45925#issuecomment-2796699152
Fixed in Airflow 3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Tasks finish despite marked as failed [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on issue #45925:
URL: https://github.com/apache/airflow/issues/45925#issuecomment-2796564337
@Rahulk14 Is the current behaviour in Airflow 3.0.0rc2 what you hoped for. I think this is now handled the time the task next heartbeats and It will be killed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Tasks finish despite marked as failed [airflow]
Posted by "Rahulk14 (via GitHub)" <gi...@apache.org>.
Rahulk14 commented on issue #45925:
URL: https://github.com/apache/airflow/issues/45925#issuecomment-2796549303
**When we mark a task as "Failed", Airflow immediately stops the running task. Here's why:**
- We're using Airflow version 3.
- In newer versions of Airflow, there's a graceful shutdown mechanism for running tasks when you mark them as failed.
**How it works in newer versions:**
- Mark as Failed → Send SIGTERM to the task → Wait → If not stopped, send SIGKILL
- This ensures the task is properly terminated.
**Comparison:**
- New Airflow: Mark as Failed → Task gets terminated properly
- Old Airflow: Mark as Failed → Task might keep running → Could result in a false success overwrite
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Tasks finish despite marked as failed [airflow]
Posted by "Rahulk14 (via GitHub)" <gi...@apache.org>.
Rahulk14 commented on issue #45925:
URL: https://github.com/apache/airflow/issues/45925#issuecomment-2796578087
@ashb Yes, this matches our observation. After marking the task as Failed, it stops shortly after — likely right after the next heartbeat.
This behavior looks good to us and resolves the original concern.
Thanks for clarifying!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Tasks finish despite marked as failed [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #45925:
URL: https://github.com/apache/airflow/issues/45925#issuecomment-2606871051
I am not sure @kaxil @ashb, @amoghrajesh but I believe that is the current behaviour in Airflow 2 (though I would expect that marking task as failed shoudl also make the supervisor to kill it **nicely** - but since you are close to that part now with TaskSDK work - is that the current behaviour and maybe we can change it in Airflow 3/fix in Airlfow 2 if so ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org