You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/02/08 10:04:57 UTC
[GitHub] [airflow] armandleopold opened a new issue #21420: KubernetesPodOperator Mark Task Failed after SUCCESS Randomly
armandleopold opened a new issue #21420:
URL: https://github.com/apache/airflow/issues/21420
### Apache Airflow version
2.2.3 (latest released)
### What happened
We are using KubernetesPodOperator to schedule basically all of our tasks.
Sometimes, randomly some TASK finish and are marked as SUCCESS and then immediately marked FAILED.
```
....
2022-02-08, 06:09:51 CET - Bootstrap - INFO - *********************************************************
2022-02-08, 06:09:51 CET - Bootstrap - INFO - *** FIN ***
2022-02-08, 06:09:51 CET - Bootstrap - INFO - *********************************************************
Event: consore-biologieclinique-interface-0.4.f3afe897abf646c6b22642cb594b8745 had an event of type Succeeded
Event with job id consore-biologieclinique-interface-0.4.f3afe897abf646c6b22642cb594b8745 Succeeded
Event: consore-biologieclinique-interface-0.4.f3afe897abf646c6b22642cb594b8745 had an event of type Succeeded
Event with job id consore-biologieclinique-interface-0.4.f3afe897abf646c6b22642cb594b8745 Succeeded
Marking task as SUCCESS. dag_id=consore-by-window, task_id=consore-biologieclinique-interface-0.4, execution_date=20220207T215900, start_date=20220208T040444, end_date=20220208T040953
Task exited with return code 0
Marking task as FAILED. dag_id=consore-by-window, task_id=consore-biologieclinique-interface-0.4, execution_date=20220207T215900, start_date=20220208T040444, end_date=20220208T040953
/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/context.py:152 AirflowContextDeprecationWarning: Accessing 'execution_date' from the template is deprecated and will be removed in a future version. Please use 'data_interval_start' or 'logical_date' instead.
/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/context.py:152 AirflowContextDeprecationWarning: Accessing 'next_ds' from the template is deprecated and will be removed in a future version. Please use '{{ data_interval_end | ds }}' instead.
/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/context.py:152 AirflowContextDeprecationWarning: Accessing 'next_ds_nodash' from the template is deprecated and will be removed in a future version. Please use '{{ data_interval_end | ds_nodash }}' instead.
/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/context.py:152 AirflowContextDeprecationWarning: Accessing 'next_execution_date' from the template is deprecated and will be removed in a future version. Please use 'data_interval_end' instead.
/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/context.py:152 AirflowContextDeprecationWarning: Accessing 'prev_ds' from the template is deprecated and will be removed in a future version.
/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/context.py:152 AirflowContextDeprecationWarning: Accessing 'prev_ds_nodash' from the template is deprecated and will be removed in a future version.
/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/context.py:152 AirflowContextDeprecationWarning: Accessing 'prev_execution_date' from the template is deprecated and will be removed in a future version.
/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/context.py:152 AirflowContextDeprecationWarning: Accessing 'prev_execution_date_success' from the template is deprecated and will be removed in a future version. Please use 'prev_data_interval_start_success' instead.
/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/context.py:152 AirflowContextDeprecationWarning: Accessing 'tomorrow_ds' from the template is deprecated and will be removed in a future version.
/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/context.py:152 AirflowContextDeprecationWarning: Accessing 'tomorrow_ds_nodash' from the template is deprecated and will be removed in a future version.
/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/context.py:152 AirflowContextDeprecationWarning: Accessing 'yesterday_ds' from the template is deprecated and will be removed in a future version.
/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/context.py:152 AirflowContextDeprecationWarning: Accessing 'yesterday_ds_nodash' from the template is deprecated and will be removed in a future version.
/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/email.py:102 PendingDeprecationWarning: Fetching SMTP credentials from configuration variables will be deprecated in a future release. Please set credentials using a connection instead.
section/key [smtp/smtp_user] not found in config
Email alerting: attempt 1
...
```
### What you expected to happen
What i expect is that the TASK is Marked as SUCCESS only .
```
...
2022-02-08, 06:09:51 CET - Bootstrap - INFO - *********************************************************
2022-02-08, 06:09:51 CET - Bootstrap - INFO - *** FIN ***
2022-02-08, 06:09:51 CET - Bootstrap - INFO - *********************************************************
Event: consore-biologieclinique-interface-0.4.f3afe897abf646c6b22642cb594b8745 had an event of type Succeeded
Event with job id consore-biologieclinique-interface-0.4.f3afe897abf646c6b22642cb594b8745 Succeeded
Event: consore-biologieclinique-interface-0.4.f3afe897abf646c6b22642cb594b8745 had an event of type Succeeded
Event with job id consore-biologieclinique-interface-0.4.f3afe897abf646c6b22642cb594b8745 Succeeded
Marking task as SUCCESS. dag_id=consore-by-window, task_id=consore-biologieclinique-interface-0.4, execution_date=20220207T215900, start_date=20220208T040444, end_date=20220208T040953
Task exited with return code 0
...
```
### How to reproduce
I don't know how to reproduce.
Ideas :
* Deploy airflow 2.2.3
* Create DAG with KubernetesPodOperator
* Schedule a task
### Operating System
debian-buster
### Versions of Apache Airflow Providers
can't have that
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
using the bitnami helm https://artifacthub.io/packages/helm/bitnami/airflow
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] armandleopold edited a comment on issue #21420: KubernetesPodOperator Mark Task Failed after SUCCESS Randomly
Posted by GitBox <gi...@apache.org>.
armandleopold edited a comment on issue #21420:
URL: https://github.com/apache/airflow/issues/21420#issuecomment-1038813178
This is strange :
The task is "Failed" but don't appears as failed in the front-end DAG view
Ho no ok, it's because the last DAG run finished Successfully.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] armandleopold commented on issue #21420: KubernetesPodOperator Mark Task Failed after SUCCESS Randomly
Posted by GitBox <gi...@apache.org>.
armandleopold commented on issue #21420:
URL: https://github.com/apache/airflow/issues/21420#issuecomment-1038811073
```
root@test-6494ccc5b8-7nlf6:/home# python -m pip list
Package Version
------------ -------
distlib 0.3.4
filelock 3.4.2
pip 21.3.1
platformdirs 2.4.1
setuptools 46.4.0
six 1.16.0
virtualenv 20.11.0
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] raphaelauv commented on issue #21420: KubernetesPodOperator Mark Task Failed after SUCCESS Randomly
Posted by GitBox <gi...@apache.org>.
raphaelauv commented on issue #21420:
URL: https://github.com/apache/airflow/issues/21420#issuecomment-1038800003
you can go inside the container and do `pip list`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] armandleopold edited a comment on issue #21420: KubernetesPodOperator Mark Task Failed after SUCCESS Randomly
Posted by GitBox <gi...@apache.org>.
armandleopold edited a comment on issue #21420:
URL: https://github.com/apache/airflow/issues/21420#issuecomment-1038813178
This is strange :
The task is "Failed" but don't appears as failed in the front-end DAG view
Ho no ok, it's because the last DAG run finished Successfully.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] raphaelauv commented on issue #21420: KubernetesPodOperator Mark Task Failed after SUCCESS Randomly
Posted by GitBox <gi...@apache.org>.
raphaelauv commented on issue #21420:
URL: https://github.com/apache/airflow/issues/21420#issuecomment-1032808667
what version of https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/#history ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] armandleopold commented on issue #21420: KubernetesPodOperator Mark Task Failed after SUCCESS Randomly
Posted by GitBox <gi...@apache.org>.
armandleopold commented on issue #21420:
URL: https://github.com/apache/airflow/issues/21420#issuecomment-1038798687
@raphaelauv I am not really familiar with how bitnami builds its images :
https://github.com/bitnami/bitnami-docker-airflow/blob/master/2/debian-10/prebuildfs/opt/bitnami/.bitnami_components.json#L7
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] armandleopold commented on issue #21420: KubernetesPodOperator Mark Task Failed after SUCCESS Randomly
Posted by GitBox <gi...@apache.org>.
armandleopold commented on issue #21420:
URL: https://github.com/apache/airflow/issues/21420#issuecomment-1038813178
This is strange :
The task is "Failed" but don't appears as failed in the front-end DAG view
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org