You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "rlbabyuk (via GitHub)" <gi...@apache.org> on 2023/12/14 10:22:50 UTC
[I] Airflow stops tracking pod with xcom_sidecar container [airflow]
rlbabyuk opened a new issue, #36218:
URL: https://github.com/apache/airflow/issues/36218
### Apache Airflow version
Other Airflow 2 version (please specify below)
### If "Other Airflow 2 version" selected, which one?
2.5.3
### What happened?
While running dag with do_xcom_push=True parameter, after pod has been created, airflow task marked as failed, but the pod continues running, so the base container is completed and the process is stuck on the xcom_sidecar container.
I've checked and the sidecar airflow container has file /airflow/xcom/return.json with content {"exit_code": "0", "errors": "", "successful_target_ids": []}
No error in the logs
In some cases task stops pod tracking while pod get created, in other case on xcom_sidecar already running
<img width="903" alt="Screenshot 2023-12-14 at 11 09 40" src="https://github.com/apache/airflow/assets/26601785/adca5886-9398-46e5-9045-f477815172e1">
<img width="891" alt="Screenshot 2023-12-14 at 11 08 18" src="https://github.com/apache/airflow/assets/26601785/4bff546e-51cf-4264-84b4-3c792ef8c2dc">
Also in my case dag has container_resources=k8s.V1ResourceRequirements(limits={"memory": "3G"})
Tried to increase the limit but it didn't help
### What you think should happen instead?
Task should not fail and pod should be deleted
### How to reproduce
Create kubernetes_pod_operator dag with the following parameters:
do_xcom_push=True,
on_finish_action="delete_pod"
Start the dag simultaneously in 10 threads
Check the container statuses and airflow logs for failed runs
### Operating System
GKE Container-Optimized OS
### Versions of Apache Airflow Providers
"apache-airflow==2.5.3",
"apache-airflow-providers-common-sql==1.8.0",
"apache-airflow-providers-cncf-kubernetes==7.3.0",
"apache-airflow-providers-ftp==3.6.0",
"apache-airflow-providers-google==10.10.0",
"apache-airflow-providers-http==4.6.0",
"apache-airflow-providers-imap==3.4.0",
"apache-airflow-providers-sqlite==3.5.0",
### Deployment
Google Cloud Composer
### Deployment details
Used image -
composer-2.5.0-airflow - 2.5.3
### Anything else?
Reproduces randomly and often
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow stops tracking pod with xcom_sidecar container [airflow]
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on issue #36218:
URL: https://github.com/apache/airflow/issues/36218#issuecomment-1859207177
After 7.3.0, we released 11 releases and some of them contain bug fixes related to xcom_push, could you try to upgrade the version of `apache-airflow-providers-cncf-kubernetes` to latest version or 7.10.0 the last one compatible with Airflow 2.5.X?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow stops tracking pod with xcom_sidecar container [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #36218:
URL: https://github.com/apache/airflow/issues/36218#issuecomment-1855575772
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow stops tracking pod with xcom_sidecar container [airflow]
Posted by "rlbabyuk (via GitHub)" <gi...@apache.org>.
rlbabyuk closed issue #36218: Airflow stops tracking pod with xcom_sidecar container
URL: https://github.com/apache/airflow/issues/36218
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow stops tracking pod with xcom_sidecar container [airflow]
Posted by "rlbabyuk (via GitHub)" <gi...@apache.org>.
rlbabyuk commented on issue #36218:
URL: https://github.com/apache/airflow/issues/36218#issuecomment-1860133082
Closing this ticket as an issue was resolved due to a resource increase
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org