You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by "mhaure-touze (via GitHub)" <gi...@apache.org> on 2025/09/08 08:28:34 UTC
[I] EksPodOperator in deferrable mode pause, restart and then wait until completion [airflow]
mhaure-touze opened a new issue, #55368:
URL: https://github.com/apache/airflow/issues/55368
### Apache Airflow Provider(s)
cncf-kubernetes, amazon
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==9.12.0
apache-airflow-providers-cncf-kubernetes==10.7.0
### Apache Airflow version
2.10.3
### Operating System
amazon linux
### Deployment
Amazon (AWS) MWAA
### Deployment details
EksPodOperator which launch a pod on a EKS cluster v1.32
### What happened
1. the operator launch the pod
2. the triggerer pause the task as "DEFERRED"
3. the triggerer send a "running" event
4. the pod launch trigger_reentry method
5. some how the task wait for pod completion
6. task stay alive until a hearbeat timeout kill it
```
ip-172-29-129-41.eu-west-1.compute.internal
*** Reading remote log from Cloudwatch log_group: airflow-data-eng-mwaa-env-Task log_stream: dag_id=waititng/run_id=manual__2025-09-05T09_23_19.615897+00_00/task_id=waititng/attempt=10.log
2025-09-05T16:22:04.599378194Z
2025-09-05T16:22:04.653984267Z
2025-09-05T16:22:04.654174133Z
...
2025-09-05T23:50:51.065801031Z
2025-09-05T23:50:51.078800975Z
2025-09-05T23:50:51.125942120Z
[Invalid date] {local_task_job_runner.py:123} ▶ Pre task execution logs
[Invalid date] {base.py:84} INFO - Retrieving connection 'aws_eks_role'
[Invalid date] {baseoperator.py:416} WARNING - EksPodOperator.execute cannot be called outside TaskInstance!
[Invalid date] {pod.py:1280} INFO - Building pod waititng-9c5b6348-8893-405e-b769-5f0ffe3ee776-n48xxaw6 with labels: {'dag_id': 'waititng', 'task_id': 'waititng', 'run_id': 'manual__2025-09-05T092319.6158970000-3503ec696', 'kubernetes_pod_operator': 'True', 'try_number': '10'}
[Invalid date] {pod.py:572} INFO - Found matching pod waititng-9c5b6348-8893-405e-b769-5f0ffe3ee776-e0pnddo5 with labels {'airflow_kpo_in_cluster': 'False', 'airflow_version': '2.10.3', 'component': 'singleuser-server', 'dag_id': 'waititng', 'kubernetes_pod_operator': 'True', 'run_id': 'manual__2025-09-05T092319.6158970000-3503ec696', 'task_id': 'waititng', 'try_number': '7'}
[Invalid date] {pod.py:573} INFO - `try_number` of task_instance: 10
[Invalid date] {pod.py:574} INFO - `try_number` of pod: 7
[Invalid date] {pod.py:584} INFO - Reusing existing pod 'waititng-9c5b6348-8893-405e-b769-5f0ffe3ee776-e0pnddo5' (phase=Running, reason=) since it is not terminated or evicted.
[Invalid date] {taskinstance.py:288} INFO - Pausing task as DEFERRED. dag_id=waititng, task_id=waititng, run_id=manual__2025-09-05T09:23:19.615897+00:00, execution_date=20250905T092319, start_date=20250906T001827
[Invalid date] {taskinstance.py:340} ▶ Post task execution logs
[Invalid date] {pod.py:146} INFO - Checking pod 'waititng-9c5b6348-8893-405e-b769-5f0ffe3ee776-e0pnddo5' in namespace 'namespace'.
[Invalid date] {triggerer_job_runner.py:631} INFO - Trigger waititng/manual__2025-09-05T09:23:19.615897+00:00/waititng/-1/10 (ID 20) fired: TriggerEvent<{'status': 'running', 'last_log_time': None, 'namespace': 'namespace', 'name': 'waititng-9c5b6348-8893-405e-b769-5f0ffe3ee776-e0pnddo5', 'eks_cluster_name': 'cluster'}>
[Invalid date] {local_task_job_runner.py:123} ▶ Pre task execution logs
[Invalid date] {base.py:84} INFO - Retrieving connection 'aws_eks_role'
[Invalid date] {pod_manager.py:713} INFO - Pod waiting-9c5b6348-8893-405e-b769-5f0ffe3ee776-e0pnddo5 has phase Running
[Invalid date] {pod_manager.py:713} INFO - Pod waiting-9c5b6348-8893-405e-b769-5f0ffe3ee776-e0pnddo5 has phase Running
[Invalid date] {job.py:229} INFO - Heartbeat recovered after 71.80 seconds
[Invalid date] {local_task_job_runner.py:266} INFO - Task exited with return code -9. For more information, see https://airflow.apache.org/docs/apache-airflow/stable/troubleshooting.html#LocalTaskJob-killed
[Invalid date] {local_task_job_runner.py:245} ▲▲▲ Log group end
```
### What you think should happen instead
I am expecting the task to alternate between a running and deferred state until pod completion/failure
- Operator mode is deferrable=true
- logging_interval is set to 600 seconds
### How to reproduce
```
import datetime
from airflow.decorators import dag
from airflow.providers.amazon.aws.operators.eks import EksPodOperator
@dag(
dag_id="wait",
start_date=datetime.datetime(2025, 8, 4),
schedule=None,
catchup=False,
)
def wait() -> None:
EksPodOperator(
task_id="wait",
aws_conn_id="aws_eks_role",
cluster_name="cluster,
deferrable=True,
namespace="namespace",
region="eu-west-1",
pod_name=f"chromium-{pipeline_config.pipeline_id}",
cmds=["/bin/sh", "-c"],
arguments=["while true; do echo 'sleeping...'; sleep 2; done"],
image="alpine:3.22.1",
on_finish_action="delete_pod",
poll_interval=60,
logging_interval=600,
)
wait()
```
### Anything else
_No response_
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] EksPodOperator deferrable=true pause, restart and then wait until completion [airflow]
Posted by "VivinAT04 (via GitHub)" <gi...@apache.org>.
VivinAT04 commented on issue #55368:
URL: https://github.com/apache/airflow/issues/55368#issuecomment-4725343644
Hi, I'd like to investigate this issue as well.
I'll first try to reproduce it against the latest provider versions as suggested above and determine whether the problem still exists. If I can reproduce it and identify a fix, I'll open a PR with my findings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] EksPodOperator deferrable=true pause, restart and then wait until completion [airflow]
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas closed issue #55368: EksPodOperator deferrable=true pause, restart and then wait until completion
URL: https://github.com/apache/airflow/issues/55368
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] EksPodOperator deferrable=true pause, restart and then wait until completion [airflow]
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on issue #55368:
URL: https://github.com/apache/airflow/issues/55368#issuecomment-4567220194
This is quite an old bug report, sorry we missed it, @mhaure-touze can you try with the latest provider version? A lot has changed in both the EKS and K8s code since September 2025.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] EksPodOperator deferrable=true pause, restart and then wait until completion [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #55368:
URL: https://github.com/apache/airflow/issues/55368#issuecomment-4543550220
cc @o-nikolas if someone on the team has time to look into this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] EksPodOperator deferrable=true pause, restart and then wait until completion [airflow]
Posted by "darshil929 (via GitHub)" <gi...@apache.org>.
darshil929 commented on issue #55368:
URL: https://github.com/apache/airflow/issues/55368#issuecomment-4699097614
working on this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] EksPodOperator in deferrable mode pause, restart and then wait until completion [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #55368:
URL: https://github.com/apache/airflow/issues/55368#issuecomment-3265162895
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] EksPodOperator deferrable=true pause, restart and then wait until completion [airflow]
Posted by "darshil929 (via GitHub)" <gi...@apache.org>.
darshil929 commented on issue #55368:
URL: https://github.com/apache/airflow/issues/55368#issuecomment-4781386503
@o-nikolas yep, #64187 looks like the right fix for the credential/kube-config half, regenerating creds and rebuilding the kubeconfig in EksPodTrigger before it delegates to KubernetesPodTrigger will clear that Expected key current-context error directly.
So between that and #66705, both halves should be sorted from cncf-kubernetes 10.17.1 and amazon 9.25.0 onward which are both newer than the 10.7.0 / 9.12.0 in the report.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] EksPodOperator deferrable=true pause, restart and then wait until completion [airflow]
Posted by "darshil929 (via GitHub)" <gi...@apache.org>.
darshil929 commented on issue #55368:
URL: https://github.com/apache/airflow/issues/55368#issuecomment-4762323844
Hi @o-nikolas
Thanks for the pointer. I took your suggestion and dug into this against current main rather than the version it was originally reported on, and I think it's largely been resolved since.
The relevant change looks like #66705 ("Re-defer task when Kubernetes pod is not completed"). It reworks `KubernetesPodOperator.trigger_reentry` so that when the trigger returns an error event but the pod is still running / pending / waiting, the operator re-defers and keeps monitoring instead of falling through and waiting for completion. That matches the pause → restart → wait-until-completion behavior described here.
A few things that point to it being fixed:
- Grepping the operator on `10.7.0` (the version reported here) for the re-defer logic turns up nothing:
```
$ git show providers-cncf-kubernetes/10.7.0:providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/operators/pod.py | grep -n "_redefer_count\|MAX_REDEFER_ATTEMPTS"
```
No matches. On `main` the same grep shows it living in `trigger_reentry`:
```
276: MAX_REDEFER_ATTEMPTS = 3
1026: redefer_count = event.get("_redefer_count", 0)
1028: if event["status"] == "error" and pod_is_not_done and redefer_count < self.MAX_REDEFER_ATTEMPTS:
1039: self.trigger_kwargs["_redefer_count"] = redefer_count + 1
```
- It first shipped in `apache-airflow-providers-cncf-kubernetes==10.17.1`, so any release from there onward should include it.
- The regression tests added alongside #66705 pass on current `main`:
<img width="1320" height="379" alt="Image" src="https://github.com/user-attachments/assets/2ec604fc-ed80-404a-96ef-7fdafc5be3b9" />
One thing worth flagging: in the original logs the re-entry is actually kicked off by a `ConfigException: Invalid kube-config file. Expected key current-context in kube-config`. #66705 stops that from immediately failing the task, but I don't think it addresses why the kube-config ends up invalid on the triggerer, that part may be closer to #61736.
@mhaure-touze would you be able to retest on `cncf-kubernetes>=10.17.1` and check whether the deferrable behavior looks correct now? If the kube-config error is gone as well, this can probably be closed. if it's still there, it might be worth tracking separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] EksPodOperator deferrable=true pause, restart and then wait until completion [airflow]
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on issue #55368:
URL: https://github.com/apache/airflow/issues/55368#issuecomment-4774392618
Thanks for the deep dive @darshil929
As for the credentials expiring or being non-existent. I think this PR I put out a little while ago should fix that: https://github.com/apache/airflow/pull/64187
What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org