You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/03/23 10:39:26 UTC
[GitHub] [airflow] sushi30 opened a new issue #22485: KubernetesPodOperator matches the task pod and foes into deadlock
sushi30 opened a new issue #22485:
URL: https://github.com/apache/airflow/issues/22485
### Apache Airflow Provider(s)
cncf-kubernetes
### Versions of Apache Airflow Providers
apache-airflow-providers-cncf-kubernetes==3.1.1
### Apache Airflow version
2.2.2
### Operating System
Centos 7
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
Airflow deployed on kubernetes
### What happened
These tasks have been running for several months. When trying to upgrade to the latest Kubernetes provider, the KubernetesPodOperator the worker pod so it does not start a new pod. This just hangs until the task times out.
```python
[2022-03-23 09:49:49,950] {kubernetes_pod.py:525} INFO - Creating pod engines-distant-sharer.5bfdc82ad54d4dc1b58367f3d6d6a94f with labels: {'dag_id': 'tag_engine_user', 'task_id': 'engines_distant_sharer', 'run_id': 'manual__2022-03-23T094931.4187280000-fe99c2456', 'try_number': '1'}
[2022-03-23 09:49:49,972] {kubernetes_pod.py:336} INFO - Found matching pod tagengineuserenginesdistantsharer.1e4548b0558448f9a7aaa69da5f1e69d with labels {'airflow-worker': '2258913', 'airflow_version': '2.2.2', 'component': 'worker', 'dag_id': 'tag_engine_user', 'kubernetes_executor': 'True', 'release': 'airflow', 'run_id': 'manual__2022-03-23T094931.4187280000-fe99c2456', 'task_id': 'engines_distant_sharer', 'tier': 'airflow', 'try_number': '1'}
```
### What you think should happen instead
The task should run succesfully
### How to reproduce
Run a kubernetes operator with the pod namespace the same as the airfloe deployment namespace.
### Anything else
I see that the label "execution_date" has been changed to "run_id". This is most likely the cause.
```python
{'dag_id': 'tag_engine', 'task_id': 'engines_stationary_sharer', 'execution_date': '2022-03-23T0730000000-b016b00b9', 'try_number': '1'}
```
```python
{'dag_id': 'tag_engine', 'task_id': 'engines_stationary_sharer', 'run_id': 'scheduled__2022-03-23T0930000000-fdfd231b4', 'try_number': '1'}
```
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] raphaelauv commented on issue #22485: KubernetesPodOperator matches the task pod and foes into deadlock
Posted by GitBox <gi...@apache.org>.
raphaelauv commented on issue #22485:
URL: https://github.com/apache/airflow/issues/22485#issuecomment-1076517146
you had this problem just after updating `apache-airflow-providers-cncf-kubernetes` ?
scenario :
1) KPO start a pod with apache-airflow-providers-cncf-kubernetes=3.0.2
2) restart airlfow with apache-airflow-providers-cncf-kubernetes=3.1.1
3) KPO do not re-attach correctly the pod and timeout
?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] raphaelauv commented on issue #22485: KubernetesPodOperator matches the task pod and foes into deadlock
Posted by GitBox <gi...@apache.org>.
raphaelauv commented on issue #22485:
URL: https://github.com/apache/airflow/issues/22485#issuecomment-1076565643
so your issue is that
`3.1.1` is not backward compatible with `3.0.2`
so the provider should had bump to v4 because it's a breaking change ? (@potiuk )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sushi30 commented on issue #22485: KubernetesPodOperator matches the task pod and goes into deadlock
Posted by GitBox <gi...@apache.org>.
sushi30 commented on issue #22485:
URL: https://github.com/apache/airflow/issues/22485#issuecomment-1077514558
I am able to reproduce this with a minimal configuration helm chart on minikube. The latest apache/airflow image is using version 3.0.2 so you need to create a custom image with the new provider version. This DAG works fine with 3.0.2 and hangs in 3.1.1:
```python
from datetime import datetime
from airflow import DAG
from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import (
KubernetesPodOperator,
)
with DAG(
dag_id="example", start_date=datetime(2022, 1, 1), schedule_interval="@once"
) as dag:
k = KubernetesPodOperator(
namespace="airflow",
name="hello",
image="debian",
cmds=["bash", "-cx"],
arguments=["echo", "10"],
labels={"foo": "bar"},
task_id="dry_run_demo",
)
```
```docker
# Dockerfile
FROM apache/airflow
RUN pip install apache-airflow-providers-cncf-kubernetes==3.1.1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sushi30 commented on issue #22485: KubernetesPodOperator matches the task pod and foes into deadlock
Posted by GitBox <gi...@apache.org>.
sushi30 commented on issue #22485:
URL: https://github.com/apache/airflow/issues/22485#issuecomment-1076347640
I do have a one-hour timeout for the *task*. This task takes 30s~1:30s when I downgrade to `apache-airflow-providers-cncf-kubernetes=3.0.2`. When I upgraded, it timed out.
It is supposed to create a *new pod* called `engines-distant-sharer-{randomString}` but instead it *matches* on the pod `tagengineuserenginesdistantsharer` which is the *airflow worker* that is running the task.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] raphaelauv edited a comment on issue #22485: KubernetesPodOperator matches the task pod and foes into deadlock
Posted by GitBox <gi...@apache.org>.
raphaelauv edited a comment on issue #22485:
URL: https://github.com/apache/airflow/issues/22485#issuecomment-1076257110
I can't reproduce with
airflow `2.2.2` - `apache-airflow-providers-cncf-kubernetes==3.1.1,`
or
`2.2.4` - `apache-airflow-providers-cncf-kubernetes==3.1.1,`
the operator can re-attach the running pod if I restart the sceduler
```log
[2022-03-23, 11:22:27 UTC] {kubernetes_pod.py:525} INFO - Creating pod airflow-test-pod.9a94208666254a0099f44c2ce811c62e with labels: {'dag_id': 'kubernetes_dag', 'task_id': 'task-one', 'run_id': 'scheduled__2022-03-13T0000000000-eef165281', 'try_number': '4'}
[2022-03-23, 11:22:27 UTC] {kubernetes_pod.py:336} INFO - Found matching pod airflow-test-pod.485275d944964e8188c0db5f10776956 with labels {'airflow_version': '2.2.2', 'dag_id': 'kubernetes_dag', 'kubernetes_pod_operator': 'True', 'run_id': 'scheduled__2022-03-13T0000000000-eef165281', 'task_id': 'task-one', 'try_number': '3'}
[2022-03-23, 11:22:27 UTC] {kubernetes_pod.py:337} INFO - `try_number` of task_instance: 4
[2022-03-23, 11:22:27 UTC] {kubernetes_pod.py:338} INFO - `try_number` of pod: 3
```
could you give us more context on the KPO ( what kind of K8S it use, in_cluster ? ... ) and the FULL airflow logs of the error
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sushi30 commented on issue #22485: KubernetesPodOperator matches the task pod and goes into deadlock
Posted by GitBox <gi...@apache.org>.
sushi30 commented on issue #22485:
URL: https://github.com/apache/airflow/issues/22485#issuecomment-1077516866
> If `3.1.1` is not managing correctly the POD's started in `3.0.2` then can you delete the pod and clear the airflow TASK and the XCOM so the task ( with `3.1.1`) can start a real new POD and not try to re attach a running POD
This has nothing to do with pods started in `3.0.2`. The KPO is not able to start new pods in `3.1.1`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sushi30 commented on issue #22485: KubernetesPodOperator matches the task pod and goes into deadlock
Posted by GitBox <gi...@apache.org>.
sushi30 commented on issue #22485:
URL: https://github.com/apache/airflow/issues/22485#issuecomment-1077474279
I cannot identify anything unique in my setup. The tasks in this setup have been working without fault for the past year or so. This broke with the change to 3.1.1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #22485: KubernetesPodOperator matches the task pod and goes into deadlock
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #22485:
URL: https://github.com/apache/airflow/issues/22485#issuecomment-1076756469
Hmm. My question Is it always the case with 3.1.1 or is it something specific for @sushi30 setup?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] raphaelauv edited a comment on issue #22485: KubernetesPodOperator matches the task pod and foes into deadlock
Posted by GitBox <gi...@apache.org>.
raphaelauv edited a comment on issue #22485:
URL: https://github.com/apache/airflow/issues/22485#issuecomment-1076517146
you had this problem just after updating `apache-airflow-providers-cncf-kubernetes` ?
scenario :
1) KPO start a pod with apache-airflow-providers-cncf-kubernetes=3.0.2
2) restart airlfow with apache-airflow-providers-cncf-kubernetes=3.1.1
3) KPO do not re-attach correctly the pod and timeout
?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] raphaelauv commented on issue #22485: KubernetesPodOperator matches the task pod and foes into deadlock
Posted by GitBox <gi...@apache.org>.
raphaelauv commented on issue #22485:
URL: https://github.com/apache/airflow/issues/22485#issuecomment-1076257110
I can't reproduce with
airflow `2.2.2` - `apache-airflow-providers-cncf-kubernetes==3.1.1,`
and
`2.2.4` - `apache-airflow-providers-cncf-kubernetes==3.1.1,`
the operator can re-attach the running pod if I restart the sceduler
```log
[2022-03-23, 11:22:27 UTC] {kubernetes_pod.py:525} INFO - Creating pod airflow-test-pod.9a94208666254a0099f44c2ce811c62e with labels: {'dag_id': 'kubernetes_dag', 'task_id': 'task-one', 'run_id': 'scheduled__2022-03-13T0000000000-eef165281', 'try_number': '4'}
[2022-03-23, 11:22:27 UTC] {kubernetes_pod.py:336} INFO - Found matching pod airflow-test-pod.485275d944964e8188c0db5f10776956 with labels {'airflow_version': '2.2.2', 'dag_id': 'kubernetes_dag', 'kubernetes_pod_operator': 'True', 'run_id': 'scheduled__2022-03-13T0000000000-eef165281', 'task_id': 'task-one', 'try_number': '3'}
[2022-03-23, 11:22:27 UTC] {kubernetes_pod.py:337} INFO - `try_number` of task_instance: 4
[2022-03-23, 11:22:27 UTC] {kubernetes_pod.py:338} INFO - `try_number` of pod: 3
```
could you give us more context on the KPO ( what kind of K8S it use, in_cluster ? ... ) and the FULL airflow logs of the error
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sushi30 edited a comment on issue #22485: KubernetesPodOperator matches the task pod and goes into deadlock
Posted by GitBox <gi...@apache.org>.
sushi30 edited a comment on issue #22485:
URL: https://github.com/apache/airflow/issues/22485#issuecomment-1077514558
I am able to reproduce this with a minimal configuration helm chart on minikube. The latest apache/airflow image is using version 3.0.2 so you need to create a custom image with the new provider version. This DAG works fine with 3.0.2 and hangs in 3.1.1:
```python
from datetime import datetime
from airflow import DAG
from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import (
KubernetesPodOperator,
)
with DAG(
dag_id="example", start_date=datetime(2022, 1, 1), schedule_interval="@once"
) as dag:
k = KubernetesPodOperator(
namespace="airflow",
name="hello",
image="debian",
cmds=["bash", "-cx"],
arguments=["echo", "10"],
labels={"foo": "bar"},
task_id="dry_run_demo",
)
```
```dockerfile
# Dockerfile
FROM apache/airflow
RUN pip install apache-airflow-providers-cncf-kubernetes==3.1.1
```
### Screenshots
Task ran on 3.0.2
Then hangs on 3.1.1:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] raphaelauv edited a comment on issue #22485: KubernetesPodOperator matches the task pod and foes into deadlock
Posted by GitBox <gi...@apache.org>.
raphaelauv edited a comment on issue #22485:
URL: https://github.com/apache/airflow/issues/22485#issuecomment-1076257110
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sushi30 commented on issue #22485: KubernetesPodOperator matches the task pod and foes into deadlock
Posted by GitBox <gi...@apache.org>.
sushi30 commented on issue #22485:
URL: https://github.com/apache/airflow/issues/22485#issuecomment-1076548189
only clarification is that regarding (3), the scheduler is able to start a
worker pod which fails to start the task pod.
On Wed, 23 Mar 2022 at 16:06, raphaelauv ***@***.***> wrote:
> you had this problem just after updating
> apache-airflow-providers-cncf-kubernetes ?
>
> scenario :
>
> 1.
>
> start a pod with apache-airflow-providers-cncf-kubernetes=3.0.2
> 2.
>
> restart airlfow with apache-airflow-providers-cncf-kubernetes=3.1.1
> 3.
>
> scheduler do not re-attach correctly the pod and timeout
>
> ?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/airflow/issues/22485#issuecomment-1076517146>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AGJAQRITIN5M74U6CMDH3LDVBM6SBANCNFSM5RNS6EWQ>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sushi30 commented on issue #22485: KubernetesPodOperator matches the task pod and foes into deadlock
Posted by GitBox <gi...@apache.org>.
sushi30 commented on issue #22485:
URL: https://github.com/apache/airflow/issues/22485#issuecomment-1076567793
I dont think is related to backward compatibility. Maybe the correction in (3) is that the KPO is failing to **start** the pod. Not reattach,.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] raphaelauv commented on issue #22485: KubernetesPodOperator matches the task pod and foes into deadlock
Posted by GitBox <gi...@apache.org>.
raphaelauv commented on issue #22485:
URL: https://github.com/apache/airflow/issues/22485#issuecomment-1076555743
`the scheduler is able to start a worker pod` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] raphaelauv commented on issue #22485: KubernetesPodOperator matches the task pod and foes into deadlock
Posted by GitBox <gi...@apache.org>.
raphaelauv commented on issue #22485:
URL: https://github.com/apache/airflow/issues/22485#issuecomment-1076328154
```log
[2022-03-23 11:18:42,437] {kubernetes_pod.py:338} INFO - `try_number` of pod: 2
[2022-03-23 11:18:42,457] {pod_manager.py:203} INFO -
[2022-03-23 11:18:42,457] {pod_manager.py:203} INFO - [2022-03-23 11:18:41,495] {dagbag.py:500} INFO - Filling up the DagBag from /opt/airflow/dags/product_dags/tag_engine.py
[2022-03-23 12:18:42,414] {timeout.py:36} ERROR - Process timed out, PID: 15
```
it look like your have a 1 hour timeout, can you check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sushi30 edited a comment on issue #22485: KubernetesPodOperator matches the task pod and goes into deadlock
Posted by GitBox <gi...@apache.org>.
sushi30 edited a comment on issue #22485:
URL: https://github.com/apache/airflow/issues/22485#issuecomment-1077514558
I am able to reproduce this with a minimal configuration helm chart on minikube. The latest apache/airflow image is using version 3.0.2 so you need to create a custom image with the new provider version. This DAG works fine with 3.0.2 and hangs in 3.1.1:
```python
from datetime import datetime
from airflow import DAG
from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import (
KubernetesPodOperator,
)
with DAG(
dag_id="example", start_date=datetime(2022, 1, 1), schedule_interval="@once"
) as dag:
k = KubernetesPodOperator(
namespace="airflow",
name="hello",
image="debian",
cmds=["bash", "-cx"],
arguments=["echo", "10"],
labels={"foo": "bar"},
task_id="dry_run_demo",
)
```
```dockerfile
# Dockerfile
FROM apache/airflow
RUN pip install apache-airflow-providers-cncf-kubernetes==3.1.1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #22485: KubernetesPodOperator matches the task pod and goes into deadlock
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #22485:
URL: https://github.com/apache/airflow/issues/22485#issuecomment-1079963090
Converting it into a discussion until we have more information.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sushi30 commented on issue #22485: KubernetesPodOperator matches the task pod and foes into deadlock
Posted by GitBox <gi...@apache.org>.
sushi30 commented on issue #22485:
URL: https://github.com/apache/airflow/issues/22485#issuecomment-1076560436
didnt see the edit. the scenario you laid out is correct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] raphaelauv commented on issue #22485: KubernetesPodOperator matches the task pod and goes into deadlock
Posted by GitBox <gi...@apache.org>.
raphaelauv commented on issue #22485:
URL: https://github.com/apache/airflow/issues/22485#issuecomment-1077515574
If 3.1.1 is not managing correctly the POD's started in 3.0.2 then can you delete the pod and clear the airflow TASK and the XCOM
so the task ( with 3.1.1) can start a real new POD and not try to re attach a running POD
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] raphaelauv edited a comment on issue #22485: KubernetesPodOperator matches the task pod and goes into deadlock
Posted by GitBox <gi...@apache.org>.
raphaelauv edited a comment on issue #22485:
URL: https://github.com/apache/airflow/issues/22485#issuecomment-1077515574
If `3.1.1` is not managing correctly the POD's started in `3.0.2` then can you delete the pod and clear the airflow TASK and the XCOM
so the task ( with `3.1.1`) can start a real new POD and not try to re attach a running POD
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #22485: KubernetesPodOperator matches the task pod and goes into deadlock
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #22485:
URL: https://github.com/apache/airflow/issues/22485#issuecomment-1079963039
> This has nothing to do with pods started in `3.0.2`. The KPO is not able to start new pods in `3.1.1`.
@sushi30 - it is very likely something environmental for you. People often argue that "it worked before so it must be backwards compatibiliity problem" where in fact there might be other - environmental factors where misconfiguration or wrong deployment caused things to "work" (or rather mask problem) before, only to be revealed when for example new library provides more thoropugh check. Or maybe a library change causes some more resource usage and you simply need to increase resources (memory/disk/the like). There can be many things that could go wrong. I would not jump into conclusion this is backwards compatibility. It might be, but does not have to - and it is not at all obvious.
That would be rather surprising to have some general problem - we do not see other people reporting problems like that one.
Do you have any logs telling more what's happening? Maybe you can take a look at the logs of K8S creating PODs and maybe they will tell you what's wrong. The informaton that PODs are "hanging" makes it impossible to diagnose what's wrong - without more details we have even less information than you have.
And looking at logs of what's happening when it fails is something that only you can do, I am afraid. Also it would be great to get some more information - which K8S version you have for example.
What I Can you also try the 2.2.5rc1 release of Airflow (We just put it up for voting). The images we have in dockerhub contain both - latest Airflow and latest cncf.kubernetes so if you could try it and see if the problem persists there would be helpfi.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sushi30 commented on issue #22485: KubernetesPodOperator matches the task pod and foes into deadlock
Posted by GitBox <gi...@apache.org>.
sushi30 commented on issue #22485:
URL: https://github.com/apache/airflow/issues/22485#issuecomment-1076322931
- It is on EKS
- `in_cluster=True`
- Using the `airflow` namespace. Same as the Airflow deployment.
No special annotations. Using the default template.
[kpo_error.log](https://github.com/apache/airflow/files/8332986/kpo_error.log)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] raphaelauv edited a comment on issue #22485: KubernetesPodOperator matches the task pod and foes into deadlock
Posted by GitBox <gi...@apache.org>.
raphaelauv edited a comment on issue #22485:
URL: https://github.com/apache/airflow/issues/22485#issuecomment-1076565643
so your issue is that
`3.1.1` is not backward compatible with `3.0.2`
so the provider should had bump to v4 because it's a breaking change ? (@potiuk )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] raphaelauv edited a comment on issue #22485: KubernetesPodOperator matches the task pod and goes into deadlock
Posted by GitBox <gi...@apache.org>.
raphaelauv edited a comment on issue #22485:
URL: https://github.com/apache/airflow/issues/22485#issuecomment-1076257110
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org