You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/08/25 09:23:10 UTC
[GitHub] [airflow] stijndehaes opened a new issue #10544: Kubernetes pod operator: More than one pod running with labels
stijndehaes opened a new issue #10544:
URL: https://github.com/apache/airflow/issues/10544
<!--
Welcome to Apache Airflow! For a smooth issue process, try to answer the following questions.
Don't worry if they're not all applicable; just try to include what you can :-)
If you need to include code snippets or logs, please put them in fenced code
blocks. If they're super-long, please use the details tag like
<details><summary>super-long log</summary> lots of stuff </details>
Please delete these comment blocks before submitting the issue.
-->
<!--
IMPORTANT!!!
PLEASE CHECK "SIMILAR TO X EXISTING ISSUES" OPTION IF VISIBLE
NEXT TO "SUBMIT NEW ISSUE" BUTTON!!!
PLEASE CHECK IF THIS ISSUE HAS BEEN REPORTED PREVIOUSLY USING SEARCH!!!
Please complete the next sections or the issue will be closed.
This questions are the first thing we need to know to understand the context.
-->
**Apache Airflow version**: 1.10.11
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`):
```
Server Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.9-eks-4c6976", GitCommit:"4c6976793196d70bc5cd29d56ce5440c9473648e", GitTreeState:"clean", BuildDate:"2020-07-17T18:46:04Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
```
**Environment**:
- **Cloud provider or hardware configuration**: AWS
- **OS** (e.g. from /etc/os-release): not relevant
- **Kernel** (e.g. `uname -a`): not relevant
- **Install tools**: not relevant
- **Others**: not relevant
**What happened**:
<!-- (please include exact error messages if you can) -->
When launching a failing kubernetesPodOperator job with 4 retries you get the following message on the second retry. This is because the failed pods still exist on kubernetes and thus the list call in the pod operator:
```python
pod_list = client.list_namespaced_pod(self.namespace, label_selector=label_selector)
```
Also returns the failed objects. A fix would be to filter on only pods that are not completed or failed.
```
[2020-08-25 08:51:17,856] {taskinstance.py:1150} ERROR - Pod Launching failed: More than one pod running with labels: dag_id=sample-python-failing,execution_date=2020-08-24T0300000000-71ba3e273,task_id=ingest-weather
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py", line 276, in execute
'{label_selector}'.format(label_selector=label_selector))
airflow.exceptions.AirflowException: More than one pod running with labels: dag_id=sample-python-failing,execution_date=2020-08-24T0300000000-71ba3e273,task_id=ingest-weather
```
**What you expected to happen**:
To start my container anew.
<!-- What do you think went wrong? -->
**How to reproduce it**:
To reproduce start a dag with a kubernetespodoperator with retries that fails. Set the retries to more then 2 as it starts happening on the thrid try.
<!---
As minimally and precisely as possible. Keep in mind we do not have access to your cluster or dags.
If you are using kubernetes, please attempt to recreate the issue using minikube or kind.
## Install minikube/kind
- Minikube https://minikube.sigs.k8s.io/docs/start/
- Kind https://kind.sigs.k8s.io/docs/user/quick-start/
If this is a UI bug, please provide a screenshot of the bug or a link to a youtube video of the bug in action
You can include images using the .md style of
To record a screencast, mac users can use QuickTime and then create an unlisted youtube video with the resulting .mov file.
--->
**Anything else we need to know**:
<!--
How often does this problem occur? Once? Every time etc?
Any relevant logs to include? Put them here in side a detail tag:
<details><summary>x.log</summary> lots of stuff </details>
-->
I will make a PR myself to fix this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] stijndehaes commented on issue #10544: Kubernetes pod operator: More than one pod running with labels
Posted by GitBox <gi...@apache.org>.
stijndehaes commented on issue #10544:
URL: https://github.com/apache/airflow/issues/10544#issuecomment-679922113
I just noticed that what I thought to do filtering on pending and running pods break the flow when restarting the scheduler. Since it will not find your completed pod. I am thinking the flow should change into:
- search for pod with the same labels (including the try_id) if it is found -> Attach to the new pod
- If it is not found we should probably always start a new one. I don't think it makes sense to attach to the pod of another try?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] matan129 commented on issue #10544: Kubernetes pod operator: More than one pod running with labels
Posted by GitBox <gi...@apache.org>.
matan129 commented on issue #10544:
URL: https://github.com/apache/airflow/issues/10544#issuecomment-822442207
Happens on `2.01`. Any updates?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] stijndehaes commented on issue #10544: Kubernetes pod operator: More than one pod running with labels
Posted by GitBox <gi...@apache.org>.
stijndehaes commented on issue #10544:
URL: https://github.com/apache/airflow/issues/10544#issuecomment-679919233
Looking at the code I notice a discrepancy between master and release 1.10.11.
In master I find this:
```python
if len(pod_list.items) == 1 and \
self._try_numbers_do_not_match(context, pod_list.items[0]) and \
self.reattach_on_restart:
self.log.info("found a running pod with labels %s but a different try_number"
"Will attach to this pod and monitor instead of starting new one", labels)
final_state, result = self.monitor_launched_pod(launcher, pod_list.items[0])
elif len(pod_list.items) == 1:
self.log.info("found a running pod with labels %s."
"Will monitor this pod instead of starting new one", labels)
final_state, result = self.monitor_launched_pod(launcher, pod_list.items[0])
else:
self.log.info("creating pod with labels %s and launcher %s", labels, launcher)
final_state, _, result = self.create_new_pod_for_operator(labels, launcher)
```
on 1.10.11:
```python
if len(pod_list.items) == 1 and \
self._try_numbers_do_not_match(context, pod_list.items[0]) and \
self.reattach_on_restart:
self.log.info("found a running pod with labels %s but a different try_number"
"Will attach to this pod and monitor instead of starting new one", labels)
final_state, _, result = self.create_new_pod_for_operator(labels, launcher)
elif len(pod_list.items) == 1:
self.log.info("found a running pod with labels %s."
"Will monitor this pod instead of starting new one", labels)
final_state, result = self.monitor_launched_pod(launcher, pod_list[0])
else:
final_state, _, result = self.create_new_pod_for_operator(labels, launcher)
```
So in the first use case on master we will always attach to the pod, and on 1.10.11 we will create a new pod. I assume that 1.10.11 is the behaviour you want however. Because in master it will never ever launch a new pod when the pod has failed it will just monitor the old one and fail again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tfedyanin commented on issue #10544: Kubernetes pod operator: More than one pod running with labels
Posted by GitBox <gi...@apache.org>.
tfedyanin commented on issue #10544:
URL: https://github.com/apache/airflow/issues/10544#issuecomment-875543490
I guess it happens when I use manual retry.
There is no label `try_number=1`, that provide uniqueness between runs. So on first manual retry pod is created correctly. And there is violation of uniqueness on next manual retries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #10544: Kubernetes pod operator: More than one pod running with labels
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #10544:
URL: https://github.com/apache/airflow/issues/10544#issuecomment-679911751
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] matan129 edited a comment on issue #10544: Kubernetes pod operator: More than one pod running with labels
Posted by GitBox <gi...@apache.org>.
matan129 edited a comment on issue #10544:
URL: https://github.com/apache/airflow/issues/10544#issuecomment-822442207
Happens on `2.0.1` as well. Any updates?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tduriez-bc commented on issue #10544: Kubernetes pod operator: More than one pod running with labels
Posted by GitBox <gi...@apache.org>.
tduriez-bc commented on issue #10544:
URL: https://github.com/apache/airflow/issues/10544#issuecomment-809128075
The problem still exists on Airflow version 1.10.14+composer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on issue #10544: Kubernetes pod operator: More than one pod running with labels
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #10544:
URL: https://github.com/apache/airflow/issues/10544#issuecomment-920098055
This should be fixed by #18070.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] stijndehaes commented on issue #10544: Kubernetes pod operator: More than one pod running with labels
Posted by GitBox <gi...@apache.org>.
stijndehaes commented on issue #10544:
URL: https://github.com/apache/airflow/issues/10544#issuecomment-679923945
Looks like it is fixed by: https://github.com/apache/airflow/pull/10230
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] stijndehaes closed issue #10544: Kubernetes pod operator: More than one pod running with labels
Posted by GitBox <gi...@apache.org>.
stijndehaes closed issue #10544:
URL: https://github.com/apache/airflow/issues/10544
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jrzdudek commented on issue #10544: Kubernetes pod operator: More than one pod running with labels
Posted by GitBox <gi...@apache.org>.
jrzdudek commented on issue #10544:
URL: https://github.com/apache/airflow/issues/10544#issuecomment-729671217
We've tested this with `1.10.12` which includes the #10230 fix, and the issue still exists. When the first job errors out, the subsequent retries reattach to the errored out pod.
```
found a running pod with labels {'dag_id': 'xxxx, 'task_id': 'xxxxx', 'execution_date': '2020-09-02T1000000000-7b0e1e2af', 'try_number': '2'} but a different try_number. Will attach to this pod and monitor instead of starting new one
```
The pod is in `Error` status in the cluster so should not be considered a candidate for re-attachment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org