You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/07/29 07:25:36 UTC
[GitHub] [airflow] SilvrDuck opened a new issue #10037: KubernetesPodOperator namespace arguments conflicts when using pod_template_file
SilvrDuck opened a new issue #10037:
URL: https://github.com/apache/airflow/issues/10037
**Apache Airflow version**: 1.10.10
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`): v1.16.11-gke.5
**Environment**:
- **Cloud provider or hardware configuration**: GKE
- **OS** (e.g. from /etc/os-release): –
- **Kernel** (e.g. `uname -a`): –
- **Install tools**: helm
- **Others**: –
**What happened**:
I tried to create a job in my DAG using the [KubernetesPodOperator](https://airflow.apache.org/docs/stable/_api/airflow/contrib/operators/kubernetes_pod_operator/index.html), and configuring it with a `.yaml` file (with the `pod_template_file` argument).
When specifying the `namespace` argument for the `KubernetesPodOperator`, I got the following error :
```
[2020-07-28 13:27:02,689] {taskinstance.py:1150} ERROR - Pod Launching failed: Cannot configure pod and pass either `pod` or `pod_template_file`. Fields ['namespace'] passed.
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py", line 291, in execute
final_state, _, result = self.create_new_pod_for_operator(labels, launcher)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py", line 338, in create_new_pod_for_operator
pod = pod_generator.PodGenerator(
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/kubernetes/pod_generator.py", line 205, in __init__
self.validate_pod_generator_args(locals())
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/kubernetes/pod_generator.py", line 553, in validate_pod_generator_args
raise AirflowConfigException(
airflow.exceptions.AirflowConfigException: Cannot configure pod and pass either `pod` or `pod_template_file`. Fields ['namespace'] passed.
```
When not specifying the argument on the other hand, I got the following error:
```
[2020-07-28 13:13:24,938] {taskinstance.py:1150} ERROR - Missing the required parameter `namespace` when calling `list_namespaced_pod`
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 984, in _run_raw_task
result = task_copy.execute(context=context)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py", line 271, in execute
pod_list = client.list_namespaced_pod(self.namespace, label_selector=label_selector)
File "/home/airflow/.local/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py", line 12803, in list_namespaced_pod
(data) = self.list_namespaced_pod_with_http_info(namespace, **kwargs) # noqa: E501
File "/home/airflow/.local/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py", line 12850, in list_namespaced_pod_with_http_info
raise ValueError("Missing the required parameter `namespace` when calling `list_namespaced_pod`") # noqa: E501
ValueError: Missing the required parameter `namespace` when calling `list_namespaced_pod`
```
(note that this error comes from the k8s python api, not from airflow)
Hence a contradiction between the two messages.
**What you expected to happen**:
I’d expect to see the job being executed on the cluster. The namespace would be specified in the `.yaml` file (or as a `KubernetesPodOperator` argument, both would be fine if working).
**How to reproduce it**:
I tried to create a minimal example to show what I’m talking about.
I have `minimal.py`:
```
from datetime import datetime, timedelta
from airflow.models import DAG
from airflow.contrib.operators.kubernetes_pod_operator import KubernetesPodOperator
default_args = {
"owner": "test",
"start_date": datetime(2019, 10, 17),
"retries": 1,
"retry_delay": timedelta(minutes=5),
}
with DAG("bug-report", schedule_interval=None, default_args=default_args) as dag:
KubernetesPodOperator(
task_id="bug-report-task",
is_delete_operator_pod=False,
in_cluster=True,
get_logs=True,
namespace="test", # this is the part causing troubles
pod_template_file="path/to/dags/minimal.yaml",
)
```
And `minimal.yaml` which is some random example job from [k8s' doc](https://kubernetes.io/docs/concepts/workloads/controllers/job/):
```
apiVersion: batch/v1
kind: Job
metadata:
namespace: test
name: bug-report
spec:
template:
spec:
containers:
- name: pi
image: perl
command: ["perl", "-Mbignum=bpi", "-wle", "print bpi(2000)"]
restartPolicy: Never
backoffLimit: 4
```
Triggering the `DAG`, I get the aforementioned errors.
**Is that an actual bug or is there something I’m missing?**
Best,
Thibault
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] leoblanc commented on issue #10037: KubernetesPodOperator namespace argument conflict when using pod_template_file
Posted by GitBox <gi...@apache.org>.
leoblanc commented on issue #10037:
URL: https://github.com/apache/airflow/issues/10037#issuecomment-705660760
Hi, I'm hitting the same bug. It seems like the docker image for 1.10.13 wasn't created yet, right? (https://hub.docker.com/r/apache/airflow/tags)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Minyus edited a comment on issue #10037: KubernetesPodOperator namespace argument conflict when using pod_template_file
Posted by GitBox <gi...@apache.org>.
Minyus edited a comment on issue #10037:
URL: https://github.com/apache/airflow/issues/10037#issuecomment-671041151
I'm trying to use `pod_template_file` with `in_cluster=True` using Airflow 1.10.11 (Docker image at [apache/airflow:1.10.11-python3.6](https://hub.docker.com/layers/apache/airflow/1.10.11-python3.6/images/sha256-5b43a4b820eb229ea6910aafcddc2d18de745a69817c0256cadc74f6faa2d3f5?context=explore)) and getting the same symptom.
I checked the [test_kubernetes_pod_operator.py](https://github.com/apache/airflow/blob/1.10.11/kubernetes_tests/test_kubernetes_pod_operator.py), but apparently the cases of `in_cluster=True` was not tested well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman commented on issue #10037: KubernetesPodOperator namespace argument conflict when using pod_template_file
Posted by GitBox <gi...@apache.org>.
dimberman commented on issue #10037:
URL: https://github.com/apache/airflow/issues/10037#issuecomment-692887069
@Minyus can you please post the error for the in_cluster issue? I think that might be separate
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman commented on issue #10037: KubernetesPodOperator namespace argument conflict when using pod_template_file
Posted by GitBox <gi...@apache.org>.
dimberman commented on issue #10037:
URL: https://github.com/apache/airflow/issues/10037#issuecomment-692883542
Ahh but ok, I see the other issue where if you don't have a namespace param it can't track the pod. Yes that's a critical bugfix.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] leoblanc commented on issue #10037: KubernetesPodOperator namespace argument conflict when using pod_template_file
Posted by GitBox <gi...@apache.org>.
leoblanc commented on issue #10037:
URL: https://github.com/apache/airflow/issues/10037#issuecomment-705660760
Hi, I'm hitting the same bug. It seems like the docker image for 1.10.13 wasn't created yet, right? (https://hub.docker.com/r/apache/airflow/tags)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman removed a comment on issue #10037: KubernetesPodOperator namespace argument conflict when using pod_template_file
Posted by GitBox <gi...@apache.org>.
dimberman removed a comment on issue #10037:
URL: https://github.com/apache/airflow/issues/10037#issuecomment-692882853
Hmm this is an interesting one.
I think when we made the pod_template_file we did so with the assumption that users would have ALL specs in the pod_template file and not attempt to modify further. If you have a pod_template_file that you want to modify before launching, you can deserialize it using the PodGenerator and then use the `pod` parameter for the K8sPodOperator.
Something like this
```
from datetime import datetime, timedelta
from airflow.kubernetes.pod_generator import PodGenerator
from airflow.models import DAG
from airflow.contrib.operators.kubernetes_pod_operator import KubernetesPodOperator
default_args = {
"owner": "test",
"start_date": datetime(2019, 10, 17),
"retries": 1,
"retry_delay": timedelta(minutes=5),
}
pod = PodGenerator.deserialize_model_file("path/to/dags/minimal.yaml")
pod.metadata.namespace="test"
with DAG("bug-report", schedule_interval=None, default_args=default_args) as dag:
KubernetesPodOperator(
task_id="bug-report-task",
is_delete_operator_pod=False,
in_cluster=True,
get_logs=True,
pod=pod,
)
```
If we documented this use-case better, would this be a reasonable solution to the problem?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman commented on issue #10037: KubernetesPodOperator namespace argument conflict when using pod_template_file
Posted by GitBox <gi...@apache.org>.
dimberman commented on issue #10037:
URL: https://github.com/apache/airflow/issues/10037#issuecomment-692966901
https://github.com/apache/airflow/pull/10963 PR for a fix. This should go out in 1.10.13.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman commented on issue #10037: KubernetesPodOperator namespace argument conflict when using pod_template_file
Posted by GitBox <gi...@apache.org>.
dimberman commented on issue #10037:
URL: https://github.com/apache/airflow/issues/10037#issuecomment-692882853
Hmm this is an interesting one.
I think when we made the pod_template_file we did so with the assumption that users would have ALL specs in the pod_template file and not attempt to modify further. If you have a pod_template_file that you want to modify before launching, you can deserialize it using the PodGenerator and then use the `pod` parameter for the K8sPodOperator.
Something like this
```
from datetime import datetime, timedelta
from airflow.kubernetes.pod_generator import PodGenerator
from airflow.models import DAG
from airflow.contrib.operators.kubernetes_pod_operator import KubernetesPodOperator
default_args = {
"owner": "test",
"start_date": datetime(2019, 10, 17),
"retries": 1,
"retry_delay": timedelta(minutes=5),
}
pod = PodGenerator.deserialize_model_file("path/to/dags/minimal.yaml")
pod.metadata.namespace="test"
with DAG("bug-report", schedule_interval=None, default_args=default_args) as dag:
KubernetesPodOperator(
task_id="bug-report-task",
is_delete_operator_pod=False,
in_cluster=True,
get_logs=True,
pod=pod,
)
```
If we documented this use-case better, would this be a reasonable solution to the problem?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman commented on issue #10037: KubernetesPodOperator namespace argument conflict when using pod_template_file
Posted by GitBox <gi...@apache.org>.
dimberman commented on issue #10037:
URL: https://github.com/apache/airflow/issues/10037#issuecomment-692967644
Also @SilvrDuck heads up that template won't work. The K8sPodOperator only takes a pod as a template (it would be bad to have two levels of retry logic so we explicitly do not allow jobs). Also the container would need to be named "base" for logs to work.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman closed issue #10037: KubernetesPodOperator namespace argument conflict when using pod_template_file
Posted by GitBox <gi...@apache.org>.
dimberman closed issue #10037:
URL: https://github.com/apache/airflow/issues/10037
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman commented on issue #10037: KubernetesPodOperator namespace argument conflict when using pod_template_file
Posted by GitBox <gi...@apache.org>.
dimberman commented on issue #10037:
URL: https://github.com/apache/airflow/issues/10037#issuecomment-695804727
@SilvrDuck PR has been cherry-picked to release branch and should go out with 1.10.13. Thank you again!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] shonik09 commented on issue #10037: KubernetesPodOperator namespace argument conflict when using pod_template_file
Posted by GitBox <gi...@apache.org>.
shonik09 commented on issue #10037:
URL: https://github.com/apache/airflow/issues/10037#issuecomment-734494647
Can someone clarify if this made it into 1.10.13? If not, when is this expected to be fixed?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #10037: KubernetesPodOperator namespace arguments conflicts when using pod_template_file
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #10037:
URL: https://github.com/apache/airflow/issues/10037#issuecomment-665075120
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman closed issue #10037: KubernetesPodOperator namespace argument conflict when using pod_template_file
Posted by GitBox <gi...@apache.org>.
dimberman closed issue #10037:
URL: https://github.com/apache/airflow/issues/10037
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Minyus edited a comment on issue #10037: KubernetesPodOperator namespace argument conflict when using pod_template_file
Posted by GitBox <gi...@apache.org>.
Minyus edited a comment on issue #10037:
URL: https://github.com/apache/airflow/issues/10037#issuecomment-671041151
I'm trying to use `pod_template_file` with `in_cluster=True` using Airflow 1.10.11 (Docker image at [apache/airflow:1.10.11-python3.6](https://hub.docker.com/layers/apache/airflow/1.10.11-python3.6/images/sha256-5b43a4b820eb229ea6910aafcddc2d18de745a69817c0256cadc74f6faa2d3f5?context=explore)) and getting the same symptom.
I checked the [test_kubernetes_pod_operator.py](https://github.com/apache/airflow/blob/1.10.11/kubernetes_tests/test_kubernetes_pod_operator.py), but apparently the cases of `in_cluster=True` were not tested well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Minyus commented on issue #10037: KubernetesPodOperator namespace argument conflict when using pod_template_file
Posted by GitBox <gi...@apache.org>.
Minyus commented on issue #10037:
URL: https://github.com/apache/airflow/issues/10037#issuecomment-671041151
I'm trying to use `pod_template_file` with `in_cluster=True` using Airflow 1.10.11 (Docker image from [apache/airflow:1.10.11-python3.6](https://hub.docker.com/layers/apache/airflow/1.10.11-python3.6/images/sha256-5b43a4b820eb229ea6910aafcddc2d18de745a69817c0256cadc74f6faa2d3f5?context=explore)) and getting the same symptom.
I checked the [test_kubernetes_pod_operator.py](https://github.com/apache/airflow/blob/1.10.11/kubernetes_tests/test_kubernetes_pod_operator.py), but apparently the cases of `in_cluster=True` was not tested well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman commented on issue #10037: KubernetesPodOperator namespace argument conflict when using pod_template_file
Posted by GitBox <gi...@apache.org>.
dimberman commented on issue #10037:
URL: https://github.com/apache/airflow/issues/10037#issuecomment-695804727
@SilvrDuck PR has been cherry-picked to release branch and should go out with 1.10.13. Thank you again!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org