You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/05/22 08:53:56 UTC
[GitHub] [airflow] fuyi opened a new issue #8965: use separate pod_mutation_hook for Kubernetes Executor pod
fuyi opened a new issue #8965:
URL: https://github.com/apache/airflow/issues/8965
**Description**
<!-- A short description of your feature -->
According to the [Kubernetes Executor doc](https://airflow.apache.org/docs/stable/kubernetes.html), pod_mutation_hook is used to mutate pod object before sending to the Kubernetes client for scheduling.
We deploy Airflow in K8S environment and use both Kubernetes Executor and KubernetesPodOperator. We need to set almost completely different set of fields for Executor pod and KubernetesPodOperator pod.
Since `pod_mutation_hook` is applied to all Pod creation requests, we have to check which type of pod in the function then add a different set of K8S fields.
Pseudo code:
```python
def pod_mutation_hook:
if pod.type == 'kubernetes executor workers pod':
pod.annotations['pod-type'] = 'kubernetes-executor-worker-pod'
if pod.type == 'PodOperator pod':
pod.annotations['pod-type'] = 'pod-operator-pod'
```
The issues:
* It is not clear what fields are applied to what Pod type
* This is not good practice for separate of concerns.
**Use case / motivation**
I propose we keep `pod_mutation_hook` for mutating PodOperator pod and use a separate hook like `executor_pod_mutation_hook` to mutate Kuberenetes Executor pod.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] j-y-matsubara edited a comment on issue #8965: use separate pod_mutation_hook for Kubernetes Executor pod
Posted by GitBox <gi...@apache.org>.
j-y-matsubara edited a comment on issue #8965:
URL: https://github.com/apache/airflow/issues/8965#issuecomment-640354651
Can't make a decision using the label?
For example,
Pseudo code:
```
def pod_mutation_hook:
if pod.labels['kubernetes_pod_operator']== 'True':
pod.annotations['pod-type'] = 'kubernetes-executor-worker-pod'
else:
pod.annotations['pod-type'] = 'pod-operator-pod'
```
The pod object have info like labels: ` {'airflow_version': '1.10.10', 'kubernetes_pod_operator': 'True'} `
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] j-y-matsubara commented on issue #8965: use separate pod_mutation_hook for Kubernetes Executor pod
Posted by GitBox <gi...@apache.org>.
j-y-matsubara commented on issue #8965:
URL: https://github.com/apache/airflow/issues/8965#issuecomment-640354651
Can't make a decision using the label?
For example,
Pseudo code:
```
def pod_mutation_hook:
if pod.labels['kubernetes_pod_operator']== 'True':
pod.annotations['pod-type'] = 'kubernetes-executor-worker-pod'
else:
pod.annotations['pod-type'] = 'pod-operator-pod'
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman commented on issue #8965: use separate pod_mutation_hook for Kubernetes Executor pod
Posted by GitBox <gi...@apache.org>.
dimberman commented on issue #8965:
URL: https://github.com/apache/airflow/issues/8965#issuecomment-702430531
@fuyi it's worth noting that there is a label injected by all KubernetesExecutor pods. Something like `is_kubernetes_executor_pod: true`. You can filter by this. I think creating a seperate hook might create some confusion. I'm going to close this for now, though if you want to propose it to the mailing list would be interested in discussing further with the larger community.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] j-y-matsubara edited a comment on issue #8965: use separate pod_mutation_hook for Kubernetes Executor pod
Posted by GitBox <gi...@apache.org>.
j-y-matsubara edited a comment on issue #8965:
URL: https://github.com/apache/airflow/issues/8965#issuecomment-640354651
Can't make a decision using the label?
For example,
Pseudo code:
```
def pod_mutation_hook:
if pod.labels['kubernetes_pod_operator']== 'True':
pod.annotations['pod-type'] = 'kubernetes-executor-worker-pod'
else:
pod.annotations['pod-type'] = 'pod-operator-pod'
```
The pod object have info like this:
labels: {'airflow_version': '1.10.10', 'kubernetes_pod_operator': 'True'}
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman closed issue #8965: use separate pod_mutation_hook for Kubernetes Executor pod
Posted by GitBox <gi...@apache.org>.
dimberman closed issue #8965:
URL: https://github.com/apache/airflow/issues/8965
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] fuyi commented on issue #8965: use separate pod_mutation_hook for Kubernetes Executor pod
Posted by GitBox <gi...@apache.org>.
fuyi commented on issue #8965:
URL: https://github.com/apache/airflow/issues/8965#issuecomment-643134458
Yes, this works, I am currently using this approach to distinguish POd types.
But would it be better to use a separate hook for separate types of Pod to separate the concern?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org