You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/08/11 10:50:42 UTC
[GitHub] [airflow] liorhar opened a new issue #10290: k8 executor ignores namespace configuration from task's executor_config
liorhar opened a new issue #10290:
URL: https://github.com/apache/airflow/issues/10290
**Apache Airflow version**:
1.10.10
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`):
```
Client Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.6-beta.0", GitCommit:"e7f962ba86f4ce7033828210ca3556393c377bcc", GitTreeState:"clean", BuildDate:"2020-01-15T08:26:26Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.6-eks-4e7f64", GitCommit:"4e7f642f9f4cbb3c39a4fc6ee84fe341a8ade94c", GitTreeState:"clean", BuildDate:"2020-06-11T13:55:35Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
```
**Environment**:
- **Cloud provider or hardware configuration**: AWS EKS
- **OS** (e.g. from /etc/os-release): official airflow docker image
- **Kernel** (e.g. `uname -a`):
- **Install tools**:
- **Others**:
**What happened**:
Airflow ignores the namespace provided in the executor_config of the task. Our use case is a centralized airflow running in one namespace scheduling pods in multiple other namespaces based on configuration.
Running the following snippet from the [example](https://github.com/apache/airflow/blob/master/airflow/example_dags/example_kubernetes_executor_config.py#L97):
```
other_ns_task = PythonOperator(
task_id="other_namespace_task",
python_callable=print_stuff,
executor_config={
"KubernetesExecutor": {
"namespace": "test-namespace",
"labels": {
"release": "stable"
}
}
}
)
```
**What you expected to happen**:
The scheduler should create the pod in the namespace as provided in the task's configuration.
I believe the problem is in `pod_generator.py` where the executor_config is overridden by the airflow.cfg kubernetes namespace configuration.
```
# Reconcile the pods starting with the first chronologically,
# Pod from the airflow.cfg -> Pod from executor_config arg -> Pod from the K8s executor
pod_list = [worker_config, kube_executor_config, dynamic_pod]
```
`dynamic_pod.namespace` is initialized with the same value as `worker_config.namespace` thus overriding the provided `kube_executor_config`
**How to reproduce it**:
configure `AIRFLOW__KUBERNETES__NAMESPACE` (or the value in `airflow.cfg`) to one namespace, then try to run a task on another namespace.
```
executor_config={
"KubernetesExecutor": {
"namespace": "test-namespace"
}
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #10290: k8 executor ignores namespace configuration from task's executor_config
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #10290:
URL: https://github.com/apache/airflow/issues/10290#issuecomment-982146093
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] johngtam edited a comment on issue #10290: k8 executor ignores namespace configuration from task's executor_config
Posted by GitBox <gi...@apache.org>.
johngtam edited a comment on issue #10290:
URL: https://github.com/apache/airflow/issues/10290#issuecomment-1062404161
@eladkal, I'm not OP, but I can confirm this still happens on 2.0.2 -- looking at more of the codeflow:
```
dynamic_pod = k8s.V1Pod(
metadata=k8s.V1ObjectMeta(
namespace=namespace,
annotations=annotations,
name=PodGenerator.make_unique_pod_id(pod_id),
labels=labels,
),
spec=k8s.V1PodSpec(
containers=[
k8s.V1Container(
name="base",
args=args,
image=image,
env=[k8s.V1EnvVar(name="AIRFLOW_IS_K8S_EXECUTOR_POD", value="True")],
)
]
),
)
# Reconcile the pods starting with the first chronologically,
# Pod from the pod_template_File -> Pod from executor_config arg -> Pod from the K8s executor
pod_list = [base_worker_pod, pod_override_object, dynamic_pod]
return reduce(PodGenerator.reconcile_pods, pod_list)
```
I see here that dynamic pod takes precedence. This dynamic pod gets its namespace from the caller of this method. Looking at the callers of this method [here](https://github.com/apache/airflow/blob/ac77c89018604a96ea4f5fba938f2fbd7c582793/airflow/executors/kubernetes_executor.py#L314), [here](https://github.com/apache/airflow/blob/5ace37a16d1773adb71c684450838e4c8e69b581/airflow/models/taskinstance.py#L2101), and [here](https://github.com/apache/airflow/blob/dc1fd89a5b95a2424737371cc2d31edea3030247/airflow/cli/commands/kubernetes_command.py#L47), I search through the code to find out where the namespace is coming from.
If it's coming from `kube_config.executor_namespace`, I can see where it's coming from [here](https://github.com/apache/airflow/blob/5ace37a16d1773adb71c684450838e4c8e69b581/airflow/kubernetes/kube_config.py#L53). Therefore, going from the code block above, dynamic_pod's namespace takes precedence, and that is determined from the config's namespace.
Therefore, dynamic_pod will ultimately override namespace without regard to `pod_override`, because there can only be one namespace. Going to explore upgrading Airflow on my own (but currently going with a workaround before then), but it doesn't seem like this particular code flow has changed much -- might be worth verifying on part of the Airflow maintainers?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #10290: k8 executor ignores namespace configuration from task's executor_config
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #10290:
URL: https://github.com/apache/airflow/issues/10290#issuecomment-968396019
This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] closed issue #10290: k8 executor ignores namespace configuration from task's executor_config
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed issue #10290:
URL: https://github.com/apache/airflow/issues/10290
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #10290: k8 executor ignores namespace configuration from task's executor_config
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #10290:
URL: https://github.com/apache/airflow/issues/10290#issuecomment-943130783
@liorhar can you please check if this issue still happens in latest Airflow version?
Kubernetes has been refactor significantly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] johngtam commented on issue #10290: k8 executor ignores namespace configuration from task's executor_config
Posted by GitBox <gi...@apache.org>.
johngtam commented on issue #10290:
URL: https://github.com/apache/airflow/issues/10290#issuecomment-1062404161
@eladkal, I'm not OP, but I can confirm this still happens -- looking at more of the codeflow:
```
dynamic_pod = k8s.V1Pod(
metadata=k8s.V1ObjectMeta(
namespace=namespace,
annotations=annotations,
name=PodGenerator.make_unique_pod_id(pod_id),
labels=labels,
),
spec=k8s.V1PodSpec(
containers=[
k8s.V1Container(
name="base",
args=args,
image=image,
env=[k8s.V1EnvVar(name="AIRFLOW_IS_K8S_EXECUTOR_POD", value="True")],
)
]
),
)
# Reconcile the pods starting with the first chronologically,
# Pod from the pod_template_File -> Pod from executor_config arg -> Pod from the K8s executor
pod_list = [base_worker_pod, pod_override_object, dynamic_pod]
return reduce(PodGenerator.reconcile_pods, pod_list)
```
I see here that dynamic pod takes precedence. This dynamic pod gets its namespace from the caller of this method. Looking at the callers of this method [here](https://github.com/apache/airflow/blob/ac77c89018604a96ea4f5fba938f2fbd7c582793/airflow/executors/kubernetes_executor.py#L314), [here](https://github.com/apache/airflow/blob/5ace37a16d1773adb71c684450838e4c8e69b581/airflow/models/taskinstance.py#L2101), and [here](https://github.com/apache/airflow/blob/dc1fd89a5b95a2424737371cc2d31edea3030247/airflow/cli/commands/kubernetes_command.py#L47), I search through the code to find out where the namespace is coming from.
If it's coming from `kube_config.executor_namespace`, I can see where it's coming from [here](https://github.com/apache/airflow/blob/5ace37a16d1773adb71c684450838e4c8e69b581/airflow/kubernetes/kube_config.py#L53). Therefore, going from the code block above, dynamic_pod's namespace takes precedence, and that is determined from the config's namespace.
Therefore, dynamic_pod will ultimately override namespace without regard to `pod_override`, because there can only be one namespace.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] DerekHeldtWerle commented on issue #10290: k8 executor ignores namespace configuration from task's executor_config
Posted by GitBox <gi...@apache.org>.
DerekHeldtWerle commented on issue #10290:
URL: https://github.com/apache/airflow/issues/10290#issuecomment-799860031
Running into this as well @liorhar, did you end up doing anything to resolve it outside of changing the order?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #10290: k8 executor ignores namespace configuration from task's executor_config
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #10290:
URL: https://github.com/apache/airflow/issues/10290#issuecomment-671874897
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org