You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Atlaoui (via GitHub)" <gi...@apache.org> on 2023/03/08 09:10:51 UTC
[GitHub] [airflow] Atlaoui opened a new issue, #29973: Template container_resources of KubernetesPod
Atlaoui opened a new issue, #29973:
URL: https://github.com/apache/airflow/issues/29973
### What do you see as an issue?
I have DAG that uses a KubernetesPodOperator to run a containerized task. I would like to be able to parameterize the resources (memory and CPU) allocated to the container, so that I can change them depending on the specific DAG run.
### Solving the problem
I tried using Airflow's params to define the container_resources argument in the KubernetesPodOperator, but I couldn't find a way to reference the parameters in the dictionary. Here's a simplified version of my code:
```python
from airflow import DAG
from airflow.contrib.operators import KubernetesPodOperator
default_args = {
'owner': 'airflow',
'depends_on_past': False,
'start_date': datetime(2023, 3, 7),
'email': ['airflow@example.com'],
'email_on_failure': False,
'email_on_retry': False,
}
with DAG(
'my_dag',
default_args=default_args,
schedule_interval='@daily',
catchup=False,
params={
"request_memory": Param("2000", type="string"),
"request_cpu": Param("4000", type="string"),
"limit_memory": Param("4G", type="string"),
"limit_cpu": Param("5000", type="string"),
}
) as dag:
task = KubernetesPodOperator(
task_id='my_task',
name='my_task',
image='my_image:latest',
namespace='my_namespace',
image_pull_policy='Always',
is_delete_operator_pod=True,
container_resources={
'request_memory': '{{ params.request_memory }}',# tried dag.params["request_memory"],
'limit_memory': '{{ params.limit_memory }}'
'request_cpu': '{{ params.request_cpu }}',
'limit_cpu': '{{ params.limit_cpu }}',
},
)```
I also tried to use k8s.V1ResourceRequirements
```
ressources = k8s.V1ResourceRequirements(
limits={
'cpu': '{{ params.limit_cpu }}',
'memory': '{{ params.limit_memory }}'
},
requests={
'cpu': '{{ params.request_cpu }}',
'memory': '{{ params.request_memory }}'
}
)
task = KubernetesPodOperator(
container_resources=resources
)
```
### Anything else
In the [doc](https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/_api/airflow/providers/cncf/kubernetes/operators/kubernetes_pod/index.html) we can read that :
`container_resources (k8s.V1ResourceRequirements | None) – resources for the launched pod. (templated)`
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] josh-fell commented on issue #29973: Template container_resources of KubernetesPod
Posted by "josh-fell (via GitHub)" <gi...@apache.org>.
josh-fell commented on issue #29973:
URL: https://github.com/apache/airflow/issues/29973#issuecomment-1460883304
@Atlaoui Trying to understand what's going on. Is this a documentation issue or are you saying that you are unable to pass a templated arg for `container_resources`?
The `container_resources` parameter is a template field like you mentioned so your example _should_ work properly. Although, your important of the KubernetesPodOperator is quite outdated (pre-Airflow 2.0). Do you/have you tried installing the Kubernetes provider package (latest is apache-airflow-providers-cncf-kubernetes==5.1.1) and updating the operator import to `airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Atlaoui closed issue #29973: Template container_resources of KubernetesPod
Posted by "Atlaoui (via GitHub)" <gi...@apache.org>.
Atlaoui closed issue #29973: Template container_resources of KubernetesPod
URL: https://github.com/apache/airflow/issues/29973
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #29973: Template container_resources of KubernetesPod
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #29973:
URL: https://github.com/apache/airflow/issues/29973#issuecomment-1459812133
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Atlaoui commented on issue #29973: Template container_resources of KubernetesPod
Posted by "Atlaoui (via GitHub)" <gi...@apache.org>.
Atlaoui commented on issue #29973:
URL: https://github.com/apache/airflow/issues/29973#issuecomment-1461834576
Ah sorry I did not realized that I was on the wrong doc version , I was using the apache-airflow-providers-cncf-kubernetes==4.2.2, I updated to 5.2.2 and it's worked
Thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #29973: Template container_resources of KubernetesPod
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #29973:
URL: https://github.com/apache/airflow/issues/29973#issuecomment-1460942477
Something does look odd here but i'm not sure if it's related to templating as both cases it looks like wrong syntax?
```
from datetime import datetime
from kubernetes import client as k8s
from airflow import DAG
from airflow.models import Param
from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator
with DAG(
dag_id="my_kube_dag",
start_date=datetime(2023, 3, 7),
schedule_interval=None,
catchup=False,
params={
"request_memory": Param("2000", type="string"),
"request_cpu": Param("4000", type="string"),
"limit_memory": Param("4G", type="string"),
"limit_cpu": Param("5000", type="string"),
}
) as dag:
task1 = KubernetesPodOperator(
namespace="default",
image="ubuntu:16.04",
cmds=["bash", "-cx"],
arguments=["lalala"],
task_id="task1",
in_cluster=False,
do_xcom_push=False,
is_delete_operator_pod=False,
container_resources=k8s.V1ResourceRequirements(
limits={
"cpu": "1",
"memory": "1G",
},
requests={
"cpu": "1",
"memory": "1",
},
)
)
task2 = KubernetesPodOperator(
namespace="default",
image="ubuntu:16.04",
cmds=["bash", "-cx"],
arguments=["lalala"],
task_id="task2",
in_cluster=False,
do_xcom_push=False,
is_delete_operator_pod=False,
container_resources=k8s.V1ResourceRequirements(
limits={
"cpu": "{{ params.limit_cpu }}",
"memory": "{{ params.limit_memory }}",
},
requests={
"cpu": "{{ params.request_cpu }}",
"memory": "{{ params.request_memory }}",
}
)
)
```
task1:
task2:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] josh-fell commented on issue #29973: Template container_resources of KubernetesPod
Posted by "josh-fell (via GitHub)" <gi...@apache.org>.
josh-fell commented on issue #29973:
URL: https://github.com/apache/airflow/issues/29973#issuecomment-1460885327
Or is the second `container_resources` arg the issue?
```py
ressources = k8s.V1ResourceRequirements(
limits={
'cpu': '{{ params.limit_cpu }}',
'memory': '{{ params.limit_memory }}'
},
requests={
'cpu': '{{ params.request_cpu }}',
'memory': '{{ params.request_memory }}'
}
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org